Accounts

This page keeps one logged-in account the way a browser profile keeps it: the same browser, platform, languages, proxy, cookies, and connection state on every run. A Device holds all of them in one file. Device::autosave keeps the file current, a Tab keeps the current page, and leyline::html reads the login form.

What a device holds

Create the device once

Pin a browser variant, set the languages and one proxy, and turn the proxy environment variables off. Then capture the device and freeze its profile.

#![allow(unused)]
fn main() {
use leyline::{Browser, Device, Platform, ProxyConfig, ProxyUrl, Session};

fn run() -> leyline::Result<()> {
let proxy = ProxyUrl::parse("http://user:secret@proxy.example:8080")?;
let session = Session::builder()
    .browser(Browser::Chrome154)
    .platform(Platform::Windows)
    .languages(["de-DE", "de", "en"])
    .proxy(ProxyConfig::from(proxy.clone()).env(false))
    .build()?;

let mut device = Device::capture(&session, Some(proxy));
device.pin_profile(&session)?;
device.strict = true;
device.jar_path = Some("account/cookies.json".into());
device.save_to("account/device.json")?;
Ok(())
}
}

Session::proxy_url() returns the configured proxy, password included, so Device::capture(&session, session.proxy_url()) captures it too. check compares the configured URL, not the exit IP the proxy assigns: a rotating proxy behind one URL passes with a new IP. To pin the exit IP, fetch it from your own IP echo service and keep it in Device::app.

Keep the proxy password out of the file

ProxyUrl serializes the full URL, password included. To keep the password out of the device file, set proxy_password_env to the name of an environment variable. save_to and Device::autosave then write the proxy URL without its password, and open and session_builder read it back from the variable. With proxy_password_env set, serializing the Device never writes the password. When the variable is not set in the environment, open and session_builder return Kind::Config with a message that names it.

#![allow(unused)]
fn main() {
use leyline::{Browser, Device, Platform, ProxyConfig, ProxyUrl, Session, Url};

const PROXY: &str = "http://user@proxy.example:8080";
const PASSWORD_VAR: &str = "ACCOUNT_PROXY_PASSWORD";

async fn run() -> Result<(), Box<dyn std::error::Error>> {
let mut url = Url::parse(PROXY)?;
url.set_password(Some(&std::env::var(PASSWORD_VAR)?))
    .map_err(|()| "the proxy URL cannot carry a password")?;
let proxy = ProxyUrl::parse(url.as_str())?;

let session = Session::builder()
    .browser(Browser::Chrome154)
    .platform(Platform::Windows)
    .proxy(ProxyConfig::from(proxy).env(false))
    .build()?;
session.get("https://shop.example/login").await?;

let mut device = Device::capture(&session, session.proxy_url());
device.proxy_password_env = Some(PASSWORD_VAR.to_owned());
device.pin_profile(&session)?;
device.save_to("account/device.json")?;
Ok(())
}
}

Open the device on each run

Device::open builds the session, runs check, loads the cookies, and restores the saved connection state. A jar_path whose file does not exist yet gives an empty jar. A device with no jar_path uses the jar in the device file, or an empty jar. Then start Device::autosave and call its shutdown before the process exits.

#![allow(unused)]
fn main() {
use std::time::Duration;
use leyline::Device;

async fn run() -> leyline::Result<()> {
let device = Device::load_from("account/device.json")?;
let session = device.open()?;
let tab = device.tab(&session);
let autosave = device.autosave(&session, "account/device.json", Duration::from_secs(2));
autosave.track(&tab);

tab.follow("/account/orders").await?;
autosave.shutdown().await?;
Ok(())
}
}

To add settings before the build, call device.session_builder()?, change the builder, build the session, then call device.check(&session)? and device.state.restore_into(&session).

Saved TLS sessions are bound to the trust that the session loaded when it was built: the pins, the contents of every root it loaded from add_ca_der, add_ca_file, SSL_CERT_FILE, and SSL_CERT_DIR, the contents of the client certificate and key files, and whether system roots are on. A saved TLS session resumes only in a session that loaded the same trust. Otherwise the connection makes a full handshake and checks the certificate again. Session::fresh_pool keeps the trust of the session it copies. Changes to the operating system’s root store are not part of the binding. The saved state contains TLS session secrets, so store it with the same protection as credentials.

check returns Kind::Config and names each field that differs:

Error::is_profile_changed() is true when the profile differs: the profile_id, or, without one, the identity, platform, brand, or user agent. It is false when only the languages, the proxy, or a strict gap differ.

Save the device as it changes

Device::autosave(&session, path, options) starts one writer task and returns a DeviceAutosave handle. options is a DeviceAutosaveOptions, or a Duration that sets its interval:

Each save writes the device file with the session’s current SessionState. With a jar_path, it also writes the jar file when a cookie changed; without one, the jar goes into the device file. For a jar without a device, use Jar::autosave; see Cookies.

Each save writes a temporary file, syncs it, and renames it, so a crash never leaves a half-written file. On Unix the files have mode 0600; on Windows they take the default ACLs of their directory. autosave panics outside a Tokio runtime.

The writer keeps its own copy of the device. Change it with DeviceAutosave::update(|device| ..); the change reaches the file with the next save. A change to the Device you called autosave on does not. DeviceAutosave::device() returns a snapshot of the writer’s copy.

DeviceAutosave::track(&tab) stores the tab’s current page in Device::page on each save. The writer keeps one tab: a second track replaces the first. On the next run, Device::tab(&session) returns a tab on that page, so the next request carries it as its Referer.

Keep your own data

Device::app holds your own data in the same atomic save, such as the account email or values a site keeps in localStorage. Leyline does not read it.

#![allow(unused)]
fn main() {
use leyline::Device;
use serde_json::json;

fn run(mut device: Device) -> leyline::Result<()> {
device.app.insert("email".into(), json!("me@example.com"));
device.app.insert("local_storage".into(), json!({ "cart_id": "c-1842" }));
let email = device.app.get("email").and_then(|v| v.as_str());
let _ = email;
device.save_to("account/device.json")?;
Ok(())
}
}

Leyline keeps no HTTP cache, localStorage, sessionStorage, IndexedDB, or service workers, because it runs no scripts. Send if-none-match or if-modified-since yourself.

Keep the connection state

Session::state() returns a SessionState, which implements Serialize and Deserialize. state.restore_into(&session) loads it, and is_empty() is true when there is nothing to save.

Device::autosave refreshes device.state on each save. Before a manual save_to, set device.state = session.state(). A ticket key holds the proxy URL without its user name and password, and a hash of them, so the saved state never holds proxy credentials.

Browse with a tab

A Tab sends each request with the current page as its initiator, so the Referer, Origin, and sec-fetch-site headers match a real click. Requests lists the calls. For an account:

• xhr, fetch, post_json, and subresource on a tab with no page fail with Kind::Request (“the tab has no page; open one first”) and send nothing. Open a page first.
• A .stream() navigation changes the page at the response head. current() reads the page, and set_current(..) sets it.
• A page script sets its own accept. Set it on the xhr builder when the site’s script does. See Fingerprints.

#![allow(unused)]
fn main() {
use leyline::Preset;

async fn run(session: leyline::Session) -> leyline::Result<()> {
let tab = session.tab();
tab.open("https://shop.example/account").await?;
let orders = tab
    .xhr("/api/orders?page=1")
    .header("accept", "application/json")
    .send()
    .await?;
let logo = tab.subresource("/static/logo.png", Preset::Image).send().await?;
println!("{} {}", orders.status(), logo.status());
Ok(())
}
}

Log in with the page’s form

The html feature, on by default, adds leyline::html and turns on the multipart feature. html::forms(document) returns every <form> as an html::Form with the values a browser would submit:

• Hidden inputs, checked checkboxes, text areas, and selected options.
• One checked radio button per name: the last one, as a browser keeps it.
• Controls outside the <form> element that name it with form="id". When two elements share the id, the first one is the form.
• No disabled controls, and no controls inside a disabled <fieldset> except in the first <legend> that is a direct child of that fieldset.
• No options inside a disabled <optgroup>. A list box (<select size>) with no selected option sends nothing.
• Line breaks in names and values converted to CRLF, as a browser sends them.
• Values with every HTML named character reference decoded, such as & and é.

Form::find(document, key) returns the first form whose id or name is key.

Tab::follow and Tab::submit_form drop the session credentials when the target origin differs from the current page: the Authorization, Cookie, and Proxy-Authorization defaults and the session bearer token, as a redirect does.

Tab::submit_form(&form) sends form.fields() as they are, with no submit button: call form.press(name) first when the site reads the button. The action resolves against the page’s <base href>, or the current page when there is none. A Post form sends the fields with Preset::FormNavigate, encoded as its enctype() says: application/x-www-form-urlencoded, multipart/form-data, or text/plain. A Get form replaces the action’s query with the fields. Both are navigations that carry the page as the Referer and change the current page.

The session redirect policy applies: after a login POST, a 301, 302, or 303 continues as a GET without the body, and a 307 or 308 repeats the POST. The response and the current page are those of the final URL.

#![allow(unused)]
fn main() {
use leyline::html::{self, Form};

async fn run(tab: leyline::Tab) -> leyline::Result<()> {
let page = tab.open("https://shop.example/login").await?.text().await?;
let token = html::meta(&page, "csrf-token").unwrap_or_default();
if let Some(mut form) = Form::find(&page, "login") {
    form.set("email", "me@example.com").set("password", "secret");
    form.press("sign_in");
    let resp = tab.submit_form(&form).await?;
    println!("{} at {}", resp.status(), resp.url());
}
let cart = tab.xhr("/api/cart").header("x-csrf-token", token).send().await?;
println!("{}", cart.status());
Ok(())
}
}

html::meta(document, name) returns the content of the first <meta> whose name or property equals name, without case; many sites put a CSRF token there. html::links(document) returns every <a href> as an html::Anchor with href(), text(), and rel(). href() is the raw attribute; pass it to Tab::follow, which resolves it.

Check the login, log in again

Leyline cannot tell whether a site considers you logged in. Request an endpoint that only a logged-in user can read, and log in again when it says no. The jar and Device::autosave keep the new cookies.

#![allow(unused)]
fn main() {
use leyline::html::Form;
use leyline::http::StatusCode;

async fn run(tab: leyline::Tab) -> leyline::Result<()> {
let me = tab.xhr("https://shop.example/api/me").send().await?;
if me.status() == StatusCode::UNAUTHORIZED {
    let page = tab.open("https://shop.example/login").await?.text().await?;
    if let Some(mut form) = Form::find(&page, "login") {
        form.set("email", "me@example.com").set("password", "secret");
        tab.submit_form(&form).await?;
    }
}
Ok(())
}
}

When several tasks share the session, each can see the 401 at once. Hold one lock, such as a tokio::sync::Mutex, around the check and the login, and check again after you take it, so one task logs in and the others reuse its cookies. examples/device.rs combines the device, autosave, the login form, and the tab.

Run tasks at the same time

Session and Tab are Clone, Send, and Sync. Session clones share the pool, the jar, the identity, and the state, so one Device::autosave covers every task. Tab clones share the current page, so give each parallel page its own session.tab().

#![allow(unused)]
fn main() {
async fn run(session: leyline::Session) -> leyline::Result<()> {
let orders = session.tab();
let messages = session.tab();
let (a, b) = tokio::join!(
    orders.open("https://shop.example/account/orders"),
    messages.open("https://shop.example/account/messages"),
);
println!("{} {}", a?.status(), b?.status());
Ok(())
}
}

To stop every task of the account, call session.shutdown(), then autosave.shutdown().await for the last save. See Sessions.

Next

Read Proxies.