Leyline user guide

Leyline is an HTTP client that mimics browsers on the wire. This guide shows you how to use it, in reading order.

Leyline has two kinds of session:

Session::browser(Browser::default()) impersonates the newest bundled Chrome on Windows. Session::new() and a builder with no browser build a plain client.

1. Quick start: add the dependency, send your first request.
2. Sessions: pick a browser, a platform, and a brand.
3. Choosing a profile: which browser and platform to claim.
4. Requests: methods, headers, bodies, presets, tabs, and downloads.
5. Responses: status, headers, Link pagination, block detection, bodies, timing, audit.
6. Streaming: stream a body in either direction, download a file, or stop a read early.
7. Redirects: the redirect limit and custom redirect policies.
8. Retries and timeouts: retry policies and the four timeouts.
9. Errors: group errors by category, map them to statuses, and find out which errors a retry covers.
10. Proxies: proxy rules, bypass lists, environment discovery.
11. Cookies: the jar, saving it, devices, and connection state.
12. WebSocket: connect over HTTP/2 or HTTP/1.1.
13. HTTP/3: the http3 feature and the protocol policy.
14. TLS trust: roots, certificate pins, client certificates, the TLS version floor.
15. Network: DNS overrides, Happy Eyeballs, socket options.
16. Logging and tracing: tracing targets and request phases.
17. Fingerprints: what a profile pins and how to audit it.
18. Features and targets: cargo features, targets, MSRV.
19. Supported platforms: the six build targets and the tools each one needs.
20. Cancellation: what happens when you drop a request or a body.
21. Testing: local plain and TLS servers, and asserting what was sent.
22. Service integration: Tower, shared sessions, and error mapping in a server.
23. Crawling: host limits, proxy pools, block detection, and tags.
24. Accounts: tabs, saved devices, and the state of a returning browser.

Reference pages beside this guide: profiles and MSRV.

The Rust blocks in these pages are doctests of the leyline-http crate. The Tower block in Requests compiles only with the tower feature. A plain cargo test does not enable it. The pull-request check in CI runs cargo test with --features leyline-http/full,leyline-http/bench-internals, which does. The toml, sh, and js blocks are not compiled. Blocks that open a socket are marked no_run, so they compile but do not send traffic.

Quick start

This chapter takes you from an empty project to a request and a parsed response. Each section links to the chapter that owns the topic.

Add the dependency

The package is leyline-http. The library it builds is leyline, so that is the name you import. Leyline is async and runs on Tokio.

[dependencies]
leyline-http = "0.1"
tokio = { version = "1", features = ["full"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"

serde and serde_json are only for the JSON examples. The first build compiles BoringSSL, so it needs a C and C++ compiler and Git. Supported platforms lists the targets and the tools.

Send a GET request

leyline::get sends one GET from a plain session. For more than one request, build a Session and reuse it: it holds the connection pool and the cookie jar.

#[tokio::main]
async fn main() -> leyline::Result<()> {
    let resp = leyline::get("https://example.com/").await?;
    println!("{} {}", resp.status(), resp.url());
    Ok(())
}

Session::get and its siblings return a RequestBuilder. Awaiting the builder sends the request. .send() does the same and returns a future you can hold in a variable.

Choose a plain or a browser session

Use a plain session for APIs. Use a browser session when the server checks that the client is a browser. SessionBuilder::user_agent replaces the default user-agent on either kind.

#![allow(unused)]
fn main() {
use leyline::{Browser, Session};

fn run() -> leyline::Result<()> {
let plain = Session::builder().user_agent("my-app/1.0").build()?;
let chrome = Session::browser(Browser::default());
let pinned = Session::builder().browser(Browser::Chrome154).build()?;
drop((plain, chrome, pinned));
Ok(())
}
}

Sessions lists the headers each kind sends.

Read the body

text() decodes the body with the charset from Content-Type, and falls back to UTF-8. json::<T>() deserializes into any DeserializeOwned type. A body that does not parse is an ErrorCategory::Decode error. Each body call consumes the response, so read status() and headers() first.

#![allow(unused)]
fn main() {
use serde::Deserialize;

#[derive(Deserialize)]
struct State {
    name: String,
    ready: bool,
}

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let page = session.get("https://example.com/").await?.text().await?;
let state = session
    .get("https://api.example/state")
    .await?
    .json::<State>()
    .await?;
println!("{} {} {}", page.len(), state.name, state.ready);
Ok(())
}
}

Send JSON, a form, and query parameters

json serializes the value and sets content-type: application/json. form sends application/x-www-form-urlencoded. query appends pairs to the URL. query and form take any iterator of (key, value) pairs where both sides are AsRef<str>. timeout with a Duration bounds the whole request, retries included; for a streamed body or a download, add TimeoutConfig::body.

#![allow(unused)]
fn main() {
use std::time::Duration;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let created = session
    .post("https://api.example/items")
    .json(&serde_json::json!({ "name": "leyline" }))
    .timeout(Duration::from_secs(10))
    .await?;
let found = session
    .get("https://api.example/search")
    .query([("q", "leyline"), ("page", "2")])
    .await?;
let login = session
    .post("https://api.example/login")
    .form([("user", "ada"), ("password", "secret")])
    .await?;
println!("{} {} {}", created.status(), found.status(), login.status());
Ok(())
}
}

See Requests and Retries and timeouts.

Handle a 404

A 4xx or 5xx response is Ok by default. Call .error_for_status() on the request builder to get a status of 400 or more as a Kind::Status error. The error keeps the status, the headers, and up to 64 KiB of the body. leyline::http re-exports the http crate, so StatusCode needs no extra dependency.

#![allow(unused)]
fn main() {
use leyline::http::StatusCode;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
match session.get("https://api.example/users/7").error_for_status().await {
    Ok(resp) => println!("{}", resp.text().await?),
    Err(err) if err.status() == Some(StatusCode::NOT_FOUND) => {
        println!("no such user: {}", err.body_text().unwrap_or_default());
    }
    Err(err) => return Err(err),
}
Ok(())
}
}

Error::category() sorts every failure into one ErrorCategory, such as Timeout, Dns, Tls, or Status, for a short message. See Errors.

Use ? with other error types

leyline::Error converts from std::io::Error and url::ParseError, so ? works on file calls in a function that returns leyline::Result. It implements std::error::Error + Send + Sync + 'static, so ? also converts it into Box<dyn std::error::Error + Send + Sync>, anyhow::Error, or your own error type.

#![allow(unused)]
fn main() {
async fn run() -> leyline::Result<()> {
let token = std::fs::read_to_string("token.txt")?;
let session = leyline::Session::builder()
    .bearer_auth(token.trim())
    .build()?;
let body = session.get("https://api.example/me").await?.text().await?;
std::fs::write("me.json", body)?;
Ok(())
}
}

Download a file

RequestBuilder::download streams the decoded body to a file in one atomic step and returns the number of bytes written. A status of 400 or more is an error. The limit caps the decoded size; None leaves only the session’s max_body_size (100 MiB by default). See Streaming.

#![allow(unused)]
fn main() {
use std::time::Duration;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let bytes = session
    .get("https://example.com/archive.tar.gz")
    .timeout(Duration::from_secs(600))
    .download("archive.tar.gz", Some(512 * 1024 * 1024))
    .await?;
println!("{bytes} bytes");
Ok(())
}
}

Find the page for your task

API clients

This chapter builds one plain client for a JSON API: a base URL, a token, timeouts, retries with a rate limit, status errors, pagination, and uploads. Each section links to the chapter with the full detail. examples/api_client.rs puts the steps together.

Build the client once

#![allow(unused)]
fn main() {
use std::time::Duration;

use leyline::http::StatusCode;
use leyline::{RetryPolicy, Session, TimeoutConfig, WaitFormat};

const BASE_URL: &str = "https://api.example/v1/";
const USER_AGENT: &str = "my-tool/1.0";
const RATE_REMAINING: &str = "x-ratelimit-remaining";
const RATE_RESET: &str = "x-ratelimit-reset";
const MAX_RATE_WAIT: Duration = Duration::from_secs(120);

fn run() -> leyline::Result<()> {
let token = std::env::var("API_TOKEN").unwrap_or_default();
let policy = RetryPolicy::transient()
    .retry_if(|resp| {
        resp.status() == StatusCode::FORBIDDEN && resp.header(RATE_REMAINING) == Some("0")
    })
    .wait_header(RATE_RESET, WaitFormat::UnixSeconds)
    .max_retry_after(MAX_RATE_WAIT)
    .retry_unsent(true);
let api = Session::builder()
    .base_url(BASE_URL)
    .bearer_auth(&token)
    .user_agent(USER_AGENT)
    .headers([("accept", "application/json")])
    .timeout(
        TimeoutConfig::new()
            .connect(Duration::from_secs(5))
            .total(Duration::from_secs(180)),
    )
    .retry(policy)
    .build()?;
drop(api);
Ok(())
}
}

Sessions has the URL resolution rules and the token scope.

Session is Clone, Send, and Sync. A clone shares the pool, the cookie jar, and every setting, so build the client once and clone it into each task.

Retry transient failures and rate limits

RetryPolicy::transient() retries idempotent methods up to 3 times on a connection error, a timeout, 429, 502, 503, and 504, with jittered exponential backoff. The setters in the example extend it:

total spans every attempt of one send, waits included. A wait that does not fit in the time left ends the retries, so set total above max_retry_after plus the time of the requests. See Retries and timeouts.

Turn statuses into errors

error_for_status() on the request turns a status of 400 or more into a Kind::Status error after the retries. The error keeps the status, the headers, and up to 64 KiB of the body; see Turn a status into an error. err.retry_after() returns the wait the policy read from the last response. err.retries_exhausted() is true when the policy wanted another try and did not make it; see Status codes are not errors.

#![allow(unused)]
fn main() {
use leyline::{Error, ErrorCategory};

fn explain(err: &Error) -> String {
    match err.category() {
        ErrorCategory::Status => match err.retry_after() {
            Some(wait) => format!("rate limited, retry in {wait:?}"),
            None => format!(
                "api error {:?} request {:?}: {}",
                err.status(),
                err.header("x-request-id"),
                err.body_text().unwrap_or_default()
            ),
        },
        ErrorCategory::Decode => format!("unexpected body: {err}"),
        ErrorCategory::Timeout => "the api timed out".to_owned(),
        other => format!("{other:?}: {err}"),
    }
}
let _ = explain;
}

See Errors.

Read JSON pages

pages() sends the request, then follows the Link: rel="next" header of each response with the same settings. See Responses.

There is no default page limit. Call limit(max) to stop after max pages. When a next link goes to another origin, the request drops the session credentials: the Authorization, Cookie, and Proxy-Authorization defaults and the session bearer token, as a redirect does.

#![allow(unused)]
fn main() {
use serde::Deserialize;

#[derive(Deserialize)]
struct Repo {
    name: String,
}

async fn run(api: leyline::Session) -> leyline::Result<()> {
let mut repos = Vec::new();
let mut pages = api.get("repos").error_for_status().pages().limit(100);
while let Some(page) = pages.next().await {
    repos.extend(page?.json::<Vec<Repo>>().await?);
}
for repo in &repos {
    println!("{}", repo.name);
}
Ok(())
}
}

Send JSON and upload files

json(&value) sends a JSON body. multipart(form) sends multipart/form-data; Part::file(path)? streams a file from disk, and mime sets the part’s content-type. See Requests.

#![allow(unused)]
fn main() {
use leyline::multipart::{Form, Part};

async fn run(api: leyline::Session) -> leyline::Result<()> {
api.post("issues")
    .json(&serde_json::json!({ "title": "hello" }))
    .error_for_status()
    .await?;

let form = Form::new()
    .text("title", "report")
    .part("file", Part::file("report.pdf")?.mime("application/pdf"));
api.post("uploads").multipart(form).error_for_status().await?;
Ok(())
}
}

Response bodies are decoded for you: the default features cover gzip, deflate, Brotli, and zstd. See Features and targets.

Sessions

A Session owns everything a request needs: the browser profile, the TLS context, the connection pool, and the cookie jar. Build one, reuse it, and clone it into tasks. This chapter covers building, deriving, stopping, and saving a session.

Build a session

Session::builder() returns a SessionBuilder. build() returns Result<Session>, so a bad proxy URL or an unknown profile fails there rather than at the first request.

#![allow(unused)]
fn main() {
use leyline::{Browser, Platform, Session};

fn run() -> leyline::Result<()> {
let session = Session::builder()
    .browser(Browser::Firefox154)
    .platform(Platform::MacOS)
    .user_agent("my-tool/1.0 (+https://my-tool.example)")
    .headers([("accept", "application/json")])
    .build()?;
let _ = session;
Ok(())
}
}

Session::new() and Session::default() build a plain session and do not fail. Session::browser(b) builds a session for browser b and does not fail either. A trust-store problem then fails each request that needs it with Kind::Tls, and an invalid proxy in the environment fails each request with Kind::Proxy. To add settings to a browser session, start from Session::builder().browser(b), whose build() reports those problems.

SessionBuilder::headers takes any iterator of (name, value) pairs, or references to pairs, where both sides are AsRef<str>. An entry replaces a default or profile header of the same name, compared without case. user_agent(value) replaces the default user-agent; it and a user-agent pair in headers set the same header, and the later call wins. On a browser session, a session user-agent also removes the profile’s sec-ch-ua client hint, so the user agent and the hint never name different browsers.

When a session has a browser and you do not call protocol, it uses ProtocolPolicy::Race if the http3 feature is on and the profile’s [h3] table sets race = true, as the bundled Chrome profiles do, and ProtocolPolicy::Auto otherwise. See HTTP/3.

Plain and browser sessions

A plain HTTP/1.1 GET sends these headers, in this order:

A plain session sends no accept-language and no browser-only headers: sec-* headers, client hints, upgrade-insecure-requests, and priority. This is true for every preset. languages turns accept-language on. A request adds the headers it needs, such as content-type, origin, referer, and cookie.

Set a token and a base URL

bearer_auth(token) sends authorization: Bearer <token> and replaces an authorization header set earlier on the builder. With a base_url, the token goes only to requests whose origin (scheme, host, and port) matches the base URL. Without one, it goes to every host. A redirect to another origin drops it. RequestBuilder::bearer_auth overrides it for one request, which is how to reach a second host. An authorization header set with headers is not scoped. Debug output masks the token.

base_url(url) resolves a relative &str or String request URL by the rules of Url::join (RFC 3986). A Url value goes out unchanged.

End the base URL with / to keep its last path segment. An invalid base URL fails build() with Kind::Config. A relative URL in a session with no base URL fails at send() with Kind::Url.

#![allow(unused)]
fn main() {
async fn run() -> leyline::Result<()> {
let api = leyline::Session::builder()
    .base_url("https://api.example/v1/")
    .bearer_auth("my-token")
    .build()?;
let repo = api.get("repos/leyline").await?;
let staging = api.with_base_url("https://staging.api.example/v1/")?;
let health = staging.get("/health").await?;
drop((repo, health));
Ok(())
}
}

with_base_url(url) derives a session with another base URL; the session token moves with it and goes to the new base URL’s origin. It returns Kind::Url for a URL that does not parse, and Kind::Config for a URL that cannot be a base.

Set the languages

languages(tags) sets accept-language in the format of the session’s browser family. The first tag has the highest weight. An invalid tag or an empty list fails build() with Kind::Config. A plain session uses the Chromium format.

Without languages, a browser session sends the value captured with its profile. SessionIdentity::languages() returns the tags the session was built with.

#![allow(unused)]
fn main() {
use leyline::{Browser, Session};

fn run() -> leyline::Result<()> {
let session = Session::builder()
    .browser(Browser::default())
    .languages(["de-DE", "de", "en-US"])
    .build()?;
let langs = session.identity().languages().map(<[String]>::to_vec);
let _ = langs;
Ok(())
}
}

Choose a browser

The Browser enum has a variant for each bundled profile. The API reference lists them, and Browser profiles gives the capture kind of each.

• Browser::latest(Family) is the newest profile of a product line that is not deprecated and whose capture kind the line accepts. Pin the line and take what the crate release carries.
• Browser::default() is Browser::latest(Family::Chrome).
• Browser::get(Family, version) returns the profile for one major version, or None. Browser::version() returns the major version.
• Browser::family() returns the product line. The engine family is the meta.family field of Browser::profile().

#![allow(unused)]
fn main() {
use leyline::{Browser, Family};

let latest = Browser::latest(Family::Firefox);
assert_eq!(latest.family(), Family::Firefox);
assert_eq!(Browser::default().family(), Family::Chrome);
}

Choose a platform

Platform sets the operating system the session claims: Windows, MacOS, Linux, Android, IOS, or Host. Platform::Host resolves to the OS you compiled for, or Windows for an unrecognized target. The platform drives Sec-CH-UA-Platform, Sec-CH-UA-Mobile, and the TCP fingerprint.

Platform defaults and call order

• A browser with no platform claims the first platform, in the order Windows, macOS, Linux, Android, iOS, that its profile covers. Chrome claims Windows, Browser::Safari26 macOS, and Browser::SafariIOS27 iOS. Session::browser(b) uses the same rule.
• A .profile() with no .platform() claims Windows and logs one info event under the leyline::session target.
• .browser() and .platform() map the browser to its sibling for the platform, in either call order. Browser::for_platform does the same.
• A session with no browser claims the host platform.

Apply a brand overlay

Edge and Opera are overlays on a Chrome profile: the HTTP/2 settings stay Chrome’s, and the User-Agent and sec-ch-ua brand list change. Edge also leaves out the trust anchors extension of the ClientHello.

#![allow(unused)]
fn main() {
use leyline::{Browser, ChromiumBrand, Session};

fn run() -> leyline::Result<()> {
let session = Session::builder()
    .browser(Browser::default())
    .brand(ChromiumBrand::Edge)
    .build()?;
let _ = session;
Ok(())
}
}

Both brands exist on desktop platforms only; a mobile platform fails build(). Edge takes its version from Chromium and fits every desktop Chrome profile. Opera supports Chromium 145 to 153, so Browser::default() with ChromiumBrand::Opera can fail build(); pin an older Chrome such as Browser::Chrome153. Brave has its own profiles, such as Browser::Brave146.

ChromiumBrand::all() lists the brands. A brand prints as its lowercase name (chrome, edge, opera), and str::parse reads it back without case. An unknown name is a Kind::Config error.

Browser::identity(platform, brand) returns the PlatformIdentity a session sends: user_agent, sec_ch_ua, and accept_language. Use it when a payload that is not a header must carry the same values. It returns None when the browser has no identity for the platform or the brand does not apply.

Session::identity() returns a SessionIdentity with the values build() resolved: identity() (None for a plain or loaded-profile session), browser(), platform(), brand(), and user_agent(). A user-agent from SessionBuilder::headers is the value user_agent() returns.

#![allow(unused)]
fn main() {
use leyline::{Browser, ChromiumBrand, Platform};

let edge = Browser::default().identity(Platform::Windows, Some(ChromiumBrand::Edge));
assert!(edge.is_some_and(|id| id.user_agent.contains("Edg/")));

let session = leyline::Session::browser(Browser::default());
let sent = session.identity();
println!("{:?} on {} sends {}", sent.browser(), sent.platform(), sent.user_agent());
}

Share a session

A clone is an Arc clone: it shares the cookie jar, the connection pool, the TLS context, and every setting. Session is Send + Sync. Pass clones into tasks instead of building a second session.

#![allow(unused)]
fn main() {
async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let worker = session.clone();
tokio::spawn(async move { worker.get("https://example.com/").await })
    .await
    .expect("task panicked")?;
Ok(())
}
}

host_limits(HostLimits) and proxy_pool(ProxyPool) are shared by clones and derived sessions too. See Crawling and Proxies.

Derive a session

A derived session changes one setting and shares the rest with its parent. None of the calls below fail except with_base_url and with_identity.

Every derived session keeps the host limits, the proxy pool, the audit setting, and the shutdown state.

• with_proxy takes impl Into<ProxyConfig>. Pool entries are keyed by proxy, so connections never cross proxies. An invalid URL or an unsupported scheme fails the first send() that uses it, with Kind::Config.
• with_identity sends another Identity. Its connections sit in their own partition of the parent’s pool, so they never cross identities, and they share the parent’s pool limits and pool_stats(). It also shares the HSTS store and the Alt-Svc knowledge. It returns Kind::Config for a plain session. See Mix and rotate identities.
• fresh_pool starts with an empty pool, TLS session cache, and HSTS store, and has its own pool limits. Call it when the next request must open new connections.
• tab() returns a Tab that keeps the current page. See Requests.

#![allow(unused)]
fn main() {
use leyline::cookie::Jar;

fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let other_user = session.with_cookie_jar(Jar::new());
let via_proxy = session.with_proxy("http://user:pass@proxy.example:8080");
let new_exit = via_proxy.fresh_pool();
let no_follow = session.with_redirect(leyline::RedirectPolicy::none());
let _ = (other_user, new_exit, no_follow);
Ok(())
}
}

Stop a session

Session::shutdown() stops a session, its clones, and every session derived from it. Requests in flight and new requests fail with Kind::Request and the message “session shut down”, and a streamed body fails on its next read. Error::is_shut_down() is true for this error, and Session::is_shut_down() is true after the call.

WebSocket handshakes and Session::preconnect follow the same rule: a call after shutdown fails with the shut-down error before it sends any bytes, and a call in flight ends with that error. An established WsConnection stays open until the caller closes it. Shutdown does not close idle pooled connections.

#![allow(unused)]
fn main() {
use leyline::Kind;

async fn run() {
let session = leyline::Session::new();
let worker = session.clone();
session.shutdown();
assert!(worker.is_shut_down());
let err = worker.get("https://example.com/").await.unwrap_err();
assert_eq!(err.kind(), Kind::Request);
assert!(err.is_shut_down());
}
}

Save and restore an identity

Identity implements Serialize and Deserialize. Browser, Platform, Family, and ChromiumBrand serialize as stable lowercase ids, such as chrome-154, windows, and edge; id() returns the id and str::parse reads it back. Display prints a label for people, not the id.

session.identity().to_identity() returns the Identity that rebuilds the session, and SessionBuilder::identity(id) builds from it.

profile_id() returns 16 hex characters: a hash of the profile data the session sends, with the platform and the brand. The same data gives the same id in every release. It is None for a plain session. A profile loaded from TOML gets an id from its TOML text. SessionBuilder::expect_profile_id(id) makes build() fail with Kind::Config when the id differs or is None; Error::is_profile_changed() is true for that error.

#![allow(unused)]
fn main() {
use leyline::{Browser, Identity, Platform, Session};

fn run() -> Result<(), Box<dyn std::error::Error>> {
let session = Session::builder()
    .browser(Browser::Chrome154)
    .platform(Platform::MacOS)
    .build()?;
let sent = session.identity();
let saved = serde_json::to_string(&sent.to_identity())?;
let profile_id = sent.profile_id().map(str::to_owned).ok_or("plain session")?;

let identity: Option<Identity> = serde_json::from_str(&saved)?;
let restored = Session::builder()
    .identity(identity.ok_or("plain session")?)
    .expect_profile_id(&profile_id)
    .build()?;
drop(restored);
Ok(())
}
}

To save the identity with the cookie jar and the proxy, use a Device; see Accounts. Session::state() returns the TLS session tickets, the Alt-Svc entries, and the HSTS store as a SessionState; see Accounts.

Keep connections warm

preconnect(url) opens the TCP, proxy, TLS, and HTTP/2 connection for an https origin and stores it in the pool, so the first request skips the handshake. It does nothing for an http URL or under ProtocolPolicy::Http1. If the origin speaks only HTTP/1.1, the session records that and returns Ok. To warm a connection through another proxy, call session.with_proxy(proxy).preconnect(url).

Before the pool reuses an HTTP/2 connection idle for 10 seconds, it sends a PING, as Chrome does. Without an answer in 2 seconds it drops the connection and opens a new one, and pool_stats().h2_ping_failures counts it. Network lists the PoolConfig settings.

#![allow(unused)]
fn main() {
async fn run() -> leyline::Result<()> {
let session = leyline::Session::browser(leyline::Browser::default());
session.preconnect("https://example.com/").await?;
println!("{:?}", session.pool_stats());
Ok(())
}
}

Trace the request lifecycle

trace() installs a listener for each phase of every request: name resolution, connect, TLS handshake, request sent, response head, and completion. Implement leyline::trace::Trace and override only the events you want; every method has a no-op default. TracingTrace is the bundled listener; see Logging and tracing.

#![allow(unused)]
fn main() {
use leyline::Session;
use leyline::trace::{Head, Trace};

struct Slow;

impl Trace for Slow {
    fn head(&self, ev: &Head<'_>) {
        if ev.elapsed.as_millis() > 500 {
            eprintln!("{} took {:?} to answer", ev.host, ev.elapsed);
        }
    }
}

fn run() -> leyline::Result<()> {
let session = Session::builder().trace(Slow).build()?;
let _ = session;
Ok(())
}
}

• Every event carries an id unique per attempt. A retry is a new attempt with a new id.
• An event fires on the task that produced it. Leyline opens an HTTP/2 or HTTP/3 connection on a task it spawns, so those events fire there, with the id of the request that started the open. A listener that blocks slows every request waiting for that connection.
• dns fires each time Leyline resolves a name for a TCP connection. Through a proxy it reports the proxy host. A direct HTTP/3 connection fires none.
• sent carries the method and the path with the query. On HTTP/2 and HTTP/3 it fires before the connection driver takes the request, and its elapsed is zero.
• head carries the response headers as received, before decompression and redirect handling.

Response::timing() needs no listener. See Responses.

Choosing a profile

A browser session has a profile and a platform. The profile sets the TLS ClientHello, the HTTP/2 settings, and the request headers. The platform sets the operating system that the session claims. This page picks both.

Start with the default

Browser::default() is Browser::latest(Family::Chrome), the newest bundled Chrome. Without .platform(), a Chrome session claims Windows.

#![allow(unused)]
fn main() {
use leyline::{Browser, Platform, Session};

fn run() -> leyline::Result<()> {
let windows = Session::browser(Browser::default());
let mac = Session::builder()
    .browser(Browser::default())
    .platform(Platform::MacOS)
    .build()?;
let _ = (windows, mac);
Ok(())
}
}

With the http3 feature, a Chrome session races HTTP/3 against HTTP/2 for an origin that advertised h3. See HTTP/3.

Pick another browser

.browser(Browser::Safari26).platform(Platform::IOS) maps to the Safari on iPhone sibling. Sessions gives the platform defaults and the call order, and Set the languages sets accept-language.

Match the platform

The platform sets Sec-CH-UA-Platform, Sec-CH-UA-Mobile, the User-Agent, and the TCP fingerprint. A site can compare the claimed OS with the TCP fingerprint, so a claim that differs from the real host is visible. Use Platform::Host to claim the OS you build for.

Keep a device stable

Browser::latest moves to newer captures in patch releases, and a release can correct the data behind a pinned variant. For an account that must keep one device, pin the variant and store session.identity().profile_id(). Pass it to SessionBuilder::expect_profile_id on start: build() then fails with Kind::Config when the profile changed, and err.is_profile_changed() is true. To keep the data itself, freeze it with Device::pin_profile. See Profile data stability and Accounts.

Read the JA4 labels

The profile reference labels the JA4 of each profile gated or estimated. A Chrome or Brave profile is estimated because Chrome shuffles the extension order on every connection. JA4 sorts the extensions before it hashes them, so the JA4 stays the same; JA3 does not. To check a live session, see Fingerprints.

Next

Read Requests to build and send a request.

Requests

Session::get and its siblings return a RequestBuilder. Chain settings on the builder, then await it. This chapter covers methods, headers, bodies, the fetch context of a browser request, and per-request overrides. The API uses the http crate’s types, re-exported as leyline::http.

Methods and URLs

The session has get, post, put, patch, delete, and head. For any other method, call request with an http::Method. Each takes a &str, a String, a &String, a url::Url, or a &url::Url.

#![allow(unused)]
fn main() {
use leyline::http::Method;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let resp = session.request(Method::OPTIONS, "https://example.com/api").await?;
println!("{}", resp.status());
Ok(())
}
}

A relative &str or String URL resolves against the session base URL; see Set a token and a base URL. A URL that does not parse does not panic: send returns Kind::Url with the url::ParseError as its source.

Headers

Header setters take anything that converts to an http::HeaderName and an http::HeaderValue, such as &str, String, or a HeaderName constant. An invalid name or value is an error from send. header appends: a second call with the same name adds a second value. headers takes an iterator of pairs. bearer_auth and basic_auth build the Authorization header. On HTTP/2 and HTTP/3 the session does not send a Host header you set, because the request authority carries the host.

#![allow(unused)]
fn main() {
use leyline::http::header::ACCEPT_LANGUAGE;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let resp = session
    .get("https://example.com/")
    .header("x-request-id", "1")
    .header(ACCEPT_LANGUAGE, "en-GB,en;q=0.9")
    .header("x-tag", "a")
    .header("x-tag", "b")
    .await?;
let _ = resp;
Ok(())
}
}

On a browser session, see Override headers safely for the headers you can change without breaking the fingerprint.

Order

Order matters to a fingerprint, so the profile’s order applies on the wire. A request header replaces every session or profile header of the same name, in that header’s position; repeated header calls for the name send every value there. A name the session does not send goes where a browser places it, such as authorization after user-agent, or else at the end.

Two methods override the order:

• header_order(&["a", "b"]) pins the wire order of the regular headers for this request, on every protocol.
• anchored(anchor, name, value) inserts one header at a HeaderAnchor slot: BeforeCchUa, AfterCchUa, AfterCchUaMobile, AfterCchUaPlatform, AfterUserAgent, AfterAccept, AfterContentType, AfterFetchDest, or BeforeAcceptEncoding.

#![allow(unused)]
fn main() {
use leyline::HeaderAnchor;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let resp = session
    .get("https://example.com/")
    .anchored(HeaderAnchor::AfterUserAgent, "x-client", "leyline")
    .header_order(&["x-client", "accept"])
    .await?;
let _ = resp;
Ok(())
}
}

To see the headers a request was prepared with, build the session with .audit(true) and read Response::request_headers.

Query parameters and bodies

query appends pairs to the URL after any query it already has. form sends pairs as application/x-www-form-urlencoded. Both take any iterator of (K, V) tuples, or references to them, where K and V are AsRef<str>. One request can carry both.

body takes a Body, or anything that converts to one: String, &'static str, Vec<u8>, &'static [u8], Bytes, or (). json sets content-type: application/json. compress(encoding) compresses a buffered body and sets Content-Encoding.

#![allow(unused)]
fn main() {
async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let search = session
    .post("https://example.com/search")
    .query([("page", "2")])
    .form([("q", "leyline"), ("sort", "new")])
    .await?;
let item = session
    .post("https://example.com/items")
    .json(&serde_json::json!({ "name": "leyline" }))
    .await?;
let raw = session
    .post("https://example.com/raw")
    .header("content-type", "application/octet-stream")
    .body(vec![0u8, 1, 2])
    .await?;
let _ = (search, item, raw);
Ok(())
}
}

To stream a request body, pass Body::stream; see Streaming.

Upload multipart forms

The multipart feature, on by default, adds leyline::multipart::{Form, Part} and RequestBuilder::multipart, which sets content-type: multipart/form-data with the form’s boundary. A file part streams from disk chunk by chunk.

#![allow(unused)]
fn main() {
use leyline::multipart::{Form, Part};

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let form = Form::new()
    .text("name", "ada")
    .file("log", "server.log")?
    .part("logo", Part::file("logo.png")?.mime("image/png"))
    .part(
        "photo",
        Part::bytes(std::fs::read("photo.jpg")?)
            .filename("photo.jpg")
            .mime("image/jpeg"),
    );
let resp = session.post("https://api.example/upload").multipart(form).await?;
let _ = resp;
Ok(())
}
}

Form::file(name, path) and Part::file(path) use the last component of path as the filename, or file when it has none or is not UTF-8. A Form::file part has no Content-Type, so the server reads it as text/plain (RFC 7578); use Part::file(path)?.mime(..) to send a type. Both return std::io::Result, which ? converts into leyline::Error. examples/multipart.rs is a complete program.

Presets

A Preset sets the sec-fetch-* headers and the header order for a fetch context: Native, Navigate, FrameNavigate, Reload, Script, Image, Xhr, Form, CrossOrigin, SameSite, and FormNavigate. Preset::Navigate is the default for a GET from a browser session.

On a browser session with no preset, a POST, PUT, or PATCH infers one from content-type: application/json gives Xhr, and application/x-www-form-urlencoded gives Form. Other types leave the preset unset. Set .preset(..) when the guess is wrong.

Form is a script that posts form data with fetch(). FormNavigate is a person who clicks the submit button of an HTML form, as on a login page; form() never infers it.

Send the requests of a page

Pass the page that sends a request to initiator. Leyline then sets Referer, Origin, and sec-fetch-site the way the browser does for the request’s preset. A navigation with an initiator is a link click. A navigation without one is a typed URL: sec-fetch-site: none and no Referer.

#![allow(unused)]
fn main() {
use leyline::Preset;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::browser(leyline::Browser::default());
let page = session.get("https://shop.example/cart").await?;
let page_url = page.url().clone();
let cart = session
    .get("https://shop.example/api/cart")
    .preset(Preset::Xhr)
    .initiator(page_url.clone())
    .await?;
let logo = session
    .get("https://cdn.example/logo.png")
    .preset(Preset::Image)
    .initiator(page_url)
    .await?;
drop((cart, logo));
Ok(())
}
}

Referer follows strict-origin-when-cross-origin. P is the page and T the target:

The second table gives the Origin header. sec-fetch-site is computed from the page and every redirect, by the same rules for Chromium and Gecko profiles. sec-fetch-user: ?1 goes only with Navigate and FormNavigate. A referer header you set wins over the computed value.

Keep the page with a tab

A Tab keeps the current page, as a browser tab does: each request sends that page as its initiator, and each navigation sets the next page. Session::tab() makes one, and clones of a tab share the page.

The page changes after each navigation response, 4xx and 5xx included, to the final URL after redirects. A transport error leaves it unchanged. Relative URLs resolve against the current page, or else the base URL. A .stream() navigation changes the page when the response head arrives. set_current(page) sets the page by hand. A script request or subresource on a tab with no page fails with Kind::Request and sends nothing: open a page first.

#![allow(unused)]
fn main() {
use leyline::Preset;

async fn run() -> leyline::Result<()> {
let tab = leyline::Session::browser(leyline::Browser::default()).tab();
tab.open("https://shop.example/").await?;
tab.follow("/cart").await?;
let cart = tab.xhr("/api/cart").await?;
let logo = tab.subresource("https://cdn.example/logo.png", Preset::Image).await?;
tab.submit("/checkout", [("step", "address")]).await?;
println!("{:?}", tab.current());
drop((cart, logo));
Ok(())
}
}

Accounts uses a tab with a saved device and submits the forms of a page.

Override the session for one request

timeout takes a Duration or a TimeoutConfig. A Duration sets total, which bounds the request up to the head of a streamed response; set TimeoutConfig::body to bound a streamed body or a download. A TimeoutConfig sets total, read, body, and response_header; an unset field keeps the session value, and None turns that timeout off. connect is session-wide, because connections are pooled. See Retries and timeouts.

#![allow(unused)]
fn main() {
use std::time::Duration;

use leyline::TimeoutConfig;
use leyline::cookie::Jar;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let guest = Jar::new();
let resp = session
    .get("https://example.com/slow")
    .proxy("http://user:pass@proxy.example:8080")
    .timeout(
        TimeoutConfig::new()
            .total(Duration::from_secs(30))
            .read(Duration::from_secs(5)),
    )
    .cookie_jar(guest.clone())
    .tag("account-7")
    .await?;
println!("{} cookies", guest.all_cookies().len());
drop(resp);
Ok(())
}
}

Send an http::Request

Session::execute sends an http::Request<Body> with its method, URI, headers, and body. It reads an optional Preset, TimeoutConfig, RetryPolicy, RedirectPolicy, and ProxyConfig from the request extensions, infers the preset from content-type as the builder does, and reads the whole response before it returns. Digest authentication and response streaming need the RequestBuilder.

#![allow(unused)]
fn main() {
use leyline::Body;
use leyline::http::{Method, Request};

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let req = Request::builder()
    .method(Method::POST)
    .uri("https://example.com/items")
    .header("content-type", "application/json")
    .body(Body::from(r#"{"ok":true}"#))
    .expect("valid request");
let resp = session.execute(req).await?;
let _ = resp;
Ok(())
}
}

The tower feature adds LeylineService, a tower_service::Service over the same send path. See Service integration.

Responses

A Response carries the status, the headers, the body, and the metadata Leyline collected while sending: the final URL, the redirect chain, the proxy, the timing, and the TLS facts. Read the metadata first, because every body call consumes the response.

Status and headers

status() returns an http::StatusCode, with is_success, is_redirection, is_client_error, and is_server_error. version() returns HttpVersion::Http1_1, Http2, or Http3, and as_str() gives "HTTP/1.1", "HTTP/2", or "HTTP/3".

headers() returns the http::HeaderMap, duplicates included. Iteration does not follow wire order; get_all yields the values of one name in the order they arrived. header(name) returns the first value as &str, or None when it is missing or holds a byte outside visible ASCII. Read headers() for the raw bytes.

#![allow(unused)]
fn main() {
async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let resp = session.get("https://example.com/").await?;
if resp.status().is_success() {
    println!("{} {}", resp.status().as_u16(), resp.version().as_str());
}
let ctype = resp.header("content-type");
let cookies: Vec<&[u8]> = resp
    .headers()
    .get_all("set-cookie")
    .iter()
    .map(|v| v.as_bytes())
    .collect();
let _ = (ctype, cookies);
Ok(())
}
}

content_length() parses Content-Length. When Leyline decodes a buffered body, it removes Content-Encoding and Content-Length from the headers. A .stream() response keeps the wire values.

request_headers() lists the headers the session prepared to send, in send order, after the preset, the cookie jar, and your headers were merged. It needs .audit(true) on the session and is empty otherwise. The values are prepared before dispatch, not captured from the transport.

trailers() iterates trailing headers in wire order. Buffered HTTP/2 and HTTP/3 responses carry them; a streamed response and HTTP/1.1 yield none.

Relay headers

relay_headers(body) returns the headers without the hop-by-hop ones: connection, each header that connection names, keep-alive, proxy-connection, proxy-authenticate, proxy-authorization, te, trailer, transfer-encoding, and upgrade (RFC 9110, section 7.6.1). RelayBody::AsReceived matches a body from into_stream(). RelayBody::Decoded matches a body from bytes(), text(), or into_decoded_stream(), and also drops content-encoding and content-length when Leyline decodes the body. When the session does not decode the coding, for example because CompressionConfig turns it off, the body arrives as sent and content-encoding stays. proxy-authenticate and proxy-authorization are dropped as hop-by-hop headers. See Service integration.

Bodies

Without .stream(), Leyline reads, decodes, and caps the whole body before send() returns. With .stream(), send() returns at the headers. The reading calls work in both modes: a buffered body returns at once, and a streamed one is drained, with the read timeout on each chunk, then decoded.

Streaming covers the streaming calls and downloads.

text() replaces invalid sequences with U+FFFD, and a leading byte order mark overrides the declared charset. Without the charset feature, on by default, text() uses lossy UTF-8.

Size cap and content coding

CompressionConfig::max_body_size (100 MiB by default) caps a buffered body and the decoded body, on every protocol. A body over the cap fails with Kind::Body, and the message names max_body_size. Set the cap on the session with SessionBuilder::compression:

#![allow(unused)]
fn main() {
use leyline::{CompressionConfig, Session};

fn run() -> leyline::Result<()> {
let session = Session::builder()
    .compression(CompressionConfig::new().max_body_size(8 * 1024 * 1024))
    .build()?;
let _ = session;
Ok(())
}
}

CompressionConfig::new() turns on gzip, deflate, br, and zstd, and a session decodes and advertises those that are also compiled in. CompressionConfig::none() turns them all off. Repeated Content-Encoding fields combine into one list. More than four codings fail with Kind::Decode. When a listed coding is turned off, the body comes back as received.

Final URL and redirects

url() returns the final URL after redirects. redirect_chain() returns each URL that answered with a redirect Leyline followed, in order, without the final URL and without user name or password. It is empty when nothing redirected. attempts() returns the number of transport attempts, 1 when no retry ran.

#![allow(unused)]
fn main() {
async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let resp = session.get("https://example.com/old").await?;
for url in resp.redirect_chain() {
    println!("via {url}");
}
println!("landed on {} after {} attempts", resp.url(), resp.attempts());
Ok(())
}
}

See Redirects.

Follow Link pagination

link(rel) returns the URL of the first Link entry with that relation. links() returns every entry as a Link with url, rel, and params. Parsing follows RFC 8288: several headers, comma-separated entries, quoted parameters, and several relations in one rel. Relative URLs resolve against url(), and the rel match ignores case.

RequestBuilder::pages() turns a request into a Pages sequence that follows Link: rel="next". Await pages.next(), or use it as a futures_util::Stream. Each follow-up is a GET with the first request’s headers, timeouts, retry policy, and error_for_status() setting. The sequence ends after a page with no next link, after an error, or at a next URL it already fetched. The first request URL and the final URL of each page, after redirects, count as fetched.

#![allow(unused)]
fn main() {
use serde::Deserialize;

#[derive(Deserialize)]
struct Item {
    id: u64,
    name: String,
}

async fn run() -> leyline::Result<()> {
let api = leyline::Session::builder().bearer_auth("my-token").build()?;
let mut pages = api
    .get("https://api.example/items")
    .error_for_status()
    .pages();
while let Some(page) = pages.next().await {
    for item in page?.json::<Vec<Item>>().await? {
        println!("{} {}", item.id, item.name);
    }
}
Ok(())
}
}

Cookies

The session jar stores Set-Cookie automatically. cookies() parses the Set-Cookie headers of the final response only. It lists every header that parses, deletions included, and does not ask the jar, so it can list a cookie the jar refused. Read the jar with session.cookies().get_cookie(&url, name). See Cookies.

Turn a status into an error

A 4xx or 5xx response is Ok. error_for_status() consumes the response and returns a Kind::Status error for a status of 400 or more, with the status, the headers, and the final URL. error_for_status_ref() makes the same check on a borrowed response, so you can still read the body of an error.

RequestBuilder::error_for_status() checks the final response after the retry policy is done, and keeps the start of the body in the error:

Leyline decides after the status line. A response below 400 is read in full as usual. An error response is read only up to these limits, so a large or slow error body cannot hold the request to the body timeout or exceed max_body_size. Decompression stops at the byte limit, so a small compressed body cannot expand past it.

The error-body timeout starts when the response head arrives, and each error response gets one window, however its body is read. If the read fails or the window ends first, the error keeps the bytes decoded so far, or no body if none arrived, with the status and headers. The error’s headers drop content-encoding when Leyline decoded the body, and content-length when the kept body is not the full declared length.

max_error_body counts decoded bytes. Over HTTP/1.1, an error body that declares a length of max_error_body or less is read with the response. If it arrives within the window, the connection goes back to the pool; otherwise Leyline closes it. A larger or unframed error body is read only until max_error_body decoded bytes. If the rest of a fixed-length error body is no larger than max_error_body, Leyline reads and discards it in the background, within the pool’s idle timeout, so the connection can be reused; this never extends the error-body window. Otherwise it closes the connection. Over HTTP/2 and HTTP/3, the stream is reset and the connection stays open. download uses the same limits for a status error.

#![allow(unused)]
fn main() {
async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let resp = session.get("https://example.com/api").await?;
if let Err(e) = resp.error_for_status_ref() {
    let ray = resp.header("cf-ray").map(str::to_owned);
    let body = resp.text().await?;
    println!("{e}: {body} {ray:?}");
}
Ok(())
}
}

Errors covers the status error and its accessors.

Proxy, timing, and TLS

proxy() returns the proxy that carried the response, with the password, or a user name without a password, replaced by ***. It is None for a direct connection. Under a ProxyPool it names the proxy of the attempt that produced the response.

Timing

timing() returns a ResponseTiming summed across redirect legs, on every protocol. It does not include the body read.

tls() returns a TlsInfo with the negotiated version, the cipher, and the peer certificate as DER (peer_cert_der). audit() returns the fingerprints the session presents when it was built with .audit(true), and None otherwise. See Fingerprints.

#![allow(unused)]
fn main() {
async fn run() -> leyline::Result<()> {
let session = leyline::Session::new().with_proxy("http://user:pass@proxy.example:8080");
let resp = session.get("https://example.com/").await?;
let t = resp.timing();
println!(
    "{} reused={} total={}ms tls={:?}",
    resp.proxy().unwrap_or("direct"),
    t.reused,
    t.total_ms,
    resp.tls().and_then(|tls| tls.version.as_deref()),
);
Ok(())
}
}

Detect a block page

block() returns a BlockSignal when the response matches a built-in rule for a bot challenge, a captcha, or a block page. BlockRules adds your own rules. See Crawling.

Print a response

{:?} prints the status, version, URL, redirect chain, headers, trailers, request headers, and timing, but not the body. URLs print with the password and the query hidden. Authorization, Proxy-Authorization, Cookie, and Set-Cookie values print as ***.

Streaming

Stream when a body is larger than you want in memory, or when you want to act on the first bytes before the last ones arrive. This chapter covers request bodies, response bodies, downloads, and flow control. Streaming needs no feature. The examples also need bytes, futures-util, and tokio-util with the io feature.

Stream a request body

Body::stream(s, len) takes any Stream of io::Result<Bytes> and an Option<u64> length. Give the length whenever you know it: Leyline then sends Content-Length on every protocol. Without it, HTTP/1.1 sends Transfer-Encoding: chunked and HTTP/2 and HTTP/3 declare no length, which some origins reject. Body::len_hint() reports the declared length.

#![allow(unused)]
fn main() {
use leyline::Body;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let file = tokio::fs::File::open("upload.bin").await?;
let len = tokio::fs::metadata("upload.bin").await?.len();
let body = Body::stream(tokio_util::io::ReaderStream::new(file), Some(len));
let resp = session
    .post("https://example.com/upload")
    .header("content-type", "application/octet-stream")
    .body(body)
    .await?;
println!("{}", resp.status());
Ok(())
}
}

When your stream returns an error, the request fails with Kind::Body, and Error::io() returns your error. Leyline does not resend, because a stream cannot be replayed. An HTTP/2 or HTTP/3 connection resets only that stream and stays open. An HTTP/1.1 connection closes, because it holds part of the request.

When a body cannot be replayed

A streamed body is consumed by the first attempt, so:

• Redirects. A redirect that turns the request into a GET follows normally. A redirect that keeps the method and body, such as a 307 or 308, fails with Kind::Redirect. See Redirects.
• Retries. The retry policy is skipped, and the first outcome is returned.

For retries or replayable redirects, buffer the body and send it as Bytes.

Stream a response body

Call .stream() on the request builder. The response arrives with the headers, and into_stream() returns the body as a BodyStream, a futures_util::Stream<Item = io::Result<Bytes>>.

#![allow(unused)]
fn main() {
use futures_util::StreamExt;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let resp = session.get("https://example.com/big").stream().await?;
let mut body = resp.into_decoded_stream(Some(64 * 1024 * 1024))?;
let mut total = 0u64;
while let Some(chunk) = body.next().await {
    total += chunk?.len() as u64;
}
println!("{total} bytes");
Ok(())
}
}

text(), bytes(), and json() also work on a .stream() response: they drain and decode it. A streamed response carries no trailers.

Decoded or raw

The decoded calls decode chunk by chunk and work on a buffered response too. limit: None means max_body_size. A body of exactly limit bytes passes. A longer one fails with Error::is_body_limit() true, and the chunk that crosses the limit is not written. A coding that CompressionConfig turns off passes through still encoded. See Responses.

Download a file

RequestBuilder::download(path, limit) sends the request with .stream() and saves the decoded body to path. It returns the number of bytes written. Response::download_to(path, limit) does the same for a response you hold.

A path with no file name fails with Kind::Request.

Bound the whole download

For a .stream() request, total covers the request up to the response head. Two timeouts bound the body:

• read limits the gap between two chunks. It stops a server that goes quiet. A gap past it is an io::ErrorKind::TimedOut error from the stream.
• body limits the whole body, from its first read. It stops a server that sends slowly but never stops. A plain Duration passed to .timeout sets only total, which ends at the head of a streamed response.

#![allow(unused)]
fn main() {
use std::time::Duration;

use leyline::TimeoutConfig;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
match session
    .get("https://example.com/image.iso")
    .timeout(
        TimeoutConfig::new()
            .read(Duration::from_secs(30))
            .body(Duration::from_secs(600)),
    )
    .download("image.iso", Some(8 * 1024 * 1024 * 1024))
    .await
{
    Ok(n) => println!("{n} bytes"),
    Err(e) if e.is_body_limit() => eprintln!("too large: {e}"),
    Err(e) => return Err(e),
}
Ok(())
}
}

See Retries and timeouts. To stop a download from your code, drop the future; see Cancellation.

Stop at a marker

RequestBuilder::read_until(limit, done) sends the request, reads and decodes the body as the chunks arrive, and stops when you tell it to. Use it when the value you need sits near the top of a large page. It streams the response itself, so the body is not buffered first, and it keeps the request’s redirects, retries, authentication, host limits, and error_for_status().

After each chunk that adds decoded bytes, Leyline calls done(body, from): body is every decoded byte so far, and from is where the new bytes start. Return true to stop. The call returns a PrefixRead: the decoded prefix in bytes, and why the read stopped in stopped_by.

LimitReached and EndOfBody mean that done found no answer, not that the answer is “no”. A zero limit reads nothing and returns LimitReached, and done is never called for an empty body.

#![allow(unused)]
fn main() {
use leyline::StopReason;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let marker = b"</head>";
let read = session
    .get("https://example.com/big")
    .read_until(256 * 1024, |body, from| {
        let start = from.saturating_sub(marker.len() - 1);
        body[start..].windows(marker.len()).any(|w| w == marker)
    })
    .await?;
match read.stopped_by {
    StopReason::PredicateMatched => println!("head is {} bytes", read.bytes.len()),
    other => println!("no </head> found: {other:?}"),
}
Ok(())
}
}

Scan from from minus the marker length plus one, as the example does: a marker can straddle two chunks, and a scan of the whole body on every chunk costs quadratic time. examples/stock_monitor.rs reports a product as in stock, out of stock, or unknown with the same pattern.

The limit counts decoded bytes. The bytes received from the network differ: fewer for a compressed body, and more by whatever the socket had already received when the read stopped. Leyline does not stop at the exact network byte where the marker ends. When it stops early, it drops the rest of the body: over HTTP/1.1 it closes the connection, and over HTTP/2 and HTTP/3 it resets the stream and keeps the connection.

total and response_header bound the request up to the response head. read bounds each wait for a chunk, and body the whole read. See Retries and timeouts.

Response::read_until does the same on a response you already have and returns only the bytes. Send that request with .stream() first; without it, the body is already buffered, and stopping early saves nothing.

Corrupt compressed data fails with Kind::Decode, a body cut short with Kind::Io, and an unknown or turned-off Content-Encoding is not decoded.

Back-pressure

A streamed response body reaches you through a bounded channel, so a slow reader slows the sender instead of growing a buffer.

Dropping a BodyStream before its end stops the transfer. See Cancellation for what happens to the connection.

Cancellation

To cancel a request, drop it: the request future, the Response of a .stream() request, or its BodyStream. Leyline then stops the request and frees what it holds. A timeout, tokio::select!, and JoinHandle::abort all cancel by dropping. What happens on the wire depends on the protocol and on how far the request got.

Drop a request before the response head

A request without .stream() reads the whole body before send() returns, so dropping its future also stops the body read.

• A connection that Leyline is dialing for the request keeps dialing. When it is ready, the pool keeps it for the next request.
• A request that waits for a HostLimits slot leaves the wait, and one that holds a slot gives it back. See Crawling.
• A download dropped before its commit starts removes its temporary .part file and leaves the target path as it was. The commit starts on a blocking thread after the body is written and synced. Once it has started, it renames the file over the target even if the future is dropped.
• A request with a streaming body stops polling the body. On HTTP/2 and HTTP/3, Leyline resets the stream with CANCEL. It polls the body only after the request starts and buffers at most 256 KiB ahead of the peer’s flow-control window.

Drop a streamed response

With .stream(), send() returns at the response head. Drop the Response or its BodyStream to stop the body. A drop is a cheap way to skip a body you do not want.

#![allow(unused)]
fn main() {
use futures_util::StreamExt;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let resp = session.get("https://example.com/file").stream().await?;
if !resp
    .header("content-type")
    .is_some_and(|value| value.starts_with("text/html"))
{
    return Ok(());
}
let mut body = resp.into_stream()?;
let mut seen = 0usize;
while let Some(chunk) = body.next().await {
    seen += chunk?.len();
    if seen > 1024 * 1024 {
        break;
    }
}
Ok(())
}
}

Cancel from your code

Prefer a timeout: .timeout(duration) bounds the request, and TimeoutConfig::body bounds a streamed body. See Retries and timeouts. To stop a request on another event, such as a shutdown signal, use tokio::select!; the branch that loses is dropped.

#![allow(unused)]
fn main() {
use tokio::sync::oneshot;

async fn run(shutdown: oneshot::Receiver<()>) -> leyline::Result<()> {
let session = leyline::Session::new();
tokio::select! {
    result = session.get("https://example.com/feed").send() => {
        println!("{}", result?.status());
    }
    _ = shutdown => println!("shutting down"),
}
Ok(())
}
}

To stop every request of a session and its clones at once, call Session::shutdown(); see Sessions.

A dropped request future returns no error, because nothing polls it, and the Trace::summary hook does not fire for it. See Logging and tracing.

Redirects

A session follows a 301, 302, 303, 307, or 308 response that has a Location header, up to 10 per request. This chapter covers what a redirect changes, how to limit or stop redirects, and how to decide each one.

What a redirect changes

• A redirect that keeps a streamed request body fails with Kind::Redirect, because the stream cannot be sent twice.
• A Location with a scheme other than http or https fails with Kind::Redirect.
• When the redirect leaves the origin of the first request, the session removes the Authorization, Proxy-Authorization, and Cookie headers you set. The cookie jar adds the cookies that match the new URL.

A request that follows a URL the server chose, such as a pages() next link, Tab::follow, or Tab::submit_form, drops the same credentials when its origin differs from the page it came from: the Authorization, Cookie, and Proxy-Authorization session defaults and the session bearer token.

Response::url is the final URL, and Response::redirect_chain lists the URLs the session left, in order. See Responses.

Limit or stop redirects

RedirectPolicy::limited(n) sets the limit; limited(10) is the default. RedirectPolicy::none() follows no redirect. At the limit, the session returns the last 3xx response, not an error. Set the policy on the session with SessionBuilder::redirect, or on one request with RequestBuilder::redirect. Session::execute reads a RedirectPolicy from the request extensions, and Session::with_redirect derives a session with another policy.

#![allow(unused)]
fn main() {
use leyline::{Browser, RedirectPolicy, Session};

async fn run() -> leyline::Result<()> {
let session = Session::builder()
    .browser(Browser::default())
    .redirect(RedirectPolicy::limited(3))
    .build()?;
let resp = session
    .post("https://example.com/login")
    .redirect(RedirectPolicy::none())
    .await?;
println!("{} {:?}", resp.status(), resp.header("location"));
Ok(())
}
}

Decide each redirect

RedirectPolicy::custom calls your function for each redirect with a RedirectAttempt: the status, the current URL, the Location value as a string, and the URLs followed so far. The URLs carry no user name or password. Return RedirectAction::Follow or RedirectAction::Stop; on Stop, the session returns the 3xx response. A custom policy follows at most 32 redirects; the next one fails with Kind::Redirect.

#![allow(unused)]
fn main() {
use leyline::{Browser, RedirectAction, RedirectPolicy, Session};

fn run() -> leyline::Result<()> {
let policy = RedirectPolicy::custom(|attempt| {
    let same_host = attempt
        .location
        .and_then(|location| attempt.url.join(location).ok())
        .is_some_and(|next| next.host_str() == attempt.url.host_str());
    if same_host && attempt.previous.len() < 5 {
        RedirectAction::Follow
    } else {
        RedirectAction::Stop
    }
});
let session = Session::builder()
    .browser(Browser::default())
    .redirect(policy)
    .build()?;
let _ = session;
Ok(())
}
}

Retries and timeouts

A RetryPolicy decides whether a whole request runs again, and a TimeoutConfig bounds how long each part of it may take. Retries are off by default: RetryPolicy::default() is RetryPolicy::none(), so a request is sent once unless you ask for more. Every timeout except total and connect is off by default.

The transient policy

RetryPolicy::transient() covers the failures that usually pass on their own: connection errors, timeouts, and status 429, 502, 503, and 504. It allows 3 retries, so 4 attempts in total, with exponential backoff from 100 ms to 1 s, factor 2, and full jitter. A server can ask for a wait of up to 60 s; max_retry_after changes that cap.

#![allow(unused)]
fn main() {
use leyline::RetryPolicy;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::builder()
    .retry(RetryPolicy::transient())
    .build()?;
let resp = session
    .get("https://example.com/flaky")
    .retry(RetryPolicy::transient().max_retries(4))
    .await?;
let _ = resp;
Ok(())
}
}

SessionBuilder::retry sets the session default. RequestBuilder::retry replaces it for one request.

Custom triggers

RetryTrigger has four variants:

A CONNECT answer ends the request with an error, not a response, so only ConnectionError matches it. Other CONNECT answers, such as 407, and a SOCKS5 reply from 3 to 6, are not retried. No other error is retried.

Start from none() or transient(). retry_on(triggers) replaces the trigger set and on_status(code) adds one status. Every other setting has a setter named after it: max_retries, initial_backoff, max_backoff, backoff_factor, jitter, max_retry_after, and allow_non_idempotent.

#![allow(unused)]
fn main() {
use leyline::{RetryPolicy, RetryTrigger};
use std::time::Duration;

let policy = RetryPolicy::none()
    .max_retries(5)
    .initial_backoff(Duration::from_millis(50))
    .max_backoff(Duration::from_secs(2))
    .backoff_factor(1.5)
    .jitter(true)
    .retry_on([RetryTrigger::ServerError, RetryTrigger::ConnectionError]);

let also = RetryPolicy::transient().on_status(408);
let _ = (policy, also);
}

Backoff for attempt n is initial_backoff * backoff_factor.powi(n), capped at max_backoff. With jitter set, the result is multiplied by a uniform random factor between 0 and 1 (full jitter). RetryPolicy::backoff(attempt) returns that value, so a retry loop of your own can wait the same way. With jitter on, each call draws a new factor.

#![allow(unused)]
fn main() {
use leyline::RetryPolicy;
use std::time::Duration;

let policy = RetryPolicy::transient().jitter(false);
assert!(policy.backoff(1) > policy.backoff(0));
assert_eq!(policy.backoff(10), Duration::from_secs(1));
}

Retry-After wins

When a retried response carries Retry-After, its value replaces the computed backoff. Leyline reads delta-seconds and an IMF-fixdate with a GMT or UTC zone. An unparsable value falls back to the computed backoff. A header set with wait_header wins over Retry-After.

max_retry_after caps the wait. RetryPolicy::transient() sets it to 60 s, and RetryPolicy::none() has no cap. When the server asks for longer than the cap, or a wait or backoff is longer than the time left before the total timeout, Leyline stops and returns the response, or the last error, as when retries run out. Error::retries_exhausted() is then true.

A Retry-After on a status that no trigger or retry_if matches is not acted on. Read it after error_for_status() with Error::retry_after():

#![allow(unused)]
fn main() {
async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
match session.get("https://api.example/jobs").error_for_status().await {
    Ok(resp) => println!("{}", resp.status()),
    Err(err) => match err.retry_after() {
        Some(wait) => println!("come back in {wait:?}"),
        None => return Err(err),
    },
}
Ok(())
}
}

See Status codes are not errors.

Retry on a condition and wait on a header

retry_if(predicate) adds a test on the response head: the status and the headers. A response is retried when a trigger matches its status or when any predicate returns true.

wait_header(name, format) names a header that says how long to wait. WaitFormat says how to read it:

Leyline reads the wait_header headers in the order you added them, then Retry-After, and uses the first value that parses. max_retry_after caps these waits too. They apply to every retried response, whichever trigger or predicate matched it. A retry after an error has no response, so it uses the computed backoff.

This policy retries a 403 that reports a used-up quota, and only that one, after the reset time:

#![allow(unused)]
fn main() {
use std::time::Duration;

use leyline::{RetryPolicy, WaitFormat};
use leyline::http::StatusCode;

const REMAINING: &str = "x-ratelimit-remaining";
const RESET: &str = "x-ratelimit-reset";

async fn run() -> leyline::Result<()> {
let policy = RetryPolicy::transient()
    .retry_if(|resp| {
        resp.status() == StatusCode::FORBIDDEN && resp.header(REMAINING) == Some("0")
    })
    .wait_header(RESET, WaitFormat::UnixSeconds)
    .max_retry_after(Duration::from_secs(120));
let session = leyline::Session::builder().retry(policy).build()?;
let resp = session.get("https://api.example/repos").error_for_status().await?;
let _ = resp;
Ok(())
}
}

A policy with max_retries(0), such as none(), never retries, so start from transient() or set max_retries.

Blocks, skip_blocks, and retries through other proxies are covered in Crawling.

The idempotency rule

Leyline retries only idempotent methods by default: GET, HEAD, OPTIONS, PUT, DELETE, and TRACE (RFC 9110, section 9.2.2). A POST or PATCH is sent once. A retry also needs a body that can be sent again. A streaming request body is retried only when no byte of it was read, for example after a connect error. See Streaming.

retry_unsent(true) retries any method when the error happened before the request was sent: a DNS, connect, TLS, or proxy error, a connect timeout, an HTTP/2 REFUSED_STREAM reset, or an HTTP/3 request that the server reports it did not process. An HTTP/3 request that the server rejects twice is retried on the same connection. The server never processed the request, so a retry cannot repeat a write. The error must still match a trigger. After a redirect got a response, an error on a later leg does not count: the server may already have processed the original request, so retry_unsent does not replay it. A 401 digest challenge is not an answer, so an error on the authorized leg still counts.

#![allow(unused)]
fn main() {
use leyline::RetryPolicy;

fn run() -> leyline::Result<()> {
let policy = RetryPolicy::transient().retry_unsent(true);
let session = leyline::Session::builder().retry(policy).build()?;
drop(session);
Ok(())
}
}

allow_non_idempotent(true) retries every failure of any method. Use it only when the endpoint is safe to repeat.

Count the attempts

Response::attempts() returns the number of attempts that produced the response: 1 without retries. When retries run out on a retryable status, the response carries the full count. Error::attempts() and trace::Summary::attempts carry the same count. See Logging and tracing.

Retries and streamed responses

A .stream() request uses the same policy, and the decision is made on the response head. After send() returns, a failure while you read the body is not retried. To try again, send a new request.

Resends outside the policy

Connection setup has no retry loop of its own. These cases send a request or connect again without asking the policy. Each happens at most once for one attempt, and the result then goes to RetryPolicy like any other:

• HTTP/1.1: a pooled keep-alive connection fails before the response. An idempotent request with a buffered or empty body goes once more on a new connection.
• HTTP/2: a pooled connection fails before the response. A request with a buffered or empty body goes once more on a new connection when the method is idempotent, or for any method after a REFUSED_STREAM reset.
• HTTP/3: a pooled connection reports that it did not send the request, for example after GOAWAY. A request with a buffered or empty body goes once more on a new connection, for any method. After an H3_REQUEST_REJECTED reset before the response head, a buffered request goes once more on the same connection.
• ProtocolPolicy::Race: when both the HTTP/3 and the HTTP/2 connect fail, Leyline falls back to Auto and connects again.

ALPN needs no second send. See One handshake per connection.

The timeouts

TimeoutConfig holds every timeout. Each setter takes a Duration or None, and None turns that timeout off.

#![allow(unused)]
fn main() {
use leyline::{Session, TimeoutConfig};
use std::time::Duration;

fn run() -> leyline::Result<()> {
let session = Session::builder()
    .timeout(
        TimeoutConfig::new()
            .total(Duration::from_secs(60))
            .connect(Duration::from_secs(5))
            .read(Duration::from_secs(15))
            .response_header(Duration::from_secs(20))
            .body(Duration::from_secs(120)),
    )
    .build()?;
let _ = session;
Ok(())
}
}

SessionBuilder::timeout takes a Duration or a TimeoutConfig. A Duration sets total, and the other settings keep their defaults. A later timeout call replaces the whole configuration, so to change connect alone, pass TimeoutConfig::new().connect(duration).

Every timeout gives an error whose is_timeout() is true. The kind depends on which one elapsed; see Fine-grained kinds.

Per-request timeouts

RequestBuilder::timeout merges over the session timeouts, setting by setting. A setting that the request sets wins, None included, and the rest keep the session value. A Duration sets total. connect stays session-wide, because connections are pooled across requests.

#![allow(unused)]
fn main() {
use leyline::TimeoutConfig;
use std::time::Duration;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();

let quick = session
    .get("https://example.com/slow")
    .timeout(Duration::from_secs(5))
    .await?;

let streamed = session
    .get("https://example.com/feed")
    .stream()
    .timeout(
        TimeoutConfig::new()
            .total(None)
            .read(Duration::from_secs(10))
            .response_header(Duration::from_secs(15)),
    )
    .await?;

let _ = (quick, streamed);
Ok(())
}
}

Bound a whole download

TimeoutConfig::body bounds the whole body from its first read, so it bounds a .stream() body and RequestBuilder::download, which streams. A plain Duration passed to .timeout sets only total, which ends at the head of a streamed response, so set body for a download. No outer timeout is needed.

#![allow(unused)]
fn main() {
use std::time::Duration;

use leyline::TimeoutConfig;

async fn run(session: leyline::Session) -> leyline::Result<()> {
let written = session
    .get("https://example.com/archive.zip")
    .timeout(TimeoutConfig::new().body(Duration::from_secs(600)))
    .download("archive.zip", None)
    .await;
match written {
    Ok(bytes) => println!("{bytes} bytes"),
    Err(err) if err.is_timeout() => println!("download took too long"),
    Err(err) => return Err(err),
}
Ok(())
}
}

A failed or cancelled download removes its temporary .part file and leaves the target path as it was. See Streaming and Cancellation.

Next

Read Errors to match on ErrorCategory and Kind.

Errors

Every call returns leyline::Result<T>, and every failure is a leyline::Error. Start with err.category(), which sorts each error into one ErrorCategory. Use err.kind() only for the finer detail of Fine-grained kinds.

leyline::Error implements std::error::Error + Send + Sync + 'static, so ? converts it into Box<dyn std::error::Error + Send + Sync> or anyhow::Error. It also converts from url::ParseError, as Kind::Url.

Group errors by category

err.category() picks the first group that applies, in this order:

ErrorCategory is #[non_exhaustive], so a match needs a _ arm.

#![allow(unused)]
fn main() {
use leyline::ErrorCategory;

async fn run() {
let session = leyline::Session::new();
if let Err(err) = session.get("https://api.example/").await {
    match err.category() {
        ErrorCategory::Timeout | ErrorCategory::Dns | ErrorCategory::Connect => {
            eprintln!("network: {err}")
        }
        ErrorCategory::Proxy | ErrorCategory::ProxyTarget => eprintln!("proxy: {err}"),
        ErrorCategory::Url | ErrorCategory::Config | ErrorCategory::Request => {
            eprintln!("fix the request: {err}")
        }
        _ => eprintln!("request failed: {err}"),
    }
}
}
}

• Config: a value or a combination of settings cannot work, for example a forced HTTP/3 policy with a proxy that cannot carry HTTP/3, or a browser with no identity for the platform. SessionBuilder::build() also returns it when a proxy environment variable is not a valid proxy URL.
• Connect: the TCP connect failed, or the socket was refused or unreachable. The connect window also covers proxy negotiation, the TLS handshake, and on HTTP/3 the QUIC handshake.
• Dns: name resolution failed, and is_dns() is true.
• Proxy, ProxyTarget: the proxy failed, or it could not reach the origin. See Proxy errors.
• Tls: the TLS handshake or the certificate check failed. A handshake failure can pass on the next try, so is_connect() and is_retryable() are true for it. A certificate, host name, or pin failure stays until the server or your trust settings change, so both are false.
• Decode: a failed content decode, or a Response::json serde failure (Kind::Json).

ErrorCategory::as_str() returns a stable snake_case label, such as "timeout", "proxy_target", or "body_limit", for logs, metrics, and client-facing error bodies. Display prints the same label.

Display of an Error prints the kind, status, message, and URL once. Walk std::error::Error::source() for the cause.

Attempts and the proxy used

err.attempts() returns the number of attempts, retries included, and 0 for an error that no request produced, such as a build() failure. err.proxy() returns the proxy URL of the last attempt, with the password redacted, or None for a direct connection. Both are set for status errors too.

#![allow(unused)]
fn main() {
async fn run() {
let session = leyline::Session::new();
if let Err(err) = session.get("https://api.example/").error_for_status().await {
    eprintln!(
        "{} after {} attempts via {}: {err}",
        err.category().as_str(),
        err.attempts(),
        err.proxy().unwrap_or("direct"),
    );
}
}
}

Status codes are not errors

A 4xx or 5xx response is Ok. To get a status of 400 or more as an error, call error_for_status() on the request builder. A failed send and a bad status then arrive as one Result. The check runs after retries, so it sees the last response, and the error keeps the start of the decoded body, within the limits in Turn a status into an error.

retry_after() returns the wait that the retry policy read, each wait_header first and then Retry-After, when the error comes from the request-builder error_for_status(). Otherwise it parses Retry-After, in seconds or as an HTTP date. It is None for an error that is not a status error. See Retries and timeouts.

retries_exhausted() is true when the last response still matched a retry trigger and the policy stopped: no retries were left, the server asked for a wait longer than max_retry_after, or the wait or backoff was longer than the time left before the total timeout.

Only a Kind::Status error carries a URL and headers. Display and Debug show the URL with the password and the query hidden.

#![allow(unused)]
fn main() {
use leyline::http::StatusCode;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
match session.get("https://api.example/users/7").error_for_status().await {
    Ok(resp) => println!("found: {}", resp.text().await?),
    Err(e) if e.status() == Some(StatusCode::NOT_FOUND) => println!("no such user"),
    Err(e) => return Err(e),
}
Ok(())
}
}

Turn a status into an error

For a response you already hold, Response::error_for_status() consumes the response and error_for_status_ref() borrows it. Both errors keep the status, URL, and headers, and no body. See Responses.

Predicates

The predicates look at the kind and at the source error:

A RetryPolicy uses is_timeout() and is_retryable() to pick its triggers. See Retries and timeouts.

Map errors to HTTP statuses

A service that calls an upstream through Leyline must answer its own client. ErrorCategory::gateway_status() gives a default answer:

#![allow(unused)]
fn main() {
use leyline::http::StatusCode;

fn to_status(err: &leyline::Error) -> StatusCode {
    err.category()
        .gateway_status()
        .or_else(|| err.status())
        .unwrap_or(StatusCode::BAD_GATEWAY)
}
let _ = to_status;
}

BodyLimit maps to 502, not to 413: the upstream response passed the limit, not the client’s request. Log Proxy and ProxyTarget apart from upstream failures, because they point at your proxy. See Service integration for a full service.

Fine-grained kinds

err.kind() returns a Kind, the layer that failed. The API map lists each Kind. Kind is #[non_exhaustive], and Kind::as_str() returns a stable lowercase label.

Because category() tests the timeout, the status, the body limit, and the TLS-layer cause first, one kind can give more than one category:

Any kind whose I/O cause is TimedOut gives Timeout. A timeout has more than one kind:

• A connect timeout is Kind::Connect, on HTTP/1.1, HTTP/2, and HTTP/3.
• A total or response_header timeout is Kind::Timeout.
• A read stall on a .stream() body is Kind::Timeout from bytes(), text(), and json(), and Kind::Io from copy_to and read_until.

So when you match on the kind, test is_timeout() first:

#![allow(unused)]
fn main() {
use leyline::{Kind, Session};

async fn run() {
let session = Session::new();
match session.get("https://example.com/").await {
    Ok(resp) => println!("{}", resp.status()),
    Err(err) if err.is_timeout() => eprintln!("timed out: {err}"),
    Err(err) => match err.kind() {
        Kind::Connect | Kind::Tls | Kind::Proxy => eprintln!("no connection: {err}"),
        Kind::Config | Kind::Url => eprintln!("fix the request: {err}"),
        _ => eprintln!("request failed: {err}"),
    },
}
}
}

Read the source error

err.tls() returns the TlsError, err.h2() the H2Error, and err.io() the std::io::Error, when that is the source. Match TlsError::Rejected for a peer that closed or reset the handshake, and TlsError::Certificate { verify_code, reason, .. } for a failed certificate check. TlsError::Proxy and TlsError::ProxyTargetUnreachable are described in Proxy errors.

Error::find(error) walks an error’s source() chain and returns the first leyline::Error in it, so an error that a tower layer or your own type wraps still maps through category().

Next

Read Proxies to set proxy rules, bypass lists, and environment discovery.

Proxies

Leyline tunnels through http://, https://, socks5://, and socks5h:// proxies. This page sets one proxy, rules per scheme, bypass lists, and environment discovery, then rotates proxies with a retry list or a ProxyPool, and tells a proxy failure from an origin failure.

An http:// proxy receives the CONNECT request and any Proxy-Authorization credentials in cleartext; an https:// proxy receives them inside TLS. A plain http:// URL goes to an http:// or https:// proxy as an absolute-form request, and through a SOCKS5 proxy over a CONNECT tunnel.

Set one proxy

Every proxy setter takes impl Into<ProxyConfig>: a URL string, a ProxyUrl, or a ProxyConfig. That covers SessionBuilder::proxy, RequestBuilder::proxy, WebSocketBuilder::proxy, and Session::with_proxy. A URL applies to every scheme.

#![allow(unused)]
fn main() {
fn run() -> leyline::Result<()> {
let session = leyline::Session::builder()
    .proxy("http://user:pass@proxy.example:8080")
    .build()?;
let _ = session;
Ok(())
}
}

build() validates the session proxy. A URL string given to a request, a WebSocket, or with_proxy is validated when a request picks it, and an invalid one fails that send() with Kind::Config. To fail early, pass a ProxyUrl: ProxyUrl::parse refuses any other scheme and a URL with no host.

#![allow(unused)]
fn main() {
use leyline::ProxyUrl;

assert!(ProxyUrl::parse("http://proxy.example:8080").is_ok());
assert!(ProxyUrl::parse("ftp://proxy.example").is_err());
}

Debug of ProxyUrl, ProxyRule, and Session, and Display of ProxyUrl, replace the password with ***. ProxyUrl implements Serialize and Deserialize, and serde keeps the full URL, password included, so protect a file that holds one.

SOCKS5 and HTTPS proxies

A socks5:// or socks5h:// proxy needs the socks feature, which is off by default. With either scheme the proxy resolves the host name.

[dependencies]
leyline-http = { version = "0.1", features = ["socks"] }

Without the feature, ProxyUrl::parse and build() refuse a SOCKS URL with Kind::Config, whether it is the session proxy, a ProxyPool entry, or the proxy from the environment. A SOCKS URL string given to a request or to with_proxy fails the first request that uses it. The message names the feature.

An https:// proxy means TLS to the proxy, then CONNECT through it. A session with certificate pins or a client certificate cannot use one, because the origin TLS identity must not go to the proxy: build() accepts the combination, and every request routed through the https:// proxy fails with Kind::Proxy. See TLS trust.

Rules per scheme

ProxyConfig holds a list of ProxyRules. A rule applies to every scheme (ProxyRule::all), to http only, or to https only. The first matching rule wins, in the order you added them. A wss:// WebSocket URL matches as https://; a ws:// URL fails with Kind::Request before any proxy is chosen. See WebSocket.

#![allow(unused)]
fn main() {
use leyline::{ProxyConfig, ProxyRule, Session};

fn run() -> leyline::Result<()> {
let proxies = ProxyConfig::new()
    .rule(ProxyRule::https("http://secure-proxy.example:8080"))
    .rule(ProxyRule::http("http://plain-proxy.example:3128"));
let session = Session::builder().proxy(proxies).build()?;
let _ = session;
Ok(())
}
}

Bypass with NO_PROXY

NoProxy::new(patterns) matches hosts that must not go through a proxy.

#![allow(unused)]
fn main() {
use leyline::{NoProxy, ProxyConfig, ProxyRule};

let bypass = NoProxy::new(["localhost", ".internal.example", "192.0.2.1"]);
let proxies = ProxyConfig::new()
    .rule(ProxyRule::all("http://proxy.example:8080"))
    .no_proxy(bypass);
let _ = proxies;
}

A pattern matches the host and every subdomain of it. A leading dot is optional, a trailing :port is stripped, and * matches every host. Matching ignores case and a trailing dot on the host.

The bypass list applies when you set it with ProxyConfig::no_proxy, and when the session proxy came from the environment. A per-request proxy() replaces the session config, bypass list included.

Environment discovery

A session with no explicit proxy reads the environment once, when it is built. It takes the first non-empty value of HTTPS_PROXY, https_proxy, HTTP_PROXY, http_proxy, ALL_PROXY, all_proxy, and applies it to every URL scheme. NO_PROXY or no_proxy seeds the bypass list.

A value that is not a valid proxy URL never falls back to a direct connection. A valid value has a scheme of http, https, socks5, or socks5h (the last two only with the socks feature) and a host, so proxy.example:8080 is invalid. Leyline checks only the value it picked and does not try a lower variable. The error names the variable, never the value.

• SessionBuilder::build() returns Kind::Config.
• Session::new() and Session::browser(..) return a session in which every request fails with Kind::Proxy, hosts in NO_PROXY included.

Fix or unset the variable, set a proxy with SessionBuilder::proxy, or turn discovery off. A proxy set on a request, a WebSocket, or with with_proxy replaces the failed state too.

#![allow(unused)]
fn main() {
fn run() -> leyline::Result<()> {
let session = leyline::Session::builder()
    .proxy(leyline::ProxyConfig::new().env(false))
    .build()?;
let _ = session;
Ok(())
}
}

Uppercase HTTP_PROXY is skipped when the process looks like a CGI handler (the httpoxy mitigation): when any of GATEWAY_INTERFACE, REQUEST_METHOD, SERVER_SOFTWARE, SCRIPT_NAME, SCRIPT_FILENAME, PATH_INFO, QUERY_STRING, SERVER_PROTOCOL, SERVER_NAME, or SERVER_PORT is set. Leyline then logs a warning on the leyline::env_proxy::cgi target.

Per-request override

RequestBuilder::proxy replaces the session proxy config for one request. A ProxyConfig::new() with no rule sends that request direct. Session::with_proxy derives a session that differs only in its proxy. It shares the cookie jar and the connection pool, and keeps the TLS context and the identity. See Sessions.

The pool is keyed by host, port, and proxy, so two proxies never share a connection, and a return to a proxy reuses its warm connections. Session::fresh_pool derives a session with a new, empty pool and TLS session cache, so the next request opens a new socket through the same proxy.

#![allow(unused)]
fn main() {
async fn run(proxy: Option<&str>) -> leyline::Result<()> {
let session = leyline::Session::new();
let mut request = session.get("https://example.com/ip");
if let Some(p) = proxy {
    request = request.proxy(p);
}
let resp = request.await?;
let rotated = session.with_proxy("http://gateway.example:8080").fresh_pool();
let _ = (resp, rotated);
Ok(())
}
}

Rotate proxies on retry

RetryPolicy::rotate_proxies(list) moves each retry to the next proxy in a list. The first attempt uses the request’s or the session’s proxy. Retry k, counted from 1, uses list[(k - 1) % list.len()]. On a session with a ProxyPool, the pool picks the proxy and rotate_proxies is ignored.

#![allow(unused)]
fn main() {
use leyline::RetryPolicy;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let policy = RetryPolicy::transient()
    .on_status(403)
    .rotate_proxies(["http://proxy-a.example:8080", "http://proxy-b.example:8080"]);
let resp = session
    .get("https://shop.example/item/1")
    .retry(policy)
    .send()
    .await?;
let _ = resp;
Ok(())
}
}

The usual retry rules apply: a trigger must match, the method must be idempotent or allowed, and the body must be replayable. A Retry-After wait still runs before the next proxy. When retries run out you get the last response, for example the 403. See Retries and timeouts.

Use a proxy pool

For bans and stickiness across many requests, give the session a ProxyPool with SessionBuilder::proxy_pool. The pool picks a proxy for each request:

1. The sticky proxy of the origin, if it is not banned and was picked within sticky_for.
2. Else the next healthy proxy, round-robin.
3. If every proxy is banned, the proxy whose ban ends first.

#![allow(unused)]
fn main() {
use std::time::Duration;
use leyline::{BlockRules, ProxyPool, Session};

fn run() -> leyline::Result<()> {
let pool = ProxyPool::new([
    "http://user:pass@proxy-a.example:8080",
    "http://user:pass@proxy-b.example:8080",
])
.sticky_for(Duration::from_secs(300))
.ban_after(3)
.ban_for(Duration::from_secs(60))
.rotate_on_block(BlockRules::statuses([403, 429]));
let session = Session::builder().proxy_pool(pool.clone()).build()?;
let _ = session;
Ok(())
}
}

After ban_after strikes, the proxy is banned for ban_for, and every origin that stuck to it loses its sticky proxy. Each retry picks the next healthy proxy and skips the one that just failed. A request with its own .proxy(..) does not use the pool. Blocks and BlockRules are described in Crawling.

ProxyPool is Clone, and clones share the health state. stats() returns one ProxyHealth per proxy, with proxy (no password), in_use, failures, and banned_until, which can be in the past when the ban has ended.

#![allow(unused)]
fn main() {
use std::time::Instant;

fn run(proxies: leyline::ProxyPool) {
let now = Instant::now();
for health in proxies.stats() {
    let banned = health.banned_until.is_some_and(|until| until > now);
    println!(
        "{} in_use={} failures={} banned={banned}",
        health.proxy, health.in_use, health.failures
    );
}
}
}

Pin one identity to each proxy

ProxyPool::identified takes (proxy, Identity) pairs. A request sent through a proxy uses a session with that proxy’s identity, so one proxy IP never shows two browsers.

#![allow(unused)]
fn main() {
use leyline::{Browser, Family, Identity, Platform, ProxyPool, Session};

fn run() -> leyline::Result<()> {
let proxies = (1..=20).map(|n| format!("http://user:pass@proxy-{n}.example:8080"));
let identities = [
    Identity::locked(Browser::latest(Family::Chrome), Platform::Windows),
    Identity::locked(Browser::latest(Family::Firefox), Platform::MacOS),
];
let pool = ProxyPool::identified(proxies.zip(identities.into_iter().cycle()));
let session = Session::builder()
    .browser(Browser::default())
    .proxy_pool(pool)
    .build()?;
let _ = session;
Ok(())
}
}

On first use of a proxy, the pool derives a session for it from the request’s session with Session::with_identity, and keeps it. That session shares the connection pool (in its own partition) and the host limits, and has its own TLS session cache. Unlike a plain with_identity session, it gets its own cookie jar, which starts empty. Each proxy has its own jar, also when two proxies share an Identity, so cookies set through one proxy never go out through another. When an origin moves to another proxy, its cookies stay in the old proxy’s jar.

The session must impersonate a browser. A session with ProxyPool::identified and no browser, profile, or identity fails at build() with Kind::Config.

Proxy errors

A failure in the proxy dial, the proxy handshake, or CONNECT is Kind::Proxy, and err.tls() returns TlsError::Proxy, whose status holds the proxy’s answer to CONNECT, for example 407.

When the proxy works but cannot reach the origin, the error is also Kind::Proxy, its category is ProxyTarget, and err.tls() returns TlsError::ProxyTargetUnreachable { reply, detail }. reply is a ProxyReply: HttpStatus(502) or HttpStatus(504) for a CONNECT answer, or Socks5(n) for a SOCKS5 reply from 3 to 6. detail holds the text.

RetryTrigger::ConnectionError retries an I/O failure on the way to the proxy and a CONNECT answer of 502, 503, or 504, and nothing else here. See Custom triggers.

Decide whether to ban a proxy

Error::is_proxy() is true when the proxy itself failed, before the tunnel was up:

• The proxy dial was refused or timed out. A timeout is also is_timeout().
• The proxy answered CONNECT with an error such as 407, 403, or 503.
• The SOCKS5 handshake or authentication failed.
• The TLS handshake with an https:// proxy failed.

It is false for ProxyTargetUnreachable and for every error after the tunnel is up, so it takes a proxy out of rotation without blaming it for the origin. A ProxyPool applies this rule for you.

#![allow(unused)]
fn main() {
use leyline::TlsError;

async fn run(session: leyline::Session, proxy: &str) -> leyline::Result<()> {
match session.get("https://example.com/").proxy(proxy).send().await {
    Ok(resp) => println!("{}", resp.status()),
    Err(err) if err.is_proxy() => println!("ban {proxy}: {err}"),
    Err(err) => match err.tls() {
        Some(TlsError::ProxyTargetUnreachable { reply, .. }) => {
            println!("origin unreachable through the proxy: {reply:?}")
        }
        _ => println!("origin error: {err}"),
    },
}
Ok(())
}
}

HTTP/3 through a SOCKS5 proxy

With the socks feature, HTTP/3 runs over a SOCKS5 proxy. Leyline opens a TCP control connection, authenticates, and sends UDP ASSOCIATE (RFC 1928, section 7). Each QUIC datagram goes to the relay with the SOCKS5 UDP header, which names the target host, so the proxy resolves it. When the control connection closes, the QUIC connection fails and leaves the pool. The connect timeout covers the control connection, UDP ASSOCIATE, and the QUIC handshake.

An http:// or https:// proxy cannot carry HTTP/3, and neither can a SOCKS proxy without the socks feature. build() rejects ProtocolPolicy::Http3 when such a proxy takes every URL, and an Http3 request fails with Kind::Config when the rules pick such a proxy for its URL. Under ProtocolPolicy::Race, a request through such a proxy takes the Auto path. MASQUE (CONNECT-UDP, RFC 9298) is not supported.

Many proxies do not support UDP ASSOCIATE. When an HTTP/3 connection through a proxy fails, Race skips HTTP/3 for that origin and proxy for 5 minutes. To keep every request on TCP, set ProtocolPolicy::Auto. See HTTP/3.

Next

Read Cookies.

Crawling

Session::crawl crawls a site for you: it follows links in scope, obeys each page’s own rules, drops copies, and reads robots.txt when you ask it to. The rest of this page covers the session settings every crawl uses: per-host limits, a proxy pool, retries, block detection, capped downloads, one log line per request, and counters for the whole crawl. Every task that shares the session shares the limits and the proxy health.

Crawl a site

session.crawl(seed) returns a Crawl, a lazy stream of CrawlPages. The defaults are polite, so the short form is a complete crawler:

#![allow(unused)]
fn main() {
use leyline::Session;

async fn run() -> leyline::Result<()> {
let mut crawl = Session::new().crawl("https://docs.example/");
while let Some(page) = crawl.next().await {
    let page = page?;
    println!("{} {} words", page.url(), page.text().split_whitespace().count());
}
Ok(())
}
}

That crawl stays on the seed’s host, sends one request at a time to each host and at most one a second, pauses a host that answers 429 or 503 and tries that page up to three times, reads the site’s sitemaps, follows redirects through the same checks as links, skips error pages, files, and block pages, and drops copies. It obeys robots.txt and each page’s robots rules unless you call ignore_robots(). Five settings change the rest:

#![allow(unused)]
fn main() {
use leyline::{HostLimits, Respect, Scope, Session};

async fn run() -> leyline::Result<()> {
let mut crawl = Session::new()
    .crawl("https://docs.example/guide/")
    .respect(Respect::AiTraining)
    .scope(Scope::Prefix)
    .limit(500)
    .host_limits(HostLimits::new().max_in_flight(2).per_second(2.0));
while let Some(page) = crawl.next().await {
    match page {
        Ok(page) => println!("{}", page.url()),
        Err(err) => eprintln!("{}: {}", err.url().map_or("-", |url| url.as_str()), err.category()),
    }
}
println!("{}", crawl.stats());
Ok(())
}
}

Unless you call ignore_robots(), the crawl reads each site’s robots.txt once, and again after 24 hours, obeys its rules, and reads its Sitemap lines. Robots groups are matched for the * agent. To read the group for a named crawler, call Session::robots(url, agent). At most 16 requests are in flight across all hosts.

For each URL, in order, the crawl:

1. Skips a URL that is not http or https, is out of scope, or was queued before. URLs are compared after the fragment is dropped, the query parameters are sorted, and percent-encoding is normalized; the request goes out as written.
2. Unless robots are ignored, skips a URL that robots.txt disallows, and uses the site’s Crawl-delay as the least spacing between its requests, up to 30 seconds. With Respect::AiTraining, it skips a URL whose site refuses AI training in a Content-Signal line (Cloudflare’s format) or a Content-Usage line (the IETF aipref draft).
3. Sends the request without following redirects. A redirect is queued like a link, so its target passes the same checks; a seed redirected to another host of the same site adds the scope of its target, and a seed redirected to another site yields only that page. A chain longer than five redirects is skipped.
4. Skips a block page (Response::block()), an error status, and a response whose Content-Type is not HTML. A response without a Content-Type is kept when its body starts like HTML, by the MIME Sniffing rules.
5. Queues the page’s links. Unless robots are ignored, it reads the page’s directives with html::Directives::read, queues nothing from a nofollow page, and skips links marked rel="nofollow".
6. Unless robots are ignored, skips a noindex page. With Respect::AiTraining, it also skips a noai page and a page whose Content-Usage header refuses AI training. noai is a convention some sites use, not a standard. A page that names another in-scope page as its canonical URL waits until the crawl has that page: it is dropped when the canonical page is kept, and kept when the canonical page is skipped or cannot be fetched. A held page counts toward limit; when the limit leaves no room to fetch the canonical page, the held page is kept. A page of 20 words or more is skipped when its text, compared word by word, matches a kept page.

The crawl reads the sitemaps that each site’s robots.txt lists on the same site, or /sitemap.xml when it lists none (and, unless robots are ignored, when robots.txt allows it). Unless robots are ignored, it opens only sitemap files that their own site’s robots.txt allows, and reads that robots.txt first for a sitemap on another host. It follows a sitemap index to the sitemaps it names on the same host, reads gzipped and plain-text sitemaps, and reads at most 50 sitemap files and 50 MiB per file; with a limit, it opens no new sitemap file once it has queued that many URLs. A sitemap that redirects is queued again at its new address and passes the same checks. Each <loc> URL passes the same checks as a link; image, video, and news <loc> entries in a sitemap’s extension namespaces are not pages and are skipped.

A request that fails comes out of the stream as an Err, and the crawl goes on. crawl.stats() returns CrawlStats: requests(), pages(), errors(), and skipped(SkipReason) for each reason above. Its Display prints one key=value line. CrawlPage has url(), the address after redirects, depth(), status(), html(), text() from html::text, and meta(name). A page body is capped by the session’s CompressionConfig::max_body_size, 100 MiB by default.

Session::shutdown() ends the crawl: requests in flight fail with the shut-down error and nothing new starts. Dropping the Crawl cancels its requests. Session::crawl needs the html feature, which is on by default.

Build the crawl session

#![allow(unused)]
fn main() {
use std::time::Duration;
use leyline::trace::{Fanout, Metrics, TracingTrace};
use leyline::{
    BlockRules, Browser, HostLimits, ProxyPool, RetryPolicy, Session, TimeoutConfig,
};

fn run() -> leyline::Result<()> {
let blocks = {
    let mut rules = BlockRules::builtin().clone();
    rules.extend(BlockRules::statuses([403, 429]));
    rules
};
let proxies = ProxyPool::new([
    "http://user:pass@proxy-a.example:8080",
    "http://user:pass@proxy-b.example:8080",
    "http://user:pass@proxy-c.example:8080",
])
.sticky_for(Duration::from_secs(600))
.ban_after(3)
.ban_for(Duration::from_secs(120))
.rotate_on_block(blocks.clone());
let metrics = Metrics::new();

let session = Session::builder()
    .browser(Browser::default())
    .host_limits(
        HostLimits::new()
            .max_in_flight(4)
            .per_second(2.0)
            .max_total_in_flight(64)
            .pause_on([429, 503]),
    )
    .proxy_pool(proxies.clone())
    .retry(RetryPolicy::transient().skip_blocks(blocks))
    .timeout(TimeoutConfig::new().body(Duration::from_secs(60)))
    .trace(Fanout::new().with(TracingTrace).with(metrics.clone()))
    .build()?;
let _ = session;
Ok(())
}
}

Clone the session into each worker task. A clone is an Arc clone: it shares the connection pool, the cookie jar, the host limits, and the proxy pool. TimeoutConfig::body bounds each body from its first read, so a slow page fails with a timeout error. See Retries and timeouts.

Limit each host

HostLimits applies per origin (scheme, host, and port), across HTTP/1.1, HTTP/2, and HTTP/3. A host name with a trailing dot is the same host as the name without it. HostLimits::default() has no limits.

HostLimits is configuration. Each session that you build from it keeps its own counters, queues, and pauses. Clones and derived sessions of that session share them; a second session built from the same HostLimits value does not.

The server’s wait comes from the same headers the retry policy reads: each RetryPolicy::wait_header, then Retry-After. See Retries and timeouts.

#![allow(unused)]
fn main() {
use leyline::HostLimits;

let limits = HostLimits::new()
    .max_in_flight(4)
    .per_second(2.0)
    .host("api.shop.example", HostLimits::new().max_in_flight(1).per_second(0.5))
    .host("*.cdn.example", HostLimits::new().max_in_flight(16))
    .max_total_in_flight(64);
let _ = limits;
}

An override uses only the max_in_flight and per_second you pass; a setting it leaves out has no limit for that host. The global cap applies to every host.

A request is admitted in this order:

1. It waits for an in-flight slot of its origin.
2. It waits while the origin is paused, then for the origin’s next rate slot.
3. It waits for a global slot. A request that loses its turn while it waits for the global slot goes back to step 2.

No request holds a global slot while it waits for a pause or the rate spacing. The rate spacing is measured when a request is admitted, and a pause is checked again just before admission.

The wait counts against the request deadline, and ends in Kind::Timeout when the deadline passes. Each retry waits its turn again. Each redirect hop is admitted against the origin it goes to, and releases the slot of the hop before. pause_on and the proxy pool’s strikes apply to the origin that sent the response, which after a redirect is not the origin of the first URL. A buffered request releases its slot when its body has been read; a .stream() request keeps it until the body ends, fails, or is dropped. A streamed body ends at its first error and releases the slot. A failed attempt releases its slot before the retry wait. Dropping a waiting request removes it from the queue.

max_in_flight counts requests to one origin across every proxy. The connection pool then applies PoolConfig::max_h1_conns_per_host to each host, port, scheme, proxy, and transport. For HTTP/1.1 through one proxy, or direct, the smaller limit bounds the requests in flight. HTTP/2 and HTTP/3 multiplex on one connection, so only max_in_flight bounds them. See Connection pool.

Obey robots.txt

Session::robots(url, agent) fetches /robots.txt for the origin of url and returns Robots, the rules for the product token agent. Pass "*" to read only the group for every crawler. The rules follow RFC 9309:

A crawl sends the session’s Authorization header (including bearer_auth) and a Cookie default header only to the seed’s origin; requests to other hosts go without them. Cookies from the session’s cookie jar follow cookie rules, which match by host and path but not by port (RFC 6265, section 8.5). Other default headers go to every host, so do not put a credential in a custom default header for a crawl.

A crawl request waits for the session’s own host limits and the crawl’s pace together, and is sent only when both allow it. A 429 or 503 that the crawl gets pauses the host in both, so the session’s own requests to that host wait out the same pause. A host that is paused, waiting for its next turn, or out of free slots holds no crawl slot. When a page or a sitemap fails in a way the retry policy retries (the session’s RetryPolicy, or three tries on transient errors, 429, and 503 when it has none), the crawl puts it back in the queue and tries again after the policy’s wait, without holding a slot. A redirect that leaves the scope counts as SkipReason::Scope. A URL longer than 2048 bytes, or more than 64 links from the seed, is dropped without a count.

A crawl that obeys robots rules holds a site’s pages until its robots.txt is read, and retries an unreachable robots.txt four times, 1, 2, 4, and 8 seconds apart, before it skips the site. A crawl with ignore_robots() never asks for robots.txt; it reads /sitemap.xml directly. A missing sitemap is not an error; a sitemap that fails after its retries counts in errors().

Up to five redirects are followed, and the rules apply to the origin you asked about. The first 512 KiB of the file are parsed. Session::robots returns an error only for an invalid URL or a shut-down session.

The groups that name agent, compared without case, are merged; with no such group, the * groups apply. A path matches the longest rule, and Allow wins a tie. * matches any run of characters, $ ends a pattern, and both sides are compared with percent-encoding normalized. /robots.txt itself is always allowed.

#![allow(unused)]
fn main() {
use leyline::{Purpose, Url};

async fn run(session: leyline::Session) -> leyline::Result<()> {
let robots = session.robots("https://shop.example/", "examplebot").await?;
let page = Url::parse("https://shop.example/private/report.html")?;
if robots.allows(&page) && robots.allows_use(&page, Purpose::AiTrain) {
    let _ = session.get(page.as_str()).send().await?;
}
println!("delay {:?}, sitemaps {:?}", robots.crawl_delay(), robots.sitemaps());
Ok(())
}
}

Robots::parse(text, agent) reads a file you already have. allows_use(url, purpose) reads the AI-use preferences of the matching group. A Content-Signal line uses search, ai-input, and ai-train with yes or no; a Content-Usage line, from the IETF aipref draft, uses search, ai-use, and train-ai with y or n, and can start with a path pattern that limits it to matching URLs. Both map to Purpose. For a use, the line with the longest path pattern that matches the URL decides; on a tie, no wins, and a use no line names is allowed. Unknown labels are ignored. The aipref labels are a draft and may change. crawl_delay() returns the group’s Crawl-delay; Session::crawl applies it unless robots are ignored. sitemaps() lists every Sitemap line in the file.

Rotate proxies with a pool

A ProxyPool keeps an origin on one proxy, strikes and bans failing proxies, and moves each retry to another healthy proxy. Its settings, strike rules, stats(), and ProxyPool::identified are described in Use a proxy pool.

Detect a block page

A block is a response, not an error. Response::block() checks it against built-in vendor rules and returns a BlockSignal with a vendor and a kind. A bare 403 or 429 is not a block signal.

BlockRules::from_toml adds rules of your own. Each [[rule]] table has vendor and kind (challenge, captcha, or block), and optional status, header, and value. The value match ignores ASCII case. BlockRules::statuses([403, 429]) makes bare statuses count as blocks, with BlockKind::Block and vendor "status". Rules match in order, so add them after the vendor rules. BlockRules::builtin() returns a reference: clone it before extend. rules.check(&resp) checks a response against a set. Clones are cheap, because the rules sit behind an Arc.

#![allow(unused)]
fn main() {
use leyline::{BlockKind, BlockRules};

async fn run(session: leyline::Session) -> leyline::Result<()> {
let mut rules = BlockRules::builtin().clone();
rules.extend(BlockRules::from_toml(
    r#"
[[rule]]
vendor = "shop"
kind = "block"
status = 403
header = "x-shop-block"
value = "1"
"#,
)?);
rules.extend(BlockRules::statuses([403, 429]));

let resp = session.get("https://shop.example/item/1").send().await?;
match rules.check(&resp) {
    Some(signal) if signal.kind == BlockKind::Captcha => println!("captcha from {}", signal.vendor),
    Some(signal) => println!("blocked by {}", signal.vendor),
    None => println!("{} bytes", resp.text().await?.len()),
}
Ok(())
}
}

Retry a block through another proxy

One BlockRules value drives three settings, in this order on each attempt:

1. ProxyPool::rotate_on_block sees every attempt. A block is a strike on the proxy that carried it and moves the origin off that proxy.
2. RetryPolicy decides whether to send again, through another proxy. skip_blocks(rules) returns a block at once, without a retry, even when its status is a trigger such as 429. Calling it again adds rules.
3. rules.check(&resp) in your code reports the block on the last response.

Use skip_blocks when a retry only repeats the block, as in Build the crawl session: the pool moves the origin and your code decides what to do. A status that the retry policy retries never reaches your code while retries remain. A Cloudflare challenge is a 403, so a policy with on_status(403) retries it before you see it.

Without a pool, RetryPolicy::rotate_proxies with on_status(403) sends each retry through the next proxy in a list. See Rotate proxies on retry.

Read the text of a page

html::text(document) returns the text a reader sees, one block per line: each heading, paragraph, list item, and table cell starts a new line, and <br> breaks a line. Runs of whitespace collapse to one space, except inside <pre>, which keeps its lines and indentation. Entities are decoded. The text of <head>, <script>, <style>, <noscript>, <template>, <svg>, and <textarea> is left out. It needs the html feature, which is on by default.

#![allow(unused)]
fn main() {
use leyline::html;

async fn run(session: leyline::Session) -> leyline::Result<()> {
let page = session.get("https://shop.example/about").send().await?.text().await?;
for line in html::text(&page).lines() {
    println!("{line}");
}
Ok(())
}
}

html::text keeps navigation, footers, and other text that repeats on every page of a site. To drop it, compare lines across pages and remove the lines that most pages share.

Obey a page’s own rules

A page can carry rules of its own, in <meta name="robots"> tags and in X-Robots-Tag headers. html::Directives::read(document, headers, agent) reads both and returns Directives:

A meta tag named robots, or named for agent without case, applies. An X-Robots-Tag value that starts with a product token and a colon, such as examplebot: noindex, applies only to that token. Other directives, such as noarchive or max-snippet, are read and ignored.

html::canonical(document) returns the href of the first <link rel="canonical">, as written in the page. Join it with Response::url() before you compare URLs.

#![allow(unused)]
fn main() {
use leyline::html::{self, Directives};

async fn run(session: leyline::Session) -> leyline::Result<()> {
let resp = session.get("https://shop.example/item/1").send().await?;
let url = resp.url().clone();
let headers = resp.headers().clone();
let page = resp.text().await?;
let rules = Directives::read(&page, &headers, "examplebot");
if rules.index() && rules.ai() {
    let canonical = html::canonical(&page).and_then(|href| url.join(&href).ok());
    println!("keep {}", canonical.unwrap_or(url));
}
Ok(())
}
}

Group errors

Error::category() tells you what to do with a failed URL. See Errors for every category.

#![allow(unused)]
fn main() {
use leyline::{ErrorCategory, Session};

enum Next {
    Done(String),
    RetryLater,
    Drop,
}

async fn crawl_one(session: &Session, url: &str) -> Next {
    let resp = match session.get(url).tag("listing").send().await {
        Ok(resp) => resp,
        Err(err) => {
            eprintln!(
                "{url}: {} after {} attempts via {}",
                err.category(),
                err.attempts(),
                err.proxy().unwrap_or("direct"),
            );
            return match err.category() {
                ErrorCategory::Timeout
                | ErrorCategory::Connect
                | ErrorCategory::Tls
                | ErrorCategory::Proxy
                | ErrorCategory::ProxyTarget => Next::RetryLater,
                _ => Next::Drop,
            };
        }
    };
    if resp.block().is_some() {
        return Next::RetryLater;
    }
    match resp.text().await {
        Ok(text) => Next::Done(text),
        Err(_) => Next::Drop,
    }
}
}

Download files with a cap

RequestBuilder::download(path, limit) streams the decoded body to a temporary file in the same directory, then renames it to path. A status of 400 or more becomes an error with the status, the URL, the headers, and the start of the body, within the status-error limits. The limit is the smaller of limit and max_body_size. When the body has no content coding and its Content-Length is over the limit, the download fails with a body-limit error before it reads the body. Any error or cancel removes the temporary file, so path never holds a partial file. See Streaming.

#![allow(unused)]
fn main() {
use std::time::Duration;

async fn run(session: leyline::Session) -> leyline::Result<()> {
let written = session
    .get("https://shop.example/catalog.csv")
    .tag("catalog")
    .timeout(Duration::from_secs(300))
    .download("catalog.csv", Some(50 * 1024 * 1024))
    .await;
match written {
    Ok(bytes) => println!("{bytes} bytes"),
    Err(err) if err.is_timeout() => println!("download took too long"),
    Err(err) => return Err(err),
}
Ok(())
}
}

Log each request with a tag

RequestBuilder::tag(tag) names a request, and TracingTrace writes one line per request with the tag, the redacted proxy, and the browser of the session that sent it. With ProxyPool::identified, that is the identity of the proxy.

INFO leyline::trace: request id=12 method="GET" url=https://shop.example/item/1 redirects=0 status=200 version=Http2 attempts=1 elapsed_ms=210 streamed=false tag="listing" proxy="http://user:***@proxy-a.example:8080/" browser=Chrome154 outcome="ok"

A .stream() request logs its body end as a separate event with the same id. See Logging and tracing.

Count requests and watch the queues

trace::Metrics counts what a session does. Metrics::new() returns an Arc<Metrics>; pass a clone to the session, through trace::Fanout when you also log, and keep one to read. snapshot() returns a MetricsSnapshot:

The latency bounds are 10, 50, 100, 250, 500, 1000, 2500, 5000, and 10000 ms; a request falls in the first bucket whose bound is at least its elapsed time, and the last pair, with bound None, counts every request over 10 s. Display of a snapshot prints one key=value line.

Session::host_stats() returns one HostStats per origin that the host limits track, with origin(), in_flight(), and waiting(). A paused origin’s requests count as waiting. Session::pool_stats() returns PoolStats with busy and idle connections. An HTTP/2 or HTTP/3 connection is busy while it has an open stream; on HTTP/1.1, busy counts the requests that hold a connection permit.

#![allow(unused)]
fn main() {
use std::time::Duration;

use leyline::ErrorCategory;
use leyline::trace::Metrics;

async fn run(session: leyline::Session, metrics: std::sync::Arc<Metrics>) {
loop {
    tokio::time::sleep(Duration::from_secs(10)).await;
    let snapshot = metrics.snapshot();
    println!("{snapshot}");
    if snapshot.errors(ErrorCategory::Proxy) > 100 {
        break;
    }
    for host in session.host_stats() {
        println!("{} in_flight={} waiting={}", host.origin(), host.in_flight(), host.waiting());
    }
}
}
}

Rotate identities

Session::with_identity derives a session with another browser identity. Use one session per identity and spread the URLs over them.

#![allow(unused)]
fn main() {
use leyline::{Browser, Family, Identity, Platform};

fn run(session: leyline::Session) -> leyline::Result<()> {
let firefox = session.with_identity(Identity::locked(
    Browser::latest(Family::Firefox),
    Platform::Windows,
))?;
let mac_chrome = session.with_identity(Identity::locked(
    Browser::latest(Family::Chrome),
    Platform::MacOS,
))?;
let _ = (firefox, mac_chrome);
Ok(())
}
}

A derived session shares its parent’s connection pool, in a partition of its own, so an identity never reuses another identity’s connection. One PoolConfig budget covers every identity, and pool_stats() counts them all. The cookie jar, the HSTS store, the Alt-Svc knowledge, the host limits, and the proxy pool are shared too. Each identity has its own TLS session cache. To give an identity its own cookies, add with_cookie_jar, or use ProxyPool::identified. See Sessions.

Stop the crawl

Session::shutdown() stops every clone and derived session: requests in flight, new requests, and the next read of a body being streamed fail with Kind::Request and the message “session shut down”. Error::is_shut_down() tells this error apart, and Session::is_shut_down() reports the state. Shutdown does not close the pooled connections. To stop a single request, drop its future or its streamed body. See Cancellation.

buffer_unordered(n) from futures_util keeps at most n request futures alive, so a large URL list does not hold one task per URL, and max_total_in_flight keeps the session’s cap across every caller. Check is_shut_down() before each URL to stop reading the list:

#![allow(unused)]
fn main() {
use futures_util::future;
use futures_util::stream::{self, StreamExt};

async fn run(session: leyline::Session, urls: Vec<String>) {
let stop = session.clone();
tokio::spawn(async move {
    if tokio::signal::ctrl_c().await.is_ok() {
        stop.shutdown();
    }
});

let open = session.clone();
let mut results = stream::iter(urls)
    .take_while(move |_| future::ready(!open.is_shut_down()))
    .map(|url| {
        let session = session.clone();
        async move { (session.get(&url).await, url) }
    })
    .buffer_unordered(32);
while let Some((result, url)) = results.next().await {
    match result {
        Ok(resp) => println!("{url}: {}", resp.status()),
        Err(err) if err.is_shut_down() => {}
        Err(err) => eprintln!("{url}: {}", err.category()),
    }
}
}
}

A URL stream built with stream::unfold over an async reader is not Unpin. Pin the combined stream with std::pin::pin! before you call .next().

Memory bounds

Stream large bodies with download or copy_decoded_to, so only one chunk is in memory.

Next

Read Accounts.

Cookies

Every session owns a Jar. It sends the matching cookies on each request and stores what each response sets, with the rules Chrome applies. This page covers reading and seeding the jar, sharing it, and saving it between runs.

Read and seed the jar

Session::cookies() borrows the jar. Every jar method that takes a URL takes a parsed &Url and returns no Result. leyline::Url is the url crate’s Url, so parse once with leyline::Url::parse(..)? and pass the same value to each call.

#![allow(unused)]
fn main() {
async fn run() -> Result<(), Box<dyn std::error::Error>> {
let session = leyline::Session::new();
session.get("https://www.example.com/login").await?;

let jar = session.cookies();
let url = leyline::Url::parse("https://www.example.com/")?;
if let Some(id) = jar.get_cookie(&url, "session_id") {
    println!("session_id={id}");
}
if let Some(header) = jar.cookie_header(&url) {
    println!("cookie: {header}");
}

jar.store_set_cookie("token=abc; Domain=.example.com; Path=/; Secure", &url);
assert_eq!(jar.remove(&url, "token"), 1);
Ok(())
}
}

A Cookie has the public fields name, value, domain, path, secure, http_only, same_site (Strict, Lax, or None), expires, creation_time, and host_only. Cookie::is_expired() tells whether the expiry has passed.

Read the cookies a response set

Response::cookies() parses the Set-Cookie headers of the final response into Cookie records. It does not read the jar, so it also lists a deletion. It skips a header that does not parse and a cookie whose Domain is a public suffix or does not cover the host. To read what the session sends, ask the jar.

#![allow(unused)]
fn main() {
async fn run() -> Result<(), Box<dyn std::error::Error>> {
let session = leyline::Session::new();
let resp = session.get("https://shop.example/login").await?;
for cookie in resp.cookies() {
    println!("set {}={} for {}", cookie.name, cookie.value, cookie.domain);
}
Ok(())
}
}

Share or fork a jar

Jar is a handle over shared state: jar.clone() and session clones see the same cookies, so a cookie one task receives reaches every other task at once.

SessionBuilder::cookie_jar(jar) starts a session from a jar you built. session.with_cookie_jar(jar) derives a session that shares the pool and every other setting but reads and writes the given jar; use it for a second identity on warm connections. See Sessions.

snapshot() is a fork: a new store with a copy of every cookie and all its attributes. A write to the copy does not reach the original. extend_from(&other) merges the cookies of other with their attributes; on the same name, domain, and path, the cookie from other wins. Together they try a step without touching a stored login:

#![allow(unused)]
fn main() {
async fn run(session: leyline::Session) -> leyline::Result<()> {
let probe_jar = session.cookies().snapshot();
let probe = session.with_cookie_jar(probe_jar.clone());
if probe.get("https://example.com/check").await?.status().is_success() {
    session.cookies().extend_from(&probe_jar);
}
Ok(())
}
}

What the jar does

The jar follows Chrome:

• A cookie is stored under the exact domain the Set-Cookie names: its Domain= value, or the request host.
• Secure cookies go only over HTTPS.
• SameSite uses the sec-fetch-site value the request sends. Leyline computes it from the initiator and every URL in the redirect chain, schemes included, so http:// to https:// on one host is cross-site. The initiator is the request’s Referer; without one, the session’s Referer when it is an absolute URL; otherwise the first URL. A preset or header that sets sec-fetch-site sets it for cookies too.
• A cross-site request sends no Strict cookie, and sends Lax cookies only on a top-level navigation (sec-fetch-dest: document, or none) with GET or HEAD. Firefox profiles also treat a request as cross-site when its redirect chain crosses sites.
• A cookie lives at most 400 days. A later Max-Age or Expires, or an Expires past the range of the platform clock, is cut to 400 days.
• A cookie whose name and value together pass 4096 octets is refused. An attribute whose value passes 1024 octets is ignored, so an oversized Path falls back to the default path, which comes from the request URL and has no cap. A cookie whose stored domain passes 1024 octets is refused. extend_from and loading a saved jar apply the same limits and skip a cookie that fails them.
• The limits are 180 cookies per exact domain and 3300 overall. Crossing a limit evicts the least recently accessed entries, 30 per domain or 300 overall.
• An expired cookie stays in memory but is never sent. get_cookie skips it; all_cookies() and snapshot() include it.

Domain scoping uses the Mozilla Public Suffix List, compiled into the crate. A cookie with Domain=example.co.uk reaches shop.example.co.uk; a host-only cookie set by www.example.co.uk reaches that host only. A Set-Cookie with Domain=co.uk, or with a domain that has no dot, is refused.

Save the jar to a file

Jar and Cookie implement Serialize and Deserialize, with times as unix milliseconds. Jar::save_to(path) writes the jar as JSON and Jar::load_from(path) reads it back. The write is atomic: a temporary file, synced, renamed over path, and the directory synced. On Unix the file has mode 0600; on Windows it takes the default ACLs of its directory.

The file is a versioned object, { "version": 1, "cookies": [..] }. load_from refuses a file with another version, or with no version, with Kind::Json.

• Saving writes every cookie that has not expired, session cookies included, as a browser that restores its last session does.
• Loading skips expired cookies and cookies past the size limits, keeps the newest of two cookies with the same name, path, and domain, applies the 180 and 3300 limits, and cuts a stored expiry to at most 400 days after the load.
• Each cookie keeps its last-access time, so eviction order survives a restart. A jar saved without that field loads with the load time.

Jar::autosave(path, interval) starts one writer task and returns a JarAutosave handle. It saves at most once per interval after a change. It panics outside a Tokio runtime.

#![allow(unused)]
fn main() {
use std::time::Duration;

use leyline::cookie::Jar;

async fn run() -> leyline::Result<()> {
let jar = if std::path::Path::new("cookies.json").exists() {
    Jar::load_from("cookies.json")?
} else {
    Jar::new()
};
let session = leyline::Session::builder()
    .browser(leyline::Browser::default())
    .cookie_jar(jar)
    .build()?;
let saver = session.cookies().autosave("cookies.json", Duration::from_secs(2));

session.get("https://shop.example/account").await?;

saver.shutdown().await?;
Ok(())
}
}

Call shutdown().await before the program exits, so the last save is not lost.

Keep a logged-in device between runs

A site can tie a login to the browser, platform, proxy, and languages as well as the cookies. Device saves all of them with the jar, and Device::autosave replaces Jar::autosave. See Accounts.

Save the connection state

TLS session tickets, Alt-Svc entries, and the HSTS store live in SessionState, beside the jar. See Accounts.

Next

Read WebSocket.

Accounts

This page keeps one logged-in account the way a browser profile keeps it: the same browser, platform, languages, proxy, cookies, and connection state on every run. A Device holds all of them in one file. Device::autosave keeps the file current, a Tab keeps the current page, and leyline::html reads the login form.

What a device holds

Create the device once

Pin a browser variant, set the languages and one proxy, and turn the proxy environment variables off. Then capture the device and freeze its profile.

#![allow(unused)]
fn main() {
use leyline::{Browser, Device, Platform, ProxyConfig, ProxyUrl, Session};

fn run() -> leyline::Result<()> {
let proxy = ProxyUrl::parse("http://user:secret@proxy.example:8080")?;
let session = Session::builder()
    .browser(Browser::Chrome154)
    .platform(Platform::Windows)
    .languages(["de-DE", "de", "en"])
    .proxy(ProxyConfig::from(proxy.clone()).env(false))
    .build()?;

let mut device = Device::capture(&session, Some(proxy));
device.pin_profile(&session)?;
device.strict = true;
device.jar_path = Some("account/cookies.json".into());
device.save_to("account/device.json")?;
Ok(())
}
}

Session::proxy_url() returns the configured proxy, password included, so Device::capture(&session, session.proxy_url()) captures it too. check compares the configured URL, not the exit IP the proxy assigns: a rotating proxy behind one URL passes with a new IP. To pin the exit IP, fetch it from your own IP echo service and keep it in Device::app.

Keep the proxy password out of the file

ProxyUrl serializes the full URL, password included. To keep the password out of the device file, set proxy_password_env to the name of an environment variable. save_to and Device::autosave then write the proxy URL without its password, and open and session_builder read it back from the variable. With proxy_password_env set, serializing the Device never writes the password. When the variable is not set in the environment, open and session_builder return Kind::Config with a message that names it.

#![allow(unused)]
fn main() {
use leyline::{Browser, Device, Platform, ProxyConfig, ProxyUrl, Session, Url};

const PROXY: &str = "http://user@proxy.example:8080";
const PASSWORD_VAR: &str = "ACCOUNT_PROXY_PASSWORD";

async fn run() -> Result<(), Box<dyn std::error::Error>> {
let mut url = Url::parse(PROXY)?;
url.set_password(Some(&std::env::var(PASSWORD_VAR)?))
    .map_err(|()| "the proxy URL cannot carry a password")?;
let proxy = ProxyUrl::parse(url.as_str())?;

let session = Session::builder()
    .browser(Browser::Chrome154)
    .platform(Platform::Windows)
    .proxy(ProxyConfig::from(proxy).env(false))
    .build()?;
session.get("https://shop.example/login").await?;

let mut device = Device::capture(&session, session.proxy_url());
device.proxy_password_env = Some(PASSWORD_VAR.to_owned());
device.pin_profile(&session)?;
device.save_to("account/device.json")?;
Ok(())
}
}

Open the device on each run

Device::open builds the session, runs check, loads the cookies, and restores the saved connection state. A jar_path whose file does not exist yet gives an empty jar. A device with no jar_path uses the jar in the device file, or an empty jar. Then start Device::autosave and call its shutdown before the process exits.

#![allow(unused)]
fn main() {
use std::time::Duration;
use leyline::Device;

async fn run() -> leyline::Result<()> {
let device = Device::load_from("account/device.json")?;
let session = device.open()?;
let tab = device.tab(&session);
let autosave = device.autosave(&session, "account/device.json", Duration::from_secs(2));
autosave.track(&tab);

tab.follow("/account/orders").await?;
autosave.shutdown().await?;
Ok(())
}
}

To add settings before the build, call device.session_builder()?, change the builder, build the session, then call device.check(&session)? and device.state.restore_into(&session).

Saved TLS sessions are bound to the trust that the session loaded when it was built: the pins, the contents of every root it loaded from add_ca_der, add_ca_file, SSL_CERT_FILE, and SSL_CERT_DIR, the contents of the client certificate and key files, and whether system roots are on. A saved TLS session resumes only in a session that loaded the same trust. Otherwise the connection makes a full handshake and checks the certificate again. Session::fresh_pool keeps the trust of the session it copies. Changes to the operating system’s root store are not part of the binding. The saved state contains TLS session secrets, so store it with the same protection as credentials.

check returns Kind::Config and names each field that differs:

Error::is_profile_changed() is true when the profile differs: the profile_id, or, without one, the identity, platform, brand, or user agent. It is false when only the languages, the proxy, or a strict gap differ.

Save the device as it changes

Device::autosave(&session, path, options) starts one writer task and returns a DeviceAutosave handle. options is a DeviceAutosaveOptions, or a Duration that sets its interval:

Each save writes the device file with the session’s current SessionState. With a jar_path, it also writes the jar file when a cookie changed; without one, the jar goes into the device file. For a jar without a device, use Jar::autosave; see Cookies.

Each save writes a temporary file, syncs it, and renames it, so a crash never leaves a half-written file. On Unix the files have mode 0600; on Windows they take the default ACLs of their directory. autosave panics outside a Tokio runtime.

The writer keeps its own copy of the device. Change it with DeviceAutosave::update(|device| ..); the change reaches the file with the next save. A change to the Device you called autosave on does not. DeviceAutosave::device() returns a snapshot of the writer’s copy.

DeviceAutosave::track(&tab) stores the tab’s current page in Device::page on each save. The writer keeps one tab: a second track replaces the first. On the next run, Device::tab(&session) returns a tab on that page, so the next request carries it as its Referer.

Keep your own data

Device::app holds your own data in the same atomic save, such as the account email or values a site keeps in localStorage. Leyline does not read it.

#![allow(unused)]
fn main() {
use leyline::Device;
use serde_json::json;

fn run(mut device: Device) -> leyline::Result<()> {
device.app.insert("email".into(), json!("me@example.com"));
device.app.insert("local_storage".into(), json!({ "cart_id": "c-1842" }));
let email = device.app.get("email").and_then(|v| v.as_str());
let _ = email;
device.save_to("account/device.json")?;
Ok(())
}
}

Leyline keeps no HTTP cache, localStorage, sessionStorage, IndexedDB, or service workers, because it runs no scripts. Send if-none-match or if-modified-since yourself.

Keep the connection state

Session::state() returns a SessionState, which implements Serialize and Deserialize. state.restore_into(&session) loads it, and is_empty() is true when there is nothing to save.

Device::autosave refreshes device.state on each save. Before a manual save_to, set device.state = session.state(). A ticket key holds the proxy URL without its user name and password, and a hash of them, so the saved state never holds proxy credentials.

Browse with a tab

A Tab sends each request with the current page as its initiator, so the Referer, Origin, and sec-fetch-site headers match a real click. Requests lists the calls. For an account:

• xhr, fetch, post_json, and subresource on a tab with no page fail with Kind::Request (“the tab has no page; open one first”) and send nothing. Open a page first.
• A .stream() navigation changes the page at the response head. current() reads the page, and set_current(..) sets it.
• A page script sets its own accept. Set it on the xhr builder when the site’s script does. See Fingerprints.

#![allow(unused)]
fn main() {
use leyline::Preset;

async fn run(session: leyline::Session) -> leyline::Result<()> {
let tab = session.tab();
tab.open("https://shop.example/account").await?;
let orders = tab
    .xhr("/api/orders?page=1")
    .header("accept", "application/json")
    .send()
    .await?;
let logo = tab.subresource("/static/logo.png", Preset::Image).send().await?;
println!("{} {}", orders.status(), logo.status());
Ok(())
}
}

Log in with the page’s form

The html feature, on by default, adds leyline::html and turns on the multipart feature. html::forms(document) returns every <form> as an html::Form with the values a browser would submit:

• Hidden inputs, checked checkboxes, text areas, and selected options.
• One checked radio button per name: the last one, as a browser keeps it.
• Controls outside the <form> element that name it with form="id". When two elements share the id, the first one is the form.
• No disabled controls, and no controls inside a disabled <fieldset> except in the first <legend> that is a direct child of that fieldset.
• No options inside a disabled <optgroup>. A list box (<select size>) with no selected option sends nothing.
• Line breaks in names and values converted to CRLF, as a browser sends them.
• Values with every HTML named character reference decoded, such as & and é.

Form::find(document, key) returns the first form whose id or name is key.

Tab::follow and Tab::submit_form drop the session credentials when the target origin differs from the current page: the Authorization, Cookie, and Proxy-Authorization defaults and the session bearer token, as a redirect does.

Tab::submit_form(&form) sends form.fields() as they are, with no submit button: call form.press(name) first when the site reads the button. The action resolves against the page’s <base href>, or the current page when there is none. A Post form sends the fields with Preset::FormNavigate, encoded as its enctype() says: application/x-www-form-urlencoded, multipart/form-data, or text/plain. A Get form replaces the action’s query with the fields. Both are navigations that carry the page as the Referer and change the current page.

The session redirect policy applies: after a login POST, a 301, 302, or 303 continues as a GET without the body, and a 307 or 308 repeats the POST. The response and the current page are those of the final URL.

#![allow(unused)]
fn main() {
use leyline::html::{self, Form};

async fn run(tab: leyline::Tab) -> leyline::Result<()> {
let page = tab.open("https://shop.example/login").await?.text().await?;
let token = html::meta(&page, "csrf-token").unwrap_or_default();
if let Some(mut form) = Form::find(&page, "login") {
    form.set("email", "me@example.com").set("password", "secret");
    form.press("sign_in");
    let resp = tab.submit_form(&form).await?;
    println!("{} at {}", resp.status(), resp.url());
}
let cart = tab.xhr("/api/cart").header("x-csrf-token", token).send().await?;
println!("{}", cart.status());
Ok(())
}
}

html::meta(document, name) returns the content of the first <meta> whose name or property equals name, without case; many sites put a CSRF token there. html::links(document) returns every <a href> as an html::Anchor with href(), text(), and rel(). href() is the raw attribute; pass it to Tab::follow, which resolves it.

Check the login, log in again

Leyline cannot tell whether a site considers you logged in. Request an endpoint that only a logged-in user can read, and log in again when it says no. The jar and Device::autosave keep the new cookies.

#![allow(unused)]
fn main() {
use leyline::html::Form;
use leyline::http::StatusCode;

async fn run(tab: leyline::Tab) -> leyline::Result<()> {
let me = tab.xhr("https://shop.example/api/me").send().await?;
if me.status() == StatusCode::UNAUTHORIZED {
    let page = tab.open("https://shop.example/login").await?.text().await?;
    if let Some(mut form) = Form::find(&page, "login") {
        form.set("email", "me@example.com").set("password", "secret");
        tab.submit_form(&form).await?;
    }
}
Ok(())
}
}

When several tasks share the session, each can see the 401 at once. Hold one lock, such as a tokio::sync::Mutex, around the check and the login, and check again after you take it, so one task logs in and the others reuse its cookies. examples/device.rs combines the device, autosave, the login form, and the tab.

Run tasks at the same time

Session and Tab are Clone, Send, and Sync. Session clones share the pool, the jar, the identity, and the state, so one Device::autosave covers every task. Tab clones share the current page, so give each parallel page its own session.tab().

#![allow(unused)]
fn main() {
async fn run(session: leyline::Session) -> leyline::Result<()> {
let orders = session.tab();
let messages = session.tab();
let (a, b) = tokio::join!(
    orders.open("https://shop.example/account/orders"),
    messages.open("https://shop.example/account/messages"),
);
println!("{} {}", a?.status(), b?.status());
Ok(())
}
}

To stop every task of the account, call session.shutdown(), then autosave.shutdown().await for the last save. See Sessions.

Next

Read Proxies.

WebSocket

The websocket feature is on by default. A WebSocket uses the session’s TLS fingerprint, proxy rules, and cookie jar. Use wss://: any other scheme, ws:// included, fails with Kind::Request before Leyline picks a proxy or connects.

Connect, send, and receive

Session::websocket(url) returns a WebSocketBuilder. Call connect(), or await the builder. send takes a WsMessage: Text, Binary, Ping, Pong, or Close.

#![allow(unused)]
fn main() {
use leyline::WsMessage;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let mut ws = session.websocket("wss://example.com/live").connect().await?;

ws.send(WsMessage::Text("ping".to_owned())).await?;
ws.send(WsMessage::Binary(vec![1, 2, 3])).await?;

if let Some(msg) = ws.recv().await? {
    println!("{msg:?}");
}
ws.close().await?;
Ok(())
}
}

When the peer closes the connection, recv first returns Some(WsMessage::Close(frame)), where frame is an Option<CloseFrame> with the close code and reason. The next recv returns Ok(None). close sends a close frame and shuts the connection down.

A send, recv, or close that fails on the transport or on a closed connection gives Kind::Io. A protocol violation, or a message over the WebSocketConfig limits, gives Kind::Body.

Set handshake options

The handshake sends the cookies that the jar holds for the URL, matched as for an https:// request. A Cookie header from the session default headers or from header() replaces them. A Set-Cookie header in the handshake response does not update the jar.

A wss:// URL follows the proxy rules of an https:// URL, including environment discovery and NO_PROXY. See Proxies.

Shutdown and host limits

The handshake takes a host-limit slot for its origin, the same slot an https:// request to that host takes, and releases it when the handshake completes or fails. An open connection holds no slot. Session::shutdown() stops a handshake in flight and fails new ones with the shut-down error. It does not close an established connection.

HTTP/2 or HTTP/1.1

WebSocketConfig::prefer_http2 defaults to true: Leyline first tries RFC 8441 extended CONNECT on a pooled HTTP/2 connection. It falls back to the RFC 6455 HTTP/1.1 upgrade on a fresh connection in two cases, and logs the reason:

• The TLS handshake does not select h2. Later WebSockets to that origin skip the HTTP/2 attempt.
• The peer does not advertise SETTINGS_ENABLE_CONNECT_PROTOCOL.

Any other error is returned, not retried. prefer_http2(false) skips the HTTP/2 attempt. protocol() returns the subprotocol the origin selected.

#![allow(unused)]
fn main() {
use leyline::WebSocketConfig;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let ws = session
    .websocket("wss://example.com/live")
    .config(
        WebSocketConfig::new()
            .max_message_size(1 << 20)
            .prefer_http2(false),
    )
    .connect()
    .await?;
println!("protocol={:?}", ws.protocol());
Ok(())
}
}

Split the connection

split() returns a WsSink with send and close, and a WsStream with recv, so one task can write while another reads.

#![allow(unused)]
fn main() {
use leyline::WsMessage;

async fn run() -> leyline::Result<()> {
let session = leyline::Session::new();
let ws = session.websocket("wss://example.com/live").connect().await?;
let (mut sink, mut stream) = ws.split();

let reader = tokio::spawn(async move {
    while let Ok(Some(msg)) = stream.recv().await {
        println!("{msg:?}");
    }
});

sink.send(WsMessage::Text("hello".to_owned())).await?;
sink.close().await?;
reader.await.expect("reader task panicked");
Ok(())
}
}

Next

Read HTTP/3.

HTTP/3

The http3 feature is on by default. It adds the Http3 and Race protocol policies and the QUIC stack. Build with default-features = false to drop the QUIC stack. This page covers when a request uses HTTP/3, the QUIC fingerprint keys of a profile, and HTTP/3 through a proxy.

Protocol policy

SessionBuilder::protocol sets the ProtocolPolicy.

Without a protocol call, a session selects Race when its profile sets race = true in [h3], as the bundled Chrome profiles do, and Auto otherwise. Session::browser(Browser::default()) selects Race; Session::new() and a build without the http3 feature select Auto.

Keep that default for a browser session: Race uses HTTP/3 where a browser would and falls back to TCP. Http3 forces QUIC on every request, the first one included, and fails where QUIC is blocked; use it only to test that an origin speaks HTTP/3. build() fails with Kind::Config when Http3 or Race meets a profile with no [h3] table, such as the bare profile.

The Alt-Svc gate

The first request to an origin never uses HTTP/3. As in Chrome, Race opens no speculative QUIC connection: it races only an origin that advertised h3 in an Alt-Svc response header. The first request goes over HTTP/2 or HTTP/1.1 and records the header; the next one can race. Response::version() returns the HttpVersion that carried each response.

#![allow(unused)]
fn main() {
use leyline::{Browser, HttpVersion, Session};

async fn run() -> leyline::Result<()> {
let session = Session::browser(Browser::default());
let url = "https://www.cloudflare.com/";

let first = session.get(url).send().await?;
println!("first: {:?}", first.version());

let second = session.get(url).send().await?;
if second.version() == HttpVersion::Http3 {
    println!("second: HTTP/3");
}
Ok(())
}
}

The second request can still use HTTP/2: the first transport to hand back a connection carries the request, and a pooled HTTP/2 connection is often ready first. The request is sent once. If both transports fail, the Auto path runs. examples/http3.rs runs this sequence.

A request is raced only when all of these hold; otherwise it takes the Auto path:

• The scheme is https and the origin is known to speak HTTP/3.
• Neither the request body nor the response is streamed.
• No proxy applies, or the proxy is socks5:// or socks5h:// and the socks feature (not a default feature) is on.

When an HTTP/3 connection to an origin fails, Race skips HTTP/3 for that origin and proxy pair for 5 minutes. A successful HTTP/3 connection to the pair ends the skip.

Session::state() saves the known origins, so a restored session can race on its first request. See Accounts.

How Alt-Svc is read

Only an h3= entry for the same port counts, and its host must be empty or equal to the origin’s host. The origin is kept until the longest ma of the matching entries, less the response’s Age, ends; without ma, 24 hours. Each Alt-Svc value replaces the one before it: ma=0 on every matching entry, clear, or a value with no usable h3= entry removes the origin. An entry whose ma is not a number is ignored. The pool keeps at most 1024 origins and drops the one that expires first to make room.

Body limits and cancellation

HTTP/3 uses CompressionConfig::max_body_size, as HTTP/1.1 and HTTP/2 do. It caps a buffered body and fails with Kind::Body; a .stream() response has no cap. See Responses.

A request that ends early resets its stream with the code Chrome sends:

When the pool drops an idle connection, Leyline sends no CONNECTION_CLOSE frame, as Chrome does not.

QUIC fingerprint

The [h3] table of a profile sets the QUIC transport parameters and the HTTP/3 settings: the flow-control limits, max_idle_timeout_secs, max_udp_payload_size, and active_connection_id_limit. These keys shape the QUIC Initial and the HTTP/3 control stream. dcid_length is required; without the other keys, a profile keeps quiche’s defaults.

Leyline speaks QUICv1 and QUICv2 (RFC 9369). It sends its first Initial in the chosen version of version_information and follows a server’s compatible version negotiation (RFC 9368) to another version in available.

A transport parameter list that sends max_datagram_frame_size (32) turns on QUIC DATAGRAM receipt. Leyline reads RESET_STREAM_AT as RESET_STREAM and ignores ACK_FREQUENCY and IMMEDIATE_ACK, so a profile can advertise reset_stream_at and min_ack_delay.

QPACK

The QPACK decoder keeps a dynamic table. The settings list sets the advertised QPACK_MAX_TABLE_CAPACITY and QPACK_BLOCKED_STREAMS; a profile without a settings list uses qpack_max_table_capacity, qpack_blocked_streams, and max_field_section_size. The decoder applies the peer’s encoder stream, holds a field section that waits for inserts (up to the advertised blocked streams), and sends Section Acknowledgment, Insert Count Increment, and Stream Cancellation. The encoder does not use the dynamic table.

Proxies

HTTP/3 goes through a socks5:// or socks5h:// proxy with the socks feature, which relays the QUIC datagrams with SOCKS5 UDP ASSOCIATE. An http:// or https:// proxy cannot carry QUIC: under Race such a request takes the Auto path, and Http3 with such a proxy fails with Kind::Config. When the cause is a missing socks feature, the message says so. See Proxies.

Next

Read TLS trust.

TLS trust

Leyline verifies every server certificate. All trust settings live in one TlsTrustConfig that you pass to SessionBuilder::tls_trust: the roots, the pins, the client certificate, and the TLS version floor. The browser profile decides what the ClientHello looks like; trust settings decide which certificates the session accepts.

Choose the roots

By default a session trusts the system trust store and the files named by SSL_CERT_FILE and SSL_CERT_DIR when they are set. add_ca_file adds a PEM file and add_ca_der adds the DER bytes of one certificate; both add to the default roots. Turn a default source off with system_roots(false) or env_roots(false). With both off, the session trusts only the roots you add.

#![allow(unused)]
fn main() {
use leyline::{Browser, Family, Session, TlsTrustConfig};

fn run() -> leyline::Result<()> {
let trust = TlsTrustConfig::new()
    .env_roots(false)
    .system_roots(false)
    .add_ca_file("/etc/myorg/ca.pem");

let chrome = Session::builder()
    .browser(Browser::default())
    .tls_trust(trust.clone())
    .build()?;
let firefox = Session::builder()
    .browser(Browser::latest(Family::Firefox))
    .tls_trust(trust)
    .build()?;
let _ = (chrome, firefox);
Ok(())
}
}

TlsTrustConfig is a value: build it once and pass a clone to each builder. tls_trust replaces the builder’s trust settings, so put every setting in one config. A file or certificate that cannot be parsed fails with Kind::Config.

Pin a certificate

add_pinned_leaf_sha256 takes the SHA-256 digest of the server’s leaf certificate in DER form. The handshake succeeds only when the chain verifies and the leaf matches a pin. A pin adds no root.

#![allow(unused)]
fn main() {
fn run() -> leyline::Result<()> {
let pin: [u8; 32] = [0; 32];
let session = leyline::Session::builder()
    .browser(leyline::Browser::default())
    .tls_trust(leyline::TlsTrustConfig::new().add_pinned_leaf_sha256(pin))
    .build()?;
let _ = session;
Ok(())
}
}

Pins apply to every host the session contacts, so give a pinned host its own session. Add a pin for each certificate you expect, including the next one before a rotation. To read the digest of a live certificate, hash the peer_cert_der field of Response::tls().

A saved TLS session resumes only in a session that loaded the same pins, roots, and client certificate, compared by content; any other session makes a full handshake.

A pin failure is a Kind::Tls error, and err.tls() returns the TlsError. With ProtocolPolicy::Http3 it is a Kind::Http3 error with a message and no source, so err.tls() returns None.

Present a client certificate

For mutual TLS, give client_identity a PEM certificate chain and its private key.

#![allow(unused)]
fn main() {
fn run() -> leyline::Result<()> {
let session = leyline::Session::builder()
    .browser(leyline::Browser::default())
    .tls_trust(
        leyline::TlsTrustConfig::new()
            .client_identity("/etc/myorg/client.pem", "/etc/myorg/client.key"),
    )
    .build()?;
let _ = session;
Ok(())
}
}

A session with pins or a client certificate cannot use an https:// proxy. A request through one fails with Kind::Proxy.

Set a TLS version floor

min_tls_version takes a TlsMinVersion: Tls10 (the default), Tls12, or Tls13. The handshake minimum is the higher of the profile’s minimum and the floor. A profile that declares no minimum starts at TLS 1.2.

#![allow(unused)]
fn main() {
fn run() -> leyline::Result<()> {
let session = leyline::Session::builder()
    .browser(leyline::Browser::Safari18)
    .platform(leyline::Platform::MacOS)
    .tls_trust(
        leyline::TlsTrustConfig::new().min_tls_version(leyline::TlsMinVersion::Tls12),
    )
    .build()?;
let _ = session;
Ok(())
}
}

A floor above the profile’s minimum changes the ClientHello, so the session no longer matches the browser it impersonates:

• Some Safari and CFNetwork profiles declare a TLS 1.0 minimum. A Tls12 floor removes TLS 1.0 and 1.1 from supported_versions.
• A Tls13 floor removes TLS 1.2 from supported_versions in every profile, with the TLS 1.2 cipher suites and the extensions only TLS 1.2 and earlier use.

A server that cannot meet the floor fails the handshake with Kind::Tls; an https:// proxy that cannot meet it fails with Kind::Proxy. Response::tls() reports the negotiated version in version. HTTP/3 always uses TLS 1.3, so the floor does not change a QUIC handshake.

Turn verification off

danger_accept_invalid_certs(true) accepts any certificate on TCP connections (HTTP/1.1 and HTTP/2). Use it against a local test server only. On those connections it also skips pin checks and turns off TLS session resumption. It has no effect on HTTP/3: QUIC always verifies the certificate and applies pins.

Next

Read Network.

Network

This page covers the layer under TLS: how a host name becomes an address, how Leyline picks between IPv4 and IPv6, which socket options it sets, and how the connection pool and the HSTS store work.

Map a host to an address

DnsConfig::resolve_host sends one host to a fixed list of addresses without a DNS lookup. Leyline replaces the port of each address with the port of the request URL, so the port you give is a placeholder. The host name still goes into SNI and the Host header, so the server sees a normal request.

#![allow(unused)]
fn main() {
fn run() -> leyline::Result<()> {
use leyline::DnsConfig;

let dns = DnsConfig::new().resolve_host(
    "api.example.com",
    [
        "203.0.113.10:0".parse().unwrap(),
        "203.0.113.11:0".parse().unwrap(),
    ],
);
let session = leyline::Session::builder()
    .browser(leyline::Browser::default())
    .dns(dns)
    .build()?;
let _ = session;
Ok(())
}
}

Hosts without an override go to the resolver. A later call for the same host replaces its list, and an empty list removes the override. Put every override in one DnsConfig: a second dns call replaces the first.

Replace the resolver

The default is tls::SystemResolver. To use your own, implement tls::Resolver and pass it to SessionBuilder::dns as an Arc<dyn Resolver>. DnsConfig holds the resolver and the overrides as one value, so several builders can share it.

Happy Eyeballs

Leyline connects to the addresses of a host IPv6 first, then alternating between the families. Inside one family, the order stays as the resolver or resolve_host gave it. Leyline starts one attempt, and starts the next when resolve_delay passes with no connection, or at once when every running attempt has failed. The first connection to complete wins.

This applies to every TCP connection, mapped hosts included. A direct HTTP/3 connection uses one address: the first IPv4 address, or the first address when the host has none.

#![allow(unused)]
fn main() {
fn run() -> leyline::Result<()> {
use std::time::Duration;
use leyline::SocketConfig;
use leyline::tls::HappyEyeballsConfig;

let he = HappyEyeballsConfig::new()
    .resolve_delay(Duration::from_millis(100))
    .attempt_limit(4);
let session = leyline::Session::builder()
    .browser(leyline::Browser::default())
    .socket(SocketConfig::new().happy_eyeballs(he))
    .build()?;
let _ = session;
Ok(())
}
}

Happy Eyeballs cannot be turned off; attempt_limit(1) tries one address only. happy_eyeballs(None) keeps the defaults.

Socket options

SocketConfig sets options on each TCP socket before it connects. SessionBuilder::socket replaces the whole value, so start from SocketConfig::new(), which carries the defaults.

The timing and size setters take a value or None, which clears it. Leyline sets TCP keepalive when any of its three values is set; to turn it off, pass None to all three.

#![allow(unused)]
fn main() {
fn run() -> leyline::Result<()> {
use std::time::Duration;
use leyline::SocketConfig;

let session = leyline::Session::builder()
    .browser(leyline::Browser::default())
    .socket(
        SocketConfig::new()
            .tcp_nodelay(true)
            .tcp_user_timeout(Duration::from_secs(20))
            .tcp_keepalive(Duration::from_secs(30))
            .tcp_keepalive_interval(Duration::from_secs(10)),
    )
    .build()?;
let _ = session;
Ok(())
}
}

Set tcp_user_timeout when a pooled connection can die without a FIN or RST, for example when a proxy or NAT drops its state. Without it, the kernel retransmits for many minutes and the request waits for its total timeout. With it, the kernel closes the socket, the request fails, and a retry policy can use a fresh connection.

Where an option is unsupported, Leyline logs one warning for it on the leyline::socket target and continues. With strict(true), the connect fails with an io::ErrorKind::Unsupported error. tcp_keepalive_retries falls back to the operating system default where it cannot be set.

SocketConfig is separate from TcpProfile, which sets the values of the JA4T fingerprint and comes from the session platform. See Fingerprints.

Connection pool

Each session has a connection pool. PoolConfig sets its limits, and every limit applies to one pool.

#![allow(unused)]
fn main() {
use leyline::PoolConfig;

fn run() -> leyline::Result<()> {
let session = leyline::Session::builder()
    .browser(leyline::Browser::default())
    .pool(PoolConfig::new().max_connections(512).max_h1_conns_per_host(8))
    .build()?;
let _ = session;
Ok(())
}
}

Clones and every derived session share the pool, except fresh_pool, which starts a new, empty one. A with_identity session keeps its connections in a partition of its own, inside the shared pool and its limits. See Sessions.

When every stream slot of an HTTP/2 connection is in use, one request waits inside the connection and later requests wait in a queue of at most 1024. The request’s timeout covers the wait, and a request dropped while it waits is never sent.

One handshake per connection

An HTTPS connection always sends the profile’s ClientHello, with its ALPN list, for example h2, http/1.1, and its ALPS extension. The server’s ALPN choice decides HTTP/2 or HTTP/1.1 on that same connection. Leyline never dials again to change the protocol, so a streaming request body to an HTTP/1.1-only origin works on the first request.

When an origin negotiates HTTP/1.1, the pool remembers it for 10 minutes, for at most 1024 origins. The next connection still sends the full ClientHello, and if the origin then answers h2, Leyline uses HTTP/2 and forgets the entry.

ProtocolPolicy::Http2 sends every request over HTTP/2 with the same ClientHello; when the server picks another protocol, the request fails with Kind::Http2. ProtocolPolicy::Http1 and a WebSocket over HTTP/1.1 offer http/1.1 only.

Limit requests per host

HostLimits limits the requests to each origin, apart from the pool limits, which count connections. See Limit each host.

HSTS

Leyline keeps an HSTS store per session (RFC 6797). It records the first Strict-Transport-Security header of each https response. The header needs max-age; includeSubDomains also covers the subdomains, and max-age=0 removes the host. IP addresses are ignored. A session that turns off certificate verification records nothing, because it cannot trust the header.

A later http:// request or redirect to a recorded host, or a covered subdomain, goes to https instead, and an explicit port 80 becomes 443. WebSocket requests are not upgraded.

Derived sessions, with_identity included, share the store; only fresh_pool starts with an empty one. Session::state() saves the store, and SessionState::restore_into loads it. See Accounts.

Next

Read Logging and tracing. Timeouts are covered in Retries and timeouts.

Logging and tracing

Leyline logs through the tracing crate and installs no subscriber. Your application installs one, for example tracing_subscriber::fmt, and filters by target. For one event per request, or counters, give the session a Trace hook.

Targets

Every target starts with leyline.

With a subscriber that reads RUST_LOG, this filter shows one line per request and the warnings from the rest of the crate. Use leyline::trace=debug to add the request phases.

RUST_LOG=leyline=warn,leyline::trace=info cargo run

Log each request

leyline::trace::TracingTrace writes one INFO event per request on the leyline::trace target, with the message request, and each phase as a DEBUG event: DNS, connect, TLS, send, response head, and completion.

#![allow(unused)]
fn main() {
use leyline::trace::TracingTrace;
use leyline::{Browser, Session};

async fn run() -> leyline::Result<()> {
let session = Session::builder()
    .browser(Browser::default())
    .trace(TracingTrace)
    .build()?;
let resp = session
    .get("https://shop.example/item/1")
    .tag("listing")
    .send()
    .await?;
println!("{} via {}", resp.status(), resp.proxy().unwrap_or("direct"));
Ok(())
}
}

The INFO event has the fields id, method, url, redirects, status, version, attempts, elapsed_ms, streamed, tag, proxy, browser, and outcome. tag, proxy, and browser appear only when they have a value. A failed request adds kind and error. The URL is redacted: the password, or a user name without a password, becomes ***, the whole query becomes ***, and the fragment is removed.

INFO leyline::trace: request id=7 method="GET" url=https://***@example.com/a?*** redirects=1 status=200 version=Http2 attempts=1 elapsed_ms=84 streamed=false outcome="ok"
INFO leyline::trace: request id=8 method="GET" url=https://shop.example/item/1 redirects=0 status=200 version=Http2 attempts=2 elapsed_ms=310 streamed=false tag="listing" proxy="http://user:***@proxy-a.example:8080/" browser=Chrome154 outcome="ok"

RequestBuilder::tag(tag) names a request, for example by job, account, or identity. The proxy field holds the proxy the request used, from the request, the session, or a ProxyPool, with the password replaced by ***. Response::proxy() returns the same value for your own log lines.

One event per request

Trace::summary receives a trace::Summary once per request, after all redirects and retries.

For a .stream() request, summary fires at the response head, so it does not cover the body read. It does not fire for a dropped request future. The done phase event fires once per attempt, so a request with two retries gives three done events and one summary.

url and original_url hold the raw URLs, with credentials and query values. Redact them with leyline::redact_url, which TracingTrace uses, before you log them.

#![allow(unused)]
fn main() {
use leyline::trace::{Summary, Trace};

struct RequestLog;

impl Trace for RequestLog {
    fn summary(&self, ev: &Summary<'_>) {
        let host = ev.url.and_then(|url| url.host_str()).unwrap_or("-");
        let status = ev.status.map_or(0, |status| status.as_u16());
        let tag = ev.tag.unwrap_or("-");
        let proxy = ev.proxy.as_deref().unwrap_or("direct");
        println!(
            "{} {} {} attempts={} {} ms tag={} proxy={}",
            ev.method,
            host,
            status,
            ev.attempts,
            ev.elapsed.as_millis(),
            tag,
            proxy
        );
    }
}

fn run() -> leyline::Result<()> {
let session = leyline::Session::builder()
    .browser(leyline::Browser::default())
    .trace(RequestLog)
    .build()?;
let _ = session;
Ok(())
}
}

A session takes one hook. trace::Fanout::new().with(a).with(b) sends each event to several, and trace::Metrics counts them. See Count requests and watch the queues.

Streamed bodies

Trace::body receives a trace::BodyEnd once for each .stream() response, when its body ends. A buffered response has no body event, because its summary already covers the body read.

BodyOutcome is #[non_exhaustive], so a match needs a _ arm. TracingTrace writes the event at INFO with the message body and the fields id, bytes, elapsed_ms, and outcome (complete, error, or dropped). An error adds kind and error.

INFO leyline::trace: body id=9 bytes=1048576 elapsed_ms=930 outcome="complete"

To write one line per request, keep the head of each streamed request by id and write the line when its BodyEnd arrives. Write other requests from summary:

#![allow(unused)]
fn main() {
use std::collections::HashMap;
use std::sync::Mutex;

use leyline::trace::{BodyEnd, BodyOutcome, Summary, Trace};

#[derive(Default)]
struct OneLine {
    heads: Mutex<HashMap<u64, String>>,
}

impl Trace for OneLine {
    fn summary(&self, ev: &Summary<'_>) {
        let head = format!("{} {:?} {:?}", ev.method, ev.status, ev.browser);
        if ev.streamed && ev.outcome.is_ok() {
            self.heads.lock().unwrap().insert(ev.id, head);
        } else {
            println!("{} {head} {} ms", ev.id, ev.elapsed.as_millis());
        }
    }

    fn body(&self, ev: &BodyEnd<'_>) {
        let head = self.heads.lock().unwrap().remove(&ev.id).unwrap_or_default();
        let outcome = match ev.outcome {
            BodyOutcome::Complete => "complete".to_owned(),
            BodyOutcome::Failed(err) => format!("failed: {err}"),
            BodyOutcome::Dropped => "dropped".to_owned(),
            _ => "other".to_owned(),
        };
        println!("{} {head} {} bytes {} ms {outcome}", ev.id, ev.bytes, ev.elapsed.as_millis());
    }
}
let _ = OneLine::default();
}

Phase events

To handle the phases in your own code, implement the other Trace methods. The sent event carries the method and the path with the query. The head event carries the status and the response headers as received, before decompression, as &http::HeaderMap. See Trace the request lifecycle.

Next

Read Service integration to use a session inside a web service.

Service integration

This page shows how to run Leyline inside a web service or a long-running worker: share sessions, relay upstream bodies, answer with the right status, use the Tower adapter, and shut down cleanly.

Share one session

Build each session once at startup, keep it in the state your framework gives to each handler, and clone it per handler. Building resolves the profile, the TLS context, and the trust store; a clone is an Arc clone that shares the pool, the cookie jar, and the identity, so it keeps the warm connections.

Session, Response, BodyStream, and Error are Send + Sync. Body and RequestBuilder are Send. leyline::http is the http 1.x crate, so its StatusCode, HeaderMap, and Request pass to and from axum 0.8 and hyper 1 with no conversion.

A service often needs a plain session for its own APIs and a browser session for sites that check the fingerprint:

#![allow(unused)]
fn main() {
use std::time::Duration;
use leyline::trace::TracingTrace;
use leyline::{Browser, Platform, Session};

#[derive(Clone)]
struct AppState {
    plain: Session,
    browser: Session,
}

impl AppState {
    fn new() -> leyline::Result<Self> {
        let plain = Session::builder()
            .base_url("https://api.example/v1/")
            .timeout(Duration::from_secs(30))
            .trace(TracingTrace)
            .build()?;
        let browser = Session::builder()
            .browser(Browser::default())
            .platform(Platform::Windows)
            .trace(TracingTrace)
            .build()?;
        Ok(Self { plain, browser })
    }
}

fn run() -> leyline::Result<()> {
let state = AppState::new()?;
let for_handler = state.clone();
let _ = for_handler;
Ok(())
}
}

See Sessions for what each kind sends. A buffered body is capped at 100 MiB by default; set CompressionConfig::max_body_size(bytes) to what your service accepts from an upstream. See Responses.

Each request can override the session timeout and proxy with .timeout(..) and .proxy(..). See Retries and timeouts and Proxies.

Relay an upstream body

Send with .stream() so that send() returns at the response head, then pass the body on as a BodyStream, a Stream<Item = Result<Bytes, std::io::Error>> that most frameworks accept as a response body. Forward the headers from relay_headers, which drops the hop-by-hop headers of RFC 9110, section 7.6.1.

Once your service sends its status and headers, a body error can only abort the body, and the client sees a truncated response. To answer 502 for an upstream body over your limit, check content_length() before you send the head, and relay with into_decoded_stream(limit) so that a body with no declared length still stops at the limit:

#![allow(unused)]
fn main() {
use leyline::{RelayBody, Session};

const RELAY_LIMIT: u64 = 8 * 1024 * 1024;

async fn run(session: &Session) -> leyline::Result<()> {
let resp = session.get("https://upstream.example/file").stream().send().await?;
if resp.content_length().is_some_and(|len| len > RELAY_LIMIT) {
    println!("answer 502");
    return Ok(());
}
let status = resp.status();
let headers = resp.relay_headers(RelayBody::Decoded);
let body = resp.into_decoded_stream(Some(RELAY_LIMIT))?;
drop((status, headers, body));
Ok(())
}
}

After the head, read bounds each chunk and TimeoutConfig::body bounds the whole body. When your client goes away, drop the stream: Leyline resets the upstream stream or closes the HTTP/1.1 connection. See Streaming and Cancellation.

Map errors to HTTP statuses

ErrorCategory::gateway_status() gives the status a gateway answers for an error: None for Status (pass the upstream status through), 504 for Timeout, 500 for Url, Config, and Request, and 502 for the rest.

#![allow(unused)]
fn main() {
use leyline::http::StatusCode;

fn status_for(err: &leyline::Error) -> StatusCode {
    err.category()
        .gateway_status()
        .or_else(|| err.status())
        .unwrap_or(StatusCode::BAD_GATEWAY)
}

fn error_body(err: &leyline::Error) -> String {
    format!(r#"{{"error":"{}"}}"#, err.category().as_str())
}
let _ = (status_for, error_body);
}

ErrorCategory::as_str() is a stable snake_case label that leaks no upstream detail. Log the failed URL with leyline::redact_url, which masks the password and the query and drops the fragment. See Map errors to HTTP statuses for the reasons behind each status.

Log one line per request

TracingTrace logs one INFO event per request on the leyline::trace target, after redirects and retries:

INFO leyline::trace: request id=7 method="GET" url=https://***@example.com/a?*** redirects=1 status=200 version=Http2 attempts=1 elapsed_ms=84 streamed=false outcome="ok"

For a .stream() request the event fires at the head, so elapsed_ms leaves out the body. To feed your own metrics, implement Trace::summary; wrap the hook in an Arc to share it between sessions. See Logging and tracing.

Use the Tower adapter

The tower feature adds LeylineService, a tower_service::Service<http::Request<leyline::Body>> whose response is http::Response<leyline::Body> and whose error is leyline::Error. It sends each request through the session, with its cookies, redirects, retries, and presets, and returns at the response head.

• The body streams decoded and capped at max_body_size. When Leyline decodes it, the response has no content-encoding and no content-length. Trailers are not exposed.
• Every other upstream header stays, hop-by-hop headers included. Filter them with leyline::relay_headers(&headers, RelayBody::AsReceived): the headers already describe the decoded body.
• Request extensions set per-request policy: Preset, TimeoutConfig, RetryPolicy, RedirectPolicy, and ProxyConfig. A ProxyConfig bypasses the session’s ProxyPool. Session::execute reads the same extensions and returns a buffered Response.
• The response carries the final leyline::Url, the HttpVersion, and the ResponseTiming as extensions, and sets version().

#![allow(unused)]
fn main() {
#[cfg(feature = "tower")]
async fn run() -> leyline::Result<()> {
use std::time::Duration;
use futures_util::StreamExt;
use leyline::http::Request;
use leyline::{Body, HttpVersion, LeylineService, Session, TimeoutConfig, Url};
use tower_service::Service;

let mut svc = LeylineService::new(Session::new());
let mut req = Request::builder()
    .uri("https://example.com/")
    .body(Body::default())
    .expect("valid request");
req.extensions_mut()
    .insert(TimeoutConfig::new().total(Duration::from_secs(10)));

let resp = svc.call(req).await?;
let final_url = resp.extensions().get::<Url>().map(Url::as_str);
let version = resp.extensions().get::<HttpVersion>();
println!("{} {:?} {:?}", resp.status(), final_url, version);

let mut body = std::pin::pin!(resp.into_body());
let mut size = 0usize;
while let Some(chunk) = body.next().await {
    size += chunk?.len();
}
println!("{size} bytes");
Ok(())
}
}

Enable it with features = ["tower"]; the example also needs tower-service = "0.3". The response body is a leyline::Body, a Stream<Item = std::io::Result<Bytes>>, so a framework that takes a byte stream, such as axum with Body::from_stream, forwards it as it is. To forward the response, keep the status and the body and filter the headers:

#![allow(unused)]
fn main() {
#[cfg(feature = "tower")]
mod example {
use leyline::http::{Request, Response};
use leyline::{Body, LeylineService, RelayBody};
use tower_service::Service;

pub async fn relay(
    mut svc: LeylineService,
    req: Request<Body>,
) -> leyline::Result<Response<Body>> {
    let resp = svc.call(req).await?;
    let (mut parts, body) = resp.into_parts();
    parts.headers = leyline::relay_headers(&parts.headers, RelayBody::AsReceived);
    Ok(Response::from_parts(parts, body))
}
}
}

Limit and time out upstream calls

LeylineService is Clone, and a clone shares the session. Build it once, store it in the application state, and clone it in each handler. To limit the upstream calls in flight, share one tokio::sync::Semaphore in the same state. To bound one call, put a TimeoutConfig in the request extensions; a timeout is then a leyline::Error like any other, and category() maps it to a status:

#![allow(unused)]
fn main() {
#[cfg(feature = "tower")]
mod example {
use std::sync::Arc;
use std::time::Duration;

use leyline::http::{Request, StatusCode};
use leyline::{Body, LeylineService, Session, TimeoutConfig};
use tokio::sync::Semaphore;
use tower_service::Service;

const MAX_IN_FLIGHT: usize = 32;
const UPSTREAM_TIMEOUT: Duration = Duration::from_secs(10);

#[derive(Clone)]
pub struct AppState {
    upstream: LeylineService,
    slots: Arc<Semaphore>,
}

impl AppState {
    pub fn new(session: Session) -> Self {
        Self {
            upstream: LeylineService::new(session),
            slots: Arc::new(Semaphore::new(MAX_IN_FLIGHT)),
        }
    }
}

pub async fn handler(state: AppState, url: String) -> StatusCode {
    let Ok(mut request) = Request::builder().uri(url).body(Body::default()) else {
        return StatusCode::BAD_REQUEST;
    };
    request
        .extensions_mut()
        .insert(TimeoutConfig::new().total(UPSTREAM_TIMEOUT));
    let Ok(_slot) = state.slots.acquire().await else {
        return StatusCode::SERVICE_UNAVAILABLE;
    };
    let mut upstream = state.upstream.clone();
    match upstream.call(request).await {
        Ok(response) => response.status(),
        Err(err) => err
            .category()
            .gateway_status()
            .or_else(|| err.status())
            .unwrap_or(StatusCode::BAD_GATEWAY),
    }
}
}
}

LeylineService is a tower_service::Service, so tower layers can wrap it too. A layer returns tower::BoxError and can wrap the leyline::Error in its own error; Error::find walks the source() chain and returns the first leyline::Error. See Map errors to HTTP statuses.

Call a service on localhost

localhost usually resolves to ::1 and 127.0.0.1. Happy Eyeballs tries IPv6 first and IPv4 after resolve_delay (250 ms by default) or when the IPv6 attempt fails. For a service that listens only on IPv4, put 127.0.0.1 in the URL. See Network.

Shut down

Call Session::shutdown(), then drop the session. Requests in flight and new requests on the session, its clones, and its derived sessions fail with Kind::Request, and a streamed body fails on its next read. Error::is_shut_down() identifies this error, so a handler can answer 503. The session holds nothing to flush, and the pooled connections stay open.

#![allow(unused)]
fn main() {
async fn run(session: leyline::Session) {
let worker = session.clone();
let task = tokio::spawn(async move { worker.get("https://api.example/jobs").await });
session.shutdown();
match task.await {
    Ok(Err(err)) if err.is_shut_down() => println!("stopped"),
    Ok(result) => println!("finished: {}", result.is_ok()),
    Err(join) => println!("task failed: {join}"),
}
drop(session);
}
}

Session::shutdown() is a plain call. The autosave handles of a cookie jar and a device are separate: JarAutosave::shutdown() and DeviceAutosave::shutdown() are async, save the last changes, and return the result of that save. Stop the server first so that the open requests finish, then stop the sessions, then await each autosave handle:

#![allow(unused)]
fn main() {
use std::time::Duration;

use leyline::{Browser, Session};

async fn run() -> leyline::Result<()> {
let session = Session::browser(Browser::default());
let cookies = session.cookies().autosave("cookies.json", Duration::from_secs(2));

let worker = session.clone();
let jobs = tokio::spawn(async move { worker.get("https://api.example/jobs").await });

tokio::signal::ctrl_c().await?;
session.shutdown();
drop(jobs.await);
cookies.shutdown().await?;
Ok(())
}
}

For a device, Device::autosave returns a DeviceAutosave; await its shutdown() in the same place. See Accounts.

Test the service

leyline::testing::TestServer stands in for the upstream. See Testing, which also shows it standing in for a forward proxy.

Testing

Test code that uses Leyline against a local server. Leyline needs no mock layer: point a session at 127.0.0.1 and check what the server received.

Choose the session for a test

Build the session as in production and change only the trust settings and the base URL. A browser session sends HTTP/1.1 to an http:// URL and HTTP/3 only to an origin that advertised it in Alt-Svc, so a local server sees HTTP/1.1 or HTTP/2.

Add the test server

The test-util feature adds leyline::testing. List the crate again under [dev-dependencies] with the feature; Cargo unifies the features, so test builds get test-util and release builds do not:

[dependencies]
leyline-http = "0.1"

[dev-dependencies]
leyline-http = { version = "0.1", features = ["test-util"] }

A handler is any Fn(&RecordedRequest) -> TestResponse + Send + Sync + 'static. It runs on a blocking thread, so a handler that blocks delays only its own response.

Use raw or header_count to check what a parsed map hides: header case, order, and duplicates.

See the leyline::testing reference for every item.

Assert what was sent

Put the body of run in a #[tokio::test] function. Use server.url(..), which holds 127.0.0.1: with localhost, Leyline tries IPv6 first, where the server does not listen.

#![allow(unused)]
fn main() {
#[cfg(feature = "test-util")]
async fn run() -> leyline::Result<()> {
use leyline::testing::{TestResponse, TestServer};

let server = TestServer::http(|request| {
    let id = request.header("x-request-id").unwrap_or("none").to_owned();
    TestResponse::new(201).header("x-echo-id", id)
})
.await?;

let session = leyline::Session::new();
let resp = session
    .post(server.url("/items"))
    .header("x-request-id", "test-1")
    .body("hello")
    .send()
    .await?;
assert_eq!(resp.status(), 201);
assert_eq!(resp.header("x-echo-id"), Some("test-1"));

let sent = server.requests().await;
assert_eq!(sent.len(), 1);
assert_eq!(sent[0].method, "POST");
assert_eq!(sent[0].target, "/items");
assert_eq!(sent[0].body, b"hello");
Ok(())
}
}

The server sees the bytes that reached it, which is the strongest check of what Leyline sent. To see what a session prepared without a server, build it with .audit(true) and read Response::request_headers(); see Fingerprints.

Answer with a sequence

queue answers each request with the next response. Use it for retries and error handling:

#![allow(unused)]
fn main() {
#[cfg(feature = "test-util")]
async fn run() -> leyline::Result<()> {
use leyline::RetryPolicy;
use leyline::testing::{TestResponse, TestServer, queue};

let server = TestServer::http(queue([
    TestResponse::new(503),
    TestResponse::new(200).body("ok"),
]))
.await?;
let session = leyline::Session::builder()
    .retry(RetryPolicy::transient())
    .build()?;
let resp = session.get(server.url("/flaky")).send().await?;
assert_eq!(resp.status(), 200);
assert_eq!(resp.attempts(), 2);
Ok(())
}
}

Test timeouts

delay holds back the head, so it trips total and response_header. chunks holds back the body, so it trips read on a .stream() response.

#![allow(unused)]
fn main() {
#[cfg(feature = "test-util")]
async fn run() -> leyline::Result<()> {
use std::time::Duration;
use leyline::TimeoutConfig;
use leyline::testing::{TestResponse, TestServer};

const LIMIT: Duration = Duration::from_millis(100);
const STALL: Duration = Duration::from_millis(500);

let server = TestServer::http(|request| match request.target.as_str() {
    "/slow-head" => TestResponse::new(200).delay(STALL),
    _ => TestResponse::new(200).chunks(["first", "second"], STALL),
})
.await?;
let session = leyline::Session::new();

let err = session
    .get(server.url("/slow-head"))
    .timeout(LIMIT)
    .await
    .expect_err("the head arrives after the limit");
assert!(err.is_timeout());

let resp = session
    .get(server.url("/slow-body"))
    .stream()
    .timeout(TimeoutConfig::new().read(LIMIT))
    .await?;
let err = resp.text().await.expect_err("the second chunk arrives after the limit");
assert!(err.is_timeout());
Ok(())
}
}

Test over TLS

TestServer::https makes a private CA and a leaf certificate for localhost and 127.0.0.1. trust() trusts that CA only, with the system and environment roots off. The server negotiates HTTP/1.1, so a browser session uses HTTP/1.1 on it.

#![allow(unused)]
fn main() {
#[cfg(feature = "test-util")]
async fn run() -> leyline::Result<()> {
use leyline::testing::{TestResponse, TestServer};

let server = TestServer::https(|_| TestResponse::new(200).body("secure")).await?;
let session = leyline::Session::builder()
    .browser(leyline::Browser::default())
    .tls_trust(server.trust())
    .build()?;
let resp = session.get(server.url("/health")).send().await?;
assert_eq!(resp.text().await?, "secure");
Ok(())
}
}

Use your own TLS server

For a server of your own, for example one that speaks HTTP/2, generate a test CA and a leaf certificate (the rcgen crate does this) and trust only that CA:

#![allow(unused)]
fn main() {
use leyline::{Session, TlsTrustConfig};

async fn run(ca_der: Vec<u8>, port: u16) -> leyline::Result<()> {
let trust = TlsTrustConfig::new()
    .system_roots(false)
    .env_roots(false)
    .add_ca_der(ca_der);
let session = Session::builder().tls_trust(trust).build()?;
let resp = session
    .get(format!("https://127.0.0.1:{port}/health"))
    .send()
    .await?;
assert_eq!(resp.status(), 200);
Ok(())
}
}

• Put 127.0.0.1 as an IP address, or localhost as a DNS name, in the leaf’s subject alternative names. To use a host name and stay on IPv4, map it with DnsConfig::resolve_host; see Network.
• Give the CA a subject name of its own, for example CN=Test CA. When the leaf’s issuer equals its subject, BoringSSL reads the leaf as self-signed and the handshake fails with verify code 18, self signed certificate. Certificate generators often give both the same default name.

See TLS trust.

Stand in for a forward proxy

TestServer can act as a forward proxy for http:// URLs. Set its URL as the proxy: the session sends the request in absolute form, so RecordedRequest::target holds the full upstream URL, and the upstream host needs no DNS entry. An https:// URL needs a CONNECT tunnel, which TestServer does not open.

#![allow(unused)]
fn main() {
#[cfg(feature = "test-util")]
async fn run() -> leyline::Result<()> {
use leyline::Session;
use leyline::testing::{TestResponse, TestServer};

let proxy = TestServer::http(|_| TestResponse::new(200)).await?;
let session = Session::builder().proxy(proxy.url("")).build()?;
session.get("http://upstream.test/path").await?;
let seen = proxy.requests().await;
assert_eq!(seen[0].target, "http://upstream.test/path");
Ok(())
}
}

Observe requests with a trace hook

A Trace hook sees every request, once, after redirects and retries. Collect the summaries to check the method, the status, and the attempt count, for example to prove that a retry ran:

#![allow(unused)]
fn main() {
use std::sync::{Arc, Mutex};
use leyline::trace::{Summary, Trace};

#[derive(Default)]
struct Recorder {
    seen: Mutex<Vec<(String, Option<u16>, u32)>>,
}

impl Trace for Recorder {
    fn summary(&self, ev: &Summary<'_>) {
        let status = ev.status.map(|s| s.as_u16());
        if let Ok(mut seen) = self.seen.lock() {
            seen.push((ev.method.to_owned(), status, ev.attempts));
        }
    }
}

async fn run(addr: std::net::SocketAddr) -> Result<(), Box<dyn std::error::Error>> {
let recorder = Arc::new(Recorder::default());
let session = leyline::Session::builder()
    .trace(Arc::clone(&recorder))
    .build()?;
let resp = session.get(format!("http://{addr}/")).send().await?;
assert_eq!(resp.status(), 200);

let seen = recorder.seen.lock().map_err(|_| "poisoned")?;
assert_eq!(seen.len(), 1);
assert_eq!(seen[0].0, "GET");
Ok(())
}
}

See Logging and tracing.

Next

Read Fingerprints to see what a browser session sends on each layer.

Fingerprints

A browser profile is a recorded description of how one browser build looks on the wire. Leyline replays it across four layers. This page covers what a profile pins, how to audit and check a session’s fingerprint, which headers are safe to override, and how to mix and rotate identities.

What a profile pins

The session takes its TcpProfile from profiles/platforms.toml through Platform::tcp_profile; SessionBuilder::tcp_profile overrides it. Each OS applies what it can:

Every OS sets TCP_NODELAY when the profile asks for it. The SYN option order in options is chosen by the kernel and cannot be set from a socket, so only the audit’s JA4T reads it. platforms.toml has rows for Windows, macOS, Linux, and iOS. Android has none, so its TcpProfile is empty and the kernel defaults apply.

Audit a session

Build the session with .audit(true), then read Response::audit(). Without the flag, audit() returns None.

#![allow(unused)]
fn main() {
use leyline::{Browser, Platform, Session};

async fn run() -> leyline::Result<()> {
let session = Session::builder()
    .browser(Browser::Chrome152)
    .platform(Platform::Linux)
    .audit(true)
    .build()?;

let resp = session.get("https://example.com/").await?;
if let Some(a) = resp.audit() {
    println!("JA4 {} JA4H {} H2 {}", a.ja4, a.ja4h, a.h2_fingerprint);
}
Ok(())
}
}

The connection fields are computed once per session from the profile. They report what Leyline built; a packet capture shows what the connection sent, and the kernel decides what the SYN carries.

Check a fingerprint with an echo service

A TLS echo service answers with the fingerprints of the connection it received. AuditData::compare checks them against the audit of the same session.

audit::Observed::from_json reads tls.ja4, tls.ja3_hash, http2.akamai_fingerprint, and the echoed request headers from http2.sent_frames[*].headers and http1.headers: the shape that https://tls.peet.ws/api/all returns. A missing field stays None. For another service, build it with Observed::new().ja4(..).ja3(..).h2_fingerprint(..).header(name, value).

#![allow(unused)]
fn main() {
use leyline::audit::{FieldOutcome, Observed};
use leyline::{Browser, Session};

async fn run() -> leyline::Result<()> {
let session = Session::builder()
    .browser(Browser::default())
    .audit(true)
    .build()?;
let resp = session.get("https://tls.peet.ws/api/all").send().await?;
let Some(audit) = resp.audit().cloned() else {
    return Ok(());
};
let observed = Observed::from_json(&resp.text().await?)?;
let report = audit.compare(&observed);
println!("match: {}", report.is_match());
if let FieldOutcome::Mismatch { expected, observed } = &report.ja4 {
    println!("ja4 sent {expected}, service saw {observed}");
}
for header in &report.headers {
    if let FieldOutcome::Mismatch { expected, observed } = &header.outcome {
        println!("{} sent {expected}, service saw {observed}", header.name);
    }
}
Ok(())
}
}

compare returns a FingerprintReport with a FieldOutcome for ja4, ja3, h2_fingerprint, and header_order, and a HeaderOutcome (lowercase name and outcome) in headers for each header the session sent, pseudo-headers excluded. header_order compares the order of the headers that both lists hold.

JA4 and the Akamai HTTP/2 fingerprint are fixed for every profile, so a difference is a Mismatch. JA3 hashes the extensions in the order sent, and Chrome shuffles that order, so for a profile with permutes_extensions a JA3 difference is Informational; for a fixed-order profile such as Firefox or Safari it is a Mismatch. A header that a proxy or middlebox rewrites shows as a Mismatch. A proxy that ends TLS changes the result.

is_match() is true when nothing is a Mismatch or Absent; FieldOutcome::is_mismatch() tests one field. Both types implement Display: println!("{report}") prints one line per field and header.

Override headers safely

On a browser session, a header you set replaces the profile’s header of the same name in its position; Requests gives the placement of a new name.

Set accept on a script request

Keep the profile’s accept on open, follow, and the navigation presets. A page script sets its own accept on a fetch() or an XHR, for example application/json. The Xhr preset sends what a fetch() without an explicit accept sends. When the site’s script sets one, set the same value: the browser sends it too, so the fingerprint does not change.

#![allow(unused)]
fn main() {
async fn run(session: leyline::Session) -> leyline::Result<()> {
let tab = session.tab();
tab.open("https://shop.example/cart").await?;
let cart = tab
    .xhr("/api/cart")
    .header("accept", "application/json")
    .send()
    .await?;
let _ = cart;
Ok(())
}
}

Mix and rotate identities

An Identity holds two browsers and a platform. The tls() browser supplies the ClientHello, the HTTP/2 and HTTP/3 settings, and the ja3, ja4, and h2_fingerprint audit values. The http() browser supplies the User-Agent, sec-ch-ua, Accept-Language, the header shape, and the header order. The platform selects the OS that the User-Agent names, the HTTP/2 platform overrides, and the TcpProfile.

rotate_hello moves through Browser::all() order and wraps. A profile can share its ClientHello with another version; its [meta] table names that version in hello, tls() returns it, and rotate_hello counts shared ClientHellos once.

Build a session from an identity

SessionBuilder::identity(identity) sets both browsers and the platform. This session sends the headers of Chrome 154 on Linux over the ClientHello, HTTP/2 settings, and HTTP/3 settings of Chrome 152:

#![allow(unused)]
fn main() {
use leyline::{Browser, Identity, Platform, Session};

fn run() -> leyline::Result<()> {
let identity = Identity::locked(Browser::Chrome154, Platform::Linux)
    .rotate_tls(Browser::Chrome152)?;
let session = Session::builder().identity(identity).build()?;
assert_eq!(session.identity().browser(), Some(Browser::Chrome154));
Ok(())
}
}

browser, profile, and identity replace each other; the last call wins. browser and profile keep the platform in either order. identity sets the platform too, so a platform call before it has no effect and one after it replaces the identity’s platform. build() returns Kind::Config when the http() browser has no identity for the platform, for example Safari 26 on Windows. A session built with SessionBuilder::profile sends that one profile on both sides.

Change the identity of a session

Session::with_identity(identity) derives a session that sends another identity; the parent keeps its own. The derived session takes the ClientHello, the HTTP/2 and HTTP/3 settings, the headers, and the audit values from the identity, as SessionBuilder::identity does.

A user-agent header from SessionBuilder::headers still replaces the identity’s User-Agent. Session::identity() reports the identity the derived session sends. See Sessions.

#![allow(unused)]
fn main() {
use leyline::{Browser, Family, Identity, Platform, Session};

fn run() -> leyline::Result<()> {
let chrome = Identity::locked(Browser::latest(Family::Chrome), Platform::Linux);
let session = Session::builder().identity(chrome).build()?;

let next_hello = session.with_identity(chrome.rotate_hello()?)?;

let firefox = chrome.switch_family(Browser::latest(Family::Firefox))?;
let switched = session.with_identity(firefox)?;
let _ = (next_hello, switched);
Ok(())
}
}

with_identity returns Kind::Config when:

• The session has no Browser: a plain session, or one built with SessionBuilder::profile.
• The http() browser has no identity for the platform.
• The session brand, Edge or Opera, has no overlay for the http() browser on the platform. Only Chromium browsers have one.
• The policy is ProtocolPolicy::Http3 or Race and the tls() browser has no [h3] table. Under Auto, Http1, or Http2 that is accepted. See HTTP/3.

Read and test the bundled profiles

ProfileRegistry::global().get(name, version) looks up a bundled profile; Browser::profile() is the short path. The profile reference lists what ships and how to load your own.

#![allow(unused)]
fn main() {
use leyline::Browser;
use leyline::profile::ProfileRegistry;

let profile = ProfileRegistry::global()
    .get("chrome", 152)
    .expect("chrome 152 is bundled");
assert_eq!(profile.meta.version, 152);
assert_eq!(Browser::Chrome152.profile().meta.browser, "chrome");
}

The offline conformance test checks each profile’s JA4 and HTTP/2 values against its recorded reference values without the network. It needs the bench-internals feature:

cargo test -p leyline-http --features bench-internals --test it fingerprint_conformance::

It checks the profile data against the reference values; it does not capture a browser.

Next

Read Features and targets.

Features and targets

This page lists the Cargo features, how Leyline’s BoringSSL coexists with other TLS crates, and how the documentation builds. For the supported targets and building BoringSSL from source, see Supported platforms.

Cargo features

The package is leyline-http and the library is leyline. The cookie jar, system trust-store loading, and streaming bodies always compile.

Each compression feature gates both the body codec and the matching RFC 8879 certificate decompressor. Certificate compression is part of the fingerprint, so a profile that lists an algorithm whose feature is off fails at build() with a message that names the algorithm and the feature. Request compression with the codec compiled out fails the same way at send().

To start from nothing and add back what you need:

[dependencies]
leyline-http = { version = "0.1", default-features = false, features = ["compression-brotli", "socks"] }

Build with RUSTFLAGS="--cfg leyline_unstable_bssl", or enable bench-internals, to reach the BoringSSL SslContextBuilder behind TlsContext. Both are outside the semver promise.

Symbol prefixing and openssl-sys

Leyline links next to openssl-sys or boring-sys in one binary:

• BoringSSL is built with -DBORINGSSL_PREFIX=LEYLINE, so every C export is LEYLINE_<name>. The C++ structs behind SSL and SSL_SESSION are LEYLINE_ssl_st and LEYLINE_ssl_session_st, and the internal C++ code is in the bssl::LEYLINE namespace. The bindings keep the plain Rust names and carry a #[link_name] for the prefixed export.
• leyline-bssl-sys declares links = "leyline_bssl", its own key, so it does not collide with a crate that claims boringssl.
• The build reads only LEYLINE_BSSL_* variables, so a value set for boring-sys never reaches Leyline.

The crates have no fips, mlkem, or rpk feature.

Documentation build

docs.rs builds leyline-http for x86_64-unknown-linux-gnu only, which keeps the build within the docs.rs limits, with features = ["full"]. The docs cover every feature except test-util and bench-internals.

The minimum Rust version is in MSRV.

Supported platforms

This page lists the targets Leyline builds for and the tools the BoringSSL build needs.

Supported targets

leyline-bssl-sys compiles BoringSSL from source with the cc crate, from the source lists BoringSSL ships in gen/sources.json, and uses the Rust bindings committed for the target. A build needs no CMake, Go, Perl, or libclang.

On any other target, including Intel macOS (x86_64-apple-darwin), 32-bit targets, and windows-gnu, the build stops with an error that names the supported targets.

The platform a session claims is separate from the build target: a Linux build can claim Windows or macOS. See Choose a platform.

Build BoringSSL

The build compiles the BoringSSL source that ships in leyline-bssl-sys. LEYLINE_BSSL_SOURCE_PATH points the build at another BoringSSL source tree. The build applies Leyline’s patches to that tree in place.

LEYLINE_BSSL_PATH links a BoringSSL that you built yourself. The build applies no patches to it, so it must carry Leyline’s patches and be built with -DBORINGSSL_PREFIX=LEYLINE.

Other variables:

• LEYLINE_BSSL_ASSUME_PATCHED skips the patches for a tree that already carries them. It needs LEYLINE_BSSL_PATH or LEYLINE_BSSL_SOURCE_PATH.
• LEYLINE_BSSL_RUST_CPPLIB replaces the C++ standard library that the build links: by default c++ on macOS, stdc++ on Linux, none on Windows.
• LEYLINE_BSSL_SYSROOT gives bindgen a sysroot when the bindgen feature is on.

The committed bindings match the headers that ship in leyline-bssl-sys. LEYLINE_BSSL_INCLUDE_PATH needs the bindgen feature of leyline-bssl-sys; without it the build stops with an error. When LEYLINE_BSSL_PATH links a BoringSSL whose headers differ, turn on the bindgen feature too.

The build needs:

• A C and C++ compiler: Xcode Command Line Tools on macOS, GCC or Clang on Linux, the MSVC build tools on Windows. The cc crate finds it, and CC, CXX, CFLAGS, and CXXFLAGS work as they do for any cc build.
• Git. Every build runs git apply for Leyline’s patches unless LEYLINE_BSSL_ASSUME_PATCHED is set. A source checkout also uses Git to fetch the BoringSSL submodule.

On Windows, the x86_64 assembly is NASM source. When NASM is on PATH, or NASM names it, the build assembles it. Otherwise it links the objects that ship in prebuilt/x86_64-pc-windows-msvc/, which CI assembles with NASM 2.16.01 from the same source and checks on every run. LEYLINE_BSSL_SOURCE_PATH needs NASM, because the shipped objects match only the bundled BoringSSL.

The build uses the macOS deployment target that Rust uses (MACOSX_DEPLOYMENT_TARGET, 11.0 by default), honors +crt-static on MSVC, and maps the source and output directories to /build, so the libraries embed no local paths. The first build compiles BoringSSL; later builds reuse it.

Windows

Install Visual Studio Build Tools with the C++ workload. A Developer Command Prompt is not required, and NASM is optional.

With the bindgen feature, also install LLVM and set LIBCLANG_PATH to its bin directory, so bindgen finds libclang.dll.

musl

BoringSSL has C++ code, so a musl build needs a musl C and C++ toolchain. The musl-gcc wrapper from musl-tools has no C++ compiler; use a full cross toolchain, such as the musl.cc builds or the one cross ships.

1. Put the toolchain on PATH, for example x86_64-linux-musl-cross/bin.

2. Set the Rust linker to the musl compiler:

   export CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER=x86_64-linux-musl-gcc
   export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-musl-gcc
   

3. Build:

   cargo build -p leyline-http --target x86_64-unknown-linux-musl
   

The build finds <arch>-linux-musl-gcc and <arch>-linux-musl-g++ on PATH; set CC_<target> and CXX_<target> to use another compiler. The musl libstdc++ links statically.

Next

Read the Profile reference for each bundled profile and the capture it comes from.

Browser profiles

A profile is one TOML file under crates/leyline/profiles/<family>/<version>.toml that describes one browser build: its TLS ClientHello, HTTP/2 SETTINGS, HTTP/3 transport, and per-platform identity. Leyline compiles the bundled profiles into the crate and indexes them in ProfileRegistry; SessionBuilder::browser selects one. This page lists what ships and where each value comes from, then covers profile stability and your own profiles.

Bundled profiles

Without .platform(), a session claims the first platform its profile covers, in the order Windows, macOS, Linux, Android, iOS, so a mobile-only browser such as SafariIOS27 claims iOS. See Sessions.

Browser::latest(family) returns the newest profile of a family whose capture is in the family’s latest_capture list in families.toml. The list defaults to browser; Safari iOS and OkHttp add emulator, and CFNetwork uses native and emulator. The build fails when a family has no such profile. A platform_browser target resolves the same way. Browser::default() is Browser::latest(Family::Chrome).

Chromium-family profiles do not store sec-ch-ua. Leyline derives it from the major version and the ch_ua_brand field in [meta], with Chromium’s GREASE brand, version, and order rule.

What the JA4 column means

A profile records a JA4 reference value under [tls.fingerprint] and an Akamai HTTP/2 reference value under [h2.fingerprint]. The offline fingerprint_conformance test computes both from the profile’s [tls] and [h2] tables and compares them with the reference values. It gates the HTTP/2 value of every profile.