Guide

Read a known URL through a bounded escalation ladder.

agbrowse fetch is not search. It reads one candidate URL through a 6-phase adaptive escalation ladder and returns evidence. Use it after a search tool or user has produced a URL and you need content extraction with safety checks.

Basic fetch

agbrowse fetch https://example.com --json
agbrowse fetch https://example.com --json --trace
agbrowse fetch https://example.com --selector main --max-bytes 60000
agbrowse fetch https://example.com --browser never
agbrowse fetch https://example.com --browser required
agbrowse fetch https://example.com --allow-third-party-reader
agbrowse fetch https://example.com --browser-session user
agbrowse fetch https://example.com --identity chrome

Escalation ladder

The fetcher executes phases in order. Content scoring runs after each phase to decide whether to escalate further.

Escalation controls

Content processing

Feed parsing

When the fetched content is an RSS, Atom, or JSON Feed document, the feed parser extracts structured items (title, date, author, summary, tags, media URL) and formats them as evidence. This runs automatically during the reader phase.

Structured table/heading extraction

The structured extractor pulls headings, tables (with caption, headers, rows), ordered/unordered lists, code blocks, and JSON-LD from fetched HTML. This supplements the prose-only Defuddle extraction with grid data that agents can reason over.

BM25 lexical reranking

When a search query is available, the BM25 filter trims a long document to its top-K most query-relevant paragraphs (preserving document order). This uses tf-idf scoring with configurable topK and minScore parameters.

Candidate discovery

The candidate discovery module extracts and ranks URLs found inside fetched pages. Each URL is classified into a lane (official, package, academic, community, realtime, archive, or fetch) with SSRF rejection, tracking-parameter normalization, deduplication, and lane-weighted scoring. This feeds into the agbrowse search pipeline for multi-URL evidence collection.

WAF profile detection

The challenge detector identifies WAF providers from response headers, cookies, body patterns, and status codes. Recognized profiles:

• Cloudflare – managed challenge + Turnstile
• Akamai – Bot Manager
• AWS WAF
• Imperva / Incapsula
• DataDome
• PerimeterX

Detection results inform escalation decisions (e.g. whether TLS-impersonation or a stealth browser might succeed) and are recorded in the attempt trace.

Security mitigations

SSRF DNS pinning

All curl-based fetches (including TLS-impersonation) pre-resolve DNS and pin the result via curl --resolve. This closes the TOCTOU window where a DNS response could change between validation and connection, preventing DNS rebinding attacks against internal networks.

Validated redirect loop

Instead of following redirects blindly with curl -L, the fetcher runs a per-hop validated redirect loop (up to 5 hops). Each redirect target is re-validated through validateFetchUrl and the DNS rebinding guard before following, ensuring no redirect chain can escape to private IP space.

Multi-hop header parse

The redirect loop correctly surfaces the real finalUrl from curl's effective URL after multi-hop redirects, so downstream evidence always records the actual landing page.

Output

In --json mode the selected content field is bounded before serialization so stdout stays parseable. Results include contentBytes, contentLimitBytes, and contentTruncated fields. Truncation means only the CLI output was compacted, not that the source was rejected.

Boundary

The fetcher detects WAF/auth/paywall signals and can report them, but it does not solve CAPTCHA, bypass access controls, or use private credentials without explicit user-session mode. Automated CAPTCHA solving, credential stuffing, and stealth are forbidden. Human assistance (browser-grade headers, user session, human resolves) is allowed only with explicit opt-in flags.