Generated on 2026-06-05.
The comparison harness runs ax-grep through Puppeteer and compares named
role output against agent-browser snapshot.
Operational rule: run comparison suites sequentially. agent-browser and
Chromium can exhaust the host when several compare:static* runs overlap. Check
for existing browser processes before a run and confirm cleanup afterward.
The historical named role overlap score is intentionally strict: it is an
exact match over normalized role:name pairs, with no fuzzy or containment
matching. It is useful for tracking whether ax-grep is reproducing
agent-browser snapshot output, but it can understate agent usefulness when the
missing items are mostly static text.
The harness also reports agentReadiness scores. These are still based on the
same exact normalized role:name matches, but split by agent-facing use:
referenceRecall: how much of theagent-browsernamed output appears inax-grep.candidatePrecision: how much of theax-grepnamed output appears inagent-browser.actionableRecall: exact recall for links, buttons, fields, tabs, and other operation targets.navigationRecall: exact recall for links, headings, landmarks, and search.contentRecall: exact recall for headings, images, table/list structure, and static text.structuralContentRecall: exact recall for headings, images, table/list structure, and other non-StaticText content roles.textRecall: exact recall for StaticText roles only.score: weighted summary for agent parsing: actionable 40%, navigation 25%, structural content 20%, precision 15%. StrictcontentRecallandtextRecallremain visible because text-heavy pages can expose useful gaps, but raw StaticText volume should not dominate the agent-usefulness score.
The static harness also emits cliAgentSummary, which scores the actual
agent-facing --agent compact JSON envelope rather than raw tree overlap. It uses
top-level agent routing status, pageCheck, structured content evidence, source links and source quality,
readability, requested verification status, follow-up nextSteps, searchResults, and suggestedActions to estimate how directly an agent can
decide whether to read, open, or retry a page. averageCliAgentScore
in gateSummary tracks that higher-level usefulness separately from
agentReadiness, which remains an exact agent-browser snapshot overlap
metric.
minCliAgentScore enforces the same readiness floor per included target, so a
weak search, page-check, or browser-retry case cannot be hidden by strong
average results.
When a per-target floor fails, weakAgentTargets lists the affected category,
URL, scores, status, and primary action for quick follow-up.
averageAgentExecutorScore is the executor-focused aggregate. It combines the
schema, routing, next, expected-outcome, signal, read-target, command,
browser-retry, continuation, response, diagnostic, and verification fields that
subagents need to run a search/page-check loop without reconstructing intent
from the raw tree.
minAgentExecutorScore applies the executor floor per included target.
The score includes action-schema completeness, so run-command actions need
both a human-readable command and raw commandArgs, read-current actions need
readFrom, and interact-browser actions need an explicit browser-interaction
signal.
averageActionSchemaScore tracks that schema completeness directly across the
gate-included targets.
averageSearchResultActionScore tracks whether compact search results include
rank-specific openResult, command, and raw commandArgs, so search agents
can open alternate results without reconstructing commands.
averageAgentRoutingIntentScore tracks whether agent.routingIntent correctly
summarizes the primary action as reading current payload, opening a URL,
searching, retrying with browser HTML, requiring browser interaction, or
stopping.
averageAgentContinuationModeScore tracks whether agent.continuationMode
maps that intent to the executor-facing mode: read, command, browser,
capture-html, inspect, or stop.
averageAgentNextScore tracks whether agent.next is a canonical executor
payload that agrees with continuationMode and mirrors the primary action's
command, read pointer, URL, browser interaction, and terminal fields.
averageAgentNextShortcutScore tracks whether top-level agent.next*
shortcuts mirror that canonical agent.next payload.
averageAgentRunbookShortcutScore tracks whether top-level agent.runbook*
shortcuts mirror the nested loop runbook contract.
averageAgentExpectedOutcomeScore tracks whether agent.expectedOutcome
describes the success condition for the next step, including read pointers when
the next step is evidence reading.
averageAgentPlanShortcutScore tracks whether top-level
agent.expectedOutcome* and agent.executionPlan* shortcuts mirror the nested
next-step contract.
averageAgentSignalScore tracks whether agent.signals exposes structured
content, verification, search result, source link, browser, response, and
diagnostic signals needed for fast agent routing.
averageContentEvidenceMetadataScore tracks whether pageCheck.contentEvidence
items include source and bounded score metadata, so agents can prioritize
semantic evidence over fallback excerpts.
averageReadabilityReasonScore tracks whether compact page checks preserve
concise readability reasons, so agents can understand why a page is readable,
thin, blocked, or worth retrying.
averageAgentReadabilityReasonScore tracks whether the compact top-level
agent summary repeats concise readability reasons, so agents can route from
the first object before drilling into pageCheck.
averageAgentPageMetadataShortcutScore tracks whether agent.page* mirrors
root page metadata such as canonical URL, language, author, dates, and
structured-data types.
averageAgentSemanticSummaryScore tracks whether agent.semanticSummary and
top-level agent.semantic* shortcuts preserve semantic tree counts, role-group
counts, heading/landmark outline flow with parent context, keyboard shortcut hints, in-page links, top role/heading/landmark/named role fields with direct paths,
interactive and focusable state, link URL, button description, image,
table/list structure, form-field input hints, description, value, resolved relation targets, choice, state, and unavailable-subtree shortcuts for quick page-shape routing.
State scoring includes parsed top-state fields so agents do not need to parse
ARIA state strings.
averageAgentBarrierShortcutScore tracks whether top-level agent.topBarrier*
shortcuts mirror the highest-priority page barrier.
averageAgentStructuredShortcutScore tracks whether top-level structured
content counts and top* shortcuts mirror the first table, FAQ, code block,
resource, media item, section paths/selectors, navigation/media structure,
provenance, offer, dataset, identity, timeline, contact point, and best structured
read-target shortcut.
averageAgentReadTargetScore tracks whether agent.readTargets points to
payload fields that actually exist and are worth reading, and whether
read-current actions mark the matching target as primary.
averageAgentTopReadTargetShortcutScore tracks whether agent.topReadTarget*
mirrors the first read-target entry for fast routing without scanning
agent.readTargets.
averageAgentAlternativeActionShortcutScore tracks whether top-level
agent.alternativeAction* shortcuts mirror the first non-primary action.
averageAgentHandoffScore and averageAgentBriefExecutorStepScore also cover
handoff detail preservation. Search handoffs must keep executable result/source
choices with snippets and command args; answer handoffs must keep selected
evidence text/reasons; read handoffs for forms and action targets must keep URL
templates, fields, selectors, methods, and encoding; diagnostic handoffs must
keep selected signals and quality gates. This prevents the compact handoff from
turning into an opaque "retry/open this" instruction.
averageAgentResultChoiceScore, averageAgentSourceChoiceScore, and
averageAgentActionListScore cover the same problem outside the handoff.
Search choices keep snippets, freshness dates, and sitelinks; source choices
keep source-link text, snippets, selectors, and executable commands; source-link
actions keep sourceLinkRef so agents can jump back to the exact
pageCheck.sourceLinks[n] item.
averageAgentTopActionShortcutScore tracks whether agent.topAction* mirrors
the first action candidate, including execution, priority, command/read target,
URL, and source-link reference.
averageAgentResultCountScore tracks whether agent.resultCount is zero for
non-search pages and at least the compact result count for search pages.
averageAgentChoiceCountScore tracks whether executable choice-count shortcuts
match their result, form, action-target, and source-link source counts.
averageAgentTopChoiceShortcutScore tracks whether agent.topChoiceKind,
path, label, URL, and command arguments mirror the first executable result,
source, form, or action-target choice for fast subagent routing.
averageAgentTopResultChoiceShortcutScore tracks whether agent.topResultChoice*
mirrors the first search result choice, including URL, rank, open-result value,
command arguments, source quality, freshness/relevance, matches, sitelinks, and
selection reason.
averageAgentTopSourceChoiceShortcutScore tracks whether source-link specific
top-level shortcuts mirror the first executable source choice, including source
type, hints, score, and selection reason.
averageAgentEvidenceCountShortcutScore tracks citation, answer-evidence,
read-target, and action count shortcuts against their agent arrays.
averageAgentTopCitationShortcutScore tracks whether agent.topCitation*
mirrors the first citation item, including path, kind, confidence, reason, URL,
and score.
averageAgentSignalCountShortcutScore tracks signal severity and failing
quality-gate count shortcuts against agent.signals and agent.qualityGates.
averageAgentTopQualityShortcutScore tracks whether agent.topSignal* and
agent.topQualityGate* mirror the first signal and quality gate for fast
accept/block routing without scanning diagnostic arrays.
averageAgentProblemShortcutScore tracks whether agent.problemSignalKind,
severity, message, and agent.failingQualityGate* mirror the first
warning/error signal and first failing quality gate, including gate severity and
score, so agents can explain blocked pages without scanning diagnostic arrays.
averageAgentSourceLinkCountScore tracks whether agent.sourceLinkCount is
zero for search pages and matches compact pageCheck.sourceLinks for ordinary
content pages.
averageAgentFormActionCountScore tracks whether top-level agent.formCount
and agent.actionTargetCount match compact pageCheck.forms and
pageCheck.actionTargets, so agents can detect hidden forms and JSON-LD/OpenSearch
actions before scanning nested page-check arrays.
averageAgentFormActionChoiceScore tracks whether agent.formChoices and
agent.actionTargetChoices preserve the compact form/action target IDs, paths,
selectors, URL templates, query inputs, submit text, first-field hints, and
methods needed for subagent form execution and selection loops.
averageAgentTopFormActionChoiceShortcutScore tracks whether top-level
form/action-target shortcuts mirror the first executable form and action target.
averagePageLinkCommandScore tracks whether compact pageCheck.primaryLinks
and pageCheck.sourceLinks include direct command and commandArgs, so
agents can open page links without reconstructing fetch flags.
averageAgentBrowserNeedScore tracks whether agent.needsBrowserHtml agrees
with the primary action: browser HTML retry actions should require browser
HTML, while URL search recovery, alternate-result recovery, read-current, and
retry-later actions should not.
averageAgentBrowserHtmlScore tracks whether browser HTML fallback payloads
preserve capture file/script data plus nested reason, command, and command-args
fields across agent.next, execution plans, executor steps, and handoff steps.
averageAgentPageKindScore tracks whether agent.pageKind mirrors the root
payload kind, so agents can route from the top-level agent object without
re-reading analysis.kind or the envelope root.
averageAgentAlternativeActionCountScore tracks whether
agent.alternativeActionCount matches the deduplicated compact follow-up
actions left outside agent.primaryAction, so agents can know whether a page
has useful alternatives before scanning nested action arrays.
averageAgentUsabilityScoreConsistency tracks whether agent.usabilityScore
matches the documented compact quality heuristic derived from status,
readability, confidence, evidence, search results, source links, and
verification status.
averageAgentEvidenceQualityScoreConsistency and
averageAgentSourceQualityScoreConsistency track whether top-level evidence
and source quality scores match the compact evidence/source arrays, so agents
can compare payload quality before reading every candidate item.
averageAgentBestReadTargetScore tracks whether agent.bestReadTarget and its
count, score, primary flag, and reason match the primary or highest-scored
agent.readTargets entry, so agents can start reading the best compact field
without sorting candidates.
averageAgentDiagnosticCountScore tracks whether top-level diagnostic severity
counts and agent.topDiagnostic* match the compact diagnostics array, so agents
can distinguish warnings from hard errors before drilling into diagnostic
messages.
averageAgentVerificationCountScore tracks whether top-level verification
requested/found/missing counts match the compact verification object, so agents
can decide whether requested evidence is complete before reading details.
averageAgentVerificationQueryScore tracks whether
agent.verificationFoundQueries and agent.verificationMissingQueries preserve
the exact matched and missing --find query lists and whether the top matched
or missing query shortcuts mirror the first items; agent.handoff and
agent.executor carry the same lists for brief subagent loops.
averageAgentResponseMetadataScore tracks whether agent.responseStatus,
agent.responseOk, agent.responseContentType, and agent.finalUrlChanged
mirror the compact envelope response fields, so agents can judge fetch health
from the top-level agent object.
averageAgentHiddenSignalScore tracks whether hidden pageCheck groups such
as hydration, API endpoints, app config, app/mobile hints, provenance,
policies, JSON-LD facts, and resource metadata are present at valid payload
paths and discoverable through at least one read target when available. This is
the executor-focused counterweight to raw accessibility-tree overlap: these
signals are often useful to subagents but absent from browser accessibility
snapshots.
averageAgentHiddenSignalCountScore tracks whether top-level
agent.hiddenSignalCount, agent.hiddenReadTargetCount, and
agent.bestHiddenReadTarget* match those hidden groups and read-target
shortcuts.
averageAgentTopHiddenSignalShortcutScore tracks whether
agent.topHiddenSignal* mirrors the first hidden metadata, API, config, or
provenance signal.
averageAgentHiddenCommandShortcutScore tracks whether hidden hydration, safe
GET-like API, and app-hint URLs keep executable top-level follow-up commands.
averageAgentBrowserAdvantageScore tracks whether those hidden pageCheck
signals create a concrete agent-browser advantage when they exist, rather than
only matching visible accessibility-tree roles.
The higher-level CLI agent score also credits hidden pageCheck signal groups
and recoverable browser-HTML retry actions. A page with little visible text can
still be useful to a subagent when it exposes metadata read targets or a
runnable browser-capture handoff.
When ax-grep produces a ready, high-scoring agent payload with content
evidence, the static comparison treats the page as usable before applying raw
tree-size failure classes such as thin-reference challenge or over-collection.
This keeps agent-browser advantage cases visible in the gate even when the raw
static tree is larger than the browser snapshot.
averageAgentCanContinueScore tracks whether agent.canContinue agrees with
the primary action execution class, so recoverable errors with runnable actions
do not look terminal and usage/input errors without actions do not look
actionable.
averageAgentPrimaryExecutionScore tracks whether agent.primaryExecution
matches agent.primaryAction.execution, so agents can route from the shortcut
field without rereading the full action object.
averageAgentPrimaryShortcutScore tracks whether agent.primaryActionName,
reason, priority, command, URL, rank, read-from, source-link reference, and
browser shortcuts mirror agent.primaryAction, so agents can continue from
top-level routing fields.
averageAgentExecutorShortcutScore tracks whether agent.executorActionName,
decision, mode, operation, confidence, terminal/continue flags, command
arguments, read-from, URL, target, and expected-outcome shortcuts mirror
agent.executor, so subagents can route the next step without parsing the full
executor object.
averageAgentHandoffShortcutScore tracks whether agent.handoffActionName,
decision, mode, operation, answer status, confidence, terminal/continue flags,
priority, command arguments, read-from, URL, target, and expected-outcome shortcuts
mirror agent.handoff, so brief loops can run from top-level fields when they
do not need the full handoff object.
averageAgentAnswerShortcutScore tracks whether agent.answerPlanStatus,
confidence, reason, next action, gap count, citation IDs, first answer-evidence
metadata, command arguments, after-interaction command, read-from, and URL
shortcuts mirror agent.answerPlan and
agent.answerEvidence, so agents can decide whether to answer or continue
without parsing the full plan object.
averageAgentSourceSearchProvenanceScore tracks whether opened-result payloads
with sourceSearch.selectedResult or sourceSearch.alternateResults expose
matching agent.readTargets, so agents can inspect original SERP provenance
before trusting or recovering from an opened page.
averageAgentSourceSearchShortcutScore tracks whether top-level
agent.sourceSearchQuery, locale, verification-query count/top query,
engine/search URL, selected rank/title/URL, selected command, and first
alternate command mirror the source-search payload for quick SERP recovery
decisions.
The command shortcuts are exposed as sourceSearchSelectedCommandArgs and
sourceSearchAlternateCommandArgs.
averageAgentRecommendedMetadataScore tracks whether search pages with a
recommendedResult repeat its URL, title, rank, source, relevance,
official-source hint, selection reason, and command args on the top-level
agent object for quick routing.
averageAgentSearchDecisionScore and averageAgentPageDecisionScore also check
top-level agent.searchDecision* and agent.pageDecision* shortcuts, so agents
can route without reopening the nested decision objects.
Terminal actions such as read-content and use-evidence are treated as
usable without executable commands when execution is read-current and a
readFrom pointer is present, because the compact payload already contains the
evidence an agent should read. Browser-interaction actions are also valid
without commands when execution is interact-browser; in those cases another
static fetch would not advance the page state.
The --agent payload intentionally removes repeated routing data: top-level
diagnostics are represented as agent.diagnosticCodes, repeated primary actions
are omitted from suggestedActions and verification/page-check action slots,
page-level alternatives are suppressed when verification has selected
use-evidence,
search-page link/action follow-ups are omitted when searchResults already
carry the decision surface, and search output is capped to the first five
results plus any out-of-window recommended result. Opened-result payloads omit
engine attempts once sourceSearch records the selected result, but keep compact
selected/alternate SERP candidates with executable open commands for failure
recovery, preserving custom fetch options such as --timeout and
--user-agent. Page checks also
skip common global-navigation headings, links, and buttons, and omit extra
external primary links when source links are already present, so repository and
documentation pages route agents toward page content instead of site chrome.
Fetch failures that still have a target URL emit an executable browser-HTML
retry command, preserving --find checks for the next run. When browser HTML is
already supplied through --html-file or --stdin, the compact agent payload
does not ask for another browser retry. Parsed search result pages keep
agent.canUseFetchedHtml true even when their page-readability score is low,
because the result cards remain usable for open/refine routing.
Captured blocker pages still keep challenge/login/paywall diagnostics; the
follow-up action changes to browser-state inspection instead of another capture
loop. HTTP error actions are status-aware, so missing URLs and transient server
errors no longer all collapse into a browser-HTML retry; missing opened search
results and opened-result verification failures can route directly to an
alternate original SERP candidate when one matches the missing --find text.
The default baseline does not force a viewport. A shared viewport can be tested
with AX_LITE_COMPARE_VIEWPORT=WIDTHxHEIGHT, but the default run is kept stable
to avoid changing the benchmark shape unexpectedly.
For state-sensitive pages, AX_LITE_COMPARE_SETUP=path/to/setup.js evaluates a
setup script in both Puppeteer and agent-browser before extraction. This keeps
exact-match scoring intact while making page state explicit.
| URL | ax-grep nodes | agent-browser lines | named role overlap | action recall | nav recall | content recall | agent score |
|---|---|---|---|---|---|---|---|
https://example.com |
4 | 3 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
https://www.wikipedia.org |
140 | 105 | 0.57 | 0.97 | 1.00 | 0.04 | 0.80 |
https://developer.mozilla.org/en-US/docs/Web/Accessibility |
315 | 286 | 0.56 | 0.74 | 0.89 | 0.15 | 0.68 |
https://news.ycombinator.com |
710 | 501 | 0.75 | 0.82 | 0.82 | 0.63 | 0.78 |
https://github.com/features |
764 | 538 | 0.90 | 0.88 | 0.95 | 0.93 | 0.92 |
https://libraries.io/npm/typescript |
382 | 609 | 0.49 | 0.95 | 0.95 | 0.17 | 0.80 |
https://www.npmjs.com/package/typescript |
16 | 15 | 0.50 | 0.67 | 0.80 | 0.50 | 0.72 |
Run with pnpm compare:korea.
| URL | ax-grep nodes | agent-browser lines | named role overlap | action recall | nav recall | content recall | agent score |
|---|---|---|---|---|---|---|---|
https://ko.wikipedia.org/wiki/%EB%8C%80%ED%95%9C%EB%AF%BC%EA%B5%AD |
5713 | 7088 | 0.44 | 0.82 | 0.85 | 0.14 | 0.69 |
https://www.hani.co.kr/ |
998 | 992 | 0.42 | 0.50 | 0.48 | 0.23 | 0.48 |
https://www.korea.kr/ |
569 | 494 | 0.47 | 0.66 | 0.69 | 0.24 | 0.59 |
https://www.yonhapnewstv.co.kr/ |
566 | 448 | 0.79 | 0.79 | 0.83 | 0.79 | 0.81 |
Run with pnpm compare:static URL....
Run with pnpm compare:static:agent for the smaller executor-focused regression
set that exercises readable pages, listings, forum-style links, and a search
diagnostic while tracking averageAgentExecutorScore.
In the current run, the gate summary includes 8 targets and excludes 4
diagnostics; averageAgentExecutorScore is 1.00 and
averageAgentHiddenSignalScore is 1.00 for the included executor targets.
averageAgentBrowserAdvantageScore is also tracked so the hidden-metadata
fixture proves more than raw accessibility-tree overlap.
The set includes a synthetic hidden-metadata gate whose browser snapshot only
contains a visible heading while pageCheck exposes 13 hidden head, script,
policy, app-link, provenance, and JSON-LD signals.
This path fetches HTML and runs extract(html) from the static entry without
Chrome, jsdom, WebView, layout, or script execution. agent-browser is used only
as the reference snapshot for comparison.
| URL | fetched bytes | static nodes | agent-browser lines | named role overlap | action recall | nav recall | content recall | agent score |
|---|---|---|---|---|---|---|---|---|
https://example.com |
528 | 5 | 3 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
https://www.wikipedia.org |
120361 | 193 | 105 | 0.57 | 0.97 | 1.00 | 0.04 | 0.77 |
https://news.ycombinator.com |
34665 | 700 | 498 | 0.74 | 0.81 | 0.81 | 0.64 | 0.77 |
https://www.yonhapnewstv.co.kr/ |
47910 | 630 | 440 | 0.51 | 0.75 | 0.78 | 0.75 | 0.72 |
Run with pnpm compare:static:diverse.
| Category | URL | class | fetched bytes | static nodes | agent-browser lines | named role overlap | action recall | nav recall | content recall | agent score |
|---|---|---|---|---|---|---|---|---|---|---|
| News index | https://www.bbc.com/news |
usable | 317290 | 672 | 427 | 0.48 | 0.52 | 0.65 | 0.55 | 0.55 |
| News article | https://www.npr.org/2025/03/11/nx-s1-5324543/ntsb-dca-mid-air-collision-american-black-hawk |
usable | 106490 | 481 | 201 | 0.33 | 0.83 | 0.89 | 0.48 | 0.70 |
| News portal stress | https://www.theguardian.com/international |
over-collected | 1429586 | 3829 | 1225 | 0.30 | 0.90 | 0.64 | 0.27 | 0.68 |
| Government service | https://www.gov.uk/foreign-travel-advice |
usable | 111369 | 714 | 698 | 0.53 | 0.97 | 0.99 | 0.49 | 0.81 |
| Accessibility guide | https://www.nottinghamshire.gov.uk/global-content/how-to-create-accessible-content/how-to-make-web-pages-accessible/checklist-web-page |
usable | 31747 | 239 | 250 | 0.49 | 0.70 | 0.76 | 0.33 | 0.61 |
| Ecommerce fixture | https://books.toscrape.com/ |
usable | 51294 | 482 | 528 | 0.61 | 0.88 | 0.91 | 0.77 | 0.82 |
| Reddit legacy | https://old.reddit.com/r/programming/ |
challenge | 136514 | 1255 | 1 | 0.00 | 0.00 | 0.00 | 1.00 | 0.20 |
| Reddit modern | https://www.reddit.com/r/programming/ |
challenge | 8438 | 53 | 1 | 0.00 | 0.00 | 0.00 | 1.00 | 0.35 |
| X social challenge | https://x.com/NASA |
needs-browser | 277862 | 38 | 35 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Instagram social challenge | https://www.instagram.com/nasa/ |
shell | 882680 | 3 | 1 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
Run with pnpm compare:tokens URL....
This serializes both browser-injected and static SSR extraction into compact
agent prompt text and estimates token cost with cl100k_base. It also measures
the recommended --agent compact JSON payload, so the benchmark can compare raw
tree prompts with the actual CLI payload agents should use. The prompt text
includes role, name, state/value, and selectors for interactive nodes.
Token gate averages skip browser references that are only a tiny shell while
static or agent output contains substantially more inspectable payload. Those
thin browser snapshots are counted separately as excludedThinBrowserReference
instead of distorting static/browser and agent/browser ratios.
| URL | browser nodes | browser tokens | static nodes | static tokens | static delta | static/browser ratio |
|---|---|---|---|---|---|---|
https://example.com |
4 | 37 | 5 | 29 | -8 | 0.78 |
https://www.wikipedia.org |
140 | 1339 | 193 | 1292 | -47 | 0.97 |
https://news.ycombinator.com |
704 | 14503 | 700 | 6356 | -8147 | 0.44 |
https://www.yonhapnewstv.co.kr/ |
568 | 14397 | 630 | 10877 | -3520 | 0.76 |
Run with pnpm compare:tokens:diverse.
| Category | URL | browser nodes | browser tokens | static nodes | static tokens | static/browser ratio |
|---|---|---|---|---|---|---|
| News index | https://www.bbc.com/news |
554 | 9617 | 672 | 6606 | 0.69 |
| News article | https://www.npr.org/2025/03/11/nx-s1-5324543/ntsb-dca-mid-air-collision-american-black-hawk |
504 | 9122 | 481 | 4152 | 0.46 |
| Government service | https://www.gov.uk/foreign-travel-advice |
722 | 19115 | 714 | 6477 | 0.34 |
| Ecommerce fixture | https://books.toscrape.com/ |
455 | 7014 | 482 | 3599 | 0.51 |
| Reddit legacy challenge | https://old.reddit.com/r/programming/ |
6 | 58 | 1264 | 9343 | 161.09 |
| X social challenge | https://x.com/NASA |
314 | 8041 | 38 | 237 | 0.03 |
| Instagram social challenge | https://www.instagram.com/nasa/ |
35 | 640 | 351 | 1883 | 2.94 |
Run with pnpm compare:static:korea-social and
pnpm compare:tokens:korea-social.
This target set covers Clien, Ruliweb, DCInside, Google/Bing/Startpage Search,
X/Twitter, and Instagram. The static comparison benchmark first tries plain HTML
fetch. If the response looks like a bot challenge, login shell, or empty
client-rendered shell, it falls back to agent-browser rendered HTML through
document.documentElement.outerHTML before running the static extractor.
Search and social targets stay in the benchmark as diagnostics, but are not included in the gate summary because their logged-out public views are anti-bot, hydration, and personalization sensitive. In this run, the gate summary includes 4 targets and excludes 5 diagnostics; the average gate agent score is 0.698 and the average static/browser token ratio is 0.395.
| Category | gate | HTML source | class | static nodes | agent-browser lines | named role overlap | action recall | nav recall | content recall | agent score | static/browser token ratio |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Clien home | gate | fetch | usable | 469 | 657 | 0.522 | 0.799 | 0.806 | 0.053 | 0.675 | 0.286 |
| Clien post | gate | fetch | usable | 573 | 1167 | 0.234 | 0.553 | 0.542 | 0.008 | 0.483 | 0.303 |
| Ruliweb post | gate | fetch | usable | 396 | 297 | 0.620 | 0.974 | 0.978 | 0.821 | 0.892 | 0.610 |
| DCInside post | gate | fetch | usable | 1217 | 358 | 0.233 | 0.951 | 0.957 | 0.429 | 0.740 | 0.381 |
| Google search | diagnostic | fetch | reference-challenge | 1 | 5 | 0.000 | 0.000 | 0.000 | 0.000 | 0.150 | 0.110 |
| Bing search | diagnostic | fetch | volatile | 152 | 126 | 0.380 | 0.719 | 0.590 | 0.091 | 0.511 | 1.249 |
| Startpage search | diagnostic | fetch | reference-challenge | 85 | 61 | 0.861 | 0.963 | 0.957 | 0.625 | 0.878 | 0.341 |
| X social | diagnostic | fetch | usable | 156 | 36 | 0.169 | 1.000 | 0.900 | 0.500 | 0.750 | 0.193 |
| Instagram social | diagnostic | fetch | usable | 36 | 115 | 0.255 | 0.293 | 1.000 | 0.130 | 0.543 | 0.145 |
Notes:
- Ruliweb can require rendered HTML fallback in some runs, but this run fetched useful static HTML directly.
- Clien matching improved after benchmark normalization started stripping icon font private-use glyphs and leading menu bullets from comparable names.
- DCInside preserved action/navigation signals, moved from
over-collectedtousable, and lowered static/browser token ratio below 0.50 after compact static extraction started pruning unnamed leaf wrappers. - Google Search returned a bot/interstitial shell in the browser reference path.
- Bing Search is volatile in this environment: fetch or rendered HTML can expose
useful search UI or unrelated image-search affordances, but the exact
reference comparison is not stable enough for a gate yet. Search diagnostics
can be classified as
volatileinstead ofusable. - Startpage can return useful fetch HTML, but this run hit a suspended-connection
captcha page in the browser-derived reference path. Embedded CSS-in-JS text is
now excluded from static names, but the target remains a
reference-challengefixture. - Instagram can alternate between login-only and fuller logged-out shells in
this environment; keep it diagnostic even when a run scores as
usable.
Run with pnpm compare:static:china-japan and
pnpm compare:tokens:china-japan.
This target set covers Chinese and Japanese encyclopedia, news, portal, forum, developer, search, and video/social pages. Search, video/social, and pages whose reference navigation fails in this environment stay in diagnostics. In this run, the gate summary includes 7 targets and excludes 6 diagnostics; the average gate agent score is 0.654 and the average static/browser token ratio is 0.544.
| Category | gate | HTML source | class | static nodes | agent-browser lines | named role overlap | action recall | nav recall | content recall | agent score | static/browser token ratio |
|---|---|---|---|---|---|---|---|---|---|---|---|
| China Wikipedia | gate | fetch | usable | 4907 | 8518 | 0.421 | 0.866 | 0.871 | 0.050 | 0.687 | 0.277 |
| People China portal | diagnostic | fetch | reference-missing | 912 | 0.000 | 1.000 | 1.000 | 1.000 | 0.850 | 0.482 | |
| Xinhua portal | gate | fetch | usable | 1041 | 1076 | 0.706 | 0.973 | 0.973 | 0.167 | 0.772 | 0.453 |
| Douban home | gate | fetch | usable | 700 | 952 | 0.757 | 0.930 | 0.893 | 0.095 | 0.738 | 0.307 |
| Baidu search | diagnostic | fetch | needs-browser | 250 | 7 | 0.000 | 0.000 | 0.000 | 1.000 | 0.200 | 0.677 |
| Bilibili home | diagnostic | fetch | usable | 231 | 217 | 0.523 | 0.754 | 0.770 | 0.436 | 0.660 | 0.371 |
| Japan Wikipedia | gate | fetch | usable | 5349 | 11774 | 0.311 | 0.599 | 0.618 | 0.072 | 0.491 | 0.649 |
| NHK News | diagnostic | fetch | reference-missing | 490 | 0.000 | 1.000 | 1.000 | 1.000 | 0.850 | n/a | |
| Qiita TypeScript tag | gate | fetch | usable | 674 | 893 | 0.645 | 0.719 | 0.741 | 0.508 | 0.693 | 0.308 |
| Hatena IT hotentry | gate | fetch | usable | 1675 | 1775 | 0.566 | 0.922 | 0.937 | 0.258 | 0.739 | 0.782 |
| 5ch board | gate | fetch | usable | 780 | 360 | 0.105 | 0.574 | 0.600 | 0.316 | 0.459 | 1.031 |
| Yahoo Japan search | diagnostic | fetch | needs-browser | 54 | 158 | 0.187 | 0.327 | 0.293 | 0.000 | 0.279 | 0.177 |
| Niconico home | diagnostic | fetch | needs-browser | 212 | 373 | 0.123 | 0.186 | 0.196 | 0.018 | 0.159 | 0.228 |
Notes:
- China Wikipedia became usable after the benchmark stopped treating Wikipedia table-of-contents section numbers as part of comparable link names and static extraction started auto-detecting wiki-like HTML to preserve more article links by default.
- Xinhua and Douban are the strongest Chinese gate targets in this run.
- People China fetches usable HTML, but
agent-browsernavigation is blocked in this environment, so the target is diagnostic until a stable reference path is available. - Baidu search is unstable across runs. It can collapse to a tiny feedback shell or expose a larger fetched search page; keep it diagnostic.
- Japan Wikipedia is usable but still has low exact content recall on the large article body.
- NHK fetches static HTML, but Puppeteer and
agent-browserboth hit HTTP/2 navigation failures in this environment. Token ratio is reported asn/awhen the browser reference is unavailable. - Qiita and Hatena are useful Japanese gate targets; Hatena remains a token-cost stress case.
- 5ch became usable after reference comparison hardening, forum thread metadata normalization, auto-detected forum link-farm limits, and pruning redundant listitem wrappers around links/buttons. It remains a token-cost stress case at roughly parity with browser injection.
- Simple static pages line up well.
example.commatched the important named roles exactly. - Wikipedia exposes a large language
<select>.ax-grepcan still unroll options for agent operation, but the comparison harness now disables option unrolling to matchagent-browser snapshotmore closely. - Wikipedia language links use both visible article-count text and descriptive
titleattributes.ax-grepnow follows accessible-name priority more closely by using link contents before title fallback. - MDN uses many custom elements.
ax-grepnow prunes simple custom-element wrappers, but host elements that expose state, ids, or shadow content still need deeper handling. - MDN ad-like placements can be excluded in comparison mode with
excludeLikelyAds. The general extractor keeps this off by default so callers do not silently lose content. - A shared comparison viewport is available through
AX_LITE_COMPARE_VIEWPORT=WIDTHxHEIGHT, but it is opt-in because responsive pages can change the benchmark shape significantly. - Hacker News relies on layout tables. The comparison harness normalizes Chrome's
LayoutTableCellrole tocelland removes punctuation-adjacent whitespace, improving overlap from 0.64 to 0.75. - The comparison harness normalizes common role vocabulary differences such as
imagevsimg,paragraphvsp, andStaticTextvstext. libraries.io/npm/typescriptis the stable package-registry-like sample.- The new agent-facing metrics show a different picture than raw overlap on Wikipedia and Libraries.io: static-text recall is low, but actionable and navigation targets are mostly preserved. That distinction better matches the goal of making pages tractable for agents.
- Korean samples cover a large encyclopedia article, two news-like pages, and a
public portal. The Korean Wikipedia page is intentionally heavy and is kept in
compare:korearather than the default sample script. hani.co.krtimed out waiting for Puppeteer network idle during the baseline run and used the DOMContentLoaded state. Keep it as a news-site stress case, but do not treat it as a tightly stable target yet.- Korean live pages can shift by a few nodes or snapshot lines between runs as headlines, ads, and embedded widgets update.
yonhapnewstv.co.krcurrently lines up best among the Korean samples across exact overlap, content recall, and agent score.- Static SSR extraction is viable for simple and server-rendered pages. It works especially well on Hacker News and reasonably on Yonhap News TV without any browser runtime.
- Static SSR extraction can prune some non-exposed menu content from HTML
alone. The most important signal so far is a collapsed control with
aria-expanded="false"andaria-controls; pruning the controlled subtree reduced Wikipedia static tokens from 11,183 to 1,292 and improved exact overlap from 0.05 to 0.57. - Static SSR extraction now skips non-semantic payload tags, summarizes large
child lists, and collapses repeated template-like subtrees. This keeps raw SSR
payloads from turning into unbounded prompt input, while preserving an
explicit
notethat nodes were omitted. - Compact static extraction prunes unnamed leaf wrappers such as decorative spans, emphasis tags, empty inputs, and line breaks. Ancestor accessible names are computed before pruning, so useful link/button names are preserved while prompt-only wrapper noise is removed.
- Static SSR extraction cannot account for computed CSS, responsive layout, client-only rendering, open shadow roots, iframe documents, or post-load DOM mutation. Treat it as a lightweight agent parsing fallback, not an AXTree replacement.
- Static SSR extraction is not automatically cheaper in prompt tokens, but it can be competitive when collapsed controlled regions are pruned. It is now slightly cheaper than browser injection on Wikipedia and still cheaper on Hacker News and Yonhap News TV.
- Token cost needs its own benchmark gate. Agent-readiness can be acceptable while prompt cost is unacceptable, especially on SSR pages with large hidden menus, language selectors, or template payloads.
- Diverse targets show why benchmark categories matter. Government, ecommerce, and article pages preserve useful action/navigation signals; large news portals are good stress tests; Reddit/X/Instagram are better treated as social/challenge fixtures because public logged-out views often collapse to shell, login, or bot-protection states.
- Diverse token results show static extraction is often cheaper on server rendered news, government, and ecommerce pages. Social sites are inconsistent: X's fetched shell is tiny compared with the browser view, old Reddit is the opposite in this environment, and Instagram exposes enough SSR payload to make static more expensive than the rendered shell.
- Shell/challenge classification is required because exact overlap and agent score can look deceptively good when both static and reference snapshots are nearly empty.
- AP News and Ars Technica were tested as additional candidates but omitted from
compare:static:diversebecause the reference snapshot timed out in this environment. Reuters returned HTTP 401 from plain fetch and is also omitted from the automated diverse set. npmjs.comcurrently serves a Cloudflare challenge in the sample environment. The baseline is useful as a challenge-page fixture, not as a package-page content fixture.
- Improve custom-element/shadow-host pruning without losing useful selector targets.
- Add explicit benchmark gates for actionable and navigation recall once a stable target set is chosen.
- Compare browser and static extraction side-by-side on the same target set to decide when the Worker-compatible path is good enough.
- Tune static pruning controls for hidden menus, select/options, and repeated template regions against the diverse benchmark set.
- Support authenticated/cached sessions for
npmjs.comif the real npm package page remains useful as a target. - Add more real WebView smoke tests once Android/iOS host projects exist.