# Layered Agentic Retrieval: A Publishing Architecture for Agent-Native Web Surfaces **Author:** Francesco Marinoni Moretto **ORCID:** [0009-0004-0096-4712](https://orcid.org/0009-0004-0096-4712) **Contact:** francesco.marinoni.moretto@gmail.com **Status:** Working paper **Date:** May 2026 ------------------------------------------------------------------------ ## Abstract As AI agents transition from read-only assistants to autonomous actors, the human-readable web is failing them. Current retrofitting approaches, such as schema.org-enhanced HTML or Retrieval-Augmented Generation (RAG), force agents to parse unstructured or weakly-structured corpora or expend finite token budgets on human-centric rendering pipelines. In response, a parallel publishing layer is taking shape: major AI operators — including OpenAI, Anthropic, Stripe, and Google — are increasingly exposing machine-legible surfaces optimized for autonomous traversal, deterministic retrieval, and reduced token consumption. This paper identifies and analyzes this emerging publishing pattern as Layered Agentic Retrieval (LAR). LAR is a publisher-side architectural commitment to surface deterministic, machine-legible data designed explicitly for agent traversal. We articulate LAR’s three core commitments — plain-text substrates, progressive disclosure, and workflow-organized hierarchies — and situate it within the lineages of RESTful hypermedia and the Semantic Web. By delineating the limits of current retrofitting approaches, this paper establishes LAR as a dedicated institutional layer through which publishers can expose machine-legible retrieval, policy, and operational surfaces to autonomous agents — the structural artifact onto which cryptographic attestation attaches and on which emerging legal provenance frameworks can operate. **Keywords:** agentic commerce, agent-mediated retrieval, web architecture, knowledge graph, semantic web, RESTful hypermedia ## Abbreviations ACP Agentic Commerce Protocol (OpenAI/Stripe). ADK Agent Development Kit (Google). A2A Agent2Agent (Linux Foundation, originally Google). HATEOAS Hypermedia as the Engine of Application State (Fielding, 2000). HAL Hypertext Application Language. JSON-LD JSON for Linked Data (W3C). LAR Layered Agentic Retrieval (this paper). MCP Model Context Protocol (Anthropic, server-side). RAG Retrieval-Augmented Generation. RDF Resource Description Framework (W3C). SPT Shared Payment Tokens (Stripe, used by ACP). UCP Universal Commerce Protocol (Google + retail coalition). WebMCP Browser-native tool exposure standard (W3C Web ML CG, distinct from MCP). ## 1. Introduction A small set of operators — OpenAI, Anthropic, Stripe, Apideck, Google’s Agent Development Kit team, WordLift, Microsoft’s NLWeb group — are publishing knowledge to AI agents in a shape that does not look like a website. The pieces of this shape have local working names; the whole does not. This paper proposes a name — Layered Agentic Retrieval (LAR) — and articulates the architectural commitments that distinguish what these operators ship from conventional web publishing or crawler-friendly retrofit. Each of these operators publishes a separate agent-facing layer, in order to expose content optimized for agent traversal rather than human rendering. While this pattern is already emerging in production, the vocabulary that lets merchants, researchers, and policy actors discuss it as a single architectural object is missing. The paper’s contribution is naming and framing, not formal specification or benchmarking. It situates LAR in its theoretical lineage, distinguishes it from adjacent approaches (notably Retrieval-Augmented Generation and schema.org-enhanced HTML), and identifies the empirical and legal questions that remain open. ## 2. The pattern emerging in production OpenAI’s developer documentation site exposes `/codex/llms-full.txt`: a single markdown file containing the complete documentation, regenerated on every doc update.[1](#note1) Google’s Agent Development Kit ships an equivalent.[2](#note2) Apideck publishes a three-tier disclosure pattern: `/llms.txt` as a summary index, `/llms-full.txt` as an enumerated index of every doc page, individual `.md` files at each leaf, and a programmatic `md/index.json` for machine traversal.[3](#note3) Other developer-tool vendors — Meteor, FastMCP, assistant-ui, Chkk, Mintlify — ship the same shape.[4](#note4) Anthropic’s published skills are organised similarly: each skill exposes its name and description in YAML frontmatter (the manifest), with SKILL.md as the entry point and additional resources — markdown documentation, executable scripts, and bundled assets — in subdirectories the agent loads only when its task requires them.[5](#note5) The surfaces[6](#note6) examined here share observable structural properties: the markdown and JSON files appear to be regenerated alongside doc updates rather than scraped from rendered HTML. This pattern is consistent with shared upstream source material with the human-facing artifacts, though it does not prove it. These concrete forms differ in their surface organisation, but three properties recur across all of them: **Root declaration.** A root surface (whether a single file or a manifest) declares what is published and links to deeper material. **Format separation.** Narrative content is held in markdown while structured content requiring deterministic parsing is held in JSON, with the two kinds separated rather than interleaved. **Explicit machine-resolvable links.** Links between files are explicit and machine-resolvable, allowing an agent to fetch the root, decide which subtree its task requires, and walk down only as far as it needs — without rendering, scraping, or inferring structure from HTML. The protocol layer is taking shape in parallel. Microsoft’s NLWeb defines how a website exposes structured data to agents through a knowledge-graph-backed natural language query interface.[7](#note7) Google’s Universal Commerce Protocol (UCP), launched at NRF 2026, standardises commerce capabilities across AI surfaces.[8](#note8) OpenAI and Stripe’s Agentic Commerce Protocol (ACP), open-sourced in September 2025, addresses the same transactional layer through agent-mediated execution within conversational surfaces.[9](#note9) WebMCP, a W3C Community Group Report (latest draft 10 February 2026), exposed in early 2026 Chrome 146 preview builds behind the “Experimental Web Platform Features” flag, lets a website declare itself as a tool surface for in-browser agents.[10](#note10) These initiatives address different layers — site-level discovery, commerce capability negotiation, agent-mediated transaction, browser-native tool exposure — but share an underlying assumption: agents need surfaces designed for them, and those surfaces will be declared, not inferred. External infrastructure providers are operationalising this layer distinction. Google’s Search Central guidance, published 15 May 2026, explicitly states that publishers “don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in generative AI search,” while a parallel section recognising autonomous agents notes that “protocols like Universal Commerce Protocol (UCP) are emerging that will allow Search agents to do more.”[11](#note11) Cloudflare’s Agent Readiness Score, launched 17 April 2026, operationalises assessment of publisher-side agent infrastructure across four dimensions (discoverability, content, bot access control, capabilities) plus commerce protocols, with adoption tracked across 108,595 successfully scanned domains as of May 2026.[12](#note12) The asymmetry is informative: Google scopes its guidance to Search optimisation while signalling that agentic experiences require different infrastructure; Cloudflare operationalises that infrastructure with empirical adoption data. The publisher/agent distinction LAR articulates is becoming externally recognised in the ecosystem the paper describes. ## 3. Three architectural commitments The operators building these surfaces deliberately, rather than as accidental byproducts of documentation tooling, share three commitments that distinguish their output from retrofit. **Plain text instead of pages.** Markdown for narrative, JSON for facts, YAML for configuration. No DOM, no JavaScript, no rendering pipeline. An agent parses these formats deterministically in milliseconds, with no model call required for extraction. The llms.txt specification states the rationale directly: context windows are too small to handle most websites in their entirety, and converting complex HTML pages (with navigation, ads, and JavaScript) into LLM-friendly plain text is difficult and imprecise.[4](#note4) **Progressive disclosure instead of flat exposure.** A manifest at the root, topic indexes one layer down, detail files at the leaves. The agent walks the tree to the depth its task requires and stops. Token budget is controlled by the agent, not by the publisher’s page boundaries. This is the structural inverse of a sitemap. Anthropic’s Claude Skill Design has codified this pattern as progressive knowledge disclosure for tool packaging, where agents see only skill names and short descriptions initially and load detailed files only when needed.[5](#note5) MajesticLabs’s hierarchical-agents pattern applies the same principle to codebase documentation.[13](#note13) **Organised by workflow instead of by topic.** A merchant’s human site is organised by what the merchant wants to say: about us, our story, our products, contact. A LAR surface is organised by what the agent was tasked to do, and the discretion the principal delegated for doing it: availability, inventory, policies, pricing, the endpoint that commits the transaction. The structure mirrors the agent’s mandate, not the merchant’s marketing. Each commitment is explicitly stated in the documentation of the operators cited above; what is proposed here is the framing that holds the three together as a single architectural pattern. The theoretical justification is the economics of the LLM context window: an agent operating under finite token budget pays a cost proportional to every byte loaded, and topic-organised structures force the agent to traverse material organised around the publisher’s narrative concerns before reaching the data its task actually requires. Workflow organisation collapses this asymmetry by aligning the structure of the surface with the structure of agent decisions, allowing token usage to scale with task complexity rather than with publisher verbosity.[14](#note14) The discipline of building a good LAR surface is therefore not to design the path the agent will follow, but to remove obstacles from every path the agent might take. The work is to expose relevant data efficiently, without forcing the agent through structure that reflects the publisher’s mental model rather than the agent’s task. These three commitments are what make the architecture layered rather than retrofit, agentic rather than crawler-friendly, retrieval-shaped rather than render-shaped. LAR is not Retrieval-Augmented Generation. RAG is a consumer-side pattern that retrieves probabilistically over unstructured or weakly-structured corpora to augment generation; LAR is a publisher-side commitment to structure the source so that retrieval can be deterministic when the consuming agent respects the layered structure.[15](#note15) Two terms are used in this paper with deliberate distinction, doing different work even when they describe the same artifact. *Machine-readable* describes formats a machine can deterministically parse — the standard sense established by RDF, HATEOAS, and the broader Semantic Web tradition. *Machine-legible* describes content a machine can meaningfully act upon. JSON exposing five generic product attributes is machine-readable; JSON exposing twenty content-rich attributes that capture provenance, occasion-fit, and operational constraints is machine-legible. The conceptual distinction between format and meaning has surfaced in adjacent literature on digital product labels and Agent Experience design; the two-term vocabulary proposed here is intended to make the distinction usable in operator and policy discourse. LAR commits to both: machine-readable formats (plain text, JSON) carrying machine-legible content (workflow-organised, identity-bound, attestation-ready). ## 4. Theoretical lineage and recent convergence Although the vocabulary proposed here is new, the pattern it names is not. Manifest-driven traversal of linked, machine-readable resources has scholarly lineage. The hypermedia-driven REST architectures articulated by Fielding (2000) — commonly known as HATEOAS — describe this shape: a client fetches a root resource, discovers links at runtime, and traverses progressively to fulfil its task.[16](#note16) The early Semantic Web vision of Berners-Lee, Hendler and Lassila (2001) extended the pattern to machine-readable linked data, anticipating agents that would compose actions across distributed semantic resources.[17](#note17) LAR is best understood as an instantiation of this lineage adapted to the LLM-agent consumer: plain text and JSON calibrated for context-window absorption rather than HAL or RDF, organisation by task-workflow rather than resource-entity, consumer that performs semantic parsing on prose rather than following typed links. The architectural innovation, where one exists, lies in the calibration to the consumer rather than in the traversal pattern itself. Recent practitioner thinking has converged on a parallel architectural pattern from the consumer side. Andrej Karpathy (OpenAI founding member, former Tesla AI Director) published his LLM Wiki sketch (2026) via a post on X and a gist on GitHub — the genre by which influential AI thinking now first reaches the field — describing an LLM agent ingesting a user’s raw sources, compiling them into a persistent markdown knowledge base, and maintaining it across queries.[18](#note18) The sketch arrives at architectural decisions that overlap with LAR’s publisher-side commitments: markdown substrate, three-layer separation, rejection of RAG as default. Karpathy’s wiki is what an agent maintains for one user’s consumption; LAR is what a publisher exposes for any agent’s traversal. The convergent design choices on opposite ends of agent-mediated information flow are the more interesting observation than the architectural overlap itself. A strict architectural critique might observe that the systems discussed above operate at different layers of the software stack: llms.txt addresses documentation discovery, MCP standardizes tool invocation, and ACP governs transactional execution. LAR does not claim these protocols are equivalent, nor that they require a shared implementation layer. The claim is narrower and structural: finite-context autonomous agents create common economic pressures across discovery, retrieval, and execution. In response, operators are increasingly exposing machine-legible institutional surfaces designed to reduce ambiguity, traversal cost, and rendering overhead for autonomous systems. LAR is proposed as a publisher-side conceptual vocabulary for understanding this convergence. ## 5. The limits of retrofit The dominant publisher-side approach to making content readable by agents is structured data embedded in HTML using schema.org vocabulary, typically expressed as JSON-LD inside `