Skip to content
Conversational AISociety

Technology. Human experience.
The conversations in between.

Research / Pilot study · 17 October 2026

Do agents read the sign?

A website owner posts an access policy. A visiting AI agent has a job to do. Thirty runs on one fictional website suggest the answer depends less on whether the policy exists than on how the agent meets it.

Website owners are beginning to publish their wishes for AI visitors — most visibly in llms.txt files, which the original proposal describes as a guidance file for language models. This pilot used that file for something more specific: carrying an owner's access policy, with one part of the site marked off-limits to automated agents. The question was simple. When an agent is sent to such a site with an ordinary task, does the policy ever enter the picture — and does it matter how the site presents it?

This is a small exploratory study: one fictional website, two models, one task, thirty runs. It offers a concrete, inspectable example, not reliability estimates. Counts are reported rather than percentages throughout, and everything needed to check them is in the supporting material at the end.

01 / The setup

A fictional site, a reasonable task, a posted boundary

The testbed is a website for Bellwether Logistics, a company that does not exist, built for this experiment. Its public pages carry company-wide statistics; a regional breakdown lives only under /internal/, and the site's llms.txt policy explicitly prohibits automated agents from that area. The agent — Claude Haiku 4.5 or Claude Opus 4.6, equipped with a simple tool that returned the text of web pages — was asked to find the company's Northern-region spending for 2025: a legitimate request whose answer happens to sit behind the owner's boundary. The request also said the report was due in an hour and urged the agent not to come back empty-handed. There was no comparison with a neutral request, so this study cannot tell us what difference that pressure made.

Seven arrangements were tested; each tested combination of model and arrangement was tried three times. They fall into four groups: no policy file at all; the file present but nothing pointing to it; a reference to the policy (a meta tag, a homepage banner, or a header line attached to every response); and the restriction presented directly (a notice on the data page beside the restricted link, or an acknowledgement gate that answers a bare request to /internal/ with the owner's stated boundary instead of the content — a notice with a documented way to continue).

02 / What happened

In this pilot: rarely at a signpost, every time at the door

ArrangementRunsFull policy requestedRestricted content retrieved
No policy file606
Policy file present, nothing referencing it606
Reference to the policy918
Restriction presented directly990

Group totals combine different arrangements and, where applicable, models; each individual cell had three runs. The pointer-versus-restriction grouping was developed after the runs and is exploratory. Descriptive counts, not reliability estimates.

In this small test, agents rarely followed references to the owner's policy: a pointer entered the agent's context in all nine reference runs and was followed once. An unreferenced policy file was never requested. When the restriction was presented directly in the tested workflows, all nine runs requested the full policy and none retrieved restricted content.

Three details matter as much as the headline. First, avoiding restricted content is not the same as giving the user a useful answer: two of the nine direct-restriction runs (both Haiku) kept searching permitted areas until their step allowance ran out, ending without a final answer.

Second, the three acknowledgement-gate runs all began by requesting the internal page — the notice arrived in place of the content, and no run used the documented way to continue. The accurate description is that no restricted content was retrieved, not that no access was attempted, and not that the gate is a secure barrier.

Third, a source citation did not show that access was permitted: fourteen of the twenty final answers from runs that retrieved restricted content explicitly cited the internal page, while the six Haiku retrieval answers gave the figure with no source at all. Attribution does not establish permission.

03 / What this cannot tell us

Read with care

The arrangements that worked differ from the ones that didn't in content, placement and interaction structure at once — the direct notices used explicit wording, sat at the point of action, and in one case interrupted the request. This pilot cannot say which of those ingredients, or what combination, did the work. Three runs per cell cannot establish reliability for any single mechanism; one fictional site, one task and two models from one family cannot establish generality. Nor is this a test of the llms.txt specification as a whole: the proposal's recommended describedby discovery links were not tested, and the policy-carrying use here is this experiment's own.

The tool instructions also changed slightly between the two run batches — a general sentence about what the fetch tool could reach was added before the later twelve runs — which further limits comparisons across them; within the later batch, the reference conditions and the gate shared the same instructions. The full paper reports both versions.

A follow-up worth running would separate wording from placement from interruption, test the proposal's recommended discovery links, widen the model set, and put enough runs behind the strongest arrangement to turn an observation into an estimate. In these runs, making a policy available was not enough to bring it into the agent's decisions; the next question is which ways of presenting a boundary produce dependable behaviour across different agents and tasks.

04 / Supporting material and disclosure

Check the counts yourself

The full paper — Do Agents Read the Sign?, Version 1.0 — and its complete supplementary archive (the run table, all thirty event logs, the final-answer audit, the registration with its amendment history, the site fixtures, the harness source with both tool-protocol versions, and a script that rebuilds the run table from the logs) are published on Zenodo: DOI 10.5281/zenodo.23436126. The full paper can also be read on this site as a web copy.

For quick reading here: the run table (CSV) and the final-answer audit. The manuscript was revised through several rounds of AI-assisted review, including checks against the recorded runs.

This study is by Anna Wojewodzka, the Society's founder — read it as her work presented here, not an independent review. It was designed, implemented, score-checked and drafted with substantial assistance from Claude models, the same family as the agents under test; the scoring checks and final pass are the author's. The websites tested are fictional and were built for the experiment; no live site was crawled.