Disclosure: This article was generated with AI and reviewed against the source transcript and the research linked below.

Suppose a repository tells a coding agent to run a particular test command before changing its payment logic. The command uses a local test database and exercises a failure that once escaped review. That instruction has an identifiable purpose. A reviewer can ask whether the command ran and whether it caught the relevant problem.

Now suppose the same file contains a long description of every folder, a collection of general programming advice, and several outdated setup instructions. Evaluating the file becomes harder. The useful requirement is buried among material whose contribution nobody has measured.

That distinction is the useful starting point for reading the research discussed in Pivot to AI’s video “Your AGENTS.md file doesn’t actually do anything”. The video is sharply skeptical. Its underlying question deserves a precise answer: what job does each instruction perform, and how would we know whether it helped?

Read the result with its scope attached

The researchers behind Evaluating AGENTS.md compared coding agents with repository context files and without them, using established benchmark tasks and tasks from repositories with developer-written files. Their September 2026 abstract reports no general improvement in task success and an average inference-cost increase exceeding 20%. It also says agents followed the instructions and identifies nonstandard coding practices as a useful purpose for these files.

Those findings describe the evaluated settings. They do not establish that every instruction file, model, task, or private repository behaves identically. They also do not justify ignoring an owner’s operating constraints. A requirement can remain necessary even when it does not improve a benchmark completion score.

The practical implication I draw is narrower: treat claims about improved agent performance as claims to test. Keep the reason for a mandatory operating rule explicit.

Separate the reason from the prose

Consider a hypothetical instruction: “Run the complete test suite for every change.” It sounds thorough, but its usefulness depends on the task and the environment. A spelling correction and a payment-state change expose different failure modes. A full suite might also require services unavailable in the current checkout.

A more reviewable instruction names the affected behavior, the appropriate check, and the response when that check cannot run. For example, a repository could identify the integration check needed after changing payment transitions, explain its local prerequisites, and require an honest report if the environment is missing.

This example is a proposed documentation practice, not a result measured by the paper. Its appeal is that it gives the instruction a testable role. Someone maintaining the repository can discover whether the named check still exists, whether it exercises the intended behavior, and whether the prerequisite description remains correct.

The same discipline applies to a rule that prevents a costly or unauthorized operation. State the operation, the boundary, and the allowed alternative. Its success criterion is respecting that boundary. Deleting it to improve a speed metric would optimize the wrong outcome.

Evaluate a documentation change like a small intervention

A team that wants to know whether a proposed context-file change improves coding performance can design a bounded comparison on its own hardware and approved tasks. This is an evaluation plan, not a recommendation to start paid agent runs.

Choose several representative tasks before seeing the results. Define what counts as a correct patch, which checks provide evidence, and what a reviewer must inspect. Record the agent version, repository revision, environment, and exact instruction-file bytes. Without those details, an apparent documentation effect could actually be a different model or a different starting point.

Compare the proposed wording with an approved baseline while retaining mandatory safety and owner rules in both conditions. Track task correctness alongside elapsed time, tool use, and review effort. A patch that arrives quickly but needs a substantial rewrite has moved work to the reviewer.

Repeated runs may differ. A single successful attempt is useful evidence about that attempt; it is a weak basis for declaring a general improvement. If a team cannot afford a broader comparison, it can still document the limits of what it observed.

Give instructions a maintenance path

A context file becomes easier to maintain when each unusual instruction points toward its source of truth. A command can point to the relevant package script. An architectural constraint can point to the decision that established it. A restricted operation can identify the owner’s policy.

That arrangement helps a human reviewer investigate contradictions. If a command changes, there is a concrete reference to update. If an exception is temporary, it can have a clear removal condition. General advice that merely restates a language guide has a different maintenance burden from a repository-specific exception.

The resulting file may become shorter, but length is an indirect measure. The stronger test is whether a maintainer can explain why each requirement exists and inspect the evidence that it is being followed.

Start with one instruction that repeatedly causes confusion. Identify the decision it should change, verify its supporting command or reference, and rewrite it so the expected behavior can be checked. Keep the old and new versions with the review result. That creates something more useful than another argument about the ideal size of an AGENTS.md file: a record of what changed and what happened.

Sources: Pivot to AI, “Your AGENTS.md file doesn’t actually do anything”; Gloaguen and colleagues, Evaluating AGENTS.md, version 3, revised September 29, 2026. The repository and evaluation examples above are illustrative proposals, not reported experiments.