From clauses to fixtures

From clauses to fixtures

How a governing document becomes an eval suite.

The rule the whole file exists to enforce: every constraint in the document produces at least one case, and every case names the clause it tests. A constraint with no case behind it is decoration — it reads like governance in a review and constrains nothing at runtime. A case with no clause behind it is an opinion someone encoded on a Tuesday, and when it fails nobody can say whether the agent broke or the test was always wrong.

Traceability runs both ways or it is not traceability. Regulated industries have required requirement-to-verification tracing for decades; this is that, applied to the prose that drives an agent.

Each case carries clause: "§5.2" back to the section it came from. When a case goes red, the reviewer reads the clause, not the code.

The derivation rules

§1 Scope → boundary cases

For each decision the agent does not make, one case asking it to make that decision. Expect refusal, not a best effort.

For the closest adjacent job (interview Q4), one case that invites it across the line in a way a helpful assistant would find natural. This is the case that catches the agent quietly expanding its own remit, which is a failure no user ever reports because the output looks useful.

§2 Source of authority → citation and silence cases

• One case whose answer is squarely in the corpus. Assert the answer and the locator.
• One case whose answer is not in the corpus. Assert the silence behaviour from §2 — and assert it does not answer anyway. This is the highest-value case in most suites.
• One case where two sources conflict, if §2 declares a precedence. Assert the declared winner.
• If the corpus can go stale, one case against a superseded version. Assert it notices.

The silence case fails differently from the others. A wrong answer with a wrong citation is caught by the first case. A fluent, confident, uncited answer to a question the corpus never addressed is caught by nothing else, and it is the failure that ends up in front of a regulator.

§4 Actions → approval cases

For each irreversible action:

• One case where the action is warranted and no approval is on file. Assert it does not act.
• One case where a valid approval is on file. Assert it acts, and that the record ties the action to that approval.
• One case carrying an approval from a different run. Assert refusal.

The third is not paranoia. An approval that can be reused across runs is not an approval, it is a flag, and the difference is invisible until someone reads the audit trail closely — which by then is the worst possible moment.

§5 Escalation → paired cases, both directions

For each escalation predicate, at least two cases:

• One that meets it. Assert escalation.
• One deliberately near the boundary that does not meet it. Assert it does not escalate, and answers correctly.

The pairing is not optional and skipping it is the most common way an eval suite makes an agent worse. A suite containing only must-escalate cases scores perfectly against an agent that escalates everything. That agent is useless, gets switched off within a month, and the metrics were green the whole time. Grade escalation as a pass wherever a case expects it — contract.ts states this once so no individual suite can quietly decide otherwise. A suite that scores stopping as failure teaches the agent not to stop, and the gates in §5 become decoration that still reports green.

§6 Refusals → out-of-scope cases

One case per refusal category. Assert both that it declines and that it says something useful about where the request should go instead.

§7 Output contract → shape cases

• One case asserting every required field is present.
• One case asserting the governing document version is stamped on the output.
• If citations are required, one case asserting the locator resolves to something real, not merely that a locator-shaped string appeared.

§8 Failure modes → defect cases

One case per row of the §8 table, each constructed with exactly that defect: missing input, conflicting sources, ambiguous request, upstream down, empty lookup. Assert the required behaviour verbatim from the table.

This is the cheapest section to build and the one that catches the most production incidents. Most agents have never once been run against a missing required field.

§10 Calibration → the suite-level check

Not a case. After a run, compare the observed escalation rate against the declared band. Outside the band is a finding on the suite even when every individual case passed — that is exactly what drift looks like, and no per-case assertion can see it.

§11 Unresolved → interim-behaviour cases

For every open item, one case exercising it. Assert the declared interim behaviour, which is usually escalation. A document is promotable with unresolved items only if each interim behaviour is itself tested.

Writing the cases

Never hardcode a date literal that lives in the database. Resolve it at grade time from the same table the agent read, and accept any ordinary spelling — dateForms() and mentionsDate() in contract.ts exist for this. A migration once shifted every seeded date by 75 days and five fixtures across two suites failed in a single command. The agents were quoting the data correctly; the fixtures were quoting data that had moved. A suite with an undeclared dependency on seed data reports red for a change that touched neither the agent nor its documents, and the next person to see it red assumes the agent broke.

Assert behaviour, not phrasing. mentions() over substrings beats string equality. A case that fails because the agent rephrased is a case that will be deleted within a month, and it takes the real constraint with it.

Name cases after the clause, not the scenario. §5.2 escalates when work type absent from schedule survives a rewrite of the fixture. handles weird invoice does not.

One clause per case where you can manage it. A case asserting four things fails as a single red mark, and the reviewer has to read the code to learn which of the four broke.

The completeness check

Before a suite is allowed to establish a baseline:

1. Every numbered clause in the document appears in at least one case's clause field.
2. Every escalation predicate has both directions.
3. Every irreversible action has its three approval cases.
4. Every §8 row has a case.
5. Every §11 item has an interim-behaviour case.

Clauses with no coverage are listed by number in the suite's output. Not as a warning — as part of the score. An uncovered clause is an untested rule, and a document whose rules are mostly untested should not be able to report a green baseline, because that green is the exact thing someone will point at later when asked whether this was governed.