A value put in {{env.*}} becomes unassertable, and the report blames the application for it #87

{{env.*}} values are redacted from everything crossing into a prompt, page snapshots included. That is the right default and it is documented. What is not documented is its consequence: a step can never assert on a value that is also an environment value, because the judge is shown *** where the page shows the thing.

Reproduction

An auth recipe signs in with {{env.TEST_EMAIL}}. A test then says:

- navigate to /account and verify the page shows admin@juice-sh.op

The page shows it. The snapshot handed to the judge does not — the literal matches a known env value and is redacted in every form. The step cannot pass, and nothing explains why: the judge reports that the expected text is not on the page, which is true of what it was given and false of the application.

Found by an outside evaluator during an unrelated retest, who diagnosed it and worked around it by rewriting the auth recipe so the literal was never an env value. That is the right workaround and nobody should have to derive it.

Why it is not simply "working as intended"

The masking exists so a password never reaches a log or a prompt. An email, a username, a tenant name, an account number or an order reference are routinely put in {{env.*}} for the same reason any test data is — so the suite is not hard-coded to one environment — and they are exactly the values a test wants to assert on.

The redaction cannot tell those apart. Every {{env.*}} value is treated as a secret, so making a value configurable silently makes it unassertable.

The failure is also the most confusing shape available: the test is right, the application is right, and the report blames the application.

Worth deciding rather than assuming

• Is the distinction the tool's to make? A secrets: versus data: split in the config would be honest — the first redacted, the second substituted and visible. It also invites a user to mark a password as data, which is the mistake this guarantee exists to prevent.
• Is a warning enough? The condition is detectable without any model: at authoring time, a step whose text contains a value that will be redacted is a step that cannot pass. That is cheap, and it turns an afternoon into a line of output — the same argument as The documented OpenAI default cannot make a single call: our schema is rejected by strict structured outputs, and the error says nothing #85's second half.
• Or is this documentation? README's Trust boundaries says redaction is "a strong default rather than a guarantee against a hostile app". It does not say what this issue is about. One paragraph there is the smallest honest fix, and it should land whatever else does.

What must not change

The redaction itself. AGENTS.md is explicit — never log a secret — and the guarantee is that every known value is masked in literal and percent-encoded form everywhere it could surface. Narrowing that to make assertions easier would trade a security property for a convenience.

Related

• A failed step cannot say whether the test or the application was wrong #53 — a failed step cannot say whether the test or the application was wrong. Here it was neither, and the report blamed the application.