Harness Authoring Guide

Harness scenarios live under scenarios/cache-harness/ and are executed through scripts/cache/* or picasso harness run.

Scenario Shape

Each scenario needs:

• A scenario directory.
• A stable scenario id.
• A harness level: H0, H1, H2, or H3.
• A token budget.
• Expected metrics.
• A privacy expectation.

Episode Packages

Episode packages are written under reports/cache-harness/episodes/<episode-id>/ and include:

• config.json
• components.json
• context-packs.jsonl
• trajectory.jsonl
• commands.txt
• verification.md
• privacy.md
• metrics.json
• summary.md

Use none-local only for local debugging. Use public-safe for anything shareable.

Evaluators

Evaluators should inspect both outcome and process:

• Did verification pass?
• Did context stay within budget?
• Did stable prefix hashes repeat?
• Did cache usage improve?
• Did privacy checks pass?
• Did any benchmark escape path appear?
• Was failure attribution actionable?

Provider Cache Scenarios

Provider cache scenarios should stay deterministic by default. Use deterministic provider usage for normal scenario runs, and reserve live provider calls for the opt-in comparison script.

Required provider coverage:

• OpenAI: stable prompt_cache_key request hint and cached-token usage parsing.
• Anthropic: cache_control placement on stable tool, system, or message blocks, plus cache creation/read usage parsing.
• Gemini: implicit cache usage parsing and explicit cachedContent handle request shape.
• vLLM: stable prefix preservation and scoped cache_salt.
• Unsupported providers: no request mutation, but cache usage and miss attribution still record provider-unsupported behavior.

Do not put raw prompts, source files, memory bodies, tool output, credentials, explicit cached-content handles, or raw salts in shareable reports. Use hashes and counts unless a local-only scenario explicitly requests none-local.

Adding A Scenario

1. Create scenarios/cache-harness/<scenario-id>/README.md.
2. Add the scenario to the relevant scripts/cache/* runner.
3. Add expected metrics and failure cases.
4. Run the cache harness and privacy scripts.
5. If the scenario touches provider behavior, add or update mocked provider adapter coverage before adding any live-provider expectation.
6. Do not mark the scenario complete until reports exist under reports/cache-harness/.