Evaluation

ariadnev states its proof boundary explicitly. Each tier below proves something specific, and none of them proves general provider parity or safe arbitrary workspace mutation.

#Static contracts

ariadnev validate lints every skill and compiles every workflow graph for structural, authority, recovery, evidence, and capability defects. --check additionally fails on README matrix drift. av run <workflow> --validate proves a canonical graph without probing any runtime — it proves the graph, not the behaviour of a provider executing it.

#Tier 1 — static skill quality

ariadnev eval scores kit skill quality from the source alone. It is free, always on, and the floor a release must clear.

#Tier 2 — behavioural suite

ariadnev eval --suite runs the source-checkout behavioural benchmark in fresh, disposable fixtures. The runner is a strict JSON argv array spawned without a shell; it receives only the case prompt on stdin and an explicit bootstrap environment allowlist rather than the CLI's ambient environment — GH_TOKEN, API keys, and NODE_OPTIONS are not inherited. Fixture suites exercise routing, trajectory, recovery, authority, and duplicate-effect behaviour against the committed graphs; they cover the scenarios they encode and are not a general correctness proof.

ariadnev eval --suite \
  --runner '["agent-command","arg-1"]' \
  --runtime-provider provider-id \
  --runtime-version exact-version \
  --model exact-model

The command emits one allowlisted, redacted JSON report and exits non-zero when Tier 1 fails, a hard release floor fails, or trusted evidence remains incomplete. Capabilities the controller cannot mediate — network, external services — stay explicitly unsupported rather than being claimed by the executor.

#Tier 3 — LLM judge (opt-in)

When ARIADNEV_EVAL_CMD is set, an LLM judge scores skills as well. It never replaces the two tiers above.

#Runtime probes

Capability-gated Codex and Claude Code probes report only the pinned runtime and model that actually ran. A pinned-runtime result says nothing about a different runtime or a different model.

#Proof-boundary ledger

Tier	Proves	Does not prove
Static contracts	Structural, authority, recovery, evidence, capability shape of the kit source.	Any runtime behaviour of a provider executing that graph.
Tier 1	Skill source quality clears the release floor.	That a skill behaves correctly at run time.
Tier 2	Committed scenarios (routing, trajectory, recovery, authority, duplicate-effect) hold in fresh fixtures.	Scenarios not encoded in the fixture suite; general correctness.
Tier 3	An LLM judge's rating of skill copy against a rubric.	Anything about actual execution or a different runtime.
Runtime probes	The pinned runtime + model that actually ran did so with the reported result.	A different runtime, a different model, or an unpinned combination.

#What is not proven

• That any provider behaves correctly at run time merely because its graph validates.
• General provider parity.
• Safe execution of arbitrary workspace mutation — active safe-change-delivery execution stays policy-denied in public builds until a real side-effect executor and approval surface exist.