Published measurement contract · v1.0

A score is only useful when its reference frame survives inspection.

Holdout estimates whether observed brand movement exceeds matched provider-wide movement. It does not convert association into causality, and it breaks a series when the evidence cannot support continuity.

Adjusted movement=Raw brand movement−Matched control movement

Immutable frame

Every observation records provider and API surface, requested and observed model identity, behavior epoch, prompt pack, retrieval policy, ground truth, grader, sampling, locale, controls, runner, commit, and evidence source.

Fail-closed comparison

Any changed measurement dimension makes two frames incompatible by default. A directed, predeclared overlap bridge must name every changed dimension and pass minimum evidence thresholds.

Clustered uncertainty

Prompt families—not repeated answers—are the independent resampling units. The interval is estimated over adjusted cluster deltas. The demo’s −1 to +9 interval crosses zero, so its verdict is inconclusive.

Provider source truth

Providers remain separate. Holdout does not silently reweight an incomplete panel or publish a universal composite. Floating model aliases are never treated as stable historical versions.

Supported statement The brand moved four points after subtracting matched control movement in this recorded fixture.

Required caveat The interval includes zero, and the fixture is fictional rather than live provider evidence.

Prohibited claim The intervention caused a four-point gain.

Operational boundary

The private canary uses an official provider API only. GPT-5.6 Sol produces schema-constrained atomic claim extraction, while deterministic rules handle exact facts wherever possible. Canary output is not inserted into the public study or passport. Recorded fixtures are labelled at every visible layer. No consumer UI is scraped.

Read the full engineering methodology, limitations, privacy policy, and beta gates in the repository’s docs/ directory.