Validation atlas

The benchmark, in the open

19 benchmark datasets/cases: 18 synthetic ground-truth stress tests plus one experimentally labelled real benchmark. Every row below is generated from a typed evidence registry that is checked, before each build, against the eval ground truth on disk — so this page cannot drift from the real planted answers.

Current bundled benchmark; synthetic unless labelled real.

18 / 18

Exact synthetic-DE verdicts reproduced

Planted ground truth; each verdict graded against evals/cases/*/ground_truth.json.

False greens

A fatal finding never reads SURVIVES. Current bundled benchmark.

False positives on clean controls

Current bundled benchmark; synthetic unless labelled real.

100%

Determinism

Byte-identical certificate on re-run — seeds are pinned.

3 / 3

Composition verdicts

Deterministic composition cases generated in memory.

~0.78

Doublet AUC (real HTO)

~20% recall at the explicit expected-rate cutoff. Real labelled data.

Reading the atlas — four evidence classes

SyntheticNegative-binomial single-cell data (2000 genes) with a known answer engineered in. The verdict can be graded exactly against the planted truth.
Real · labelledA real dataset carrying experimental hashtag-oligo (HTO) multiplet labels. Scored against those wet-lab labels, not a planted answer.
Real · literatureA published biological claim re-checked against a public, same-experimental-design cohort standing in for a data-use-controlled dataset. Evaluates the finding's robustness, not a planted label.
DemoA real-workflow walkthrough with no graded outcome — shown to illustrate the pipeline end-to-end, never counted as a benchmark result.

Evidence class

Expected verdict

Status

23 of 23

SyntheticC1_batch_perfectconfound
Does not survive
Condition perfectly aliased with batch — indistinguishable from a batch artifact.
Planted finding:
GENE_1999 · up · RED
Sentinel:
Batch / condition confounding
SyntheticC2_pseudoreplicationconfound
Does not survive
Cell-level significance from a handful of donors — the classic pseudoreplication false positive.
Planted finding:
GENE_1999 · up · RED
Sentinel:
Pseudoreplication
SyntheticC3_group_imbalanceconfound
Survives
Uneven group sizes are flagged, but the effect still survives every defensible re-analysis.
Planted finding:
GENE_1999 · up · GREEN
Sentinel:
Group imbalance
SyntheticC4_covariate_confoundconfound
Does not survive
A covariate tracks the condition — the apparent effect is not identifiable.
Planted finding:
GENE_1999 · up · RED
Sentinel:
Covariate confounding
SyntheticC5_partial_confound_survivesconfound
Survives
Partial batch overlap is flagged, yet the design-matrix model still recovers the effect.
Planted finding:
GENE_1999 · up · GREEN
Sentinel:
Batch / condition confounding
SyntheticD1_discriminatediscriminate
At risk
Two findings in one dataset — one robust, one fragile — must be separated, not lumped.
robust:
GENE_1999 · up · GREEN
fragile:
GENE_1998 · up · AMBER
SyntheticE1_confounded_downdirectionality
Does not survive
A down-regulation claim that is really a confounded batch shift.
Planted finding:
GENE_1999 · down · RED
Sentinel:
Batch / condition confounding
SyntheticE2_degenerateadversarial
Does not survive
A degenerate, adversarial input that must not slip through as a real effect.
Planted finding:
GENE_1999 · up · RED
SyntheticF1_qc_fragilefragility
At risk
The result flips with the QC threshold — fragile, not fatal. Reported as at-risk.
Planted finding:
GENE_1999 · up · AMBER
SyntheticF2_detest_fragilefragility
At risk
Sensitive to the choice of DE test — survives some, not all. At-risk.
Planted finding:
GENE_1999 · up · AMBER
SyntheticF3_clustering_resolutionfragility
At risk
Depends on the Leiden clustering resolution — flagged as resolution-fragile.
Planted finding:
GENE_1999 · up · AMBER
SyntheticQ3_knockdown_faileddata_quality
Does not survive
The perturbation never took — the target was not actually knocked down.
Planted finding:
GENE_1999 · up · RED
Sentinel:
Knockdown failed
SyntheticR1_robust_uprobust
Survives
A clean up-regulated marker that holds across every defensible pipeline.
Planted finding:
GENE_1999 · up · GREEN
SyntheticR2_robust_downrobust
Survives
A clean down-regulation that survives the full multiverse.
Planted finding:
GENE_1999 · down · GREEN
SyntheticR3_robust_clean2robust
Survives
A second clean survivor — a true positive must still read as SURVIVES.
Planted finding:
GENE_1998 · up · GREEN
SyntheticS1_multiple_testingstatistical
At risk
Survives raw significance but is fragile once multiple testing is honoured.
Planted finding:
GENE_1999 · up · AMBER
SyntheticS2_double_dippingstatistical
Does not survive
Groups were defined from the same expression now being tested — circular inference.
Planted finding:
GENE_1999 · up · RED
Sentinel:
Double dipping
SyntheticS3_effect_trivialstatistical
Does not survive
Statistically significant but biologically negligible — significance is not effect size.
Planted finding:
GENE_1999 · up · RED
Sentinel:
Trivial effect size
Real · labelledHTO-labelled doublet benchmarkdoublet detection
A real single-cell dataset with experimental hashtag-oligo multiplet labels — the only benchmark scored against wet-lab truth rather than a planted answer.
Doublet score AUC:
~0.78
Recall at expected-rate cutoff:
~20%
One real labelled dataset. VERITAS surfaces doublet-driven risk with an honest recall at the explicit expected-rate cutoff — it does not claim to detect every doublet.
Real · literatureExcitatory-neuron mechanism re-checkliterature evaluation
Unscored
A published excitatory-neuron differential-expression claim re-checked on a public, same-experimental-design cohort standing in for a data-use-controlled dataset.
Cohort:
GSE174367 (public stand-in)
Outcome:
robustness re-check (unplanted)
Uses GSE174367 as a same-design public stand-in — NOT the exact data-use-controlled dataset, and not two independent cohorts. Evaluates robustness of the stated mechanism, not a planted label.
Real · literatureMicroglia mechanism re-checkliterature evaluation
Unscored
A published microglia differential-expression claim re-checked on the same public same-design cohort.
Cohort:
GSE174367 (public stand-in)
Outcome:
robustness re-check (unplanted)
Uses GSE174367 as a same-design public stand-in — NOT the exact data-use-controlled dataset, and not two independent cohorts.
DemoPBMC demonstration workflowdemonstration
Unscored
A real peripheral-blood-mononuclear-cell workflow run end-to-end to illustrate the pipeline — shown, not scored.
Outcome:
unscored walkthrough
A demonstration of the full pipeline on real PBMC data. Carries no graded benchmark result.
DemoComposition benchmark casescell-type composition
Three deterministic cell-type-composition benchmark cases, generated in memory to exercise composition-claim verification (donor-level vs cell-pooled proportions).
Cases:
3 / 3 expected verdicts
Storage:
generated in memory (not .h5ad)
Deterministic composition cases generated in memory. Distinct from the 18 synthetic DE .h5ad cases; not stored as .h5ad files.

Verify a case yourself

Several of these synthetic cases are bundled as one-click samples in the console. Sign in and run one against the live engine.

Try a sample How verdicts are decided