Validation Suite

11 preregistered checkpoints across 7 pipeline stages — all must pass

The pipeline enforces a gate-and-proceed model. The validate_pipeline.py orchestrator reads all per-stage *.json outputs and exits with code 1 if any critical checkpoint fails, blocking downstream Nextflow steps.

11 Total checkpoints

9 Pipeline-blocking (critical)

2 Report-only (non-critical)

7 scVI sub-tests

Stage 0 — Preregistration Integrity

Checkpoint 0 · Global · CRITICAL

Preregistration hash verification

The SHA256 hash of the canonical JSON serialization of PREREG must match the recorded value in PREREG_HASH. Any modification to analysis parameters after hash registration constitutes a preregistration deviation and must be declared in the manuscript.

SHA256 exact match required

How to set the hash:

# After finalizing config/preregistration.py:
python3 -c "from config.preregistration import compute_hash; print(compute_hash())"
# → Paste output into PREREG_HASH = "..." in the same file
# → git commit and push
# → Register on OSF/AsPredicted, paste DOI into registration_doi field

Stage 1 — Alignment QC

Checkpoint 1a · Per-sample · CRITICAL

Spatial barcode mapping rate

Fraction of post-QC cells with valid puck coordinates (direct match or 1-Hamming correction). Low rates indicate puck file mismatch, barcode degradation, or poor RNA quality.

≥ 70% cells spatially mapped

Checkpoint 1b · Per-sample · CRITICAL

Velocity-compatible unspliced coverage

Two metrics from Velocyto output: (1) fraction of genes with detectable unspliced counts; (2) median unspliced/spliced ratio. Both must be in range for dynamical velocity to be well-estimated.

≥ 40% genes with unspliced · Median U/S ratio: 0.10–0.40

Failure remediation:

Symptom	Likely cause	Action
Spatial mapping < 70%	Wrong puck file paired to sample	Re-check `sample_sheet.csv`
Unspliced < 40%	GTF missing intron annotations	Re-run STAR with `eisar`-generated intron GTF
U/S ratio > 0.40	Genomic DNA contamination	Add DAPI/live-dead sort step; tighten `max_pct_mito` threshold
U/S ratio < 0.10	snRNA degradation; very fast splicing	Consider using `GeneFull` (pre-mRNA) as proxy

Stage 2 — Seurat QC

Checkpoint 2 · Per-sample · CRITICAL

Minimum cell count post-QC

Per-sample cell count after applying all exclusion criteria (min genes, max % mito, min/max counts). Samples below threshold are excluded from the merged analysis entirely — not imputed or padded.

≥ 1,000 cells per sample

Exclusion criteria (frozen in PREREG["exclusion_criteria"]):

{
    "cells": {
        "min_genes_detected": 200,
        "max_pct_mitochondrial": 20.0,
        "min_counts": 500,
        "max_counts": 25000,      # doublet proxy
    },
    "samples": {
        "min_cells_post_qc": 1000,
        "min_iop_elevation_mmhg": 15,
        "min_spatially_mapped_fraction": 0.70,
    }
}

Stage 3 — scVI Validation (7 tests)

All 7 tests must pass. The validation module produces validation_report.json with per-test passed, metric, threshold, and details fields. The Nextflow VALIDATE_PIPELINE process reads this file and blocks downstream steps if any test fails.

Test 1 · Posterior predictive check · CRITICAL

Count distribution fidelity

For a random subset of 1,000 genes: draw 100 posterior samples from the scVI model and compare each gene's predicted distribution to observed counts via 2-sample KS test. Genes pass if KS p > 0.05.

≥ 50% of tested genes pass (KS p > 0.05)

Test 2 · Batch silhouette · CRITICAL

Batch correction quality

Compute silhouette scores for batch labels (should be low — batches mixed) and cell type labels (should be preserved — cell types separated) in the scVI latent space. A combined score captures the tradeoff.

bio_silhouette − batch_silhouette > 0.5

Test 3 · Reconstruction accuracy · CRITICAL

Expression recovery

For 200 randomly sampled cells: compare scVI-imputed normalized expression to observed counts (normalized to 10,000) via Spearman correlation. Low median r suggests the model has collapsed or underfits.

Median Spearman r > 0.4 across HVGs

Remediation: Increase n_latent to 50–100. The default 30 may be insufficient for 20-sample datasets spanning 4 disease timepoints.

Test 4 · Marker preservation · CRITICAL

Known retinal marker gene DE

For each cell type × marker gene pair: test whether the marker remains significantly differentially expressed in its expected cell type vs. all others post-integration (Mann-Whitney U, adj. p < 0.01, log₂FC > 0.5).

≥ 70% of marker-cell_type pairs preserved

Remediation: Add major cell types to categorical_covariate_keys. scVI may otherwise absorb cell-type variance into the batch correction.

Test 5 · Velocity compatibility · NON-CRITICAL

scVelo readiness

Run scVelo stochastic velocity on the scVI neighbor graph as a pre-flight check. Mean velocity confidence < 0.7 indicates the spliced/unspliced ratio is too noisy for VelocityKernel in CellRank.

Mean velocity confidence > 0.7 · Mean velocity length > 0.1

Fallback: If this test fails, substitute PseudotimeKernel for VelocityKernel in CellRank. Declare the substitution as a preregistration deviation.

Test 6 · Stability · CRITICAL

Seed stability (5 independent runs)

Train scVI five times with different random seeds. Cluster each latent representation with k-means (k = 15). Compute pairwise adjusted Rand index across all seed pairs. Low ARI indicates the model is not learning stable structure.

Mean pairwise ARI > 0.8

Note: This is the slowest test (~5× training time). Use --skip_t6_stability during development and enable for the final registered run.

Test 7 · Biological validation · CRITICAL

Three biological sub-checks

Sub-check 7a: AUROC > 0.7 for RGC and Müller glia markers in annotated clusters.
Sub-check 7b: Reactive gliosis markers (Gfap, Vim, Lcn2) increase baseline → late.
Sub-check 7c: Trem2 enrichment near ONH in spatially-mapped cells (Mann-Whitney p < 0.05).

≥ 60% of sub-checks pass

Stage 4 — Velocity QC

Checkpoint 4 · NON-CRITICAL (fallback available)

Dynamical velocity model fit quality

Reads velocity_qc.json produced by scVelo. Mean velocity confidence and length are computed after dynamical model fitting (more stringent than the pre-flight stochastic check in scVI Test 5).

Mean velocity confidence > 0.7 · Mean velocity length > 0.1

Stage 5 — CellRank

Checkpoint 5a · CRITICAL

Kernel weights locked before fate extraction

Verifies that kernel_weights.json exists and contains a numeric locked_velocity_weight. The validation orchestrator also checks that this file's modification timestamp precedes fate_adata.h5ad.

File exists with numeric locked_velocity_weight

Checkpoint 5b · Per terminal state · CRITICAL

Terminal state recovery AUROC

Cells labeled as a terminal state by marker criteria should have high fate probability for that state. AUROC quantifies how well fate probabilities discriminate labeled terminal cells from all others.

AUROC > 0.75 per terminal state

Expected terminal states

State	Up-regulated markers	Down-regulated markers
`RGC_apoptotic`	Casp3, Casp9, Bax, Atf3, Sox11	Rbpms, Thy1, Sncg, Pou4f1
`Muller_reactive`	Gfap, Vim, Lcn2, Osmr, Stat3	Rlbp1, Glul, Clul1
`Microglia_DAM`	Trem2, Apoe, Lpl, Cst7, Cd68	P2ry12, Cx3cr1, Tmem119, Sall1
`Astrocyte_reactive_ONH`	Gfap, Serpina3n, C3, Lcn2	Aldh1l1, Glul, S100b

Stage 6 — Spatial Statistics

Checkpoint 6a · NON-CRITICAL (outcome reporting)

Primary endpoint significance

Mann-Whitney U result for RGC apoptotic fate Moran's I: mid vs baseline. This checkpoint reports the primary endpoint result — it does not block the pipeline regardless of outcome. Significance is reported as-is.

Reported; p_adj < 0.05 is preregistered significance level

Checkpoint 6b · CRITICAL

Spatial permutation validity

Verifies that permutation-based Moran's I p-values are non-degenerate (not all identical, which would indicate a permutation bug). Checks that ≥ 50% of computed p-values are unique.

≥ 50% unique p-values across all Moran's I tests

Stage 7 — Targets

Checkpoint 7 · NON-CRITICAL

Therapeutic target table completeness

Verifies that therapeutic_targets.csv was produced with the required columns. Non-critical because a failed OpenTargets query (e.g., API downtime) produces partial but still usable results.

File present · gene, composite_score, n_lineages, tier1_tractability columns present

Validation Report Output

Running validate_pipeline.py produces two files:

pipeline_validation_report.json — machine-readable, suitable for automated CI
pipeline_validation_report.txt — human-readable summary

════════════════════════════════════════════════════════════
PIPELINE VALIDATION SUMMARY
════════════════════════════════════════════════════════════
  Passed: 11/11
  Critical failures: 0
  Status: ✅ PROCEED
════════════════════════════════════════════════════════════

Tip

CI integration. The validation orchestrator exits with code 0 (proceed) or 1 (blocked). This means you can add it as a Nextflow step with errorStrategy 'terminate' and treat a clean run as a green CI signal before committing results to the manuscript analysis directory.