Bigger gene sets win,
and it has nothing
to do with biology.
A 200-gene program returns more hits than a 30-gene one regardless of
what either does — the way a raw crime count always ranks big cities as the most
dangerous. Across all 50 Hallmark programs scored against 9,837 CRISPRi
knockdowns in K562, program size alone explains 46.5% of what looks
like discovery, and 56–75% of the variance is explained without reference
to what any program does.
The range is a range because one of our six features is computed
from the same matrix as the outcome, so part of 0.751 is arithmetic. 0.561 is the
figure that survives that objection, and we never quote the top alone.
Use it on your own screen
Two minutes, before the year.
Your enrichment tool already gave you the whole input: set name,
how many genes each set had, how many came back significant. Drop that file
below — it is read in your browser and goes nowhere. This page makes no
network call of any kind, and a build-failing test enforces that, so your
results cannot leave this tab.
Nothing read yet. Three things will appear here, in
order: the verdict on how much of the ranking set size explains, where that
sits against 1,272 published CRISPR screens, and which of the top-ten entries
survive the size correction. Get them from the button above — our own
screen, one click — or from a file of yours.
14 evaluations. Ten came back negative, one returned no
verdict when our own power rule fired against us, and all 14 are reported here.
No gene-level claim is made anywhere, and the build fails if one appears.
The reversal matrix: every knockdown scored against every program. 9,837 CRISPRi knockdowns (rows) scored against all 50 MSigDB Hallmark programs (columns), in K562. Blue means switching that gene off pushed the program down; red means it pushed it up. Programs are ordered left to right by how many knockdowns moved them. The dashed line marks the held-out program.
Three of the ten negative findings
Most of it is not biology
A model that never looks at what a program does predicts most of how
reversible it appears. 11 of 50 programs return nothing at all.
We wrote down before running that if this cleared 60%, it becomes the finding
rather than the failure. It cleared, at 0.751.
The obvious filter is wrong 20 of 50
We built the quality filter anyone would build. 20 programs
fail it and produce hits anyway; only 1 passes it and produces
nothing. The program we held out fails our own filter and still ranks
11th of 50.
Our generalisation test failed
10 programs from a different collection, not scored until the model was
finished. Only 1 of 10 was measurable, so by our own pre-registered
rule the evaluation is underpowered and inconclusive. Balanced accuracy
0.4375 — worse than chance, 0 true positives. We did not refit.
Take it with you
The same check, in your pipeline.
The runner above is this repository’s own math ported into the
page, and a parity test fails the build if the two ever disagree on a number.
For repeated use — every screen, every collection, CI — install the
packaged tool. Same input, same verdict, plus a --json flag and a
replication check the page does not carry.
1 · install, from a clone
git clone https://github.com/alejandro-publius/denali
pip install -e denali/packages/denali-audit
denali audit my_results.csv
denali rerank my_results.csv --top 10
No column renaming: it reads g:Profiler, DAVID,
clusterProfiler, Enrichr/GSEApy, MAGeCK gene_summary, fgsea, GSEA desktop,
drugZ and BAGEL2 output as-is (denali formats). An R²
is not a judgement until you know what normal looks like — the
field’s median is 0.224.
2 · what the CLI adds
denali audit results.csv --json
denali replication results.csv --hits-b screen2
It grades its own correction. Re-audit the re-ranked
list and the score has to drop, or the correction did not work on
your data — and you know that before you publish, not after. On ours:
0.4649 CONFOUNDED → 0.0000 NOT SIZE-DOMINATED. The tool still
refuses to say the survivors are real; it reports what was carried by
size, not what to chase.
Replication has a price too. When two screens agree,
denali replication measures how much of the agreement is set
size — both screens confounded the same way agree for the same wrong
reason.
Or connect it to your agent
The frozen matrix is an MCP server, so any agent can query it directly. No API
key, no account, nothing hosted — it runs on your machine against files in
the repo.
{
"mcpServers": {
"denali": {
"command": "python",
"args": ["-m", "src.mcp_server"]
}
}
}
It also refuses. Ask it about a bare gene symbol and it
declines, citing the -0.019 concordance that makes any single-gene answer
irreproducible. Ask it to rank or nominate and it declines again, citing its own
predictor’s failure. Those refusals are not advisory text the model may ignore
— they are code that returns before the data is read, and eleven tests
assert they still fire. The build-time scope guard stops us publishing a
gene-level claim; this is what stops an agent extracting one.
Paste into Claude Code, Cursor, or any MCP client. Two tools:
reversibility(program) and provenance(). The first
returns the measured rank, the share measurability alone predicts, the residual,
and a generated next experiment — and for a program we never scored it
volunteers the predictor’s own failure before answering. There is deliberately
no backend and no key to manage: this page is a single static file that
makes zero network calls, enforced by five tests. It is served from GitHub Pages
for convenience, and that copy is byte-identical to the file in the repository
— downloading it and opening it offline gets you the same page, because
every figure, font and number is already inlined. A backend would mean this demo
could fail in front of you.
The loop, running
The agent chooses what to look at next, and decides when it has seen enough.
It starts knowing nothing. At each step it picks a program by a
stated policy, reads that program's frozen evidence, updates its estimate of how
much of apparent reversibility is explained by measurability, and emits a next
experiment. It halts on its own when the estimate stops moving. Nothing here is
scripted to a fixed answer — change the policy or the halt rule and it visits
different programs and stops somewhere else.
0 of 50 visited
—
running R²: how much of what it has seen is explained by
measurability alone
No steps taken. The agent has
read nothing yet.
The running R² is computed in the browser from two frozen columns
— observed Rp and the measurability model's prediction — over
exactly the programs the agent has chosen to read. It is not fitted here and no
value is pre-computed for it. It is not the pre-registered statistic: that
one is an OLS fit over all 50 programs, penalised for six parameters,
and it is 0.5606–0.7511. This is a plain goodness-of-fit on a subset, which
over all 50 rows comes to 0.7816. The agent reports both when it
halts, and names the gap, because a number from an early stop is worth less than
a number from a full read and it should say so itself.
The filter anyone would build is wrong 20 times out of 50. Each point is one program. Left: programs that fail our measurability filter. Right: programs that pass. Height is how many knockdowns moved the program, on a log scale. The shaded region is the failure mode: 20 of 50 programs fail the filter and produce hits anyway. Only one program passes the filter and returns nothing. Circled in red is the held-out program — it fails the filter on an expression ratio of 0.92, and still ranks 11th of 50 with 773 hits. We built a filter that would have discarded our own best result, and we only found that by scoring every program rather than the ones the filter approved.
Added after the freeze
“It replicated in a second cell line” is the strongest evidence a hit
list ever gets. We measured what that evidence is worth.
A second, independently screened cell line — RPE1, 2,386
knockdown targets against K562’s 9,837 — scored with the same frozen
code. Two questions: does the size effect hold there, and when the two screens
agree, is that biology?
0.2758
size alone, in RPE1. Pre-registered bar was 0.25, fixed and
hashed before the sweep ran. It clears by 0.026 — thin, and
we say so. p = 0.0001069, 49 of 50 scoreable
+0.663
raw rank agreement between the two screens. This is the number
a replication claim rests on
+0.493
the same agreement after removing set size from both.
26% of the replication was set size
0.6
of the top 10 programs in the second screen, predictable from
set size alone. Observed overlap 0.8; chance 0.2
Six of the top ten programs in an
independent cell line can be predicted using nothing but how many genes are in
each set. Both screens are confounded the same way, so agreeing for the same
wrong reason looks exactly like agreeing for the right one.
The RPE1 arm was pre-registered; the concordance measurement was
not, and is labelled post-freeze wherever it appears. RPE1 covers 24.3% of K562’s
targets and that quarter is disproportionately essential genes — our own
coverage control, which fails at 94.1% versus 11.3%. So this is a
generalisation test, not a replication, and the number above is a measurement on
these two screens rather than a general estimate of anything. Both run on anyone
else’s paired screens: audit_screen.py --hits-b.
Added after the freeze · and it failed twice
Everything above used the cleanest annotation in biology. Most people use the
messiest one.
Hallmark is 50 hand-curated sets spanning 6× in size. Gene
Ontology Biological Process is 7,538 sets spanning 398×, and it is the
most-used gene-set collection there is. We pre-registered a prediction: the size
confound should get worse as the annotation gets looser. 793 sets across
four collections, scored on Modal.
98%
of Hallmark sets can be scored against a genome-scale screen
46%
of GO Biological Process sets can. The median GO-BP set declares
20 genes and has 8 measured
wrong
our prediction, in direction. GO-BP 0.2905 and Reactome 0.1846
are below Hallmark, not above
none
verdict issued. Our own power rule — 150 of 250 sets
scoreable — fired on three of four collections
More than half of the most-used gene-set
collection in biology cannot be evaluated against this screen at all. That is the
annotation meeting the assay, not the biology.
Two failures and we are reporting both. The prediction was wrong in
direction, which we state rather than let an underpowered result quietly bury. And
the power rule fired before the deciding statistic could be applied, so no verdict
is issued and those R² figures carry none — the rule was fixed before the
run for exactly this case and it cost us the headline. What survives is
descriptive, was not the question we asked, and is labelled so wherever it
appears.
One held-out row carries the whole result. REACTOME_SCAVENGING_OF_HEME_FROM_PLASMA drew
the highest prediction of all 10 — Rp 5.26 — on
1 measured gene. It returned 0 hits. The model predicted strongly
because the program looked measurable on the features it could see, and the
program returned nothing because it was not measurable at all. The failure and the
finding are the same fact.
Control
The one positive
Run unchanged on a program it had not been developed against, the pipeline puts
that pathway's master regulator at rank 2 of 11,258 scored perturbations — more
than the 9,837 unique genes, because some are targeted twice. Eleven of
seventeen canonical members land in the extreme 10%, against 1.7 expected by
chance, binomial p = 7.0×10⁻⁸, with the correct sign at both ends of the ranking.
That same program fails our measurability filter on an expression ratio of
0.92 and still returns 773 hits.
This is a control, not a discovery. It says the ranking works. It
does not say the ranking found anything, and we do not claim that it did.
20 genes queried independently. 19 came back with the same paper. We used a literature retrieval tool to attach one citation per gene, then audited it instead of trusting it. Left: 20 genes, queried independently in a blind probe. Right: the papers returned. Red shows a single zebrafish knockout-methodology paper returned for 19 of the 20 genes — including genes with no connection to it or to each other. Across the full 113-gene set, 34 distinct sources cover everything and one review accounts for 50.4% of them. Our top-cited source for ATF3 was a paper about integrating single-cell data across species, which has nothing to do with ATF3. This is a pointer layer, not an evidence chain, and we label it as one. If you build on retrieval, measure your retrieval.
What this does not claim
- Not that measurement is the cause. A post-freeze check, prompted by a
critique rather than our plan, split the six features: measurement-only reaches
0.152, set-construction-only reaches 0.697. The number stands; the
attribution belongs to how gene sets are defined.
- No gene-level result. Guide-pair concordance is -0.019 — two
independent guides against the same gene give uncorrelated scores. Pathway-level
claims only, and no novel gene is named anywhere in this project.
- Not generalisable, on our own evidence. The held-out evaluation was
underpowered and inconclusive, and its binary axis failed outright at 0.4375.
- One cell line, unstressed. Everything is K562. Our first program
returned a null because it was never switched on in those cells — measurable is
not the same as engaged, and our gate tested the wrong one.
- The evidence layer is a pointer layer. 34 distinct sources cover
113 genes, one review holds 50.4% of them, and 19 of 20 blind-probe
genes returned the same unrelated paper. Not an evidence chain, and we do not
describe it as one.
Tool chain
4 of these 9 were installed, authenticated and
run. 3 changed anything in results/frozen/. The gap between
those two numbers is the honest tool count, and every row on the wrong side of it is
still listed. A reviewer can check the whole column with one command —
grep -rn "modal\|esm\|benchflow\|benchling" src/ returns the strings on
this page and no import. The test suite runs that grep, so if any of them ever enters
the pipeline this table fails the build instead of quietly becoming false.
Declined, with the reason
A tool we could have run and chose not to is a different fact from one
that would not install. Collapsing the two is how a tool count stops meaning anything.
We got one of these wrong, twice, and caught it ourselves. We
had Proto recorded as broken — pip install proto-language fails at
import. Proto does not publish that package. We had tested a name collision and
filed the result as Proto's status, which is the same mistake as the
sundial collision we had already caught and warned about in the same
document. The real install succeeds. It is the fourth error in this project found by
us rather than by a reviewer, and it is in LIMITATIONS.md §7 with the
other three.
The result is callable
An MCP server exposes the frozen matrix as two tools, reversibility(program)
and provenance(). It reads results/frozen/ and recomputes nothing.
Ask about one of the 50 scored programs and it returns the measured rank, the
share of that score measurability alone predicts, the residual that could be biology,
and a generated next experiment. Ask about anything else and it takes the third branch:
{
"program": "HALLMARK_A_PROGRAM_WE_NEVER_SCORED",
"status": "UNSCORED",
"note": "Not in the frozen matrix. Supply measurability features to get a prediction, or score it on the existing pipeline.",
"prediction_uncertainty_sd": 0.5766,
"predictor_validation": "FAILED on held-out data: balanced accuracy 0.4375, worse than chance, 0 true positives. The predictor is reported, not endorsed.",
"scope_limit": "Pathway-level only. Guide-pair concordance is -0.019, so gene-level calls are not reproducible and no novel gene is named. Between 56% and 75% of variance in apparent reversibility is explained by how the programs were defined -- chiefly their size -- not by their biology. A post-freeze check split the features: measurement quality alone reaches only adj R2 0.152, set construction alone 0.697."
}
No caller asked whether the predictor works. The tool says so anyway,
in the same response as the prediction, with the number that condemns it —
balanced accuracy 0.4375 against a coin's 0.500 — and the words reported, not
endorsed. That string is not written into the page; it is imported from the
module the server answers with, so if one drifts the build fails.
For scale on how hard the held-out step is: Arc Institute's Virtual
Cell Challenge wrap-up (6 December 2025, 300+ final submissions) reported that
perturbation-prediction models are “not yet consistently outperforming naive
baselines across all metrics”. That is a different and much larger task than
ours — predicting expression responses, not program-level movement — so it is not
a defence of our number. It is the reason we treated a held-out failure as the
expected outcome to design for rather than a result to bury.