What checked means, and what it doesn't

Drylab labels one of its results checked: predicting which yeast genes are essential, meaning the cell cannot grow without them. The predictions were scored against measurements of genes the model never saw during training.^[1]

Why not just count the right answers

Most genes are not essential. A model that answered 'not essential' every time would be right most of the time, and useless. So the score we use stays honest when one answer is much rarer than the other, and a model that guesses scores nothing.

Before calling it checked, Drylab also requires the model to pass gated checks on the same held-out genes, including beating simple baselines: predicting the average, predicting no change, and copying the nearest known perturbation. The pass marks are fixed before anything is scored.^[2]

Held out means held out

The genes it was scored on were set aside before training and never used to train or tune the model. That separation is the difference between checked and theory. A score on data the model has already seen is a measure of memory, not of prediction.

What it does not cover

• →Checked applies to this test: yeast gene essentiality, at this tier.
• →Drylab's drug screens are labeled theory, on every result, until they are checked against real data.
• →A good score on essentiality says nothing, on its own, about growth rates, double deletions or other organisms.

1. Notes
2. [1]Drylab, how we check: checked means matched held-out measurements. Source ↗
3. [2]Drylab docs, How we check: gated checks against baselines on held-out data. Source ↗