The same recipes, scored on texts they never saw.

The pipeline is the notebooks' own, in a different order. Everything below was produced by scripts/evaluate_heldout.py with the 2023 library versions (scikit-learn 1.2.2, imbalanced-learn 0.10.1).

On domain 1, logistic regression reaches macro-F1 0.697 (0.685 to 0.709) and AUC 0.960 (0.954 to 0.965). Compared like for like with 2023, the change sits in the copied class: the leaky split had it catching 0.998 of the machine texts, and on texts it never saw it catches 0.789 (0.757 to 0.817) (Wilson 95%), while human recall stays about the same (0.933 then, 0.945 now; see the 2023 numbers). The printed 2023 F1 of 0.9643 is the human-class F1 on a 50:50 validation set, so it is not comparable with macro-F1 on the real 97:3 mix.

A logistic regression on the text length alone reaches AUC 0.932 (0.923 to 0.941): machine texts in domain 1 are about half as long as human ones. The bag-of-words detector adds +0.027 (+0.019 to +0.036) AUC over length on the same texts. Domain 2 has only 100 test texts, 20 of them human, and the intervals show it. Under each detector is its mean ± standard deviation over ten re-drawn splits. The main domain-2 split ranks 2 of 10 on logistic regression's macro-F1 (1 is the best), so its headline numbers sit on the favourable side.

The recipe gives LogisticRegressionCV 50 lbfgs iterations. In 36 of the 66 fits on this page that use that budget, every cross-validation fit stopped at the limit, and in 38 of them the chosen C was the largest value in the grid. For the domain-1 recipe that is all 50 cross-validation fits and the final fit (51 convergence warnings), with C = 10,000. The 2023 notebooks silenced the warnings; the script now records them for every fit. The inner folds also hold copies of the same text on both sides, because the copies are made before cross-validation splits the training part, so inner-fold accuracy keeps rising with C. SGD on domain 1 also ran all 1,000 of its epochs without meeting its stopping rule.

The table re-fits the recipe with each problem removed in turn, on the same training and test texts. The headline numbers above stay the recipe as run, because that is what the team built. This section shows how much they depend on the fitting.

Both detectors scored the same texts, so the comparison uses only the texts where they disagree (McNemar's test) and resamples texts in pairs for the differences. On domain 1, logistic regression is ahead on every measure, and it was ahead on macro-F1 in 10 of 10 splits. On domain 2 the 15 disagreements give no evidence either way (p = 0.30).

A detector is calibrated if, among texts it gives P(human) = 0.8, about 80% are human. Points on the diagonal are calibrated, the bars are Wilson 95% intervals and a point's area shows how many texts fall in its bin. The hinge-loss SGD model has no probabilities of its own, so it gets Platt scaling, a logistic regression on its cross-validated scores.

Each row is a detector, each column a test part. The shaded cells are in-domain. Trained on domain 1 and tested on all 500 domain-2 texts, logistic regression reaches AUC 0.553 (0.483 to 0.621), not distinguishable from chance (the interval includes 0.5). Pooling both domains into one model (correctly this time, see the audit) is worse than a dedicated model on domain 1 on every measure (macro-F1 +0.040 (+0.032 to +0.048), own minus pooled). On domain 2 it is worse in accuracy (+0.140 (+0.050 to +0.230), p = 0.004), while the macro-F1 (+0.135 (−0.014 to +0.273)) and AUC (+0.113 (−0.098 to +0.316)) differences are not distinguishable from zero.

The machine texts came from 5 generators in domain 1 and 4 in domain 2 (the course labelled them only by number). To mimic a new model appearing after training, logistic regression was re-fitted five times, each time leaving one domain-1 generator out of training, and scored on that generator's test texts.

Leaving a generator out does not hide its prompts. The bag of words includes the prompt, and between 118 and 140 of each generator's 140 test texts answer a prompt that the other generators' training texts also answer. The second view removes every training text (human or machine) that answers one of those prompts, and compares a model that keeps the generator with one that leaves it out.

Five ways to handle the imbalance, all applied to the training part only and all scored on the same test texts with logistic regression. On domain 1 the model with no re-balancing at all has the best macro-F1, 0.809 (0.791 to 0.825), the best AUC and near-perfect calibration. The 2023 recipe minus that model is −0.111 (−0.128 to −0.095) in macro-F1 on the same texts.

As fitted in 2023 that comparison is partly between regularisation regimes: cross-validation chose C = 10,000 for the recipe and 0.046 without re-balancing. The converged tabs re-fit every strategy with the solver run to convergence and C chosen on folds grouped by source text. No re-balancing is still best on all three measures in domain 1, and the recipe's gap becomes −0.163 (−0.181 to −0.145). In domain 2 the strategies are indistinguishable as fitted in 2023: the recipe minus no re-balancing is −0.025 (−0.084 to 0.000). With converged fits the recipe falls behind, and the difference is −0.168 (−0.277 to −0.057). The recipe does catch more machine texts, at the price of flagging far more human ones. The decision record discusses that trade-off.

A bootstrap interval treats the training data as fixed. Re-drawing the split (seeds 90051 to 90060) and re-fitting shows how much the split itself moves the scores. Each seed is used twice: once for a random split stratified by class and generator, and once for a split that keeps every prompt on one side. Each dot is one split, the blue bar is the mean, the ring marks the main split (seed 90051), and the numbers are mean ± standard deviation and the range.

The report's validation table as printed, with intervals computed from the re-derived confusion matrices: Wilson intervals for accuracy and bootstrap intervals (10,000 resamples, seed 90051) for F1. The domain-1 intervals are narrow because there were 10,000 validation rows. They describe sampling noise only. The leak is a bias, and no interval can reveal a bias.

The per-class recalls below show where the bias sat. In each domain only the minority class was copied, so only its 2023 recall is inflated. The class that was never copied scores about the same on clean texts.

The aggregates behind this page, exactly as the script wrote them. They contain no text, token sequence or per-item prediction. To reproduce them, run uv run scripts/evaluate_heldout.py from the repository root (about 40 minutes on a 10-core laptop, most of it in the converged sensitivity fits).