Disease spread (infected count against time, population 50)

How many of 50 individuals are infected at a given time? The task asks for a PyMC (a Python library for probability models) model of this count, fitted to 80 simulated observations at times between 0 and 2. The counts at 20 more times in the same range are held out for scoring.

The Qwen3.6-35B-A3B language model wrote four programs in separate calls and four in a two-round loop, where critiques of the first two proposals went into the prompts for proposals 3 and 4.

Eight of the ten programs scored. A Monte Carlo standard error (MCSE) measures the error from estimating a score with 200 posterior draws. All six Qwen programs that scored beat the floor model (one shared probability p ~ Beta(1, 1), with Binomial(50, p) at every time) on held-out log score, but were more than two MCSEs below the reference, a program we wrote knowing how the environment generates its data. Their scores ranged from -60.7518 to -59.8279, against -45.6085 for the reference.

PredictionsCopy Anchor Link

At time 0.17 the reference program predicts between 5 and 13 infected, and the held-out count there is 4. That count is the only one of the 20 held-out counts outside the reference's band, so the reference covers 19. In every panel, a program's band spans the 5th to the 95th percentile of the counts simulated from its fitted model. The floor model's approximately level sampled band contains four of the twenty held-out counts. The two programs without a score have panels with their observations and no band.

Full-screen interactive predictions

Scores and coverageCopy Anchor Link

Higher scores are better when comparing programs on this same held-out task. Disease-spread scores use probabilities of integer counts, while dugong scores use densities for continuous lengths, whose values depend on the length units. Coverage counts how many of the held-out counts fall inside a program's band.

Single attempt 2 scores 14.2194 below the reference, far more than the combined MCSE of the two fits, sqrt(0.0452^2 + 0.2392^2), which treats the fits as independent. Each of these MCSEs comes from the 200 draws of its fit, in ten consecutive batches of 20: within a batch we average each draw's influence on the total held-out log score, that is, how much the draw raises or lowers that total compared with an average draw, in a linear approximation (the delta method). The MCSE is then the standard deviation of the ten batch means divided by the square root of ten.

Loop proposals 3 and 4, the two programs from the second round of the loop, use a negative-binomial likelihood, which allows more than 50 infections in a population of 50. Their bands contain 19 of the 20 held-out counts, and the upper end of each band exceeds 50 at 8 of the 20 held-out times. Their held-out log scores, -60.5719 and -60.7518, are below that of loop proposal 1, the binomial proposal from the first round (-59.9324).

Four-chain fitsCopy Anchor Link

We also fitted the eight programs that scored with four chains of 1,000 draws after 1,000 tuning steps per chain, using the same programs, priors, and data. All six generated programs that scored remain above the floor model and more than two MCSEs below the reference. All eight fits have largest R-hat below 1.01, bulk and tail effective sample sizes of at least 400, and no divergences.

Four-chain fit data (CSV)

Programs