Dugong growth (length against age)

We prompted the Qwen3.6-35B-A3B language model for programs in PyMC (a Python library for probability models), and it wrote eight, each modelling a dugong's length against its age: four single attempts and four proposals from a two-round loop in which a critic gives feedback on fitted predictions.

We fitted each program to 80 dugongs from one simulated dataset, and scored it on the lengths of 20 held-out dugongs from the same dataset, all aged within the range of the 80. The reference is a program we wrote knowing how the environment generates its data.

PredictionsCopy Anchor Link

One held-out dugong, aged 1.21, measured 2.44. At that age the reference program's band (the 5th to the 95th percentile of the lengths simulated from its fitted model) runs from 1.47 to 2.31, so this dugong's length lies above the band. Four other held-out lengths lie outside the reference's band, so the reference covers 15 of the 20. The floor model uses the same predictive distribution at every age. Its approximately level sampled band covers 16 of the 20 held-out lengths, one more than the reference, yet its log score is the lowest of the ten.

Full-screen interactive predictions

Scores and coverageCopy Anchor Link

All ten programs scored. Of the eight written by the Qwen3.6-35B-A3B language model, all scored above the floor model and two (single attempts 2 and 4) above the reference by more than two Monte Carlo standard errors (MCSEs, the errors from estimating scores with 200 posterior draws) on the 20 held-out observations of this one simulated dataset. The gains of single attempts 2 and 4 over the reference, 1.3617 and 1.3319 in log score, come from the youngest held-out dugong, aged 0.16 and measuring 0.64. That length falls below the reference's band and inside the bands of the two programs, which score 1.5198 and 1.5036 above the reference there (all ten programs at that age). Across the other 19 held-out dugongs, the reference scores 0.1581 and 0.1718 higher in total. Loop proposal 4 scored above the reference but within two MCSEs. Coverage counts the held-out lengths inside a program's band.

We split the 200 draws of a fit into ten consecutive batches of 20, and associate with each batch the mean influence of its draws on the total held-out log score, a draw's influence being how much it raises or lowers that total compared with an average draw, in a linear approximation (the delta method). Dividing the standard deviation of these ten batch means by the square root of ten gives the MCSE. Two programs' MCSEs combine as the square root of the sum of their squares, which treats the fits as independent. Loop proposal 4 gains 0.0465 over the reference, with a combined MCSE of 0.1162, so it stays within two MCSEs.

Four-chain fitsCopy Anchor Link

We also fitted the same programs, priors, and data with four chains of 1,000 draws after 1,000 tuning steps per chain. For these fits, the MCSE uses batches of 100 consecutive draws within each chain. We sample the reference's posterior over |lam|, whose prior density for r > 0 is p(r) + p(-r), with p the Normal(0.4, 0.5) density of lam, which leaves its predictive distribution unchanged. All eight language-model programs still score above the floor model, and loop proposal 2 joins single attempts 2 and 4 in scoring above the reference by more than two MCSEs, while loop proposal 4 stays within two MCSEs. Nine of the ten fits have their largest R-hat below 1.01, both bulk and tail effective sample sizes (ESS) of at least 400, and zero divergences. The exception is loop proposal 4, which has five divergences.

Four-chain fit data (CSV)

Programs