Supporting results
Read the report · Research results · PDF report · HTML report
Corpora
The brain-coral task, on maximum diameter across reef sites, is one of 220 training packages for PyMC (a Python library for probabilistic programming). We have 323 packages in all, with the other 103 kept for evaluation. We split them by whole scenario groups, so that every group is on one side of the split.
For these tasks, other language models wrote example programs (teacher programs). Of the 8,468 in the bank of 30 Sep, GPT-6 Astra wrote 7,280 and GPT-6.1 Sol wrote 1,188.
| Collection | Size | Composition |
|---|---|---|
| PyMC task packages | 323 packages | 220 for training (81 scenario groups) and 103 for evaluation (39 groups), with no group on both sides; by family, 60 forward tasks (data simulated from a generating process that the prompt describes), 134 survey counts and 129 survey detections |
| PyMC reinforcement learning (RL) environments, 30 Sep to 1 Oct | 375 tasks | 358 tasks in four new families, plus 17 earlier tasks (8 for training, 9 held out for evaluation) |
| The four new RL families | 358 tasks | Two forward-task cohorts, one survey-count family and one survey-detection family, the two survey families being synthetic and calibrated on a real species survey; 342 of the 358 also exist in LazyPPL, a probabilistic programming library in Haskell |
| Packaged RL tasks, 20 Oct | 4 tasks | One from each of four scenario groups, each with a fixed reference fit; none among the 323 packages; the 20 Oct run trains on these four |
| PyMC teacher candidate programs | 35,475 candidates on 140 tasks | Example programs that several model families generated and refined |
| Teacher ensemble round, 4 Oct | 1,346 distinct programs | From five teachers, asked for two programs per package on 135 training packages |
| PyMC supervised fine-tuning (SFT) admission lists | 1,346 records on 1 Oct; 1,187 pairs on 4 Oct | Pairs of prompt and program on which SFT trains a model to write the program from its prompt; a different set from the 1,346 programs of the 4 Oct ensemble round |
| LazyPPL admitted pairs | 3,671 pairs | On the 155 eligible tasks; a separate ranked pool holds 2,207 programs on 213 tasks, 1,600 of these programs on the eligible tasks, and its 213 tasks are not the tasks of the PyMC supervised fine-tuning pairs |
| SynthStats/ppl-synthesis-sft-bootstrap (Hugging Face asks you to accept its terms for access), 1 Oct | 899 SFT rows and 724 held-out rows | SFT rows: PyMC 473 (285/94/94 train/validation/test) and LazyPPL 426 (256/85/85); held-out rows: PyMC 373 and LazyPPL 351; the prompt checker accepted all 595 data-checkable claims across 404 rows |
| Earlier sets, Mar to Aug | 162 rows; 189 trajectories | A PyMC SFT set of 162 rows (146/8/8), and a mixed set of 189 agent trajectories, the recorded steps of an agent on a task (155 PyMC, 34 LazyPPL), stored in ATIF, an interchange format for agent trajectories |
| Unstated assumptions, from an audit of the 220 training reference programs | 1,243 confirmed | Assumptions that the task prompts do not state: 1,196 generic modelling choices, 3 domain facts and 44 other |
We fitted the PyMC and LazyPPL versions of the forward tasks to the same visible rows and compared their posterior means. The comparison covered 371 parameters in 69 task specifications, and all 261 parameters in the 46 specifications of the second forward cohort. Across these two cohorts, the means differed by at most 0.70 PyMC posterior standard deviations (SD).
| Forward tasks compared | Parameters | Difference between the PyMC and LazyPPL posterior means, in PyMC posterior SD |
|---|---|---|
| 69 task specifications | 371 | Median 0.056, 95th percentile 0.24, maximum 0.56, none above 1 |
| Second forward cohort, all 46 specifications | All 261 | All less than 1 (median 0.074, maximum 0.70) |
Data used in training and data held out
E8 and E11 below compare two trained versions of GLM-5.3-Flash with the untrained model. Both come from GLM-5.3-Flash's run of 21 October, in which it made four attempts (episodes) at each of the 220 training packages and none at the 103 evaluation packages that both comparisons use. So both comparisons are made on held-out packages.
| Date | Model and training | Data used |
|---|---|---|
| 26 October | Qwen3.5-0.8B, trained from the supervised warm start in three arms: subtrajectory balance (SubTB), group relative policy optimization (GRPO), and SubTB at a learning rate of 1e-6, all with low-rank adapters of rank 32 | 220 training packages from 81 groups, 32 prompts with 4 episodes each per update, over 8 updates in each arm |
| 21 October | GLM-5.3-Flash as a tool-using agent (it requests observations through tools, then submits a program), trained on the fit reward at weight 1.0 | 14 updates (changes to the model's weights), each on up to 16 packages × 4 episodes drawn in order from the 220 training packages, covering all 220 packages and 880 episodes |
| 21 October | Checkpoint 000003 of that run (its weights as saved after update 3), evaluated in E8 | 48 packages and 192 episodes, over 3 updates |
| 21 October | Final checkpoint of that run, evaluated in E11 | All 14 updates |
| 21 October | GLM-5.3-Flash as a tool-using agent, trained with fit at weight 1.0 and a judge score at weight 0.25 | 220 training packages, 4 episodes per package, over 14 updates |
| 21 October | Qwen3.5-0.8B as a tool-using agent on the open stack (Harbor, mini-SWE-agent and SkyRL), in its supervised warm start, which trains it to imitate teacher episodes | 193 verified teacher episodes (1,930 tool actions) on training packages from 63 of the 81 training groups, over 2 epochs (passes through these episodes) and 98 updates |
| 20 October | GLM-5.3-Flash as a tool-using agent | The 4 packaged tasks (see Corpora): 32 episodes over 2 optimizer steps |
| 19 October | GLM-5.3-Flash as a tool-using agent, trained by group relative policy optimization (GRPO), which compares each episode's reward with those of the other episodes on the same task | 3 updates of 8 episodes on a single training package (a forward task) |
| 18 October | Qwen3.5-4B in a control run, trained with the trajectory balance (TB) objective on all of its weights | One update, from a schedule of 24 prompt rows on 23 of the 220 training packages |
| 16 October | GLM-5.3-Flash | 3 updates of 4 prompts × 8 answers, drawn from training-side packages |
| 5 October | Qwen3.5-0.8B, a supervised fine-tuning pilot | 92 teacher programs on 32 LazyPPL tasks, with one program held out for validation. On that program, the negative log-likelihood (NLL) per token (lower is better) fell from 1.3391 at epoch 0 (before training) to 0.8352 at the selected epoch 4 |
| May | Qwen supervised warm start on Tinker (a hosted training service) | 1 training example, 1 optimizer update |
| Kept out of training | Rule |
|---|---|
| 879 programs that teachers wrote for evaluation packages, and 393 for packages held back from the split | Excluded from training and tuning, and from teacher examples and reward data |
| The 103 evaluation packages and the 9 earlier held-out tasks | Reserved for evaluation: E8 and E11 measure the 103 packages, and other measurements on the 323 packages use the training side |
Training runs
The coral episode under Predicting coral size in the report comes from one of these runs, the run of 21 October, which trained GLM-5.3-Flash on the 220 training packages. Of the utility of 2.2276 that its program earned, 1.5 came from admission, which the verifier grants when it accepts and scores a program, and 0.7276 came from predictive quality.
| Code | Run | Training | Population |
|---|---|---|---|
| T2 | 16 Oct, GLM-5.3-Flash | 3 updates, one seed (a single run) | Development (training-side packages used to check a run): ten fixed packages, one answer per package; one of the ten comes from a group used in the updates |
| T3 | 18 Oct, a separate draw from the T2 checkpoints | The T2 run: C0 is its initial checkpoint, U3 its checkpoint after update 3 | Development: the 10 fixed packages and 30 packages from groups the updates did not use, 8 answers per package; one of the 10 fixed packages comes from a group used in the updates |
| T4 | 20 Oct, GLM-5.3-Flash as a tool-using agent | 2 optimizer steps (updates); the agent calls the tools list_candidates, query, inspect_data, run_program and submit, with a budget of 8 queries | In-sample (measured on the tasks that the run trained on): 4 tasks, 16 episodes per policy (the model before or after its 2 steps); the initial policy's episodes are its training episodes at the initial weights |
| T5 | 19 Oct, GLM-5.3-Flash as a tool-using agent | Group relative policy optimization (GRPO) with the reference settings of the Tinker cookbook, the published training recipes of the Tinker service: 3 updates of 8 episodes on one task | In-sample: that one task; the final scorer refitted each program on all 64 training rows |
| T7 | 30 Sep, Qwen3.5-4B | Offline comparison of the training objectives reference-weighted trajectory balance (TB, VarGrad) and Dr. GRPO, a variant of GRPO: 8 updates per objective on 128 cached programs, one seed | Held-out evaluation (tasks kept out of training) and in-sample (the training tasks) |
| T8 | 21 Oct, GLM-5.3-Flash as a tool-using agent | The 220-package run, followed here on one training package for maximum diameter of brain coral across reef sites, under a query budget | In-sample: that one training package |
| Code | What we measured | Result |
|---|---|---|
| T8 | Utility of the coral episode | 2.2276: 1.5 for admission and 0.7276 for predictive quality |
Evaluations and baselines
The baselines cover untrained models, the teachers that wrote our example programs, and agents on BoxingGym (a benchmark of environments for experimental design and model discovery). The evaluations compare checkpoints of our training runs with the same model before training, and inference in LazyPPL with inference in PyMC.
| Baseline | Date and model | What was measured | Result | Population |
|---|---|---|---|---|
| E2 | Other models, untrained, on the same 17 tasks | Whether a program's score depends on the verifier version: 4 samples for each task, under the original verifier and under a later version, which also accepts discrete data declared with pm.Data through a cast | Among the nine models scored under both versions, all 107 programs scored by both get the same score from each | untrained |
| E3 | 4 Oct: Qwen3.5-4B, untrained, and the teachers | LazyPPL programs on the 155 admitted tasks, and whether each is accepted by the LazyPPL scorer after runtime checks and numerical probes of likelihood normalization | Qwen3.5-4B, in a separate draw of 620 samples at up to 8,192 tokens each: 2 get through the typechecker Teacher programs accepted by the scorer: GPT-6 Astra: 604 of 620 GPT-6.1 Sol: 605 of 620 Kimi K3: 510 of 620, with at least one accepted program on 138 of the 155 tasks Claude Sonnet 4.6: 249 of 310 Claude Haiku 4.5: 163 of 310 | untrained model and teachers, training-split tasks |
| E4 | 13 Feb 2026: seven API models on 10 BoxingGym environments, in a snapshot of the multi-model BoxingGym results | Mean z-score against the BoxingGym baseline (lower is better), over the 1,946 runs left of 5,710 rows once we kept query budgets (the number of experiments an agent may run) of at least 10 and mean z-scores between -100 and 100 inclusive. BoxingGym defines one baseline predictor for each environment, which its code calls the naive predictor, and records the mean and the standard deviation of that predictor's errors. For dugongs, that predictor draws 10,000 simulated measurements, averages them, and predicts that average at every age. A z-score is a model's mean error minus the baseline's mean error, divided by the baseline's standard deviation, so 0 matches the baseline, +1 is a mean error one baseline standard deviation larger, and -1 is one baseline standard deviation smaller. Entry V1 below uses a different baseline, a refit of the reference program. At a query budget of 10, we group runs with matching task and replication settings into cells. The matched comparison uses only the 20 cells that contain runs from all seven models. In the matched comparison, each of the ten environments has equal weight and the 95% intervals come from resampling whole environments (200,000 draws). In the pooled comparison, each row has equal weight and the intervals come from the snapshot's 10,000-draw bootstrap over rows. | DeepSeek-V3.2: pooled +0.296 [+0.122, +0.532]; matched +0.245 [-0.194, +0.760]; 369 pooled rows, 295 matched rows GPT-4o: pooled +0.255 [+0.136, +0.387]; matched +0.258 [-0.126, +0.648]; 224 pooled rows, 157 matched rows MiniMax-M2.1: pooled +0.185 [+0.071, +0.315]; matched +0.262 [-0.128, +0.694]; 311 pooled rows, 273 matched rows GPT-5.1-Codex-Mini: pooled +0.586 [+0.310, +0.966]; matched +0.279 [-0.158, +0.771]; 238 pooled rows, 174 matched rows Kimi-K2: pooled +0.262 [+0.130, +0.408]; matched +0.352 [-0.105, +0.863]; 302 pooled rows, 271 matched rows GLM-4.7: pooled +0.546 [+0.257, +0.904]; matched +0.405 [-0.127, +1.061]; 314 pooled rows, 260 matched rows Qwen3-32B: pooled +0.275 [+0.115, +0.464]; matched +0.506 [+0.040, +0.993]; 188 pooled rows, 165 matched rows +0.122 on the 700 runs in optimal experimental design mode, and +0.466 on the 1,246 in discovery mode +0.538 on the 115 runs with a probabilistic programming language, and +0.330 on the 1,831 without one (two-sided Welch t test, p = 0.134) | no training (API models) |
| E5 | 30 Sep and 4 Oct: five API models querying BoxingGym environments over several turns | Programs with a finite held-out score, each model writing a PyMC program and a LazyPPL program for the observations available after 0, 1, 3, 5, 7 and 10 successful queries | Of 225 recorded episodes, from a 420-episode grid of model, environment, goal and seed, 191 contain both programs at all six query counts PyMC: a finite score in 761 of 1,165 scoring jobs LazyPPL: a finite score in 235 of 1,167 scoring jobs, the program failing to compile in 883 of them | held-out evaluation (20 held-out points per episode) |
| E9 | 21 Oct: BoxingGym's own plain-scientist agent, run with Qwen3-32B on Bedrock, information gain and probabilistic-programming assistance off | Mean MSE (mean squared error, as BoxingGym computes it; lower is better) on the dugongs environment, seeds 1 to 5, with ten successful experiments per seed and ten evaluation questions at each budget | 0.149 with no experiments, 0.631 after one, 0.650 after three, 0.537 after five, 0.552 after seven and 0.640 after ten; every seed ended worse than it started | no training (API model) |
| E10 | 21 Oct: BoxLM's own model search (run_box_loop), run with Qwen3-32B and PyMC 5.13.1 | Programs that received a PSIS-LOO score (Pareto-smoothed importance-sampling leave-one-out, as BoxLM computes it) on the ten-observation dugongs data set, over three rounds in which the search proposed eight programs each, with a critic between rounds | 6, 8 and 8 programs in the three rounds, 22 of the 24 in all; the best score on this data set was 11.868 | no training (API model, one data set) |
| Evaluation | Date and comparison | Evaluated on | Result | Population |
|---|---|---|---|---|
| E6 | 4 to 5 Oct: LazyPPL against PyMC with NUTS (the No-U-Turn sampler) | The models of 18 posteriors from posteriordb, a public collection of reference posteriors, fitted in PyMC with NUTS and in LazyPPL, as translations of the PyMC programs, with a fixed-mixture Metropolis-Hastings kernel, on the visible rows and again on the full rows | After convergence and reference checks pass, agreement requires every parameter's Monte Carlo intervals (error multiplier 4) to lie within 0.2 PyMC SD for mean differences and 0.8 to 1.25 for SD ratios in both row scopes; a wholly outside interval gives disagreement, and other cases are inconclusive. Comparing marginal means and standard deviations, 7 of the 18 posteriors agree in both scopes (visible rows and full rows) on every one compared, 11 are inconclusive and none disagrees Counting the scopes separately, 14 of the 36 comparisons agree, 22 are inconclusive and 0 disagree | no model trained |
| Entry | Date and setting | Design | Population |
|---|---|---|---|
| E1 | 29 Sep: Qwen3.5-0.8B, Qwen3.5-2B and Qwen3.5-4B, untrained | PyMC programs that got a score, of 68 per model (4 samples for each of 17 tasks, at temperature 1.0 and at most 2,048 tokens each), scored by the verifier, which checks a program before fitting it to the task's data | untrained |
| E8 | 21 Oct: checkpoint 000003 of that day's run on the 220 training packages (GLM-5.3-Flash as a tool-using agent, with low-rank adapters of rank 32, trained with the Tinker cookbook recipe, which the Tinker training service publishes), against the untrained model | All 103 evaluation packages, with 2 episodes per package and model, 206 per model; the comparison is paired by package, and the 95% interval comes from 10,000 resamples of the 39 whole groups | held-out evaluation (103 packages in 39 groups, none used in training) |
| E11 | 21 Oct: the final checkpoint of the same run, after 14 updates over all 220 training packages, against the untrained model, with the same agent, recipe and verifier as E8 | All 103 evaluation packages, with 8 episodes per package and model, 824 per model; the comparison is paired by package, and the 95% interval comes from 10,000 resamples of the 39 whole groups | held-out evaluation (103 packages in 39 groups, none used in training) |
Verifier and reward results
We also tested the verifier and the rewards on which we train policies. On 20 Oct, for instance, we gave it an honest control program and four forged programs that write its scoring state or its result record themselves. Against one verifier version, the control scored and all four forgeries were refused at model build.
| Entry | What was tested | Result | Population |
|---|---|---|---|
| V1 | 10 Jul. Held-out ELPD (expected log predictive density: how well a program fitted to the visible rows predicts the held-out rows) as a reward for reference programs. Each task's reference program against a baseline that refits it with its ordinary predictors held at constants computed from the visible rows (a mean or a most frequent value, chosen by how many distinct values the column has) and its structural inputs, such as group membership, offsets and censoring indicators, left at their real values | The reference programs scored below the baseline in 25 of the 44 evaluations. Of the 26 evaluations on out-of-distribution splits, whose held-out rows differ systematically from the visible ones, 21 were below the baseline: ordered tail, 14 below and 3 not; group hold-out (whole groups held out), 6 below and 2 not; spatial region (one region held out), 1 below and 0 not. Of the 18 evaluations on random rows, 4 were below the baseline and 14 were not | held-out ELPD of reference programs |
| V2 | 1 Oct. Stability of the reward across seeds: programs on 17 tasks, each scored with 4 random seeds | 896 scoring jobs, of which 802 scored and 94 were refused by the verifier's checks. 201 of the 818 pairs of programs (24.6%) change order from one seed to another, the program ranked higher under one seed ranking lower under another; among the 669 pairs not tied within one standard error (their scores differ by more than one standard error), 52 (7.8%) change order | reference and program pairs |
| V3 | The reference ELPDs (the held-out ELPD of each task's reference program), as our protocol computes them, against a long reference run of NUTS (the No-U-Turn sampler), on five programs | Equal-weight mean of the five program percentage differences: -0.5% of the gap between reference and baseline ELPD (nominal 95% Student t interval with 4 degrees of freedom: -1.3% to +0.3%) | reference programs |
| V4 | 20 Oct. Four constructed forgeries and one honest control, run against one verifier version; the forgeries write their own scoring state or result record | All 4 forgeries were refused at model build, the step that turns submitted code into a PyMC model; the control scored | constructed programs |
| V4 | 12 Jul. LazyPPL scoring of programs that emit arbitrary finite log-likelihoods, against an honest program | The programs that emit arbitrary log-likelihoods outscored the honest one, with training log reward 233,827.998 against 0.912 | constructed programs |
| V4 | Saved answers, each scored on two machines | All 4 LazyPPL pairs matched exactly; all 4 PyMC pairs were within 7.99 nats (natural-log units), including one exact match | saved answers |
| V5 | 18 Oct. An LLM judge (a language model that grades programs against a rubric, a written list of criteria), with one judge model, on a contrast set of one good program and flawed programs | The good program received a rubric score of 1.0 (3 of 3), and the flawed programs received scores between 0.5 and 0.625 | constructed programs |
| V6 | Our UAI 2026 paper on likelihood hacking, Likelihood hacking in probabilistic program synthesis, which appears in the UAI 2026 proceedings: a training run of 29 steps, in which Qwen3-4B-Instruct-2507 is trained with GRPO (a reinforcement-learning method) and LoRA rank 32 (a low-rank update of its weights) to write PyMC programs for three-dimensional Bernoulli data (d=3), each program rewarded by the log marginal likelihood of the training split (the log of the probability that the program gives to its training data) | 23,053 completions (programs that the policy generated): 17,522 valid, 804 with pm.Potential (the PyMC construct that adds an arbitrary term to the log density) | in-sample |
| V6 | The same paper and setting: a training run of 28 steps | 22,257 completions: 17,042 valid, 606 with pm.Potential | in-sample |
| V6 | A heuristic classifier applied to the pooled completions of seven untrained LLMs | Fewer than 1% labelled as likelihood hacking | completions of seven untrained LLMs, pooled |
| V6 | SafePyMC, the paper's graph checker for PyMC, applied to each valid generated record of four runs | Rejected all 116 programs with positive reward; in a separate curated test, rejected all 20 discovered high-reward programs and accepted the honest baseline | 37,339 valid generated records from four runs (two large and two short) |
| V6 | A numerical audit, by direct computation or a lower bound | Normalization violations (probabilities over all possible data that do not add up to one) established for 86; 22 unresolved | 108 programs from the two large runs, 2 of them from a final step outside the run totals |
| V6 | Translation audit for SafeStan, the paper's static checker for Stan (another probabilistic programming language): a language model attempted a SafeStan translation of each model and labelled the change needed 'as-is', 'minor' or 'major' | A language model labelled 721 of the 1,163 models (62%) 'as-is' or 'minor' | 1,163 public Stan models |