SynthStats demonstrations

SynthStats team

What a demonstration is

From the ages and lengths of 80 simulated dugongs, Qwen3.6-35B-A3B wrote eight programs in PyMC (a Python library for probability models) that predict length from age. Four of the programs came from single calls and four from a two-round loop, and all eight were scored on 20 held-out dugongs.

Dugongs: the reference program

We wrote the reference ourselves, knowing how the simulator generates its data.

import numpy as np
import pymc as pm


def model(data):
    """Dugong length against age: the growth curve and priors of BoxingGym's
    dugongs environment, with a Normal likelihood and a fixed noise scale.
    """
    age = np.asarray(data["age"], dtype=float)
    y = np.asarray(data["length"], dtype=float)
    with pm.Model() as dugongs_model:
        age_data = pm.Data("age_data", age)
        alpha = pm.Normal("alpha", mu=2.0, sigma=0.2)
        beta = pm.Normal("beta", mu=1.5, sigma=0.5)
        lam = pm.Normal("lam", mu=0.4, sigma=0.5)
        mu = alpha - beta * pm.math.abs(lam) ** age_data
        pm.Normal("length", mu=mu, sigma=0.25, observed=y)
    return dugongs_model

Growth slows with age in both dugong fits

Dugong growth: predictions of the reference and of single attempt 1, with visible and held-out observations.

Each score above a panel adds up the log predictive densities of the 20 held-out lengths, so higher is better. Coverage is the number of held-out lengths falling inside the 90% band (15 of 20 for the reference, below the nominal 18), regardless of how much density is put on each of them.

The second task: disease spread

Disease spread: predictions of the reference and of single attempt 1, with visible and held-out infected counts.

Disease spread comes, like dugong growth, from BoxingGym, a benchmark of simulated scientific environments: a program predicts how many of 50 individuals are infected at a given time.

Every scored model-written program beat the floor model, which ignores the input

Held-out predictive scores of all twenty programs on one simulated dataset of 20 held-out points per task; two programs have no score.

Dugong growth: score and coverage

Program	Held-out log score ± MCSE	Coverage
Reference	-4.5586 ± 0.0761	15 of 20
Floor model	-14.0898 ± 0.1042	16 of 20
Single attempt 1	-4.6292 ± 0.0731	16 of 20
Single attempt 2	-3.1969 ± 0.0190	18 of 20
Single attempt 3	-4.6876 ± 0.0539	16 of 20
Single attempt 4	-3.2267 ± 0.0204	18 of 20
Loop proposal 1 (round 1)	-4.9166 ± 0.1399	16 of 20
Loop proposal 2 (round 1)	-4.6634 ± 0.0950	16 of 20
Loop proposal 3 (round 2)	-7.4306 ± 0.1294	16 of 20
Loop proposal 4 (round 2)	-4.5121 ± 0.0879	17 of 20

Single attempts 2 and 4: gains over the reference of 1.3617 ± 0.0784 and 1.3319 ± 0.0787, both more than two MCSEs. Loop proposal 4: gain 0.0465 ± 0.1162, within two MCSEs, on this one simulated dataset (20 held-out points).

Monte Carlo standard error (MCSE): error from 200 posterior draws, using batch means of the delta-method influence over ten batches of 20 draws.

Disease spread: score and coverage

Program	Held-out log score ± MCSE	Coverage
Reference	-45.6085 ± 0.0452	19 of 20
Floor model	-262.1291 ± 0.5574	4 of 20
Single attempt 1	-59.9733 ± 0.2544	17 of 20
Single attempt 2	-59.8279 ± 0.2392	17 of 20
Single attempt 3	-60.0128 ± 0.2010	17 of 20
Single attempt 4	failed at model build	no predictions
Loop proposal 1 (round 1)	-59.9324 ± 0.1301	17 of 20
Loop proposal 2 (round 1)	failed at model build	no predictions
Loop proposal 3 (round 2)	-60.5719 ± 0.0794	19 of 20
Loop proposal 4 (round 2)	-60.7518 ± 0.1383	19 of 20

No model-written program beat the reference on this one simulated dataset (20 held-out points).

Monte Carlo standard error (MCSE): error from 200 posterior draws, using batch means of the delta-method influence over ten batches of 20 draws.

Predictions of all ten dugong programs

Dugong length against age, with predictions from all ten programs.

At the youngest ages, single attempts 2 and 4 start lower than the reference (low enough for their bands to contain the held-out dugong measuring 0.64 at age 0.16, which the reference misses), while the sampled band of the floor model, which ignores age, is approximately level.

Disease-spread observations and program predictions

Infected count against time, with predictions for the eight scored programs; two failed at model build.

Here the reference misses only one held-out count (4 at time 0.17, below its interval from 5 to 13), while the panels for single attempt 4 and loop proposal 2, which failed at model build, show the observations alone without a band.