Demonstrations

For dugong growth, the prompted language model Qwen3.6-35B-A3B wrote eight programs in PyMC (a Python library for probability models): four single attempts and four proposals from a two-round critic loop. On the 20 held-out observations of this one simulated dataset, two scored above our reference by more than two Monte Carlo standard errors (the errors from estimating scores with 200 posterior draws); we wrote the reference knowing how the environment generates its data. Single attempts 2 and 4 exceeded the reference by 1.4059 (Monte Carlo standard error 0.1048) and 1.3761 (0.1051). Both gains come from the youngest held-out dugong, aged 0.16: there the two programs score 1.5402 and 1.5240 above the reference, while across the other 19 held-out dugongs the reference scores 0.1343 and 0.1480 higher in total. Loop proposal 4 had a positive gain of 0.0907 (0.1355), within two Monte Carlo standard errors of the reference. For disease spread, all six programs that scored beat the floor model (a program which ignores the input, predicting with the same distribution for every input) and scored below the reference.

Both tasks come from BoxingGym, a benchmark of simulated scientific environments, and in each task the language model received a description in English together with 80 observations of one simulated dataset, the other 20 being held out for scoring. Of its eight programs, four are single attempts and the other four come from a loop of two rounds. In each round the loop fits two proposals and calls the language model as a critic, and the first round's critic comments go into the prompt for the second round.

A program that fails one of the checks before fitting gets no score. The others are fitted to the 80 visible observations, with 200 tuning iterations followed by 200 retained posterior draws in one chain with seed 0; each draw is then used 20 times to simulate a new observation, giving 4,000 predictions at every input (with PyMC 5.21.1 and ArviZ 0.23.4). Their held-out log scores, the sums of their log predictive densities at the 20 held-out observations, can only be compared within one task, higher being better.

Generation and fitting costsCopy Anchor Link

Sampler seconds are the part of the scoring seconds spent drawing posterior samples. While the loop runs, it also attempts a fit of each of its four proposals, using 200 tuning iterations and 200 retained draws in one chain, with an acceptance target of 0.90 and seed 42. These fits are part of how the loop works rather than part of scoring, and sampling for them took 50.95 of the loop's 87.87 total elapsed seconds on dugong growth, against 11.82 of its 45.80 total elapsed seconds on disease spread. The independent scoring fits use an acceptance target of 0.80 and seed 0.

Dugong growthCopy Anchor Link

In this environment, the length of a dugong is simulated from its age with the environment's own growth curve, priors (the distributions of the parameters before any data) and noise.

The taskCopy Anchor Link

Write a PyMC probabilistic model of dugong length as a function of age. Your model must be a Python
function `def model(data):` that returns a `pm.Model` instance, reading its data from the `data`
dictionary and declaring an observed likelihood for `length`.

Lengths of dugongs (sea cows) recorded at known ages. 80 observations are provided.

The prompt also contained a table of the columns, together with the summary statistics of the 80 visible rows. The reference program below can be thought of as the environment's growth curve written in PyMC: three parameters with the environment's own priors, plus a fixed noise scale.

import numpy as np
import pymc as pm

def model(data):
    """Dugong length against age: the growth curve and priors of BoxingGym's
    dugongs environment, with a Normal likelihood and a fixed noise scale.
    """
    age = np.asarray(data["age"], dtype=float)
    y = np.asarray(data["length"], dtype=float)
    with pm.Model() as dugongs_model:
        age_data = pm.Data("age_data", age)
        alpha = pm.Normal("alpha", mu=2.0, sigma=0.2)
        beta = pm.Normal("beta", mu=1.5, sigma=0.5)
        lam = pm.Normal("lam", mu=0.4, sigma=0.5)
        mu = alpha - beta * pm.math.abs(lam) ** age_data
        pm.Normal("length", mu=mu, sigma=0.25, observed=y)
    return dugongs_model

Scores and predictionsCopy Anchor Link

Monte Carlo standard errors (MCSEs) use batch means of each draw's delta-method influence on the total held-out log score, in ten consecutive batches of 20 draws; MCSEs for differences combine the two fits in quadrature, adding the two variances and taking the square root, under the independent-fit approximation.

At the youngest ages, single attempts 2 and 4 start lower than the reference, and their bands cover the held-out length of 0.64 at age 0.16, which the reference misses. The floor model, which ignores age, has a sampled predictive band that is approximately level.

Disease spreadCopy Anchor Link

The taskCopy Anchor Link

Your goal is to be able to reliably predict the number of infected individuals at specific times. Conduct experiments to learn about the environment and make predictions based on your observations.

Write a PyMC function `def model(data):` returning a `pm.Model`. Read the supplied data dictionary and declare an observed likelihood for `Infected_Count`. The final scorer performs fitting; do not sample at module scope.

Synthetic disease-spread observations in a population of 50 individuals; Time lies strictly between 0 and 2.

80 observations are provided.

Scores and predictionsCopy Anchor Link

The reference misses only one held-out count, the 4 infected at time 0.17, which lies below its interval from 5 to 13. The panels of the two programs that failed at model build show their observations without a band.

This page is also available as one self-contained HTML file: the HTML report.