Validation

Two checks back the atlas: the population simulation layer was compared with an established open-source tool, and every model was implemented a second time from its paper alone, with separate code, and compared with the atlas implementation. Both checks were made within this project; none has yet been made by outside people.

Simulation layer against mrgsolve

The dosing and population machinery was run against mrgsolve 2.0.1 (R) on identical models, doses and time grids, with tight solver tolerances on both sides (2026-11-04). The scripts are in validation/mrgsolve/.

caselargest relative differenceresult
IV bolus, repeated doses (ii, addl)8.7e-10agree
infusions every 12 h, two overlapping infusion streams, infusion longer than ii5.4e-10agree (rates add)
oral dose with bioavailability 0.6 and lag 0.75 h; lag longer than ii7.4e-9agree (mrgsolve fails on lag longer than ii with addl; explicit dose records agree)
bioavailability on a fixed-rate infusion (scales duration) and on a fixed-duration infusion (scales rate)5.7e-10agree
steady state: bolus, oral with lag, infusion shorter and longer than ii, turnover model, steady state after earlier doses7.2e-9agree
steady state with lag longer than ii4e-9 against 200 explicit dosesQSPAtlas agrees with the explicit doses; mrgsolve fails
reset (EVID 3) and reset with dose (EVID 4) during an infusion3.3e-10agree
new infusion after a resetQSPAtlas matches the closed form (17.470 at 12 h); mrgsolve keeps the old infusion's stop event and ends the new infusion 6 h early
doses at the same time; a bolus at the moment one infusion stops and another starts5.5e-10agree
population: one-compartment oral, correlated log-normal CL and V, proportional error, 5,000 and 20,000 subjectswithin Monte Carlo erroragree; the zero-random-effect prediction equals the deterministic model in both tools

Dataset cases

NONMEM-style dataset files (validation/mrgsolve/datasets/) were read by both tools, read_nmdata in QSPAtlas and mrgsim(data = ...) in mrgsolve, with mrgsolve's default record handling (data set order kept, recsort = 1, nocb = TRUE) on a two-compartment model with depot and a turnover response, clearance and central volume scaled by a covariate WT (2026-11-05). One row per observation record:

caselargest relative differenceresult
record order: observations before and after dose records at the same time (bolus, oral, infusion start), observation at an ADDL dose time4.7e-9agree
time-varying covariate, NOCB1.4e-9agree
time-varying covariate, LOCF1.4e-9 (states)states agree; the output at a record where the covariate changes differs (see below)
EVID 2 records carrying covariate changes1.4e-10agree; no output rows for EVID 2
SS=2: 10 every 24 h at 8 h plus 15 at 20 h; 10 every 12 h plus 5 every 24 h; oral SS=2 on an infusion SS=16.1e-9 (PK)PK agrees; the turnover response differs (see below)
steady-state constant infusion (AMT 0, RATE 30, SS 1), alone, continued by an explicit infusion, and with a modelled rate (RATE -1)1.7e-10agree; in both tools the infusion ends at the record, as the NONMEM Users Guide describes
modelled rate (RATE -1) and duration (RATE -2) with bioavailability 0.5, also with SS=12.6e-10agree
reset (EVID 3) and reset with dose (EVID 4) with observations before and after them6.1e-9agree
SS=1 record while ADDL doses of an earlier record are still to come (bolus doses; infusions, one running at the SS=1 time)6.9e-10 on the 14 records the cancellation does not reach (before the first cancelled dose, and a control subject without SS); up to 0.93 after itdiffer after the SS=1 record: mrgsolve keeps the pending ADDL doses (see below)

The same semantics are tested against closed forms in test/dataset.jl (SS=2 against superposed and explicit dose histories, the constant infusion against Css = R / CL).

Dose control with random effects and between-occasion variability

A one-compartment model with first-order absorption, bioavailability F = 0.6 exp(ηF) WT / 70 and lag ALAG = 0.75 exp(ηLAG) on the depot (computed by the model's dosecontrol), and clearance CL = 2 exp(κ) with κ of the occasion (OCC column, NOCB; mrgsolve selects κ by OCC in $MAIN, the NONMEM indicator-variable way), 2026-11-05. In the exact case every subject has SUPPLIED η and κ values (data set columns in mrgsolve, η and κ arguments in QSPAtlas), so individual curves are compared record by record; in the distribution case both tools draw their own random effects.

caselargest relative differenceresult
fixed η and κ: two occasions, doses at the start of each6.2e-11agree
fixed η and κ: WT changes on the second dose record (that dose uses WT 95)1.1e-10agree
fixed η and κ: addl doses, and addl doses running into occasion 2 and a new WT (they keep their record's F and ALAG in both tools)3.8e-11agree
fixed η and κ: occasion labels 1, 2, 1 (the first κ comes back)9.0e-11agree
sampled η (F, ALAG) and κ (CL, two occasions), 5,000 subjects: 5th, 50th, 95th percentiles at 14 timeslargest difference 1.77 bootstrap standard errors (none above 2)agree within Monte Carlo error

The same features are tested against closed forms in test/iov.jl (log-normal AUC under log-normal F, the lag distribution, covariance and independence of the κ draws, exact switching at occasion boundaries).

For the population case both tools were also compared with a near-exact reference (closed form, 4 million draws). QSPAtlas sampling with one million draws matches the reference to 1.7e-3 (largest z-score 1.96).

Differences in convention, documented in Population simulation:

  • Where steady state starts. QSPAtlas iterates dosing cycles from the model's initial conditions, mrgsolve from the state at the steady-state record. QSPAtlas and mrgsolve agree on every steady-state case above; they can differ for models with several periodic solutions or slow, history-dependent states. How NONMEM initialises its steady-state computation depends on the routine and was not compared with NONMEM output here.
  • Observations at a dose time see the state after the dose (mrgsolve recsort = 3) unless tagged :pre; datasets set the tag from the record order, and ADDL doses come before an observation at the same time (mrgsolve's default; not checked against NONMEM).
  • LOCF covariates. With nocb = FALSE, mrgsolve evaluates the output of a record with the previous record's covariates (the record's own values start after it); QSPAtlas uses the record's values at the record. The states agree; an output that depends on the covariate (here the concentration, through the volume) differs at the change records. NOCB is the convention the NONMEM documentation describes; under NOCB QSPAtlas and mrgsolve agree.
  • SS=2 with an endogenous baseline. mrgsolve adds the whole steady state of the SS=2 dose, including the turnover response's own baseline (the response jumps from 100 to 200 with no drug effect when IMAX = 0). QSPAtlas adds the periodic state minus the dose-free periodic state, so the baseline is not counted twice. For PK compartments starting at zero the two are the same and agree to 6e-9.
  • Steady-state constant infusion. The infusion ends at the record (NONMEM Users Guide); mrgsolve 2.0.1 behaves the same, although its user guide describes the infusion as never-ending.
  • SS=1 after a record with ADDL doses. QSPAtlas cancels the ADDL doses of earlier records that are still to come at the SS=1 time, as the NONMEM help on SS describes ("infusions in progress or pending additional doses are cancelled"); not compared with NONMEM output. mrgsolve 2.0.1 stops the running infusion but keeps the later ADDL doses: its output agrees to 7.0e-10 with QSPAtlas run with those doses written as dose records of their own (case d8_ss_addl, ss_addl_qspatlas.jl and ss_addl_mrgsolve.R). QSPAtlas on the dataset and QSPAtlas with the cancelled doses left out by hand agree exactly.
  • A reset cancels lagged doses that have not yet been given and keeps later addl doses.
  • A lagged dose entering exactly at an observation time. QSPAtlas reports the state after the dose (as for any dose at an observation time); mrgsolve 2.0.1 reports the state before it. Found while setting up the IOV cases (a lag of exactly 0.75 h on a 0.75 h observation); the cases above use lags that do not coincide with observation times.

Third implementation from the authors' own code

For four models the authors published their NONMEM or Monolix code. That code was translated to rxode2 (R) with nonmem2rx and monolix2rx (or by hand where the code is printed in a PDF), run with the parameter values and inputs the atlas uses (the code often carries initial estimates, so this checks equations, structure, dosing and units, not the values), and compared point by point with the atlas. Scripts: validation/authors_code/.

modelroutecaseslargest relative difference
Ouerdani 2015nonmem2rx (both control streams)Figure 4 mouse and patient; mouse AUC 220 and 6575.2e-10
Hutchinson 2016monolix2rx and a hand translation (identical)Figures 2a, 2c, 2e and 6a1.7e-10 (2.5e-10 in the window cases with a tighter atlas solve)
Lindauer 2017nonmem2rx, uneditedmouse PK, receptor occupancy and tumour volume at 0, 0.1, 1.4 and 5 mg/kg1.2e-8
Stein 2019by hand (code printed in the supplement PDF)typical curve and three comedication orders2.2e-12

Differences between the code and the paper that this check exposed (none affects an atlas target): in Ouerdani's code a control run (DOSE = 0) switches all drug terms off, while the paper's Eq. 2 with beta_alpha = 0 keeps alpha = alpha0 for any exposure; in Hutchinson's code the normalisation factor is N = Nmax x dose + 1, not Eq. 5's Nmax; Hutchinson's Figure 6 chemotherapy is not in the code. Stroh and Fornari published no runnable code (Stroh's supplement has the reaction matrices, checked entry by entry; Fornari's code is truncated).

Second implementations of the six models

Each model was implemented a second time from its paper and supplement alone, without access to the atlas code, in plain OrdinaryDiffEq without ModelingToolkit, by a separate AI agent session within this project (see How models are built). The second implementation made its own choices for every unstated input. It compared simulations only: nothing was re-digitised, so digitising errors are not covered by this check. Where both implementations chose an unstated input (a dose group, a starting value) by matching the figure, their agreement is not independent evidence for that choice. Results:

modelagreement with the atlas implementation
Ouerdani 2015Figure 4 curves agree to 0.003 mm³ (mice) and 0.06 mm (patient K); the patient P curves differ by 0.2 mm at the start only, because the two implementations read the starting size from the figure as 66.8 and 67 mm (66.8 vs 67 mm). The same dose group (100 mg/kg) was chosen independently.
Hutchinson 2016All 22 Figure 2 curves agree to within 2e-5 of each curve's range. The rebuild found a reading of Figure 6 (resistance clock from the start of chemotherapy, chemotherapy scale 0.01) that brings panels b to d close; without independent evidence for it, it is reported as a sensitivity run and Figure 6 stays C.
Lindauer 2017Identical base values (-1.302 and -0.401 with the Table 1 KdegPD1) and the same six bars that do not match the figure, with the same unstated choices made independently.
Fornari 2019Same closure fixes. The rebuild found that reading the rat PK volumes per kg reproduces Figure 3 (see the model page).
Stroh 2019The reaction network built from the text matches the supplement matrices entry for entry (234 and 84 entries).
Stein 2019Same reading of the variability (standard deviations) and the same result: early phase inside the published bands, late phase below them in every variant tried.

No transcription error was found in any atlas model. One second implementation (Fornari) changed the reading of a units question that decides a figure grade; see that model's page.