Regression tolerance study — cross-build drift

Purpose: choose the numeric tolerances for the GSAS-II regression tests (refined_parameters.csv, *_unit_cell_report.csv) that absorb cross-build / cross-OS floating-point drift while still catching real code regressions.

Key premise (established by the Windows/Linux cross-platform validation runs): refinements are byte-deterministic per machine (same input → byte-identical output on repeated runs). Therefore there is no run-to-run scatter to measure; the meaningful quantity is the cross-build delta between a platform’s output and the committed reference (which was generated on Linux). This document tabulates that delta.

Method

For each example with a committed refined_parameters.csv, run pixi run kicker --no-server examples/<name>/input.json --output out_drift/<name> and compare the generated refined_parameters.csv to the committed reference, per-parameter, grouped by the category column. Metric: relative difference |gen - ref| / max(|ref|, 1e-30) on the value and esd columns.

Windows results (Python 3.13.14, GSASII 5.7.9, win-64 source build)

Reference = committed Linux output. Full example set at measurement time (including several examples that are not part of the public example set); all shapes matched exactly.

Per-example overall `value` max relative drift

Example	value max rel	worst parameter
example_DRX_33_atomrefine	2.95e-3	`0:0:Size;i`
example_DRX_33_anisoADP	3.95e-4	`1::AU22:4`
example_DRX_33	1.13e-4	`0:0:Size;mx`
example_DRX_33_strainonly	8.5e-6	`:0:Back;2`
example_LaB6	0	(identical)

(The measurement covered a larger example set — several examples not part of the public example set also reproduced the Linux reference byte-for-byte; example_LaB6 is the byte-identical case shown here.)

Per parameter-class max relative drift (across all examples)

Category	max `value` rel	max `esd` rel
`other` (size/strain: Size, Mustrain)	2.95e-3	3.6e-4
`atom_occupancy` (Afrac)	6.2e-4	1.5e-4
`atom_displacement_anisotropic` (Uaniso)	3.9e-4	3.9e-6
`atom_displacement_isotropic` (AUiso)	3.1e-4	4.9e-4
`scale`	9.2e-5	1.1e-4
`background` (Chebyshev/Back)	5.7e-5	5.8e-6
`phase_other` (coord shifts dA*)	4.0e-5	2.0e-5
`reciprocal_metric_tensor` (cell A0..A5)	2.3e-7	1.3e-4
`instrument_broadening` (U,V,W,X,Y,Z)	0	1.3e-7
`background_peak`	0	0

Reading

Size/strain broadening (other) is the lone outlier at ~3e-3 — expected: crystallite-size and microstrain terms are strongly correlated and ill-conditioned. It drives the atomrefine example’s overall drift.
Every other class is ≤ 6.2e-4 on value; esd ≤ 4.9e-4 everywhere.
Cell / reciprocal-metric-tensor terms are ~2e-7 — effectively exact, confirming the unit_cell report tolerance (rtol=1e-4) has enormous margin.

Chosen tolerances

Implemented in tests/subprocess_utils.py:

unit_cell_match: rtol=1e-4, atol=1e-6 — validated (all 30 unit-cell comparisons pass; cell drift ~2e-7 gives ~500x margin).
refined_params_match: rtol=1e-3, atol=1e-4 for all parameter classes, except correlated size/strain broadening (category other: Size, Mustrain), which uses loose_rtol=1e-2. Rationale: every non-size/strain class drifts <6.2e-4 (value) / <4.9e-4 (esd) across builds, so the tight default is a strong regression guard; correlated size/strain is ill-conditioned (the classic size-strain separation problem, plus the poorly-determined Lorentzian/Gaussian ;mx mix terms) and is only reproducible to ~1% (observed max 2.95e-3), so it gets ~3.4x margin at 1e-2. The looser tolerance is scoped to that one class so well-conditioned parameters keep their tight guard.
Rwp: abs < 0.1 pp (observed delta ~0.00).

Validation: with these tolerances all 9 examples’ refined_parameters.csv and all 30 *_unit_cell_report.csv comparisons pass on win-64.

Follow-up (RESOLVED 2026-08-11): keep the loose tolerance

example_DRX_33_atomrefine is the only example needing the loose class tolerance. Its recipe refines correlated size and strain broadening (+ the ;mx mix terms) on top of atom parameters, which is inherently ill-posed. Making it well-conditioned was investigated on Linux and not adopted as a design decision; the baseline recipe + other → loose_rtol=1e-2 is kept. Evidence (all variants run on linux-64, GSASII 5.7.9):

Variant	Outcome
Fix size = 10 µm, refine strain (mirrors `strainonly`)	Diverges — “Invalid cell metric tensor” for phase 0 (DRX_33), cell → −1.48e6 Å, refinement fails, no output
Fix size ≈ refined optimum (3.25 / 11.24 µm), refine strain	Diverges identically — not a value-tuning issue; removing the refined size DOF destabilizes the simultaneous atom+cell+strain refinement
Refine size, fix strain	Converges but Rwp degrades to 14.2% and `Size;mx` stays pathological (−1352 ± 100)
Fix only the `;mx` LG-mix terms, keep size+strain	Converges, Rwp 11.25% (fit preserved), worst mix terms gone — but size+strain remain co-refined (still needs some tolerance, unverifiable cross-build from Linux alone)

Additionally, any edit to this recipe breaks two committed TOPAS goldens — test_declarative_examples_inp_and_xye_match_golden[atomrefine] (golden INP, regenerable on Linux) and test_declarative_examples_roundtrip_reproduce_committed[atomrefine], which round-trips against a real Windows TOPAS _results.csv under output/topas/ (not regenerable on Linux — needs tc.exe). Since the candidate change diverges and every variant entangles the Windows-only TOPAS path while not cleanly achieving tight-and-good-fit, the design decision is to keep the baseline recipe and the scoped loose tolerance. Caveat: re-attempting this change requires a Windows TOPAS session to regenerate the roundtrip reference end-to-end.

Linux results (Python 3.13.14, GSASII 5.7.9, linux-64 conda-build)

Reference = committed Linux output. Same example set, run once each (refinements are byte-deterministic per machine, so a single run fully characterizes the platform — no N× repeat needed). All shapes/labels matched exactly.

Per-example overall `value` / `esd` max relative drift (Linux gen vs committed ref)

Example	value max rel	esd max rel	worst parameter
example_DRX_33_atomrefine	4.61e-6	7.87e-7	`0:0:Size;i`
example_DRX_33	0	0	(identical)
example_DRX_33_anisoADP	0	0	(identical)
example_DRX_33_strainonly	0	0	(identical)
example_LaB6	0	0	(identical)

Every measured example except one reproduces the committed reference byte-for-byte. The lone non-zero (atomrefine, 4.6e-6) is on 0:0:Size;i — the same correlated-size term that is worst on Windows — and is non-zero only because the committed reference was minted on a slightly different Linux build. All *_unit_cell_report.csv comparisons are exactly 0 (global unit-cell max rel drift = 0.00e+00).

Windows-value vs Linux-value (cross-platform delta)

Because the current Linux output equals the committed Linux reference to within ≤4.6e-6, the Windows “vs reference” column above is effectively the Windows↔Linux cross-platform delta. Restating per class:

Category	Win↔Linux `value` rel (≈ Win vs ref)	Linux vs ref	tight guard (1e-3) margin
`other` (Size, Mustrain)	2.95e-3	4.6e-6	fails tight → uses `loose_rtol=1e-2` (3.4× margin)
`atom_occupancy` (Afrac)	6.2e-4	1.1e-6	1.6×
`atom_displacement_anisotropic`	3.9e-4	0	2.6×
`atom_displacement_isotropic`	3.1e-4	7.7e-7	3.2×
`scale`	9.2e-5	0	11×
`background`	5.7e-5	1.9e-7	18×
`phase_other`	4.0e-5	0	25×
`reciprocal_metric_tensor` (cell)	2.3e-7	0	4300×
`instrument_broadening`	0	0	∞
`background_peak`	0	0	∞

The drift is one-sided in magnitude (Linux≈exact, Windows carries all the cross-build noise) but the class ordering is identical on both platforms: correlated size/strain is the sole outlier; cell/metric terms are essentially exact.

Sign-off on tolerances (Linux)

unit_cell_match rtol=1e-4, atol=1e-6 — CONFIRMED. Linux drift is exactly 0 on all 13 unit-cell reports; Windows is ~2e-7. Margin is ~500× (Windows) to ∞ (Linux). No change.
refined_params_match rtol=1e-3, atol=1e-4 default — CONFIRMED. Every non-other class drifts ≤6.2e-4 cross-platform (≥1.6× margin) and ≤1.1e-6 on Linux. Holds on both platforms.
other → loose_rtol=1e-2 — CONFIRMED. Windows size/strain drift 2.95e-3 (3.4× margin); Linux 4.6e-6. The one-class exception is still the minimal loosening that admits the correlated size/strain noise without weakening any other guard.
Rwp abs < 0.1 pp — CONFIRMED (Δ ≈ 0 on Linux).

Recommendation: keep the shared tolerances as-is; do not adopt a per-OS scheme. A per-OS tolerance would only help if the two platforms needed different values, but Linux is a strict subset of Windows drift (Linux ≤ Windows for every class), so the Windows-sized tolerance already covers both. A single cross-platform tolerance is simpler and correct.

LF byte-stability (Linux)

All Linux-generated refined_parameters.csv, *_unit_cell_report.csv, *_peak_list_report.csv, and fit_profile.txt contain zero CRLF sequences (pure LF). Diffs against the committed LF references appear only in numeric value/esd fields (and only for atomrefine), never in line endings or structure — confirming the lineterminator="\n" / newline hardening produces byte-stable output on Linux.