Schema Evolution History

This document provides a concise summary of PowderLine’s schema evolution for context. For detailed commit history, see the repository’s git log. Current schema: 0.26.0

Schema 0.26.0: Per-Parameter Refinement Flags Honored (Current)

Status: Active (Q3 2026)

Key Changes:

  • Per-parameter refine flags are now honored (behavioral change, no shape change) for the three parameter classes GSAS-II exposes only as lumped flags:

    • Unit-cell parameters (a, b, c, alpha, beta, gamma) — previously ANY true flag refined the whole symmetry-allowed cell; false flags on other cell parameters were silently ignored.

    • Atomic coordinates (x, y, z) — previously any true flag refined all three together.

    • Anisotropic displacement components (U11..U23) — previously any true flag refined all symmetry-allowed components together. A parameter now refines iff it is present with refine_flag=true; absent or false means fixed. Internally, PowderLine emits GSAS-II “Hold” constraints for the not-refined degrees of freedom (new module powderline.constraints).

  • Symmetry-linked parameters refine together (group-OR): parameters coupled by the phase’s Laue class or site symmetry (e.g. cubic a=b=c; monoclinic a/c/beta; x=y on an (x,x,z) site) form one degree-of-freedom group that refines if any member is requested. Flags on symmetry-fixed parameters (e.g. cubic angles) have no effect. PowderLine does not validate flag/symmetry consistency — producing consistent flags is the recipe author’s (or an upstream recipe builder’s) responsibility (see KI-01/KI-02 in docs/known_issues.md).

  • Partial parameterization is equivalent to explicit flags: listing only the parameters you wish to refine (absence = fixed) is equivalent to listing all with explicit true/false flags. Shipped examples use the explicit style for clarity.

  • Fix: a phase parameterization that omits the peak_broadening section no longer raises (set_phase_parameterization previously called .get on the None that model_dump emits for an absent section); minimal recipes now run.

Rationale:

  • The recipe is the point of truth: a refine_flag: false that is silently overridden (e.g. a hexagonal example’s c refined despite c: false in schema 0.25) misrepresents what the refinement actually did. Schema 0.26 makes the flags mean what they say.

Migration: Update "schema_version" from "0.25.4" to "0.26.0" in all recipe JSON files, then review your refine flags: any parameter previously piggy-backing on a lumped flag (e.g. cubic b, c marked false next to a: true, or y/z false next to x: true) is now genuinely held. Set every parameter you want refined to true explicitly (symmetry-linked partners refine together regardless, but explicit flags keep the recipe honest).

Breaking Changes:

  • No changes to recipe format.

  • Refinement behavior changes for recipes whose per-parameter flags disagree within a symmetry-linked group or across a formerly-lumped class: previously-ignored false flags are now honored, so fewer parameters may vary than before.

xrd_data validation hardening (within 0.26.0 — no version bump): XRDDataModel now rejects ill-formed data instead of passing it through to GSAS-II. It requires: non-empty, equal-length arrays; all-finite tth/Itth/Itth_weights (no NaN/inf — catches, e.g., a weight of 1/esd² with esd=0); strictly increasing tth; and weights >= 0 with at least one > 0. Negative intensities remain valid (background-subtracted data legitimately dips below zero — Itth is checked for finiteness only, not sign). This is a validation tightening, not a shape or kicker.py output change, so the version is unchanged and previously-valid recipes are unaffected (the empty-array example_template skeleton is intentionally not a runnable recipe). Rationale: PowderLine is strict and never repairs xrd_data; transforming raw data (unit conversion, weight derivation, negative-intensity handling) is the job of the file reader/parser that builds the block.

Schema 0.25.4: Consistent Failure Metadata

Status: Superseded by 0.26.0

Key Changes:

  • Consistent traceback in all run_refinement() failure exits (behavioral change): failure exits 1 and 3 (histogram loading error, executor exception) now capture and return the Python traceback string under a 'traceback' key. Exit 2 (executor returned success=False) forwards the executor’s traceback if present, or None. Previously only the outermost unexpected-exception exit (exit 4) included a traceback.

  • run() normalizes traceback key: run() now calls result.setdefault('traceback', None) so callers can always key into result['traceback'] without a KeyError, regardless of which failure exit triggered.

Rationale:

  • Consistent traceback across all failure exits simplifies debugging without requiring callers to handle multiple return shapes

Migration: Update "schema_version" from "0.25.3" to "0.25.4" in all recipe JSON files. No other recipe changes required.

Breaking Changes:

  • No changes to recipe format

Schema 0.25.3: Refinement Failure Detection and Windows Support

Status: Superseded by 0.25.4

Key Changes:

  • Robust refinement failure detection (behavioral change): execute_rietveld_refinement and execute_spf_refinement now return success=False when the refinement produces no Rwp (convergence failure), instead of silently returning success=True. All success=False paths in run_refinement() now consistently include 'rwp': None and forward the executor’s error message rather than a generic fallback string.

  • Windows (win-64) platform support: pixi.toml now targets both linux-64 and win-64. Includes flang/flang-rt_win-64 Fortran toolchain, an ar.exe wrapper (scripts/ar_wrapper.c) for MSVC archive compatibility, scripts/win_activate.bat. Shell scripts enforce LF line endings via .gitattributes.

Rationale:

  • A refinement that produces no Rwp has not converged; returning success=True was misleading and caused silent data quality issues downstream

  • Windows support is required for deployment on user workstations at partner institutions

Migration: Update "schema_version" from "0.25.2" to "0.25.3" in all recipe JSON files. No other recipe changes required.

Breaking Changes:

  • No changes to recipe format

  • Code that expected success=True when a refinement produced no Rwp must now handle success=False

Schema 0.25.2: Programmatic Python API

Status: Superseded by 0.25.3

Key Changes:

  • powderline.run(): Primary programmatic entry point. Accepts dict | RecipeModel, output_dir: Path, and execution_mode parameter ('auto' | 'server' | 'subprocess'). Eliminates the need for callers to manage raw JSON primitives or re-read output files.

  • powderline.validate(): Standalone schema validation without GSAS-II execution. Returns a RecipeModel or raises pydantic.ValidationError. Enables fast CI-friendly recipe linting.

  • run_id: UUID4 attached to every execution path. Enables log correlation and result tracing across concurrent or sequential runs. Not present when validate_only=True.

  • Structured DataFrame returns: run() returns pd.DataFrame objects for all tabular outputs (fit_profile, unit_cell_data, peak_list_data, refined_parameters, spf_peaks, spf_convergence_diagnostics). Callers never receive raw dicts or lists. This eliminates column-index bugs that arose when callers parsed the 15-column reflection list by position.

  • DataFrame normalization boundary: run_refinement() continues to return JSON-serializable primitives (required for HTTP transport). run() converts them to DataFrames at the public API boundary — keeping the server/client architecture intact.

  • GSASClient input polymorphism: submit_simulation() now accepts Path | dict | RecipeModel. Previously only Path was accepted, requiring callers to serialize models back to disk before submitting.

  • GSASClient auto-start + retry: Automatic server start on first call; 3-attempt exponential backoff (0.5 s → 1.0 s → 2.0 s) on ConnectError only. HTTP 4xx/5xx and timeout errors are not retried.

  • SimulationResponse (HTTP): FastAPI response model now carries all structured data fields (fit_profile, unit_cell_data, peak_list_data, refined_parameters, spf_peaks, spf_convergence_diagnostics), enabling rich API consumers without a separate file-read step.

  • run(validate_only=True): Fast validation path returns a slim summary dict (success, schema_name, schema_version, phases, refinement_cycles, simulation_mode) with no run_id and no GSAS-II invocation.

Rationale:

  • Direct run_refinement() calls required callers to reconstruct results from files or parse raw primitive dicts; run() removes this boilerplate at the correct abstraction level

  • Standalone validate() makes recipe linting a one-liner in CI pipelines without importing GSAS-II

  • run_id makes it possible to correlate results, logs, and output files from batch or concurrent runs without relying on timestamps or output-directory names

  • Returning DataFrames (not raw dicts) at the public API boundary is the correct place for this transformation: it keeps HTTP serialization clean while giving programmatic callers the right type immediately

  • Polymorphic GSASClient input removes the most common adoption friction: having to write a RecipeModel to a file before calling submit_simulation()

  • Auto-start + retry makes server mode transparent for new users and removes the need to manually manage server lifecycle in scripts

Migration: Update "schema_version" from "0.25.1" to "0.25.2" in all recipe JSON files. No other recipe changes required.

Breaking Changes:

  • No changes to recipe format

  • run() callers using the method= keyword argument must rename it to execution_mode=

  • GSASClient.submit_simulation() callers passing recipe as a positional Path continue to work unchanged; passing as dict or RecipeModel is now also valid

What NOT to do:

  • Do not add DataFrame conversion inside run_refinement() — it must remain JSON-serializable for HTTP transport

  • Do not bypass run() for new programmatic callers; run_refinement() is an internal execution engine, not a public API

  • Do not add retry logic for HTTP 4xx/5xx errors — these indicate recipe or server bugs that should surface immediately

Schema 0.25.1: Refined Parameters Export + Extended SPF Returns

Status: Superseded by 0.25.2

Key Changes:

  • New output: refined_parameters.csv with 9 columns including phase/atom associations and ESDs

  • Updated unit cell reports: Now include esd column (3 total columns instead of 2)

  • Per-column float formatting: Values use %.6e, ESDs use %.8e for appropriate precision

  • SPF DataFrames: powderline.run() now returns spf_peaks and spf_convergence_diagnostics as pandas DataFrames; previously only written to files

  • Schema validator: GSASII_SPF schema now validates payload.refinement_controls.single_peak_fitting_mode at parse time, not at runtime

  • run_refinement() signature: recipe_path parameter removed (was accepted but never used); method default changed from 'direct' to 'server'

  • SimulationResponse: Added spf_peaks and spf_convergence_diagnostics optional fields to FastAPI response model

Rationale:

  • Comprehensive ESD reporting enables uncertainty quantification for all refined parameters

  • Machine-readable CSV format enables automated downstream analysis

  • Per-column formatting preserves significant figures for small ESDs while keeping values concise

  • SPF DataFrame return makes single-peak-fitting results immediately usable programmatically without re-reading output files

  • Schema-level SPF validation gives earlier and clearer error messages

Migration: Update "schema_version" from "0.25" to "0.25.1" in all recipe JSON files. No other recipe changes required.

Breaking Changes:

  • run_refinement() callers that passed recipe_path= keyword argument must remove it

  • run_refinement() callers that relied on method='direct' default must pass it explicitly (new default: method='server')

Schema 0.25: Simplification

Status: Superseded by 0.25.1

Key Changes:

  • Payload-based structure: All recipe fields moved under "payload" key

  • Fixed output filenames: dummy.gpx, dummy.lst, <phase>_unit_cell_report.csv (independent of sample names)

  • Removed metadata fields: sample_name, recipe_description (manage externally)

  • Schema identification: Required "schema_name" field ("GSASII_Rietveld" or "GSASII_SPF")

  • Permanent removal of multi-strategy system: Sequential and iterative refinement workflows removed (not planned for future development)

  • Peak broadening model field: Added "model" field to support future extensibility (uniaxial, ellipsoidal)

  • Size/strain renaming: "size" → "isotropic_size", "strain" → "isotropic_strain"

Rationale:

  • Payload structure separates schema metadata from recipe content

  • Fixed filenames eliminate variability in testing/automation

  • Multi-strategy refinement didn’t work as intended; workflows now split into two schema_name options

  • Sample identity belongs in external metadata (directory names, databases)

Migration: For recipes from schema 0.24 or earlier:

  1. Add "schema_name" field ("GSASII_Rietveld" or "GSASII_SPF")

  2. Wrap all recipe fields in "payload" object

  3. Remove "sample_name" and "recipe_description" fields

  4. Update strategy field: remove "strategy" from refinement_controls, use schema_name instead

Schema 0.24: Multi-Strategy Refinement System

Status: Deprecated (replaced by schema 0.25)

Key Features:

  • Four refinement strategies: peaks_only, structural_only, sequential, iterative

  • Single peak fitting (SPF) integrated with structural refinement

  • Advanced convergence diagnostics with aphysical parameter detection

  • Optional trajectory/comparison outputs

Why Removed: Multi-strategy system proved complex with limited benefit. Schema 0.25 simplified to two workflow types selected by schema_name field.

Breaking Changes:

  • Required refinement_controls with strategy field

  • Moved refinement_cycles from top-level to refinement_controls

  • Renamed Gaussian parameter: pv_gaussian_sigma → pv_gaussian_sigma_sq

  • Moved use_instrument_profile into single_peak_fitting_mode block

Schema 0.23: Initial Multi-Strategy Exploration

Status: Deprecated

Key Features:

  • Introduced basic strategy concept (peaks vs structural)

  • Optional refinement_controls block

  • Initial single peak fitting integration

Notes: Experimental implementation that led to full multi-strategy system in 0.24

Schema 0.22: ADP Field Requirements

Status: Deprecated

Key Change: Made ADP (Atomic Displacement Parameter) field required in both structure and parameterization sections for all atoms.

Rationale: Eliminated ambiguity about which displacement parameters were active. Previously optional with complex fallback logic.

Breaking Change: Validation fails if ADP missing from atom definitions. No default value provided.

Feature: Structure and parameterization ADP values can differ (e.g., CIF provides Uaniso, but refinement uses Uiso).

Schema 0.21 and Earlier

Status: Historical

Early development versions focused on basic Rietveld refinement capabilities:

  • JSON recipe structure establishment

  • GSAS-II G2scripts API integration

  • Basic parameter setting/refinement

  • CIF-based structure loading

  • Single-phase refinements

Origin (Schema 0.21–0.24)

PowderLine began as a minimal proof-of-concept to confirm that GSAS-II could be driven from a JSON file via the G2scripts API. The initial scope established the directory structure and documentation, a JSON schema derived from 3–5 real examples, JSON loading and schema validation in kicker.py, output written to a directory, at least three refinement types working end-to-end, and basic tests covering JSON loading and validation.

Features deferred from that initial scope (REST API, structured output parsing, configuration system, programmatic Python API) were subsequently delivered in schema 0.25.1 and 0.25.2.

Schema Design Philosophy

Example-Driven Development: Features are added when real refinement examples need them, not speculatively. Schema evolution is guided by committed examples in examples/ directory.

Schema-First Validation: Pydantic models provide clear error messages before GSAS-II operations. Fail fast with helpful diagnostics.

Progressive Enhancement: Uses ConfigDict(extra='allow') to permit field additions without breaking existing recipes.

Separation of Concerns: Recipe JSON contains only refinement parameters. Sample identity, descriptions, and documentation live in external files (directory names, DESCRIPTION.md, databases).

Migration Philosophy

  • Explicit schema versions: Every recipe declares its "schema_version". The code accepts one current version — 0.26.0 — and recipes written for older versions must be migrated to it following the per-version notes above.

  • Clean breaks with migration guidance: Obsolete features are clearly deprecated, and each version entry documents the migration steps rather than maintaining indefinite compatibility shims.

  • Git history is source of truth: This document provides high-level context only.