PowderLine Developer Guide

This guide provides comprehensive context for developers working on PowderLine (and for AI coding assistants used alongside them). Read this first to understand the project architecture, design decisions, and development workflows.

Project Overview

What is PowderLine? PowderLine automates crystallographic refinement from a declarative JSON “recipe” and drives interchangeable refinement engines — GSAS-II by default, with Bruker TOPAS v7 and open-source easydiffraction as optional alternatives. It takes a recipe describing refinement parameters and produces standardized output reports. The goal is to make powder diffraction refinements reproducible, automatable, and accessible.

Scope:

Command-line tool that runs refinements from JSON recipes, dispatching to interchangeable engines (GSAS-II by default; TOPAS v7 and easydiffraction optional)
Schema 0.26.0: Simplified two-workflow system (Rietveld refinement or single peak fitting)
Schema validation using Pydantic
Single-phase and multi-phase Rietveld refinements; phase-less single peak fitting
Standard output reports (fit profiles, unit cell parameters with ESDs, peak lists, refined_parameters.csv)
Programmatic Python API: powderline.run() with DataFrame returns
Example-driven design: schema evolves based on real refinement cases

Design Philosophy:

Example-driven: Don’t build features speculatively - add them when real examples need them
Schema-first: Validate inputs before any GSAS-II operations
Fail fast: Detect errors early with helpful messages
Reproducible: Same JSON should produce identical refinement results

Schema Structure

Current schema: 0.26.0 (payload-based structure with fixed output filenames)

Top-Level Structure

All recipes use a three-key structure:

{
  "schema_name": "GSASII_Rietveld",  // or "GSASII_SPF"
  "schema_version": "0.26.0",
  "payload": {
    // All refinement parameters here
  }
}

Key field: schema_name - Selects the workflow executor:

"GSASII_Rietveld": Full Rietveld structural refinement
"GSASII_SPF": Single peak fitting (peak-based analysis without crystal structure)

Payload Structure

The payload contains all refinement parameters organized by concern:

"payload": {
  "xrd_data": {...},           // Required: Diffraction pattern
  "instrument": {...},         // Required: Beamline/instrument config
  "phases": {...},             // Required for Rietveld, null for SPF
  "background": {...},         // Optional: Background model (defaults to empty if omitted)
  "refinement_controls": {...}, // Optional: Cycles, convergence
  "fit_range": [min, max],     // Optional: 2θ range limits
  "single_peaks": {...},       // Required for SPF, optional otherwise
  "single_peak_fitting_mode": {...} // For SPF workflows
}

Required fields vary by schema_name:

Rietveld (GSASII_Rietveld): Must have phases defined
SPF (GSASII_SPF): Must have single_peaks and single_peak_fitting_mode

Output Files (Fixed Names)

PowderLine uses fixed output filenames independent of sample identity:

dummy.gpx: GSAS-II project file (reopenable in GUI)
dummy.lst: Human-readable refinement log (Rwp, parameter tables)
<phase>_unit_cell_report.csv: Refined unit cell (one per phase)
<phase>_peak_list_report.csv: Reflection list (one per phase)
fit_profile.txt: Observed/calculated/difference intensities

Rationale: Fixed filenames eliminate variability in automation/testing. Sample identity should be managed externally (directory names, databases, metadata systems).

Refined Parameters Export with ESDs

File: refined_parameters.csv (9 columns)

PowderLine exports all refined parameters with estimated standard deviations (ESDs) after each refinement. This provides a structured, machine-readable record of refinement results with uncertainty quantification.

Column structure:

parameter_name: GSAS-II internal parameter identifier (e.g., ::0:Scale, ::A0, 0::A0)
descriptive_name: Human-readable description (e.g., “Scale factor for phase LaB6”, “Background coefficient A0”)
phase_name, phase_idx: Phase associations for phase-specific parameters (null otherwise)
atom_name, atom_idx: Atom associations for atom-specific parameters (null otherwise)
value: Refined parameter value
esd: Estimated standard deviation from covariance matrix
category: Parameter classification (instrument, background, cell, atom_position, atom_displacement, HAP, peak_broadening)

Implementation: export_refined_parameters_csv() in kicker.py

ESD calculation:

Primary source: GSAS-II covariance matrix (proj.data['Covariance'])
Fallback: A .lst-parsing fallback exists in kicker.py (extract_refined_params_from_lst) but is currently disabled at its call site. It is preserved pending a test that confirms covariance matrix ESDs match what .lst parsing previously produced. Until that test exists, the covariance matrix is the sole ESD source.
- Note: The .lst fallback was incomplete — it only covered instrument, background, and unit cell parameters, not atom or HAP parameters.
Unit cell ESDs: Uses G2lat.getCellEsd() to convert reciprocal metric tensor ESDs to direct lattice parameters
- Function: calculate_cell_esds_from_A_matrix() in kicker.py
- GSAS-II stores cell as reciprocal metric tensor (A-matrix); ESDs require conversion

Parameter name parsing:

parse_parameter_associations() in kicker.py: Extracts phase/atom indices from GSAS-II parameter names
get_descriptive_param_name() in kicker.py: Generates human-readable descriptions
build_phase_name_mapping() in kicker.py: Maps phase indices to names
build_atom_name_mapping() in kicker.py: Maps atom indices to labels

Phase-specific unit cell reports: In addition to the comprehensive refined_parameters.csv, each phase generates <phase>_unit_cell_report.csv with 3 columns:

parameter: a, b, c, alpha, beta, gamma, volume
value: Refined unit cell parameter
esd: Estimated standard deviation

Note on simulations: Simulation mode (all refine_flag=false) does not generate refined_parameters.csv since no parameters are refined. Unit cell reports still generated with esd=0.0.

Schema Evolution

See SCHEMA_HISTORY.md for historical context.

IMPORTANT — Schema Version Discipline: The GSAS-II server caches kicker.py at startup. If kicker.py changes (even without recipe format changes), a running server will use stale code. The schema_version in recipes is validated against EXPECTED_SCHEMA_VERSION in schema.py, so bumping the version forces a server restart. You MUST bump EXPECTED_SCHEMA_VERSION whenever kicker.py output behavior changes (new output files, format changes, column additions, etc.), even if the recipe input format is unchanged. Update all input.json files to match.

Domain Essentials

Parameter Types

PowderLine organizes refinement parameters into logical groups:

Structural Parameters (Phase-specific):
- Unit cell: a, b, c, α, β, γ define crystal lattice dimensions
- Atom parameters: positions (x, y, z), occupancy, thermal parameters
- Space group: symmetry operations (defined in CIF, not refined)
Intensity Parameters (Phase-specific):
- Scale factor: Converts calculated intensities to absolute scale
- Preferred orientation: Non-random crystallite orientations (not implemented yet)
Peak Shape Parameters (Phase-specific):
- Sample broadening: Crystallite size, microstrain
Background Parameters:
- Chebyshev polynomials: Smooth curved background
- Single peaks: Individual pseudo-Voigt peaks for known impurities

Background Default Behavior

The background field is optional in the payload. If omitted (background: null):

No Chebyshev coefficients are initialized
No background single peaks are added
GSAS-II uses an empty background (effectively zero)

This is useful when background has been pre-subtracted from XRD data.

Instrument Parameters:
- Instrument broadening: U, V, W (Gaussian), X, Y, Z (Lorentzian)
- Profile asymmetry: Axial divergence, sample transparency
- Wavelength: X-ray or neutron wavelength
- Zero shift: 2θ offset correction
- Polarization: For synchrotron data

The `[value, refine_flag, min, max]` Pattern

Throughout the JSON schema and code, you’ll see parameters represented as 4-element lists:

"wavelength": [0.45236, false, null, null]

value: Current parameter value
refine_flag: true = refine this parameter, false = hold fixed
min/max: Parameter bounds (placeholders for future - GSAS-II may not support bounds)

This mirrors GSAS-II’s internal parameter representation.

Phase vs Histogram Parameters

Critical distinction (confusing in GSAS-II):

Phase parameters: Belong to the crystal structure (scale, unit cell, atoms, space group, sample broadening)
Histogram parameters: Belong to the measurement (instrument broadening, background)

In multi-phase refinements, each phase has its own structural parameters, but histogram parameters are shared across phases (except scale factors, which are phase-histogram pairs).

Glossary

CIF: Crystallographic Information File - standard format for crystal structures (In PowderLine, structural information is provided as embedded dictionaries, not CIF files)
GPX: GSAS-II project file (binary format)
LST: GSAS-II refinement log file with Rwp and parameter tables
CHI: Two-column ASCII file (2θ, intensity) for powder diffraction data (Note: PowderLine uses embedded data arrays, not files. This format is for reference only)
INSTPRM: GSAS-II instrument parameter file (Note: PowderLine uses embedded instrument parameters, not files. This format is for reference only)
Recipe: PowderLine’s JSON input describing a refinement workflow
Multiplicity: Number of symmetry-equivalent reflections
d-spacing: Interplanar spacing for an hkl reflection
2θ (two-theta): Diffraction angle in degrees
hkl: Miller indices describing a crystallographic plane

GSAS-II Integration

Why Direct Dictionary Manipulation?

PowderLine manipulates the proj.data dictionary directly rather than using GSAS-II’s phase/histogram objects. This decision was made because:

Phase object ambiguity: Unclear how G2Phase initialization maps to parameter setting
Explicit control: Direct dictionary access makes parameter locations clear
Less abstraction: Easier to debug when you can see exact data structure
Trade-off: More brittle if GSAS-II internal structure changes (acceptable risk for Phase 1)

The `proj.data` Dictionary Structure

GSAS-II stores all project data in a nested dictionary. Key access patterns:

# Instrument parameters (list format: [current_value, value, refine_flag]). See kicker.py for setting functions
curr_wavelength = proj.data[hist.name]['Instrument Parameters'][0]['Lam'][1] # before setting, can get value from initialization (e.g., when xrd_data is added)
proj.data[hist.name]['Instrument Parameters'][0]['Lam'][1] = wavelength
proj.data[hist.name]['Instrument Parameters'][0]['Lam'][2] = refine_flag

# Background parameters
proj.data[hist.name]['Background'] = [
    [chebyshev, no_peaks, background_type],  # Background metadata
    {'nDebye': 0, 'debyeTerms': [], ...},     # Debye-Scherrer params
    {'nPeaks': n, 'peaksList': [...]},        # Single peaks
]

# Histogram-atom parameters (HAP)
# Change phase intensity scale given a value and refine_flag from payload for a phase's parameterization:
proj.data['Phases'][phase_name]['Histograms'][hist.name]['Scale'][0] = value
proj.data['Phases'][phase_name]['Histograms'][hist.name]['Scale'][1] = refine_flag

# Change isotropic size broadening for a given phase from the payload for a phase's parameterization:
proj.data['Phases'][phase_name]['Histograms'][hist.name]['Size'][0] = model # assume model is "isotropic" in this example
proj.data['Phases'][phase_name]['Histograms'][hist.name]['Size'][1][0] = value # set isotropic crystallite size value
proj.data['Phases'][phase_name]['Histograms'][hist.name]['Size'][2][0] = refine_flag # set isotropic size value refinement flag (if not None)

# Change isotropic mustrain broadening for a given phase from the payload for a phase's parameterization:
proj.data['Phases'][phase_name]['Histograms'][hist.name]['Mustrain'][0] = model # assume model is "isotropic" in this example
proj.data['Phases'][phase_name]['Histograms'][hist.name]['Mustrain'][1][0] = value # set isotropic mustrain value
proj.data['Phases'][phase_name]['Histograms'][hist.name]['Mustrain'][2][0] = refine_flag # set isotropic mustrain value refinement flag (if not None)

# Phase parameters
proj.data['Phases'][phase_name]['General']['Cell'] = [refine_cell, a, b, c, alpha, beta, gamma, volume]

Known GSAS-II Quirks

These inconsistencies exist in GSAS-II (as of 2025) and are documented in kicker.py comments:

Size/Strain Documentation Conflict:
- GSAS-II API does not document structure of dicts / lists well
- PowderLine uses correct extended format
Unit Cell Refinement:
- GSAS-II natively exposes a single refine flag for all 6 unit cell parameters; since schema 0.26 PowderLine honors per-parameter flags instead.
- See Unit Cell Refinement Semantics (Schema 0.26) for the full mechanism and symmetry-coupling rules.
Reflection List Headers:
- GSAS-II provides 15-column reflection arrays without headers
- PowderLine adds headers manually: h, k, l, multiplicity, d_spacing, 2theta, etc.

Peak Broadening Models

Schema Enhancement: Size and strain broadening support model branching with schema-level validation.

Size Broadening Models

PowderLine supports three size broadening models matching GSAS-II’s capabilities:

Model	Parameters	GSAS-II Structure	Status
isotropic	`isotropic_size`, `LG_eta`	`Size[0] = 'isotropic'` `Size[1][0]` = size `Size[1][2]` = LG_eta	✅ Implemented
uniaxial	`uniaxial_equatorial`, `uniaxial_axial`, `hkl_direction`, `LG_eta`	`Size[0] = 'uniaxial'` `Size[1][0:2]` = eq/ax sizes `Size[3]` = [h,k,l]	⚠️ NotImplementedError
ellipsoidal	`S11`, `S22`, `S33`, `S12`, `S13`, `S23`, `LG_eta`	`Size[0] = 'ellipsoidal'` `Size[4][0:6]` = tensor params	⚠️ NotImplementedError

Usage (isotropic only):

"size_broadening": {
  "model": "isotropic",
  "isotropic_size": [100.0, true, null, null],
  "LG_eta": [0.5, false, null, null]
}

Future models raise NotImplementedError at schema validation:

# This will fail during recipe validation:
"size_broadening": {
  "model": "uniaxial",  # Error: Not yet implemented
  ...
}

Strain Broadening Models

Strain broadening supports three models with increasing complexity:

Model	Parameters	GSAS-II Structure	Status
isotropic	`isotropic_strain`, `LG_eta`	`Mustrain[0] = 'isotropic'` `Mustrain[1][0]` = strain `Mustrain[1][2]` = LG_eta	✅ Implemented
uniaxial	`uniaxial_equatorial`, `uniaxial_axial`, `hkl_direction`, `LG_eta`	`Mustrain[0] = 'uniaxial'` `Mustrain[1][0:2]` = eq/ax strains `Mustrain[3]` = [h,k,l]	⚠️ NotImplementedError
generalized (Stephens)	Symmetry-dependent parameters	Complex Laue class-specific	⚠️ NotImplementedError (Phase 2)

Usage (isotropic only):

"strain_broadening": {
  "model": "isotropic",
  "isotropic_strain": [0.001, true, null, null],
  "LG_eta": [0.8, false, null, null]
}

Note on Stephens Model: The generalized strain model uses Stephens formalism where the number and meaning of parameters depend on the crystal’s Laue class (cubic, hexagonal, etc.). Implementation deferred to Phase 2 due to complexity.

Implementation Details

Model Branching (kicker.py):

def set_phase_size_broadening(proj, hist, phase_name, size_dict, print_info=False):
    """Set size broadening with model-specific branching."""
    model = size_dict.get('model', 'isotropic')  # Default for backward compat

    # Set model type in GSAS-II
    proj.data['Phases'][phase_name]['Histograms'][hist.name]['Size'][0] = model

    # Branch on model
    if model == 'isotropic':
        _set_isotropic_size_broadening(...)  # Current implementation
    elif model in ['uniaxial', 'ellipsoidal']:
        raise NotImplementedError(f"{model} size broadening not yet supported")

Schema Validation (schema.py):

class SizeBroadening(BaseModel):
    model: Literal["isotropic", "uniaxial", "ellipsoidal"] = "isotropic"
    # ... parameters for all models ...

    @model_validator(mode='after')
    def validate_model_implementation(self) -> Self:
        if self.model != 'isotropic':
            raise NotImplementedError(
                f"{self.model} size broadening not yet implemented. "
                "Use 'isotropic' model instead."
            )
        return self

Design Rationale:

Schema documents complete API (all models) even if not implemented
Clear error messages guide users to supported features
Foundation for Phase 2 implementation without breaking changes
Validation happens early (recipe parsing), not during refinement

Architecture & Design Decisions

Module Responsibilities

src/powderline/kicker.py (the refinement engine):

CLI entry point via argparse
Recipe loading, template detection, validation
GSAS-II project initialization
All parameter setting functions (background, phases, instrument, unit cell, etc.)
Refinement execution
Output generation (CSV reports, fit profiles)
Role: Complete workflow orchestration + implementation

src/powderline/schema.py:

Pydantic models matching JSON recipe structure
Nested validation: RecipeModel → InstrumentModel, PhaseModel, BackgroundModel
Custom validators for coefficient counts, list lengths, fit range
Role: Input validation and type safety

src/powderline/gsas_server.py / gsas_client.py:

Persistent FastAPI server keeping GSAS-II loaded in memory, and the HTTP client that talks to it (with in-process subprocess fallback)
Role: Execution backends for run()’s server/auto modes

src/powderline/topas/ + src/powderline/easydiff/ + src/powderline/engine.py:

Alternate translation engines — TOPAS v7 (recipe → .inp/.xye, tc.exe runner, results round-trip; standalone topas-kicker CLI) and easydiffraction (recipe → easydiffraction Project, lmfit fit) — plus the engine dispatcher behind powderline.run(recipe, output_dir, engine="gsasii"|"topas"|"easydiffraction"). Both alternate paths import zero GSAS-II; the gsasii branch is a transparent pass-through to kicker.run.
Role: Alternate refinement engines + engine dispatch

tests/:

test_example_LaB6_regression.py: End-to-end regression test with exact output matching
conftest.py: Pytest fixtures for paths and test data
Role: Quality assurance - detect refinement result changes

Monolithic Design Rationale: Phase 1 keeps all implementation in kicker.py intentionally:

Simplifies development during schema evolution
Easy to understand complete workflow in one file
Refactoring to modules planned for Phase 2 when API stabilizes
Comments saying “upstream” or “will be handled elsewhere” refer to future refactoring

Schema Design: Why `extra='allow'`?

Nearly all Pydantic models use ConfigDict(extra='allow') to permit fields not explicitly defined (the one exception is RefinementParameterModel, which uses extra='forbid'):

class RecipeModel(BaseModel):
    model_config = ConfigDict(extra='allow')

Rationale:

Schema evolution: Example-driven design means new fields emerge as examples are added
Backwards compatibility: Old recipes continue working when new fields are added
Flexibility: Users can add custom metadata without validation errors
Trade-off: Typos in field names won’t be caught (acceptable for Phase 1 iteration speed)

Future phases may tighten validation to extra='forbid' once schema stabilizes.

The “Upstream” Pattern

You’ll see several TODO markers like # TODO: validation will be handled upstream throughout kicker.py. This means:

Currently: Validation happens in helper functions (e.g., check coefficient count in set_chebyshev_background)
Future: Validation moves to Pydantic schema (separation of concerns)
Not implemented yet: Still evolving what validation logic belongs where

The pattern reflects Phase 1’s iterative approach - implement in helper functions first, refactor to schema later.

File-less Schema Architecture

PowderLine uses file-less JSON payloads. All data (XRD patterns, structures, instrument parameters) is embedded directly in JSON. File handling (loading XRD data from .chi files, parsing CIF structures, reading instrument parameters) occurs upstream of PowderLine.

Embedded Data Format:

XRD data: “xrd_data” contains “tth”, “Itth”, “Itth_weights” arrays
Structure: “structure” contains “space_group”, “unit_cell”, “atoms” dictionaries
Instrument: “instrument” contains “initialization” and optional “parameterization” dictionaries

This design enables PowderLine to focus on refinement logic while delegating file I/O to upstream tools.

Phase Name Resolution

In PowderLine, phase names come from the phase_name field in the embedded structure dictionary:

# Phase name is explicitly specified in structure
"phases": {
  "LaB6": {  # This becomes the phase identifier
    "structure": {
      "phase_name": "LaB6",  # Must match the key above
      "space_group": "P m -3 m",
      ...
    }
  }
}

The phase name is used consistently throughout:

Phase dictionary keys
Structure phase_name field (must match key)
Parameterization references
Output file naming

This ensures consistent phase identification throughout refinement and output generation.

Server Architecture (Three-Layer Execution Stack)

powderline.run() dispatches through three layers:

powderline.run() / CLI
        │  validates recipe, normalises DataFrames
        ▼
GSASClient.submit_simulation()
        │  routes: HTTP server  ──or──  in-process fallback
        ▼
run_refinement()
        │  single execution engine, returns JSON-serializable primitives
        ▼
GSAS-II (GSASIIscriptable)

_submit_via_subprocess is in-process, not an OS subprocess. Despite its name, this method calls run_refinement() directly in the current interpreter. The name has been retained for API stability; if crash-isolation is needed in a future phase it can be upgraded to multiprocessing without changing callers.

execution_mode vs method — two distinct concepts:

powderline.run() accepts execution_mode= ('auto' | 'server' | 'subprocess') as its public input parameter.
run_refinement() accepts method= as its internal parameter (values: 'server' | 'subprocess' | 'test').
Both run_refinement() and GSASClient include method as a result dict output key (result['method']) recording which execution path was used.
These are not a renamed pair — they serve different roles at different layers of the stack.

The FastAPI server (gsas_server.py) keeps GSAS-II imported in one persistent process, amortising the ~2.6 s import cost across all requests. Requests are handled synchronously (one-at-a-time) by design — concurrency is not needed for the current single-user CLI and programmatic-API use cases.

After modifying any Python source file (kicker.py, schema.py, or any other module under src/), restart the server so the running process loads the new code:

pixi run gsas-server restart

The server loads kicker.py at startup and caches it. Changes are invisible to the running server until it is restarted.

MP Pipeline Status

The Materials Project integration (mp_interface.py / mp_simulate.py / simulation_builder.py) targets the current mp-api client (pinned >=0.46,<1 in pixi.toml) and recipe schema 0.26.0.

Updated for mp-api 0.46.x (previously written against older client behavior):

Single summary fetch — get_structure() uses client.materials.summary.search(material_ids=[...], fields=[...]) (one request for structure + symmetry + energetics) instead of the legacy get_structure_by_material_id() convenience method.
Conventional-cell standardization — MP’s computed structure is often a primitive cell; it is now standardized via SpacegroupAnalyzer(...).get_conventional_standard_structure() so the emitted space-group symbol matches the emitted cell. (Previously a centered-lattice primitive cell was paired with the conventional H-M symbol — wrong phase for anything non-primitive; LaB6 only worked because its primitive and conventional cells coincide.) The symbol is derived from the standardized structure itself (spglib condensed style with screw-axis underscores stripped, e.g. Pm-3m, I41/amd), which kicker.py normalizes via G2spc.StandardizeSpcName.
MPID stringification — search results now return material_id as str; previously raw MPID objects crashed material_id.startswith('mp-') on the --formula path.
Polymorph selection — search_by_formula() returns results sorted by energy_above_hull (None last); the CLI lists all candidates and auto-selects the most stable one, printing the choice.
Partial occupancy support — sites are extracted per species (site.species.items()), carrying real occupancies into the recipe (AtomStructure.occupancy); disordered structures no longer crash on site.specie. get_structure() reports is_ordered.
Error taxonomy — MPInterfaceError base with MPAuthError, MPNotFoundError, MPConnectionError; the CLI prints targeted hints instead of a generic wrapped exception.
Metadata fixes — crystal_system now holds the actual crystal system (previously the space-group number, now under space_group_number); nelements now holds the element count (previously atom count, now under num_atoms); energy_above_hull and theoretical added.
Key handling — API key resolution is config file first, then the MP_API_KEY environment variable (ConfigLoader.get_mp_api_key()); MPInterface is a context manager that closes the HTTP session.
Server output-visibility guard — a GSAS-II server can be reachable over localhost yet have a divergent filesystem view (another node; a sandbox with a private /tmp), reporting success while its output files are invisible to the client. GSASClient verifies fit_profile.txt was freshly written client-side after a “successful” server run (stat-compare before/after — a stale file from a previous run into the same output dir does not count) and falls back to in-process execution (or returns a structured error when fallback is disabled). mp-simulate --no-server skips the server entirely, and its .chi export is built from the in-band fit_profile result data rather than re-reading the file. Tests: tests/test_gsas_client_visibility.py, tests/test_mp_simulate_cli.py. KI-12 tracks the follow-up that removes the shared-filesystem assumption altogether (server returns output artifacts in-band).
Strict config validation — SimulationRecipeBuilder validates the simulation_defaults block against its allowed shape: unknown keys at any level (typos, the retired flat layout) and invalid values (null from an empty YAML key, strings where numbers belong) raise a ValueError naming the offending key path; the CLI exits with that message. These previously passed RecipeModel validation (the instrument block is Any-typed passthrough) and failed opaquely inside GSAS-II.

Historical (schema 0.21 → 0.26 fixes, retained by the regression suite): payload wrapper, schema_name, isotropic_size/isotropic_strain renames, recipe= keyword for GSASClient.submit_simulation().

Regression guard: tests/test_mp_interface.py (mocked MPRester — real pymatgen Structures, no live API) and tests/test_mp_pipeline.py run as part of the standard test suite (pixi run test):

pixi run pytest tests/test_mp_interface.py tests/test_mp_pipeline.py -v

Limitation: mp_simulate.py requires a live Materials Project API key to fetch structures. The test suite uses mocks and a synthetic LaB6 fixture (mp_lab6_structure_data in conftest.py) so no API key is needed for testing.

Future: File-less Output Design

Currently run_refinement() always writes output files to output_dir. Two options are under consideration for making file output optional:

Option A — save_files flag (default False)

result = powderline.run(recipe, output_dir=None, save_files=False)
# Returns DataFrames; no files written

Simplest API change; easy to add without breaking existing calls
Requires output_dir to be optional throughout the stack

Option B — optional output_dir (default None)

result = powderline.run(recipe)          # no files
result = powderline.run(recipe, output_dir=Path("/tmp/out"))  # files + DataFrames

Slightly more discoverable (“no output_dir = no files”)
Requires output_dir=None guard at every write site in run_refinement()

Both options are backwards-compatible additions. This will be evaluated alongside the Phase 2 module refactoring work.

Development Workflow

Environment Setup

PowderLine uses pixi for reproducible conda environments:

# Install dependencies (first time)
pixi install

# Activate environment
pixi shell

# Run refinement
pixi run kicker examples/example_LaB6/input.json

# Run tests
pixi run test

# Validate recipe only (no refinement)
pixi run kicker --validate-only examples/example_LaB6/input.json

PYTHONPATH Configuration: The kicker task in pixi.toml sets PYTHONPATH=$PWD/src:$PYTHONPATH to enable from powderline.schema import RecipeModel imports.

Execution Modes

PowderLine supports three execution modes for running refinements:

1. Auto-Detect Mode (Default)

pixi run kicker input.json

Behavior: Tries server → auto-starts if needed → falls back to subprocess
Performance: 0.1-0.3s per simulation (if server available), 1-2s (if fallback)
Use case: Normal development workflow - “just works”
Output: Shows “🚀 server mode” or “🐢 subprocess mode” at end

2. Force Server Mode

pixi run kicker input.json --use-server

Behavior: Uses server only, fails if unavailable, no fallback
Performance: 0.1s per simulation (10x+ faster than subprocess)
Use case: Batch processing, CI/CD where performance matters
Auto-start: Attempts to start server automatically if not running
Error handling: Exits with clear error if server can’t be started

3. Force Subprocess Mode

pixi run kicker input.json --no-server

Behavior: Direct in-process execution using loaded GSAS-II libraries, skip server entirely
Performance: 1-2s per simulation (slower; GSAS-II imported on first use)
Use case: Debugging, HPC/Slurm batch jobs, environments where the server is not suitable
Output: Returns the same full structured result (DataFrames) as server mode

Managing the Server

Start server manually:

pixi run gsas-server start   # Starts in background
pixi run gsas-server status  # Check if running
pixi run gsas-server stop    # Shutdown server

Configure server port (if default port 19471 is in use):

# Set custom port before starting server
export POWDERLINE_SERVER_PORT=19472
pixi run gsas-server start

# Client will automatically detect the port from the powderline_gsas_server.port
# file in the platform temp directory (e.g. /tmp on Linux, %TEMP% on Windows)
pixi run kicker input.json  # No additional configuration needed

View server logs (powderline_gsas_server.log in the platform temp directory):

tail -f /tmp/powderline_gsas_server.log   # Linux; on Windows: %TEMP%\powderline_gsas_server.log

Server benefits:

GSAS-II libraries loaded once (2.6s startup amortized across requests)
Direct function calls using pre-loaded libraries (no subprocess overhead)
Persistent process (no repeated Python interpreter startup)
Ideal for batch simulations (10+ recipes)
Connection retry logic (3 attempts with exponential backoff) handles transient issues

Server caching behavior:

The server keeps Python modules loaded in memory for performance. This means:

✅ Production benefit: 4-6x speedup (first run ~12s, subsequent ~2-3s)
⚠️ Development caveat: Code changes require server restart to take effect

During active development:

# After making code changes to kicker.py or other modules:
pixi run gsas-server restart   # gracefully restart with updated code

# Next run will use the freshly started server with current code
pixi run kicker input.json

For deterministic integration tests:

# Restart the server first so tests run against current code
pixi run gsas-server restart
pixi run pytest tests/test_schema.py -q

See TROUBLESHOOTING.md (“Stale Server (Old Code Running)”) for details on stale server issues.

When to use each mode:

Auto-detect: Default for interactive use
–use-server: Batch processing, automation, performance-critical
–no-server: Debugging kicker.py, isolated testing, server issues

Performance Comparison (Post-Fix ✅)

Mode	LaB6 Simulation	DRX_33 Multi-phase	Typical Use
Server (–use-server)	0.1s	0.3s	Batch jobs
Auto-detect (default)	0.1-2s	0.3-3s	Interactive
Subprocess (–no-server)	1.1s	3s	Debugging

Speedup: Server mode is 10.67x faster than subprocess for LaB6 simulations.

Times measured on NSLS-II workstation post-fix

Adding a New Example

Examples drive schema evolution. To contribute:

Create directory: examples/example_N/
Add required files:
- input.json - complete recipe
- DESCRIPTION.md - what refinement demonstrates, data source, expected results
Run refinement: pixi run kicker examples/example_N/input.json
Commit outputs: output/ directory with .gpx, .lst, .csv files
Update schema if new fields needed
Add regression test if example becomes quality reference

See examples/CONTRIBUTING_EXAMPLES.md for detailed guidelines.

Running Tests

# All tests
pixi run test

# Specific test file
pixi run pytest tests/test_example_LaB6_regression.py

# With verbose output
pixi run pytest -v

# See stdout/stderr
pixi run pytest -s

Regression test strategy:

Exact matching of Rwp, unit cell parameters, peak counts
Detects any changes in GSAS-II refinement algorithm
Reference outputs committed to git for comparison
If test fails: inspect .lst file to determine if change is expected or bug

Note on test speed: All tests currently run as a single suite (pixi run test). Introducing @pytest.mark.slow / @pytest.mark.integration markers to allow skipping GSAS-II-dependent tests in CI is planned.

Template File Detection

PowderLine prevents users from accidentally running template files with three-level detection:

Path check: Filename/directory contains “template”
Required fields check: Missing payload.xrd_data, payload.instrument, or payload.phases
Placeholder text check: Fields contain "XXXX" or "TODO" or "REPLACE"

If detected, shows error with helpful message directing to real examples.

Key Gotchas & Implementation Notes

Output File Formats

After refinement, PowderLine generates (fixed filenames):

dummy.gpx: GSAS-II project file (can reopen in GUI)
dummy.bak0.gpx: Backup before refinement started
dummy.lst: Human-readable refinement log with Rwp, parameter tables, correlations
fit_profile.txt: 2θ, observed, calculated, background, difference intensities
<phase>_unit_cell_report.csv: Refined unit cell parameters per phase
<phase>_peak_list_report.csv: Reflection list (hkl, d-spacing, 2θ, intensities) per phase

For single peak fitting (SPF) mode:

single_peaks_report.txt: Peak fit results (position, width, intensity per peak)
peak_convergence_diagnostics.txt: Convergence warnings for aphysical parameters

Note: Output filenames are now fixed (dummy.*) regardless of sample name. This simplifies automation and testing by eliminating filename variability.

Parameter Correlation Warnings

GSAS-II may print warnings about highly correlated parameters (e.g., background coefficients):

** Warning: parameters 0 and 1 are 99.8% correlated **

This is normal for over-parameterized models. Common causes:

Too many Chebyshev background terms
Refining both instrument and sample broadening simultaneously
Limited 2θ range

Not necessarily a problem if Rwp is good and parameters are physically reasonable.

Chebyshev Background Coefficients

Indexing starts from 0th order (constant term):

"coefficients": [
  [100.0, true, null, null],    // 0th order (constant)
  [-50.0, true, null, null],    // 1st order (linear)
  [10.0, true, null, null]      // 2nd order (quadratic)
]

Number of coefficients must match num_coefficients field (validated in schema).

Single Peak Background Format

Each single peak is a pseudo-Voigt with 4 parameters:

"background": {
  "single_peaks": {
    "positions": [[35.5, false, null, null]],           // 2θ position
    "intensities": [[50.0, true, null, null]],          // Peak height
    "pv_gaussian_sigma": [[0.1, false, null, null]],    // Gaussian width (σ)
    "pv_lorentzian_gamma": [[0.05, false, null, null]]  // Lorentzian width (γ, HWHM)
  }
}

All lists must have same length (validated in schema). Useful for known impurity peaks that are part of the background.

Note: pv_mixing_eta parameter not currently implemented in schema.

Single Peak Fitting (Peak List Mode)

Earlier schemas introduced dedicated single peak fitting capabilities via the top-level single_peaks field (distinct from background.single_peaks), maintained in the current schema.

This enables refining individual peaks with direct control over position, intensity, and pseudo-Voigt width parameters—useful for:

Materials without crystal structure (amorphous, liquids, polymers)
Unknown phases requiring peak characterization before structure solution
Detailed peak shape analysis independent of structure
Le Bail fitting (fit peak intensities without structural model)

Key Features:

Independent width refinement: With use_instrument_profile: false, peak widths (σ, γ) refine directly rather than being derived from instrument parameters (U, V, W, X, Y, Z)
Coexistence: Both background.single_peaks and single_peaks can be used in same recipe—they populate different GSAS-II structures
Extended output: Generates single_peaks_report.txt with refined parameters plus calculated FWHM and integral breadths

JSON Structure:

// use_instrument_profile lives in refinement_controls.single_peak_fitting_mode
// (false = direct width refinement), not inside single_peaks:
"single_peaks": {
  "positions": [[21.3, true, null, null], [...]],     // 2θ positions
  "intensities": [[100.0, true, null, null], [...]],  // Peak intensities
  "pv_gaussian_sigma_sq": [[0.15, true, null, null], [...]], // Gaussian width variances (σ²)
  "pv_lorentzian_gamma": [[0.08, true, null, null], [...]] // Lorentzian widths (γ, HWHM)
}

Implementation Details:

GSAS-II Location: Peaks stored in proj.data[hist.name]['Peak List']['peaks'] (not Background)
Format: Each peak is [pos, pos_flag, intensity, int_flag, σ², σ²_flag, γ, γ_flag]
- Note: GSAS-II uses variance (σ²) internally, not σ
Instrument mode: Set via G2pwd.setPeakInstPrmMode(use_instrument_profile)
- False (default): Peak widths refined independently
- True: Widths constrained by instrument parameters (U, V, W, X, Y, Z)

Output File: single_peaks_report.txt

Tab-separated file with 10 columns per peak:

position_2theta  intensity  sigma  gamma  fwhm_gaussian  fwhm_lorentzian  fwhm_pseudovoigt  integral_breadth_gaussian  integral_breadth_lorentzian  integral_breadth_pseudovoigt

Calculated Values:

FWHM values:
- Gaussian: FWHM_G = 2√(2ln2) × σ ≈ 2.355σ
- Lorentzian: FWHM_L = 2γ
- Pseudo-Voigt: Thompson et al. (1987) approximation
Integral Breadths: Area under peak / peak height
- Gaussian: IB_G = FWHM_G × √(π/(4ln2)) ≈ 1.065 × FWHM_G
- Lorentzian: IB_L = π × γ = (π/2) × FWHM_L
- Pseudo-Voigt: Weighted combination using mixing parameter η

Verification: FWHM values are cross-checked against GSAS-II’s G2pwd.getgamFW(γ, σ) with warning if >1% difference.

Example: See examples/example_LaB6_singlepeakfit/ for complete demonstration.

Schema Classes:

SinglePeaksBackground: For background-associated peaks (background.single_peaks)
SinglePeaks: For Peak List fitting (single_peaks at top level) - includes use_instrument_profile

Error Handling Strategy

PowderLine fails fast with actionable error messages:

File loading errors: Check paths relative to JSON, file permissions
Template detection: Redirect to real examples
Validation errors: Show which field failed Pydantic validation
Histogram creation errors: Check XRD data format, instrument file compatibility
Refinement errors: Check parameter ranges, correlation warnings in .lst
Output errors: Check directory permissions, disk space

All errors exit with code 1 for automation compatibility.

Future Roadmap

Phase 2 (Next): Recipe generation API

Python functions to build JSON programmatically
Template system for common refinement types
Possible FastAPI wrapper for web service

Phase 3: Output parsing

Structured extraction of all refined parameters to JSON/CSV/HDF5
Quality metrics beyond Rwp (χ², GOF, R-Bragg)

Phase 4: Dash GUI for recipe building

Visual parameter selection
Live preview of refinement setup

Phase 5: Dash GUI for results visualization

Interactive plots of fit profiles
Parameter evolution across refinement cycles
Comparison between refinements

Contributing

Key principles for future work:

Add features when real examples need them (not speculatively)
Schema evolves based on examples
Maintain backwards compatibility when possible
Document GSAS-II quirks as you discover them
Add regression tests for new examples

Questions? Open an issue or discussion on GitHub.