Add an engine or a capability#

Goal: add a refinement engine behind a new gateway, or teach an existing gateway a new recipe field.

You need: a development setup (Contributing), the engine installed somewhere you can run it, and the layout in Architecture.

The gateway contract#

A gateway is a package that powderline.registry finds through the powderline.gateways entry-point group. Its factory returns a frozen registry.Gateway:

from powderline.registry import Gateway

def get_gateway() -> Gateway:
    return Gateway(name="myengine", capabilities=capabilities, validate=validate, run=run)

Member	Contract
`name`	The registry name, which is also the `schema_name` prefix (`myengine.rietveld`)
`capabilities()`	A dict with at least `name`, `workflows` (the schema names), `engine_version_spec` (a PEP 440 specifier) and the schema `DECLARATIONS`
`validate(recipe, *, verbose=False)`	Schema layer only. Takes a dict or a validated model and returns the validated model. Exactly these keywords
`run(recipe, output_dir, , verbose=False, validate_only=False, allow_unsupported_engine_version=False, *own_options)`	Runtime layer. Validates, checks the engine version, imports the engine lazily, runs it, and returns the standardized result dict

The schema layer must import no engine. powderline.validate() runs it in the default environment, where no engine is installed.

An in-repo gateway lives in src/powderline/gateways/<engine>/. An out-of-tree gateway is a separate distribution that declares the same entry point. For tests and development, registry.register_gateway(name, factory) registers a gateway in code without an entry point.

Add an engine#

Write the schema module (schema.py, engine-free).
- Subclass core’s phase block in the engine’s parameter shape: class MyPhase(Phase[BoundedRefinableParameter]), or Phase[RefinableParameter] if the engine has no usable bounds. Add the engine’s phase fields with their own validator names and 'after' validators only.
- Model the instrument, background and controls in the engine’s own terms and units. Every quantity carries json_schema_extra={"unit": ...}, which the field tables read. Keep extra='forbid'. A block that may be left out is left out, never null.
- Add no default that silently replaces an engine behavior. Either the value is required, or the default is documented and reported with a structured warning.
- Write the recipe models as subclasses of schema_core.EngineRecipe, with schema_name narrowed to a Literal and two class variables: gateway_name and schema_declarations = DECLARATIONS.
- Declare the versions exactly:
```
DECLARATIONS = compat.schema_declarations(
    engine_schema_version="0.1.0",
    supported_engine_schemas="==0.1.0",
    requires_core_schema="==0.1.0",
)
```
  EngineRecipe runs the check. A declaration is widened explicitly later, never written as a range (Policies).
- Export SCHEMA_NAMES and validate_recipe = schema_core.recipe_validator({<schema_name>: <model>, ...}, engine="<engine>", title="<Engine>Recipe"). It accepts a dict or a validated model and reports an unknown schema_name at schema_name.
Write every engine rule as a validator. Refuse what the engine cannot express, and what it would silently alter (a floor, a limit, a snapped value), naming the value to write. Use core’s building blocks rather than copies:
- absent_not_null and omit_absent for blocks that may be left out;
- unique_phase_names, fit_window, fit_limits_on_data, fit_range_problems, outside_window_problems and refined_flag_problems for the payload rules;
- refined_flags, custom_error and raise_problems to report every problem at once.
Register every warning code the engine emits in powderline.exceptions.WARNING_CODES (owner, level), and add its row to the warnings table in docs/reference/messages.md. tests/test_warning_codes.py holds the registry, the code and the table in agreement.
Write the gateway module (gateway.py) with get_gateway(), capabilities(), validate(), run() and ENGINE_VERSION_SPEC. If the engine is a pixi dependency, add ENV and INSTALL_COMMAND. In run():
- With validate_only=True, return registry.validate_only_summary(...) and skip the engine version check.
- Otherwise call registry.check_engine_version(...). Raise EngineNotAvailableError with env and install_command when the engine is missing.
- Define installed_engine_version() in the module that defines run. powderline.run() calls it to check that the engine is importable before it touches the output directory.
- Import the engine lazily, inside run().
Return the standard result. Use the same keys and types as the other gateways (see Outputs).
- Compute the statistics with powderline.fitstats over the window’s weighted points, with P read from the engine after the run. Put the engine’s own values under engine_details, with parameters_requested and parameters_varied.
- Hand powderline.roundtrip the refined values by recipe path ({path: (value, esd)} in recipe units), plus the flagged paths the engine did not vary and those that ended at a limit. roundtrip.build_output and roundtrip.attach build the output recipe, the dispositions and their files.
- Write the report files with powderline.reports. Keep the engine’s native files.
Never return a failed result. Raise EngineExecutionError when the engine ran and failed, or ExecutionBackendError when it did not run, with a reason from powderline.results.REASONS. A non-finite calculated pattern is a failure (diverged). results.raise_if_failed raises on a failed envelope you received.

Declare the entry point in pyproject.toml, then run pixi reinstall powderline --all:

[project.entry-points."powderline.gateways"]
myengine = "powderline.gateways.myengine.gateway:get_gateway"

Add a pixi feature and environment for the engine’s dependencies, so the default environment stays engine-free. Pin the engine exactly and keep the pin inside ENGINE_VERSION_SPEC. Put the engine’s test task in its feature.

Check: pixi run powderline <recipe.json> --validate-only works in the default environment, pixi run powderline --doctor lists the gateway, and a run in the engine’s environment returns status ok.

Tests to write#

Follow the pattern of the existing gateways. Mark each test with exactly one suite marker; tests that need the engine get the engine’s marker, so pixi run test stays green without it.

Schema tests (tests/test_<engine>_schema.py): every rule and every refusal, with the error path.
Phase canary: add the engine’s phase model to ENGINE_PHASES in tests/test_phase_canary.py. It runs core’s must-fail recipes through it.
Import block: prove that the schema layer and import powderline load no engine module.
Space groups: compare every setting the engine accepts with gemmi (orbits or operators).
Fixtures and goldens: hand-derived native recipes with a README of the derivations, and the engine’s output recorded with its version, under tests/data/<engine>/native/.
Result contract: the same keys and types as the other engines.
Pins: tests/test_gateways.py checks that the pixi pin lies inside ENGINE_VERSION_SPEC.

The goldens are the references. A change in what the engine is asked to do (the input file, the project, the input dictionary) is proved against them.

Docs to update#

Add the recipe models to _recipe_models() in scripts/gen_schema_field_tables.py and regenerate the field tables. Add a section for the new tables to Fields. The support matrix regenerates itself.
Write the engine page under docs/engines/ and add a column to the feature table on Engines.
Add a catalog entry in docs/catalog/engine_behaviour.yaml for every engine behavior a user can meet, with PowderLine’s policy. Regenerate with scripts/gen_catalog.py. Anything the engine does silently is written up before the gateway ships.
Add the engine schema’s entry to Versions and a CHANGELOG bullet.

Add a capability to an existing engine#

The same discipline, smaller.

Find the ground truth. Read and run the engine to see what it does with the quantity. If the engine’s behavior is unknown, ask the maintainers rather than infer it.
Add the field to the engine’s schema.py, with its unit, validators and refusals. A structural quantity shared by every engine goes into schema_core as a building block, and each engine’s phase model adds it in its own parameter shape. March–Dollase preferred orientation is built this way (schema_core.MarchDollase).
Consume it in the runtime layer from the validated model. For gsasii, put the read in a non-frozen module. A read inside a frozen module is a behavior change and needs the maintainers’ consent (Policies). Map the refined parameter back to its recipe path (gsasii paths.py) so it reaches the output recipe.
Prove it. Add a fixture and a golden that show the value reaches the engine and does what the recipe says. Prefer a formula check on a simulated pattern as the gate, and an example as the demonstration.
Add an example if it is a real case: examples/<case>/ with its DESCRIPTION.md and, for gsasii, the committed output.
Record it. Regenerate the field tables, add catalog entries for anything the engine does silently, and update Versions and the CHANGELOG.

If the new field changes what an existing recipe means, it is a deliberate change: a schema version bump, a CHANGELOG entry, and a note for users (Policies).