Pipeline tests

tests/test_tabascal_pipeline.py runs the whole of run_tabascal end to end and compares the result against recorded reference values. These are the tests that catch a change in what tabascal actually infers, as opposed to the unit tests, which check individual functions.

Each case runs run_tabascal.py in a subprocess against a simulation downloaded from the epfl-radio-astro/rfi-simulations HuggingFace dataset, then parses the run’s stdout.

pixi run -e dev pytest tests/test_tabascal_pipeline.py            # double precision
pixi run -e dev pytest tests/test_tabascal_pipeline.py --x64 false # single precision

--x64 selects the precision requested of the subprocess (it is written into the generated config as model.precision). Cases marked requires_double — the SGP4/phase trajectory components, which need fp64 for orbit propagation — are skipped under --x64 false.

What is asserted

Each case is a PipelineTestConfig carrying its references:

  • chi2_ref — the reduced chi² at the optimisation point. A scalar is asserted at 1% relative tolerance; a (lo, hi) tuple is asserted as inclusive bounds. One value covers both precisions — see “Which architecture to record on” below.

  • metrics_ref — optional truth-based metrics at the opt point, as {quantity: {metric: ref}} where quantity is ast, rfi or gains. Only the metrics listed are checked, so a case can assert just what it cares about. Each value takes the same scalar-or-(lo, hi) form as chi2_ref.

The truth metrics come from print_truth_metrics (tabascal/tab_tools.py). The two normally worth asserting are:

  • NRMSE(noise) — the residual against the thermal-noise floor. This is the science-meaningful yardstick (below 1 means the residual is sub-noise) and the most architecture-stable normalisation. Lower is better.

  • bias_significance — the coherent component of the error, in sigma:

    bias_significance = sqrt(2 * N_eff) * |ME| / RMSE

    where |ME| is the magnitude of the mean error and N_eff is the correlation-deflated effective sample size (_effective_sample_size). It is a “no significant coherent bias” guard — it catches e.g. RFI leaking systematically into the recovered sky — not a tight value to match.

Reading bias_significance correctly

bias_significance is a detectability measure, not an error magnitude, and it does not move with NRMSE(noise). Note that RMSE is in the denominator and N_eff is inside the square root, so the significance can rise while the fit strictly improves.

This happened when the astronomical fringe rate was corrected (RiemannVis case, double precision):

before

after

ast RMSE

1.989e-01

1.700e-01

−14.5% (better)

ast |ME| (bias)

1.205e-02

1.217e-02

+1.0% (flat)

ast N_eff

54

113

+109%

ast bias_significance

0.6

1.1

+83%

The total error dropped 14.5% and the coherent bias barely moved, yet the significance nearly doubled. Both surviving factors push the same way: N_eff more than doubled, so sqrt(2 * N_eff) grew ~44%, and the shrinking RMSE in the denominator added another ~17%. The same bias measured over more effectively-independent samples, against a smaller residual, is simply resolved more sharply. A rising significance is only a problem if |ME| itself is growing, or if it approaches the bound — check the absolute bias before concluding anything from the sigma value alone.

Beware of rounding too: the significance is printed to one decimal, so the rfi value in that same change reads as a jump from 0.1 to 0.2 when the underlying values are 0.106 → 0.232.

Re-recording the references

When a change intentionally alters what the model infers, the recorded values no longer describe it and must be re-recorded. Do not widen the tolerances to make the existing numbers fit — that silently discards the regression signal.

Run with --record-refs to skip the assertions and print the measured values instead. This is the point of the flag: you do not have to make the assertions pass, or hand-edit the file, just to find out what the new values are.

pixi run -e dev pytest tests/test_tabascal_pipeline.py --record-refs -s
pixi run -e dev pytest tests/test_tabascal_pipeline.py --record-refs -s --x64 false

-s is required — the values are printed, and pytest captures stdout without it. For one case:

pixi run -e dev pytest "tests/test_tabascal_pipeline.py::test_pipeline[GPGains]" \
    --record-refs -s

Output per case, shaped to be read straight across into the config literal and its arch table:

--- measured: GPGains [double] ---
    chi2_ref=0.8874142592424018,
    ast: {'NRMSE(noise)': 0.2617, 'NRMSE(signal)': 0.09797, 'RMSE': 0.17, 'bias_significance': 1.1}
    rfi: {'NRMSE(noise)': 0.4279, 'NRMSE(signal)': 0.0253, 'RMSE': 0.278, 'bias_significance': 0.2}
    gains: {'NRMSE(signal)': 0.0005328, 'RMSE': 0.0005328, 'bias_significance': 1.1}

Then:

  1. Paste the chi2_ref scalar. Record both precisions and check they agree before pasting either — see below.

  2. Move any metrics_ref bound whose measured value now falls outside it, keeping roughly the existing relative width. Leave the bounds that still hold — a change that only moves some of them is more credible than one that moves all.

  3. Update the arch table comment in that case, so the next person can see what was measured where.

  4. Record in the commit message why the values moved, with the before/after numbers. A reference change is indistinguishable from a regression without it.

Filling in the GPU and x86 arch rows

The dev environment is CPU-only; the CUDA jaxlib lives in cuda12-dev. To fill in a GPU row, and to pin the CPU row to the CPU backend on a machine that also has a GPU:

# GPU rows
pixi run -e cuda12-dev pytest tests/test_tabascal_pipeline.py::test_pipeline \
    --record-refs -s
pixi run -e cuda12-dev pytest tests/test_tabascal_pipeline.py::test_pipeline \
    --record-refs -s --x64 false

# CPU rows on a GPU machine — otherwise jax picks the GPU
JAX_PLATFORMS=cpu pixi run -e dev pytest tests/test_tabascal_pipeline.py::test_pipeline \
    --record-refs -s

Target ::test_pipeline explicitly rather than -k test_pipeline: the latter also selects test_pipeline_sharded_equivalence and test_pipeline_multiprocess, which --record-refs does not apply to and which take several minutes.

conftest.py pins CUDA_VISIBLE_DEVICES=0, so a GPU run is single-device and does not exercise the sharded solve. Three cases are requires_double and are skipped under --x64 false, so a single-precision run records three cases, not six.

Which architecture to record on

Any of them. Both precisions and all three architectures currently agree well inside the 1% tolerance, which is why each case carries one scalar rather than a per-precision split.

Measured across ARM (Apple silicon), x86 CPU and an NVIDIA GPU, in both precisions:

  • Double is architecture-stable to 5.5e-8 relative in the worst case (the two SGP4 orbit cases, where the propagation amplifies rounding; the non-orbit cases agree to ~1.7e-10), and the printed truth metrics agree to every digit shown.

  • fp32 agrees with fp64 to 2.4e-5 relative on chi2, and to the printed precision on every truth metric. The offset is the same 2.4e-5 on ARM, x86 and CUDA alike — a precision effect, not an architecture one.

So a reference recorded on any platform in either precision is canonical for all of them. Prefer the CI/x86 value when you have it, since that is what gates the PR.

This was not true of the pre-#103 real-space GP model, where fp32 convergence rate differed markedly between architectures (ARM reached chi2 ~0.92 in 100 iterations while x86 was still at ~1.13, and the rfi residual nearly doubled between GPU and x86). That is what the per-precision (lo, hi) bounds existed to absorb. The Fourier model removed the need for them. If you reintroduce a component whose fp32 convergence is architecture-dependent, that split has to come back — measure both precisions on at least two architectures before collapsing a reference to a scalar.

If a double value differs between two machines by much more than 1e-6, suspect a stale reference rather than an architecture difference. That is what the earlier “ARM runs ~0.7% high” comments in this file turned out to be: the references had drifted while staying inside the 1% tolerance, so nothing failed until a real change pushed them over.

Adding a case

Append a pytest.param(PipelineTestConfig(...), id="...") to the relevant list (trajectory_configs, rfi_vis_configs, gains_configs, …); they are concatenated into all_configs. Give it an explicit id — that id is the test name and the handle for -k. Record its references with --record-refs as above.