fix: cap loader mislabels at-or-below-cap data pixel_scales

[Jammy2211]

Overview

Under PYAUTO_SMALL_DATASETS=1, cap_array_2d_for_small_datasets handles one case and silently drops the other: data larger than the 16x16 cap is cropped and rebuilt at SMALL_DATASETS_PIXEL_SCALES (0.6), but data already at-or-below the cap early-returns, keeping the caller's uncapped pixel_scales (0.1). A capped simulator writes its data at 0.6, so the loader mislabels the frame 6x — +/-0.8" instead of +/-4.8".

Off-centre galaxies then fall outside the mislabelled frame, their non-negative linear intensity solve correctly returns exactly 0.0, total_luminosity becomes 0, and min(5 * 0.5 * 0**0.6, 5.0) collapses a UniformPrior to lower == upper == 0.0. The resulting PriorException surfaces four steps downstream, in autolens_workspace/scripts/group/slam.py, naming neither the loader nor the pixel scale.

Reproduced on clean main in PyAutoHeart Workspace Smoke run 30790463134 (group/slam.py and group/slam.ipynb). This is the root cause; the fault is in PyAutoArray, not in any workspace script.

Plan

• Make the at-or-below-cap branch rebuild the array at the capped pixel scale when the cap is active, mirroring what the crop branch already does.
• Correct the docstring, which currently documents that branch as a deliberate no-op.
• Replace the two unit tests that assert the buggy behaviour as intended, and add one covering the rebuild.
• Validate end-to-end against group/slam.py, plus the four scripts whose committed 15x15 datasets change scale, plus one crop-path script.
• Report whether the two sibling scaling_relation/slam NEEDS_FIX parks now clear.

Detailed implementation plan

Affected Repositories

• PyAutoArray (primary, and the only repo edited)

Not edited: autolens_workspace and HowToLens appear in the originating prompt's Repos: header, but both legs are already done — the group/slam no_run line was removed by autolens_workspace PR #312 (a9b7ac1a), and HowToLens has no group/ scripts and no matching no_run entry. autolens_workspace is currently claimed by another task and is used here for validation runs only (writes land in gitignored dataset/ and output/).

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoArray	main	clean

No task in active.md claims PyAutoArray — no conflict.

Suggested branch: feature/small-datasets-loader-pixel-scales

Implementation Steps

1. autoarray/util/dataset_util.py, in cap_array_2d_for_small_datasets: replace the if h <= cap_h and w <= cap_w: return array_2d, pixel_scales early return (lines 41-42) with a rebuild returning Array2D.no_mask(values=array_2d.native.array, pixel_scales=SMALL_DATASETS_PIXEL_SCALES) and SMALL_DATASETS_PIXEL_SCALES. Shape is preserved — this branch must NOT crop.

   Returning a corrected scalar alone is not enough: the Array2D is constructed before the call and carries its own geometry. A first prototype that only fixed the scalar still failed.

   The PYAUTO_SMALL_DATASETS != "1" guard above is untouched, so non-smoke behaviour is unchanged.

2. Same file, docstring lines 15-18: currently promises "Returns (array_2d, pixel_scales) unchanged ... array_2d.shape_native is already at-or-below the cap (16, 16)". Rewrite to state that under an active cap, at-or-below-cap data is relabelled to 0.6, because such data was written by a capped simulator at that scale.

3. test_autoarray/util/test_dataset_util.py: test__env_set__shape_already_at_cap__returns_inputs_unchanged and test__env_set__shape_below_cap__returns_inputs_unchanged both assert result is array and pixel_scales == 0.08 — they encode the bug as intended behaviour. Rewrite both to assert the rebuild: result is not array, pixel_scales == SMALL_DATASETS_PIXEL_SCALES, shape preserved (not cropped), pixel values preserved.

   Leave test__env_unset__returns_inputs_unchanged and both crop-path tests as-is — they are the guard that this fix stays scoped to the cap-active path.

Validation

• pytest test_autoarray/.
• scripts/group/slam.py under the standard capped smoke env (PYAUTO_TEST_MODE=2, PYAUTO_SMALL_DATASETS=1, etc.), with dataset/ and output/ cleared, expecting exit 0 and all six SLaM searches running. Already confirmed green against a monkeypatched loader; re-run against the real edit.
• The four scripts with committed 15x15 datasets that will now receive 0.6 instead of 0.1: double_einstein_ring, mass_stellar_dark, scaling_relation, extra_and_scaling_galaxies. This is the declared behaviour change and the main risk.
• One crop-path script, to confirm that branch is untouched.
• Then check whether imaging/features/scaling_relation/slam and multi_galaxy/features/scaling_relation/slam now pass — scaling_relation is among the four affected datasets, so their 0.0-luminosity NEEDS_FIX park may clear. Un-parking them is a follow-up PR in autolens_workspace, not part of this one.

Explicitly out of scope

Flooring or guarding the collapsed prior, in group/slam.py or anywhere else. An off-frame profile solving to exactly zero intensity is the correct result of a non-negative solve. (A 2026-05 clamp on two sibling group SLaM scripts treated this symptom; that is not the pattern to extend.)

Known residual risk

The shape <= cap implies 0.6 inference is an inference, not a measurement. Every committed and generated imaging dataset checked is either cropped-and-relabelled to 0.6 or written by a capped run at 0.6, but a genuinely small real-scale dataset added later would be mislabelled. Follow-up prompt on simulators recording their own scale to be filed separately.

Key Files

• autoarray/util/dataset_util.py — cap_array_2d_for_small_datasets, the fix and its docstring
• test_autoarray/util/test_dataset_util.py — the two tests asserting the bug, plus the new rebuild test

Original Prompt

Click to expand starting prompt

PYAUTO_SMALL_DATASETS loader keeps uncapped pixel_scales for at-or-below-cap data

Type: bug
Target: PyAutoArray
Repos:

• PyAutoArray
• autolens_workspace
• HowToLens
  Difficulty: small
  Autonomy: supervised
  Priority: normal
  Status: formalised

PYAUTO_SMALL_DATASETS loader keeps uncapped pixel_scales for at-or-below-cap data.

Root cause of the group/slam PriorException (supersedes draft/bug/autolens/group_slam_priorexception_limits.md). The fault is in PyAutoArray, not in any workspace script.

autoarray/util/dataset_util.py cap_array_2d_for_small_datasets handles one case and silently drops the other:

• data LARGER than the 16x16 cap -> crops it AND rebuilds it at SMALL_DATASETS_PIXEL_SCALES (0.6). Correct.
• data already AT-OR-BELOW the cap (because a capped simulator just wrote it at 0.6) -> early-returns, keeping the callers uncapped pixel_scales (0.1). Wrong.

Consequence: the frame is mislabelled 6x (plus/minus 0.8 arcsec instead of plus/minus 4.8 arcsec). Off-centre galaxies fall outside it, their non-negative linear intensity solve correctly returns exactly 0.0, total_luminosity becomes 0, and min(50.50**0.6, 5.0) collapses a UniformPrior to lower==upper==0.0. The PriorException is four steps downstream of the fault.

FIX (about 4 lines): in the at-or-below-cap branch, when the cap is active, rebuild the Array2D at SMALL_DATASETS_PIXEL_SCALES and return that scale, mirroring what the crop branch already does. Returning a corrected scalar alone is NOT enough: the Array2D is constructed before the call and carries its own geometry. A first prototype that only fixed the scalar still failed. Also update the docstring, which currently documents only the crop case and calls the early return a no-op.

NO workspace script changes are needed. pixel_scales=0.1 is a TRUE statement about the dataset in normal operation and belongs in a tutorial script; the cap silently invalidates it and the loader must correct it.

PROVEN on clean main, standard capped smoke env (PYAUTO_TEST_MODE=2, PYAUTO_SMALL_DATASETS=1), fresh dataset:

• unmodified scripts/group/slam.py + patched loader -> EXIT 0, all six searches ran
• unmodified script, unpatched loader -> PriorException at slam.py:307
• script with pixel_scale hardcoded to 0.6, unpatched loader -> EXIT 0 (confirms the scale, not the science, was the problem)

Evidence the capped data really is 0.6 arcsec/px: the simulated 16x16 image has clumps at (+3.6,+2.4) and (-4.8,-4.8) arcsec under 0.6, matching the declared extra-galaxy centres (3.5,2.5) and (-4.4,-5.0). Under 0.1 they would be at (0.55,0.45) - nowhere near.

Safety of the >=cap-implies-0.6 inference: every committed and generated imaging dataset checked is either cropped-and-relabelled to 0.6 or written by a capped run at 0.6. It is still an inference; a genuinely small real-scale dataset added later would be mislabelled. See the follow-up prompt on simulators recording their own scale.

BEHAVIOUR CHANGE TO VALIDATE: four datasets have committed 15x15 files (double_einstein_ring, mass_stellar_dark, scaling_relation, extra_and_scaling_galaxies). Scripts loading them will now receive 0.6 instead of 0.1. That is a correction (the data genuinely is 0.6) but must be spot-checked.

ALSO IN SCOPE: remove the group/slam NEEDS_FIX line from autolens_workspace/config/build/no_run.yaml, and the dead one from HowToLens/config/build/no_run.yaml (HowToLens has no group/ scripts at all).

VALIDATION PLAN: pytest test_autoarray/ (add an at-or-below-cap test case); clean-dataset and clean-output run of scripts/group/slam.py under the standard capped smoke env expecting exit 0; the four committed-15x15 scripts; one crop-path script to confirm that branch is untouched.

DEAD ENDS ALREADY RULED OUT, do not redo: (1) the 132-vs-35 should_simulate split is NOT drift - commit 0f294fc70 scoped that migration to smoke-tested scripts deliberately, and since dataset/ is gitignored, exists() and should_simulate are equivalent on a clean CI checkout. (2) An FOV-preserving rewrite of the cap is NOT needed for this bug. (3) Flooring or guarding the collapsed prior is wrong - an off-frame profile solving to exactly zero intensity is the correct result of a non-negative solve.

[Jammy2211]

Library PR Created

PR: #431 (pending-release)

The at-or-below-cap branch of cap_array_2d_for_small_datasets now rebuilds the Array2D at SMALL_DATASETS_PIXEL_SCALES, mirroring the crop branch. Shape preserved; crop branch untouched; no behaviour change when the cap is unset.

Validation: pytest test_autoarray/ 929 passed · group/slam.py exit 0 with all six SLaM stages genuinely run · crop path byte-identical on 209x209 cosmos_web_ring · imaging/start_here.py (uncapped) exit 0.

Two corrections to the plan on this issue

1. The declared behaviour-change risk is not real. The plan flagged four datasets with committed 15x15 files (double_einstein_ring, mass_stellar_dark, scaling_relation, extra_and_scaling_galaxies) as needing spot-checks. Measured on today's main: of 31 committed dataset FITS, exactly one is at-or-below the cap — dataset/imaging/slacs1430+4105/psf.fits (11x11) — and PSFs never route through the capper (imaging/dataset.py caps data and noise_map only). double_einstein_ring and extra_and_scaling_galaxies do not exist as scripts or datasets; the other two use generated, gitignored data. Zero committed datasets change scale.

2. The two parked siblings split. imaging/features/scaling_relation/slam now exits 0 (6 real searches) — its NEEDS_FIX park can be removed in a separate autolens_workspace PR. multi_galaxy/features/scaling_relation/slam clears the 0.0-luminosity cause but then hits a separate latent bug: slam.py:863 derives image_half_width from the script's hardcoded pixel_scale (0.1) while the mask uses dataset_full.pixel_scales (now correctly 0.6), giving an empty mask and a zero-size reduction in ConvolverState. Not a regression — unpatched main fails earlier on the documented luminosity error. It stays parked; filed separately.

Gate

Heart was RED at ship time. This PR fixes part of the YELLOW workspace validation not passing (cloud#30790463134) reason, not any RED reason; the three RED reasons (PyAutoFit: 1 commit(s) behind origin, install verification FAILED (testpypi; checks D), release validation FAILED (stage integrate)) are unrelated to this change. Shipped to PR-open under an explicitly human-authorized corrective-PR exception. Merge and release remain human.

[Jammy2211]

Shipped

Merged: #431 → 17885f38 on main.

Post-merge verification against the canonical install (not the worktree): scripts/group/slam.py under the capped smoke profile, cleared dataset + output → exit 0, 6 searches genuinely run, 0 cached resumes. CI green on 3.12 and 3.13.

Completion record: PyAutoMind/complete/2026/08/small-datasets-loader-pixel-scales.md. Worktree and branch cleaned up.

Follow-ups filed (not part of this issue)

• draft/bug/autolens_workspace/script_local_pixel_scale_vs_dataset_pixel_scales.md — the latent bug this fix exposed in multi_galaxy/features/scaling_relation/slam. Scoped as a sweep, not a one-line fix: any script mixing a hardcoded pixel_scale literal with dataset-derived geometry has it, and it is invisible in uncapped runs.
• draft/maintenance/workspaces/unpark_imaging_scaling_relation_slam.md — now unblocked by this merge; removes the NEEDS_FIX park for imaging/features/scaling_relation/slam, which passes with this fix. Explicitly does not un-park the multi_galaxy sibling.

Note for whoever picks those up: the originating prompt's Repos: header and its "four datasets have committed 15x15 files" risk statement were both stale/false when re-measured — treat its remaining unverified claims (e.g. the proposed simulator-scale follow-up) with the same suspicion.