fix: HST DQ-bits dial — final_bits="0" rejects every DQ-flagged pixel

[Jammy2211]

Overview

autoreduce/drizzle/combine.py:drizzle_kwargs_for never sets final_bits / driz_sep_bits, and no adapter sets them either — final_bits, driz_sep_bits and mdriztab appear nowhere in the repo. Every reduction therefore inherits drizzlepac's package default final_bits = "0" (drizzlepac/pars/astrodrizzle.cfg:101), which treats no DQ bit as good and rejects every flagged pixel, including the hot, warm and blob pixels that calacs/calwf3 have already corrected.

This produces structured defects on both HST instruments we support: zero-coverage holes on WFC3/IR (DQ 512 blobs, PJ011646, closed validation issue #25) and high-noise stripes through lens cores on ACS/WFC F814W SLACS-gold, which the legacy SLACS reductions do not have. STScI's own MDRIZTAB reference files — read directly from our CRDS cache, see below — pass most of these bits as usable, and do so as a function of exposure count.

Two separable deliverables: the bits dial (leg 1), and a local weight-deficit guard that can actually detect this defect class (leg 2). Expect two PRs.

Plan

• Control test first. Re-drizzle one striped SLACS target at the current 0 and at the MDRIZTAB value, diff the weight and noise maps. If the stripes do not move, the cause is elsewhere (exposure count, dither geometry, genuine bad columns) and leg 1 is not the fix — report that rather than shipping the dial anyway.
• Leg 1 — bits dial. Expose DQ bits as an adapter default plus a TargetSpec override, keyed on exposure count per the MDRIZTAB table. Record the value and its provenance in reduction.json. Unit-test drizzle_kwargs_for across the N boundaries. Document in docs/design/wfc3.md and docs/design/hst_acs_pipeline.md.
• Leg 2 — local weight-deficit guard. Both current guards only see total coverage loss, so the ACS stripe class passes silently. Add a per-column / local-vs-median weight-ratio diagnostic that fires inside protect_radius_arcsec. Landing this first gives the control test an objective pass/fail instead of an eyeball judgement.
• Do not enable mdriztab=True wholesale — it would silently revert the deliberate lensing deviations in stage 3 (0.05″/pix, pixfrac 0.8, north-up).
• Mask-only policy unchanged: bad pixels are masked, never inpainted.

Verified evidence — STScI's own MDRIZTAB values

Read out of the cached CRDS reference files under scripts/cache/crds/, so these are facts about the pipeline STScI ships, not recollection:

Detector	MDRIZTAB file	numimages	driz_sep_bits	final_bits
ACS/WFC	acs/37g1550cj_mdz.fits	1	65535	65535
ACS/WFC	"	≥2	336	336
WFC3/UVIS	wfc3/2ck18260i_mdz.fits	1	65535	65535
WFC3/UVIS	"	≥2	336	336
WFC3/IR	wfc3/3562021pi_mdz.fits	1	65535	65535
WFC3/IR	"	2	65535	528
WFC3/IR	"	≥4	528	528

1. 336 = 16 + 64 + 256 — hot, warm and saturated pixels are usable for multi-exposure ACS/UVIS because the dark subtraction already corrects them. We reject all three.
2. 528 = 512 + 16 for WFC3/IR — STScI passes exactly the blob bit that caused the PJ011646 hole. PJ011646 had 5 exposures, landing on the numimages ≥ 4 row, so under standard practice that hole could not have occurred.
3. The value is exposure-count dependent, and single-exposure data uses 65535 — every bit good. That is the SLACS SNAP regime, and an independent explanation for why the legacy SLACS maps look clean: with one exposure there is nothing to fill a masked pixel with, so the standard recipe keeps flagged pixels rather than punching holes. Any fix must be N-aware; a flat constant is wrong.

Detailed implementation plan

Affected Repositories

• PyAutoReduce (primary)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoReduce	main	clean

Suggested branch: feature/hst-dq-bits-dial

The two symptoms (one root cause)

WFC3/IR — zero-coverage holes. On snapshot data with tiny dithers, detector-fixed DQ 512 (blob) regions produce structured zero-coverage holes that the packaging guard correctly refuses. PJ011646 (program 14653, F160W, 5 exposures, 2–6 px dithers) failed with a single 123-px hole at r = 5.3″, outside the 3.8″ science mask, DQ 512 at the same detector pixels in all five exposures (issue #25). Aris's trusted reduction of the same data has no hole. reduce_pj011646.py carries a documented monkeypatch workaround until this ships.

ACS/WFC — high-noise stripes through lens cores. F814W SLACS-gold noise maps show stripes of elevated noise, some straight through the deflector cores, absent from the legacy SLACS reductions (user report 2026-08-04). Mechanism: rejecting hot (16) and warm (64) pixels on top of the genuine bad columns (4, 128) removes pixels that are partly column-organised on an aged ACS CCD (trap columns, CTE trails), so IVM weight along those columns is reduced rather than zeroed. Noise rises by sqrt(N/(N-1)) per lost exposure — ×1.41 at N=2, ×1.22 at N=3, ×1.15 at N=4, ×1.08 at N=7 — so few-exposure targets stripe visibly and many-exposure ones do not.

Implementation Steps

Leg 2 first (the detector):

1. autoreduce/noise/rms.py — mask_isolated_bad_pixels currently tests bad = ~isfinite(noise) | (noise <= 0). A stripe with reduced-but-nonzero IVM weight is finite and positive, so it passes every check, including the protect_radius_arcsec (1.5″) protection whose entire purpose is to guarantee "the lens itself must be clean", and the structured-defect clustering check that would otherwise refuse a column. Add a local weight/noise-deficit check that fires on finite-but-degraded coverage inside the protect radius.
2. autoreduce/drizzle/diagnostics.py — weight_uniformity is a global RMS/median over the whole cutout against a 0.2 limit; the slacs0008 spike measured 0.066 and a handful of degraded columns cannot move it. Add a local (per-column and/or local-vs-median) statistic beside it and record both in reduction.json.
3. Unit-test both against a synthetic weight map with one degraded column through the centre — this is the regression that proves the guard now sees the defect.

Leg 1 (the fix):

4. autoreduce/instruments/acs_wfc.py, wfc3_ir.py (and wfc3_uvis if present) — add DQ bits to default_drizzle_kwargs, keyed on exposure count per the MDRIZTAB table.
5. autoreduce/target.py — TargetSpec override so the bits are a first-class user dial, consistent with how final_pixfrac/final_kernel are treated in stage 3.
6. autoreduce/drizzle/combine.py — drizzle_kwargs_for already receives n_exposures, so the N-keyed lookup lands there naturally.
7. Record chosen bits + provenance (adapter default vs user override) in reduction.json so existing datasets stay re-derivable.
8. Unit-test drizzle_kwargs_for across the N=1/2/4 boundaries per adapter — numpy/astropy only, no drizzlepac in test_autoreduce/.
9. Document the choice and the blob/hot-pixel physics in docs/design/wfc3.md and docs/design/hst_acs_pipeline.md.

Trap to avoid: do not simply set mdriztab=True. MDRIZTAB supplies the whole parameter set — final_scale, final_pixfrac, final_kernel, final_rot — and hst_acs_pipeline.md stage 3 deviates from those deliberately and with justification. Verify the precedence rules before trusting it; the safer route is to read the bits columns and keep our own explicit kwargs.

Key Files

• autoreduce/drizzle/combine.py — drizzle_kwargs_for, the single fix site; already takes n_exposures
• autoreduce/instruments/acs_wfc.py, autoreduce/instruments/wfc3_ir.py — default_drizzle_kwargs, currently bits-free
• autoreduce/target.py — TargetSpec, for the user override
• autoreduce/noise/rms.py — mask_isolated_bad_pixels, the guard blind to finite-but-degraded noise
• autoreduce/drizzle/diagnostics.py — weight_uniformity, the global-only statistic
• docs/design/hst_acs_pipeline.md — stage 3 deviation table, stage 4 noise recipe, mask-only policy (line ~368)
• docs/design/wfc3.md — WFC3 deltas, blob physics
• scripts/cache/crds/references/hst/{acs,wfc3}/*_mdz.fits — the MDRIZTAB evidence above

Related but distinct — do not fold in

• driz_cr (median/blot CR rejection) systematically erodes flux at steep-gradient cores — consider LACosmic per-frame masking as the default #61 driz_cr erodes flux at steep-gradient cores → LACosmic option
• Tier-1 empirical ePSF is built from the CR-rejected mosaic — field-star cores get masked before DAOStarFinder ever sees them #62 tier-1 ePSF built from the CR-rejected mosaic

All three concern how DQ/CR masking degrades products, but they touch different stages.

PyAutoMind/draft/research/pyautoreduce/acceptance_noise_rebaseline.md re-baselines the SLACS parity numbers. A bits change moves the IVM weights and therefore the noise maps, so run that study after this lands or its numbers will need redoing.

Original Prompt

Click to expand starting prompt

The prompt was filed 2026-07-10 as the WFC3/IR-only wfc3_ir_dq_bits_dial.md and rescoped HST-wide on 2026-08-04 after the ACS SLACS-gold noise-map report; the MDRIZTAB read showed both instruments share one root cause and one fix site. Current version: PyAutoMind/active/hst_dq_bits_dial.md.

Original 2026-07-10 text:

PyAutoReduce wfc3_ir (and HST generally) lacks a DQ-bits dial: autoreduce/drizzle/combine.py drizzle_kwargs_for never sets final_bits/driz_sep_bits, so AstroDrizzle's default treats every DQ-flagged pixel as unusable. On WFC3-IR snapshot data (tiny dithers) detector-fixed DQ 512 blob regions then produce structured zero-coverage holes in the mosaic that the packaging guard (correctly) refuses: PJ011646 (program 14653, F160W, 5 exposures, 2-6 px dithers) failed with a single 123-px hole at r=5.3 arcsec, outside the 3.8 arcsec science mask, DQ 512 at the same detector pixels in all five exposures (validation issue #25). Standard IR practice passes 512 (blobs) and often 64 (warm px) as usable; Aris's trusted reduction of the same data has no hole. Fix: expose a bits dial on the adapter/TargetSpec (adapter default for wfc3_ir should include 512), document the choice + blob physics in docs/design/wfc3.md, unit-test drizzle_kwargs_for; also consider whether the packaging guard should distinguish defects inside vs outside the target's science aperture. The slacs1430-style validation script reduce_pj011646.py carries a documented monkeypatch workaround until this ships.

2026-08-04 addition (ACS leg), from the user report:

Do you know what is normally done with the dead-columns in HST data? The SLACS lenses in yours/Amy's analysis don't seem to have these 'stripes' of high noise due to the bad columns, but a lot of the lenses are getting these stripes straight through the cores. (noise maps of the F814W slacs-gold)

[Jammy2211]

Both legs merged in #70 (99558128e). Leaving this open deliberately — the control test this issue asks for first has not been run, and it is the gate on leg 1's defaults.

Shipped

• Leg 2 (c514eec) — drizzle/diagnostics.py:check_local_weight_deficit, evaluated inside the same 1.5″ radius as protect_radius_arcsec, reporting the science-region median weight and the worst row/column median as fractions of the cutout median. Recorded in reduction.json at drizzle.local_weight_deficit. Records, never raises — the 0.9 limit is uncalibrated, so a fatal guard would refuse datasets that are likely fine. test_the_existing_guards_are_blind_to_the_same_map pins the blindness: one synthetic striped map that weight_uniformity and mask_isolated_bad_pixels both pass clean.
• Leg 1 (52342b7) — adapters carry the MDRIZTAB rows as (min_exposures, driz_sep_bits, final_bits); TargetSpec.final_bits/driz_sep_bits override at every N; reduction.json records each value and its source. mdriztab=True deliberately not used.

281 passed / 15 skipped, up from 265/15 on main. CI green on 3.12 and 3.13.

Still owed — please run before these defaults reach a release

Re-drizzle one striped SLACS target at the old 0 and at the MDRIZTAB value, and diff the weight and noise maps. Leg 2's diagnostic now scores that comparison objectively, and calibrates its own provisional limit at the same time. If the stripes do not move, revert the adapter defaults rather than shipping the dial — the cause would then be exposure count, dither geometry, or genuine bad columns. The obligation is written into docs/design/hst_acs_pipeline.md stage 3 as well, not only here.

Two corrections to this issue's body, both resolved in favour of the reference files:

1. The shorthand "wfc3_ir 65535 at N=1, 528 at N≥2" flattens the MDRIZTAB table quoted directly above it. The rows give N=2–3 as driz_sep_bits 65535 with final_bits 528 — the two columns differ, so the adapter carries both rather than one bits value per exposure count. PJ011646 (5 exposures) lands on 528 either way.
2. reduce_pj011646.py does not exist anywhere in this repo (no pj011646 reference at all), so "carries a documented monkeypatch workaround until this ships" is stale — there was nothing to unwind.

Downstream, unchanged: acceptance_noise_rebaseline.md must run after this, since the bits move the IVM weights and therefore the noise maps.

---

Generated by Claude Code