[segran2]

Missing or rejected source readings and recording gaps can leave Home use or Solar unknown. This adds explicitly requested, audited historical estimates while preserving original observations and quality. Fixes #1750; related earlier work: #1667 and #1668.

The API/UI exposes an estimated covered Home use subtotal separately when the range total remains unknown. It does not promote that subtotal to a complete total or measured energy. Current master's source-hold and 10% unknown-time policy remain unchanged; short outages may already produce an accepted estimated total, without a repair.

Two offline repair methods are available: reconstruction from explicitly selected retained AC source-power series, and endpoint-mean estimates for Home use and Solar. The endpoint helper computes (power before + power after) / 2 × gap hours for incomplete minutes only. Home uses retained minute source-energy means; Solar uses retained ten-second means. Both bounds and an explicit maximum distance are required. Missing endpoints never imply zero; two observed night-time zeros give 0 Wh.

The endpoint workflow keeps Core collecting during backup, preparation and application. An independent checked SQLite online backup is required before applying a reviewed hashed plan. Apply briefly reserves the SQLite writer, rechecks evidence and atomically publishes audit metadata plus daily-cache invalidation. No live database is replaced. Original samples, source energy, quality, configuration and newer unrelated measurements remain intact. A matching retry is idempotent; stale, conflicting or tampered evidence fails closed. Endpoint corrections supersede source-power corrections only in the derived view; both audits remain.

Corrections participate in daily totals, ledger buckets and aggregate charts, remain estimated, and survive portable backup/restore and retention. Estimated Solar is excluded from measured.pv_wh. Totals expose completed-through timestamps and do not extrapolate an unfinished writer tail. Raw history and training/control inputs are unchanged. Valid SemVer build metadata retains native updates for the installed current slot; published trials still require exact release identity.

Current-master integration — 24 October 2026

Head 53f5130 merges master 2d7c8eb without rewriting earlier commits.

• Preserve master's newer source-hold and day-completeness policy. The first integrated test run exposed fixtures that assumed a single short outage must remain unknown. Fixtures now explicitly model retained pre-source-hold rejected minutes or sustained invalid configuration. Short PV/EV/writer/topology events remain accepted estimates; longer unsupported gaps retain partial coverage.
• Preserve repair rejection, explicit interpolation bounds, original quality and independent measured totals. No repair production logic or control settings were changed by this follow-up.
• Fix one rendered quality-note bug: complete daily totals with estimated Solar no longer incorrectly claim that current totals have missing readings. A new regression fails before the fix and passes afterward, including combined Home/Solar estimates and genuine incomplete totals.
• Refresh the changeset. The previous body incorrectly called this PR a draft; GitHub currently marks it ready for review. The owner completed visual review of all five fixed examples on 24 October 2026 and reported that all five were clear.

Validation on the integrated tree

• Full make verify passes: Go suites, scripts, native worker checks, vet and build.
• 884 web tests pass, with no failures or skips.
• Targeted state/API race tests pass for Endpoint, Repair and HouseholdSubtotal.
• The first integrated runs failed on the obsolete short-outage fixture assumptions; those failures are recorded and resolved by the compatibility changes above. No golden or known-failure exemption was added.
• Docker-context and live Avahi checks were skipped because this host has no Docker daemon.
• All executed GitHub CI checks pass on 53f5130, including full Go race, web, full stack, vet and Linux builds. Three conditional driver/module jobs were skipped. Test run: https://github.com/srcfl/ftw/actions/runs/40100658760.
• The real renderer and existing card markup/styles were checked in a local browser against five fixed examples: partial Home use, estimated Solar, both flows estimated, complete zero and no data. The owner visually accepted all five examples; this remains fixed-fixture evidence, not live-box acceptance.

The existing positive review applies to 6554c67, before this integration; the follow-up needs review. On 24 October 2026 the owner reviewed the rendered examples at http://127.0.0.1:18751/ and answered “Tydligt i alla fem exempel” (clear in all five examples). This satisfies the requested human visual review for those examples; it does not constitute a code review, merge approval or live-data repair approval. No household database, installed FTW or live control was modified.

Operator workflow: docs/history-endpoint-repair.md and docs/history-repair.md. This does not change the forward-looking source-hold or dark-PV policies introduced by other work.

Review follow-up — 27 October 2026

Commit ab3fe94bcbdac69122bcbada1d2b67279ff6e557 addresses the two “worth-a-question” findings and the endpoint CLI coverage gap from the 26 October review. It is a fast-forward follow-up; prior commits/authorship and ready-for-review status are preserved.

• Retry evidence: newly prepared source-power repair plans include a logical digest of the complete working-history schema and table contents. A retry checks it even when every correction minute already matches. Only this plan's exact correction rows and recomputable daily-cache contents are excluded; unrelated estimates, original data and schema changes are rejected. Daily caches are invalidated on retries too, so excluded cache contents cannot supply the returned result. Byte-level working/backup hashes remain checked. Legacy plans without the new digest may apply to their unchanged original working copy but cannot authorize an unchecked post-apply retry; prepare a new copy/plan for that workflow.
• Independent backup integrity: endpoint apply runs the existing cancellable SQLite quick_check helper on the caller-supplied independent backup before acquiring the live writer or publishing corrections. A matching generation is insufficient. Corruption aborts with no repair metadata published.
• CLI coverage: endpoint prepare/apply now accept an injected output writer. Tests cover argument/date/gap/reason rejection, private plan creation, refusal to overwrite an existing plan, reviewed-hash enforcement, successful JSON responses, idempotent apply and malformed plan rejection. This introduces no new command or network API.

Final-tree validation

• New corruption and retry regressions were first run against the old code and failed: a corrupt backup whose generation remains readable was accepted, and all three unrelated-data/estimate/schema changes were accepted on retries. They pass after the fix.
• Focused state and both repair CLI suites pass. Existing idempotence and atomicity cases remain green; additional cases verify rejection of unchecked legacy retries and rebuilding altered daily caches.
• Focused state/CLI go test -race passes.
• All 884 web tests pass. No rendered UI changes were made by this follow-up; the earlier visual acceptance remains scoped to its reviewed examples.
• Go build and vet pass. Compose-migration, container-boundary, release-workflow, native-solver, protocol-boundary and cloudlink-boundary targets pass; Docker-context/live Avahi checks are skipped where Docker is unavailable. Whitespace check passes.
• Full make verify is not green in this environment: TestABoxReportReachesTheGatewayThroughTheRelay (energyreport: “gateway got []”) and TestAReportReachesTheGatewayThroughTheRelay (statsrelay: “gateway answered 400”) fail. Both also fail on a clean checkout of unchanged prior PR head 53f5130dc027b2ac91d762fafa7c506b468bc8bb, and repeated focused runs reproduce them. No relay code/test was changed, bypassed or disabled. All other Go packages in the final full run passed. The cause of those baseline failures remains outside this follow-up; prior green CI above belongs to the old head, not this commit.

This follow-up operated only on temporary test databases and repository code. No household database was repaired, installed box updated, control changed, release published or merge performed. New-head GitHub CI/review remains to be assessed.

Relay test investigation resolved — 27 October 2026

Commit f40de9615a56f794584e6f45eb0b73383d0d6012 resolves the two baseline failures recorded above. The fake gateway used a fixed date while the real client's report envelope used the host's wall-clock date. Once those dates diverged, the unchanged today/yesterday envelope check rejected the request with inner HTTP 400.

• Keep statsrelay.New on the normal wall clock. Add an explicit-clock constructor so the shared test fixture gives its client and gateway the same immutable clock.
• Regression cases for years 2000 and 2040 and an offset crossing UTC midnight failed before this fix and pass afterward.
• Explicitly retain rejection of envelopes sent two days ago or in the future; today/yesterday still pass. No replay/freshness window is widened.
• A full race run then exposed a separate fixture-only problem: the retired-asset test wrote more than three days of observations in one transaction and exceeded the unchanged 30-second storage timeout under SQLite race instrumentation. Write the same ordered observations in bounded batches instead. No production timeout, sample, assertion or test is removed.

Final-tree validation:

• Full make verify passes, including the formerly failing relay tests, Go suites, vet, build and repository checks. Docker-context/live Avahi checks retain the documented environment skips.
• Relay, energyreport and fleetping suites passed five repetitions after the clock fix. The final batched-fixture tree also passes their complete go test -race run (energyreport approximately 196 seconds).
• Whitespace check passes. No web assets changed; the prior 884-web-test evidence remains applicable to the unchanged web tree.
• GitHub Actions on the preceding repair commit ab3fe94 completed successfully in all six workflows. New-head CI for this follow-up is checked separately; those earlier green runs are not attributed to this new commit.

The earlier local make verify failure is historical evidence, resolved by this follow-up. This does not change a running box, repair a household database, relax a protection limit or authorize merging/releasing the PR.

Current-master integration and final validation — 27 October 2026

Merge commit 0d62873ff41890489b0925603a1a029434d8cae9 integrates master 0d54327a36a11f7953de3cf4257376358246000a without rewriting previous commits. Master independently contains a relay clock seam (Client.WithClock); keep that implementation and its fixture binding instead of retaining this branch's duplicate NewWithClock constructor. Keep the new fixed-date/UTC-boundary/expiry regressions, bounded fixture batches and all history-repair review fixes. The only merge conflict was the fixture's clock binding; other master changes are preserved.

• Integrated-tree full make verify passes: Go suites, vet, build, native worker and repository checks. Docker-context/live Avahi retain the environment skips.
• 898 web tests pass, zero failures/skips, on the integrated tree.
• Focused integrated relay/energyreport race tests pass. The previous complete relay/energyreport/fleetping race run passed before this integration; the 196-second full energyreport race run is not claimed as a new run on the merge commit.
• Whitespace check against current master passes. The upstream generated release changelog has existing trailing spaces in the merge delta; it is left unchanged.

This integration resolves the new conflict with master. GitHub CI for this exact merge commit is tracked separately below; prior-head CI remains historical evidence. The PR remains unmerged and the installed household box is unchanged.

Final-head GitHub CI — 27 October 2026

All six Actions workflows completed successfully on 0d62873ff41890489b0925603a1a029434d8cae9: test, repo hygiene, brand-cleanup, changeset-check, Core Linux binaries and offline history converter. The test workflow includes successful normal Go, Go race-detector, web, full-stack and registry checks; the three conditional driver/module jobs were skipped. The PR is conflict-free at this check. This is CI acceptance of the exact integrated head, not a merge or release approval. The earlier relay-test blocker recorded above is resolved.

[miravoss26]

Reviewed. Adds audited historical-estimate repair for Home use/Solar gaps plus a "household subtotal" status field for the still-incomplete current day. Fixes #1750.

Correctness: The estimate machinery is careful about what it touches. endpointOriginalValid/planEndpointRepair only fill minutes that are genuinely invalid (missing power, balance mismatch, non-finite/negative values) and bound each gap by real recorded power on both sides within MaxGapMS — anything wider or unbounded gets skipped with a reason, never guessed. ApplyEndpointRepair re-verifies the plan hash against freshly-recomputed evidence before writing (so stale review approvals can't silently apply against changed data), requires an independently-verified backup (checked via os.SameFile, not just path string), detects conflicting overlapping repairs, and is idempotent on re-apply. The CLI tools (ftw-history-endpoints, ftw-history-repair) are local, file-based, human-run with a mandatory --reason — not network-exposed, so no new authz surface.

On the display side, household subtotal is correctly kept out of measured.load_wh and never promoted to the complete total — api_household_subtotal.go's own doc comment states the invariant and the tests (TestHouseholdSubtotalStatusAndDailyKeepIncompleteTotalSeparate) check it round-trips through both /api/status and /api/energy/daily. Web-side changes are plain textContent, no innerHTML/XSS surface.

Test coverage is thorough on both the Go and web sides (estimate-vs-measured separation, malformed-subtotal rejection, quality-label plumbing). No secrets, no new deps, no new network destinations.

Safe to merge from my read.

[miravoss26]

Adds an offline history-repair feature: reconstructs missing Home-use/Solar readings via retained AC source-power series or endpoint-mean estimates, behind a backup → hashed plan → apply flow that keeps originals and audits the correction. Read the full diff (29 files, ~2500 added lines across go/internal/state + the two new CLI tools).

Checked the PR body's safety claims against the actual code rather than taking them at face value:

• worth-a-question — history_repair.go's ApplyHistoryRepair retry fast-path (when every plan minute already matches history_load_estimates) skips re-hashing the full working-history file, so a retry wouldn't catch tampering of unrelated rows introduced between two apply calls. ApplyEndpointRepair in the sibling endpoint-repair path re-derives the full evidence digest from the live DB on every call, retry included — that's the stronger pattern and probably what this one should match. Not reachable in the normal single prepare→apply flow, only if the workdir is touched in between.
• worth-a-question — the docs call the required backup "integrity-checked"; ApplyEndpointRepair only checks inode-difference + a matching generation row, never runs PRAGMA integrity_check/quick_check on the caller-supplied backup (the offline tool's own self-made backup does get quick_check, just not this path).
• nit — ftw-history-endpoints's CLI (flag parsing, plan-file write, JSON output) has no test coverage, unlike its sibling ftw-history-repair, which does. The underlying library functions are well tested either way.
• Verified the one named bug fix is real: web/energy-today.js was unconditionally seeding the gaps-note text whenever today was truthy; the new pv_quality==="estimated"" branch now appends instead of overwrites, and the new regression test in history-repair-quality.test.mjs` fails pre-fix / passes post-fix.
• Security screen is clean: no secrets, no new deps, no network calls, no shell/exec. SQLite access is parameterized throughout — the handful of string-concatenated SQL fragments only splice in fixed Go-literal identifiers, never plan- or request-controlled values.
• Test coverage on the risky paths (double-apply, transaction rollback on conflict, tampered plan/backup/working-copy, stale evidence, forced-trigger-failure atomicity) is genuinely above-average for a change this size, not happy-path-only.

Safe to merge from my read. The two worth-a-question items aren't blockers — flagging them so whoever merges can decide whether they're worth a quick follow-up or just noted for later.

[miravoss26]

Reviewed (large diff, focused on the two repair pathways — the actual risk surface, a tool that rewrites historical household energy data).

history_repair.go / ftw-history-repair CLI — never opens the owner's live DBs for writing: PrepareHistoryRepair does VACUUM INTO a backup copy, then operates only on a further corrected/ copy. The plan is content-hashed at every stage (backup state/history, working evidence, working history/state, plan digest itself); ApplyHistoryRepair requires the caller to supply the plan's own SHA-256 back, re-verifies the backup files are unchanged, re-verifies the working copy is unchanged, and is idempotent on retry (allApplied check against history_load_estimates + evidence-hash re-check) without re-inserting. repairFileHash rejects symlinks and repairCopyFile uses O_EXCL — reasonable defense against a swapped file between prepare and apply. repairIntegral's trapezoidal integration refuses to interpolate across a gap bigger than MaxGapMS and requires full-minute coverage before accepting a reconstructed value; planHistoryRepair only fills minutes that are genuinely missing (never overwrites existing data or an existing estimate row). TestRepairCommandPrepareThenApplyChangesHistoricalCopy exercises the full CLI prepare→apply round trip and checks the actual reconstructed Wh value (250 Wh for a known gap), not just that it ran.

history_endpoint_repair.go / ApplyEndpointRepair — different shape: writes a small additive marker into the live energy_ledger_meta (never touches raw samples), gated on an independent backup passing quick_check and matching the live DB's migration generation (os.SameFile + EvalSymlinks guard against the backup path aliasing the source), a fresh re-derivation of the plan's evidence hash against current data (stale plans can't replay against drifted history), and a conflict check against any already-recorded repair for the same minute. TestEndpointRepairRejectsCorruptIndependentBackup, TestEndpointRepairRejectsChangedEvidence, and TestEndpointRepairRollsBackWhenCacheInvalidationFails all exercise the failure paths, not just the happy one.

No secrets, no new network destinations, no new dependencies; SQL throughout is parameterized. The human gate here is the design (a person reviews plan.json and supplies its exact hash back) — that part is inherently a judgment call, not something a diff read can second-guess. CI green (Core + race detector + full stack + web).

Safe to merge from my read.