Skip to content

feat(reporting): hold reporting-subscription owed sets in the PostgreSQL ledger - #1729

Open
bokelley wants to merge 7 commits into
nextfrom
claude/issue-1695-subscription-storage
Open

bokelley wants to merge 7 commits into
nextfrom
claude/issue-1695-subscription-storage

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Part of #1695, slice 3 (storage), per the owner's decision: widen the key to match TS. It stacks on #1728, #1725 and #1723, so it contains their commits. Review only feat(reporting): hold reporting-subscription owed sets in the PostgreSQL ledger.

Maintenance upgrade (breaking storage boundary)

The PostgreSQL generation key becomes (account_id, consumer_id, delivery_config_id, delivery_config_version, offering_id), matching adcontextprotocol/adcp-client#3517's one-generation-per-offering identity. Account-level generations keep offering_id = ''.

Upgrade: stop every reporting worker, run create_schema() once with this release, then restart. Workers of earlier releases refuse the widened schema through their exact manifests, and mixed-version workers are unsupported. next is a beta line, so per the owner the maintenance upgrade is acceptable. It is documented in docs/reporting-ledger-migration.md ("Reporting-subscription generations: the widened key").

The widening lives in reporting_ledger_account_generations.sql, the existing generation-key guard. It runs under the schema advisory lock and ACCESS EXCLUSIVE locks, losslessly and idempotently:

  • Configurations: reporting_configurations gains offering_id ('' for existing rows), and the primary key includes it.
  • Obligations: reporting_obligations gains offering_id, and the reporting_obligations_period_key unique index includes it.
  • Consumer statuses: reporting_consumer_statuses gains offering_id.
  • The SDK's own references are re-pointed at the widened key: reporting_reconciliation_records, reporting_materializer_discovery, reporting_status_scope_checkpoints and reporting_production_generations.
    • Each gains offering_id with CHECK (offering_id = ''), so they keep serving account-level generations only until each feature serves subscriptions.
    • Each has a named foreign key.
    • Their fresh CREATE TABLE definitions match.
  • A foreign key outside the SDK that references reporting_configurations refuses the upgrade with the maintenance error instead of being dropped.
  • Identity check: a fresh schema ends identical to an upgraded one, and to one upgraded twice. I checked all 1,739 reporting_* catalog objects locally.

I chose an in-place upgrade rather than refusing create_schema on the old key, so every historical upgrade gate (new code on an earlier schema) keeps working. Old workers are refused by the manifests, as the repo's guard requires.

Store

  • Owed sets: PgReportingLedgerStore implements ReportingSubscriptionLedger, so the planner and service accept it:

    • list_generation_obligations;
    • record_owed_set_progress (monotonic GREATEST, only for subscription generations);
    • owed_set_through.

    Horizons live in the private adcp_reporting_owed_set_progress table. The adcp_ prefix keeps it outside the reporting_* catalog contract, like the lease turns.

  • Status snapshots carry the owed-set horizons, so scope_closed matches the in-memory store.

  • Queries:

    • Every configuration, obligation and consumer-status query carries offering_id. The row codecs append it as a trailing column.
    • Leasing selects offering_id = '' only, so an owed set is never leased.
    • Production lookups pin account generations. Delivery and materializer lookups select by the key's own offering.
  • Legacy import (migrate_legacy_reporting) fills offering_id = ''.

  • Status notifications: a store with them enabled refuses an owed set (UNSUPPORTED_FEATURE), because status scope checkpoints are still keyed per account generation.

Manifests

  • Regenerated from a clean migration chain, with fingerprints computed by the package's own catalog reader:
    • outbox;
    • outbox/required_status_schema;
    • materializer;
    • production.
  • Rule: existing keys get updated fingerprints, replaced foreign keys are swapped for the named ones, and new columns and constraints are added only where the manifest already covered that table.
  • Pinned counts: the union of manifests grows 767 → 774 and the materializer manifest 187 → 189. Those pins in the migration tests are updated.

Tests

  • New: tests/conformance/reporting/test_reporting_subscription_ledger_pg.py (core Postgres lane) covers:
    • match: all with four offerings in one generation tuple, sparse and idempotent writes, and a horizon that never regresses;
    • no leasing of owed sets, and progress refused for account generations;
    • two principals' visibility, scope_closed, and status ingest storing offering_id;
    • lossless, idempotent widening of a four-column schema with an existing row;
    • an adopter foreign key refusing the widening;
    • status notifications refusing owed sets.
  • Locally against PostgreSQL 16:
    • the generation, caller-ownership, reconciliation, production, materializer, receipt and feed migration suites pass;
    • the core lane passes except the wheel and historical-artifact tests, which need more disk than the sandbox has; CI runs them.

This does not adopt beta.2.

🤖 Generated with Claude Code

bokelley and others added 6 commits October 30, 2026 03:20
…ng ledger

A principal's reporting subscription is now executable by the ledger. The
engine's frozen owed sets become ordinary obligations of one configuration
generation per account, owning principal, subscription_id, version and
resolved offering_id, aligned with the TypeScript SDK.

- ReportingConfigurationGenerationKey gains an optional offering_id, set
  only for subscription generations. ledger_asdict omits it when unset, so
  every digest, work key and persisted scope of an account generation is
  byte-identical to before; a guard test forbids bare asdict in reporting.
- SubscriptionObligationPlanner writes sparse, immutable obligations for
  periods ending by the engine's horizon, then advances the owed set's
  completeness horizon. Stores opt in through ReportingSubscriptionLedger;
  the in-memory store implements it, and the planner refuses other stores.
- The producer never grid-fills an owed set and reads an owed obligation
  only once its period has ended. Owed sets are not leased; the service
  runs them through run_configuration.
- get_reporting_status reports offering_id per period, keeps one wire
  generation per id, version and purpose, and leaves scope_closed false
  while the scope ends after the owed set's horizon.
- sync_reporting_status resolves a statement through the caller's owed
  obligation for its period and report definition; a period the owed set
  does not hold fails with INVALID_STATUS_PERIOD.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…pshot

The closed-payload allowlist in reporting_reconciliation_validate now names
offering_id, so the required-schema manifest carries the function's new
fingerprint, computed from a clean migration chain. The public API snapshot
records ReportingSubscriptionGeneration and ReportingSubscriptionLedger.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The engine now persists its state through a compare-and-swap store, so
frozen_owed_sets reads one snapshot of the principal's state, as
owed_periods and subscription_coverage do, instead of the removed
process-local lock.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…Service

ReliableReportingService(reporting_subscriptions=...) now owes and serves a
principal's reporting subscriptions. Without the option, capabilities,
responses and worker turns are unchanged.

- Each worker turn plans every listed principal's frozen owed set, then
  executes it with the account configurations. A subscription generation is
  installed through the same trusted configure path: account_context
  receives it with configuration.subscription set, and the service validates
  the source route, currency and advertised offering as for an account
  configuration. The horizon is the earlier of the engine's and the turn
  clock.
- A subscription that cannot be planned is isolated, recorded in
  turn.subscription_errors and reported to planning_error_handler or the
  worker error handler; it never stops another subscription or an account
  configuration.
- Capabilities gain configuration_tasks, the subscriptions block,
  standard_dimensions, the feature id and the principal section at the
  served release; callers below 3.3 see the pre-3.3 block, and adcp#8585
  offering fields reach only 3.3.0-beta.2 callers.
- get_reporting_status summary and periods views gain
  subscription_coverage. Ledger reads stay scoped to the caller, and
  resolve_principal is checked against the caller resolver.
- The planner takes an install hook and resolves currency per generation;
  the engine exposes its offerings and status_retention_days.

Production options, managed or receipt-reconciled subscription offerings
and PostgreSQL owed sets are refused or not yet served.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…on grant

Access means access. A loss of authority opens no window: the account is
unknown to every task, sync_reporting_receipts and sync_reporting_status
included. A seller that wants a buyer to finish reconciliation keeps a
reduced grant instead of revoking.

- AuthorizedAccount carries the grant's allowed_tasks, persisted in the
  engine state document only when set, so unscoped documents are unchanged.
- Coverage continues only while the grant lists get_reporting_status; a
  grant reduced below that ends coverage as a revocation does.
- ReportingSubscriptionEngine.grants_task answers from the last reconcile.
- For a caller holding subscription generations on the account,
  ReliableReportingService checks get_reporting_status, exact revision
  reads, sync_reporting_status and sync_reporting_receipts against the
  grant, and refuses an absent task with ACCOUNT_NOT_FOUND.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…SQL ledger

The PostgreSQL generation key widens to account, consumer,
delivery_config_id, delivery_config_version and offering_id, matching the
TypeScript ledger, so several offerings of one subscription version coexist
with account generations, which keep offering_id ''.

This is a stopped-worker maintenance upgrade. Stop every reporting worker,
run create_schema once with this release, then restart: workers of earlier
releases refuse the widened schema. Under the schema lock the upgrade adds
offering_id to configurations, obligations and consumer statuses, widens
the primary key and the obligation period index, and re-points the SDK's
own references at the widened key. Those referencing tables hold
account-level generations only, enforced by a CHECK. A foreign key outside
the SDK refuses the upgrade instead of being dropped. The upgrade is
lossless and idempotent, and a fresh schema ends identical to an upgraded
one.

- PgReportingLedgerStore implements ReportingSubscriptionLedger: owed-set
  obligations, a monotonic completeness horizon in the private
  adcp_reporting_owed_set_progress table, and horizons in status snapshots.
- Every configuration, obligation and status query carries offering_id;
  an owed set is never leased; production, delivery and materializer
  lookups stay on account generations.
- A store with status notifications enabled refuses an owed set until the
  status checkpoints hold one per offering.
- The required-schema manifests and the pinned object counts are
  regenerated from a clean migration chain.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — PostgreSQL reporting-storage slice (key-widening to include offering_id).

No blocking findings. The reviewer confirmed the key-widening migration (reporting_ledger_account_generations.sql) is idempotent (early RETURN on 5-col PK), lossless (DEFAULT ''), and fails closed on outside-SDK FKs with test coverage. Account happy path preserved (offering_id='' pins), owed sets never leased, record_owed_set_progress monotonic under account lock, notifications refuse owed sets, and ledger_asdict keeps digests byte-stable. Public API additions are additive and carry the correct feat: signal.

High-risk flag is true only for src/adcp/decisioning/handler.py (modified), but the reviewer surfaced no medium-or-higher concerns on that file, so the modification is presumed safe (does not trip row 5). gated_paths is false; no author team gate; no prior decision. With zero findings, rows 1–8 do not fire and the outcome falls through to row 9.

Non-blocking note: core wheel/historical-artifact tests deferred to CI for disk; the owed-set path is exercised by the new PG16 conformance suite.

The materializer manifest now lists the discovery table's offering_id
column, CHECK and foreign key, so the installed and exact catalog counts
grow by those objects. The PostgreSQL owed-set test imports psycopg only
after the pool fixture, so it skips cleanly where psycopg is absent.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — subsequent pass, clean.

This is a follow-up to a prior approve (head 7c080b1). The delta since then is three test-only files: catalog count repins (materializer 187→189, scope 794→801, both matching the regenerated manifests' +7/+2 growth pinned in the PR body) and relocating from psycopg.errors import RaiseException inside the isolated_reporting_pool context so the fixture's pytest.importorskip("psycopg") skips cleanly before the import. All sound.

No critical, high, or medium findings in this run. No prior blockers to re-surface.

Gate checks:

  • Author team gates: no no-auto-approve match.
  • gated_paths: false; review_decision: APPROVED (no hard path gate triggered).
  • high_risk: true via src/adcp/decisioning/handler.py (modified), but the reviewer surfaced no medium-or-higher concern on it, so the modification is presumed safe — the flag alone does not escalate.
  • No (deleted) high-risk reasons.
  • Prior decision was approve, so sticky escalation (row 6) does not apply.

Rows 1–8 do not fire; falls through to row 9 → approve.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant