Skip to content

feat(client-contract): leave out party totals by default and order columns for reading - #24

Merged
jm-rivera merged 1 commit into
devfrom
feat/reading-defaults
Oct 26, 2026
Merged

jm-rivera merged 1 commit into
devfrom
feat/reading-defaults

Conversation

@jm-rivera

Copy link
Copy Markdown
Contributor

A client fetch now leaves out party totals by default. With an open party role, the long layout used to hold group and whole-total rows such as DAC countries, total beside their members, so summing value over every donor to Kenya gave 15.2bn for 2022 where the members sum to 3.5bn, with no warning. The build now posts "without_party_totals": true unless the caller passes party_totals (Python) or partyTotals (TypeScript). A role the caller binds keeps what it binds, so a total named by its dcid still comes back. The wire flag is unchanged. The README states what a ref records under the default, that artifact matching ignores the flag, and that newest_version names the newest plain build, which a cut ref's content_version need not equal.

The new column-order rule (rules/column-order.json) sets the order in which a client presents a dataset's columns: time period, provider, recipient, subject, purpose, value, unit, flow type and provenance first, the label column winning over the code column, then the served order, with stat_var last. A stacked dataset puts part first. A client hashes the downloaded file before ordering, so digests and refs keep the server's bytes. The validator reads each file's served order from its Parquet schema with pyarrow and checks every recorded columns list against the rule applied to it, so a hand edit to columns fails. A missing or malformed rule file fails the validator and the capture tool.

Dev had moved 12 commits past the last recording, so every scenario is re-recorded at apis c9a1b905 and served/openapi.json is recaptured. With the flag stripped, request bodies in the 120 existing scenarios are unchanged; the remaining differences are extra running polls and four recipient labels dev renamed. Two new recorded scenarios, fetch-spec-party-totals-left-out (96 rows) and fetch-spec-party-totals-kept (111 rows), pin the default and the option. The capture tool's pacing check now keeps the first three running polls, since this build polled running five times. The tools tests need pyarrow, and ci.yml installs it.

This stacks on #23. The SDK consumes it after apis bumps its spec-kit pin.

Testing: uv run tools/validate_client_contract.py reports 122 valid scenarios; tools/tests passes 537 tests and python/client-contract 105, including fault rows that fail when the party-totals or column checks are removed.

@jm-rivera
jm-rivera changed the base branch from docs/client-waiting-rules to dev October 26, 2026 08:08
…lumns for reading

A fetch now posts "without_party_totals": true unless the caller passes
party_totals (Python) or partyTotals (TypeScript). A role the caller binds
keeps what it binds. The wire flag is unchanged; only the client default moves.

The new column-order rule (rules/column-order.json, nine slots with vectors)
puts time period, provider, recipient, subject, purpose, value, unit, flow type
and provenance first, then the served order, with stat_var last. A client
hashes the downloaded file before it orders the columns, so digests and refs
keep the server's file.

Every scenario is re-recorded against apis dev at c9a1b905, and
served/openapi.json is recaptured. Two new recorded scenarios,
fetch-spec-party-totals-left-out and fetch-spec-party-totals-kept, pin the
default and the option.

Every accepted ref example now carries the rendering, parts, content
version and sentence of a member its scenario served, so ref-one-member
pins the plain rendering from fetch-spec-party-totals-kept and
ref-answer-member comes from answer-definite-fetch. The validator checks
both, with a fault row each.

Every scenario is re-recorded against the deployed dfo-apis dev app at d80df9f5f, with the served document captured from that commit. The long-build scenarios start at 1967 and 1968, years dev had not built.
@jm-rivera
jm-rivera force-pushed the feat/reading-defaults branch from 41c47f8 to ca207f0 Compare October 26, 2026 09:31
@jm-rivera
jm-rivera marked this pull request as ready for review October 26, 2026 09:42
@jm-rivera
jm-rivera merged commit a1d723b into dev Oct 26, 2026
22 checks passed
@jm-rivera
jm-rivera deleted the feat/reading-defaults branch October 26, 2026 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant