Skip to content

feat(metamodel): @of is optional on @agg:collect — the whole-object rollup (#335) - #345

Merged
dmealing merged 25 commits into
mainfrom
feat/issue-335-whole-object-rollup
Aug 24, 2026
Merged

feat(metamodel): @of is optional on @agg:collect — the whole-object rollup (#335)#345
dmealing merged 25 commits into
mainfrom
feat/issue-335-whole-object-rollup

Conversation

@dmealing

Copy link
Copy Markdown
Member

Closes #335.

The gap

A projection could roll related rows up into an array of one column, and had no way to roll them up into an array of objects. origin.aggregate @agg: collect required @of, so "every supplier's {id, name} for this product" was inexpressible — the shape had to be a second round-trip, or a hand-written view, which is unmanaged and invisible to meta verify --db. When origin.collection retired in 0.24.0 this became the one coverage gap the retirement guide had to state rather than close.

What lands

@of becomes OPTIONAL on collect, and omitting it means a whole-object rollup. The carrying field.object @isArray @objectRef collects each related row as its declared value object.

{ "field.object": {
    "name": "supplierBriefs", "isArray": true, "@objectRef": "SupplierBrief",
    "children": [
      { "origin.aggregate": { "@agg": "collect", "@via": "Product.suppliers" } }
    ]
}}
  • The declared value object IS the exposure. Members bind to the @via terminal entity's fields by name; a field the entity has but the value object omits is simply not projected. That carries the Prelude to #210: retire redundant payload origin-typing (Kotlin, Python) and close Java's KNOWN_GAPS entry as moot #270 guarantee (a curated value object must not silently become the full entity) down to the DDL tier. Name matching rather than extends is deliberate — it keeps one value object collectable from two different entities. The convention is written into the byte-gated origin.aggregate registry prose so no port has to infer it from TypeScript.
  • Eight load errors, in all five ports — carrying field must be a field.object declaring @objectRef; that @objectRef must name an object.value; @via is required; the path must be to-many; @distinct is refused; @orderBy keys must resolve against the @via terminal entity; every value-object member must match a terminal field; and a matched member must agree on both type axes (subtype and array-ness).
  • A new error code, ERR_COLLECT_WHOLE_OBJECT, for the five refusals that would otherwise have shared ERR_INVALID_ORIGIN. Not taxonomy for its own sake: the shared corpus compares error code + source and never message text, and ERR_INVALID_ORIGIN is exactly what a loader that still requires @of emits for this same metadata — so five of the eight negative fixtures passed against three ports containing none of the rules. Proven: the unported ports went 4 → 9 failures once the code was distinct.
  • View lowering, both dialects. Postgres emits COALESCE(jsonb_agg(jsonb_build_object(...) ORDER BY <pk> ASC) FILTER (WHERE <pk> IS NOT NULL), '[]'::jsonb)jsonb not json, because PG's json type has neither an equality nor an ordering operator.

Half B — array filter/sort validation

Two commits (9a09724, 1de6601) give @sortable the subtype validation @filterable already had, and stop an array field claiming a filter-operator band.

This makes previously-loading metadata fail to load, so it is breaking on the metadata contract — but it is the previously-wrong-acceptance ruling this release already invokes twice (#342, and before that the 0.19.1 @min clamp and the 0.21.6 like case-sensitivity fix): such a declaration emitted a like/eq rule against a text[] column, i.e. SQL that cannot execute. It was never validly expressible. Bounds, stated rather than hoped:

  1. In-repo cost is zero — a structural per-field-node scan of 1321 JSON and 124 YAML files found no field carrying both isArray: true and @filterable/@sortable: true.
  2. What breaks was already broken — a runtime failure becomes a load failure.
  3. Nothing ever taught adopters to write it — never documented, never exemplified, never emitted by any generator.

Residual exposure is not zero and the changelog says so: an adopter who wrote the form independently auto-adopts a load error, because pre-1.0 ^0.24.x resolves it.

Two findings the real-engine round-trip paid for

Both were invisible to emitted-SQL text, and only the emit → apply → introspect → re-diff gate could see them:

  1. The SQLite emitter was wrong and the unit test agreed with it. On SQLite 3.44.0 the in-aggregate ORDER BY destroys the JSON subtype, so the column returned an array of quoted strings while the generated type declared an array of objects. The obvious json(json_object(...)) wrapper does not survive the ORDER BY (seven variants probed). Fix: build the ordered array, then re-wrap element-by-element through json_each, which iterates in array order — ordering survives and json(value) restores the subtype. Dropping ORDER BY was rejected (an author's @orderBy would silently do nothing); a correlated ordered subquery also works but loses multi-hop @via.
  2. A field.long member arrives inside the rollup as a JSON number while the same value as a top-level BIGINT arrives from node-postgres as a string. Inherent to JSON, lossy above 2^53. Pinned with a comment rather than normalised away.

Verification

scripts/ci-local.sh on this exact tree (merge-base bcef4d9c8), every lane read from its trailing OK — line, never piped:

Lane Result
--quick PASSED — 16/16, incl. leak-scan, #337's shipped-doc-examples gate, metamodel-version bump, peer-range bounds, TS conformance, ts unit suites, completeness-gate (mutation, killed 787), doc-template + embedded-library drift
--only ts-slow PASSED — migrate-ts real-PG, runtime-ts real-PG dialect matrix, integration-tests (ts): persistence conformance 229 pass / 0 fail
--only java PASSED — conformance java + kotlin, java-reactor, integration-tests java + kotlin
--only csharp PASSED — conformance csharp, integration-tests csharp 118/118
--only python PASSED

Coordination

The 0.24.1 release cut owns the merge — landing this is part of that cut, not a separate action.

🤖 Generated with Claude Code

https://claude.ai/code/session_015WgNA6uDN9qxu2nA471rAq

dmealing and others added 25 commits August 23, 2026 19:47
…ract it lands in (#335)

Design for making @Of optional on origin.aggregate @agg:collect. Ships in two
halves: the rollup itself, and two holes in the queryable-projection contract that
the rollup would otherwise land a column into — the filter allowlist never consults
isArray, and @Sortable has no subtype validation at all. Both are live today.

Records the decision NOT to mint a subtype or @kind for "document" vs "relational"
projections: ADR-0037's subtype test fails, @kind is the physical-storage axis with
read-only-ness derived from it, and the query contract is already per-field, which
is the right level.

Four claims in the issue did not survive verification against the code, including
that the @objectRef-must-be-a-value rule already exists — it is payload-scoped and
never reaches a projection-hosted field, so the branch's main #270 guard would have
been absent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ng it (#335)

A parallel change (#342) already moved metamodelVersion 0.10 -> 0.11, and that bump
is unreleased — v0.24.0 shipped 0.10. The gate baselines against the last release
tag so a cycle needs one bump, not one per PR, so this change needs no edit at all.

Records the two conditions that would change it: 0.11 shipping before this lands,
or Half B being judged breaking.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… it (#335)

A rule with no negative fixture is a blind spot one layer up — "no fixture covers
it" and "every port enforces it" look identical on a green suite. Three findings in
the parallel #342 work were exactly this shape, including one where the ungated arm
was the precise place a port had diverged.

Makes Half B's negative cases shared conformance fixtures rather than port-local
unit tests, and records the measurement that motivates it: a structural scan of
1321 JSON + 124 YAML files finds zero fields carrying both isArray:true and
filterable/sortable:true, so nothing in the corpus exercises an array field through
the filter tier at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…#335)

"Potentially breaking" was too flat. Three bounds now stated: in-repo cost measured
at zero, what breaks was already emitting SQL that cannot execute, and — the
distinction that actually matters — nothing ever taught adopters to write it.

That last point separates this from a sibling change in the same window that is the
same category and a different magnitude: a retired spelling that appeared as a
worked example in a shipped skill has a plausible acquisition route; a silently
wrong emission nobody was told to write does not.

Also records that this version question gets asked when #335 is cut, on its own
evidence, rather than folded into a ruling already made about a different change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…urface it lands in (#335)

Fifteen tasks, 86 steps, TDD throughout. Half B first (two loader rules making the
filter/sort contract honest about array columns) so Half A lands into a correct
tier, then the loader change across four ports, the seven byte-gated prose copies,
the TypeScript view column and its lowering, and a real-engine round-trip on both
dialects.

Carries the traps that have bitten this repo before as inline warnings rather than
lore: `dotnet test` printing Passed! over a failed compile, a Maven or ci-local run
piped through tail reporting tail's exit status, the integration round-trip living
in a package no per-package run touches, and the fresh-worktree precondition that
looks like a regression in codegen-ts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…335)

Ports the two loader validation rules from TypeScript (task 3) to C#, Java
and Python:

- @filterable: true on an ARRAY field now errors ERR_FILTERABLE_UNSUPPORTED_
  SUBTYPE (extends the existing no-operator-band check in each port).
- @Sortable: true on an ARRAY field, or a field subtype with no operator
  band, now errors ERR_SORTABLE_UNSUPPORTED_SUBTYPE — a wholly new pass in
  all three ports, wired into each port's validation pipeline.

Array-ness is read through each port's resolving accessor (C#/Java
field.ResolvedIsArray()/isArrayType(), Python field.resolved_is_array()),
matching the accessor already used by the @agg:collect check, per ADR-0039.

Java additionally gains the ERR_SORTABLE_UNSUPPORTED_SUBTYPE message-prefix
constant in ErrorMessageConstants (the ErrorCode enum member already
existed; the message ledger did not).

Turns fixtures/conformance/error-filterable-array-field and
error-sortable-array-field green on all three ports.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgNA6uDN9qxu2nA471rAq
Both corrections were established by execution, not review.

Task 5's model was illegal in two ways unrelated to this feature: a projection
cannot `extends` an entity, and a projection's identity must `extends` an entity
identity rather than declare fresh @fields. Replaced with the shape the corpus
already uses, verified to load clean and to produce exactly the one intended
error for the whole-object form.

Task 9's grep returns 9 files, not 7 — two are this plan and its spec quoting the
sentence. Scoped the grep to product files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#335)

An @of-absent origin.aggregate @agg:collect is now a WHOLE-OBJECT rollup:
the carrying field must be a field.object (isArray, @objectref) and the
collect assembles the related rows as an array of that value object rather
than an array of one scalar column. @via is required (no @Of entity to
infer the relationship from), @distinct is refused (a guaranteed no-op on
the object form), and @orderby resolves against the @via terminal entity
via a new _viaTerminalEntityNode helper.

The @objectref must resolve to an object.value — #210's payload-scoped
value-only rule never reaches a projection-hosted field, so this branch
enforces it itself (ERR_SUBTYPE_RULE_VIOLATION), closing the same hole
0.20.16 (#270) closed for payload-typed fields.

The scalar collect-with-@Of path is untouched.
…ollect (#335)

Fix round 1 on the whole-object collect review: two of the six must-enforce
rules had no test arm, so deleting the _validateViaPath/_checkAggregateCardinality
call and the entire _viaTerminalEntityNode call site still passed all six
original tests.

- Add "@via provably to-one" arm (ERR_ORIGIN_CARDINALITY via a single
  reference hop, inherently to-one).
- Add a 3-entity A.b.c chain arm proving @orderby resolves against the @via
  TERMINAL entity (C) rather than the head (A) or a middle hop (B).
- Gate the @orderby call on `hops !== undefined` — _viaTerminalEntityNode
  does not reproduce _validateViaPath's malformed-@via guard, so an invalid
  @via was emitting a second, misleadingly-scoped @orderby error alongside
  the real one.
- The three arms that previously discriminated only by the shared
  ERR_INVALID_ORIGIN code now also assert each rule's distinctive message
  fragment.

Both new arms verified by commenting out their corresponding source lines
(cardinality call; the whole hasOrderBy block) and confirming the matching
arm — and only that arm — fails, then restoring.
…via terminal (#335)

The lowering for an @of-absent origin.aggregate @agg:collect (the whole-
object rollup added in the prior commit) now projects EXACTLY the declared
value object's members, matched by name against the @via terminal entity:

- An unmatched member fails with a new code, ERR_COLLECT_MEMBER_UNRESOLVED.
  Failing open here is how #270 turned a curated value object into the full
  entity, invisible in a diff because the metadata still read as curated.
- A member whose field.<subType> differs from the matched entity field's
  fails with ERR_INVALID_ORIGIN — the object-form analogue of the scalar
  @Of path's element-type check, same #185 type-preserving doctrine.

Both sides are read with the resolving accessor (children()), never own*(),
per ADR-0039 — a value object or entity may bring members via extends, and
an own-only read would silently skip them, which is exactly the bug class
this guard exists to prevent.

ERR_COLLECT_MEMBER_UNRESOLVED is registered across all six error ledgers
(ERROR-CODES.json, TS/Python/C#/Java ErrorCode enums, and Java's separate
ErrorMessageConstants message ledger — discovered on an earlier #335 task
to be a sixth site, not five). ERR_INVALID_ORIGIN is reused, not re-added.

Two new arms added to the existing validation-collect-whole-object.test.ts
(the model helper gained an optional voFields parameter so SupplierBrief's
member list can vary per test without touching any existing call site).
Each arm's pin was verified by commenting out its corresponding check,
confirming that arm alone failed, then restoring to a clean diff.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgNA6uDN9qxu2nA471rAq
…rms (#335)

Half A conformance fixtures for the @of-absent origin.aggregate @agg:collect
whole-object rollup: one positive fixture plus eight negative arms covering
every rule the loader enforces on the @of-absent branch (the scope grew from
the brief's original six to eight — @distinct and @orderBy-terminal landed
after the brief was written). Models are lifted from the verified
validation-collect-whole-object.test.ts harness; jsonPaths and the positive
expected.json are copied verbatim from the runner's actual output, never
hand-derived.

error-collect-no-object-ref intentionally asserts two errors (ERR_INVALID_ORIGIN
from the new rule plus the pre-existing ERR_OBJECT_FIELD_WITHOUT_OBJECT_REF,
which fires on any field.object missing @objectref regardless of collect) —
both are inherent to the shape. error-collect-via-to-one's value object is
narrowed to a single id member (matching the mis-cardinality path's actual
terminal) so member-resolution can't raise a second, unrelated
ERR_COLLECT_MEMBER_UNRESOLVED alongside the intended ERR_ORIGIN_CARDINALITY.

Conformance suite: 588 -> 606 pass (+18 = 9 fixtures x lint + conformance).
Full metadata package: 2507 -> 2525 pass, 0 fail.
…335)

The per-member type-agreement check added in Task 6 cites the #185
type-preserving doctrine but implemented only half of it: it compared
`member.subType !== match.subType` and ignored array-ness entirely. A
value-object member declared `field.string` matching a terminal-entity
`field.string isArray: true` (or the reverse) passed validation, and the
lowering would project an array into a scalar member.

The sibling this rule names, `_checkPassthroughType`, already compares BOTH
axes via the `field.<subType>[]` label and carries a comment explaining why
subtype names never contain "[]", so equal labels mean same subType AND same
array-ness. This check now does the same thing, so the two halves of one
doctrine are enforced the same way.

Found by the whole-branch review. Caught before Task 8 deliberately: porting
a known-incomplete rule into three more ports and then fixing four is
strictly worse than fixing the reference implementation first.

Also corrects the ERR_FILTERABLE_UNSUPPORTED_SUBTYPE ledger description,
which still described only the subtype cause after this branch gave the code
a second trigger (array field) in all four ports. Its sibling
ERR_SORTABLE_UNSUPPORTED_SUBTYPE already documented both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgNA6uDN9qxu2nA471rAq
…even (#335)

Task 7 shipped eight rules and nine fixtures after a scope correction the ledger
recorded but this plan never absorbed: member TYPE agreement and @orderBy-resolves-
against-the-@via-terminal had no place in the original six. Following the stale text
literally would have under-ported two rules into three ports — precisely the silent
cross-port omission the shared corpus exists to prevent.

Also records the trap that makes Task 8's own signal untrustworthy: five of the eight
negative fixtures assert ERR_INVALID_ORIGIN, which is ALSO what an unported loader
emits for the old "missing @Of" error. Those five pass vacuously before any porting
work begins, so "conformance green" does not mean the rules were implemented. The
step now names the guard-neutralisation technique the Task 7 reviewer used, and
repeats the no-git-stash prohibition (worktrees share one stash list).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgNA6uDN9qxu2nA471rAq
…ct rules (#335)

All four cleanup reviewers independently flagged the same thing, and the
efficiency pass measured it: `_validateViaPath` already holds the terminal
entity in `currentObj` when it returns, and throws it away — so
`_viaTerminalEntityNode` existed only to recover it by walking the identical
path a second time, and the whole-object branch called that helper TWICE.
Three walks where one would do. On a 1103-object estate the two redundant
walks measured ~56ms of a 778ms load — 7.2%.

Worse than the cost: the duplicate re-derived the ADR-0042 package-resolution
rule by hand, so the loader carried two independent @via path-walkers that had
to be kept in agreement. The branch's own comments admitted the divergence
("_viaTerminalEntityNode does not reproduce _validateViaPath's malformed-shape
guard") and both call sites needed a redundant `hops !== undefined` gate to
paper over it. Those comments were the smell, not the invariant.

`_validateViaPath` now returns `{ hops, terminal }` — the shape
`_validateFromPath` already uses for the same reason. `terminal` is defined
exactly when `hops` is (all five early exits return undefined; the sole
success path has walked to the end), so the gate is structural rather than
commented. `_viaTerminalEntityNode` is deleted: 44 lines, and three ports that
no longer have to reimplement it in Task 8.

Also extracted, because the helper set IS the cross-port porting contract —
every port mirrors this file's `_check*`/`_validate*` decomposition 1:1, and
Java's validateOriginNode is already ~330 lines:
  - `_checkCollectMembers` — the member-resolution + type-agreement rules.
    Measured nesting in the @of-absent arm drops from 10 levels to 4.
  - `_typeLabel` — the `field.<subType>[]` idiom, previously inline in four
    places with its invariant re-explained each time. `_checkPassthroughType`
    now shares it.

Fixture fix (found by the altitude pass): error-collect-orderby-not-terminal
declared `name` only on B, the MIDDLE hop. Task 7's review proved it
discriminates middle-resolution — but a port resolving @orderby against the
HEAD (the likelier mistake, since the walk starts there) also errored and so
PASSED. Entity A now declares `name` too. Verified by mutation: resolving
against the head fails the fixture now and did not before.

Python's `@sortable` check inlined the literal "sortable" where the other three
ports used their constant; now imports FIELD_ATTR_SORTABLE.

Verified: typecheck clean, 2526 pass / 0 fail, conformance 606/0. The extracted
helper was mutation-checked (disabling it fails 3 unit tests + 2 fixtures, so
it is reachable and pinned, not dead code). Python conformance is unchanged at
4 failed / 451 passed — the known interim red for Half A, which Task 8 ports.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgNA6uDN9qxu2nA471rAq
#335 changes the registered vocabulary: `@of` becomes OPTIONAL on
`origin.aggregate @agg: collect`, which is a rule relaxation an adopter can
observe, plus eight new load errors. Under ADR-0035 Amendment 2 that moves the
metamodel contract's number, and pre-1.0 a breaking metamodel change moves the
MINOR.

It is 0.12, NOT the 0.11 the design spec's §A7 claimed. 0.11 is CLAIMED — by
#342's XOR relaxation, shipping in 0.24.1; origin/main's 2af77e7 names it the
sole claimant precisely so a parallel branch would not reason "v0.24.0 shipped
0.10, so 0.11 is free" and collide. This branch already carries #342 in its
ancestry, which is what its 0.11 was recording; #335's own change needs the
next number.

No gate would have caught this. #335's only registry footprint is `@of`'s
byte-gated DESCRIPTION PROSE, and check-metamodel-version.mjs WARNs on prose
rather than classifying it — a rule can change with no machine-readable
footprint (#210's only manifest edit was a `rules` string). Confirmed
empirically: the gate reported "declared 0.11 — ok" before this commit and
"declared 0.12 — ok" after. Human decision, made by Doug.

Written by `check-metamodel-version.mjs --set 0.12` so all five sites move
together — the manifest plus all four port constants. A partial edit only
fails in the forgotten port's registry-conformance lane, which is
release-tag/dispatch-only for three of them.

Verified: all five sites read 0.12 with no residual 0.11, gate green,
typecheck clean, metadata 2526/0, Python registry-conformance 10/10.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgNA6uDN9qxu2nA471rAq
Five of the eight negative conformance fixtures for the @of-absent
@agg:collect asserted ERR_INVALID_ORIGIN — which is ALSO what a loader that
still REQUIRES @Of emits for that exact metadata. The corpus compares only
`code` + `source` and never message text (spec/conformance-tests.md), so those
five passed vacuously against three ports that implement none of the rules:
Python conformance failed 4 of the 9, not 9 of the 9.

ERR_COLLECT_WHOLE_OBJECT now carries the five refusals raised by the
whole-object arm itself — carrier is not a field.object declaring @objectref;
@via absent; @distinct declared; an @orderby key that does not resolve against
the @via TERMINAL entity; a value-object member whose declared field.<subType>[]
disagrees with the matched terminal field's. The three arms that already had
distinct codes keep them (ERR_SUBTYPE_RULE_VIOLATION for a non-value
@objectref, ERR_ORIGIN_CARDINALITY for a to-one @via,
ERR_COLLECT_MEMBER_UNRESOLVED for an unmatched member).

The @orderby arm runs through _validateOrderByKeys, shared with the scalar @Of
arm and with origin.first, so the helper gained a trailing
`code: ErrorCode = "ERR_INVALID_ORIGIN"` parameter. Only the whole-object call
site passes the new code; the other two keep their envelope byte-for-byte, so
no existing fixture, port or adopter-visible behaviour changes.

Registered in all SIX ledgers (ERROR-CODES.json, the TS/Python/C#/Java enums,
and Java's separate ErrorMessageConstants). C#'s FixtureLint rejects an
unregistered code in a fixture, which is independent proof the registration
took in the shared ledger.

Signal verified by execution on every unported port, not by reading: Python
9 failed / 568 passed (was 4 failed / 568 passed), C# 9 failed / 979 passed,
Java 606 run / 9 failures — in each case exactly the nine #335 fixtures, so
Task 8 now has a real red-before/green-after gate per rule. TS is unchanged
green: metadata 2526/0, conformance 606/0, typecheck clean, dotnet build with
no `error CS`, `mvn -pl metadata compile` clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgNA6uDN9qxu2nA471rAq
Ports Half A's eight loader rules for an @of-absent origin.aggregate @agg:collect
to C#, Java and Python. The nine shared conformance fixtures are the contract;
all four ports now agree on every one.

The eight rules, in the order they fire: the carrying field must be a
field.object declaring @objectref (ERR_COLLECT_WHOLE_OBJECT); that @objectref
must resolve to an object.value (ERR_SUBTYPE_RULE_VIOLATION — #210's rule is
payload-scoped and never reaches a projection-hosted field, so this branch
enforces it itself); @via is required (ERR_COLLECT_WHOLE_OBJECT); @distinct is
refused (ERR_COLLECT_WHOLE_OBJECT); the @via path must be to-many
(ERR_ORIGIN_CARDINALITY); every @orderby key must resolve against the @via
TERMINAL entity (ERR_COLLECT_WHOLE_OBJECT); every value-object member must match
a terminal field by name (ERR_COLLECT_MEMBER_UNRESOLVED); and a matched member
must agree on BOTH type axes (ERR_COLLECT_WHOLE_OBJECT).

Each port mirrors the TS decomposition 1:1 rather than inlining, because the
helper set IS the porting contract and Java's validateOriginNode was already
~330 lines: WalkedViaPath (validateViaPath now returns hops AND the terminal, so
the terminal is not recovered by a second walk carrying a second copy of the
ADR-0042 package-resolution rule), _checkCollectMembers, and _typeLabel.
validateOrderByKeys gains an optional error code — Java as a 7-arg overload, C#
and Python as a defaulted parameter — so only the whole-object call site reports
ERR_COLLECT_WHOLE_OBJECT while the scalar @Of and origin.first sites keep their
envelope byte-for-byte.

ADR-0039 throughout: resolving reads for @objectref (C# Attr, Java
hasMetaAttr/getMetaAttr, Python get_meta_attr), resolving children() on both the
value object and the terminal entity, and the resolving array accessor inside
the type label (C# ResolvedIsArray, Java isArrayType, Python resolved_is_array).
@via stays an OWN read — origin.* never inherits (ADR-0029).

Green is not the evidence here, and this cut does not rely on it. Every one of
the 24 (8 rules x 3 ports) guards was neutralised individually against a backed-up
copy and the corpus re-run: in all 24 cases EXACTLY the one fixture that gates
that rule failed, and no other. Each file was restored and `diff -q` confirmed
byte-identical to its backup afterwards; no `git stash` was used, since worktrees
share one stash list.

Two C# mutations initially tripped CS0162 under warnings-as-errors, so those were
re-run with a runtime-opaque condition rather than a statically dead one — a
detail worth keeping: a neutralisation that will not compile proves nothing.

Suites: Python 1849 passed / 0 failed; C# 1696 passed / 0 failed across all four
test projects with no `error CS`; Java 1494 run / 0 failures / BUILD SUCCESS;
TypeScript unchanged at 2526 / 0 with typecheck clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgNA6uDN9qxu2nA471rAq
…copies (#335)

Until now the loader was LOOSER than the contract it ships: five ports accept an
@of-absent @agg:collect while every registry copy still said @Of was "Required
for count/sum/avg/min/max/collect".

Five prose strings change, byte-identically across the copies that carry them:

- @Of — required for count/sum/avg/min/max; OPTIONAL for collect, where absent
  means a whole-object rollup of the field's declared @objectref value object.
- @agg and the origin.aggregate type description — collect is an array rollup of
  the @Of column OR of the carrying field.object's value object.
- @distinct — not supported on a whole-object collect, with the reason (a
  guaranteed no-op whenever the value object carries the primary key).
- the origin.aggregate `rules` string — the whole rule, including the one thing
  a port could not infer from the others.

That last point is the reason this is not a one-line edit. The VO-member↔column
binding is NAME MATCHING (member name == terminal field name), deliberately not
`extends` — which keeps one value object collectable from two different entities,
something `extends` would forbid. It is a NEW implicit convention in a metamodel
whose stated doctrine is "extends is THE mechanism", so leaving it undocumented
would have left four ports to each infer it from the TypeScript. It is now
written into the byte-gated contract, along with the @via-terminal @orderby scope
and the both-axes type agreement.

Six copies carry the long form and are regenerated or edited in lockstep;
`server/csharp/.../Origin/OriginSchema.cs` carries its own shorter wording (it is
a C# runtime schema, not part of the byte-gated manifest) and is edited to match
in substance. `expected-registry.json` and the `metamodel-docs` fixtures are
GENERATED — emitted from the TS registry and by `scripts/regen-metamodel-docs.ts`
rather than hand-edited; the regenerated manifest differs from the committed one
in exactly the five intended strings and nothing else.

§A7 of the design spec is corrected in the same commit, since it documents this
same contract. It claimed the change rode #342's unreleased `metamodelVersion`
0.11 and needed no edit. The re-check that section itself demanded then fired:
origin/main's 2af77e7 names 0.24.1 as the sole claimant of 0.11, so #335 needs
0.12 — already shipped in 7c99536a7. The superseded reasoning is recorded rather
than deleted, because a reviewer independently re-derived it and called it
correct: "v0.24.0 shipped 0.10 and main already carries the unreleased bump" is
true and still wrong, because it never asks who else has claimed 0.11.

Registry conformance green in all four ports (TS 8/0 — the only runner that
compares description prose; Python 84/0; C# 25/0; Java RegistryManifest 3/0,
which compares structure rather than prose). TS metadata 2526/0.
`check-metamodel-version.mjs` reports the four prose changes as WARN and passes:
"additive since v0.24.0; declared 0.12 — ok".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgNA6uDN9qxu2nA471rAq
…ollup (#335)

CLOSES THE MERGE GATE. Since the loader started accepting an @of-absent
@agg:collect, `extract-view-spec.ts` still bailed on `if (!of_) continue;` ABOVE
the collect branch — so the metadata LOADED clean and codegen silently DROPPED
the column. Proven empirically before this commit: the positive conformance
fixture extracted `columns = [productId]` with `supplierBriefs` simply gone, i.e.
a CREATE VIEW missing a column the generated type still declares. This branch was
not mergeable until this landed.

A new `collectObjectAgg` SelectColumn kind carries a MEMBER LIST rather than one
source column — deliberately a separate kind, not an arm of `collectAgg`: a union
would force `isInflationSensitive` and `buildGroupBy` to re-narrow. Both are
updated; a whole-object collect is unconditionally inflation-sensitive, because
the loader refuses @distinct on this form, so it can never be the distinct case
that makes `collectAgg` safe.

Extraction resolves the related entity from @via's TERMINAL hop, exactly as the
any/all arm does — there is no @Of entity. The value object's declared member list
IS the exposure: a field the terminal entity has but the VO omits is not
projected, which is the #270 guarantee this rollup has to keep. The JSON key is
the MEMBER name while the value reads the terminal entity's PHYSICAL column; those
differ whenever a field carries @column, and a test pins that they stay distinct.

SQL: `jsonb`, not `json`. Verified against a real PG 15 — `json` has neither an
equality nor an ordering operator, so the `json_agg(json_build_object(…) ORDER BY
…)` form does not run at all. Postgres gets
COALESCE(jsonb_agg(jsonb_build_object(…) ORDER BY <pk> ASC) FILTER (WHERE <pk> IS
NOT NULL), '[]'::jsonb); SQLite gets json_group_array(json_object(…)) with
json_array() as the empty value. In-aggregate ORDER BY needs SQLite >= 3.44, which
is not a new constraint — the scalar collect already emits it and D1's baseline is
pinned there.

Default element order is the related entity's PK ascending, not "value ascending":
ordering rows by a serialized object is meaningless. An explicit @orderby leads
with the PK appended as a tie-break. The SCALAR arm keeps its no-tie-break
behaviour deliberately — adding one would change emitted SQL for every existing
project using @orderby — and three no-churn tests pin that asymmetry, byte for
byte, in the same file.

Tasks 10 and 11 of the plan are ONE commit rather than two. Splitting them would
leave an intermediate commit that does not BUILD: adding the union member without
its emit branch fails `tsc` with TS2345 where view-ddl-emit falls through to
renderFirst. That exhaustiveness failure is a feature of the union — it is what
guarantees no kind can be added and silently unlowered — but it makes the split
commit a bisect trap.

codegen-ts 1312 pass / 0 fail (7 new DDL + 5 new extract), package typecheck clean,
whole-workspace `bun run --filter '*' build` and `typecheck` clean. The two
issue-214-read-half-compile failures seen mid-task were the documented
unbuilt-workspace precondition (missing `@metaobjectsdev/runtime-ts/drizzle-fastify`),
not a regression — they pass after the build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgNA6uDN9qxu2nA471rAq
…SQLite (#335)

Golden SQL is not evidence for new DDL, and this cut is the proof of why. Both
view-lifecycle probes gain a `WeekBrief` value object and a `weekBriefs` column
carrying an @of-less collect, beside the existing scalar `weekLabels` collect so
both arms run through one view on one engine. Each asserts the full
emit -> apply -> introspect -> re-diff-is-EMPTY convergence gate, then the actual
values: two related rows give an array of objects in related-PK order, zero
related rows give [] and not null.

The SQLite emitter is CHANGED here, by what the probe found. On SQLite 3.44.0 —
D1's pinned baseline — the in-aggregate ORDER BY clause destroys the JSON
subtype:

  json_group_array(json_object(...))                   -> nests correctly
  json_group_array(json_object(...) ORDER BY ...)      -> array of QUOTED STRINGS
  json_group_array(json(json_object(...)) ORDER BY ...)-> array of QUOTED STRINGS

So the column came back as ["{\"id\":1,...}", ...] — an array of strings — while
the generated type declared an array of objects, and a json() wrapper on the
argument does not survive the ORDER BY either. Dropping the ORDER BY was not an
option: element order would stop being deterministic and an author's @orderby
would silently do nothing. The emitter now builds the ordered array first and
re-wraps it element by element through json_each, which iterates in array order,
so the ordering survives while json(value) restores each element's subtype. Still
a grouped LEFT JOIN — unlike origin.first it needs no correlation info, so a
multi-hop @via lowers exactly as a single-hop one does. Postgres was already
correct and is verified, not assumed by symmetry.

The unit test in codegen-ts is updated to pin the new SQLite shape, so the two
tiers cannot drift.

Second finding, pinned rather than normalised away: a `field.long` member arrives
inside the rollup as a JSON NUMBER, while the same value as a top-level BIGINT
column arrives from node-postgres as a STRING. That asymmetry is inherent to JSON
— jsonb has no bigint — and is lossy above 2^53. Both id types are asserted
explicitly with a comment, so a future change to either tier has to face it.

Ran against a throwaway Testcontainers Postgres, not a local one: this box has
sibling-project Postgres containers up, `METAOBJECTS_TEST_PG_URL` was confirmed
unset, and startPostgres() provisions its own `metaobjects-test-<uuid>` container
on a fresh port. view-lifecycle-pg 17/0, view-lifecycle-sqlite 1/0, codegen-ts
projection suite 148/0, package typecheck clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgNA6uDN9qxu2nA471rAq
… replaced, not deleted (#335)

The retirement guide's "delete the child and change nothing else" advice is
correct for a payload or object.value host and SILENTLY WRONG for a view-kind
projection: a projection field with no origin.* is treated as a plain base
column, so the emitted DDL becomes `SELECT base."supplierBriefs"` against a
column that does not exist. The metadata loads clean; the failure surfaces at
apply. §2 now splits by host — 2a keeps the delete for payload/value hosts, 2b
replaces the child with `origin.aggregate @agg: collect @via: …` for projections
and names the three constraints the old subtype did not have (@via required,
@distinct refused, members must match the @via terminal on both type axes), each
of which is a load error, so a mistake there fails loudly.

The section that stated the gap as a future promise is replaced by the migration
itself, now that #335 has shipped.

fixtures/conformance/README.md's "coverage genuinely lost" entry is closed with
the reasoning, not just a status flip: the array-of-value-object-carrying-an-origin
shape lives in the dedicated `collect-whole-object` fixture rather than back in
`flattened-kitchen-sink`, because that fixture's concern is `@storage: flattened`
and a whole-object rollup exercises none of it — folding them would make each
harder to read in a five-language investigation. It also names where the shape is
round-tripped against real engines.

TS conformance 606/0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgNA6uDN9qxu2nA471rAq
…will look (#335)

The CHANGELOG entry leads with the gap rather than the feature: a projection
could roll related rows up into an array of one COLUMN and had no way to roll
them into an array of OBJECTS, so the shape needed a second round-trip or a
hand-written view — unmanaged, and therefore invisible to `meta verify --db`. It
records the rulings that will otherwise be re-litigated: why the declared value
object is the exposure and binds BY NAME rather than by `extends`; why @distinct
is refused by choice rather than engine limit; why the new error code exists at
all (the corpus compares code + source and never message text, so five negative
fixtures passed against three ports containing none of the rules); why jsonb and
not json; why SQLite needs the json_each re-wrap; and the bigint-inside-jsonb
asymmetry an adopter will hit above 2^53. Half B's array filter/sort rules are
recorded in the same entry.

The authoring skill is updated in all SIX copies (the source plus its five
byte-gated conformance mirrors). That matters for a specific reason: #342 shipped
with the skill teaching a form the loader had just made ILLEGAL. This is the same
failure in the other direction — a skill that cannot express the new form leaves
adopters reaching for a hand-written view, which is exactly what the feature
exists to remove. agent-context-conformance 5/0.

`downstream-metadata-decisions.md` gains it as a Step 0 entry rather than a
feature note, because "I need an array of objects" is the case most often
mistaken for missing vocabulary: the @Of form reads as the only form, and the
natural next thought is a custom view.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgNA6uDN9qxu2nA471rAq
@dmealing
dmealing merged commit 18d7cd9 into main Aug 24, 2026
1 check passed
@dmealing
dmealing deleted the feat/issue-335-whole-object-rollup branch August 24, 2026 00:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FR: @agg: collect with @of optional — whole-object rollup (the designated origin.collection re-entry shape)

1 participant