[akbashev]

Builds on #336. Only the last commit, "Report the system model's usage from Foundation Models 27", belongs to this PR.

Why

The system model reports no usage: its responses and snapshots are always zero. From OS 27, Foundation Models reports usage, and with #336 ALM reports it the same way: a response's usage is its final request, and session.usage counts every request.

What

• From OS 27, a response's and each snapshot's usage is Foundation Models' usage for the final request.
• The earlier requests come from the Foundation Models session's usage. The system model makes a session for each request, so its usage is that response's total. ALM's session.usage counts every request, as Foundation Models' does.
• When the delegate stops at a tool call, the response has no Foundation Models usage of its own. The session's usage when each call started tells the stopping round from earlier ones: calls of the same round all see the same usage.
• Before OS 27, on compilers before Swift 6.4, and on tvOS, where Foundation Models has no usage, the counts stay zero.

Measured on macOS 27 with a tool round: the response reported 186 input and 29 output tokens, and session.usage 320 and 49, the first request's 134 input tokens plus the final request's 186.

Tests

• stopSplitsTheStoppingRoundFromEarlierOnes, with no model: a stop in the second round, after two calls in the first, and a stop in the first round.
• On the device, when the system model is available:
  • responseReportsTheFinalRequest.
  • streamedResponseReportsTheFinalRequest: the running total never goes down.
  • stoppedResponseCountsItsRequests.

[akbashev]

Rebased onto main now that #336 is merged.

[akbashev]

@mattt
Rebased onto main after #357, and the system model now counts its requests the way #357 does: each round's usage is reported as the next tool call starts, so a response that throws after a round (a failing tool, say) keeps that round's usage. Test: failedResponseCountsTheRoundsItCompleted, which fails on the rebase alone (the session counts 0) and passes with the change.

[mattt]

Thanks, @akbashev. Counting each round as the next tool call starts looks right to me.

One case still loses usage, though: the final round is only counted when the response returns. If anything throws after Foundation Models returns — a model that wraps this one, say, or structured content that fails to convert — that round's usage is dropped, and for a response with no tool calls, that's all of it. The Anthropic and Ollama models report each request as it completes, which is what RequestUsageChannel expects. Could you report the remaining session usage as soon as Foundation Models returns, before the content is converted, and add a test for a response that throws after that?

Non-blocking: failedResponseCountsTheRoundsItCompleted passes without checking anything when the model answers without calling the tool, and its catch accepts any error.

[akbashev]

Thanks, @mattt. Done: once Foundation Models returns, the rounds' remaining usage is reported before the content is made, so a response that throws after that — a wrapping model, or content that fails to convert — keeps it. responseThatThrowsAfterFoundationModelsReturnsKeepsItsUsage wraps the system model in one that throws once it returns, for text and structured content; both counted nothing before the change.

failedResponseCountsTheRoundsItCompleted now expects the tool's own error, so it fails if the model answers without calling the tool. Rebased onto main.