feat(io): batched footer resolve on rest_ioctx - #188
Open
ran-yuan-rui wants to merge 9 commits into
Open
Conversation
Resolving N objects' parquet footers costs N independent blocking suffix probes, each on a fresh TCP+TLS connection (the synchronous metadata path shares DNS/TLS-session state but not live connections). Add rest_ioctx::resolve_footer_objects(paths, on_result, stop): batched submission with streamed per-entry completion. One curl multi driven on the caller's thread carries every probe (and HEAD fallback) of a batch, reusing its pooled connections across entries, with per-entry semantics identical to open_io_object(path, parquet_footer_probe): verified-206 window, 200/416/unverifiable-206 HEAD fallback, the same retry policy per entry, per-attempt re-authorization and ETag capture, and the same perf-snapshot attribution. Each result carries a stashless io_object (size + validation tag) plus the footer window as a separate payload whose buffer is a lease on an ioctx-wide byte budget: bytes return when the buffer is freed, bounding resolve-ahead memory without attaching budget-held state to long-lived objects. While any transfer is active the engine acquires budget non-blockingly; a blocking, stop-aware wait happens only at zero active transfers. Batches FIFO-serialize per ioctx; a queued batch cancels out of the queue without side effects. Delivery is exactly-once per input occurrence on the caller's thread; cancellation aborts in-flight transfers and delivers one operation_canceled per undelivered entry; a throwing callback cancels the remainder and rethrows the first exception after the sweep. Config: footer_resolve_max_inflight (default derives n_reactors * max_connections; 0 disables the API) and footer_resolve_stash_budget (default 2 * inflight * footer_probe_bytes). admission_control gains try_acquire plus reserved/peak accessors, and the ioctx perf snapshot reports the budget's live/peak bytes.
…gauges The connection-reuse case retains all twelve footer payloads in its results while sizing the stash budget at four windows; under the payload-is-the-lease contract the resolve call then blocks waiting for bytes only its own caller could free. Size the budget to the retained payload count. The budget case now also asserts the snapshot gauges directly (reserved == peak == budget while payloads are held; reserved drops to zero with peak still at budget after release), and the scheduling-loop checks use fixed assertion counts so the suite's case and assertion totals stay stable.
Memory now tracks the ledger: every fallback, terminal, and cancel path destroys the probe buffer before releasing its budget reservation, probe buffers reserve exactly the window up front so growth can never exceed the lease, and an explicit footer_resolve_stash_budget smaller than footer_probe_bytes is rejected at submission (a sub-window budget cannot be honored as a hard cap). Entry submission is exception-safe: an authorizer or curl setup failure becomes that entry's error and its siblings continue, and an unwind guard detaches any easy handle still attached to the batch multi before the owning entries are destroyed. Path parsing failures likewise become per-entry errors instead of aborting the batch with no callbacks. The event loop drops its full-vector scans for a backoff counter plus a deadline min-heap, giving O(N + R log N) scheduling instead of a worst-case O(N^2), and the per-submit clock read only happens when perf_instrumentation is on.
Three cases on the batched footer resolve: multiple malformed-206 objects under a two-window budget complete alongside healthy siblings with the reservation gauge back at zero; an authorizer that throws for one key delivers that exception to that entry alone, with no GET issued for it and every sibling succeeding; and an unparsable URI in the batch is isolated as that entry's error, while an explicit stash budget smaller than the probe window is rejected at submission. The loopback harness gains per-key malformed-206 scripting and the mock authorizer gains per-key exception injection.
After the first callback exception nothing may be delivered as success: completion draining, new-entry admission, and retry resubmission all stop at the first recorded throw (or a stop request), leaving undrained completions to the cancel sweep. A driver failure mid-batch (a curl multi error) now runs the same sweep before rethrowing, so every undelivered entry still receives exactly one canceled result on every exit path. An explicit footer_resolve_stash_budget of zero is no longer silently treated as the derived default — it falls under the sub-window rejection like any other too-small budget. And a zero footer_probe_bytes now matches the single-probe path exactly: batch entries skip the suffix GET and start at the HEAD fallback, with no lease taken.
…nobs A deterministic GET barrier on the loopback server holds three requests and releases them together, so a callback that throws on the first delivery faces simultaneously-ready siblings: the case pins one success, every remaining entry canceled, only the initial three GETs on the wire, the original exception rethrown, and no callback after return. An explicit zero stash budget throws before any authorization or network request, falling under sub-window rejection rather than reading as the derived default. And a zero probe window matches the single-probe path: no GET and no reservation per entry, one HEAD each, with size and quoted ETag from the HEAD and a null footer.
mock_authorizer is a test-only double, but it lived under include/, so the install rules shipped it to every consumer and it surfaced in the public API docs. Its only in-tree users are three test files, and its peer test fixture (loopback_range_server.hpp) already lives under test/io/rest. Pure rename plus the three include-path updates; no content or CMake changes -- fresh installs no longer include it once it is moved out of include/.
ran-yuan-rui
marked this pull request as ready for review
August 23, 2026 05:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The existing footer-probe path starts with a blocking suffix GET on a short-lived easy handle. The process-wide CURLSH shares DNS and TLS-session state, but not its connection cache, so repeated probes each create a new TCP/TLS connection.
This adds
rest_ioctx::resolve_footer_objects(paths, on_result, stop). A per-call curl multi keeps at mostfooter_resolve_max_inflighttransfers active and reuses connections while the call is running. Results are delivered one input at a time, serially on the caller's thread, without waiting for the whole batch. This PR adds the cuCascade mechanism only; Sirius scan-preparation wiring will follow separately once the sirius-side io parity settles.Each entry follows
open_io_object(path, parquet_footer_probe)behavior. A verified 206 returns the suffix window. Responses that cannot establish a valid suffix window fall back to HEAD. A zero-byte probe window uses HEAD directly. Retries re-authorize each attempt and preserve ETag and perf-counter attribution.Each input occurrence receives exactly one callback, including duplicate paths. Per-entry failures do not cancel siblings. Cancellation reports
operation_canceledfor entries not yet delivered. If a callback throws, remaining entries are canceled before the first exception is rethrown. Invalid submissions fail before any callback or network request.The returned
io_objectcarries the path, object size, and validation tag, but not the footer bytes. Footer bytes are returned in a separate payload backed by an ioctx-widefooter_resolve_stash_budgetlease. The reservation is released when the payload is freed. Concurrent calls are FIFO-serialized per ioctx, andfooter_resolve_max_inflight = 0disables the API.Supporting changes:
exec::admission_controlgainstry_acquire(),reserved(), andpeak_reserved().rest_perf_snapshotreports current and peak footer-payload reservations.The PR also moves
mock_authorizer.hppfrominclude/cucascade/io/rest/totest/io/rest/. It is a test-only fixture that was being installed and included in public API documentation. The move preserves its contents and updates its three in-tree includes; the move itself requires no CMake change. Fresh installs no longer contain the header.Validation
The 20 footer-resolve cases, all failing before the mechanism landed, cover:
cucascade_io_tests: 135/135 green, 1624 assertions.Sirius rebuilt successfully with cuCascade
c0a2607onintegrate_cucs_ioat8e7ce16b.make s3-testpassed 90 of 92 cases; the two remaining failures are the existing retry-log observability cases tracked in #175. The later test-only header move does not affect that result.