ci(macrobenchmark): Run the startup benchmark on Sauce Labs and recover its results - #5909
Open
runningcode wants to merge 15 commits into
Open
ci(macrobenchmark): Run the startup benchmark on Sauce Labs and recover its results#5909runningcode wants to merge 15 commits into
runningcode wants to merge 15 commits into
Conversation
The sentry-uitest-android-macrobenchmark module currently only runs on a locally connected device. This wires it to Sauce Labs so we can evaluate whether the cold-start timeToInitialDisplay benchmark can run on the real-device cloud already used by our other benchmarks. This is a proof-of-concept, gated behind workflow_dispatch and kept off the per-PR path. Device-guard errors are intentionally not suppressed: on non-rooted, unlocked-clock cloud devices the guards (UNLOCKED, DEBUGGABLE, ...) are expected to fire, and seeing which ones fire is the point of the POC. It also tells us whether timeToInitialDisplay can be retrieved from Sauce artifacts (benchmark JSON) or must be parsed from the device log. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drop the assembleMacrobenchmark Makefile target and invoke the two gradle assemble tasks directly from the workflow. Only one workflow uses them, so the extra indirection isn't worth it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
saucectl cannot retrieve Macrobenchmark's output. Its artifacts.download.match filters a hardcoded list of assets Sauce hosts for a job -- device.log, junit.xml, video.mp4, network.har, crash.json, screenshots.zip -- and never reads the device filesystem, so benchmarkData.json and the per-iteration perfetto traces stay stranded on the device. The POC's "*-benchmarkData.json" match line could never have matched anything; drop it. Drive the run through the Real Device Access API instead, which exposes adb shell and pullFile: reserve a device, install both APKs, run the instrumentation, then pull the results off. This also recovers the perfetto traces, the only way to resolve sub-millisecond SDK changes that timeToInitialDisplay cannot see. The API is an open-beta add-on, so the workflow probes entitlement before spending a Gradle build on it, and defaults to a one-iteration smoke run.
…rride App Storage upload answers 201, not 200, so the strict status check rejected every upload before a device was even reserved. Also drop --iterations: androidx.benchmark.iterations only feeds the microbenchmark path (BenchmarkStateLegacy, MicrobenchmarkPhase). Macrobenchmark reads its iteration count from the test source and ignores the argument, so skipping AOT compilation is the only way to shorten a run that still writes results -- dryRunMode would shorten it but forces outputEnable to false, producing no benchmarkData.json to retrieve. Track installations by installationId rather than by the app reference, which Sauce may echo back normalised.
…g a session The first run reached the API but matched zero devices: Real Device Access descriptors are not the ids used in .sauce/*.yml, which carry an OS version and a region suffix (Google_Pixel_9_Pro_XL_15_real_sjc1). Fetch the catalog unfiltered and match client-side against both descriptor id and name, then open the session with the resolved concrete id rather than a pattern. On no match, print every available Android device so a naming mismatch is diagnosable from a single run -- the endpoint's own deviceId filter cannot distinguish "no devices on this account" from "your pattern matched nothing". Also run the no-match check before --probe-only returns. It previously came after, so the probe reported success and the real run failed on it minutes later.
An empty catalog answers 200 exactly like a populated one, so the first probe reported the API as reachable while there was in fact nothing to run on. Report device availability and the other regions' catalogs too, so an unentitled account is distinguishable from a misconfigured region.
On a public-cloud account /devices answers 200 with an empty list, which made the probe look like a missing entitlement. /devices/status returns the whole fleet -- 392 devices, all public, 368 available -- and its descriptors are exactly the ids already used in .sauce/*.yml, so the original Google_Pixel_9_Pro_XL_15_real_sjc1 is back as the default. Prefer AVAILABLE devices when a pattern matches several, and report each match's state so an IN_USE device explains itself.
The Real Device Access API can pull files off a device, but it cannot reserve one on a public-cloud account: POST /sessions rejects every request with deviceClasses=[PRIVATE_DEVICE], and the request body has no way to ask for a public device. Everything up to reservation worked -- App Storage upload, device resolution from /devices/status, the session request itself -- so this is a commercial limit, not a technical one. Revert to saucectl and drop the script; git history has it if we ever lease private devices, since that route would also return the per-iteration perfetto traces. Instead have the benchmark echo its own benchmarkData.json into logcat in chunks, which reaches CI as device.log, the one asset Sauce does hand back. scripts/parse-macrobenchmark-log.py reassembles it, validates the JSON, and writes a timeToInitialDisplay table to the job summary.
The recovery worked on the device but not in the parser. Sauce returns
device.log as JSON lines -- {"tag", "message", "level", ...} -- so the payload
arrives with its quotes escaped and cannot be regexed straight out of the raw
line. Decode each entry and match against its message, falling back to the raw
line so the same parser still handles `adb logcat` output from a local run.
Enable pipefail in the workflow step too. Piping into tee meant the step exited
on tee's status, so this very failure reported success while producing no
results.
Verified against the device.log from run 31117974318: timeToInitialDisplayMs
min 443.5 / median 477.3 / max 571.2 over 12 iterations on a Pixel 9 Pro XL.
Document the three routes that do not work -- artifacts.download.match, Macrobenchmark's own reporting channels, and the Real Device Access API on a public-cloud account -- so the next person does not re-derive them.
Chunks were merged across every *.log found, so two devices each emitting the same number of chunks would overwrite each other's indices and reassemble into one bogus document. Collect per file and report when more than one log carries results, which also drops the cross-file total comparison that was standing in for this check. Build the summary table and per-iteration details in one pass while here.
… env The repo has no submodules -- no .gitmodules, no gitlink entries -- so submodules: 'recursive' does nothing; sentry-native-ndk is a Maven dependency pinned in gradle/libs.versions.toml. GRADLE_ENCRYPTION_KEY is never read as env.* or $GRADLE_ENCRYPTION_KEY either; setup-gradle takes the secret directly via cache-encryption-key, which stays.
📲 Install BuildsAndroid
|
Older hardware would resolve SDK-init regressions better, but Google_Pixel_4_13_real_us could not be allocated in three attempts -- each queued ~61 minutes before Sauce gave up with a concurrency retry-timeout, leaving an empty device log and tests="0". The Pixel 9 Pro XL allocates reliably and is the device this pipeline is proven green on.
runningcode
force-pushed
the
no/macrobenchmark-sauce-results
branch
from
August 7, 2026 12:51
9bf03e7 to
c1cd5bd
Compare
Contributor
Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 6b019b7 | 319.84 ms | 333.15 ms | 13.31 ms |
| d15471f | 322.58 ms | 396.08 ms | 73.50 ms |
| b750b96 | 421.25 ms | 444.09 ms | 22.84 ms |
| d217708 | 411.22 ms | 430.86 ms | 19.63 ms |
| 604a261 | 380.65 ms | 451.27 ms | 70.62 ms |
| 5b1a06b | 352.27 ms | 413.70 ms | 61.43 ms |
| fcec2f2 | 357.47 ms | 447.32 ms | 89.85 ms |
| 7414e9b | 370.39 ms | 422.18 ms | 51.79 ms |
| fcec2f2 | 311.35 ms | 384.94 ms | 73.59 ms |
| f6cdbf0 | 314.19 ms | 357.59 ms | 43.40 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 6b019b7 | 0 B | 0 B | 0 B |
| d15471f | 1.58 MiB | 2.13 MiB | 559.54 KiB |
| b750b96 | 1.58 MiB | 2.10 MiB | 533.20 KiB |
| d217708 | 1.58 MiB | 2.10 MiB | 532.97 KiB |
| 604a261 | 1.58 MiB | 2.10 MiB | 533.42 KiB |
| 5b1a06b | 0 B | 0 B | 0 B |
| fcec2f2 | 1.58 MiB | 2.12 MiB | 551.50 KiB |
| 7414e9b | 0 B | 0 B | 0 B |
| fcec2f2 | 1.58 MiB | 2.12 MiB | 551.51 KiB |
| f6cdbf0 | 0 B | 0 B | 0 B |
…ated call @Suppress("DEPRECATION") covered the whole candidateDirs expression, including externalCacheDir, which is not deprecated -- so a future deprecation there would have been silently swallowed. Scope it to getExternalMediaDirs and record why it cannot be migrated: it is deprecated in favour of MediaStore, which returns content URIs rather than the filesystem path androidx.benchmark writes to.
The retrieval rationale these comments carried is documented in the module README, so removing the duplicated prose loses nothing. Also drops cache-encryption-key from the Setup Gradle step, leaving this workflow without the Gradle configuration-cache encryption the other workflows still configure.
runningcode
commented
Aug 7, 2026
| # Temporary scaffolding: workflow_dispatch cannot target a workflow that does not exist on | ||
| # the default branch yet, so trigger on pushes to this branch while we validate the flow. | ||
| # Remove before merge. | ||
| push: |
Contributor
Author
There was a problem hiding this comment.
We want to remove this before merging and just use workflow_dispatch until we decide how we want to use this.
runningcode
marked this pull request as ready for review
August 7, 2026 14:36
runningcode
requested review from
0xadam-brown,
adinauer,
markushi and
romtsn
as code owners
August 7, 2026 14:36
runningcode
added
the
sanity-check
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📜 Description
Runs the
sentry-uitest-android-macrobenchmarkcold-start benchmark on a Sauce Labs real device and gets the numbers back into CI. See the results here.In a future PR, we will either decide to post this as PR comments OR run this as a pre-release task against the last relase. But for now I'm just getting the mechanism working.
Q: Why is this so complicated?
A: Sauce does not provide any APIs to pull files off the devices unlike firebase test lab or other tools. Here's a link to Sauce docs in case you find something I didn't. So to work around this we echo the
benchmarkData.jsoninto logcat in chunks and sauce does give us back adevice.log.scripts/parse-macrobenchmark-log.pyuses that to reassemble thebenchmarkData.jsonand parse it.The things my clanker tried are documented in the README of the module so others can check/verify.
Note: sauce labs devices run with unlocked CPU clocks so the benchmarks will be unstable!!
💡 Motivation and Context
Get a reliable Sdk init benchmark!
💚 How did you test it?
Green run on a Pixel 9 Pro XL (api 35) — 31159874391 — recovered all three metrics over 12 iterations:
📝 Checklist
sendDefaultPIIis enabled.🔮 Next steps
Before this leaves draft:
push:trigger on this branch from the workflow — it only exists becauseworkflow_dispatchcannot target a workflow absent from the default branch. It is markedRemove before mergein the file.if-no-files-found: erroron the upload step. A failed Sauce run currently uploads nothing while the step reports success.Follow-ups, out of scope here:
#skip-changelog