Repository navigation
Fingerprint telemetry exceptions by function names, not module paths - #4086
Merged
Merged
Conversation
The daily telemetry error report keys each stack frame in an exception's fingerprint by its module path and function name, so moving a function to another module changes the fingerprint. These tests describe keying each frame by its function name alone, without leading underscores, under fingerprint algorithm 2: - the same exception raised through functions that moved to other modules gives the same fingerprint, and so does a helper renamed from a private `_name` to a public `name`; - a renamed function still gives a different fingerprint (this passes already); - frames are still chosen by their paths: the three innermost frames in application packages, or of any kind without them; - the issue state and details name fingerprint algorithm 2; - an issue the job opened under algorithm 1 is neither updated nor reopened, and a new issue opens for the failures instead; - an algorithm 1 issue whose title someone changed to the new fingerprint is matched by its title and rewritten with an algorithm 2 state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The daily telemetry error report keyed each stack frame in an exception's fingerprint as `module path:function name`. Moving a function to another module changed the fingerprint, so the group's issue stopped being updated and a duplicate opened. The recent file-length splits moved many functions into new modules, and made some private helpers public. - `normalization.fingerprint_frames` still chooses the three innermost frames whose paths are in application packages (or the three innermost of any kind without them), but keys each by its function name alone, without leading underscores. `frame_key`, which returned `path:function`, is now `frame_location`, which returns the path and function separately; the frame extraction tests compare those pairs. - `FINGERPRINT_ALGORITHM` is 2. The algorithm number is hashed into every fingerprint, so message and failed-request groups get new fingerprints too, although their inputs are unchanged. - The sync needs no change. It matches an issue by its state comment only when the state names the current algorithm, and otherwise by the fingerprint in its title, which still holds the algorithm 1 value. So algorithm 1 issues are never updated, reopened or commented on, and a group that still fails opens a new issue. - The observability README describes the new frame keys and what happened to algorithm 1 issues. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4086 +/- ##
===========================================
+ Coverage 50.82% 87.83% +37.01%
===========================================
Files 254 270 +16
Lines 14635 16058 +1423
Branches 2339 2626 +287
===========================================
+ Hits 7438 14105 +6667
+ Misses 6570 1284 -5286
- Partials 627 669 +42 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4085
Fixes #4071
#4089 is stacked on this PR. It changes which record represents a failed simulation worker, which moves some failures between groups, and relies on this PR's algorithm 2 instead of needing algorithm 3, so it should merge right after this one.
What was wrong
The Public API's daily telemetry error report fingerprints an exception by its service family, its exception class and its three innermost stack frames in application packages. Each frame was keyed as
module path:function name, so moving a function to another module gave every exception raised through it a new fingerprint: the group's issue stopped being updated and a duplicate opened. The recent file-length splits moved many functions into new modules, and made some private helpers public.What changed
normalization.fingerprint_framesstill chooses the three innermost frames whose paths are in application packages (or, without any, the three innermost of any kind), but now keys each one by its function name alone, with leading underscores removed._resolve_regionandresolve_regiongive the same key. Any other rename still changes the fingerprint, which is accepted. Two functions with the same name in different modules now give the same key; with three frames, the exception class and the service family also in the fingerprint, this should rarely merge different failures.frame_key, which returnedpath:function, is replaced byframe_location, which returns the normalized path and the function name separately, andfunction_key, which removes the leading underscores.FINGERPRINT_ALGORITHMis now 2. The issue state comment and the issue details record it.gcp/observability/README.mddescribes the new frame keys and what happens to issues from algorithm 1.changelog.d/telemetry-fingerprint-function-names.changed.mddescribes the change.Message and failed-request groups
Their fingerprint inputs are unchanged. Their fingerprints still change, because the HMAC input starts with
FINGERPRINT_ALGORITHM, the issue state records one algorithm number for every kind, and the sync matches an issue's state only when it names the current algorithm. Keeping them would need a separate algorithm number per kind, which isn't worth it for the two such issues that exist (#4067 and #4070).Issues opened under algorithm 1
Five exist: #4067 (message group), #4068 (closed as completed), #4069, #4070 (failed-request group) and #4071.
sync._issue_fingerprintreads an issue's fingerprint from its state comment only when the state names the current algorithm; otherwise it reads the fingerprint at the end of the title. The job never changes titles, so these issues are indexed under their algorithm 1 fingerprints, which no algorithm 2 group produces. As a result:/<country_id>/report(20e14b2f6c33dcfa) #4068 won't reopen if itsStaleDataErrorreturns. A new issue opens instead, without theregressionlabel.Someone can close the five by hand. Alternatively, an issue can be carried over by replacing the fingerprint at the end of its title with its group's algorithm 2 fingerprint: the job then matches it by its title and rewrites its marked section with an algorithm 2 state. A test covers this.
Testing
The first commit adds the tests and they fail; the second commit is the fix.
test_records.py: an exception raised through functions that moved to other modules gives the same fingerprint, as does a helper renamed from_resolve_regiontoresolve_region, in its old or new module. A renamed function (_find_region) still gives a different fingerprint; this passes before and after. The exception's fingerprint inputs are now the function names.test_normalization.py: the three existing frame-selection tests expect function names, and a new test checks that leading underscores are removed (__getitem__becomesgetitem__). Frames are still chosen by path: library frames are skipped when there are application frames.test_rendering.py: the state comment and the issue details name algorithm 2.test_sync.py: an open, or completed and closed, issue from algorithm 1 for the same failures is neither updated, reopened nor commented on, and a new issue opens. An algorithm 1 issue whose title was changed to the new fingerprint is updated and gets an algorithm 2 state.frame_keytoframe_locationin the six frame-extraction tests, which now compare (path, function) pairs. Their expected paths and functions are unchanged.pytest tests/unit/telemetry_error_issues: 254 passed onmaster; 12 failed and 250 passed after the first commit; 262 passed after the second.master, the rules tool tests, the generated-files check anduv lock --checkpass. No inline suppressions were added (94noqa, 9 type or pyright ignores, as onmaster).🤖 Generated with Claude Code