Skip to content

Fingerprint telemetry exceptions by function names, not module paths - #4086

Merged
anth-volk merged 2 commits into
masterfrom
telemetry-fingerprint-function-names
Oct 27, 2026
Merged

anth-volk merged 2 commits into
masterfrom
telemetry-fingerprint-function-names

Conversation

@anth-volk

@anth-volk anth-volk commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4085
Fixes #4071

#4089 is stacked on this PR. It changes which record represents a failed simulation worker, which moves some failures between groups, and relies on this PR's algorithm 2 instead of needing algorithm 3, so it should merge right after this one.

What was wrong

The Public API's daily telemetry error report fingerprints an exception by its service family, its exception class and its three innermost stack frames in application packages. Each frame was keyed as module path:function name, so moving a function to another module gave every exception raised through it a new fingerprint: the group's issue stopped being updated and a duplicate opened. The recent file-length splits moved many functions into new modules, and made some private helpers public.

What changed

  • normalization.fingerprint_frames still chooses the three innermost frames whose paths are in application packages (or, without any, the three innermost of any kind), but now keys each one by its function name alone, with leading underscores removed. _resolve_region and resolve_region give the same key. Any other rename still changes the fingerprint, which is accepted. Two functions with the same name in different modules now give the same key; with three frames, the exception class and the service family also in the fingerprint, this should rarely merge different failures.
  • frame_key, which returned path:function, is replaced by frame_location, which returns the normalized path and the function name separately, and function_key, which removes the leading underscores.
  • FINGERPRINT_ALGORITHM is now 2. The issue state comment and the issue details record it.
  • gcp/observability/README.md describes the new frame keys and what happens to issues from algorithm 1. changelog.d/telemetry-fingerprint-function-names.changed.md describes the change.

Message and failed-request groups

Their fingerprint inputs are unchanged. Their fingerprints still change, because the HMAC input starts with FINGERPRINT_ALGORITHM, the issue state records one algorithm number for every kind, and the sync matches an issue's state only when it names the current algorithm. Keeping them would need a separate algorithm number per kind, which isn't worth it for the two such issues that exist (#4067 and #4070).

Issues opened under algorithm 1

Five exist: #4067 (message group), #4068 (closed as completed), #4069, #4070 (failed-request group) and #4071. sync._issue_fingerprint reads an issue's fingerprint from its state comment only when the state names the current algorithm; otherwise it reads the fingerprint at the end of the title. The job never changes titles, so these issues are indexed under their algorithm 1 fingerprints, which no algorithm 2 group produces. As a result:

  • Matching: none of them matches a group. Nothing else iterates over unmatched issues, so they are left as they are.
  • Updating: their marked sections keep the last algorithm 1 report and are no longer refreshed.
  • Reopening: [telemetry] policyengine-api: StaleDataError on /<country_id>/report (20e14b2f6c33dcfa) #4068 won't reopen if its StaleDataError returns. A new issue opens instead, without the regression label.
  • New issues: a group that still fails opens a new issue once it meets the thresholds, within the limit of five new issues per run.
  • Nothing fails: the old state comments still parse, and an issue whose fingerprint matches nothing is skipped.

Someone can close the five by hand. Alternatively, an issue can be carried over by replacing the fingerprint at the end of its title with its group's algorithm 2 fingerprint: the job then matches it by its title and rewrites its marked section with an algorithm 2 state. A test covers this.

Testing

The first commit adds the tests and they fail; the second commit is the fix.

  • test_records.py: an exception raised through functions that moved to other modules gives the same fingerprint, as does a helper renamed from _resolve_region to resolve_region, in its old or new module. A renamed function (_find_region) still gives a different fingerprint; this passes before and after. The exception's fingerprint inputs are now the function names.
  • test_normalization.py: the three existing frame-selection tests expect function names, and a new test checks that leading underscores are removed (__getitem__ becomes getitem__). Frames are still chosen by path: library frames are skipped when there are application frames.
  • test_rendering.py: the state comment and the issue details name algorithm 2.
  • test_sync.py: an open, or completed and closed, issue from algorithm 1 for the same failures is neither updated, reopened nor commented on, and a new issue opens. An algorithm 1 issue whose title was changed to the new fingerprint is updated and gets an algorithm 2 state.
  • The fix commit also renames frame_key to frame_location in the six frame-extraction tests, which now compare (path, function) pairs. Their expected paths and functions are unchanged.
  • pytest tests/unit/telemetry_error_issues: 254 passed on master; 12 failed and 250 passed after the first commit; 262 passed after the second.
  • ruff format and lint, the quality guards, the rules guard against master, the rules tool tests, the generated-files check and uv lock --check pass. No inline suppressions were added (94 noqa, 9 type or pyright ignores, as on master).

🤖 Generated with Claude Code

anth-volk and others added 2 commits October 28, 2026 00:43
The daily telemetry error report keys each stack frame in an exception's
fingerprint by its module path and function name, so moving a function to
another module changes the fingerprint. These tests describe keying each
frame by its function name alone, without leading underscores, under
fingerprint algorithm 2:

- the same exception raised through functions that moved to other modules
  gives the same fingerprint, and so does a helper renamed from a private
  `_name` to a public `name`;
- a renamed function still gives a different fingerprint (this passes
  already);
- frames are still chosen by their paths: the three innermost frames in
  application packages, or of any kind without them;
- the issue state and details name fingerprint algorithm 2;
- an issue the job opened under algorithm 1 is neither updated nor
  reopened, and a new issue opens for the failures instead;
- an algorithm 1 issue whose title someone changed to the new fingerprint
  is matched by its title and rewritten with an algorithm 2 state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The daily telemetry error report keyed each stack frame in an exception's
fingerprint as `module path:function name`. Moving a function to another
module changed the fingerprint, so the group's issue stopped being updated
and a duplicate opened. The recent file-length splits moved many functions
into new modules, and made some private helpers public.

- `normalization.fingerprint_frames` still chooses the three innermost
  frames whose paths are in application packages (or the three innermost
  of any kind without them), but keys each by its function name alone,
  without leading underscores. `frame_key`, which returned
  `path:function`, is now `frame_location`, which returns the path and
  function separately; the frame extraction tests compare those pairs.
- `FINGERPRINT_ALGORITHM` is 2. The algorithm number is hashed into every
  fingerprint, so message and failed-request groups get new fingerprints
  too, although their inputs are unchanged.
- The sync needs no change. It matches an issue by its state comment only
  when the state names the current algorithm, and otherwise by the
  fingerprint in its title, which still holds the algorithm 1 value. So
  algorithm 1 issues are never updated, reopened or commented on, and a
  group that still fails opens a new issue.
- The observability README describes the new frame keys and what happened
  to algorithm 1 issues.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.83%. Comparing base (ad26e90) to head (fe8ad13).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #4086       +/-   ##
===========================================
+ Coverage   50.82%   87.83%   +37.01%     
===========================================
  Files         254      270       +16     
  Lines       14635    16058     +1423     
  Branches     2339     2626      +287     
===========================================
+ Hits         7438    14105     +6667     
+ Misses       6570     1284     -5286     
- Partials      627      669       +42     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@anth-volk
anth-volk marked this pull request as ready for review October 27, 2026 21:25
@anth-volk
anth-volk merged commit 3fde3da into master Oct 27, 2026
15 checks passed
@anth-volk
anth-volk deleted the telemetry-fingerprint-function-names branch October 27, 2026 21:25

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant