fix(crashtracking): stop resolving thread symbols with libunwind in the receiver

[taegyunkim]

Summary

Keep libunwind for remote stack walking, stop using it for redundant and crash-prone ELF symbol lookup, and let blazesym do the one symbolization pass with an ELF fallback when the application is already gone.

Why?

The crashtracker uses a separate receiver process because the crashing application's signal handler cannot safely parse binaries, collect other thread stacks, or upload a report itself.

When all-thread collection is enabled, the receiver ptrace-attaches each application thread and uses libunwind's remote API to walk its stack. Stack walking and symbolization are separate operations:

unw_get_reg_remote / unw_step_remote
    -> collect instruction and stack pointers

unw_get_proc_name_remote
    -> search the target's ELF symbol tables for a function name

The second operation ran once per frame per thread inside unwind_remote_thread. For affected processes, libunwind segfaulted while searching an ELF symbol table:

unw_get_proc_name_remote
  -> _Ux86_64_get_proc_name
    -> _Ux86_64_get_proc_name_by_ip
      -> _Uelf64_get_proc_name
        -> _Uelf64_get_proc_name_in_image
          -> _Uelf64_lookup_symbol_closeness  # receiver crashes here

This destroys the original crash report. Thread collection runs before builder.build() and async_upload_to_endpoint, so a receiver that crashes here never uploads the application's report.

The risky lookup was also redundant on successful paths. CrashInfo::enrich_callstacks subsequently asks blazesym to symbolize the same thread frames and overwrites the libunwind function name. If blazesym failed, the libunwind name previously remained as an accidental fallback, but retaining that fallback meant risking the entire report. An uploaded report with unresolved addresses is preferable to losing all crash data.

How did we detect this?

The Python receiver was unintentionally auto-instrumenting itself in affected SSI and ddtrace-run environments. The parent application's PYTHONPATH included ddtrace/bootstrap, and that path was forwarded to the receiver. On startup, the receiver ran the full ddtrace preload and started its own nested crashtracker.

That nested crashtracker did not cause the libunwind fault, but it made the fault observable:

Application
  -> Receiver R1 processes the application crash
      -> R1 crashes in _Uelf64_lookup_symbol_closeness
          -> R1's nested crashtracker starts Receiver R2
              -> R2 uploads R1's crash report

The resulting reports identify _dd_crashtracker_receiver as the crashed process and contain instrumentation threads such as TelemetryWriter, RemoteConfigPol, RemoteConfigSub, and SignalUploader.

In a 120-hour dd-trace-py 4.13.x window we found 5,492 receiver crashes. 5,431 (98.9%) contain the libunwind get_proc_name / _Uelf64_lookup_symbol_closeness stack. The worst-affected customer produced 5,389 receiver crashes against only 47 delivered application crash reports.

Representative receiver crash and matching events in Datadog Logs

Without the receiver's accidental self-instrumentation, the libunwind crash and application report loss would still occur, but the receiver death would be largely silent.

dd-trace-py #19735 fixes that separate auto-instrumentation bug by removing the exact ddtrace/bootstrap directory from the receiver's inherited PYTHONPATH while preserving the injected ddtrace package path. It prevents the receiver from starting tracing, profiling, and a nested crashtracker. That PR intentionally remains draft until this libdatadog fix lands, because merging it first would remove the telemetry that exposes these receiver deaths without preventing the deaths or recovering the lost application reports.

How does this PR fix the problem?

1. Keep remote stack walking. The receiver still uses unw_init_remote, unw_get_reg_remote, and unw_step_remote to collect every thread's instruction and stack pointers.
2. Remove receiver-side libunwind symbol lookup. unw_get_proc_name_remote is no longer called, so the receiver does not enter the faulting ELF symbol-table search.
3. Use blazesym once. CrashInfo::enrich_callstacks remains responsible for converting addresses into function names, files, and line numbers.
4. Handle an exited target process. Process-based blazesym symbolization requires /proc/<pid>. The sidecar receiver can outlive the application, so this PR falls back to the ELF path and virtual relative_address already recorded by normalize_ip. This needs no live process.
5. Degrade instead of losing the report. If both blazesym paths fail, the frame retains its IP, SP, and any normalized build ID/path/relative address; the symbolization error is added to log_messages; and the receiver still uploads the report.

How did we test this?

A standalone reproducer and fundamental ELF parser hardening are available in DataDog/libdatadog-libunwind #14, with the underlying fork change in DataDog/libunwind #3. The reproducer ptrace-stops a helper in a valid shared library, corrupts its on-disk GNU hash metadata after loading, and calls unw_get_proc_name_remote() on that frame. Before the parser fix it terminates with SIGSEGV in the same call chain observed in telemetry; after the fix it returns -UNW_ENOINFO.

cargo run -p bin_tests --bin prebuild
cargo test -p bin_tests --test crashtracker_bin_test
cargo test -p libdd-crashtracker --features generate-unit-test-files --lib
cargo clippy -p libdd-crashtracker --all-targets

Local Linux x86_64 results:

• 45/45 crashtracker integration tests passed. This includes the three multi-thread collection tests that assert worker_fn_0 and worker_fn_1 are present in error.threads[].stack.frames[].function, proving blazesym still names remotely collected thread frames without unw_get_proc_name_remote.
• 161 unit tests passed, 1 ignored.
• cargo clippy passed.
• Added test_symbolization_after_process_exit. It normalizes a frame while the process is available, then symbolizes against a nonexistent process and verifies that the recorded ELF path and virtual offset recover my_function.
• Disabling the ELF fallback makes the new unit test fail and also makes test_crash_tracking_sidecar_multi_thread_collection fail with failed to open proc maps file /proc/<pid>/maps, confirming that the fallback covers a real path rather than an artificial unit-test condition.

[github-actions]

📚 Documentation Check Results

⚠️ 1156 documentation warning(s) found

📦 libdd-crashtracker - 1156 warning(s)

---

Updated: 2026-08-18 22:25:35 UTC | Commit: df01c7a | missing-docs job results

[github-actions]

🔒 Cargo Deny Results

⚠️ 3 issue(s) found, showing only errors (advisories, bans, sources)

📦 libdd-crashtracker - 3 error(s)

Show output

error[vulnerability]: Invalid pointer dereference in `fmt::Pointer` impl for `Atomic` and `Shared` when the underlying pointer is invalid
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:61:1
   │
61 │ crossbeam-epoch 0.9.18 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
   │
   ├ ID: RUSTSEC-2026-0204
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0204
   ├ Affected versions of `fmt::Display` dereference the underlying pointer. This causes a invalid pointer dereference e.g., when a pointer created with `Atomic::null` or `Shared::null`. `fmt::Debug` impls and pre-0.9 `fmt::Display` impls, which do not dereference pointers, are not affected by this issue.
   ├ Announcement: https://github.com/crossbeam-rs/crossbeam/pull/1276
   ├ Solution: Upgrade to >=0.9.20 (try `cargo update -p crossbeam-epoch`)
   ├ crossbeam-epoch v0.9.18
     └── crossbeam-deque v0.8.5
         └── rayon-core v1.12.1
             └── rayon v1.10.0
                 └── criterion v0.5.1
                     ├── (dev) libdd-crashtracker v1.0.0
                     └── (dev) libdd-ddsketch v1.1.1
                         └── libdd-telemetry v7.0.0
                             └── libdd-crashtracker v1.0.0 (*)

error[vulnerability]: h2 unbounded empty DATA frames
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:99:1
   │
99 │ h2 0.4.6 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
   │
   ├ ID: RUSTSEC-2026-0258
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0258
   ├ The h2 crate, used internally by hyper, had a flaw that would accept and queue empty DATA frames without limit.
     If streams were not actively drained, this could lead to unbounded memory usage, or a panic if the length overflows.
     
     Low severity.
     
     Patched in v0.4.16.
   ├ Announcement: https://github.com/hyperium/hyper/security/advisories/GHSA-q83h-524g-xf6h
   ├ Solution: Upgrade to >=0.4.16 (try `cargo update -p h2`)
   ├ h2 v0.4.6
     └── hyper v1.6.0
         ├── httpmock v0.8.0-alpha.1
         │   └── (dev) libdd-telemetry v7.0.0
         │       └── libdd-crashtracker v1.0.0
         ├── hyper-rustls v0.27.7
         │   └── libdd-common v5.2.0
         │       ├── libdd-capabilities-impl v4.0.0
         │       │   ├── libdd-crashtracker v1.0.0 (*)
         │       │   ├── libdd-shared-runtime v3.0.0
         │       │   │   └── libdd-telemetry v7.0.0 (*)
         │       │   └── (dev) libdd-telemetry v7.0.0 (*)
         │       ├── (build) libdd-crashtracker v1.0.0 (*)
         │       ├── libdd-shared-runtime v3.0.0 (*)
         │       └── libdd-telemetry v7.0.0 (*)
         ├── hyper-util v0.1.17
         │   ├── httpmock v0.8.0-alpha.1 (*)
         │   ├── hyper-rustls v0.27.7 (*)
         │   └── libdd-common v5.2.0 (*)
         └── libdd-common v5.2.0 (*)

error[unsound]: Rand is unsound with a custom logger using `rand::rng()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:210:1
    │
210 │ rand 0.8.5 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0097
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0097
    ├ It has been reported (by [@lopopolo](https://github.com/lopopolo)) that the `rand` library is [unsound](https://rust-lang.github.io/unsafe-code-guidelines/glossary.html#soundness-of-code--of-a-library) (i.e. that safe code using the public API can cause Undefined Behaviour) when all the following conditions are met:
      
      - The `log` and `thread_rng` features are enabled
      - A [custom logger](https://docs.rs/log/latest/log/#implementing-a-logger) is defined
      - The custom logger accesses `rand::rng()` (previously `rand::thread_rng()`) and calls any `TryRng` (previously `RngCore`) methods on `ThreadRng`
      - The `ThreadRng` (attempts to) reseed while called from the custom logger (this happens every 64 kB of generated data)
      - Trace-level logging is enabled or warn-level logging is enabled and the random source (the `getrandom` crate) is unable to provide a new seed
      
      `TryRng` (previously `RngCore`) methods for `ThreadRng` use `unsafe` code to cast `*mut BlockRng<ReseedingCore>` to `&mut BlockRng<ReseedingCore>`. When all the above conditions are met this results in an aliased mutable reference, violating the Stacked Borrows rules. Miri is able to detect this violation in sample code. Since construction of [aliased mutable references is Undefined Behaviour](https://doc.rust-lang.org/stable/nomicon/references.html), the behaviour of optimized builds is hard to predict.
    ├ Announcement: https://github.com/rust-random/rand/pull/1763
    ├ Solution: Upgrade to >=0.10.1 OR <0.10.0, >=0.9.3 OR <0.9.0, >=0.8.6 (try `cargo update -p rand`)
    ├ rand v0.8.5
      ├── libdd-common v5.2.0
      │   ├── libdd-capabilities-impl v4.0.0
      │   │   ├── libdd-crashtracker v1.0.0
      │   │   ├── libdd-shared-runtime v3.0.0
      │   │   │   └── libdd-telemetry v7.0.0
      │   │   │       └── libdd-crashtracker v1.0.0 (*)
      │   │   └── (dev) libdd-telemetry v7.0.0 (*)
      │   ├── (build) libdd-crashtracker v1.0.0 (*)
      │   ├── libdd-shared-runtime v3.0.0 (*)
      │   └── libdd-telemetry v7.0.0 (*)
      ├── libdd-crashtracker v1.0.0 (*)
      ├── (dev) libdd-ddsketch v1.1.1
      │   └── libdd-telemetry v7.0.0 (*)
      └── proptest v1.5.0
          └── (dev) libdd-common v5.2.0 (*)

advisories FAILED, bans ok, sources ok

---

Updated: 2026-08-18 22:26:58 UTC | Commit: df01c7a | dependency-check job results

[pr-commenter]

Benchmarks

Comparison

Benchmark execution time: 2026-08-18 22:33:14

Comparing candidate commit 97f8eba in PR branch taegyunkim/crashtracker-receiver-symbolization-fix with baseline commit aab513c in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 1 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Candidate

Candidate benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	97f8eba	1787091806	taegyunkim/crashtracker-receiver-symbolization-fix

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
receiver_entry_point/report/2644	execution_time	3.537ms	3.569ms ± 0.018ms	3.566ms ± 0.010ms	3.577ms	3.602ms	3.614ms	3.655ms	2.49%	1.310	3.323	0.49%	0.001ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
receiver_entry_point/report/2644	execution_time	[3.566ms; 3.571ms] or [-0.069%; +0.069%]	None	None	None

Baseline

Baseline benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	aab513c	1787075760	main

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
receiver_entry_point/report/2644	execution_time	3.547ms	3.574ms ± 0.017ms	3.572ms ± 0.009ms	3.581ms	3.605ms	3.627ms	3.641ms	1.92%	1.173	1.906	0.47%	0.001ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
receiver_entry_point/report/2644	execution_time	[3.572ms; 3.576ms] or [-0.065%; +0.065%]	None	None	None

[datadog-datadog-prod-us1-2]

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 85.71%
• Overall Coverage: 76.40% (+0.01%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 97f8eba | Docs | View more details | Give us feedback!}

[taegyunkim]

@codex review

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR removes in-receiver symbol name resolution during ptrace unwinding (capturing only IP/SP), and improves post-processing symbolization by adding an ELF/offset fallback when the target process has already exited.

Changes:

• Stop resolving frame symbol names via unw_get_proc_name_remote during ptrace collection; capture raw addresses only.
• Update thread context streaming/capture APIs to drop the “resolve frames” parameter.
• Enhance StackFrame::resolve_names to fall back from /proc/<pid> symbolization to ELF + virtual offset, and add a regression test for “process already exited”.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
libdd-crashtracker/src/receiver/receive_report.rs	Removes passing the “resolve frames” setting into thread collection after collector no longer supports per-frame symbolization.
libdd-crashtracker/src/receiver/ptrace_collector.rs	Removes remote proc-name symbolization and updates thread-capture APIs accordingly.
libdd-crashtracker/src/crash_info/stacktrace.rs	Adds ELF/offset fallback symbolization and a test ensuring symbolization works after the process exits.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: ab7cbb3bc1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[dd-octo-sts]

Artifact Size Benchmark Report

aarch64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so	8.20 MB	8.20 MB	0% (0 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a	89.46 MB	89.47 MB	+0% (+2.03 KB) 👌

aarch64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so	11.06 MB	11.06 MB	+0% (+312 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a	100.63 MB	100.64 MB	+0% (+2.07 KB) 👌

libdatadog-x64-windows

Artifact	Baseline	Commit	Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll	26.62 MB	26.62 MB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib	94.56 KB	94.56 KB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb	180.67 MB	180.67 MB	0% (0 B) 👌
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib	772.95 MB	772.95 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll	8.72 MB	8.72 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib	94.56 KB	94.56 KB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb	25.64 MB	25.64 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib	51.05 MB	51.05 MB	0% (0 B) 👌

libdatadog-x86-windows

Artifact	Baseline	Commit	Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll	23.20 MB	23.20 MB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib	96.04 KB	96.04 KB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb	185.56 MB	185.55 MB	-0% (-8.00 KB) 👌
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib	759.56 MB	759.56 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll	6.74 MB	6.74 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib	96.04 KB	96.04 KB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb	27.58 MB	27.58 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib	48.60 MB	48.60 MB	0% (0 B) 👌

x86_64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a	79.72 MB	79.73 MB	+0% (+1.70 KB) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so	9.15 MB	9.15 MB	0% (0 B) 👌

x86_64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a	95.39 MB	95.39 MB	+0% (+1.64 KB) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so	11.16 MB	11.16 MB	+0% (+264 B) 👌