Skip to content

skill(apm-integrations): match the sibling module's test DSL (no Java tests in a Groovy family) - #12164

Draft
jordan-wong wants to merge 1 commit into
masterfrom
skill/test-dsl-match-sibling-20260807
Draft

skill(apm-integrations): match the sibling module's test DSL (no Java tests in a Groovy family)#12164
jordan-wong wants to merge 1 commit into
masterfrom
skill/test-dsl-match-sibling-20260807

Conversation

@jordan-wong

Copy link
Copy Markdown
Contributor

🤖 Generated with APM Instrumentation Toolkit

[Reference / skill-fix PR — draft.] From the toolkit eval-research thread; a sharpening of the apm-integrations agent skill, not an instrumentation change.

What

Fixes a self-contradiction in references/tests.md:

  • Line 7 mandates "Write Groovy/Spock tests ... Full Java instrumentation test support is not yet available."
  • Lines ~136–175 illustrate the no-banner-comments rule with a fully Java AbstractInstrumentationTest example, and nothing states which wins.

Adds an unambiguous rule — match the sibling module's test DSL; a Groovy family stays Groovy — and fences the Java examples as style-only illustrations for modules already on the Java/JUnit DSL (not a license to migrate a Groovy family to Java).

Why (evidence)

Surfaced by the 2026-08-07 cache-category gap sweep on jedis-3.0 (reference PR #12080). jedis-3.0 is otherwise clean, but the regen wrote its tests in Java/JUnit (src/test/java, AbstractInstrumentationTest) while master and its siblings (jedis-1.4, jedis-4.0) use Groovy/Spock (src/test/groovy). The generation transcript shows the agent read the Groovy prose and a sibling cache library's Java test (lettuce-5.0) and followed the Java exemplar.

That Java choice is the direct trigger of the spurious _dd.svc_src CI failure a prior report mis-attributed to pure master-side infra: the Java DSL's TagsMatcher.defaultTags() (which carries a // TODO Keep porting default tag logic gap) enforces exact default tags the Groovy DSL tolerates. Fixing the skill (so the family stays Groovy) is higher-leverage than patching TagsMatcher.

Domain-agnostic: the contradiction is in the shared skill, so any Java instrumentation generation faces the same fork. Surfaced via cache but not cache-specific.

Scope

One file, references/tests.md, +3/−1. No instrumentation code changes.

🤖 Generated with APM Instrumentation Toolkit

… tests in a Groovy family)

Surfaced by the 2026-08-07 cache category gap sweep (jedis-3.0, reference
PR #12080). jedis-3.0 is otherwise clean, but the regen wrote its tests in
Java/JUnit (src/test/java, AbstractInstrumentationTest) while master and
its siblings (jedis-1.4, jedis-4.0) use Groovy/Spock (src/test/groovy).

Root cause is a self-contradiction in this skill file: line 7 mandates
"Write Groovy/Spock tests ... Full Java instrumentation test support is not
yet available", but lines ~136-175 illustrate the no-banner-comments rule
with a fully Java AbstractInstrumentationTest example and nothing says which
wins. The generation transcript shows the agent read the Groovy prose AND a
sibling cache library's Java test (lettuce-5.0) and followed the Java
exemplar. That Java choice is the direct trigger of the spurious
`_dd.svc_src` CI red a prior report mis-filed as pure master-side infra:
the Java DSL's TagsMatcher.defaultTags() (which carries a
"// TODO Keep porting default tag logic" gap) enforces exact default tags
the Groovy DSL tolerates.

Fix: add an unambiguous "match the sibling module's test DSL; a Groovy
family stays Groovy" rule next to the mandate, and fence the Java examples
as style-only illustrations for modules already on the Java DSL — not a
license to migrate a Groovy family. Domain-agnostic (all Java
instrumentation), surfaced via cache but not cache-specific.

Draft reference — from the toolkit eval-research thread.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jordan-wong jordan-wong added tag: ai generated Largely based on code generated by an AI or LLM tag: apm integration toolkit Changes generated by DataDog/apm-instrumentation-toolkit as part of IDM AIT experimentation labels Aug 7, 2026
@dd-octo-sts

dd-octo-sts Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.01 s 13.97 s [-0.6%; +1.3%] (no difference)
startup:insecure-bank:tracing:Agent 13.02 s 13.00 s [-0.8%; +1.2%] (no difference)
startup:petclinic:appsec:Agent 17.47 s 17.32 s [-0.3%; +2.0%] (no difference)
startup:petclinic:iast:Agent 16.74 s 17.46 s [-8.3%; +0.1%] (no difference)
startup:petclinic:profiling:Agent 17.42 s 17.39 s [-1.1%; +1.5%] (no difference)
startup:petclinic:sca:Agent 17.33 s 17.32 s [-0.8%; +0.9%] (no difference)
startup:petclinic:tracing:Agent 16.13 s 16.65 s [-7.3%; +0.9%] (no difference)

Commit: 7c00fb4e · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tag: ai generated Largely based on code generated by an AI or LLM tag: apm integration toolkit Changes generated by DataDog/apm-instrumentation-toolkit as part of IDM AIT experimentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant