Add smoke tests documentation by PerfectSlayer · Pull Request #12282 · DataDog/dd-trace-java

PerfectSlayer · 2026-08-25T11:46:37Z

What Does This Do

This PR introduces documentation for the new smoke test framework.

Motivation

Help with discovery and adoption.

Additional Notes

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Once approved, use merge queue to merge the PR

Jira ticket: [PROJ-IDENT]

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e352616c17

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-25T11:54:51Z

+## Adding a new smoke-test module
+
+1. Register it in `settings.gradle.kts` (the `:dd-smoke-tests:*` list).
+2. Create `dd-smoke-tests/<name>/build.gradle` with the `application` and `com.gradleup.shadow` plugins, `testImplementation project(':dd-smoke-tests')`, and a `Test` task that publishes the app fixture:


Apply the smoke-test convention plugin

When a new module follows this recipe, the listed plugins omit dd-trace-java.module.smoke-test, even though that convention applies the repository's Java and test configuration, including JUnit Platform setup. The parent dd-smoke-tests/build.gradle only configures existing Test tasks and does not apply the convention to subprojects, so a JUnit 5 smoke test created from these instructions can be undiscovered or otherwise misconfigured; include the convention plugin in the required plugin list.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-25T11:54:51Z

+3. Write the application fixture under `src/main/java`, and the test under `src/test/java`.
+4. Refresh the module's `gradle.lockfile`.
+
+The parent `dd-smoke-tests/build.gradle` already gives every subproject the agent jar, the build directory and the `:dd-smoke-tests` framework, so do not re-declare those.


Require the framework dependency in each subproject

The parent dd-smoke-tests/build.gradle supplies the agent-jar and build-directory properties, but it does not add testImplementation project(':dd-smoke-tests') to subprojects. Therefore the claim that the parent provides the framework contradicts step 2 and may lead readers to remove or omit the dependency, leaving classes such as SmokeServerApp unavailable; restrict this sentence to the two system properties and keep the framework dependency explicit.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-25T11:54:51Z

+                span()
+                    .operationName("spring.handler")
+                    .resourceName("MyController.success")
+                    .childOfPrevious()));


Match the Spring handler span type

For the Spring MVC application represented by this example, a spring.handler span has the web span type, but an omitted SpanMatcher.type(...) is not a wildcard: SpanMatcher defaults to requiring a null or empty type. Consequently this advertised complete test keeps polling and then fails against a correct trace; add .type("web") to both handler matchers.

Useful? React with 👍 / 👎.

That’s not the point of the example… 😞

chatgpt-codex-connector · 2026-08-25T11:54:51Z

+Use `getFlatMessages()` and `waitForFlat(...)` for almost everything.
+The tracer batches events, so an `app-started` you are looking for usually arrives nested inside a `message-batch`.
+
+One telemetry assertion runs at the first `afterEach` without you writing it: `assertTelemetryReceived()` checks that at least one message reached the backend.


Qualify the automatic telemetry assertion

When an app uses a separately registered shared backend, AbstractSmokeApp.afterEach explicitly skips assertTelemetryReceived() because backend.isShared() is true. The unconditional statement here can therefore let multi-app tests assume telemetry is validated when no automatic assertion runs; document the owned-backend restriction and direct shared-backend tests to assert through backend.telemetry() themselves.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-25T11:54:52Z

+## Adding a new smoke-test module
+
+1. Register it in `settings.gradle.kts` (the `:dd-smoke-tests:*` list).
+2. Create `dd-smoke-tests/<name>/build.gradle` with the `application` and `com.gradleup.shadow` plugins, `testImplementation project(':dd-smoke-tests')`, and a `Test` task that publishes the app fixture:


Configure an entry point for the fixture jar

Following this module recipe leaves the application plugin's mainClass unset and does not add a Main-Class manifest attribute. The resulting shadowJar is therefore not a runnable fixture for SmokeServerApp.jar(...) or SmokeCliApp.jar(...), and the child JVM exits immediately when launched with java -jar; include an application { mainClass = ... } block or equivalent manifest configuration.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-25T11:54:52Z

+| `allowedErrorLogs("...")`   | One test's known-noisy lines, matched by substring                |
+| `errorLogFilter(predicate)` | Full control over what counts as an error; replaces the allowlist |
+| `skipErrorLogCheck()`       | Tests that are *about* error cases                                |


Treat error-log exemptions as class-wide

These are builder settings on the static app extension, and the automatic assertNoErrorLogs() scans the entire log accumulated since class launch at afterAll; none of these options can be scoped to one test method. Using allowedErrorLogs(...) or skipErrorLogCheck() for a single error-case method therefore suppresses matching failures from every other method in the class, so describe them as suite-wide and advise isolating such cases in a separate class or performing explicit log assertions.

Useful? React with 👍 / 👎.

dd-octo-sts · 2026-08-25T12:09:23Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.99 s	14.01 s	[-1.0%; +0.7%] (no difference)
startup:insecure-bank:tracing:Agent	12.94 s	12.98 s	[-1.0%; +0.4%] (no difference)
startup:petclinic:appsec:Agent	17.37 s	17.22 s	[-0.2%; +1.9%] (no difference)
startup:petclinic:iast:Agent	16.86 s	17.47 s	[-7.7%; +0.8%] (no difference)
startup:petclinic:profiling:Agent	16.83 s	17.19 s	[-6.5%; +2.3%] (no difference)
startup:petclinic:sca:Agent	17.38 s	17.24 s	[-0.2%; +1.9%] (no difference)
startup:petclinic:tracing:Agent	16.16 s	16.32 s	[-7.0%; +4.9%] (unstable)

Commit: e352616c · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

datadog-datadog-prod-us1-2

The new module recipe does not work as written. The guide also says that telemetry checks always run, but the code skips them for shared backends and apps without the agent.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit e352616 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

datadog-datadog-prod-us1-2 · 2026-08-25T12:14:01Z

+## Adding a new smoke-test module
+
+1. Register it in `settings.gradle.kts` (the `:dd-smoke-tests:*` list).
+2. Create `dd-smoke-tests/<name>/build.gradle` with the `application` and `com.gradleup.shadow` plugins, `testImplementation project(':dd-smoke-tests')`, and a `Test` task that publishes the app fixture:


Complete the Gradle module recipe

Contributors cannot create a working smoke-test module from this recipe.

Assertion details

Input: A contributor copies step 2 to create a module that starts its fixture with .jar(APPLICATION_JAR).

Expected: The recipe must include the convention plugin, the ShadowJar import, and an executable main-class configuration.

Actual: The recipe omits the smoke-test convention plugin and the ShadowJar import. It also does not configure a main class for the fixture that SmokeServerApp.jar(...) starts with java -jar. The copied script fails during Gradle configuration or produces a jar that cannot start.

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

datadog-datadog-prod-us1-2 · 2026-08-25T12:14:01Z

+Use `getFlatMessages()` and `waitForFlat(...)` for almost everything.
+The tracer batches events, so an `app-started` you are looking for usually arrives nested inside a `message-batch`.
+
+One telemetry assertion runs at the first `afterEach` without you writing it: `assertTelemetryReceived()` checks that at least one message reached the backend.


State the telemetry check limits

A test can omit telemetry validation while its author believes that the framework performs it.

Assertion details

Input: A smoke test uses a shared backend or calls noAgent().

Expected: The guide must state that the automatic check runs only for an agent-instrumented app with an owned backend. Other cases need an explicit assertion.

Actual: AbstractSmokeApp.afterEach skips this assertion when the backend is shared or when the app has no agent.

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

PerfectSlayer added 2 commits August 25, 2026 13:43

feat(dynamic-config): Migrate smoke test to JUnit

186a713

feat(doc): Add smoke tests documentation

e352616

PerfectSlayer requested a review from a team as a code owner August 25, 2026 11:46

PerfectSlayer requested review from mcculls and removed request for a team August 25, 2026 11:46

PerfectSlayer added comp: testing Testing type: documentation labels Aug 25, 2026

chatgpt-codex-connector Bot reviewed Aug 25, 2026

View reviewed changes

datadog-datadog-prod-us1-2 Bot reviewed Aug 25, 2026

View reviewed changes

Base automatically changed from bbujon/smoke-tests-dynamic-config to master August 25, 2026 13:16

gh-worker-dd-mergequeue-cf854d Bot requested review from a team as code owners August 25, 2026 13:16

gh-worker-dd-mergequeue-cf854d Bot requested review from claponcet, manuel-alvarez-alvarez and mhdatie and removed request for a team August 25, 2026 13:16

Navigation Menu

Uh oh!

Add smoke tests documentation - #12282

Add smoke tests documentation#12282
PerfectSlayer wants to merge 2 commits into
masterfrom
bbujon/smoke-tests-documentation

PerfectSlayer commented Aug 25, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Aug 25, 2026

Uh oh!

chatgpt-codex-connector Bot Aug 25, 2026

Uh oh!

chatgpt-codex-connector Bot Aug 25, 2026

Uh oh!

PerfectSlayer Aug 25, 2026

Uh oh!

chatgpt-codex-connector Bot Aug 25, 2026

Uh oh!

chatgpt-codex-connector Bot Aug 25, 2026

Uh oh!

Uh oh!

chatgpt-codex-connector Bot Aug 25, 2026

Uh oh!

dd-octo-sts Bot commented Aug 25, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Uh oh!

datadog-datadog-prod-us1-2 Bot Aug 25, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

PerfectSlayer commented Aug 25, 2026

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

dd-octo-sts Bot commented Aug 25, 2026

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1-2 Bot Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1-2 Bot Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant