feat: DH-21975: Gc benchmarking - #421
Draft
stanbrub wants to merge 117 commits into
Draft
Conversation
cpwright
reviewed
Mar 27, 2026
cpwright
left a comment
cpwright
left a comment
There was a problem hiding this comment.
All the benchmarks are going to do work. I have some concerns that we might do too much work related to the timestamp calculation and traversing a UnionSourceManager (merge) where not necessary.
| merge([ | ||
| read('/data/timed.parquet').view(formulas=[${loadColumns}])${headRows} | ||
| ] * ${scaleFactor}).update_view([ | ||
| 'timestamp=timestamp.plusMillis((long)(ii / ${rows}) * ${rows})' |
There was a problem hiding this comment.
Is there a reason we can't use the timestamp from the file? I have a few worries about doing rowset calculation as part of the benchmark (to come up with ii).
For the actual test benchmarks, without a select we would also just prefer more/bigger parquet files to avoid the overhead of going through the merge data structures. We might even be able to get away with symlinks to have the data just repeate itself.
Collaborator
Author
There was a problem hiding this comment.
For the "train" benchmarks, since we don't use Scale Factors, that section of code will not be hit. This is only used when we are doing merges to simulate larger data sets. So for the nightly runs, this will happen BEFORE the "select" into memory, which is not included in the measurement. But for the "train" benchmarks, we only read timestamps directly from the parquet file(s), and that only if they are used in the benchmark (like for rollingtime).
There was a problem hiding this comment.
Pull request overview
Adds GC benchmarking infrastructure, training workloads, event collection, dashboards, heap-pressure analysis, and more robust remote execution.
Changes:
- Adds local Parquet generation and six-category training benchmarks.
- Adds GC/heap dashboards, reports, and consolidated data tooling.
- Adds matrix workflows and remote-server lifecycle improvements.
Reviewed changes
Copilot reviewed 56 out of 72 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
src/test/resources/io/deephaven/benchmark/util/filerfolder/folder1/filer2.txt |
Adds directory-size fixture. |
src/test/resources/io/deephaven/benchmark/util/filerfolder/filer1.txt |
Adds directory-size fixture. |
src/test/java/io/deephaven/benchmark/util/FilerTest.java |
Tests byte-size calculation. |
src/test/java/io/deephaven/benchmark/api/QueryLogTest.java |
Updates query-log expectations. |
src/main/resources/io/deephaven/benchmark/run/profile/queries/dashboards/train_dashboard.dh.py |
Adds GC training dashboards. |
src/main/resources/io/deephaven/benchmark/run/profile/queries/dashboards/pullup.py |
Consolidates benchmark datasets. |
src/main/resources/io/deephaven/benchmark/run/profile/queries/dashboards/heap_dashboard.dh.py |
Adds heap-pressure dashboard. |
src/main/resources/io/deephaven/benchmark/run/profile/queries/dashboards/benchmark_functions.dh.py |
Adds event and consolidated-file loading. |
src/main/resources/io/deephaven/benchmark/run/profile/queries/compare.py |
Updates product comparisons. |
src/main/java/io/deephaven/benchmark/util/Filer.java |
Adds file creation and sizing utilities. |
src/main/java/io/deephaven/benchmark/generator/LocalParquetGenerator.java |
Adds direct Parquet generation. |
src/main/java/io/deephaven/benchmark/generator/ColumnDefs.java |
Adds copying and larger caches. |
src/main/java/io/deephaven/benchmark/generator/AvroKafkaGenerator.java |
Strengthens Kafka delivery settings. |
src/main/java/io/deephaven/benchmark/controller/DeephavenDockerController.java |
Extends startup timeout. |
src/main/java/io/deephaven/benchmark/connect/BarrageConnector.java |
Improves cleanup and message limits. |
src/main/java/io/deephaven/benchmark/api/Snippets.java |
Adds event tables and Parquet links. |
src/main/java/io/deephaven/benchmark/api/QueryLog.java |
Simplifies query-log placement. |
src/main/java/io/deephaven/benchmark/api/BenchTable.java |
Integrates local Parquet datasets. |
src/main/java/io/deephaven/benchmark/api/BenchQuery.java |
Cleans query-scope variables. |
src/main/java/io/deephaven/benchmark/api/BenchPlatform.java |
Corrects JVM argument collection. |
src/main/java/io/deephaven/benchmark/api/BenchLog.java |
Relocates engine logs. |
src/main/java/io/deephaven/benchmark/api/BenchEvents.java |
Persists benchmark events. |
src/main/java/io/deephaven/benchmark/api/Bench.java |
Integrates events and per-test logs. |
src/it/java/io/deephaven/benchmark/tests/train/UserFormulaTrainTest.java |
Adds UDF training workload. |
src/it/java/io/deephaven/benchmark/tests/train/UpdateByTrainTest.java |
Adds UpdateBy workload. |
src/it/java/io/deephaven/benchmark/tests/train/TrainTestRunner.java |
Adds training orchestration and telemetry. |
src/it/java/io/deephaven/benchmark/tests/train/OrderedTrainTest.java |
Adds ordered aggregation workload. |
src/it/java/io/deephaven/benchmark/tests/train/NaturalJoinTrainTest.java |
Adds natural-join workload. |
src/it/java/io/deephaven/benchmark/tests/train/InlineFormulaTrainTest.java |
Adds inline-formula workload. |
src/it/java/io/deephaven/benchmark/tests/train/FilterTrainTest.java |
Adds filter workload. |
src/it/java/io/deephaven/benchmark/tests/train/AggByTrainTest.java |
Adds aggregation workload. |
src/it/java/io/deephaven/benchmark/tests/standard/StandardTestRunner.java |
Adds local loading and event capture. |
src/it/java/io/deephaven/benchmark/tests/standard/kafka/KafkaTestRunner.java |
Configures log package roots. |
src/it/java/io/deephaven/benchmark/tests/standard/file/FileTestRunner.java |
Configures log package roots. |
src/it/java/io/deephaven/benchmark/tests/compare/CompareTestRunner.java |
Configures log package roots. |
pom.xml |
Adds Parquet writer dependency. |
docs/heap/gc-heap.md |
Documents heap-pressure findings. |
docs/gc/train-dashboard.md |
Documents dashboard setup. |
docs/gc/gc-report.md |
Documents GC throughput findings. |
.github/workflows/remote-benchmarks.yml |
Improves remote setup and publishing. |
.github/workflows/matrix-exist-remote-benchmarks.yml |
Adds script-generated matrices. |
.github/workflows/adhoc-exist-remote-benchmarks.yml |
Clears existing servers first. |
.github/workflows/adhoc-auto-remote-benchmarks.yml |
Clears between benchmark runs. |
.github/scripts/setup-test-server-remote.sh |
Reuses repository checkouts. |
.github/scripts/run-ssh-local.sh |
Adds SSH retries. |
.github/scripts/run-benchmarks-remote.sh |
Caps runner heap. |
.github/scripts/manage-deephaven-remote.sh |
Improves image selection and host paths. |
.github/scripts/clear-test-server-remote.sh |
Adds remote cleanup. |
.github/scripts/build-server-distribution-remote.sh |
Reuses server builds. |
.github/scripts/build-matrix.sh |
Converts TSV profiles to matrices. |
.github/scripts/build-docker-image-remote.sh |
Reuses Docker images. |
.github/scripts/build-benchmark-artifact-remote.sh |
Reuses benchmark artifacts. |
.github/scripts/adhoc.sh |
Improves workflow error reporting. |
.github/resources/adhoc-benchmark-docker-compose.yml |
Exposes the host data directory. |
.github/matrix/gc-train-heap-fail.sh |
Defines heap-pressure runs. |
.github/matrix/gc-train-1000-100.sh |
Defines GC comparison runs. |
Suppressed comments (2)
.github/matrix/gc-train-1000-100.sh:25
train.staticInchas no consumer in the repository; TrainTestRunner runs both modes whenever its configured factors are positive. Thus rows markedIncstill run and publish Static benchmarks. Wire this setting into mode selection or remove the unused axis and adjust the matrix scope.
local opts="$2 $BASE -DPeriodicUpdateGraph.targetCycleDurationMillis=$3 -Dtrain.incLoadTarget=$4 -Dtrain.staticInc=$5"
src/main/resources/io/deephaven/benchmark/run/profile/queries/dashboards/train_dashboard.dh.py:134
- The new event pipeline never emits
server_state_log/ugp.cycle.time; it emitsugp.cycle.cost/duration_rows, with cycle time stored induration. As written, the on-budget table has no rows for these benchmark runs. Align this filter/value column with TrainTestRunner or change the producer schema.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+73
to
+75
| while (!isClosed.get() && recCount < maxRecordCount) { | ||
| for (int i = 0; i < colCount; i++) { | ||
| row[i] = columnDefs.nextValue(i, recCount, maxRecordCount); |
|
|
||
| # emit <label> <gc> <cycle_ms> <inc_cycle_factor> — outputs regular and _huge variants | ||
| run() { | ||
| local opts="$2 $BASE -DPeriodicUpdateGraph.targetCycleDurationMillis=$3 -Dtrain.incLoadTarget=$4 -Dtrain.staticInc=$5" |
| STEP=4 | ||
|
|
||
| BASE="-XX:+AlwaysPreTouch -XX:+UseTransparentHugePages -XX:+UseStringDeduplication -XX:+UseCompactObjectHeaders -DServerStateTracker.reportIntervalMillis=1000" | ||
| CYCLE="-DPeriodicUpdateGraph.targetCycleDurationMillis=1000 -Dtrain.incLoadTarget=1.00" |
|
|
||
| for combo in "${COMBOS[@]}"; do | ||
| read -r cls gc min_heap max_heap <<< "$combo" | ||
| for h in $(seq ${max_heap} -${STEP} ${min_heap}); do |
| def get_jitter_table(): | ||
| gc_events = bench_result_sets.join(bench_events, ['benchmark_name','origin','set_id'], | ||
| ['type','start','duration','name','value']).sort(['set_id']) | ||
| jitter = gc_events.where(["type=`server_state_log`", "name=`ugp.cycle.time`"]) \ |
| rec = Recording() | ||
| rec.setName("benchmark") | ||
|
|
||
| enabled_events=['jdk.GarbageCollection','jdk.GCHeapSummary'] |
Comment on lines
+137
to
+138
| cycles = total.join(under, ['benchmark_name','set_id'], ['under']) \ | ||
| .update(['pct_under_budget = 100.0 * under / total']) |
Comment on lines
+64
to
+65
| delegate.useLocalParquet(true); | ||
| delegate.setRowFactor(maxRowFactor); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GC benchmarking infrastructure and analysis for understanding how each garbage collector (G1, ZGC, Shenandoah, Parallel) uses heap across Deephaven operation categories.
Infrastructure:
Analysis (
docs/):docs/gc/gc-report.md— GC throughput rankings from the original 48G-heap training benchmarks (static + incremental, JDK 17/25)docs/heap/gc-heap.md— Heap-pressure study showing how each GC behaves when constrained to minimal heap per operation category.