Skip to content

[Pipe] Optimize OPC UA TsFile last-value transfer - #18433

Open
Caideyipi wants to merge 4 commits into
apache:masterfrom
Caideyipi:perf/opcua-tsfile-last-value
Open

[Pipe] Optimize OPC UA TsFile last-value transfer#18433
Caideyipi wants to merge 4 commits into
apache:masterfrom
Caideyipi:perf/opcua-tsfile-last-value

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Description

Optimize the OPC UA sink's client-server TsFile path by reading each timeseries' last point directly from TsFile metadata instead of materializing every Tablet.

Metadata fast path

  • Read last points synchronously with TsFileLastReader; only BLOB/OBJECT-style types without values in statistics need their last chunk read.
  • Preserve the original measurement schema alongside each TimeValuePair, so logical types such as TIMESTAMP, STRING, and BLOB are not degraded by TsPrimitiveType.
  • Validate the number of values returned against independently read metadata because TsFileLastReader.hasNext() logs and suppresses IOExceptions.
  • Reuse the existing OPC UA row-transfer code for both embedded-server and client modes.

Compatibility and event semantics

  • Use the fast path only for client-server events without mods, time/path filtering, or privilege filtering.
  • Keep the existing Tablet parser as the fallback for filtered events and metadata read failures.
  • Retain the Tablet path for pub-sub mode, which must publish the full event stream.
  • Preserve TsFile event reference counting and close ownership across both paths.

Tests

  • Added real non-aligned tree and aligned table TsFile coverage.
  • Covered statistics-backed values, BLOB last-chunk reads, TIMESTAMP schema preservation, DATE conversion, aligned time-column filtering, null values, and null table tags.
  • Added an opt-in manual benchmark comparing the legacy Tablet path with the metadata last-value path, including tunable workload and final-value equivalence validation; skipped by default.
  • Spotless and Checkstyle pass; the 5 focused unit tests pass.

Manual performance result

Measured locally on 2026-08-11 with Windows 11 64-bit, an ASUS Zenbook UX3404VC (Intel Core i9-13900H, 14 cores/20 threads), JDK 17.0.15, and Maven 3.8.8.

Command:

mvn -Ddevelocity.off=true -q -pl iotdb-core/datanode -Dtest=OpcUaSinkTsFilePerformanceTest -Diotdb.opcua.tsfile.last-value.perf.enabled=true -Diotdb.opcua.tsfile.last-value.perf.iterations=50 -Diotdb.opcua.tsfile.last-value.perf.rounds=5 test

Workload: 2 devices, 32 measurements/device (1 BLOB measurement), 50,000 rows/device, 3,200,000 points, and a 12.32 MiB TsFile; 1 warmup iteration and 50 measured iterations per round across 5 alternating rounds. Three independent JVM runs were completed; the table reports the median across those runs (each run reports its own 5-round median).

Path CPU time/file Allocated/file Peak heap delta/file
Legacy Tablet path 156.250 ms 312.235 MiB 336.709 MiB
Metadata last-value path 0.625 ms 0.639 MiB 19.456 MiB
Change 250.00x speedup 99.8% reduction 94.2% reduction

Observed CPU speedup across the three runs was 166.67x-254.50x; allocation reduction stayed at 99.8%, and peak-heap reduction was 94.0%-94.6%. The benchmark's final-value maps were equal in every run, and each run passed the JUnit correctness check (Tests run: 1, Failures: 0, Errors: 0).


This PR has:

  • been self-reviewed.
  • added Javadocs for non-trivial methods.
  • added comments explaining non-obvious behavior.
  • added unit tests covering the new code paths.

Key changed/added classes
  • OpcUaSink
  • IoTDBOpcUaClient
  • OpcUaNameSpace
  • OpcUaSinkTsFileMetadataTest
  • OpcUaNameSpaceMetadataTest

Comment on lines +608 to +610
if (Objects.isNull(timeseriesDataTypes)) {
throw new IOException();
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add clearer message. Or return null instead.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in 7eca368 using the suggested null option: device/type/count metadata inconsistencies now return null and immediately trigger the existing Tablet fallback instead of throwing a message-less IOException.

…ast-value

# Conflicts:
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/sink/protocol/opcua/OpcUaSink.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants