[spark] Support time-range incremental batch reads

[Yohahaha]

Summary

Support time-range incremental batch reads in the Spark connector, so pipelines can read only the data written within a [start, end) window (start inclusive, end exclusive) — the building block for an hourly/daily incremental job.

New TVF fluss_incremental_between_timestamp(table, start[, end]), registered via FlussSparkSessionExtensions:

-- the past hour, computed in SQL
SELECT * FROM fluss_incremental_between_timestamp(
  'log_table',
  date_format(now() - INTERVAL 1 HOUR, 'yyyy-MM-dd HH:mm:ss'));

The table argument accepts table, database.table or catalog.database.table. The bound arguments accept a string (epoch millis or yyyy-MM-dd HH:mm:ss), an integral epoch-millis value, a DATE (start of that day) or a TIMESTAMP/TIMESTAMP_NTZ, all in the Spark session time zone, and may come from any constant expression (column references are rejected). Both bounds are resolved during analysis — omitting end fills in Spark's current_timestamp() — so the window is pinned by the statement: rows committed while the query is planned stay out of it, and re-executing the same relation reads the same window.

New options backing the TVF, also usable from the DataFrame API. They are per-query read options and deliberately not read from session configuration, so a window can never leak into later reads; they are batch-only and ignored by streaming:

Option	Default	Meaning
scan.incremental.start.timestamp	(none)	Enables the incremental read; inclusive lower bound.
scan.incremental.end.timestamp	(none)	Exclusive upper bound; unset means "up to the latest committed data".

Read semantics per table type:

• Log table: the records appended within the window.
• Primary key table: the keys inserted/updated in the window, folded to their latest value as of the window end (changelog range only, no kv snapshot; keys only deleted in the window are excluded).
• Lake-enabled table: same as above, but always read from Fluss only — an incremental read never unions the lake snapshot.

Exactness of the window. The start bound positions the scan through the server's timestamp-to-offset lookup, but the scan stops at the latest offset and both bounds are then applied to each record's commit timestamp by the reader (FlussTimeRange, carried on the input partition), terminating the partition at the first record at or after end. That keeps [start, end) exact even for data already tiered to remote storage, where a timestamp lookup is only as accurate as the sparse server-side time index — and it avoids sending a driver-computed "now" to the server, which would be rejected as a future timestamp under clock skew.

Retention. A window reaching further back than what Fluss still retains (table.log.ttl) is not an error: the dropped part simply yields fewer rows, or none. The result is always a genuine subset of the requested window, so a caller never sees foreign data — there is no separate out-of-range mode to configure, and no extra listOffsets(EARLIEST) round trip at planning time.

Invalid windows fail fast instead of silently changing semantics: a blank or unparseable timestamp, an end without a start, and (in the TVF, at analysis time) a start that is not strictly before the end.

Also:

• scan.startup.mode is documented as affecting streaming reads only; plain batch reads remain full-table, so default behavior is unchanged.
• Fixes an unrelated leak where SET spark.sql.fluss.* values were merged into the catalog-wide configuration shared by every table, instead of a per-scan copy.
• The scan description now reports the resolved window, e.g. FlussScan: [fluss.t], Type: [Append] [TimeRange: [1767225600000, 1767312000000)].
• Docs: website/docs/engine-spark/reads.md (new "Time-Range Batch Read" section) and options.md.

Fixes #3842

Test Plan

• Unit tests FlussOffsetInitializersTest: option gating, blank/malformed values, end-without-start, and the time range handed to the reader.
• Integration tests SparkTimeRangeTvfTest (partitioned tables by default): log-table window incl. the pinned upper bound, primary key changelog folding, non-partitioned log/pk tables, equivalence of all timestamp argument forms, invalid usage, and that bounds are never taken from session configuration.
• Integration tests SparkLakeTimeRangeReadTest: log and pk tables skip the lake/kv snapshot.
• mvn spotless:check passes on the affected modules.

🤖 AI-assisted changes - reviewed by human developer

[Yohahaha]

@YannByron @fresh-borzoni @luoyuxia @beryllw PTAL, this is an actual customer requirement.

[YannByron]

Thanks for the PR — the feature direction makes sense and the docs are unusually thorough. I traced the PK changelog-fold semantics (snapshotId = -1 -> reader skips the snapshot -> SortMergeReader drops delete rows) down through FlussUpsertPartitionReader and SortMergeReader, and it does behave as described.

I have three must-fix findings before this lands, all on the incremental upsert path plus the option parsing. Details are inline; summary:

1. read.optimized=true combined with an incremental PK read silently returns zero rows, with no error. This is the one I would call a genuine bug.
2. failOnTimestampOutOfRange uses .get and Enumeration.withName, and is evaluated eagerly for every batch read — so a bad value of an incremental-only option breaks plain full-table reads with a bare NoSuchElementException.
3. createIncrementalUpsertPartitions is missing the empty-range guard that the append path gained in this same PR, so empty buckets each still spin up a Spark task and a Fluss connection.

I also collected some non-blocking notes (short-circuiting the lake-snapshot probe in incremental mode, end-side handling versus the server's strict ts > now rejection in Replica#getOffsetByTimestamp, the scan.incremental.* naming versus Fluss's existing scan.startup.* vocabulary, and the Thread.sleep-based test timing). I left those out to keep this review focused — happy to post them separately if useful.

[Copilot]

Pull request overview

Adds time-range incremental batch reads to the Fluss Spark connector, exposing the feature both as per-query scan options and as a SQL table-valued function (TVF), with documented semantics for log / primary-key / lake-enabled tables.

Changes:

• Introduces scan.incremental.{start,end}.timestamp + scan.incremental.timestamp.out-of-range and wires them through split planning for append/upsert tables (including retention guard behavior).
• Adds SQL TVF fluss_incremental_between_timestamp(table, start[, end]) via FlussSparkSessionExtensions, translating TVF arguments into per-relation scan options.
• Adds unit/integration tests and updates Spark connector docs to describe time-range batch read semantics and option scoping.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
website/docs/engine-spark/structured-streaming.md	Clarifies streaming startup-mode limitation and points users to incremental batch reads for bounded windows.
website/docs/engine-spark/reads.md	Documents time-range batch reads, TVF usage, DataFrame API options, and retention/window validation semantics.
website/docs/engine-spark/options.md	Splits session-level options vs per-query read options and documents new incremental scan options.
fluss-spark/fluss-spark-ut/src/test/scala/org/apache/fluss/spark/SparkTimeRangeTvfTest.scala	End-to-end TVF coverage for log/pk/partitioned tables, option scoping, and timestamp literal forms.
fluss-spark/fluss-spark-ut/src/test/scala/org/apache/fluss/spark/read/FlussOffsetInitializersTest.scala	Unit tests for incremental option parsing, window validation, and retention-guard decisions.
fluss-spark/fluss-spark-ut/src/test/scala/org/apache/fluss/spark/lake/SparkLakeTimeRangeReadTest.scala	Integration coverage ensuring incremental reads on lake-enabled tables stay Fluss-only (no lake/kv snapshot union).
fluss-spark/fluss-spark-common/src/main/scala/org/apache/fluss/spark/SparkFlussConf.scala	Adds config options/constants/enums for incremental batch reads.
fluss-spark/fluss-spark-common/src/main/scala/org/apache/fluss/spark/read/SplitPlanner.scala	Implements incremental planning branches, retention guard hooks, and empty-range handling.
fluss-spark/fluss-spark-common/src/main/scala/org/apache/fluss/spark/read/FlussOffsetInitializers.scala	Adds incremental option parsing, timestamp parsing, window validation, and retention-guard helpers.
fluss-spark/fluss-spark-common/src/main/scala/org/apache/fluss/spark/read/FlussMicroBatchStream.scala	Updates stopping-offset initializer wiring for streaming.
fluss-spark/fluss-spark-common/src/main/scala/org/apache/fluss/spark/FlussSparkSessionExtensions.scala	Registers the TVF resolver and injects supported TVFs into Spark extensions.
fluss-spark/fluss-spark-common/src/main/scala/org/apache/fluss/spark/catalyst/plans/logical/FlussTableValuedFunctions.scala	Implements TVF logical plan + argument normalization into scan options and resolution to a V2 relation.
fluss-spark/fluss-spark-common/src/main/scala/org/apache/fluss/spark/catalyst/analysis/FlussTableValuedFunctionResolver.scala	Analyzer rule to rewrite unresolved TVF nodes into DataSourceV2Relation.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[fresh-borzoni]

@Yohahaha Thank you for the PR, left some commments, PTAL

[Yohahaha]

@luoyuxia @beryllw @fresh-borzoni PTAL!

[fresh-borzoni]

@Yohahaha Thank you, left one comment, PTAL

[fresh-borzoni]

This can't fire anymore stopOffset is always latest, so an empty window still gets a partition.

If there's data after the window the start offset lands past the end, the reader keeps nothing, and LogChangesIterator throws NoSuchElementException on .head.

[Yohahaha]

thank you for catching this real bug! fixed with return empty iter when log records is empty.