Add PyLucene integration and CPU/GPU end-to-end tests

[nvzm123]

Summary

This ports NVIDIA/cuvs-lucene#174 into java/cuvs-lucene following the cuvs-lucene move into the cuVS monorepo. It adds PyLucene integration support and a pytest-owned CPU/GPU end-to-end suite.

The test layout follows the current cuVS/cuML test_*.py convention:

• java/cuvs-lucene/src/test/python/test_pylucene_end_to_end.py owns test cases, parametrization, assertions, and reporting.
• java/cuvs-lucene/src/test/python/pylucene_test_support.py contains reusable PyLucene index/search helpers.
• java/cuvs-lucene/src/test/java/com/nvidia/cuvs/lucene/PyLuceneTestSupport.java provides test-only codecs, execution-path diagnostics, and graph verification.
• Tests run directly with python -m pytest -q -s src/test/python/test_pylucene_end_to_end.py from java/cuvs-lucene once the documented PyLucene, jar, JVM, and native-library environment is available.

The standard thin cuvs-lucene jar remains the artifact under test; Lucene and cuvs-java remain external classpath dependencies, and the Java diagnostics stay in target/test-classes rather than the published jar.

The port also:

• fixes every default codec provider to Lucene 10.2's Lucene101Codec delegate instead of dynamically selecting an older outer codec
• requires the PyLucene wrapper's Lucene version to match the POM before JVM initialization
• makes scalar-quantized format initialization lazy so SPI discovery does not eagerly construct it
• adds the correctly typed scalar HNSW provider API while retaining the legacy JVM descriptor with an actionable failure
• preserves the existing binary-provider descriptors and correctly typed APIs
• uses cuVS 26.10's padded-dataset view API when serializing GPU-built indexes
• documents the monorepo build/test flow and current cuVS installation links

Apache does not publish PyLucene 10.2.0, so the full suite uses a custom PyLucene wrapper generated against the same Lucene 10.2.0 sources as this project.

PyLucene test coverage

The suite explicitly proves these paths:

• CPU HNSW build and HNSW search
• GPU CAGRA build followed by one-layer or three-layer HNSW search
• GPU CAGRA build and CAGRA search

GPU-required cases assert the concrete accelerated writer, reader, and query implementations and fail on unavailable cuVS or CPU fallback. CPU cases construct and report a stock Lucene HNSW path.

Coverage includes:

• a single-live-document CAGRA-search case
• one and ten segments
• 10-to-1 and 100-to-10 force merges
• CAGRA searchWidth values 1, 16, and 32
• deleted-document exclusion
• selective filtering through all three execution paths
• persisted HNSW graph degree and layer verification
• cold-JVM codec and vector-format SPI discovery
• an exact PyLucene/Lucene version preflight

Vectors and queries are deterministic, and expected neighbors are computed by brute force. Assertions verify rank-one self matches where applicable, exact hit counts, no duplicates, inactive or filter-rejected document exclusion, and a configurable recall floor.

CAGRA configurations use graphDegree=32 and intermediateGraphDegree=64. Cases construct enough vectors to avoid cuVS graph-parameter clamping, including 24,832 vectors for the three-layer case.

Validation

Validated from java/cuvs-lucene on an NVIDIA A10G with matching cuVS Java/native 26.10 and Lucene/PyLucene 10.2 environments:

• bash -n build.sh: passed
• python -m compileall -q src/test/python: passed
• python -m pytest --collect-only -q src/test/python/test_pylucene_end_to_end.py: 24 tests collected
• mvn -q spotless:check: passed
• mvn -Dtest=TestBackCompat,TestAcceleratedHNSWDeletedDocuments test: 17 tests, 0 failures, 0 errors, 0 skipped
• mvn -q clean verify -Dtests.seed=5A17C10120260817: 316 tests, 0 failures, 0 errors, 30 skipped
• python -m pytest -q -s src/test/python/test_pylucene_end_to_end.py: 24 passed in 39.03 seconds
• minimum observed HNSW recall: 0.85; all CAGRA-search cases: 1.0

The Maven suite retains existing warnings from randomized small-dataset graph clamping, JVM vector/native access, and intentional fallback or invalid-configuration cases. The PyLucene suite emitted no cuVS graph-clamping or CPU-fallback warnings; its only warning was the JVM notice for the incubating vector module.

Known randomized-test failure

Seed CC0EA94328BAB3E5 exposes an existing failure in TestCuVSVectorsFormat.testRandomWithUpdatesAndGraph: IllegalStateException: Index not found for field:field at CuVS2510GPUVectorsReader.java:425.

The same seed reproduces identically on this branch and on the untouched base commit 0fa5ebe5, including the secondary seed and stack trace. It is therefore not introduced by this port. Reproduction command from java/cuvs-lucene:

mvn -q -Dtest=TestCuVSVectorsFormat#testRandomWithUpdatesAndGraph \
  -Dtests.seed=CC0EA94328BAB3E5 \
  -Dtests.locale=ti-Ethi-ET \
  -Dtests.timezone=Greenwich test

The fixed-seed full validation listed above passes.

Follow-up multithreaded concurrency coverage is tracked in NVIDIA/cuvs#2407.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cjnolet]

We shouldn't be hosting python files in java directories. Please move to cuvs-bench.

[cjnolet]

Again, this should be hosted in cuvs-bench, not in cuvs-lucene.

[cjnolet]

I mentioned this in the prior PR in cuvs-lucene- you should not need to list special instructions in the docs for running the pylucene tests. They should be part of the cuvs-bench python test suite. Please remove all this stuff from the docs. You also should not need to explicitly state how to run the java tests for cuvs-lucene- running cuvs-lucene tests should be enough to run all the java tests in cuvs-lucene.

[cjnolet]

This still needs some work.