Skip to content

Fix iterable filters for combined metadata - #393

Merged
thodson-usgs merged 3 commits into
DOI-USGS:mainfrom
thodson-usgs:fix/392-combined-metadata-iterables
Aug 26, 2026
Merged

Fix iterable filters for combined metadata#393
thodson-usgs merged 3 commits into
DOI-USGS:mainfrom
thodson-usgs:fix/392-combined-metadata-iterables

Conversation

@thodson-usgs

@thodson-usgs thodson-usgs commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • classify waterdata.get_combined_metadata returns no rows for iterable query parameters #392 as a code bug: get_combined_metadata documents iterable filters, but comma-joined GET values return no matches while equivalent CQL2 queries work
  • route multi-value combined-metadata filters through the existing CQL2 POST path
  • add offline regression coverage for monitoring-location IDs, site types, agency codes, and state-name translation
  • update repository guidance to identify both monitoring-locations and combined-metadata as CQL2 exceptions

Closes #392.

Live API verification

Verified against the Water Data API on 2026-08-26 with site_type_code, agency_code, and state_name. Every field produced the same result:

Collection Scalar GET Comma-separated GET Equivalent CQL2 POST
monitoring-locations HTTP 200, 1 feature HTTP 200, 0 features HTTP 200, 1 feature
combined-metadata HTTP 200, 1 feature HTTP 200, 0 features HTTP 200, 1 feature

PR #233 recorded the historical monitoring-locations failure as HTTP 400. The live endpoint now fails silently with an empty HTTP 200 response instead, but CQL2 remains necessary for correct multi-value behavior.

Forced-chunk stress testing

Compared complete live result sets after sorting by record ID:

  • two states × two site types, filtered to daily discharge: 113 records both unchunked (1 fetch) and with parallel_chunks(4) (4 fetches)
  • four states × four sites × two parameter codes: 16 records both unchunked (1 fetch) and with parallel_chunks(8) (8 fetches)
  • scalar and one-item iterable forms: identical 4-record GET results and identical URLs
  • projected empty result: identical 0×6 frame unchunked and after four scalarized chunks
  • existing get_monitoring_locations path: identical 2-record result unchunked and after eight chunks

Adjacent observations were not introduced by this change:

  • limit=1 can return overlapping pages, but the same duplicate IDs reproduce on the unchanged scalar GET path; the unique-ID set matches limit=50
  • max_rows can select a different valid subset after partitioning because the API provides no global order; complete uncapped sets match
  • an all-null projected string column can infer object versus pandas string under fan-out while IDs and values remain identical; this is generic chunk-frame combination behavior

Validation

  • python3 -m pytest tests/waterdata_test.py -k 'combined_metadata or monitoring_locations or construct_api_requests' -q (13 passed)
  • python3 -m pytest tests/waterdata_chunking_test.py -q (118 passed)
  • python3 -m pytest tests/waterdata_utils_test.py -q (94 passed)
  • python3 -m coverage run -m pytest tests/ (998 passed, 12 live tests deselected)
  • python3 -m coverage report -m (98%)
  • python3 -m ruff check .
  • python3 -m ruff format --check .
  • python3 -m mypy
  • python -m xenon --max-absolute C --max-modules B --max-average A dataretrieval
  • complexipy dataretrieval
  • lint-imports (8 contracts kept)

@thodson-usgs
thodson-usgs marked this pull request as ready for review August 26, 2026 13:45
@thodson-usgs
thodson-usgs requested a review from ehinman August 26, 2026 13:46
@thodson-usgs
thodson-usgs merged commit 708757e into DOI-USGS:main Aug 26, 2026
11 checks passed
@thodson-usgs
thodson-usgs deleted the fix/392-combined-metadata-iterables branch August 26, 2026 19:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

waterdata.get_combined_metadata returns no rows for iterable query parameters

1 participant