[Performance] Filtered pod CPU query cannot reuse broader per-series rate state #891

Example

An observability recording rule and a pod-specific drill-down read the same CPU counter and 5-minute window:

sum by (cluster, namespace, pod, container) (rate(container_cpu_usage_seconds_total[5m]))

sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="production",pod="checkout-123"}[5m]))

Both queries get valid plans, but no candidate reuses the broad query's rate state for the filtered query. Removing the namespace/pod predicates enables selected sharing of the scan, time range, rate build, and rate finalization.

Tested on origin/main at c3be284c97c1f7ecdc490d78ddecf3e5f78c2007 on 2026-10-22. This is a concrete reproducer related to #664's filter/coverage-selection work, rather than a report of incorrect CPU values.

User or operational impact

A recording rule maintains CPU data across pods, then an SRE agent asks about one pod. The planner cannot exploit the overlap in this tested workload, so it selects independent rate computations. With raw retention disabled, it still maintains two independent 10-pane chains (20 stored outputs).

Metric and target

Target: enumerate at least one legal candidate that reuses per-series rate state and applies the pod/namespace selection before the filtered query's final aggregation. Stage 3 should compare that candidate with independent computation; it need not always select sharing.

No measured latency or memory improvement is claimed. The immediate acceptance metric is candidate coverage and successful execution of the shared candidate.

Representative workload and environment

Synthetic planning facts: 1,000 counter series; samples every 15 seconds; ingestion at 1,000/15 samples per second; both queries repeat every 30 seconds, are predictable, and require exact results.

Save this exact tested workload as cpu.json:

{
  "query_workload": {
    "language": "promql",
    "repeating_queries": [
      {
        "query": "sum by (cluster, namespace, pod, container) (rate(container_cpu_usage_seconds_total[5m]))",
        "demand": {
          "fixed_interval": 30000
        },
        "requirements": {
          "accuracy": {
            "max_error": 0.0
          }
        },
        "predictability": {
          "predictable": {
            "known_at": null
          }
        }
      },
      {
        "query": "sum by (pod) (rate(container_cpu_usage_seconds_total{namespace=\"production\",pod=\"checkout-123\"}[5m]))",
        "demand": {
          "fixed_interval": 30000
        },
        "requirements": {
          "accuracy": {
            "max_error": 0.0
          }
        },
        "predictability": {
          "predictable": {
            "known_at": null
          }
        }
      }
    ]
  },
  "data_workload": {
    "arrival": {
      "ingesting": {
        "rows_per_second": 66.66666666666667
      }
    },
    "sources": {
      "container_cpu_usage_seconds_total": {
        "sample_interval": 15000,
        "series_count": 1000,
        "metric_type": "counter"
      }
    }
  }
}

cargo run -p asap-devtools --bin stage_pipeline -- \
  --workload cpu.json --enforce-accuracy-guarantees --out cpu.default.json
cargo run -p asap-devtools --bin stage_pipeline -- \
  --workload cpu.json --enforce-accuracy-guarantees --exhaustive \
  --max-combinations 10000 --max-displayed 100 --out cpu.exhaustive.json

Control: replace only the second query with sum by (pod) (rate(container_cpu_usage_seconds_total[5m])) and rerun. This was tested and selects shared rate state.

Current baseline

• Default search separates the filtered workload into two components and selects P1+P1 without cross-query sharing.
• Exhaustive search enumerates 4 logical / 9 physical candidates and selects independent P1; none of its logical candidates offers cross-query sharing. This is not only a default-search component split.
• Unfiltered control selects P8-m1, sharing scan, time range, SummaryAgg rate build, and FinalizeExactAccumulator.
• With --deployment-capabilities '{"raw_data_retained":false}', the original filtered workload selects two independent ingestion-time rate chains, not one shared chain.

Correctness constraints

Retain namespace, pod, cluster, container, and full per-series identity as needed. Apply the filtered consumer's label selection without restricting the broad consumer. Handle counter resets per original series before summing; do not merge independent counters and then compute their rate. Refuse reuse when retained labels or state coverage cannot answer the requested filter.

Measurement method and acceptable variance

Use the fixed workload above with exact accuracy checks. Inspect all exhaustive logical candidates for reuse and compare default/exhaustive selection costs. Candidate coverage should be deterministic; runtime speed is not an acceptance threshold in this report.

Add end-to-end fixtures with multiple pods/namespaces and counter resets. Require the shared candidate to agree with independent evaluation for both queries, including a nonexistent pod filter. These numerical checks remain follow-up work; the current evidence covers planning and executor compilation only.

Related: #664 (filter push-down and coverage selection), #580 (sharing/coverage parity).