[Performance] A 30-day offset trend exceeds 90s planning at a 30s refresh cadence #896

Example

A single GitLab-style 30-day trend query compares the current 30 days with the preceding 30 days:

(sum(sum_over_time(sli_aggregations:pipelines_created_total:rate_5m[30d])) - sum(sum_over_time(sli_aggregations:pipelines_created_total:rate_5m[30d] offset 30d))) / sum(sum_over_time(sli_aggregations:pipelines_created_total:rate_5m[30d] offset 30d))

With a 30-second repeat interval, both default and exhaustive search exceed a 90-second process deadline without writing a stage document. Changing only the repeat interval to one hour selects a valid plan in 28.032 seconds.

Tested 2026-10-22 at c3be284c97c1f7ecdc490d78ddecf3e5f78c2007. This is a focused workload related to the broader search-performance tracker #879 and long-window work in #731.

User or operational impact

A dashboard trend query can spend over 90 seconds in planning even though a slower-refresh version is executable. Window duration and refresh interval need to be practical planner inputs without constructing or compiling prohibitive amounts of intermediate work.

Metric and target

Record planning wall time, peak RSS, candidates checked, and the work performed per candidate while varying cadence. Investigate how much time goes into candidate search, physical pane expansion, executor compile checks, and serialization before choosing a fix.

Keep the same valid optimum and guarantees as exhaustive search on tractable cases. The 90 seconds here is the probe deadline, not an established product requirement or proof that #879's 3-minute goal is violated. A proposed follow-up target is the existing #879 planning-time budget, with results measured on its reference environment.

Representative workload and environment

Linux x86_64, local Cargo debug build, Rust 1.98.1. The machine was shared; these are smoke measurements, not a controlled performance benchmark. No peak-RSS or CPU profile was captured. The query is instantiated from ASAPAgents' shortened GitLab production-dashboard template, with no optional label filter. The recorded rate is declared as a gauge.

Save as trend.json:

{
  "query_workload": {
    "language": "promql",
    "repeating_queries": [
      {
        "query": "(sum(sum_over_time(sli_aggregations:pipelines_created_total:rate_5m[30d])) - sum(sum_over_time(sli_aggregations:pipelines_created_total:rate_5m[30d] offset 30d))) / sum(sum_over_time(sli_aggregations:pipelines_created_total:rate_5m[30d] offset 30d))",
        "demand": {
          "fixed_interval": 30000
        },
        "requirements": {
          "accuracy": {
            "max_error": 0.0,
            "failure_probability": 0.0
          }
        },
        "predictability": {
          "predictable": {
            "known_at": null
          }
        }
      }
    ]
  },
  "data_workload": {
    "arrival": {
      "ingesting": {
        "rows_per_second": 66.66666666666667
      }
    },
    "sources": {
      "sli_aggregations:pipelines_created_total:rate_5m": {
        "sample_interval": 15000,
        "series_count": 1000,
        "metric_type": "gauge"
      }
    }
  }
}

cargo build -p asap-devtools --bin stage_pipeline
# Start the timer after building, not around cargo compilation.
timeout 90s target/debug/stage_pipeline --workload trend.json \
  --enforce-accuracy-guarantees --max-displayed 100 --out trend.default.json
timeout 90s target/debug/stage_pipeline --workload trend.json \
  --enforce-accuracy-guarantees --exhaustive --max-combinations 10000 \
  --max-displayed 100 --out trend.exhaustive.json

The original probe used Python subprocess deadlines of 90 seconds. The shell commands express the equivalent cap; they do not include build time.

For the control, change only demand.fixed_interval from 30000 to 3600000 and rerun default search.

Current baseline

The hourly selected plan shares repeated historical computations and keeps two ingestion-time chains of 720 one-hour panes, exposing 1,440 stored outputs. Search checks 1,176 physical candidates. The 30-second configurations produced no final report, so their candidate counts and actual materialized pane counts are unknown. A 30-day window has 86,400 potential 30-second panes; this motivates profiling pane handling but does not establish the bottleneck.

Correctness constraints

Preserve the current window, the 30-day historical offset, repeated-denominator sharing, exact sum semantics, and the retention needed for both periods. Preserve distinctions between cadence, pane width, and evaluation phase. Do not remove legitimate candidates with an unproven cap or silently change the query's refresh interval.

Measurement method and acceptable variance

Repeat after a warm build on a controlled machine; vary cadence (30s, 1m, 5m, 1h) with all other inputs fixed, and report multiple timing samples, peak RSS, stage timings, and candidate counts. Use smaller windows to compare default and exhaustive selection costs and execution outputs. The existing observations establish the deadline exceedance on the tested machine, not an exact speedup ratio.