update sdk with new adds

[luke-e-schaefer]

Summary — v0.19.0: EvaluationV2, benchmark paradigm

Brings the SDK to parity with the Nucleus EvaluationV2 UI. Evaluation creation is benchmark-scoped only — dataset/slice-scoped creation (an unreleased earlier iteration of this branch) was removed before release as the platform moved to the benchmark paradigm.

Added

• Benchmarks. create_benchmark() (members from item_ids, (dataset_id, ref_id) items pairs, a slice_id, or a dataset_id — exactly one required; membership frozen at creation), list_benchmarks(), get_benchmark(), update_benchmark(), delete_benchmark(), list_benchmark_items(), plus the Benchmark resource (refresh() / update() / delete() / items() / create_evaluation_v2()) in nucleus/benchmark.py.
• Benchmark evaluations. create_benchmark_evaluation_v2(benchmark_id, model_run_id, ...) — every benchmark item is scored (uncovered items count as FN, keeping leaderboard scores comparable). EvaluationV2 exposes benchmark_id, rollup_groups, exclusion_rules, exclusion_stats. No only_items_with_predictions / eval-scope slice_id (invalid in the benchmark paradigm; the server rejects the former).
• Rollup groups. RollupGroup (class_name + labels) as the primary label configuration on benchmark eval create and presets (create/update_evaluation_v2_preset(rollup_groups=...), mutually exclusive with legacy allowed_label_matches; parsed from both key casings). Preset seeding prefers rollup_groups, falling back to legacy matches.
• Exclusion rules. MetadataExclusionRule / LabelExclusionRule / BoxAreaExclusionRule via exclusion_rules on eval create and presets.
• Presets. list/create/update/delete_evaluation_v2_preset + EvaluationV2Preset resource; preset= seeds benchmark eval creation (explicit args override).
• Results. EvaluationV2.charts() (mAP, per-class AP, confusion matrix, PR/F1 curves, TIDE, AP by size) and examples() (paginated TP/FP/FN; match_type optional) with EvaluationV2FilterArgs filtering; cancel() / retry(); Dataset.evaluation_label_schema().
• Benchmark leaderboards. leaderboard_ranking(metric_type, benchmark_ids, ...) (metrics: MAP_50, MAP_50_95, AP_SMALL/MEDIUM/LARGE, PRECISION, RECALL, F1; scope/collapse) and leaderboard_f1_curve(benchmark_ids, ..., top_n=5).
• Filter schema discovery. EvaluationV2.filter_schema() / get_evaluation_v2_filter_schema() return the evaluation's filter vocabulary (gt_labels, pred_labels, metadata_fields with inferred value types).
• New DTOs: EvaluationV2FilterSchema, LeaderboardRankingEntry, LeaderboardF1CurveEntry, BenchmarkItemsPage.

Removed (relative to earlier commits on this branch — never released)

• create_evaluation_v2 (dataset/slice-scoped), create_evaluations_v2_batch, BatchEvaluationResult, only_items_with_predictions. Reading existing evals (including old slice-scoped ones) still works.

Fixed

• CI build_test pylint failure (pre-existing W0718 on the per-job catch) — suppressed with a black-stable # pylint: disable-next line.

Tests / Version

• tests/test_benchmarks.py, tests/test_leaderboard.py, plus rollup/benchmark coverage in the eval/preset test files (59 unit tests, mock-based).
• pyproject.toml → 0.19.0; single consolidated CHANGELOG entry (the unreleased 0.18.9 section was folded in).

Server dependency: the leaderboard and filter-schema methods call new REST mirrors (POST /nucleus/leaderboard/ranking, POST /nucleus/leaderboard/f1Curve, GET /nucleus/evaluationsV2/:id/filterSchema) that ship separately in scaleapi. Unit tests pass regardless; live calls 404 until that deploys. All benchmark CRUD / benchmark-eval / rollup-group / preset endpoints are already live (DE-8209).

resolves https://linear.app/scale-epd/issue/DE-8209

Greptile Summary

This PR brings the Python SDK to parity with the Nucleus EvaluationV2 UI under a benchmark paradigm: benchmarks are frozen ground-truth item sets that model runs are evaluated against, enabling comparable leaderboard scores across runs.

• Benchmarks (nucleus/benchmark.py, NucleusClient): full CRUD (create_benchmark, list_benchmarks, get_benchmark, update_benchmark, delete_benchmark, list_benchmark_items) plus a Benchmark resource object with refresh/update/delete/items/create_evaluation_v2.
• Benchmark evaluations (create_benchmark_evaluation_v2): supports rollup_groups (new primary label config), legacy allowed_label_matches*, exclusion rules, and preset seeding; adds cancel, retry, filter_schema to EvaluationV2; migrates charts() from GET to POST.
• Presets, exclusion rules, leaderboard (evaluation_v2_preset.py, evaluation_v2_exclusions.py): new EvaluationV2Preset with per-user CRUD; MetadataExclusionRule, LabelExclusionRule, BoxAreaExclusionRule; leaderboard_ranking and leaderboard_f1_curve client methods backed by new server endpoints.

Confidence Score: 5/5

• Safe to merge. The change is a well-structured additive surface — new dataclasses, resource methods, and client methods — with 59 mock-based unit tests. No existing behavior is broken; the charts() GET→POST migration matches the server's new requirement.
• All findings are style or API-usability nits (empty-list fallback in preset parsing, missing BoxAreaExclusionRule guard, update_benchmark unable to clear optional fields, inconsistent key constants). None affect correctness of the live happy-path flows. The core benchmark CRUD, evaluation create, leaderboard, and preset flows are logically correct and covered by tests.
• nucleus/evaluation_v2_preset.py (allowed_label_matches fallback with or) and nucleus/evaluation_v2_exclusions.py (BoxAreaExclusionRule missing bound guard) are worth a second look before the next SDK release, but neither blocks this one.

Important Files Changed

Filename	Overview
nucleus/benchmark.py	New file — clean Benchmark dataclass with from_json, refresh, update, delete, items, and create_evaluation_v2 instance methods that delegate to NucleusClient; no logic issues found.
nucleus/evaluation_v2.py	Adds RollupGroup, _parse_rollup_groups, _parse_allowed_label_matches, _parse_json_field helpers; extends EvaluationV2 with benchmark_id, rollup_groups, exclusion_rules, exclusion_stats; adds cancel, retry, filter_schema; migrates charts from GET+QS to POST; makes match_type optional in examples. Logic is sound.
nucleus/evaluation_v2_exclusions.py	New file with MetadataExclusionRule, LabelExclusionRule, BoxAreaExclusionRule. BoxAreaExclusionRule lacks the client-side guard its docstring implies ("at least one bound required"), so an empty rule reaches the server before failing.
nucleus/evaluation_v2_preset.py	New EvaluationV2Preset dataclass with from_json, update, delete; handles both camelCase/snake_case response keys and JSON-encoded fields. allowed_label_matches fallback uses or rather than an explicit is not None check, which incorrectly treats an empty list as absent.
nucleus/init.py	Large addition of benchmark CRUD, benchmark evaluation create, preset CRUD, and leaderboard methods. update_benchmark cannot clear optional fields (None-guard silently no-ops); create_benchmark_evaluation_v2 uses inline string literals for legacy label-match keys where constants exist.
nucleus/data_transfer_object/evaluation_v2.py	Adds EvaluationV2FilterSchema, LeaderboardRankingEntry, LeaderboardF1CurveEntry, BenchmarkItemsPage; extends filter args with gt_area_range and slice_ids. Required fields on schema models look correct.

Sequence Diagram

sequenceDiagram
    participant User
    participant NucleusClient
    participant API

    User->>NucleusClient: "create_benchmark(name, slice_id=...)"
    NucleusClient->>API: POST /benchmarks
    API-->>NucleusClient: "{benchmark_id, ...}"
    NucleusClient-->>User: Benchmark

    User->>NucleusClient: "create_benchmark_evaluation_v2(benchmark_id, model_run_id, rollup_groups=..., preset=...)"
    Note over NucleusClient: Seed from preset if no explicit label config
    NucleusClient->>API: "POST /benchmarks/{id}/evaluationsV2"
    API-->>NucleusClient: "{evaluation_id}"
    NucleusClient->>API: "GET /evaluationsV2/{evaluation_id}"
    API-->>NucleusClient: EvaluationV2 payload
    NucleusClient-->>User: EvaluationV2

    User->>NucleusClient: evaluation.wait_for_completion()
    loop Poll status
        NucleusClient->>API: "GET /evaluationsV2/{id}"
        API-->>NucleusClient: status
    end

    User->>NucleusClient: "evaluation.charts(iou_threshold=0.5)"
    NucleusClient->>API: "POST /evaluationsV2/{id}/charts"
    API-->>NucleusClient: EvaluationV2Charts

    User->>NucleusClient: leaderboard_ranking(metric_type, benchmark_ids)
    NucleusClient->>API: POST /leaderboard/ranking
    API-->>NucleusClient: List[LeaderboardRankingEntry]
    NucleusClient-->>User: ranked entries

Loading

_{Reviews (10): Last reviewed commit: "Route benchmark eval create through the ..." | Re-trigger Greptile}

[jaypsiri]

Had a few questions, but mostly looks good to me! I would defer to @vinay553 and @edwinpav's review on this though