Repository navigation
feat(gooddata-eval): score dashboard layout and the exact filter bar - #1910
Conversation
|
No actionable comments were generated in the recent review.
 |
| Layer / File(s) | Summary |
|---|---|
Expectation validation and applicability packages/gooddata-eval/src/gooddata_eval/core/agentic/dashboard_skill.py |
Validates layout and filter-bar expectation shapes, checks tab entries, and enables each check only when its expectation is present. |
Layout and filter-bar comparisons packages/gooddata-eval/src/gooddata_eval/core/agentic/dashboard_skill.py, packages/gooddata-eval/tests/test_agentic_dashboard_skill.py |
Checks section count and order, titles, chart placements, and filter contents. Tests cover matching and mismatching layouts and filters, including malformed expectations and grouped filters. |
Evaluation result integration packages/gooddata-eval/src/gooddata_eval/core/agentic/dashboard_skill.py, packages/gooddata-eval/tests/test_agentic_dashboard_skill.py |
Runs applicable checks and includes their results and failures in evaluation output. Tests cover empty runs and cases without these expectations. |
Priority:
Estimated code review effort: 3 (Moderate) | ~25 minutes
Change: Feature
Suggested reviewers: myhoai, frankhuynh
Merge Risk: 
e7e1a
-
This adds opt-in layout and filter-bar scoring for dashboard evaluation, with no change to existing items that do not use the new expectation keys. No concrete merge-blocking issue was found in the supplied changes.
Pre-merge checks: 
-
Check name Status Explanation Resolution Docstring Coverage Docstring coverage is 56.25% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold. -
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title clearly and concisely describes the two main changes: scoring dashboard layout and exact filter-bar expectations in gooddata-eval. Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 56.25% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 32 functions across 2 files.
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
-
Finishing Touches: 💡 1
-
📝 Generate docstrings: 💡
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
A rabbit checks each chart in place,
Then sorts the filters with care and grace.
Sections line up, tabs pass the test,
Grouped filters join the rest.
The dashboard’s checks report their view,
And carrots wait when work is through.
Comment @coderabbitai help to get the list of available commands.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1910 +/- ##
==========================================
+ Coverage 85.75% 85.83% +0.07%
==========================================
Files 343 343
Lines 25182 25322 +140
==========================================
+ Hits 21595 21735 +140
Misses 3587 3587 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Moving or removing a widget, and removing or retyping a filter, leave nothing for the existing dashboard checks to see. `charts_matched` and `preserved_widgets` ask whether a chart is still on the dashboard, so a move that never happened, or left a second copy behind, passes, and so does a removal that never happened. `filters` ignores filters beyond the expected ones, and `tab_filters` reads only tabbed dashboards and needs every filter's key in advance, which the agent mints at run time for a new or re-added filter.
Two opt-in expectation keys close that:
- `layout`: per tab, the exact sections in order, each with its exact charts in order (an id, or `{"title"}` for an authored chart), and the section title when stated. Scored as `dashboard_layout_correct`.
- `filter_bar`: per tab, the exact filters, the kept ones by key under `filters` and the ones with a minted key matched by content under `added`. Anything left over fails, an empty filter group included. A grouped filter counts by its own key and names its group's title as `group`; without it, the filter must stand outside every group, so a move into or out of a group fails. It reads a dashboard without tabs too. Scored as `dashboard_filter_bar_correct`.
An entry without `tab` applies only to a dashboard with one tab, and must be the only entry. `tab_filters` now runs through the same check, so the two read filter groups alike; no existing item names a group, and re-scoring 103 saved `tab_filters` runs changes no verdict. A malformed entry is rejected before the first request. Without either key, nothing changes for existing items.
Re-scored against 93 saved runs of the QA-29514 items on ecommerce_demo, the results match the expected outcome of every run.
jira: QA-29514
risk: low
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
22b5f37 to
e7e1a44
Compare
Summary
Two opt-in expectation keys for the
agentic_dashboard_skillevaluator. They let eval items cover moving and removing widgets and removing or retyping filters (QA-29514). The existing checks cannot see any of these:charts_matchedandpreserved_widgetsonly check that a chart is still on the dashboard. A move that never happened passes, so does a move that left a second copy behind, and so does a removal that never happened.filtersignores filters beyond the expected ones, so a filter that should be gone is invisible to it.tab_filtersis exact, but it reads tabbed dashboards only and needs each filter's key in advance. The agent mints a new key at run time for a filter it adds, or removes and re-adds (seen when a list filter is turned into a text filter).layout{"title"}for an authored chart), and the section title when stateddashboard_layout_correctfilter_barfilters); minted-key ones by content (added, matched one-to-one). Anything left over fails, an empty filter group included. A grouped filter counts by its own key and names its group's title asgroup; without it, it must stand outside every group. Works on dashboards without tabsdashboard_filter_bar_correcttabapplies only to a dashboard with one tab, and must be the only entry.tab_filtersnow runs through thefilter_barcheck, so both read filter groups alike. No existing item names a group; re-scoring 103 savedtab_filtersruns changes no verdict._validate_expectation, before any request.Test Plan
pytest packages/gooddata-eval/tests: 2528 passed, 1 skipped. AddsTestLayoutandTestFilterBar, using patch operations trimmed from real runs: a move between tabs, a removal that empties a section, and a list filter turned into a text filter. Both new scores are also added to the "empty run fails every applied check" test.ruff format,ruff check,ty check --project packages/gooddata-eval packages/gooddata-eval/src: clean.ecommerce_demopreview workspace on stg11). The new checks agree with the outcome expected for every run; the 3 failing runs are the already-diagnosed ones. A negative control on the unpatched dashboards failslayouton all 10 move/delete items.tests/tavern-e2e. The companion gdc-nas commit adds both scores tocombo_report._DASHBOARD_CONDITIONAL_FIELDS.JIRA: QA-29514
Risk: low. Test-tooling package. Both checks are opt-in, so existing items and old traces score the same.
🤖 Generated with Claude Code
Summary by CodeRabbit