Most of the great threads here focus on iterating Genie Space quality: curating instructions, adding SQL expressions, fixing failures.
What I see discussed less often is the step that makes all of that repeatable: how you measure whether a Space is actually getting better.
A Genie Space drifts as tables, definitions, and user questions change. Without a benchmark, you only find out it regressed when a user does.
So I wanted to start a thread on treating evaluation as a first-class practice, pairing a small benchmark suite with production monitoring, and hear how others are doing it.
Metrics that actually matter
Benchmark accuracy: a numeric score across your benchmark questions. In Chat mode, a response is marked Good when the SQL or result set matches ground truth (tolerant of sort order and rounding to ~4 significant digits).
Failure rate: the fast signals are SQL errors, empty result sets, extra columns, or incorrect single-cell answers.
Manual-review share: benchmarks without a SQL ground truth need manual review. A high share usually means the benchmark pack isn't operationalised yet.
Coverage of real questions: benchmark the questions users actually ask, and include 2 to 4 phrasings of each so you test language variation, not one canonical wording.
User-feedback trend: thumbs up/down, “Fix it”, “Request review”, weekly message volume and active users from the Monitor tab.
Ownership and cadence: a named owner and a regular (monthly works well) review to harvest failed questions and feed them back.
Benchmark tests worth including
1. Business-logic regression tests
Prove Genie understands your definitions, not just generic SQL. Examples: “What is our current open pipeline by region?” (pipeline = forecast category Pipeline, excluding closed stages), “What is the win rate this quarter?”, “Show high-value orders by deal size”. Lock the metric logic down so it can't silently drift.
2. Rephrasing tests
Try one question with multiple phrasings: “open pipeline by region,” “pipeline split by region,” or “how much pipeline in each region right now.” The Pass condition is that all variations map to the same ground-truth SQL or result set.
3. Synonym and entity-matching tests
The classic real-world failure: users say one thing, the data stores another. “Car sales in Florida” when the column stores FL. Without entity matching, Genie may write ILIKE '%Florida%'; with it, state = 'FL'. The same pattern applies to product names, statuses, departments, and country/state codes.
4. Join-accuracy tests
Questions that force a correct join, for example, “open pipeline by region” joining opportunities to accounts before aggregating. If they fail, the fix is usually explicit join relationships or SQL expressions, not more vague text instructions.
5. Clarification-behaviour tests
Include prompts where Genie should ask a follow-up instead of guessing, for example, “sales performance breakdown” with no time range, channel, or KPI. This tests whether it knows when not to answer prematurely.
6. Verified-answer / trusted-asset tests
For high-value, board-reporting-style questions where drift is costly (“Open pipeline for EMEA”, “Win rate for Q2”, “Large orders in Germany last month”), use parameterised example SQL so that Genie can return verified answers.
7. Agent-mode quality tests
For multi-step/narrative tasks not reducible to one SQL result, grade with an LLM judge against an evaluation note. Example: “Summarise the main drivers of pipeline change this quarter and highlight outliers by region”. Pass criteria might require a regional breakdown, date-range coverage, and explicit mention of outliers.
A starter pack that you can try out now!
Pick 5 to 10 top business questions
Consider each with 2 to 4 alternate phrasings
Try to perform mostly Chat-mode tests with SQL ground truth
Finally, include a few Agent-mode tests for narrative/exploratory tasks
The operational loop
The pattern that seems to work: score, keep the scan history, run a monthly review with users, then feed back failed questions as new examples or instructions.
A nice variant: admins spot repeated prompts in monitoring, then encode the efficient query into config. Call it an observe, a benchmark, or a promotion to trusted logic flow.
Over to you: What's in your benchmark pack, and what's your review cadence? Are you scoring in Chat mode, Agent mode, or both, and which failure class (definitions, synonyms, or clarification) bites you most?