Most Genie quality conversations start the same way. A business user says an answer looks wrong, someone opens the space, reads the generated SQL, spots the missing filter or the wrong join, and adjusts an instruction. That work matters, and it is a good skill to have. On its own, though, it only ever fixes the single answer in front of you.
The question one would care about is one level up.
How do you know whether your Genie Space is getting better or worse over time, across the hundreds of questions you never personally see?
That is an evaluation practice, and it is different from debugging.
Debugging tells you why one answer failed. Evaluation tells you whether quality is trending in the right direction.
Here is how you could build that practice on what the platform already offers you.
1. Start with a benchmark set, not a hunch
Genie has a built-in Benchmarks feature, and it is the natural foundation. A benchmark is a set of test questions you run to measure a space's response accuracy, and each agent supports up to 500 of them. You add a question, pick a mode, and attach a SQL query that represents the correct answer. That SQL is your ground truth.
The scoring is stricter than a thumbs-up. In chat mode, a response is rated good when it matches your SQL, or returns the same result set even if sorted differently, or matches numeric values to four significant digits. Anything else is rated badly. In agent mode, an LLM judge scores the response against the evaluation criteria you write. Either way, you now have a repeatable number instead of a feeling.
2. Make accuracy a trend, not a snapshot
The point of a benchmark set is that you rerun it. Improve an instruction, add a worked example, curate a table, then rerun the set and see whether overall accuracy moved. The Evaluations tab keeps that history, so you can watch the accuracy line over time rather than guessing whether your last change helped.
This is the habit worth forming. Every meaningful change to a space becomes a small experiment with a before-and-after number. Genie Code can even analyse a run across all questions and suggest context improvements, which turns a failed benchmark into a concrete next edit rather than a vague worry.
3. Close the loop with real users
A benchmark set reflects the questions you thought of. Real users will always surprise you, so the second half of the practice is capturing what actually happens in production. Request Review lets a user flag an answer for the space authors to verify or correct. Authors see the original prompt and the generated SQL, then either confirm the answer or fix the query, and the user hears back.
The valuable move is to treat verified questions as feedstock. When a reviewed question turns out to be common, promote it into your benchmark set with its corrected SQL as ground truth. Now, real usage is continuously widening the very thing you measure against, and yesterday's surprise becomes today's regression test.
4. When to reach past built-in benchmarks - is MLflow a good fit for me?
Benchmarks cover a lot, and for most spaces, they are the right first tool. There is a clear line where you outgrow them, though, and it is worth knowing where that line sits.
Reach for the broader MLflow evaluation harness when you need things that benchmarks do not give you. That includes custom LLM judge scorers for dimensions like groundedness, evaluation across an agent that chains several tools rather than a single space, richer trend and cost metrics pulled from traces, and production monitoring that scores live traffic without ground truth answers. In practice, you use mlflow.evaluate() With an evaluation dataset of representative questions and optional expected answers, and you get per-row rationales plus aggregate pass rates.
A simple rule of thumb. Use Genie Benchmarks to measure and improve a single space against known correct SQL. Step up to MLflow when you need custom judges, multi-step agents, or always-on production scoring. Start with the first, then graduate to the second when the questions get harder than the built-in tool is designed to answer.
A practice you can actually run
Putting it together, here is the loop I would set up for a space that matters.
Baseline: Prepare the space with clean tables and metadata, then hand-test the common questions to gauge where you stand.
Build the benchmark set: Capture those common questions with authoritative SQL as ground truth, including a few different phrasings of the same intent.
Run and read: Execute the set, look at good versus bad, and note which questions fail and why.
Improve and rerun: Fix the instructions and examples, rerun the set, and confirm that the accuracy line has moved up in the Evaluations tab.
Feed the loop: Pull commonly reviewed questions from Request Review into the benchmark set, so coverage grows with real usage.
Escalate when needed: When you need custom judges, multi-step agents, or live production scoring, move that space onto MLflow evaluation.
Why this matters
Trust in a Genie Space is not a one-time setup task. Data changes, questions drift, and instructions that were right last quarter quietly go stale. A space with an evaluation practice catches that drift as a number moving, while a space without one only finds out when a user complains. The tools to run this are already here. The habit is the part you have to build.
A few questions for you:
Do you run a benchmark set on your important spaces, and how many questions does it hold?
How do you decide when a space has outgrown built in benchmarks and needs MLflow evaluation?
What is your cadence for rerunning benchmarks and folding in reviewed questions?
If you want to share and learn more than just Genie evaluation patterns, connect with me on LinkedIn: linkedin.com/in/lingeshwarankanniappan
References:
Building confidence in your Genie Space with Benchmarks and Ask for Review, Databricks blog: databricks.com/blog
Genie Benchmarks, Databricks documentation: docs.databricks.com
Agent evaluation with MLflow, Databricks documentation: docs.databricks.com