Did the system find the right evidence?
Recall, precision, relevance, correct metadata and permission enforcement.
MADAI / AI EVALUATION
Evaluation that separates the model, retrieval and action.
One overall impression is not enough. We measure source retrieval, answer quality, tool selection, action execution, safety and operating cost separately.
In brief
Production evaluation is a versioned test system. It compares models, prompts, retrieval and tools and catches regressions before a change reaches users.
01 / FRAMEWORK
Measure the system before you scale it.
Architecture, data, models and automation are designed around specific work scenarios and tests, not around a general impression from a chat interface.
Was the answer correct and grounded?
Correctness, completeness, citations, uncertainty, format and domain rules.
Did it use the right tool safely?
Tool selection, parameters, step order, approval gate and final state.
02 / IMPLEMENTATION
What has to be ready before production
The result must be repeatable, traceable and usable in operations. Model quality, retrieval, tool actions, permissions, latency and human decisions need separate failure categories.
- 01Golden set
A versioned set of real queries, sources and expected outcomes.
- 02Failure taxonomy
Retrieval, reasoning, tool use, permissions, latency and UX failure categories.
- 03Regression gate
Automated checks before changing model, prompt, chunking or connector.
- 04Production feedback
Separate user feedback, incidents and sampled expert review.
03 / DECISIONS
Questions that directly affect architecture.
The answers determine pilot scope, autonomy, model selection and operating cost.
Is a general model benchmark enough?+
No. General benchmarks help with pre-selection, but production decisions need tasks, language, data and risk from the actual system.
How should RAG be evaluated?+
Evaluate retrieval and generation separately. Otherwise you cannot tell whether the model answered badly or never received the right evidence.
How should agents be tested?+
Check not only the final result but also the route: chosen tool, parameters, permissions, approvals, side effects and safe stop behaviour.
Start with a task that can be tested.
We define the data, expected output, risks and evaluation set before choosing the model and integrations.