SaaS, AI & Developer Infrastructure
AI Evaluation API
You can tell when your model output got worse because a customer emails you. That is not a measurement system, and it will not survive your next prompt change.
- AI product companies
- Agent platforms
- LLM tooling vendors
- Enterprise AI teams
- Model hosting providers
What your customers would see
example interfaceMockupIllustrative mockup of the interface we would build. Figures shown are examples, not results from a live deployment.
The problem this solves
AI features ship faster than the ability to tell whether they still work. A prompt gets tweaked, a model version changes underneath you, a retrieval index is rebuilt, and output quality shifts in a way nobody notices until a customer complains about something specific. The team then argues about whether it was ever better, because no one has a number from before.
The reason this stays unsolved is that evaluation is more tedious than it is difficult. You need a held-out set of cases with known good answers, graders that correlate with human judgement, a harness that runs the same suite on every change, and the discipline to keep the eval set from leaking into the prompts being tested. None of that is research. All of it gets deprioritised against features.
It is also the kind of thing a platform should sell rather than rebuild. Every AI product on your platform needs regression testing, and the infrastructure — case storage, grader orchestration, run comparison, cost accounting — is identical regardless of what the customer's model does. One build, every tenant, priced per evaluation run.
Who buys this
AI product companies, Agent platforms, LLM tooling vendors, Enterprise AI teams, Model hosting providers
What the API returns
What comes back
- Per-case results with the grader's reasoning, not just a pass or fail
- Aggregate scores by capability, case category and difficulty band
- Regression comparison between runs, with the specific cases that changed
- Hallucination and groundedness scoring against supplied reference material
- Inter-grader and human-agreement statistics, so the grader itself is measured
- Cost and latency recorded per run alongside quality, since the three trade against each other
- Statistical significance on score differences, so a two-point move is not reported as an improvement
- Eval-set hygiene reporting: contamination checks, case staleness and coverage gaps
Endpoints
| Method | Path | Returns |
|---|---|---|
| POST | /v1/evals | Define an evaluation suite with cases, graders and thresholds |
| POST | /v1/runs | Run a suite against a model, prompt or pipeline version |
| GET | /v1/runs/{id} | Run results with per-case detail and aggregate scores |
| POST | /v1/runs/compare | Compare two runs and return the cases that regressed |
| POST | /v1/graders/calibrate | Measure a grader against human-labelled judgements |
| GET | /v1/suites/{id}/hygiene | Contamination, staleness and coverage report for a suite |
Request and response
illustrativeRequest
{
"tenant_id": "supportai",
"suite_id": "suite_support_answers_v9",
"candidate": {
"label": "prompt_v14 + retrieval_v3",
"endpoint": "https://api.yourplatform.com/internal/answer"
},
"baseline_run_id": "run_8812",
"graders": ["groundedness", "answer_correctness", "refusal_appropriateness"],
"require_significance": 0.95
}Response
{
"run_id": "run_8914",
"suite_id": "suite_support_answers_v9",
"cases": 1240,
"completed_at": "2026-10-22T11:02:41Z",
"cost_usd": 18.42,
"mean_latency_ms": 2110,
"scores": {
"answer_correctness": { "value": 0.871, "baseline": 0.864, "delta": 0.007,
"significant": false, "ci95": [0.852, 0.890] },
"groundedness": { "value": 0.942, "baseline": 0.905, "delta": 0.037,
"significant": true, "ci95": [0.928, 0.956] },
"refusal_appropriateness": { "value": 0.780, "baseline": 0.841, "delta": -0.061,
"significant": true, "ci95": [0.754, 0.806] }
},
"regressions": [
{ "case_id": "c_0441", "category": "out_of_scope_question",
"was": "refused correctly", "now": "answered from unrelated passage" },
{ "case_id": "c_0912", "category": "multi_hop",
"was": "correct", "now": "correct but ungrounded citation" }
],
"grader_calibration": { "groundedness_vs_human_kappa": 0.78, "sample": 120 },
"hygiene": { "contamination_suspected": 0, "cases_older_than_12m": 214 }
}Where the data comes from
| Your own production traffic | Sampled real queries, de-identified and labelled, which make a far better eval set than anything synthetic |
| Human-labelled judgements | A sample graded by people who know the domain — without this there is no way to know whether an automated grader means anything |
| Public benchmark suites | Useful for coarse comparison and for nothing else, since widely published sets are the ones most likely to be in a model's training data |
| Customer-supplied reference material | The documents an answer is supposed to be grounded in, which is what makes hallucination measurable rather than subjective |
| Adversarial and edge-case sets | Deliberately constructed cases for refusals, prompt injection, ambiguity and out-of-scope questions |
| Model and prompt version history | Metadata linking each run to exactly what was tested, because an unattributed score is not evidence of anything |
The intelligence layer
The difference between a demo and a service is mostly in this section. These are the parts that keep the answer trustworthy once real traffic is hitting it.
- Eval sets held strictly out of prompt development, with contamination checks, because the fastest way to a great score is to accidentally test on your examples
- Graders calibrated against human judgement and reported with agreement statistics, so a grader's own reliability is visible
- Stratified case sampling by category and difficulty, since an aggregate score dominated by easy cases moves for the wrong reasons
- Statistical testing on run differences, with confidence intervals reported instead of bare deltas
- Groundedness evaluation against supplied references rather than against the grader's own knowledge
- Pairwise and reference-free grading where no single correct answer exists, with the limitations of each stated in the output
- Cost and latency captured per run, because a quality gain bought with four times the tokens is a trade-off and not a win
- Drift monitoring on production samples, which catches a silent upstream model change that no code deployment would explain
What it changes for the business
Changes you can ship with confidence
A regression suite turns prompt and model changes from a leap of faith into a measured decision.
Faster debugging
Per-case regression lists point at the specific behaviour that broke rather than at a score that moved.
Evidence for buyers
Enterprise customers increasingly ask how you measure your AI features; a documented harness answers that in one page.
Usage revenue from every tenant
Each customer building on your platform needs their own suites, so evaluation volume grows with their development activity.
How it fits into your product
How it embeds
- White-label API and dashboards under your brand, with no MLPivot marks
- CI integration so suites run on every prompt, model or retrieval change rather than when someone remembers
- Per-tenant suites, cases and graders with strict isolation, since eval sets are commercially sensitive
- Webhooks on regression and threshold breach into your existing alerting
- Cost caps and sampling controls per tenant, because a full suite on every commit gets expensive quickly
- In-tenant deployment where production traffic used as eval material cannot leave your customers' environment
Included in every build
- Authentication, scoped API keys and per-customer rate limiting
- Input validation with predictable, documented error responses
- Versioned endpoints so your integration never breaks without notice
- Request and response logging for audit and dispute resolution
- Latency, error-rate and usage monitoring with alerting
- OpenAPI specification, sandbox environment and client examples
- Backups, failover design and a documented recovery procedure
- An agreed support arrangement once the service is live
How we would build it
about 12 weeksDiscovery and data review
Weeks 1–2
We agree the exact question the API answers, review the sources you already hold, check what has to be licensed, and write down the response contract your developers will code against.
Architecture and prototype
Weeks 3–4
You get a written architecture plus a working endpoint on real data — narrow, but genuinely callable — so integration can be tested against something concrete rather than a document.
Production build
Weeks 5–10
Pipelines, models, storage, authentication, rate limiting, logging, tests and documentation. Security and observability are part of this phase, not a later one.
Integration and handover
Weeks 11–12
We support your engineers through their integration, run load tests at your expected volume, and agree how the service is monitored and supported once it is live.
Commercial models
There is no fixed price list, because the cost is driven by data complexity, integration count, model requirements, request volume and the support level you need. Scope is agreed in writing before development starts.
Fixed-scope prototype
A defined slice of the service, built and priced as one piece of work. The usual first step when the business case needs proving internally.
Fixed-price build
An agreed scope against a written architecture, delivered for an agreed price. Best when the requirements are already clear.
Build then retainer
We build the service, then operate and improve it under a monthly agreement covering monitoring, updates and model maintenance.
White-label product
The service runs under your brand, your domain and your pricing, with MLPivot invisible to your customers.
Questions buyers ask
Can you tell us our model is good?
We can tell you how it scores on a specific set of cases with specific graders, which is a narrower claim and the only honest one. An evaluation is a measurement of a sample, not a property of a model. If the eval set does not represent what your customers actually ask, a high score means your eval set is easy.
Are benchmark scores worth anything?
Less than their prominence suggests. Public benchmarks leak into training data, get optimised against directly, and reward the narrow behaviour they measure. They are useful as a smoke test and as a shared vocabulary. Treating a leaderboard position as evidence about your own workload is a mistake, and we would rather build you a small suite of your own cases than report a published benchmark number.
Should we trust a vendor's evaluation of their own model?
No, and that includes evaluations we help build if the cases come from the same team tuning the prompts. A vendor-run eval of a vendor's model has every incentive problem you would expect: case selection, grader choice and run cherry-picking all point one way. The structural fix is an eval set held by whoever is buying the outcome, with human-labelled calibration they control.
Can an LLM grade another LLM's output reliably?
For some tasks, within limits that need measuring. Model graders show position and verbosity biases and tend to agree with outputs that resemble their own style. That is why calibration against human labels is part of the build and the agreement statistic is returned with every run. Where agreement is poor, the right answer is more human grading, not a more confident grader.
How many cases do we need?
Enough to detect the size of change you care about, which for small quality differences is more than teams expect. A few hundred well-stratified cases will catch obvious regressions; distinguishing a one-point improvement reliably needs considerably more. The API reports confidence intervals so an underpowered suite is visible rather than misleading.
Related blueprints
Web Research API
Structured research answers with source references and freshness metadata, built as the retrieval layer behind your AI agent platform.
API Observability Intelligence API
Anomaly detection, latency trend analysis, error clustering and incident forecasting for API traffic, embedded in your developer platform.
Document Extraction API
Structured fields pulled from PDFs, scans and forms, built as the extraction layer your ERP, CRM or accounting product embeds.
Want this built for your platform?
Tell us what your product does today and what you want another system to be able to ask it. We will come back with an approach, a scope and a price.