SaaS, AI & Developer Infrastructure
Content Moderation API
The day your product lets users post something, you acquire a moderation problem. The question is whether you face it deliberately or in response to an incident.
- Marketplace and community platforms
- Social and messaging software
- Dating and gaming platforms
- Review and listing sites
- Trust and safety vendors
What your customers would see
example interfaceMockupIllustrative mockup of the interface we would build. Figures shown are examples, not results from a live deployment.
The problem this solves
Moderation arrives unplanned. A product adds comments, reviews, listings or messages, and for a while the volume is small enough that someone on the team reads the reports. Then it is not, and the response is usually a keyword blocklist — which blocks a town called Scunthorpe, misses every deliberate evasion, and gives nobody a record of why anything was removed.
The genuinely hard part is not detecting slurs. It is context. The same sentence can be harassment, a quotation of harassment being reported, satire, or a community using reclaimed language among themselves. Automated classification is poor at telling these apart and will stay poor, because the distinction often depends on who is speaking, to whom, and with what history. A system designed as though the classifier decides is a system that will remove the wrong things at scale.
What works is a signals layer feeding a process that keeps humans in the consequential decisions. Scored policy categories with confidence, queue prioritisation so reviewers see the worst first, automatic action only where the evidence is overwhelming and the harm of inaction is high, and an audit record for every decision. That is buildable, measurable, and the right thing for a platform vendor to provide to every tenant on it.
Who buys this
Marketplace and community platforms, Social and messaging software, Dating and gaming platforms, Review and listing sites, Trust and safety vendors
What the API returns
What comes back
- Per-policy category scores with confidence, rather than a single harmful or safe verdict
- The specific spans that triggered each category, so a reviewer sees the evidence
- Evasion-aware text handling: character substitution, spacing, transliteration and coded terms
- Image, video-frame and audio signals alongside text, where the content type requires it
- Severity and priority scoring to order a review queue by actual harm
- Recommended action per item under your policy, including an explicit escalate-to-human
- Context signals: account history, reporting patterns, coordinated posting and velocity
- A full audit record per decision, retained for appeals and regulatory reporting
Endpoints
| Method | Path | Returns |
|---|---|---|
| POST | /v1/moderate | Score an item against your policy categories |
| POST | /v1/moderate/batch | Score a batch of items for backlog or bulk review |
| GET | /v1/queue | Review queue ordered by severity and confidence |
| POST | /v1/decisions | Record a reviewer decision, which feeds calibration |
| POST | /v1/policies | Define policy categories, thresholds and action rules |
| GET | /v1/reports/transparency | Aggregate decision statistics for transparency reporting |
Request and response
illustrativeRequest
{
"tenant_id": "marketplace_eu",
"item": {
"id": "msg_881204",
"type": "text",
"locale": "en-GB",
"content": "if you show up here again you'll regret it",
"surface": "buyer_seller_message"
},
"context": {
"author_account_age_days": 3,
"prior_upheld_reports": 1,
"thread_participants": 2
},
"policy_id": "pol_marketplace_v6"
}Response
{
"item_id": "msg_881204",
"scored_at": "2026-10-22T12:08:19Z",
"categories": [
{ "policy": "threats_of_violence", "score": 0.81, "confidence": "medium",
"spans": [ { "start": 22, "end": 38, "text": "you'll regret it" } ] },
{ "policy": "harassment", "score": 0.64, "confidence": "medium" },
{ "policy": "hate_speech", "score": 0.03, "confidence": "high" }
],
"context_signals": {
"new_account": true,
"prior_upheld_reports": 1,
"coordinated_posting": false
},
"severity": 72,
"recommended_action": "escalate_to_human",
"auto_action_eligible": false,
"auto_action_blocked_because": "ambiguous intent; consequential action requires review",
"appeal_reference": "apl_ref_881204",
"audit": { "policy_version": "pol_marketplace_v6", "model_version": "mod_2026_09" }
}Where the data comes from
| Your own reviewed content | Items your moderators have already actioned, with the decision and the policy cited — the only training data that reflects your actual policy |
| Your written policy | The classifier has to be built against the policy as written, so vague policy text becomes a modelling problem; tightening it is part of the work |
| Open moderation research datasets | Published hate speech, harassment and toxicity corpora with licences permitting commercial use, useful for a baseline and inadequate alone |
| Known-illegal content hash lists | Industry hash-matching services for content that must be detected and reported, accessed under the programmes that govern them |
| Account and behavioural signals | Age, reporting history, posting velocity and network patterns, which frequently carry more signal than the text itself |
| Reviewer disagreement data | Cases where moderators disagreed, which mark the genuinely ambiguous regions of your policy rather than model error |
The intelligence layer
The difference between a demo and a service is mostly in this section. These are the parts that keep the answer trustworthy once real traffic is hitting it.
- Multi-label scoring per policy category instead of one toxicity number, because your policies have different thresholds and different consequences
- Span-level evidence returned with every category, since a reviewer deciding in seconds needs to see why
- Evasion normalisation for substitutions, spacing, homoglyphs and transliteration, retrained as evasion patterns shift — which they do in response to enforcement
- Per-language models rather than translate-then-classify, as translation destroys exactly the register and slang the decision depends on
- Calibration against your own reviewer decisions, with precision and recall reported separately per category and per language
- Threshold design that separates automatic action, queue prioritisation and no-action, with automatic action reserved for high-confidence high-harm categories
- Appeal-outcome tracking as a primary quality metric, because overturned decisions are the clearest evidence of classifier error
- Monitoring for drift, for sudden volume shifts suggesting a coordinated campaign, and for category-level fairness across language and dialect groups
What it changes for the business
Reviewers see the right things first
Severity ordering means limited moderator time goes to the most harmful items rather than to whatever arrived most recently.
Defensible decisions
A per-decision audit record with policy version and evidence is what you need when a user appeals or a regulator asks.
Coverage as you grow
Volume increases do not require a proportional increase in reviewers, which is usually the constraint that forces bad shortcuts.
One layer for every tenant
Marketplace, community and messaging customers on your platform all need this, and the policy configuration differs more than the engineering does.
How it fits into your product
How it embeds
- White-label API with your brand, your policy names and your thresholds throughout
- Synchronous scoring for pre-publication checks, asynchronous batch for backlogs and re-review
- Per-tenant policies, so each customer on your platform configures its own categories and actions
- An embeddable review queue with span highlighting and decision capture, or raw JSON for your own tooling
- Webhooks into your enforcement, appeal and escalation workflows
- Region-pinned deployment and retention controls for jurisdictions with specific requirements
Included in every build
- Authentication, scoped API keys and per-customer rate limiting
- Input validation with predictable, documented error responses
- Versioned endpoints so your integration never breaks without notice
- Request and response logging for audit and dispute resolution
- Latency, error-rate and usage monitoring with alerting
- OpenAPI specification, sandbox environment and client examples
- Backups, failover design and a documented recovery procedure
- An agreed support arrangement once the service is live
How we would build it
about 20 weeksDiscovery and feasibility
Weeks 1–3
This one depends heavily on data access, so we start by proving the sources exist, can be licensed for your use, and actually carry the signal the model needs. If they do not, you find out here.
Architecture and baseline model
Weeks 4–6
A written architecture and a baseline model measured against a holdout period, so you know what accuracy you are buying before the full build starts.
Production build
Weeks 7–16
Data pipelines, model training and serving, historical archive, validation harness, API, security, monitoring and documentation.
Validation and integration
Weeks 17–20
Backtesting against held-out history, accuracy reporting your team can audit, integration support and an agreed retraining cadence.
Commercial models
There is no fixed price list, because the cost is driven by data complexity, integration count, model requirements, request volume and the support level you need. Scope is agreed in writing before development starts.
Fixed-scope prototype
A defined slice of the service, built and priced as one piece of work. The usual first step when the business case needs proving internally.
Fixed-price build
An agreed scope against a written architecture, delivered for an agreed price. Best when the requirements are already clear.
Build then retainer
We build the service, then operate and improve it under a monthly agreement covering monitoring, updates and model maintenance.
White-label product
The service runs under your brand, your domain and your pricing, with MLPivot invisible to your customers.
Questions buyers ask
Can this moderate automatically without humans?
For a narrow band of cases, and we would resist widening it. Automatic removal makes sense where confidence is very high and the harm of leaving content up is severe — hash-matched known-illegal material being the clearest example. Everything else should prioritise a queue rather than take action. A system that auto-removes on a 0.7 score will remove satire, quoted abuse and reclaimed language, at volume, with no record anyone reads.
How well does it handle satire and reclaimed language?
Badly, in the sense that any automated classifier handles them badly. Irony, quotation of abuse in order to report it, in-group use of reclaimed slurs and regional humour all look like the thing they reference. We can reduce false positives with context signals and per-community calibration, and we can make the classifier abstain rather than guess. We cannot make it reliably correct, and a vendor claiming otherwise has not measured it on real content.
Does it work the same across languages and cultures?
No. Quality varies sharply by language, and norms vary by community within a single language — what counts as abusive in one market is ordinary directness in another. We report precision and recall per language and per category rather than one headline number, and where a language's performance is not good enough for prioritisation we would say so rather than ship it quietly.
What about the people doing the review?
Worth designing for explicitly. Reviewing harmful content causes real psychological harm, and a system that routes the worst material to humans efficiently also concentrates that exposure. Practical measures belong in the build: blurring and audio muting by default, graduated exposure, rotation away from the most severe queues, volume caps per shift, and not using reviewer throughput as the primary performance metric. If you intend to operate a moderation function, access to clinical support for the people doing it is not optional.
Can you guarantee compliance with online safety regulation?
No. We can build the mechanisms regulation expects — decision records, appeal references, transparency statistics, response-time measurement, flagged-content routing — and we can build them to the specification your legal advisers set. Whether your overall process complies is a legal judgement about your platform, not a property of an API, and any vendor promising compliance in a product description is overselling.
Related blueprints
Translation and Localisation API
Industry-specific translation with managed terminology and localisation checks, built as the multilingual layer inside your product.
AI Evaluation API
Accuracy tests, hallucination checks, regression suites and benchmark runs, built as the evaluation layer inside your AI product.
Document Extraction API
Structured fields pulled from PDFs, scans and forms, built as the extraction layer your ERP, CRM or accounting product embeds.
Want this built for your platform?
Tell us what your product does today and what you want another system to be able to ask it. We will come back with an approach, a scope and a price.