SaaS, AI & Developer Infrastructure
Data Quality and Deduplication API
Every platform that lets customers import records eventually becomes a platform full of duplicates. Your support team knows this better than your roadmap does.
- CRM vendors
- Marketing automation platforms
- ERP and finance software
- Data warehouse and iPaaS vendors
- Customer data platforms
What your customers would see
example interfaceMockupIllustrative mockup of the interface we would build. Figures shown are examples, not results from a live deployment.
The problem this solves
The arithmetic is unforgiving. A CRM with two hundred thousand contacts, three import sources and a web form will accumulate duplicates at a rate nobody plans for: the same person as J. Smith, Smith John and jsmith@, the same company as Northbridge Ltd, Northbridge Limited and NORTHBRIDGE LTD. Reports stop reconciling, two sales people call the same prospect, and the customer blames your software rather than their process.
Matching is harder than it looks because the easy cases are already handled. Exact-match and simple fuzzy comparison catch perhaps the obvious half. The remainder are the ones that matter: a company that moved address, a person who married and changed surname, two genuinely different people with the same name at the same employer, a supplier recorded once with a VAT number and once without. Each decision has a cost if you get it wrong, and merging two records that are not the same thing is far more expensive than leaving them apart.
For a platform vendor this is an unusually good thing to embed. Deduplication is not a feature your customers evaluate against competitors; it is a reason they stop complaining. One integration improves every tenant, and record volume gives you a usage-based line that grows as your customers' databases grow.
Who buys this
CRM vendors, Marketing automation platforms, ERP and finance software, Data warehouse and iPaaS vendors, Customer data platforms
What the API returns
What comes back
- Candidate duplicate pairs with a match score and the fields that drove it
- Clustered duplicate groups rather than pairwise lists, so a four-way duplicate is one decision
- Merge recommendations with field-level survivorship — which value to keep from which record
- Normalised values: names, addresses, phone numbers, emails, company names and identifiers
- Validation results: deliverable email syntax and domain, phone format by country, address resolvability
- Entity resolution across record types, so a contact links to the right company
- Confidence bands with an explicit review tier between automatic merge and automatic keep
- Audit records for every merge, including what was overwritten, so a merge can be reversed
Endpoints
| Method | Path | Returns |
|---|---|---|
| POST | /v1/match | Score a record against a candidate set and return matches |
| POST | /v1/dedupe | Find and cluster duplicates across a submitted dataset |
| POST | /v1/normalise | Standardise names, addresses, phones and identifiers |
| POST | /v1/validate | Validate contactability and format for a record |
| POST | /v1/merge/plan | Field-level survivorship plan for a duplicate cluster |
| GET | /v1/jobs/{id} | Status and results of a batch deduplication job |
Request and response
illustrativeRequest
{
"tenant_id": "acme_crm",
"record_type": "company",
"record": {
"name": "NORTHBRIDGE LTD.",
"address": "Unit 4, Carlow Rd, Dublin",
"vat_number": "IE6388047V",
"domain": null
},
"match_against": "tenant_companies",
"thresholds": { "auto_merge": 0.97, "review": 0.80 }
}Response
{
"request_id": "mat_55a19c",
"normalised": {
"name": "Northbridge Ltd",
"name_key": "northbridge",
"address": { "line1": "Unit 4, Carlow Road", "city": "Dublin", "country": "IE" },
"vat_number": "IE6388047V",
"vat_valid": true
},
"matches": [
{
"record_id": "cmp_00418",
"score": 0.98,
"decision": "auto_merge",
"evidence": [
{ "field": "vat_number", "weight": 0.61, "agreement": "exact" },
{ "field": "name", "weight": 0.27, "agreement": "normalised_exact" },
{ "field": "address", "weight": 0.10, "agreement": "partial" }
]
},
{
"record_id": "cmp_02211",
"score": 0.84,
"decision": "review",
"evidence": [ { "field": "name", "weight": 0.27, "agreement": "normalised_exact" } ],
"note": "Same name, different VAT number — likely a separate legal entity"
}
],
"merge_plan": {
"survivor": "cmp_00418",
"field_sources": { "name": "cmp_00418", "address": "incoming", "vat_number": "both_agree" }
}
}Where the data comes from
| Your tenants' own records | The dataset being cleaned; processed on their instruction, isolated per tenant and never pooled to improve matching for anyone else without permission |
| Official company registers | Registry identifiers, legal names and status, which resolve company matching far better than name similarity ever will |
| VAT and tax identifier validation services | Official validation endpoints, where a single confirmed identifier settles a match that fuzzy name comparison cannot |
| Postal address reference data | Licensed address files per country for standardisation and resolvability — licensing differs sharply by country and is checked per market |
| Email and phone validation infrastructure | Domain and mailbox checks and numbering-plan reference data, used without sending test messages to real people |
| Human merge decisions | Your reviewers' accept and reject actions, which are the most valuable training signal available and specific to each tenant's data |
The intelligence layer
The difference between a demo and a service is mostly in this section. These are the parts that keep the answer trustworthy once real traffic is hitting it.
- Blocking and candidate generation before scoring, because comparing every record to every other record does not scale past small datasets
- A probabilistic matching model with learned field weights, since an exact VAT match should outweigh an exact name match by a wide margin
- Transitive clustering with conflict detection, so A matching B and B matching C does not silently merge two different companies
- Asymmetric thresholds that reflect real cost: a wrong merge destroys data, a missed duplicate merely annoys, and the defaults are set accordingly
- Field-level survivorship rules per tenant, including recency, source priority and completeness, rather than a single winning record
- Reversible merges with a full audit record, because every deduplication project eventually needs to undo something
- Precision and recall measured on a human-labelled sample per tenant, reported separately — a single accuracy figure hides which error you are making
- Monitoring of duplicate creation rate by source, which usually identifies the import or form that is causing the problem
What it changes for the business
Reports that reconcile
Duplicate removal fixes the counting problems your customers currently blame on your reporting.
Fewer embarrassing moments
Two sales people calling the same prospect, or two invoices to the same supplier, stop being a weekly occurrence.
Cleaner data for everything else
Matching and normalisation make every downstream feature — scoring, segmentation, forecasting — measurably better.
Usage revenue tied to growth
Record volume rises as your customers' databases grow, so the line item expands without a renegotiation.
How it fits into your product
How it embeds
- White-label API with your brand and domain, invisible to your tenants' end users
- Real-time matching on record creation to prevent duplicates at the point of entry, plus batch jobs for existing backlogs
- Per-tenant configuration of thresholds and survivorship rules, because a finance system and a marketing list need different settings
- An embeddable review queue for the middle confidence band, or raw JSON for your own interface
- Per-tenant keys and metering so deduplication can be priced per record or per matched pair
- In-tenant or region-pinned deployment where customer records cannot leave their jurisdiction
Included in every build
- Authentication, scoped API keys and per-customer rate limiting
- Input validation with predictable, documented error responses
- Versioned endpoints so your integration never breaks without notice
- Request and response logging for audit and dispute resolution
- Latency, error-rate and usage monitoring with alerting
- OpenAPI specification, sandbox environment and client examples
- Backups, failover design and a documented recovery procedure
- An agreed support arrangement once the service is live
How we would build it
about 12 weeksDiscovery and data review
Weeks 1–2
We agree the exact question the API answers, review the sources you already hold, check what has to be licensed, and write down the response contract your developers will code against.
Architecture and prototype
Weeks 3–4
You get a written architecture plus a working endpoint on real data — narrow, but genuinely callable — so integration can be tested against something concrete rather than a document.
Production build
Weeks 5–10
Pipelines, models, storage, authentication, rate limiting, logging, tests and documentation. Security and observability are part of this phase, not a later one.
Integration and handover
Weeks 11–12
We support your engineers through their integration, run load tests at your expected volume, and agree how the service is monitored and supported once it is live.
Commercial models
There is no fixed price list, because the cost is driven by data complexity, integration count, model requirements, request volume and the support level you need. Scope is agreed in writing before development starts.
Fixed-scope prototype
A defined slice of the service, built and priced as one piece of work. The usual first step when the business case needs proving internally.
Fixed-price build
An agreed scope against a written architecture, delivered for an agreed price. Best when the requirements are already clear.
Build then retainer
We build the service, then operate and improve it under a monthly agreement covering monitoring, updates and model maintenance.
White-label product
The service runs under your brand, your domain and your pricing, with MLPivot invisible to your customers.
Questions buyers ask
Will it merge records automatically?
Only above a threshold you set, and we would push back on setting it low. Three bands work better than two: automatic merge where the evidence is near-certain, automatic keep-apart where it is weak, and human review in between. The review band is where the value is, and a vendor promising full automation is quietly accepting wrong merges on your customers' behalf.
How good is matching without a shared identifier?
Noticeably worse, and that is worth saying plainly. With a registry number, VAT number or verified email, matching is close to settled. On name and address alone it is genuinely ambiguous — two different people can share a name and an employer, and a company can have three legitimate entities at one address. Those cases return as review items rather than as confident matches.
Can you enrich records as well as clean them?
Within limits. Validating and standardising data the tenant already holds is straightforward. Appending new personal data from third-party sources is a different activity with consent, lawful-basis and licensing questions attached, and in some jurisdictions it is the part that creates liability. We would scope enrichment separately and only from sources whose terms permit it.
Do you use our tenants' data to improve the model for everyone?
Not unless they have agreed to it. Default behaviour is per-tenant isolation: a tenant's merge decisions improve matching for that tenant only. Cross-tenant learning can be worth a lot, and it needs an explicit basis in your terms rather than a silent default.
What accuracy should we expect?
We would measure it rather than quote it, on a labelled sample of your own data during the prototype. Precision and recall get reported separately because they trade against each other, and the right operating point depends on whether a wrong merge in your product is an inconvenience or a data-loss incident.
Related blueprints
Document Extraction API
Structured fields pulled from PDFs, scans and forms, built as the extraction layer your ERP, CRM or accounting product embeds.
Enterprise Search API
Search across a customer's connected documents and structured records, built as the search layer inside your own product.
Website Change Monitoring API
Alerts when pages your customers are permitted to monitor change — prices, terms, policies, stock or staff listings — as a feature inside your product.
Want this built for your platform?
Tell us what your product does today and what you want another system to be able to ask it. We will come back with an approach, a scope and a price.