SaaS, AI & Developer Infrastructure
Website Change Monitoring API
Your customers want to know when a page they care about changes. Doing that reliably is mostly diffing; doing it legitimately is mostly knowing which pages you are allowed to fetch at all.
- Competitive intelligence software
- Compliance and risk platforms
- Procurement software
- Sales intelligence vendors
- Pricing tools
What your customers would see
example interfaceMockupIllustrative mockup of the interface we would build. Figures shown are examples, not results from a live deployment.
The problem this solves
Change monitoring sounds like a weekend project and behaves like a long-running one. A naive implementation fetches a URL, compares the HTML to last time, and fires an alert. In practice it fires on a rotating testimonial, a session identifier in a script tag, a cookie banner variant and an A/B test, and your customers stop reading the alerts within a fortnight. The useful product is the one that distinguishes a changed price from a changed carousel.
The second half of the problem is the part that gets vendors into trouble. Automated fetching is governed by each site's terms of use, its robots directives and, in some jurisdictions, by law about circumventing technical measures. A monitoring feature that accepts any URL a customer pastes and fetches it aggressively is a feature that will eventually generate a cease-and-desist letter addressed to you, not to your customer. Being able to say plainly what the service will and will not fetch is a commercial asset.
Done properly, this is a strong embedded product. Monitoring is inherently recurring: every page checked today must be checked again tomorrow, which makes usage-based pricing natural and churn visible early. And one integration into a procurement or compliance platform can monitor pages on behalf of every customer on it.
Who buys this
Competitive intelligence software, Compliance and risk platforms, Procurement software, Sales intelligence vendors, Pricing tools
What the API returns
What comes back
- Semantic change detection on the content that matters, not raw HTML diffs
- Classified change types: price, availability, terms, personnel, contact details, document replacement
- Before-and-after extracts with the changed region highlighted
- Materiality scoring so a typographical fix does not alert like a price rise
- Structured extraction of monitored elements, defined by CSS or semantic selector
- PDF monitoring for terms, tariffs and published policy documents
- Change history per URL, with a retained snapshot of what was observed
- Per-URL compliance status: robots directives, terms assessment and the crawl policy in force
Endpoints
| Method | Path | Returns |
|---|---|---|
| POST | /v1/monitors | Create a monitor for a URL, with selectors and a check frequency |
| GET | /v1/monitors/{id}/changes | Change history with classified types and extracts |
| POST | /v1/monitors/{id}/check | Trigger an immediate check within policy limits |
| GET | /v1/changes | Recent changes across a tenant's monitors, filterable by type |
| POST | /v1/compliance/check | Pre-flight assessment of whether a URL may be monitored |
| WS | /v1/stream | Push material changes as they are confirmed |
Request and response
illustrativeRequest
{
"tenant_id": "procurehub",
"url": "https://supplier.example.com/pricing",
"frequency": "daily",
"watch": [
{ "name": "list_price", "selector": ".plan-enterprise .price", "type": "currency" },
{ "name": "terms_link", "selector": "a[href*='terms']", "type": "document" }
],
"alert_on": ["price", "terms", "availability"],
"min_materiality": 40
}Response
{
"monitor_id": "mon_7781",
"url": "https://supplier.example.com/pricing",
"compliance": {
"robots_allows_path": true,
"crawl_delay_respected_s": 10,
"terms_reviewed": "2026-09-30",
"policy": "fetch_and_extract_for_internal_use",
"redistribution_permitted": false
},
"last_checked": "2026-10-22T06:12:00Z",
"changes": [
{
"change_id": "chg_30441",
"detected_at": "2026-10-22T06:12:00Z",
"type": "price",
"materiality": 82,
"element": "list_price",
"before": "EUR 1,200 / month",
"after": "EUR 1,380 / month",
"delta_pct": 15.0
},
{
"change_id": "chg_30442",
"detected_at": "2026-10-22T06:12:00Z",
"type": "terms",
"materiality": 61,
"element": "terms_link",
"detail": "Linked PDF replaced; clause 7 notice period changed from 30 to 60 days"
}
],
"suppressed_noise": [ "rotating testimonial", "cookie banner variant" ]
}Where the data comes from
| Pages the customer is entitled to monitor | Their own sites, supplier and partner pages they have a relationship with, and public pages whose terms permit automated reading |
| Robots directives and site policy | Fetched and respected per host, including crawl delay and disallowed paths — checked before a monitor is accepted, not after a complaint |
| Official publication sources | Regulator pages, tariff schedules and company registries, many of which publish feeds or APIs that remove the need to fetch pages at all |
| Vendor-supplied feeds | Where a monitored party offers an API or price feed, using it is faster, cheaper and legally clean — we look for this first |
| Snapshot archive | Retained extracts so a change can be evidenced later; retention scoped to what the source's terms and your customers' needs permit |
| Customer-authorised credentials | Only for pages behind a login the customer legitimately holds, and only where the site's terms allow automated access with those credentials |
The intelligence layer
The difference between a demo and a service is mostly in this section. These are the parts that keep the answer trustworthy once real traffic is hitting it.
- Content extraction before comparison, so diffs run on meaningful text and structured values rather than markup
- Noise suppression learned per host: rotating content, session tokens, timestamps, banner variants and experiment branches
- Change classification into price, availability, terms, personnel and contact categories, each with its own materiality model
- Materiality scoring calibrated against which changes customers actually acted on, measured from their own alert interactions
- Document monitoring that compares extracted PDF text clause by clause rather than by file hash, since a re-save changes the hash and nothing else
- Per-host politeness budgets — conditional requests, caching, back-off on errors and a hard ceiling on request rate regardless of the frequency a customer asks for
- Rendering only where a page genuinely requires it, because headless rendering is both slower and a heavier load on the monitored host
- Monitoring of the monitors: silent failures, blocked responses and layout changes that break a selector are surfaced rather than reported as 'no change'
What it changes for the business
Alerts people still read
Materiality scoring and noise suppression are the difference between a feature customers rely on and one they mute in week two.
Recurring usage revenue
Every monitored page is a recurring check, so revenue scales with the number of things your customers care about.
Defensible compliance story
Being able to tell an enterprise buyer exactly which pages you fetch, how often and on what basis closes deals that a vague answer loses.
Evidence trail
Retained snapshots turn 'their terms changed' from an assertion into something a procurement or compliance team can file.
How it fits into your product
How it embeds
- White-label API and alerting under your brand, with MLPivot invisible to your customers
- Webhooks and WebSocket push into your existing notification and workflow systems
- Per-tenant monitor quotas and scoped keys, so you can price by monitored page or by check volume
- A pre-flight compliance endpoint your UI can call before a customer saves a monitor, so refusals are explained at creation time
- Embeddable change-history and diff components, or raw JSON for your own interface
- Region-pinned deployment and configurable snapshot retention where your customers require it
Included in every build
- Authentication, scoped API keys and per-customer rate limiting
- Input validation with predictable, documented error responses
- Versioned endpoints so your integration never breaks without notice
- Request and response logging for audit and dispute resolution
- Latency, error-rate and usage monitoring with alerting
- OpenAPI specification, sandbox environment and client examples
- Backups, failover design and a documented recovery procedure
- An agreed support arrangement once the service is live
How we would build it
about 12 weeksDiscovery and data review
Weeks 1–2
We agree the exact question the API answers, review the sources you already hold, check what has to be licensed, and write down the response contract your developers will code against.
Architecture and prototype
Weeks 3–4
You get a written architecture plus a working endpoint on real data — narrow, but genuinely callable — so integration can be tested against something concrete rather than a document.
Production build
Weeks 5–10
Pipelines, models, storage, authentication, rate limiting, logging, tests and documentation. Security and observability are part of this phase, not a later one.
Integration and handover
Weeks 11–12
We support your engineers through their integration, run load tests at your expected volume, and agree how the service is monitored and supported once it is live.
Commercial models
There is no fixed price list, because the cost is driven by data complexity, integration count, model requirements, request volume and the support level you need. Scope is agreed in writing before development starts.
Fixed-scope prototype
A defined slice of the service, built and priced as one piece of work. The usual first step when the business case needs proving internally.
Fixed-price build
An agreed scope against a written architecture, delivered for an agreed price. Best when the requirements are already clear.
Build then retainer
We build the service, then operate and improve it under a monthly agreement covering monitoring, updates and model maintenance.
White-label product
The service runs under your brand, your domain and your pricing, with MLPivot invisible to your customers.
Questions buyers ask
Can we monitor any URL our customers paste in?
No, and we would build the product to refuse. Monitors are accepted against a policy: robots directives are respected, hosts that disallow the path are rejected, rate is capped per host regardless of the requested frequency, and sites whose terms prohibit automated access are not fetched. Your customers will occasionally be annoyed by a refusal. That is far cheaper than the alternative.
What is the difference between reading a page and redistributing it?
A large one, legally and commercially. Fetching a public page to detect that a number changed is a different act from storing its content and serving it to your customers as your own data. We design these monitors to return the fact of the change, a minimal extract for evidence, and a link to the source — not a mirror of someone else's site. Where a customer wants redistributable content, that needs a licence from the publisher.
Will this work on sites with bot protection?
Sometimes, and we will not build around it when it does not. If a site uses technical measures to prevent automated access, circumventing them is exactly the behaviour that creates legal exposure and, in several jurisdictions, breaks the law. The honest options are an API or feed from that party, a commercial data licence, or accepting that the page is not monitorable.
How often can pages be checked?
Daily and hourly are routine for most hosts. Sub-minute monitoring of a third party's site is rarely defensible and often indistinguishable from a denial-of-service pattern, so the service caps per-host rate independently of what a customer configures. For genuinely real-time needs, the answer is a feed from the source.
Who is liable if a monitored site objects?
Worth settling before launch rather than after. The service makes the requests, your brand is on the product, and your customer created the monitor. We would help you write the policy, implement it in the API so it is enforced rather than aspirational, and keep per-request logs that show what was fetched and on what basis. Your own counsel should review the allocation of risk in your terms.
Related blueprints
Web Research API
Structured research answers with source references and freshness metadata, built as the retrieval layer behind your AI agent platform.
Data Quality and Deduplication API
Duplicate detection, record matching, normalisation and validation, built as the data hygiene layer inside your CRM, ERP or marketing platform.
Document Extraction API
Structured fields pulled from PDFs, scans and forms, built as the extraction layer your ERP, CRM or accounting product embeds.
Want this built for your platform?
Tell us what your product does today and what you want another system to be able to ask it. We will come back with an approach, a scope and a price.