Website API Error Handling: 402s, Unknowns and Retries

Good error handling for a per-domain checks API means treating results as four states, not two: pass, fail, unknown and error. Retry transient errors with backoff, stop and alert on authentication or balance errors, and never convert unknown into pass. This keeps bad data out of your reports and cost under control.

The state model

Most home-made integrations record only success or failure and quietly lose information. Use four states.

State	Meaning	Example
Pass	Check ran, condition met	DMARC record present
Fail	Check ran, condition not met	No SPF record
Unknown	Check ran but could not conclude	DNS lookup did not complete
Error	Check did not run	Timeout, 5xx, 401, 402

The distinction between unknown and error is subtle but useful. Unknown is a result the service chose to report. Error means your call never produced one. BusinessMCP’s website checks API documents two unknown cases: an unreadable robots.txt returns “unknown”, not “allowed”, and a DNS lookup that does not complete is reported as “unknown”, not as a failure of the domain.

Key takeaway: the dangerous bug is collapsing unknown into pass. It produces a clean report for domains nobody actually checked.

HTTP status handling

Status	Action	Retry?
200	Parse and store	n/a
400-class input errors	Log, mark the domain invalid, skip	No
401	Stop the run, alert: key missing or revoked	No
402	Stop the run, alert: balance empty	No, until topped up
429	Back off and slow down, if the service returns it	Yes
5xx	Transient; retry with backoff	Yes, limited
Timeout	Treat as error; retry once or twice	Yes, limited

The 402 case is documented: according to BusinessMCP’s data API overview, once the trial and balance are used up, calls are refused with HTTP 402 insufficient_balance, and you are never charged later. The vendor does not publish numeric rate limits for the API, so whether and how a 429 appears is something to observe, not assume; the table row is a defensive default. Confirm all codes in the API reference.

A retry policy that will not burn money

Per-call billing changes the usual retry calculus. Blind retries on a billable endpoint can multiply cost.

1. Retry only errors where no result was produced (timeouts, connection resets, 5xx).
2. Use exponential backoff with jitter: for example 2, 4, then 8 seconds.
3. Cap at three attempts per domain and check.
4. Do not retry results the service reported, including unknown, in the same run. Schedule them for the next pass instead.
5. Keep an idempotency habit: record that a call was attempted before sending, so a crash does not cause a double call.

Whether a failed attempt is billed depends on the vendor’s terms. BusinessMCP states that a site that cannot be read fails and is refunded rather than graded; for other cases, such as a timeout on your side, check the current terms and test with the free credit before assuming.

Unknowns: what to do with them

An unknown is information that the answer is not available now. Three sensible policies:

• Defer. Re-run unknowns on the next scheduled pass.
• Corroborate. For robots.txt unknowns, fetch the file yourself and look at the status code; a 5xx and a 404 mean different things under RFC 9309.
• Escalate on persistence. Raise an alert only if unknown repeats across two or three runs.

Report unknowns as their own column or colour. Executives reading a dashboard should be able to see that 4 of 100 domains were not conclusively checked.

Special case: client-rendered sites

The AI readiness check reads served HTML only and runs no JavaScript, so script-rendered content can be missed. BusinessMCP says results flag when this is likely. Treat that flag as a caveat on the score, not as an error: store it, show it next to the grade and avoid raising a “fix your site” ticket on a low score that is really a rendering limit.

Logging that helps later

For every call store domain, check, timestamp, HTTP status, latency and the raw body. When behaviour surprises you, such as a sudden cluster of unknowns, the raw bodies and latencies tell you whether the cause is your network, the target domains or the service. It also lets you re-parse history without paying again.

Put it into practice

Start from the Python client and add the state model and retry rules above. Check the budget effects against the pricing scenarios, and wire the 402 alert into your scheduled monitor so it never stops silently.

Status codes, refund rules and limits are set by the vendor and can change, so verify them in current documentation.

Putting the state model in code

A small enumeration keeps the logic honest. Define the four states as constants, and make your parser return one of them for every response. Then write every downstream function, such as reporting, alerting and diffing, to accept all four. If a function can only handle pass and fail, the compiler or a test should force you to decide what it does with unknown.

Function	Pass	Fail	Unknown	Error
Report	Green	Red	Grey, counted separately	Listed as not run
Diff	Compare	Compare	Do not alert, defer	Do not alert, retry
Cost tracking	Billed	Billed	Per vendor terms	Per vendor terms
Escalation	None	Ticket	After two runs	Immediately for 401 and 402

Testing the unhappy paths

Happy-path tests prove little for an integration like this. Write tests that feed your parser recorded bodies for each case: a normal pass, a normal fail, an unknown, a malformed JSON body, an empty body and a body with a field missing. Replay them against your reporting code. Simulate network failures with a mock that raises timeouts, and confirm that the retry cap holds and no more than three attempts occur.

You can capture real examples during the free trial. BusinessMCP gives $5 of credit for 7 days, which per its pricing page is enough for hundreds of calls, more than needed to collect samples of each state, including one from a domain that is deliberately unreadable.

Communicating limits to stakeholders

Error handling is not only technical. When you report results, state coverage: “96 of 100 domains checked conclusively, 4 unknown”. People reading the report will otherwise assume completeness. Add a one-line note about known limits, such as the check reading served HTML without running JavaScript. That honesty protects you when someone asks why a client-rendered site scored low, and it keeps trust in the numbers you do report. For how the audit side of this works at volume, see bulk domain audits.