Bulk Domain Audit: Check Many Websites Without a Crawler

A bulk domain audit runs the same checks over a list of domains and ranks the results by severity. With a per-domain checks API you loop over the list, store every raw response, and sort by the worst findings. At BusinessMCP’s published rates, auditing 100 domains with all four checks costs about $6 per run.

Who needs one

Agencies onboarding clients, holding companies with dozens of brand sites, web teams inheriting a sprawl of microsites, and email teams with many sending domains. BusinessMCP lists “auditing a portfolio” and “pre-flighting a sending domain” among the uses of its website checks API, which is the scenario this article follows.

Step 1: Build a clean domain list

Garbage in, money out. Because each call is billed, clean the list first.

1. Export domains from your registrar, DNS provider or CRM.
2. Lower-case everything and strip schemes, paths and ports.
3. Deduplicate, and decide whether www variants count as one site.
4. Remove parked and retired domains, or tag them to be checked less often.
5. Reject anything over 253 characters, the documented maximum for the domain field.

Step 2: Choose checks per domain type

Not every domain needs every check.

Domain type	Useful checks	Cost per domain
Public marketing site	AI readiness, AI crawler access	$0.04
Site that should be agent-ready	Add WebMCP check	$0.05
Sending domain	Email authentication	$0.01
Full review	All four	$0.06
Parked or redirect-only	Email authentication only	$0.01

Costs use the vendor’s published rates of $0.03 for the readiness grade and $0.01 for each other check. See website data API pricing for scenario tables.

Key takeaway: tag each domain with a profile before you audit. Running every check on every domain is the easiest way to triple your bill for results you will ignore.

Step 3: Run the loop safely

A short Python client is all you need. A few production habits make a large run safer:

• Cap spend. Stop after a fixed number of calls and print the count.
• Throttle gently. The vendor says the API lacks the free tools’ hourly limits but gives no numeric rate limit, so start at a few requests per second and watch for errors.
• Store raw bodies. Keep the unparsed response, timestamp and status for every call so you can re-parse later without paying again.
• Treat 402 as a stop sign. It means the balance is empty; abort and alert rather than looping.
• Resume, don’t restart. Record completed domain and check pairs so an interrupted run continues cheaply.

Step 4: Rank the findings

Severity should drive what gets fixed first. A reasonable default ordering:

Priority	Finding	Why
1	Sending domain with no DMARC record	Deliverability and spoofing exposure
2	SPF permanent error	Mail may fail authentication; see SPF lookup limits
3	robots.txt blocking all AI crawlers unintentionally	Often a leftover staging rule
4	Very low AI readiness grade	Several fixable basics missing
5	Unknown results	Retry first; investigate if persistent

Your own priorities may differ. A publisher that deliberately blocks AI crawlers will treat finding 3 as intended policy, not a defect. Record an “expected” flag per domain so deliberate choices stop appearing in every report.

Step 5: Report in a way people act on

A spreadsheet with one row per domain and one column per check works. Add the previous run’s value, so changes stand out, and a one-line recommended fix. The vendor’s email check grades results with fixes for failures, and the readiness check provides per-category detail, so you can often copy the suggested fix rather than writing your own. Keep unknown as a separate colour from fail.

Keep unknowns honest

Do not collapse unknown into pass. BusinessMCP documents that an unreadable robots.txt returns unknown rather than allowed, and that an incomplete DNS lookup is reported as unknown rather than a domain failure. In a 100-domain run, a handful of unknowns is normal. Retry them in a later pass; if they persist, check by hand. Our note on error handling covers the pattern.

After the first audit

Turn the one-off into a routine. Re-run monthly, diff against the last run, and alert on regressions only. Scheduled website monitoring describes how to do that without alert fatigue. Over a year, monthly audits of 100 domains with all four checks cost about $72 at today’s published rates.

Rates and features can change, so re-verify the vendor’s pricing page before you budget a large programme.

A sample results table

Here is the kind of summary a first audit tends to produce, with invented domains and illustrative values to show the format, not real data.

Domain	Readiness	AI crawlers	Email auth	Notes
client-a.example	high	allowed	pass	Baseline
client-b.example	low	blocked	fail (no DMARC)	Staging rule in production
client-c.example	medium	allowed	unknown	Retry next pass
client-d.example	very low	unknown	pass	Client-rendered, check flagged rendering limit

Notice how few rows are clean. A first audit across a real portfolio almost always finds several quick wins, which is why running one is worth the few dollars it costs.

Choosing which API calls to make

Not every question needs a paid call. Use the free browser tools for a handful of spot checks, such as the AI readiness grader, and use the API for the repeated, scripted runs. This keeps credit for the work that needs automation. The vendor’s free tools are rate limited per visitor, so they suit a few domains, not a hundred.

Governance and ownership

Audits produce findings; findings need owners. Before you run one, decide who fixes what. DNS records belong to whoever manages DNS, robots.txt to the web team and DMARC policy to whoever owns email. Put the owner in your domain list so the report can be routed instead of landing in a shared inbox where it is ignored. Record exceptions: a domain parked on purpose, a site that blocks AI crawlers by policy, a legacy domain scheduled for retirement.

Re-running and tracking progress

The value of an audit compounds when you repeat it. Keep every run, compute the number of domains in each state and track it over time: fewer fails, fewer unknowns, a rising median readiness grade. A simple line of figures in a monthly report keeps the work visible. For the monitoring layer that catches regressions between audits, see scheduled website monitoring, and for the interpretation of any one email result, see the DMARC checker article. The wider BusinessMCP data API catalogue lists other lookups, though none of them replaces the per-domain checks described here.