Discovery
01How the crawl works
- Fetch the start URL and extract same-host internal links
- Read robots.txt for Sitemap: lines when present
- Parse sitemap / sitemap index XML (public text fetch, SSRF-hardened)
- Prioritize taxonomy-looking paths, then BFS more internal URLs
- Hard cap: about 36 HTML pages and 200 sitemap URLs per run
- Optional: competitor homepage + sitemap themes (not rank tracking)
Taxonomy
02What we classify
- Service / product pages
- Use-case and industry pages
- Comparison and alternatives pages
- FAQ / help hubs
- Blog / guides / resources
- About / company trust
- Case studies / customers / proof
- Pricing / plans
- Contact / demo / trial conversion paths
Each area is marked dedicated (matching page fetched),weak (path or language only), or missing.
Scoring
03How the architecture score is calculated
- Weighted taxonomy points: dedicated = full, weak = partial, missing = zero
- Thin crawls are capped (single-page HTML cannot earn an exceptional grade)
- Evidence coverage reflects pages fetched vs budget and sitemap breadth
- Recommendations suggest concrete paths/titles for missing or weak areas
Limits
04What we never claim
- Keyword volume, difficulty, or ranking gaps
- That a missing path proves the page does not exist anywhere (crawl budget is capped)
- Full site inventory equal to a site mirror or enterprise crawler
- That competitor “themes” mean they outrank you