Get indexed · 4 min read
Make sure your pages can be indexed
Crawlable, non-blocked, non-noindex pages get indexed. Here's how to avoid the common blockers.
Don't block the crawler
Our crawler (CGMSearchBot) respects robots.txt. If your robots.txt disallows a path, those pages won't be crawled or indexed. Make sure the pages you want found aren't blocked.
Watch for noindex
A page with a noindex robots meta tag is deliberately excluded from search. It's easy to leave noindex on by accident after a site launch or migration — check that your important pages don't carry it.
- •Remove Disallow rules for pages you want indexed.
- •Remove noindex meta tags from pages that should rank.
- •Return a normal 200 response — error pages can't be indexed.
Tip: Our crawler reads the HTML your server returns and doesn't run JavaScript. Make sure your key content and links are present in the server-rendered HTML.
More in Get indexed
How CGM Search ranking works
A plain-English look at how CGM Search decides which pages appear in the web results on cgmimm.com — and what you control.
Verify your site
Verification proves you own the domain. Until you verify, none of your pages are eligible for CGM Search.
Submit a sitemap
A sitemap hands us the full list of pages you want indexed — the fastest way to get crawled completely.