Plans & Pricing

Plans & Pricing

Access Plans & Pricing Now

What services are you interested in? (Select all that apply) *

What Can We Help With

What is Robots.txt

Definition

Robots.txt is a file that tells search engines which pages on your site to crawl and which pages not to to crawl.
Flat Elements 05

You don’t want your server to be overwhelmed by Google’s crawler or to waste crawl budget crawling unimportant or similar pages on your site.

Resources