What is a Crawler?
A crawler is an internet program designed to browse the internet systematically. Crawlers are most commonly used as a means for search engines to discover and process pages for indexing and showing them in the search results.
In addition to crawlers that process HTML, some special crawlers are also used for indexing images and videos.
In the real world, the main web crawlers to know are the ones used by the world’s top search engines: Googlebot, Bingbot, Yandex Bot, and Baidu Spider.
Good vs. Bad Crawlers
Think of a good crawler as a bot that can help your site, primarily by adding your content to a search index or by helping you audit your website. Other hallmarks of a good crawler are that it identifies itself, follows your directives, and adjusts its crawling rate to keep from overloading your server.
A bad crawler is a bot that adds no value to a website owner and may have malicious intent. Bad crawlers may fail to identify themselves, circumvent your directives, create unnecessary loads on servers, and even steal content and data.
Types of Crawlers
There are two main types of crawlers:
- Constant-crawling bots: These perform a crawl 24/7 to discover new pages and recrawl older ones (e.g., Googlebot).
- On-demand bots: These will crawl a limited number of pages and perform a crawl only when requested (e.g., Ranktracker Site Audit bot).
Why is Website Crawling Important?
The primary purpose of a search engine crawler is to find out what’s on your website and add this information to the search index. If your site isn’t crawled, then your content will not appear in the search results.
Website crawling isn’t just a one-time event – it’s an ongoing practice for active websites. Bots will regularly recrawl websites to find and add new pages to the search index while also updating their information about existing pages.
While most crawlers are associated with search engines, there are other types of crawlers out there. For example, the Ranktracker Site Audit bot can help you see what’s wrong with your website in terms of SEO.
How Do Crawlers Work?
In a nutshell, a web crawler like Googlebot will discover URLs on your website through sitemaps, links, and manual submissions via Google Search Console. Then it will follow the “allowed” links on those pages.
It does this while respecting the robots.txt rules, as well as any “nofollow” attributes on links and on individual pages.
Some websites – those with more than 1 million pages that are updated regularly or those with 10 thousand pages of content that changes daily – may have a limited “crawl budget.” This refers to the amount of time and resources the bot can devote to a website in a single session.
Crawl Priorities
Because of the limited capacity of crawl budgets, crawlers operate by a set of crawl priorities. Googlebot, for example, considers the following:
- PageRank of the URL
- How often the page(s) are updated
- Whether or not the page is new
In this way, the crawler can focus on crawling the most important pages on your site first.
Mobile vs. Desktop Crawler Versions
Googlebot has two main versions: Googlebot Desktop and Googlebot Smartphone. These days, Google uses mobile-first indexing, which means that its smartphone agent is the primary Googlebot used for crawling and indexing pages.
Different versions of a website can be presented to these different types of crawlers. Technically, the bot identifies itself to a web server using the HTTP request header User-Agent, along with a unique identifier.
Best Practices for a Crawl-Friendly Website
To ensure that your website is ready for crawling, there are several steps we recommend. Follow them in order to give your key pages the best chance of indexing and ranking.
1. Check Your Robots.txt File
The robots.txt file is the file on your website that communicates with those bots, using a series of crawler directives. Ensure that it’s not disallowing good bots from any pages or sections that you want to be indexed. Use tools like Google’s robots.txt tester to check for errors.
2. Submit Sitemaps
Submitting your sitemap is a critical step. A sitemap lists all the pages of your website you want to be indexed. In Google Search Console, you can submit your sitemap under Index > Sitemaps. This process is similar for other search engines, like Bing Webmaster Tools.
3. Use Crawler Directives Wisely
The robots.txt file uses directives to tell crawlers which pages are allowed or disallowed to be crawled. It’s important to allow important pages in your site’s navigation to be crawled. Any page-level directives won’t be seen if the content is disallowed to be crawled in your robots.txt file.
4. Provide Internal Links Between Pages
Internal linking helps search engines understand what each page is about and helps the crawler discover pages in the first place. Internal links also help you shape how PageRank flows throughout your site.
5. Reduce 4xx’s and Unnecessary Redirects
4xx errors signal to a crawler that the content at that URL does not exist. Use tools like Ranktracker Site Audit to fix these pages or set up a redirect to a live page. Also, eliminate unnecessary redirects and redirect chains to ensure smooth crawling.
6. Use Ranktracker Site Audit to Find Crawlability and Indexability Issues
The Ranktracker Site Audit tool can help by checking all noindexed pages and nofollowed links on your site. It can uncover any broken pages or excessive redirects, including redirect chains or loops, and point out any orphan pages.
FAQs
Is Crawling and Indexing the Same Thing?
No. Crawling refers to the process of discovering publicly accessible web pages and other content. Indexing refers to search engines analyzing these pages and storing them in their search index.
What are the Most Active Crawlers?
The most popular search engine crawlers include Googlebot, Bingbot, Yandex Bot, and Baidu Spider. For more details, check out this Imperva Bot Traffic study. Interestingly, AhrefsBot, which powers the entire link database, was found to be the second most active crawler after Googlebot.
Do Crawlers Hurt My Website?
While most crawlers do not pose harm to your website, there are bad crawlers that can. Harmful crawlers may hog your bandwidth, causing pages to slow down, and may attempt to steal data or scrape content from your site.
