Is Google Crawling Your Site a Lot? Potential Issues Explained

Intro

Google's Gary Illyes has warned that sudden increases in site crawling might indicate underlying problems, such as hacking or issues with infinite spaces.

Understanding Crawling Spikes

According to Gary Illyes, an Analyst at Google, a sudden surge in crawling activity on your website should be approached with caution. While it might seem like a positive sign, it can often indicate deeper problems.

Illyes advises:

“Don’t get happy prematurely when search engines unexpectedly start to crawl like crazy from your site.”

There are two common issues to be aware of: infinite spaces and website hacks.

Infinite Spaces and Crawling Spikes

Infinite spaces refer to sections of a website, such as calendar modules or endlessly filterable product listings, that generate unlimited potential URLs. This can cause search engine crawlers to become overly active.

Illyes explains:

“If your site generally has pages that search users find helpful, crawlers will get excited about these infinite spaces for a time.”

To mitigate this, he recommends using the robots.txt file to block crawlers from accessing infinite spaces.

Hacked Sites and Crawling Frenzy

Another concerning cause of increased crawling is a security breach. Hackers might inject spam onto a reputable site, initially tricking crawlers into indexing what appears to be new content.

Illyes notes:

“If a no-good-doer somehow managed to get access… they might flood your site with, well, crap… crawlers will get excited about these new pages for a time and happily crawl them.”

Responding to Crawling Spikes

Illyes suggests treating unexpected sharp increases in crawling as a symptom of potential issues until the root cause is identified.

“Treat unexpected sharp increases in crawling as a symptom… until you can prove otherwise. Or, you know, maybe I’m just a hardline pessimist.”

Addressing Hacked Sites

Illyes provides a guide for dealing with hacked sites, emphasizing a systematic recovery process.

Tips from Google's Video on Fixing Hacked Sites

1. Identify the Vulnerability: Use tools like Google’s Webmaster Tools to detect issues.

2. Fix the Vulnerability: Close the security hole by updating software, changing passwords, etc.

3. Clean the Hacked Content: Remove spam, malware, defaced pages, or other injections.

4. Harden Security: Implement additional security measures such as firewalls and frequent updates.

5. Request a Review: Once the site is clean, request Google to review and remove any security warnings.

Google’s video highlights that the review process is faster for malware issues compared to spam issues.

Additional Tips from Google’s John Mueller

• Use the URL Removal Tool: Quickly deindex hacked pages.

• Improve Overall Site Quality: Focus on enhancing site quality beyond just removing hacked content.

• Expect Lingering Impacts: It may take months to regain Google’s trust fully.

Importance of Website Security

Website security is crucial for maintaining user trust and search engine rankings. Sudden spikes in crawling activity could indicate serious issues that need immediate attention.