What Does Crawlability Mean in SEO?
In Search Engine Optimisation (SEO), crawlability refers to how well search engine crawlers like Googlebot can access, read, and understand your website’s content. Crawlability issues can negatively affect a website’s search engine rankings.
What Is the Difference Between Crawling and Indexing in SEO?
Crawling and indexing are both essential processes in SEO, but they serve different purposes:
- Crawling: This is the process where search engine bots (often called “crawlers” or “search engine spiders”) scan the web to discover new or updated content. They follow internal links on a website, analyse the relevant content, and report back to the search engine.
- Indexing: After a page is crawled, it may be added to the search engine’s index, which is a database of all the web pages that the search engine knows about. Indexing is the process of storing and organising the additional content found during crawling so that it can be retrieved and displayed in search results.
In summary, crawling is about discovery, while indexing is about storage and retrieval.
Importance of Crawlability
If crawlers can’t crawl your website or find your different pages, your site and pages won’t be ranked in search results, leading to a loss of organic traffic, conversions, and revenue.
A page cannot be fully indexed without being crawled. While it’s rare, Google can sometimes index a URL without crawling it, relying on the URL text and the anchor text of its backlinks. However, in such cases, the page title and description won’t appear in the search results.
Crawlability is crucial, not just for Google. Other specialised crawlers also need to access website pages for different purposes. For example, the AhrefsSiteAudit bot crawls pages to assess SEO health and identify any technical SEO issues.
What Factors Influence a Website’s Crawlability?
1. Page Discoverability
For a page to be crawled, it must first be discovered by the crawler. Pages that aren’t included in the sitemap or lack internal links (referred to as orphan pages) can’t be found by the crawler and, therefore, can’t be crawled or indexed. To ensure a page is indexed, it should be both in the sitemap and have internal links pointing to it.
2. Nofollow Links
Googlebot does not follow links that have the “rel=nofollow” attribute. If a page is only linked through a nofollow link, it effectively has no crawling links, making it invisible to the crawler.
3. Robots.txt File
The robots.txt file instructs web crawlers on which parts of your site they are allowed to access. If a page is disallowed in the robots.txt file, it won’t be crawlable.
4. Access Restrictions
Specific restrictions can prevent crawlers from accessing certain pages. These can include login requirements, user-agent blacklisting, or IP address blacklisting, all of which can hinder a page’s crawlability.
How Often Does Google Crawl a Website?
Typically, Google crawls websites every three days to 4 weeks. However, this depends on several factors, including:
- Content Updates: Google crawls sites more frequently that are regularly updated.
- Popular Pages: Google also tends to crawl more popular pages regularly.
- Site Size: The size of your site can also influence how often Google crawls your pages.
