A client recently was recently looking at the crawl stats of his website’s pages and he was disappointed to see some of the core pages of his website weren’t crawled as frequently as he would have liked. But if you’re faced with low crawl rates, you need to ask yourself how often should Google be crawling your website. Here’s the answer I gave to our client.
Getting crawled frequently on the face of it seems like a sound idea. The more often you’re site is crawled the better chance you have of ranking highly in Google therefore you’ll have great SEO right? Well, it depends.
The number of times a page is crawled is influenced by a number of factors, but it’s mainly around crawlability, freshness and how often Google thinks the page needs to be crawled.
The latter is a key point. It costs Google money to crawl a page. So they crawl them as infrequently as they can get away with. For some pages this will be every couple of days, some every couple of weeks, others every couple of months and some every six months. Pages that aren’t likely to change will be crawled less frequently. These pages include product pages, category pages etc.
It doesn’t really matter how often a page is crawled if that page hasn’t changed, because it’s the cache of the page and how that page features in the index that matters. If Google crawled the page every day and it was always exactly the same it wouldn’t have anything to update. Likewise, if Google crawled the site every day and it was always changing but not in a meaningful way, it might decide to come back to the page less frequently.
How To Increase The Number Of Times A Page Is Crawled By Google
Onsite Work To Increase The Frequency Google Crawls A Webpage
- Have sitemaps in place.
- Make sure the most important pages have prominent links throughout the site
- Update the page’s content more frequently
- Have a page that features some topically relevant content
- Add feeds from the blog to product and category pages
- Regularly update the blog
Offsite Work To Increase The Frequency Google Crawls A Webpage
- Do more social media
- Do more PR
- Increase the velocity of links being built to those pages
My suggestion is often to add a “You may also like” list item section to the bottom of a product or product category page, which pulls through the titles of blogs from relevant blog categories and links back to them. This would be a quick way to see if the frequency of page crawls could be improved.
The above assumes that everything is set up properly and there are no crawlability issues with your site. Just check you’ve got sitemaps that are submitted to Google Search Console and they feature the relevant links. Also, check those links to make sure the pages are followed and indexable.
Hope this helps, let us know how you get on and if you’d like us to take a look at any crawl issues you may be experiencing on your website, please get in touch.