Carl Adrian57mo agoGoogle has their own unique algorithms to determine how much crawl budget to allocate to websites.
For new sites, Google generally crawls the site pretty quickly, provided:
- They are able to discover links on your site easily (proper linking structure and sitemaps)
- Your site doesn't take too long to load
- It has been added to the search console (if it is not added, Google may take a longer time to find your site)
Every site has a limited "crawl budget" by Google. After all, the company can't be spending all its crawling resources only on a single website.
Generally a higher crawl budget is allocated for sites that are:
- Older: Sites that have been around longer generally get more priority
- Faster: Sites that load fast allow Google to gather data quicker, and gives Google the perception that it can handle a higher crawl volume.
- Popular: More backlinks to the site indicate that this site is relevant and people trust it.
- Have lots of new content: Sites that seem to be updated very frequently generally "deserve" more crawl budget too.
Few tips and tricks to increase Google's crawl rate (seems to work for me):
- Manually update your uploaded sitemaps in Google Search Console.
- If you have a large site, do not upload ALL your pages - focus on CONTENT PAGES only. (Pages that have lists / directories / search / similar content with different parameters should be avoided at the start)
- Try your best to avoid having any downtime for your site - the moment Google detects the site is down for too long, it will PAUSE / REDUCE crawling significantly.
- Get more quality backlinks for your site (1 - 2 quality a day is fine)
The rule of thumb is this: Google places a higher priority on websites that are more trustworthy, popular, and relevant.
Anything you do that falls into those 3 areas above will yield you good results.
