BEAMSTART Logo

How do I speed up Google's crawl rate for my new website quicker?

I made a new website but Google is crawling my pages very slowly. How do I increase the number of crawled pages? My site is a pretty large site with over 100k pages.

Alfred Lee

Malaysia


3 Responses


Carl Adrian

Partnerships & Content Coordinator at BEAMSTART

Google has their own unique algorithms to determine how much crawl budget to allocate to websites.

For new sites, Google generally crawls the site pretty quickly, provided:

  • They are able to discover links on your site easily (proper linking structure and sitemaps)
  • Your site doesn't take too long to load
  • It has been added to the search console (if it is not added, Google may take a longer time to find your site)

Every site has a limited "crawl budget" by Google. After all, the company can't be spending all its crawling resources only on a single website.

Generally a higher crawl budget is allocated for sites that are:

  • Older: Sites that have been around longer generally get more priority
  • Faster: Sites that load fast allow Google to gather data quicker, and gives Google the perception that it can handle a higher crawl volume.
  • Popular: More backlinks to the site indicate that this site is relevant and people trust it.
  • Have lots of new content: Sites that seem to be updated very frequently generally "deserve" more crawl budget too.

Few tips and tricks to increase Google's crawl rate (seems to work for me):

  • Manually update your uploaded sitemaps in Google Search Console.
  • If you have a large site, do not upload ALL your pages - focus on CONTENT PAGES only. (Pages that have lists / directories / search / similar content with different parameters should be avoided at the start)
  • Try your best to avoid having any downtime for your site - the moment Google detects the site is down for too long, it will PAUSE / REDUCE crawling significantly.
  • Get more quality backlinks for your site (1 - 2 quality a day is fine)

The rule of thumb is this: Google places a higher priority on websites that are more trustworthy, popular, and relevant

Anything you do that falls into those 3 areas above will yield you good results.


Latest Jobs


Wallace Ho

COO @ BEAMSTART

Google sets aside a crawl budget for each site, and this value rises/falls depending on:

  • Number of pages indexed
  • Site speed
  • Quality of content
  • Site popularity (backlinks / trust score, etc.)

We've had scenarios where a few hundred thousand pages were crawled within a matter of weeks, and some other scenarios where very few pages (less than 10%) was crawled even after 2 months.

Fundamentally the focus should be to:

  • Create great content
  • Grow your site popularity organically (backlinks, media coverage, etc.)
  • Avoid duplicate content (if you duplicate content, add additional value on top of it)
  • Ensure all your pages are indexable/crawlable (proper linking, sitemaps)


Shaun Ling

CEO of iPRIMA MEDIA GROUP

Google crawling highly depends on many factors:

  • Site age and popularity
  • Site crawlability (linking, etc.)
  • Available crawl budget
  • Site speed
  • Site uptime/downtime (if site goes down during crawl, Google will reduce the crawl rate)
  • How frequent new content is created

BEAMSTART

BEAMSTART is a global entrepreneurship community, serving as a catalyst for innovation and collaboration. With a mission to empower entrepreneurs, we offer exclusive deals with savings totaling over $1,000,000, curated news, events, and a vast investor database. Through our portal, we aim to foster a supportive ecosystem where like-minded individuals can connect and create opportunities for growth and success.

© Copyright 2024 BEAMSTART. All Rights Reserved.