1

I have a project which aggregates public information on companies, startups, founders etc. for a total of >2 million tracked entities worldwide.

Yes, every single page is high quality, manually curated and vetted as much as possible to provide the best value to the customer. Trouble is, google is only crawling my website at the rate of 300 pages/day. That's 109.5K pages per year. At this rate, Google will never index my entire website.

To reiterate, I have over 2 million pages I'd like to get indexed. Here's what I've done so far:

  1. Squeezed out every bit of performance, so that Googlebot has more 'compute quota' to spend on my site
  2. Ensured high quality SEO to signal that yes this is a good website google, come and crawl please.
  3. Ensured high user value, again, I've made sure it is a good website that provides a valuable service and I'm seeing high CTRs/low bounce rates even at positions of 15+.

My website is two months, and only about 15k pages are indexed till now. I know this problem can be solved. If you google any of these: site:crunchbase.com site:owler.com site:zoominfo.com (which are my competitors), they have literally tens of millions of pages indexed. Forget competing, right now I'm just fighting for a seat at the table.

How to increase my indexing rate? I need something far higher than 300 pages/day, as much as possible really. (For reference, 100k pages/day would still take 20 days to scan my whole website)

(P.S: If anyone wants, I can link to the website here)

Stephen Ostermiller
  • 98,758
  • 18
  • 137
  • 361
Aditya Anand
  • 111
  • 3
  • 2
    I find it hard to believe that you are able to reliably gauge the quality of 2.1 million pages, let alone vouch that all of them are high quality. – tripleee Aug 03 '20 at 09:28
  • Although the other question is old, it is still relevant. Additionally, if you want new answers to an old question, you can add a bounty to attract more attention to it. But this site (and network's) policy is not to ask the same question more than once, even if the other question is significantly older. – John Conde Aug 03 '20 at 11:36
  • Edits should not be used to remove the text from a post. I have rolled back your destructive edit. – Stephen Ostermiller Aug 03 '20 at 13:41

1 Answers1

5

You need two things:

  1. Time
  2. External links

Googlebot will eventually increase its crawl rate and crawl your entire site. Getting 2 million pages crawled could take a year or two. Even then Google may choose to index a small percentage of them. Getting 90% of your pages indexed could take five years.

The best way to speed up Google's crawl rate is by getting high quality deep links from other sites. If your content is indeed high quality, that will happen naturally. As people find your site, a small percentage of them will create links to it. You can help things along by asking people to link to your site when appropriate. However, resist the temptation to create links yourself. Creating links by auto-posting and buying links is especially dangerous and likely to get your site kicked out of Google entirely.

Stephen Ostermiller
  • 98,758
  • 18
  • 137
  • 361