1

Hello I have a website where anyone can post anything. I have a file called:

article.php?id=123

Lets say I have 400 articles. How does google find all of them. When I get a sitemap generator to generate a sitemap it dosnt see them and google isnt indexing them. How do I get Google to index these pages?

Jelly
  • 11
  • 2
  • How long have you waited? It can take weeks if not months for Google to index your site. As it is, there is not enough actionable detail for us to answer this question. Have you tried using Google Search Console - Fetch as Google to see if Google can see you content correctly? If not, that is your first step. As well, make sure your content is linked to. Sitemaps generally do nothing. They are primarily used for auditing whether Google can effectively crawl your site. If not, then the sitemap may be used to spot check your site. Your site should be crawlable. Do not rely on a sitemap. – closetnoc Dec 02 '16 at 17:18
  • its indexed all my non php pages. – Jelly Dec 02 '16 at 17:33
  • and yes i used fetch with google and it fetched them but I want it to do it automatically like it does with regualr php pages – Jelly Dec 02 '16 at 17:34
  • Sitemaps are not needed to get content indexed, nor do they actually help do so. See The Sitemap Paradox. – Stephen Ostermiller Dec 02 '16 at 18:57
  • It takes weeks if not moths. Google generally does start right away, but will go according to it's own priorities. Search engines are not real-time nor can they be. It is impossible. The web is just too huge. – closetnoc Dec 02 '16 at 19:00
  • Using Fetch as Google will not generally be fast. The regular googlebot will fetch your page 1-2 days later to make it official. However, the reason why I suggested using Fetch as Google is to know if there are any errors. Are there any errors? – closetnoc Dec 02 '16 at 19:02
  • @StephenOstermiller It depends if the content is crawlable (ie. well linked internally). – MrWhite Dec 02 '16 at 19:04

1 Answers1

2

When I get a sitemap generator to generate a sitemap it dosnt see them

Since you are already allowing users to generate content you would seem to be in the best position to generate the sitemap; not an external tool. When the sitemap changes, you can optionally ping Google.

If an external sitemap generator is unable to find these pages then it's because the new pages aren't being linked to and your site cannot be crawled successfully.

In addition to the XML sitemap, you should probably be generating some kind of HTML index/sitemap that enables the new pages to be crawled easily.

do it automatically like it does with regualr php pages

These are regular PHP pages. article.php?id=123 might sometimes be referred to as a dynamic URL, but really there's no difference.

anyone can post anything

Sounds like a moderation nightmare?! Is this content worthy of being indexed??

MrWhite
  • 42,784
  • 4
  • 49
  • 90
  • 1
    If a CMS is used, it is far better to have the CMS generate the sitemap. External tools often fail. I do not like to use ping, though it is a perfectly valid option. Google, once they know about a sitemap, will check for updates when it cares. Otherwise, G prefers to crawl a site. It is likely a linking issue, which you mention, and potentially an issue with whether the pages work as designed. That is my biggest worry. Hence the Fetch as Google recommendation. It is possible that Google may have already tried to fetch a page and it failed for some reason. Worth the OP checking out. – closetnoc Dec 02 '16 at 19:19