5

Possible Duplicate:
Is it possible for web crawlers to see static pages without following a link to them?

I have some URLs (some pdfs and static HTML files) in my website that I want only few people to know. These URLs dont have any pointing links from my website or any other source.

So my question is this: Can google crawl an URL (say some static HTML file) that has no incoming link?

Community
  • 1
Thale
  • 215
  • 2
  • 7

3 Answers3

6

Usually no but you have to be really sure that the URL is not present anywhere on the web, in your sitemap if you are publishing it. You also have to be careful on web server access statistics if you make them public.

In addition You can always use the robot.txt file to tell Google not to crawl the URLs.

But this is just security by obfuscation, if you really want to protect them use a proper way (authentication/authorization)

Matteo
  • 556
  • 5
  • 12
  • 1
    +1 for adding the note about authentication/authorisation. The only real way to guarantee search engines won't be able to reach these pages is to have them in a secure area in such a way that even requesting the path directly triggers the authentication. – Zhaph - Ben Duguid Feb 24 '12 at 14:39
6

Yes, Google will find it somehow!

They monitor people's browser/search history via Google Accounts/Toolbars/Social Networks and the like - then use that data to augment and prioritize their crawler.

So if a user visits your page while logged into a google account with it's search history tracking enabled Google may find out about your page. You also can't control what users post to social media sites and the like.

You can prevent it's inclusion in google's index though, robots.txt, a simple text file which sits in the www root directory of your server, will stop the GoogleBot in it's tracks.

House your non-google pages in a single directory and exclude like this:-

User-agent: *
Disallow: /your-directory-name/

As @Matteo and @Zaph note this isn't real protection and won't stop determined users finding your content. I use .htpasswd to block areas on my sites in conjunction with Coffee Cup Website Access Manager which outputs hashed htpasswd files and uploads them to your site, and provides multi user management.

toomanyairmiles
  • 12,555
  • 2
  • 26
  • 49
  • 2
    Note that this doesn't stop people from snooping at your robots.txt file and having a look themselves at directories and files that you list there. Your best bet is probably to password protect the directory as well as mention it in robots.txt. – Andy Smith Feb 24 '12 at 15:00
  • I don't think this answer makes sense. How can "monitoring search history" help Google index pages if the files aren't being returned in search results? – DisgruntledGoat Feb 24 '12 at 23:51
  • @DisgruntledGoat in essence Google uses input from the search history in your Google account/toolbar/site search/analytics to 'train' its bot. Put another way, in some respects, you direct the crawler. – toomanyairmiles Feb 25 '12 at 00:16
  • @toomany but I don't see how that leads to discovery of URLs. Let's assume Google doesn't know about a URL. Then you search Google for something that would match that URL. Of course that URL is not returned as Google doesn't know about it. So how can it get the URL from your search history when it's not there? And Google Analytics (and Adsense for that matter) are completely separate, they don't share data. – DisgruntledGoat Feb 25 '12 at 00:40
  • @DisgruntledGoat User visits previously unknown URL (search history etc enabled) > Google discovers URL > Google indexes URL – toomanyairmiles Feb 25 '12 at 01:01
  • @toomany Unless you mean the user is typing the URL in their address bar while on a Google search page (which is quite rare, especially in this question's situation), then there is no way for Google to "track" that page or know that you visited it. – DisgruntledGoat Feb 25 '12 at 01:13
  • Actually I'm not sure it's possible to track URLs the user types in their address bar, in which case there is simply no way for Google to pick up that URL. – DisgruntledGoat Feb 25 '12 at 01:18
  • Save for logging the visit via a user installed toolbar, or the end site's analytics, or your accounts search history (i.e. web history which google logs unless you disable it). You are thinking it only logs searches which isn't the case. – toomanyairmiles Feb 25 '12 at 01:19
  • @DisgruntledGoat please do some research on the subject or read the answer before you down vote an answer no one else seems to have a problem with. – toomanyairmiles Feb 25 '12 at 01:21
  • let us continue this discussion in chat – DisgruntledGoat Feb 25 '12 at 01:25
  • @DisgruntledGoat http://support.google.com/accounts/bin/answer.py?hl=en&answer=65396&topic=14150&ctx=topic – toomanyairmiles Feb 25 '12 at 01:27
0

As long as there is no incoming link or anything that can point google towards the file (sitemap, open directory structure, etc..) then I believe that it shouldn't be indexed. alternatively you could put the files in a folder and block it through the robots file.

Vince P
  • 640
  • 4
  • 15