4

Let's say I have a URL that is not linked to from anywhere, and let's say the file portion is very long, so that it's unlikely it could be accidentally entered. Something like sdfnWERvwrw4529vFWGgwgGW42.htm.

I'm not talking about the domain-- those care found through registrars. And assume that my site does not give out directory indexes (which I think has been the default in Apache for some time). And it's not indexed anywhere on the site, for example in a site map. (Since I built this server myself from the ground up, it has very little 3rd party software and I know it's not doing anything like that.)

A similar question was asked here and here but they're not the same questions; they are asking about how the site itself is discovered. I'm asking about a single page.

(If you want to know, it's a page where I keep some information that's not really sensitive, that I use for convenience, but I still don't want public.)

felwithe
  • 143
  • 4

1 Answers1

7

There are many ways that secret URLs may be discovered:

  • Anybody with access to the URL has the option of sharing the URL or linking to it.
  • Web servers are often configured to create listings of all files in a directory by default. Your secret file may end up in such a listing.
  • Web servers are often configured to create sitemaps or RSS feeds of URLs. Your secret URLs could inadvertently be included.
  • Any web log analysis or analytics may leak URLs if reports are publicly available or insecure.
  • Your page may use 3rd party resources such as scripts, CSS, or images that send the URL to those third parties as referrers when the page is viewed in a browser.
  • Your page may link to other sites and the URL is sent to the other sites as a referrer when the link is clicked. Many sites publish a list of referring URLs which allows search engines to find the secret URL.
  • URLs are visible in the browser address bar. Anybody shoulder surfing will be able to see the URL and know its secret.
  • If you email your URL it may be scanned by automated systems such as those at Gmail.
  • Browsers could be configured to send all URLs viewed by users to third parties such as the browser maker, or the author of extensions. However, browsers don't typically come configured like this because major browsers want to respect privacy.
  • HTTP URLs are sent in plain text over the internet, anybody between your client and your server could read the URL. Many ISPs scan all traffic and record the URLs that are visited.
  • Malware on your user's computer could see what URLs are being browsed or find URLs in the browser cache
  • Malware on your server could see what URLs are being served or find URLs in your server logs.

Of those, URLs leaking through referrers is the most common way that secret URLs become public and subsequently get indexed by search engines.

Secret URLs can sometimes work OK, but it is not as good as using proper authentication over HTTPS.

For further reading:

Stephen Ostermiller
  • 98,758
  • 18
  • 137
  • 361
  • Thank you for the detailed reply. I think I have all of those covered (even the referer-- I use the "no-referer" directive in the header, and have checked that both Firefox and Chrome honors it), except malware, which pretty much by definition is something I don't know about. But I think I'm still going to hold off on putting anything sensitive on the page, even the low-sensitivity thing I was going to, simply because there are so many ways. Thanks. – felwithe May 03 '18 at 19:06
  • If you have malware that isn't something that authentication is likely to help much with anyway. – Stephen Ostermiller May 03 '18 at 19:09
  • Someone who is actually permitted to view the page could do what many people I know do and rather than type the URL into the address bar, use a search engine. This would reveal the URL to that search engine which I am almost certain would be an invitation to index the page. – Steve Nov 17 '21 at 09:19