3

My logs are showing bingbot / msnbot appending %2Findex.php to some of my urls. Any idea why? This makes them invalid and if any users were redirected with it, it would not take them to the correct page. I've poked around a bit and noticed that Findex is a common string in a referrer URL, but it is not valid on my website.

UPDATE

Some Details From the log: These IPs are both listed as owned by Microsoft

157.55.39.21 - - [11/Jan/2016:10:09:02 -0800] "GET /index.php?grade=7&page=chemistry&page2=crystallization%2Findex.php HTTP/1.1" 200 7308 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

207.46.13.150 - - [11/Jan/2016:10:00:22 -0800] "GET /index.php?grade=1&page=Games%2Findex.php HTTP/1.1" 200 6427 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

MrWhite
  • 42,784
  • 4
  • 49
  • 90
mseifert
  • 249
  • 2
  • 10
  • "Findex is a common string in a referrer url" - what do you mean by this? You can get anything in a referer URL. What server are you on? Can you post an example of such a URL please. %2F is an encoded slash and on Apache, an encoded slash in the URL-path part of the URL will typically generate a 404. "redirected with it" - what response is returned for such URLs? – MrWhite Jan 12 '16 at 08:38
  • Have you checked the IP addresses that are requesting the URL's? – Mike -- No longer here Jan 13 '16 at 01:29
  • @w3dk - I have edited the post with url information from the log. I am on the shared host WebHostingHub. The server's response to this page is a 200 OK. I don't have a Findex.php on my server, yet msnbot is requesting this page. I looked at what services use a Findex.php and discovered that many search engines' do. Further, I looked at the referral urls from search engines and discovered that many have Findex.php in them. – mseifert Jan 13 '16 at 05:35
  • @Mike - yes and they are from Microsoft. I have updated and included the ips in the post. – mseifert Jan 13 '16 at 05:36
  • Apart from the trailing %2Findex.php, is this URL otherwise valid? How is this URL generated? Where do the values for the page and page2 URL parameters come from? – MrWhite Jan 13 '16 at 09:43

1 Answers1

2

I don't have a Findex.php on my server, yet msnbot is requesting this page.

It's not Findex.php, it's %2Findex.php, which (when unescaped) is /index.php. It's also part of the query string, rather than the URL-path (which explains why your server doesn't reject it - as it would on Apache by default if it was in the URL-path).

Do you see similar requests from anywhere else?

It's not uncommon to see quirky requests from scripts checking for vulnerabilities, etc. However, for this to come from bingbot my best guess is that bingbot must have found this URL somehow. Either from a URL (or form submission) on your own site, or from a malformed link to your site from elsewhere. Although if the later then you would perhaps expect to see requests for this URL from other sources.

MrWhite
  • 42,784
  • 4
  • 49
  • 90
  • I forgot that the % is the encoding for the following 2 chars after it. That helps understand the first aspect of it. I would also guess that bingbot must have found this URL from another site. I have not seen Google doing this and I would expect so if it was from my own site. I will be on the lookout going forward. Thanks. – mseifert Jan 14 '16 at 07:51
  • 1
    I finally created a script to import my log into a db so I could search it. I imported one month's data or 110,000 records. There were 10 instances of this malformed link and all came from msnbot. Interesting. Don't know what really to conclude - but I won't worry about it. Thanks again. – mseifert Jan 16 '16 at 06:11