6

I'm seeing some traffic on our classic ASP website from a verified Bing bot that seems to be following a current user of mine.

For instance, if this user adds an item to their shopping cart, 2 to 24 hours later, a Bing bot comes by and tries to add the same items to its cart. I have now confirmed that everything this user does on my site is mirrored by several Bing bots within 24 hours.

I'm wondering if anyone out there has seen something like this, and if there is a known cause for it? Our site is password protected and I've never seen (in 15 years) a spider bot try and index or hit any of these protected URLs before, so I'm very curious as to what's happening here.

Thanks!

EDIT: To be clear: These Bing bots do not actually reach the pages my user visits, as they are all password protected. They try and access the URL, then get redirected to my login page because they have not been authenticated.

Doctor McDoctor
  • 63
  • 6
  • As long as you are making good usage of SSL, Canonical Links, Noindex, and Robots.txt then things like this are nothing really to worry about... however, it sounds like you are having cart abandonment problems which normally occurs when the site allows a page to be accessed when something is added to the cart... so, check all your robots, noindexs and ensure that no checkout process such as STEP 1, 2, 3, shipping, your cart(1) and so forth are noindex or blocked via robots.txt – Simon Hayter Feb 01 '16 at 21:16
  • Thanks for your comment. I'm sorry, I don't know what you mean by cart abandonment problems. This user is finalizing his orders. I only used that as an example. Another example, any searches he makes, the bing bots will also make. They mirror everything. – Doctor McDoctor Feb 01 '16 at 21:34

3 Answers3

3

Maybe that user has a toolbar from Bing installed. This toolbar might submit visited URLs to Bing, and the Bingbot might (try to) crawl these URLs then.

From the searchengineland.com article How The Bing Toolbar May Accidentally Submit Private Pages & Ads:

Microsoft has confirmed that they do discover and index URLs that they find through users surfing the Internet with the Bing Toolbar installed.

[…]

Duane Forrester, a Senior Product Manager at Microsoft, told us:

Yes, as with some other toolbars, the Bing toolbar (when permitted by the user) may record the open (not https) web sites that a user visited to add to our knowledge of the internet in order to improve the search results we provide to users. This in turn (if the site permits) may indeed lead to our crawling links we have discovered. […]

(The article is from 2012. I don’t know if things have changed since then.)

Community
  • 1
unor
  • 21,739
  • 3
  • 46
  • 117
  • Although I can't confirm, (my user says he's not aware of any Bing toolbars), I think this is the best explanation. Oddly, the Bing bots are indeed trying to crawl SSL pages, so maybe they've changed this since 2012? I really do appreciate your help with this. It's been driving me crazy. And in all of my searches, I haven't come across that article. Thank you! – Doctor McDoctor Feb 05 '16 at 00:48
1

I actually experienced this at one point on my site.

What you can do to solve the problem is to make the links the search engine robots are not supposed to access as POST based links.

Here's how you do it perfectly with good compatibility for just about every browser invented:

Create a form with a method post. For example:

  <form action="http://example.com/robots-cant-access-this.asp" method="POST">
  <input type="hidden" name="parametertoscript" value="valueforscriptparameter">
  <input type="hidden" name="secondparametertoscript" value="valueforsecondscriptparameter">
  <input type="hidden" name="nthparametertoscript" value="valuefornthscriptparameter">
  <input type="submit" value="button label to show to public">
  </form>

Then in your asp code, you need to check that the request method is POST, and if it is not, then don't continue with the cart process. In fact, return a 410 so robots don't continue to try to access urls you don't want them to access.

I'm not fluent enough with ASP, but here's pseudo code to get you started:

Get value of request method
If Request method equals post then
     process requested page to client
else
     issue HTTP error 410 GONE
     print error
end if
exit
Mike -- No longer here
  • 13,610
  • 4
  • 28
  • 60
  • 1
    I really appreciate your answer. Forgive me if I misunderstand you, but these pages are all password protected, so the Bing bot isn't actually getting through. The bot tries to access the page (previously visited by my user), then gets redirected to my login page because they are not properly authenticated. Beside this, I'm much more interested in how the heck something like this is happening. I do believe your fix is rather clever though. – Doctor McDoctor Feb 02 '16 at 03:06
  • Maybe it is not the real bing bot. Check the IP address to see. – Mike -- No longer here Feb 02 '16 at 05:26
  • 1
    I've checked about a dozen IPs so far and they appear to be legitimate Bing bots. (Using Bing bot verify tool and nslookup.) Thanks for the suggestion. – Doctor McDoctor Feb 02 '16 at 17:37
1

Do you happen to send some of these urls by mail? In that case a nosniff header could help to prevent mail clients from sniffing the page.

Hotmail would use Bing-bot, Gmail would use the Google crawler and so on.

Here is a nosniff snippet for htccess:

# Add headers to all responses.
<IfModule mod_headers.c>
  # Disable content sniffing, since it's an attack vector.
  Header always set X-Content-Type-Options nosniff
</IfModule>
Neograph734
  • 419
  • 1
  • 3
  • 10
  • This is very interesting. I know that my webserver does not send out any emails with these links, but it's entirely possible that a sales agent or even some of my customers might be. Reading up on the X-Content-Type-Options: nosniff header leads me to think that it's intended for other uses though? Still, I have added this header to IIS. I guess we'll see what happens. Thanks so much for the recommendation. – Doctor McDoctor Feb 03 '16 at 23:57