5

We periodically (more frequently recently) have very agressive crawling activity coming from EC2 instances that give us a user agent that looks like AF_ID=<xxxxxx>.

I've looked around for common User-Agent formats and I cannot seem to find any references to these User Agents. Do any of you know what these would be, or is it just a generic user-agent used by EC2 instances?

MrWhite
  • 42,784
  • 4
  • 49
  • 90
rbieber
  • 121
  • 3
  • Do you have sofware on your server that accesses the pages internally? Its possible to set user-agents to anything melodramatically. Makes it possible that its a bot or software. – Frank Jun 26 '12 at 07:19

1 Answers1

2

It's just some random scam/spam crawler hosted on EC2. My only advice, add this to your .htaccess :

RewriteCond %{HTTP_USER_AGENT} .*AF_ID=* [NC]
RewriteRule ^.*$ http://english-1335426938.spampoison.com/English [R=303,L] 
#http://tinyurl.com/troll-face-bot :)
Dark
  • 154
  • 4
  • And you're basing that "random spam" assessment on what? – Su' Jul 31 '12 at 18:34
  • 1
    On the fact that it's not an usual user agent, and agressive crawling activity. It's the usual flag to ban... – Dark Aug 06 '12 at 09:24
  • 2
    @Su' respectable bots link to their policy page in a url in the user-agent – ZJR Aug 31 '12 at 04:18