Shortcut: If you have been warned specifically about the IPMI vulnerability (watch out for the tag: 750x7ipmi in the subject),
Synopsis
Attackers enter linux machines by means of IPMI or RFB console access, install a rootkit and launch a bitcoin miner. Additional functions may include: distribution of hacking/mining software, attacking other machines, possibly stealing passwords.
About this document
This writeup sums up what ACOnet-CERT has learnt during the investigation of an incident. It turned out that many machines were involved, so we set up this page in the hope it will be useful. It aims at helping sysadmins and security teams to
This writeup refers to one particular campaign, which may or may not correspond to anyones particular situation. Please keep also in mind that the attackers are likely to change their methodology at some point in time, i.e. what's written on this page will become outdated sooner or later. We welcome feedback and updates though (preferably by mail to cert@aco.net).
Distribution on a "need to know" basis is fine with us. It is recommended to simply pass on a link to this page, so that updates can reach the persons involved. Please don't link to this page on a public web site.
Thanks: We would like to thank all those people who have shared their knowledge with us and have provided important hints which helped us a lot in our own work.
Disclaimer: Any information on this page is provided without warranty, may contain errors, misunderstandings and can be misleading, obsolete or otherwise inaccurate. In no way may ACOnet or the University of Vienna be held liable for damages or whatever can cause liability in which jurisdiction ever.
The following IOCs have been observed on machines involved in the "750x7" hack, but may also be present under other, unrelated circumstances.
With some luck, the backup logs may show when files were created or altered, even if they have been removed since. Things to watch for are e.g.:
Any findings in the backup log can also help establishing a timeline.
Note that the attackers hide their tracks by use of a rootkit. It is recommended to investigate the machine by booting from a know to be safe image, otherwise the output of ps, ls etc. may be misleading. For a quick preliminary check, the rootkit can likely be circumvented by first executing a command like export LD_PRELOAD=/lib/libc.so.6 (that's on bash, please make sure to point to the correct libc).
/tmp like do, update, rc_local_found.minerd, or starting with _- (underscore-dash - these should apparently be hidden by the rootkit)./lib/libncom.* or /lib64/libncom.* and /etc/ld.so.preload pointing to this library (beware of the rootkit, see above)./usr/local/bin/ssh.Specific notes regarding a possibly vulnerable IPMI interface:
As far as we could observe, the attackers intrude the system in one of at least two, possibly three ways:
ssh client was dropped in /usr/local/bin. Though we haven't analysed it, chances are that this binary collects the user's passwords as they log into other machines from the compromised one.On the compromised machine, libncom seems to provide access to the attacker. From what we have found about libncom, it hooks some of libc's system calls used by the system's deamons (be it ssh, ftpd, httpd, ...) and immediately opens a (root)shell when the attacker connects. By doing so, the rootkit would bypass any access controls (even tcpwrappers) built into the server, allowing her to get shell access through any service listening to the outside world.
The primary goal of the attackers being the bitcoin mining, minerd is downloaded and installed.
To avoid detection, the libncom rootkit is installed. From this point on detection may be difficult, allthough the rootkit doesn't seem to always work properly.
A number of directories and files were touched during installation of various software. Places to look at are /tmp, /usr/bin and /usr/bin, /opt and directories that are seldom looked at by humans eg. /mnt or device directories.
In one case (so far),
We are to date not aware of:
The usual advice is to disconnect and then reinstall the compromised machine. Considering that a rootkit is used and that the system is manually modified by the attackers, this is probably a good advice.
Be careful to close the IPMI/RFB-vulnerability before getting back online, otherwise the attack can be repeated anytime.
Have the users change their passwords on the local machine and, if applicable, on machines they connected to via ssh. Be aware that the attackers could also have copied the private ssh keys, so these keypairs would need to be replaced as well.
Check for possible lateral movement and intrusions of equally vulnerable servers. Make sure IPMI/RFB can't be exploited on any machine.
Countermeasures likely not to be effective:
None so far.
Deducing the haker's nationality from the network location of the bitcoin master server (China) seems compelling, but may well be completely wrong. During the investigation, we have seen command traffic from several different countries. Any of the machines involved, this also includes the bitcoin master server, may itself have been hacked turning the alleged attacker into the victim. Therefore, we strongly recommend against jumping to conclusions.
Truth is: We don't know who or where the hackers are.
Note that if you use of the tools and information on this page or following any of it's links, you do so at your own risk.
Baseboard Management controllers (BMC) with IPMI:
Others
ACOnet-CERT welcomes feedback, preferably by e-mail to cert@aco.net. If you are aware of other sites covering this topic, please let us know.
Q: Are you saying we have hacked you?
On the contrary! The evidence we found indicates that hackers appear to have your systems on their radar.
Q: I have received a notification from ACOnet-CERT regarding the 750x7 issue. How do you know?
Note: The following applies only if you received a notification by ACOnet-CERT regarding this issue.
We had to analyse a security breach in our constituency (that is: a site was hacked and we looked into it). We found serveral pieces of data with IP addresses. We did our best to interpret these and notify the owners of these addresses. Any background information we can give is on this wiki page.
Q: Can you prove it? Send me logfiles!
Short answer: Sorry, that can't be done.
Long story: As part of handling an incident, we made an effort to
For us, although we're used to handle these things automatically, it was a large number of contacts. Dealing with gazillions of bounces and autoreplies posed a considerable workload. That said, it was absolutely worth the hassle! Our mission is, after all, to make the internet more secure.
We regret though, that we must decline requests for individual log files or "proof needed by our customer to start the investigation". This would require manual research in every single case – we simply need to protect ourselves from getting overwhelmed, as this would impair our mission. We believe that all the information needed to start the investigation can be found on this wiki page and will be happy to improve it if need be.
Q: Can you check if my site is secure?
Testing for vulnerabilities could arguably be interpreted as hacking. We never launch "hacker tools" against sites outside of our constituency.
Q: What does the name 750x7 stand for?
A.: Nothing in particular. We felt it necessary to clearly distinguish this case/pattern from others like, for instance, the bitcoin mining malware for windows that was found a couple of years ago. Since the attack we investigated had no outstanding characteristics, we couldn't figure out an obvious name. Eventually, we went for an "opaque character string".