Shortcut: If you have been warned specifically about the IPMI vulnerability (watch out for the tag: 750x7ipmi in the subject),
- your machine(s) has been found by a hacker using metasploit;
- we found his scan results on a hacked machine that we have analysed;
- you can jump directly to specific notes regarding IPMI.
Bitcoin Mining Hack "750x7" - Technical Details for Detection & Recovery
Synopsis
Attackers enter linux machines by means of IPMI or RFB console access, install a rootkit and launch a bitcoin miner. Additional functions may include: distribution of hacking/mining software, attacking other machines, possibly stealing passwords.
...
Disclaimer: Any information on this page is provided without warranty, may contain errors, misunderstandings and can be misleading, obsolete or otherwise inaccurate. In no way may ACOnet or the University of Vienna be held liable for damages or whatever can cause liability in which jurisdiction ever.
Indicators of Compromise
The following IOCs have been observed on machines involved in the "750x7" hack, but may also be present under other, unrelated circumstances.
Network
- Traffic to 119.78.232.8. This is the bitcoin master server at the time of this writing.
- Inbound ssh connections that can't be attributed to legitimate users.
- Possibly outgoing scanning activity, in particular for port 623 and 5900.
- Possibly outgoing scans for http/htts (port 80/443).
On the backup
With some luck, the backup logs may show when files were created or altered, even if they have been removed since. Things to watch for are e.g.:
...
Any findings in the backup log can also help establishing a timeline.
On the machine
Note that the attackers hide their tracks by use of a rootkit. It is recommended to investigate the machine by booting from a know to be safe image, otherwise the output of ps, ls etc. may be misleading. For a quick preliminary check, the rootkit can likely be circumvented by first executing a command like export LD_PRELOAD=/lib/libc.so.6 (that's on bash, please make sure to point to the correct libc).
- Unexplained reboots.
- rkhunter reporting the libncom rootkit.
- we've heard that /lib/libproc-3.2.8.so has been replaced on some machines, not detected by rkhunter (seen on ubuntu so far, unclear whether other distributions are affected)
- Presence of files in
/tmplikedo,update,rc_local_found. - Presence of files in /usr/bin like
minerd, or starting with_-(underscore-dash - these should apparently be hidden by the rootkit). - Presence of
/lib/libncom.*or/lib64/libncom.*and/etc/ld.so.preloadpointing to this library (beware of the rootkit, see above). - CPU usage that can't be accounted for. The miner process might only be visible when evading the rootkit (see above).
- CPU usage by processes like metasploit, nmap, minerd.
- Presence of
/usr/local/bin/ssh. - Some tools may have been upgraded or installed (gnu auto*, Python, JRE), metasploit, nmap.
...
- A vulnerable machine doesn't necessarily get hacked. To
- The vulnerability, if present, should nevertheless be fixed ASAP.
- If the interface is exposed to untrusted networks (i.e. the Internet), the attacker we observed would try to access the system
- by guessing just a username. This is possible if the so called "cipher 0" is enabled, which implies that no password is required.
- to crack the password of an IPMI user after retrieving the Hash. This is possible with weak or moderately complex passwords.
- However, a cracked password (3b above) may not be exploitable when the user is disabled, the attack would then fail. ACOnet-CERT has no data whether this is the case and can't detect this either, as this would require us to try to attack ourselves.
- See also the vendor's documentation and make sure the firmware is up to date - see the links below.
Intrusion
As far as we could observe, the attackers intrude the system in one of at least two, possibly three ways:
...
On the compromised machine, libncom seems to provide access to the attacker. From what we have found about libncom, it hooks some of libc's system calls used by the system's deamons (be it ssh, ftpd, httpd, ...) and immediately opens a (root)shell when the attacker connects. By doing so, the rootkit would bypass any access controls (even tcpwrappers) built into the server, allowing her to get shell access through any service listening to the outside world.
Alterations of the System
The primary goal of the attackers being the bitcoin mining, minerd is downloaded and installed.
...
- any user accounts being created.
- manipulation of the logfiles.
Remediation
The usual advice is to disconnect and then reinstall the compromised machine. Considering that a rootkit is used and that the system is manually modified by the attackers, this is probably a good advice.
...
- changing the root password
- using tcpwrappers (hosts.allow / hosts.deny)
- firewalling ssh while leaving any other services accessible
Attribution
None so far.
Deducing the haker's nationality from the network location of the bitcoin master server (China) seems compelling, but may well be completely wrong. During the investigation, we have seen command traffic from several different countries. Any of the machines involved, this also includes the bitcoin master server, may itself have been hacked turning the alleged attacker into the victim. Therefore, we strongly recommend against jumping to conclusions.
Truth is: We don't know who or where the hackers are.
| Anker | ||||
|---|---|---|---|---|
|
Note that if you use of the tools and information on this page or following any of it's links, you do so at your own risk.
...
- Dell iDRAC: Best Practices for Security for iDRAC, IPMI, SNMP
- Dell iDRAC: Vulnerability Note VU#843044 (Dec. 2014)
- Cisco: IPMI Security Vulnerabilities
- Dan Farmer about IPMI security: http://fish2.com/ipmi/
- Metasploit: A Penetration Tester's Guide to IPMI and BMCs
- Article about ipmi vulnerabilities: Many servers expose insecure out-of-band management interfaces to the Internet
Others
- rkhunter (Rootkit detection tool): http://rkhunter.sourceforge.net
Contact and Feedback
ACOnet-CERT welcomes feedback, preferably by e-mail to cert@aco.net. If you are aware of other sites covering this topic, please let us know.
FAQ
Q: Are you saying we have hacked you?
...