Bitcoin Mining Hack "750x7" - Technical Details for Detection & Recovery

Synopsis

Attackers enter linux machines by means of IPMI or RFB console access, install a rootkit and launch a bitcoin miner. Additional functions may include: distribution of Hacking/Mining software, attacking other machines, possibly stealing passwords.

About

This writeup sums up what ACOnet-CERT has learnt during the investigation of an incident. It turned out that many machines were involved, so we set up this page in the hope it will be useful. It aims at helping sysadmins and security teams to

• verify if a machine has been compromised
• assess the impact of the compromise
• recover from the incident.

This writeup refers to one particular campaign, which may or may not correspond to anyones particular situation. Please keep also in mind that the attackers are likely to change their methodology at some point in time, i.e. what's written on this page will become outdated sooner or later. We welcome feedback and updates though (preferably by mail to cert@aco.net).

Distribution on a "need to know" basis is fine with us. It is recommended to simply pass on a link to this page, so that updates can reach the persons involved. Please don't link to this page on a public web site. 

Thanks: We would like to thank all those people who have shared their knowledge with us and have provided important hints which helped us a lot in our own work. 

Disclaimer: Any information on this page is provided without warranty, may contain errors, misunderstandings and can be misleading, obsolete or otherwise inaccurate. In no way may ACOnet or the University of Vienna be held liable for damages or whatever can cause liability in which jurisdiction ever.

Indicators of Compromise

The following IOCs have been observed on machines involved in the "750x7" hack, but may also be present under other, unrelated circumstances.

Network

• Traffic to 119.78.232.8. This is the bitcoin master server at the time of this writing.
• Inbound ssh connections that can't be attributed to legitimate users.
• Possibly outgoing scanning activity, in particular for port 623 and 5900.
• Possibly outgoing scans for http/htts (port 80/443).

On the machine

Note that the attackers hide their tracks by use of a rootkit. It is recommended to investigate the machine by booting from a know to be safe image, otherwise the output of ps, ls etc. may be misleading. For a quick preliminary check, the rootkit can likely be circumvented by first executing a command like export LD_PRELOAD=/lib/libc.so.6 (that's on bash, please make sure to point to the correct libc).

• Unexplained reboots.
• rkhunter reporting the libncom rootkit.
• Presence of files in /tmp like do, update, rc_local_found.
• Presence of files in /usr/bin like minerd, or starting with _- (underscore-dash - these should apparently be hidden by the rootkit).
• Presence of /lib/libncom.* or /lib64/libncom.* and /etc/ld.so.preload pointing to this library (beware of the rootkit, see above).
• CPU usage that can't be accounted for. The miner process might only be visible when evading the rootkit (see above).
• Presence of /usr/local/bin/ssh.
• Some tools may have been upgraded or installed (gnu auto*, Python), metasploit, nmap.

On the backup

With some luck, the backup logs may show when files were created or altered, even if they have been removed since. Things to watch for are e.g.:

• minerd, 
• ssh
• ld.so.preload
• _-*
• automake, autoconf, ...

Any findings in the backup log can also help establishing a timeline.

Intrusion

As far as we could observe, the attackers intrude the system in one of at least two, possibly three ways:

• IPMI (remote management as in e.g. ILO, DRAC etc, port 623 tcp/udp). This protocol has severe weaknesses which, if not properly mitigated, would allow an attacker to reboot the machine into a live linux image. He would then mount the root disk, alter it (thereby circumventing any of the target system's security controls), and then reboot the compromised system.
• RFB (remote framebuffer as in e.g. VNC). We currently have no information on how exactly the attackers exploit this protocol, but they are actively scanning for it.
• Possibly by using compromised accounts. We observed that an ssh client was dropped in /usr/local/bin. Though we haven't analysed it, chances are that this binary collects the user's passwords as they log into other machines from the compromised one.

On the compromised machine, libncom seems to provide access to the attacker. From what we have found about libncom, it hooks some of libc's system calls used by the system's deamons (be it ssh, ftpd, httpd, ...) and immediately opens a (root)shell when the attacker connects. By doing so, the rootkit would bypass any access controls (even tcpwrappers) built into the server, allowing her to get shell access through any service listening to the outside world.

Alterations of the System

The primary goal of the attackers being the bitcoin mining, minerd is downloaded and installed.

To avoid detection, the libncom rootkit is installed. From this point on detection may be difficult, allthough the rootkit doesn't seem toalways work properly.

A number of directories and files were touched during installation of various software. Places to look at are /tmp, /usr/bin and /usr/bin, 

We are to date not aware of:

• any user accounts being created
• manipulation of the logfiles

Remediation

FAQ

Q: What does the name 750x7 stand for?

A.: Nothing in particular. We felt it necessary to clearly distinguish this case/pattern from others like, for instance, the bitcoin mining malware for windoes that was found a couple of years ago. Since the attack we investigated had no outstanding characteristics, we couldn't figure out an obvious name. Eventually, we went for an "opaque character string".