[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE : [cobalt-security] Forensics on a hacked server



Hello,

This is quite impressive indeed. One bottom-line question I wonder about is
this: is there any way to be secure from these kinds of attacks ? Our RAQ
has been hacked a couple of weeks ago, and I don't sleep well yet after this
shock. After reading this E-mail I bet I won't sleep very well for another
couple of weeks.

Alain

---

Alain Fontaine		
IT Consultant

VAlain S.A.	
27 rue Knupp
L-9535 Wiltz		
Luxembourg

www.valain.lu


-----Message d'origine-----
De : cobalt-security-admin@xxxxxxxxxxxxxxx
[mailto:cobalt-security-admin@xxxxxxxxxxxxxxx] De la part de Michael Stauber
Envoyé : mercredi 21 mai 2003 18:15
À : cobalt-security@xxxxxxxxxxxxxxx
Objet : [cobalt-security] Forensics on a hacked server


 Hi all,

I just finished forensics on a customers RaQ4 which had been hacked. The box

was used to send SPAM and although some formmail scripts were present none
of 
'em was responsible for the SPAM. Kinda unusual, right?

Patchlevel: All patches installed, except the new C37 Kernel.

Third party software: MySQL-3.23.37-1, php-4.1.2-3, OpenWebmail, Webalizer
and 
OpenSSH-3.4p1.

According to the timestamps in /var/lib/cobalt the patches were mostly 
installed in a timely manner.

After poking around for a while I found out that the box had been hacked and

'lo and behold: It is the most beautiful hack I have seen so far.

chkrootkit-0.40 reports *nothing* at all. The report it generates looks just

perfectly fine. No modified binaries, no LKMs, no trojans. Show that output 
to someone who knows chkrootkit and he'd say: "So what? Looks good!"

chkrootkit-0.39 produces the same neat report, except that it bugs out on
the 
LKM test:

###
### Output of: ./chkproc -v -v
###
ERROR: Thread display not implemented.
********* simple selection *********  ********* selection by list *********
-A all processes                      -C by command name
-N negate selection                   -G by real group ID (supports names)
-a all w/ tty except session leaders  -U by real user ID (supports names)
-d all except session leaders         -g by session leader OR by group name
-e all processes                      -p by process ID
T  all processes on this terminal     -s processes in the sessions given
a  all w/ tty, including other users  -t by tty
g  all, even group leaders!           -u by effective user ID (supports
names)
r  only running processes             U  processes for specified users
x  processes w/o controlling ttys     t  by tty
*********** output format **********  *********** long options ***********
-o,o user-defined  -f full            --Group --User --pid --cols
-j,j job control   s  signal          --group --user --sid --rows
-O,O preloaded -o  v  virtual memory  --cumulative --format --deselect
-l,l long          u  user-oriented   --sort --tty --forest --version
                   X  registers       --heading --no-heading
                    ********* misc options *********
-V,V show version       L  list format codes  f  ASCII art forest
-m,m show threads       S  children in sum    -y change -l format
-n,N set namelist file  c  true command name  n  numeric WCHAN,UID
-w,w wide output        e  show environment   -H process heirarchy
OooPS!


"rpm -Va" reports nothing unusual either with the exception of these (I cut 
the unimportant crap out):

M......   /usr/bin/wall
Unsatisfied dependencies for cpio-2.4.2-16: /sbin/rmt
.M......   /bin/mount
.M......   /bin/umount
.M......   /bin/ping
.M......   /usr/bin/sperl5.00503
.M......   /usr/bin/suidperl
S.5....T   /usr/sbin/crond
SM5....T   /usr/bin/recover

Especially ...

rpm -V procps
rpm -V fileutils
rpm -V net-tools
rpm -V util-linux

... report nothing unusual, which is pretty neat.

The only dead giveaway is that "netstat -an|grep LISTEN" only produces the 
following output:

[root install]# netstat -an|grep LISTEN
unix  0      [ ACC ]     STREAM     LISTENING     698    /var/run/ndc
unix  0      [ ACC ]     STREAM     LISTENING     2538   
/var/lib/mysql/mysql.sock
unix  0      [ ACC ]     STREAM     LISTENING     1165   /tmp/.s.PGSQL.5432


So I installed a good LSOF from an RPM and look at the report of that:

]# lsof -n|grep LISTEN
]#

Ah, no network services running? I don't think so! ;o)

I reinstalled the "procps" and "net-tools", "util-linux" and "fileutils"
RPMs 
from the Cobalt FTP server and retried my "ps axf" and "netstat -an|grep 
LISTEN" commands. No changes in the output. 

I then looked for suspicious files and directories and found these:

/tmp/sdk386/
/usr/include/sdk386/

In there were quite a nice collection of tools. Among 'em a modified
"libnet", 
a couple of network sniffers for various purpose, scanners, a mass emailer 
and the backdoor stuff.

Intercepting the "|grep LISTEN" output certainly was done in runtime with a 
kernel modification, but I was unable to find the actual kernel module on
the 
box.

The actual running hacker processes were finally found with the following 
shell script:

----------------------------------------------------------------------------
----------
#!/bin/bash
cd /proc
for i in `seq 1 33000`; do test -f $i/cmdline && (cat $i/cmdline; echo " - "

$i); done
----------------------------------------------------------------------------
----------

Apparently a kernel module is also loaded which hides the processes and 
network connections which the hacker runs with his stuff.

In essence this is pretty impressive. That rootkit is apparently
specifically 
designed to fool the latest chkrootkit and it masks itself pretty well. 

Had the hacker been a little bit more careful ("netstat -an" not reporting 
anything) and had he hidden his files a little better, then this thing would

have gone almost unnoticed.

My bottom line: Do you still trust chkrootkit? Think again.

It still is a useful tool, but without watching the filesystem for actual 
changes (with Tripwire, FCHeck or similar tools) you might be coaxed into 
believing that everything is fine when it is not.

How the hackers got in? Difficult to say, but it could be a number of
factors 
about which I don't want to speculate in public.

-- 

With best regards,

Michael Stauber

_______________________________________________
cobalt-security mailing list
cobalt-security@xxxxxxxxxxxxxxx
http://list.cobalt.com/mailman/listinfo/cobalt-security