RE: [cobalt-security] Forensics on a hacked server

["Mich Connect" <bernie@xxxxxxxxxx>]

Mike,

Can I contact you offline to discuss some forensics on our servers.

B Johnson

-----Original Message-----
From: cobalt-security-admin@xxxxxxxxxxxxxxx
[mailto:cobalt-security-admin@xxxxxxxxxxxxxxx] On Behalf Of Michael Stauber
Sent: Wednesday, May 21, 2003 12:15 PM
To: cobalt-security@xxxxxxxxxxxxxxx
Subject: [cobalt-security] Forensics on a hacked server


 Hi all,

I just finished forensics on a customers RaQ4 which had been hacked. The box

was used to send SPAM and although some formmail scripts were present none
of
'em was responsible for the SPAM. Kinda unusual, right?

Patchlevel: All patches installed, except the new C37 Kernel.

Third party software: MySQL-3.23.37-1, php-4.1.2-3, OpenWebmail, Webalizer
and
OpenSSH-3.4p1.

According to the timestamps in /var/lib/cobalt the patches were mostly
installed in a timely manner.

After poking around for a while I found out that the box had been hacked and

'lo and behold: It is the most beautiful hack I have seen so far.

chkrootkit-0.40 reports *nothing* at all. The report it generates looks just

perfectly fine. No modified binaries, no LKMs, no trojans. Show that output
to someone who knows chkrootkit and he'd say: "So what? Looks good!"

chkrootkit-0.39 produces the same neat report, except that it bugs out on
the
LKM test:

###
### Output of: ./chkproc -v -v
###
ERROR: Thread display not implemented.
********* simple selection *********  ********* selection by list *********
-A all processes                      -C by command name
-N negate selection                   -G by real group ID (supports names)
-a all w/ tty except session leaders  -U by real user ID (supports names)
-d all except session leaders         -g by session leader OR by group name
-e all processes                      -p by process ID
T  all processes on this terminal     -s processes in the sessions given
a  all w/ tty, including other users  -t by tty
g  all, even group leaders!           -u by effective user ID (supports
names)
r  only running processes             U  processes for specified users
x  processes w/o controlling ttys     t  by tty
*********** output format **********  *********** long options ***********
-o,o user-defined  -f full            --Group --User --pid --cols
-j,j job control   s  signal          --group --user --sid --rows
-O,O preloaded -o  v  virtual memory  --cumulative --format --deselect
-l,l long          u  user-oriented   --sort --tty --forest --version
                   X  registers       --heading --no-heading
                    ********* misc options *********
-V,V show version       L  list format codes  f  ASCII art forest
-m,m show threads       S  children in sum    -y change -l format
-n,N set namelist file  c  true command name  n  numeric WCHAN,UID
-w,w wide output        e  show environment   -H process heirarchy
OooPS!


"rpm -Va" reports nothing unusual either with the exception of these (I cut
the unimportant crap out):

M......   /usr/bin/wall
Unsatisfied dependencies for cpio-2.4.2-16: /sbin/rmt
.M......   /bin/mount
.M......   /bin/umount
.M......   /bin/ping
.M......   /usr/bin/sperl5.00503
.M......   /usr/bin/suidperl
S.5....T   /usr/sbin/crond
SM5....T   /usr/bin/recover

Especially ...

rpm -V procps
rpm -V fileutils
rpm -V net-tools
rpm -V util-linux

... report nothing unusual, which is pretty neat.

The only dead giveaway is that "netstat -an|grep LISTEN" only produces the
following output:

[root install]# netstat -an|grep LISTEN
unix  0      [ ACC ]     STREAM     LISTENING     698    /var/run/ndc
unix  0      [ ACC ]     STREAM     LISTENING     2538
/var/lib/mysql/mysql.sock
unix  0      [ ACC ]     STREAM     LISTENING     1165   /tmp/.s.PGSQL.5432


So I installed a good LSOF from an RPM and look at the report of that:

]# lsof -n|grep LISTEN
]#

Ah, no network services running? I don't think so! ;o)

I reinstalled the "procps" and "net-tools", "util-linux" and "fileutils"
RPMs
from the Cobalt FTP server and retried my "ps axf" and "netstat -an|grep
LISTEN" commands. No changes in the output.

I then looked for suspicious files and directories and found these:

/tmp/sdk386/
/usr/include/sdk386/

In there were quite a nice collection of tools. Among 'em a modified
"libnet",
a couple of network sniffers for various purpose, scanners, a mass emailer
and the backdoor stuff.

Intercepting the "|grep LISTEN" output certainly was done in runtime with a
kernel modification, but I was unable to find the actual kernel module on
the
box.

The actual running hacker processes were finally found with the following
shell script:

----------------------------------------------------------------------------
----------
#!/bin/bash
cd /proc
for i in `seq 1 33000`; do test -f $i/cmdline && (cat $i/cmdline; echo " - "

$i); done
----------------------------------------------------------------------------
----------

Apparently a kernel module is also loaded which hides the processes and
network connections which the hacker runs with his stuff.

In essence this is pretty impressive. That rootkit is apparently
specifically
designed to fool the latest chkrootkit and it masks itself pretty well.

Had the hacker been a little bit more careful ("netstat -an" not reporting
anything) and had he hidden his files a little better, then this thing would

have gone almost unnoticed.

My bottom line: Do you still trust chkrootkit? Think again.

It still is a useful tool, but without watching the filesystem for actual
changes (with Tripwire, FCHeck or similar tools) you might be coaxed into
believing that everything is fine when it is not.

How the hackers got in? Difficult to say, but it could be a number of
factors
about which I don't want to speculate in public.

--

With best regards,

Michael Stauber

_______________________________________________
cobalt-security mailing list
cobalt-security@xxxxxxxxxxxxxxx
http://list.cobalt.com/mailman/listinfo/cobalt-security