[cobalt-security] I got Hacked, here's how I fixed it.

["Lance Rushing" <lance_rushing@xxxxxxxxxxx>]

First off I'd like to thank all the frequent contributors to this list.  The
archives provided much assistance in fixing the server.

I get a call from a client that portions of their web site aren't working.
Mysql was throwing an error about not being able to write to the /tmp
directory.  'No big deal' it think and telnet to the server.

Upon telneting I am instantly logged in!!!  No username required no
password.  I'm just at a 'bash# ' prompt.  BASTARDS!! A freaking root kit.

===========================
The Diagnosis

===========================

I checked Sun for any recent patches.   I've been lazy/busy the last months
and missed two updates.  I apply the patches and start going about cleaning
up the server.

Obviously /bin/login was infected because it wasn't authenticating.  I take
a look at the file, last modified Sep 9 17:00.  Also the file has all of its
sucSadAi attributes set.  I look around on the server and bind several other
binaries that are infected with the same date and all of the attributes set.

/bin/ps
/usr/bin/top
/bin/login
/bin/netstat
/sbin/ifconfig
 /usr/bin/pstree
/usr/bin/find
/usr/bin/slocate
/usr/bin/locate
/usr/bin/updatedb

I also find /usr/sbin/xntps and startup scripts for it in /etc/rc.d.

 So after some archive searching I get chkrootkit and run it.

Checking `pstree'... INFECTED
Checking `login'... INFECTED
Checking `ifconfig'... INFECTED
Searching for t0rn's v8 defaults... Possible t0rn v8 (or variation) rootkit
installed
Searching for Showtee... Warning: Possible Showtee Rootkit installed
Checking `lkm'... You have     4 process hidden for ps command

===========================
The Treatment

===========================

I remove the xntps from the startup scripts and reboot.

Next I replace all of the infected files.  You really shouldn't reinstall
the whole rpm because it might overwrite another binary that might be newer
(patched).  So you need to unpack the rpm and replace only the infected
files with the uninfected originals.

Find all the suspicious files:
 Method 1:
 Using lsattr should show you all files that have some special attribues set
 The following command will search the whole computer.
  lsattr -aR * 2> /dev/null | grep -ve "^--------" | grep -ve ":$" |
grep -ve ^$
 If you want to find ones that are just in certian directories:
  lsattr -aR /bin 2> /dev/null | grep -ve "^--------" | grep -ve ":$" |
grep -ve ^$
 Also check /sbin /usr/bin /usr/sbin /usr/local and /etc

 Method 2:
 Verify the rpms for the files in a binary directory
  rpm --verify -f /bin/*

 Example output: (see 'man rpm' for more info)
  S.5..UG.   /usr/bin/md5sum
  means the size, md5, User and Group are different than it's rpm
  so it needs to be replace:


1. Remove all of the locking attributes. (if necessary)
     chattr -s -u -c -S -a -d -A -i  [filename]

2. Figure out which rpm the binary came from.
 a. rpm -qf [filename]

 b. Check out which package that rpm in
  grep [rpm_name] /var/lib/cobalt/

 c1. If the rpm came from a package (other than the OS-#)

  Download the pkg again: (if necessary)
  mkdir /tmp/downloads
  cd /tmp/downloads
  wget ftp.cobalt.com/pub/packages/raq4/eng/[packge_file]

  Unpack it:
  mkdir /tmp/unpacked_pkgs
  tar -xzf [packge_file] -C /tmp/unpacked_pkgs

  Get the rpm:
  mkdir /tmp/rpms
  cp /tmp/unpacked_pkgs/RPMS/[rpm_file] /tmp/rpms

 c2. If the rpm came from the iso.

  Download the iso and md5sum: (if necessary)
  mkdir /tmp/iso
  cd /tmp/iso
  wget ftp.cobalt.com/pub/iso/960-RAQ4R101BU.iso.gz
  wget ftp.cobalt.com/pub/iso/960-RAQ4R101BU.iso.md5sum
  (This is the iso that my raq used, your iso name might be different)

  Unzip it:
  gunzip 960-RAQ4R101BU.iso.gz  (takes a while)

  Md5 verify:
  md5sum 960-RAQ4R101BU.iso
  cat 960-RAQ4R101BU.iso.md5sum

  Mount it:
  mkdir /mount/loop
  mount -o loop  /tmp/iso/960-RAQ4R101BU.iso  /mnt/loop

  Verify it is for your cobalt:
  cat /etc/build
  ls /mnt/loop/nfsroot-x86/bto-sync/production
  Make sure your build is reflected in the name of the directory.

  Get the rpm:
  mkdir /tmp/rpms
  cd mnt/loop/nfsroot-x86/bto-sync/production/3100R_1.155/rpms
  (My cobalt is a 3100R_1.155, yours might vary)
  cp [rpm_file] /tmp/rpms

 d. Unpack the rpm
  cd /tmp/rpms
  rpm2cpio [rpm_file] | cpio -idmv --no-absolute-filenames

 e. Replace the infected file
  cp -f /tmp/rpms/bin/sample /bin/sample   or:
  cp -f /tmp/rpms/sbin/sample /sbin/sample


Remove/fix any config files
look at /etc/inetd.conf ( I had some in.cfinger thing running)
also /etc/ld.so.hash  ( only delete once you have /bin/login working
correctly)



===========================
Conclusion

===========================

1. Stay up on updates more often.
2. get ipchains up
3. check logs often and then lock bad ip's out with ipchains
4. http://www.average.org/cobalt/quick-security-guide.html

===========================
Useful links

===========================

Unpack rpms:
http://la-samhna.de/library/compile/unpacking.html

Unpacking pkgs:
http://list.cobalt.com/pipermail/cobalt-users/2002-July/074123.html
ftp://ftp.cobalt.sun.com/pub/developer/TechNotes/DTN10-0-5.pdf
cat /usr/local/sbin/cobalt_upgrade

Mounting ISOs:
http://xpt.sourceforge.net/techdocs/linuxdisk/Mount/linuxdisk10.001.html

chkrootkit:
http://list.cobalt.com/pipermail/cobalt-security/2002-November/006842.html


Recovering from a hack:
http://www.incidents.org/archives/intrusions/msg00464.html
http://marc.theaimsgroup.com/?l=cobalt-users&m=104009524721034&w=2
t0rn:
http://list.cobalt.com/pipermail/cobalt-security/2001-July/002678.html
http://www.solarspeed.net/news/4.php
http://list.cobalt.com/pipermail/cobalt-security/2001-August/003161.html


Thanks and good luck,
Lance Rushing