Re: [cobalt-security] RaQ3i are still vulnerable AFTER BIND update..

[Michael Stauber <cobalt@xxxxxxxxxxxxxx>]

Hi Paul,

> One of my Cobalt RaQ3i server was completely shafted on Wednesday night and
> after a few hours i'm 100% sure its the t0rn rootkit and probably:
>
> The Lion worm is similar to the Ramen worm.

I've done security audits for several ISP's and have seen many infected 
Cobalt boxes this year. I was able to manually clean out and to secure about 
a dozend of boxes infected with t0rn, Lion and Adore, but in most cases an OS 
restore was the only feasible way to go. 

Why? That depends on the degree and duration of the infection. The longer the 
bad guys have been into your box, the more likely they fortified their 
position and the more nasty surprises are hidden somewhere. 

Aditionally an OS restore is more economic, as it'll take you out of business 
for 1-3 hours. Snooping around on the system and cleaning it out manually 
usually takes 4-8 hours and there's still no guarantee that it's really clean 
afterwards.

To regain the trust and to be sure that nobody else "owns" the box you have 
to wipe it clean with the OS restore. Then you install all the patches and 
install additional security measures. Like those that RAQport.com and others 
offer.

If you haven't done so already you might want to run "chkrootkit" from 
http://www.chkrootkit.org/ on your machine. It detects about 20 rootkits and 
their variants.

> The box certainly shows all of those signs.. it has about 60 sites on it
> and is a DNS server. It was definately patched with BIND Update 4.0.1
> RaQ3-All-Security-4.0.2-9353.pkg.

With great confidence I can say that none of the 80 Cobalt's I've patched, 
secured and worked on the last half year has been compromised by Li0n, Ramen 
or other worms. So unless proven otherwise I still tend to think that the 
Bind-8.2.3 which comes with RaQ3-All-Security-4.0.2-9353.pkg and 
RaQ4-All-Security-1.0.2-9353.pkg is safe. Then again, I always modified 
/etc/named.conf to run the named service as user named, group named. I still 
think that it's hillarious (if not maliciously dumb) that SUN/Cobalt doesn't 
implement this crucial and very effective security measure instead of running 
bind as root:root.

> I am in the process of rebuilding the machine but I would still like to be
> able to check for sure if this hacker/script really has wiped out
> everything in my /home directory... i can't do much of anything seeing as
> its replaced my ls, ps, top, etc. and (maybe not a feature of the toolkit)
> has made my filesystem Read-only. 

Do you know the "Swiss Army Knife of embedded Linux"? 

See: http://busybox.lineo.com/

Here is a blurb from their website: "BusyBox combines tiny versions of many 
common UNIX utilities into a single small executable. It provides minimalist 
replacements for most of the utilities you usually find in fileutils, 
shellutils, findutils, textutils, grep, gzip, tar, etc. BusyBox provides a 
fairly complete POSIX environment for any small or embedded system. The 
utilities in BusyBox generally have fewer options than their full featured 
GNU cousins; however, the options that are included provide the expected 
functionality and behave very much like their GNU counterparts."

So after compiling busybox you got untainted system tools like ps, ls, top 
and thelike. It's a great tool to diagnose hacked boxes, especially when 
you're not sure which onboard utilities you can still trust and which not.

> Does anyone know this bugger inside out
> and at least help find a way to get the filesystem writeable and maybe then
> I can start to remove obvious entries from scripts like inetd.conf an
> rc.sysinit and replace good copies of files.

The problem is that someone most likely inserted a kernel module at runtime 
which now protects parts of the filesystems and makes them read only for user 
root. When you look through your startscripts you might be able to find where 
this LKM is loaded, but that most likely won't help you as you can't edit the 
file in question.

The only worthwile thing you can do is to remove the harddisk from the RaQ 
and to put it as slave into another Linux machine. Then mount the partitions 
from the "infected" hardrive and either repair the files, or pull off the 
data which you want to recover for re-usage after an OS restore.

Anyhow ... best of luck with this and let me know if you have any additional 
questions.

-- 

With best regards,

Michael Stauber
SOLARSPEED.NET