Re: [cobalt-security] Re: Cobalt Raq 4 Hacked

["Lance Rushing" <lance_rushing@xxxxxxxxxxx>]

Well, I guess I'll chime in.

We had a RaQ4 (raid) that when bad last week.   I don't know if it was a
hack or hardware failure.  Around 2 pm Tuesday, the machine went dead to we
requests.  Luckily I had a ssh tunnel open (for using IMAP) and started
trying to figure out what when wrong.  But as I started typing commands I
was getting alot of "command not found" from the command prompt.  Well the
entire /bin directtory was gone!!, /etc was empty.  and /var was empty also.

Well I wasn't feeling so happy. After the ISP tried to fix it with some disk
tools and gave up.  I went over and got the hard drive.  Took it home,
mounted it up to another linux box and got all of /home off (thankyou
Jesus!!).  I had an 10 month old backup in the form of a raq migration
utility dump.

I returned the harddrive and they reinstalled the machine. And I patched it
with all of the latest.  After uploading the mu file and running it (1.2
gig). and then uploading a tar.gz of the /home (2.2 gig) I had the machine
limping along pretty good.  But then it started to die again.  I was on
console and I think it was the 'whereis' command I ran got a "command not
found".  A quick cd to /dir showed that it only had half the files it should
have.  I immediatily shutdown the machine to prevent further data loss.
This was now Wendnesday night.

Two failuers or possible hacks in two days!?!?!? ugh.  I don't know if I got
hacked again, or it is a hardware failure.

Well to keep the story shortish, we decided to abonded the RaQ completely,
got set up on a P4 Hyper thread box with RH9.  Got all 120+ domains set up
on it.  My isp was able to reroute all 100 ips to the new box.  Instead of
using sendmail, I set up postfix. Instead of proftpd, we are on vsftpd now.

I've kept the same directory structure for the RaQ and I am considering
writing a control pannel to handle web/users/ftp/email for the new box.  But
none of our customers really ever used it anyway.

Over all I'm sad that I had so work so much to get the server back up, but
I'm happy to have the cobalt behind me.

Goodbye cobalt,
Lance


Plug for my isp: Here is the link for the new server
http://www.cari.net/Apps-template/apps-template.html?service_key=141 we went
too.


----- Original Message ----- 
From: "Jelmer Jellema" <lists@xxxxxxxxxxxxxxx>
To: <cobalt-security@xxxxxxxxxxxxxxx>
Sent: Thursday, November 06, 2003 01:54
Subject: RE: [cobalt-security] Re: Cobalt Raq 4 Hacked


> Tripwire works just fine, though it will take you a while to get the right
> policy-entries in (if I recall correctly starting with the redhat version
of
> a prefab policy file saves you some time but if you use the
> cobalt-management admin system you should add stuff).
>
> I run it from cron every x hours and it reports nicely everytime I changed
> something. Ofcourse I cannot know whether it failed to report some hack,
but
> I still feel secure (as did you until...).
>
> I could send you - off list - a policy file I use on a Raq4, with some
> site-specifik stuff out. But as I don't use the cobalt management stuff
any
> more, this is not very well included.
>
> I take it you also have chkrootkit installed.
>
> Jelmer
>
> -----------------------------------------------------------------
> Jelmer Jellema - Spin in het Web
> http://www.spininhetweb.nl
> Spin in het Web: Alle Touwtjes In Handen
> -----------------------------------------------------------------
>
>
> > -----Oorspronkelijk bericht-----
> > Van: cobalt-security-admin@xxxxxxxxxxxxxxx
> > [mailto:cobalt-security-admin@xxxxxxxxxxxxxxx] Namens Cory
> > Hollingsworth
> > Verzonden: woensdag 5 november 2003 16:46
> > Aan: cobalt-security@xxxxxxxxxxxxxxx
> > Onderwerp: [cobalt-security] Re: Cobalt Raq 4 Hacked
> >
> >
> > > Subject: [cobalt-security] Cobalt Raq 4 Hacked
> > >
> > > Hi all.
> > > Today at 2:00 in the morning our RaQ4 was hacked.
> > >
> > > The hack replaced all files named index.* with own
> > > hacker content.
> > >
> > > We have had this issue a couple of weeks ago. That
> > > time the hack afected all index.* files under
> > > /home/sites.
> > >
> > > Now the hack affected all index.* files under / so the
> > > Control Panel is affected too.
> > >
> > > I must note a week ago the server wass rebuilded formm
> > > scratch and ALL upgrades were applied.
> > >
> > > I have chkrootkit and portsentry installed. None of
> > > those reported strange activity.
> > >
> > > I have checked open ports in the server and theres' no
> > > strange ports opened.
> > >
> > > I want to know if somebody has experienced the same
> > > issue, and any help will be apreciated..
> > >
> > > TIA
> > > Pablo
> >
> > I've had a similar problem with a Raq 4 last week.  Instead
> > my issue was someone using the server as a DOS attack
> > machine.  Their attack removed /var/log and turned off all
> > network services into the machine.  Also unbenounced to me on
> > an earlier date Apache stopped logging in the
> > /home/sites/XXX/log sub directories.
> >
> > Even though the machine was fully patched, the Apache logging
> > had halted before I applied the latest patches.  It may have
> > been compromised months before it was used as a DOS attack machine.
> >
> > Their DOS attack I believe unintentionally wiped out the root
> > filesystem so the machine could no longer boot.  It took a
> > couple of days of hacking at the serial console to get the
> > machine working well enough to ssh the accounts and data off
> > the machine.
> >
> > I've since moved everything onto a Raq550 I had hanging
> > around as a spare.  The only things lost were the users
> > passwords as I couldn't easily load them from the old shadow
> > file as the Raq550 stores passwords in a BTREE in /var/db/shadow.db.
> >
> > As of yet I haven't figured out how the machine was
> > exploited.  But we had 600+ users on the machine with CGI, so
> > it is not infeasible the exploit could have been in user CGI
> > rather than the core services.
> >
> > Any one have any luck with tripwire on Raqs?  That is the
> > only thing I can think of trying at this time.
> >
> > _______________________________________________
> > cobalt-security mailing list
> > cobalt-security@xxxxxxxxxxxxxxx
> > http://list.cobalt.com/mailman/listinfo/cobalt-security
> >
>
> _______________________________________________
> cobalt-security mailing list
> cobalt-security@xxxxxxxxxxxxxxx
> http://list.cobalt.com/mailman/listinfo/cobalt-security
>