[cobalt-security] RaQ1 hacked

[Carrie Bartkowiak <ravencarrie@xxxxxxxx>]

Hey guys, 
There's another list member who's had his RaQ1 hacked twice now - 
within 18 hours of restoring it with all patches and putting the RaQ2 
OS on it and getting it back online, it was hacked again. 
He'd like some advice. He's tried posting to the list but it won't 
come through for some reason, so I'm trying for him.

The first letter is the one he sent me, the second is some questions 
that I asked him to answer, so that y'all will have more info.

CarrieB

FIRST LETTER:
I had a RaQ1 running with all the software patches installed in the
requested order from Cobalt and had Anon FTP running on one site. It 
got
hacked into on 08/15/01 and I received an email from the server 
advising
that there was a problem with crond, to restart the server. Not 
thinking I
went ahead and restarted it, it would not get past "Starting Up" on 
the lcd
screen. We removed the drive and accessed it from a linux pc and 
found that
somebody had been in it and changed some files in the startup and had 
it
referencing Intel compiled files. Not knowing what other damage had 
been
done I used the RaQ1 OS restore disk and decided to update it to the 
RaQ2
software.

        I did this and followed the instructions for installing the 
patches in the
proper order, even the one that is out of order regarding RaQ1 
hardware
using RaQ2 software. Once all the patches were installed we put it 
back
online and I uploaded the sites back to it. It went online on 
Wednesday
08/22/01 at 1336. On Thursday 08/23/01 at 0920 I again received an 
email
from the server that there was a problem with crond, to restart the 
server.
I have yet to do this.

        Going through the log files there are a large amount of 
information
missing on late Wednesday night and early Thursday morning. Then at 
around
0840 there are several failed login attempts to telnet, then at 
around
0915-0920 there are several files that have been changed and or 
created. I
am still every 15 minutes getting an email saying cannot execute 
binary
file "/bin/ps".

        I have a lot of log files that just do not look right. Under 
one log file
there is a startup "fork" message that is there when I first reloaded 
the
software and another when it was shutdown and reconnected to the 
network.
Around 0925-0950 there are 7 entries of Startup "fork". From there on 
I am
also seeing this message
kernel: Swansea University Computer Society IPX 0.34 for NET3.035
kernel: IPX Portions Copyright (c) 1995 Caldera, Inc.
kernel: Appletalk 0.17 for Linux NET3.035
modprobe: can't locate module binfmt-0
in the messages log.  This message repeats itself at a random time 
frame,
from 2 minutes apart to almost 9 minutes apart.

        I don't know what to do. If it was a fresh load with all of 
the supposed
security patches with NO Anon FTP running, how did they get in? They 
had
less than 18 hours total to hit it.

If you can help or direct me I would appreciate it very much. I have 
all of
the log files and corrupted/created files saved on my pc and 
compressed in
zip format in case he comes back and tries to block me out or clean 
up the
log files completely

SECOND LETTER:
I appreciate your replying to my questions. I have had this server 
for
about two and a half years, without any trouble. I am more of a 
dos/windows
person than linux so most of the stuff you had me do below is kind of 
new.
My main job is a police officer and I have the server and small 
business on
the side for some stress relief (and it isn't relieving anything 
right
now). We (me and one of our criminal investigators) have started 
tracking
the user of the ip that logged into this server to try and get a 
name. Here
in Virginia, computer hacking if considered malicious is a felony and 
who
knows what we might be able to find. I have below the questions you 
posted
to me along with the answers I have for them. Seeing that he hit it 
again
this morning I am going to go through the logs again right now and 
see what
I can find.

C - Do any of your users, besides you (admin), have telnet access?
P - Only Me

C - What funny stuff shows up when you do a "last | less" (no quotes)
from the command line?
P - I can account for all except for
admin ttyp0        ust001        Thu Aug 23 0914-0922 (00:08)
admin ttyp0 ust001        Sun Aug 26 0921-0926 (00:04)

C - Is there any funny stuff in /root/.bash_history? (Read this with 
pico.)
P - Read .bash_history in /root      0 lines read

C - Is there anything besides ftp and pop3 service uncommented in
/etc/inetd.conf?
P - ftp, telnet, pop-3, imap are the only ones uncommented

C - When you do a "top -c" from the command line, what strange 
processes do
you see running?
P - The ones that show up running are:
top -c, init, kflushd, kswapd, md_thread(twice), nfsiod (three),
/usr/sbin/ht (three), /sbin/mgetty, update (bdfl, /sbin/kernel

C - You said you removed the hard drive, so I am hoping that you have 
this
RaQ in your possession and it's not co-located somewhere.
P - It's co-located about 12 miles from me at my ISP. They have 
around 12
or 13 various RaQ products and have also had several of them hit in 
the
last few weeks. It seems from what we have back traced that the hits 
are
coming from Canada (on mine) and they have also got hits from Europe. 
I do
not feel that anyone at the local isp is doing the hacking as I know 
all of
them and they have been business partners for several years. They 
have
requested our services for prosecuting hackers several times over the 
last
couple of years also.

C - Then before you put it back online, install:SSH (disable telnet 
after
installing SSH and confirming that it works)
P - Exactly what is SSH and how do you get to command line with 
Telnet
disabled?

C - PortSentry (www.psionic.com) Logcheck (www.psionic.com)
P - I have the Portsentry and tried to install it on Friday night and 
could
not get it to run. It compiled with no errors, installed with no 
errors but
would not run. Not sure what I may have done wrong, I thought I 
followed
the simple instructions pretty closely, Guess not.

C - IPChains (can get an RPM of this just about anywhere)PMFirewall 
(to
configure IPChains for you)
P - What does this do that the above two (Portsentry and SSH) not do?

C - TripWire (keep a copy of the tripwire database on a different 
machine
or on CD; have TripWire run a database check every night to see what 
files,
if any, have been changed and email you about them)
P - Also about this.

C - Take away shell access for all of your users besides yourself. 
There is
no reason any of them need it; besides to hack or cause havoc. And 
that's
more usernames/passwords out there floating around for someone to 
crack or
find and get into sensitive parts of your machine.
P - No one else on the machine has any special access other than 
FrontPage
access on a couple sites, the other two sites I manage for the 
customers.

C - I wish I could be of more help, but without access to the machine 
I
can't dig around and see how he came in. I'm also not nearly as much 
of a
security expert as the guys on the security list. But I do know that 
BEFORE
that machine goes online, it should be braced like a fortress, and 
IPChains
is the only way you're going to be able to do that.
P - At this point I have spent a considerable amount more than I
anticipated to on redoing the machine and now I have to reload it 
again, at
least I don't have to find the special network card and cd again.

C - If you like, I can try posting your question to the security list 
and
see if it goes through?
P - If you would please.


That's it!
I've given him some instructions on how to install all of this stuff, 
answered his questions about IPChains, Tripwire, etc.
His name is Patrick.
As I have no experience with the RaQ1's and 2's, I'm hoping some of 
the gurus on the list can help him out. I'm not sure what little 
differences there are in using these programs on the MIPs vs. the 
Intel processor.

Carrie