[cobalt-security] Analog shooting load average to 107.50? (no typo)

[Michael Stauber <cobalt@xxxxxxxxxxxxxx>]

Hi all,

some might say that this is not a security related issue. However, as it 
threatened the stability of my server and hindered normal operations I tend 
to consider it at least disturbing, if not problematic. So that's why I want 
to share this event with the participants of this list.

Last night at 5 a.m. european time one of my RaQ3's paged me with the report 
that the cpu was heavily loaded. The load average was listed with 46.20. I 
instantly logged in, did an "uptime", "ps auxw" and shutdown httpd, admserv, 
sendmail, mysql and named to reduce the system load.

Despite that the system load spiked to 107.50 and it took almost 40 minutes 
to come down to normal levels of 1.00 and less. Below is the output of the 
command "(date; uptime; cat /proc/meminfo)" at that time:

Tue Jun 26 05:45:49 CEST 2001
  5:45am  up 20:27,  1 user,  load average: 107.50, 52.92, 41.37
 
        total:    used:    free:  shared: buffers:  cached:
Mem:  130936832 34758656 96178176  7782400  2695168 12529664
Swap: 134692864 43425792 91267072
MemTotal:    127868 kB
MemFree:      93924 kB
MemShared:     7600 kB
Buffers:       2632 kB
Cached:       12236 kB
SwapTotal:   131536 kB
SwapFree:     89128 kB

Network traffic was light at that time and there were 15-20 http requests and 
a few DNS connections running. Maybe 5-6 of 'em. 

Relevant running processes at that time:

USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root     31334  0.0  0.0  1156    0 ?        SW   04:02   0:00 [crond]
root     31335  0.0  0.0  1528    0 ?        SW   04:02   0:00 [run-parts]
root     31336  0.0  0.0  1516    0 ?        SW   04:02   0:00 [logrotate]
root     31337  0.0  0.0  1212    0 ?        SW   04:02   0:00 [logrotate]
root     31338  0.0  0.0  1516    0 ?        SW   04:02   0:00 [sh]
root     31339  0.7  0.0  9252    0 ?        SW   04:02   0:45 [split_logs]
root      2099  0.0  0.0  1516    0 ?        SW   04:56   0:00 [sh]
root      2100  0.7  5.3  9632 6840 ?        D    04:56   0:21 
/usr/local/sbin/analog -U/home/sites/www.XXX.com/logs/web.cache -CCA

I know that the machine could need more memory (additional 256MB on 
order), but hey, just splitting 20 hours worth of logfiles for a couple of 
domains (40 or so) shouldn't put that kind of stress on the machine, right?

Does anyone know a way to permanently renice analog so that it runs with a 
lesser nice-level? Or would that cause any undesireable side effects?

Thanks in advance for any pointers. I'll put a pizza on the machine tonight. 
Should be nice and crispy in the morning. ;o)

-- 

Mit freundlichen Grüßen / Best regards

Michael Stauber

 Stauber Multimedia Design ____ Phone:  +49-6471-923812
 Hauptstrasse 31 ______  D-56244 Goddert ______ Germany
 SMD.NET ___ SOLARSPEED.NET ___ FORUMWORLD.COM