Re: [cobalt-security] traffic question

[Matthew Nuzum <cobalt@xxxxxxxxxxxxx>]

On Tue, 2002-02-26 at 23:15, Edwin Groenescheij wrote:
    Matthew Nuzum wrote:
    > 
    > If you offer e-mail or ftp services, you're going to have to save and
    > process these logs.  I found a tool a while back on sunsite
    > (http://metalab.unc.edu) that will convert e-mail log files into common
    > log format so that it can be processed by standard web log processing
    > tools.  ProFTP also supports using the same log format as apache, so it
    > is easily processed.
    
    You don't really need to process these log files yourself, the
    Cobalt RaQ server already does that for you (see the *.log files
    in /home/sites/site/logs/). These log files are generated each
    night at 4.00AM (unless you changed the default logrotate/cron
    settings).
Great!  Thanks for the info on that.  I hadn't really looked at the
per-site logs before.

The benefit to working with the main, raw log files is that they contain
the data for all sites.  If you have 200 small, virtual sites, that
means you'd have 200 pre-processed log files to deal with.  Even though
these are in a very usable form already, the high number of separate
files might make them difficult to manage.

As a side note, if you have some tenacity and a hands-on attitude, I've
done some research into converting logfiles into SQL.  They can then be
inserted into an database that supports creating custom reports.  You
can then use your favorite report generator to see the data any way you
like.

As a matter of fact, I have a simple one in place that works very well
for web-traffic.  In a MySQL database, 10,000 page views takes aprox 3MB
(we only record page views, not hits).  Periodically, we normalize this
so that it takes much less space than than.

Once done, it's very easy to create some real simple reports for
bandwidth, page-views in a period of time, etc.

We have it updating at approx 15 minute intervals, so that our reports
are near-time.  I'll bet you could knock out a solution workable for you
in a day or two.