« My Weblog is a Source of Information on Gatorade | Main | Thinking about Bob Mould »

March 14, 2005

When webalizer Can't Deal with Largeness of Log Files

It appears to me that webalizer can't deal with large log files on it's own. When I say large, I mean over 3 Gb.

I've run into this before, you attempt to webalize your logs and get this silly error:

webalizer -c /home/stats/webalizer.conf -o /home/stats/www 2005_all.log
Webalizer V2.01-10 (Linux 2.6.9-1.667) English
Error: Can't open log file 2005_all.log
The 2005_all.log is a mergelog of the access_log files since January 1st from the load-balanced web servers. The merged log file is 3.2 Gb. The file is definitely there, and I'd just processed another file in the same directory. The only thing I can think is that beyond a certain size, webalizer just has to say no. I couldn't seem to find any information.

I've run into this in the past, and always broke down and did a grep to break the file up into one-month segments, but I didn't want to accept that. Then I wondered, "does webalizer handle a log file piped to it?"

Sure enough, if I cat the log file in it works:

cat 2005_all.log | webalizer -c /home/stats/webalizer.conf -o /home/stats/www
I've been using webalizer for some time, but have been hearing more about awstats lately. I like the fact that awstats preserves more than a single year of data, where webalizer overwrites the previous year's data. I wonder how fast the perl processing is, compared to webalizer's C.

Posted by mike at March 14, 2005 5:46 PM