June 7, 2004
syslogd stops? is that allowed?
Over the weekend I had a machine kept complaining about disk space on /var and periodically would alert me that syslogd was stopping. We've had floods of logging to /var/log/syslog in the past when administrators are sending out mass email reminders about finishing end-of-year online evaluations. I rotated the logs once, then twice. Today I figured I'd better check and see what was up.
Come to find out the problem was a cron entry that didn't have stderr properly redirected to stdout and was attempting to send email locally (the machine doesn't allow), resulting in sendmail in a spiral of attempting to send messages, but having to stick them in the queue for another attempt. By the time I looked the queue was up to ~26,000 messages. Everytime sendmail would start through them a huge dump would go through syslogd and at some point became too much.
I fixed the output problem changing cron to include:
. . . 1>>sync.log 2>>error.log
Removed the messages in /var/spool/mqueue and we're back in business.
Posted by mike at June 7, 2004 12:09 AM