May 14, 2003
Server Hosed with syslog Attempts
Today we had a machine serving HTTP/HTTPS go down, but not really. The machine would not respond to SSH/HTTP/HTTPS, ping requests were successful. I was able to connect with console and determine no no obvious problem with the machine (poking through the logs, looking at cpu and memory).
Would have suspected network issues but other machines in the same cabinet/router were fine.
After 10 minutes of determining nothing looked wrong I was asked to reboot the machine, the problem went away. Troubling to a person who believes with Unix/Linux machines that executing shutdown is only for moving machines or installing hardward. My faith was shaken.
Spent the evening going through the logs and found that in the 30 minute downtime our firewall blocked 18,400 attempts to syslog to another machine over the network . . . something that I wasn't aware was in our syslog.conf. In that time the machine did get some SSH/HTTP/HTTPS traffic out, but very little.
The working theory is that the overwhelming number of attempts for log packets seriously hampered the ability for other packets to get out.
Reading up on syslog turned up some interesting information:
the messages are unauthenticated and there is no mechanism to provide verified delivery and message integrity - IETF
Posted by mike at May 14, 2003 3:40 PM