News for hackers. Stuff that matters.
Re:Centralized Logging With Rsyslog by psadauskas on Tuesday October 05, 2010 @03:13PM (#1760991)
I've been pushing to implement this for our application, but I'm told that we used to, and had to turn it off because it would saturate the IO of the logging server.

Has anyone else experienced this? Is it just a simple configuration tuning problem?

Re:Centralized Logging With Rsyslog by moe on Tuesday October 05, 2010 @05:08PM (#1761500)
it would saturate the IO of the logging server.

Rather unlikely unless your deployment is very large or you're doing extraordinarily expensive filtering/binning on the sink-host.

The first bottleneck is normally diskspace, not disk I/O. Those logs pile up very quickly, depending on how long you retain them.

The raw network and disk I/O, however, are rarely of concern. Before you approach either limit you're already logging to the tune of ~300G per hour - and have probably switched to a distributed architecture of some sort long ago.

Writing broad streams of sequential text is very cheap.

Making sense of what you wrote, ideally before actually writing it, but at the very least before being forced to purge it due to storage constraints, is the difficult part. ;-)

Re:Centralized Logging With Rsyslog by d4rt on Tuesday October 05, 2010 @03:31PM (#1761069)
I've had to up the maximum simultaneous connections, but otherwise I've been able to log extremely large volumes of data from 1,000s of machines, with syslog-ng. I'd assume the same is true with rsyslog.

Looking at limits on open files, and considering tcp timeouts and cookies were all we needed. Total volume was in the order of high 10s of GB.

If you want to unpack offline, contact details are in my profile.

Re:Centralized Logging With Rsyslog by gmcquillan on Tuesday October 05, 2010 @03:18PM (#1761018)
It definitely helped to separate the incoming queue from the actions queue on the central loghost. Unfortunately, I don't have good data comparing the two. At this time, our loghost is an EC2small, so relatively susceptible to I/O disruption, and we haven't had any problems.

If you're running a whole datacenter's worth of logs into your loghost, then you might want to consider a distributed approach. For medium sized companies, I don't see why rsyslog wouldn't do the trick.

Re:Centralized Logging With Rsyslog by illumin8 on Tuesday October 05, 2010 @05:00PM (#1761466)
It depends - if you're logging from thousands of machines or services, you might want to use UDP logging. Then, provided you don't saturate the network connection, you should be fine. UDP is connectionless, so you definitely won't have as much overhead.

On the other hand, a lot of high security environments want to encrypt their syslog traffic using something like stunnel, which introduces OpenSSL overhead as well as TCP connection overhead. With thousands of clients and lots of encryption going on you definitely are going to hit some limits sooner rather than later. Check kernel parameter net.ipv4.ip_local_port_range (on Linux) and make sure you have a large enough range to accomodate all of the clients.