rafaelsteil.com

Apache remote logging with rsyslog

One of the challenges of running a cluster of webservers is to decide where to store the log files for post-processing, considering they are relevant for you for any reason. The first approach most people think is to keep them locally and sync to a central server using rsync from time to time. However, such approach can still lead to missing entries if the machine goes down between synchronizations, which is specially true when using Amazon AutoScaling to dynamically add and remove servers from the grid (which is our case).

A much better approach is to log everything to a remote server right on spot, instead of relying on local files. The most common solution is to use rsyslog, which comes with default with the majority of Linux distributions, or syslog-ng, which claims to be a better tool than rsyslog.

In order to make it work with rsyslog, three things are necessary: configure the webserver (Apache, in my case) to delegate the logging to an external program (/usr/bin/logger), configure the log server to accept connections either in UDP or TCP ports, and configure the client (which should be the same machine where the webserver is running) to send the information to the server.

 To configure Apache, open its configuration file and change the directive CustomLog to it pipes the contents to an external program, like this:

CustomLog "|/usr/bin/logger -t apache -p local6.info" combined

The next step is to configure rsyslog client, which usually is located at /etc/rsyslog.conf, where we need at least specify that the level “local6.info” should be sent remotely – you may choose to send everything to the remote log server, however in my case I only want to send the Apache stuff. It works like this:

# Syntax:
# <level> @<IP>:<port>
local6.info @10.11.12.13:514

For “<level>” you can pass “*.*” to send everything. The single “at” symbol (@) means that the connection will be made via UDP, while two @ is for TCP.

The last step is the server, also in /etc/rsyslog.conf, where we need to enable the UDP module and specify where the Apache logs should be written. The following is the minimum you need:

$ModLoad imuxsock.so
$ModLoad imklog.so

# Provides UDP syslog reception
$ModLoad imudp.so

# The port where to listen
$UDPServerRun 514

# Write all apache logs to this file (please note the comma)
$template apacheAccess,"/var/log/apache_access_log"

# If the log's tag is "apache" and matches
# the defined level, send it to a specific file
if $syslogtag == 'apache' then {
    local6.info ?apacheAccess
    & ~
}

That’s all you need.

One last thing: officially each line is limited to 4kb by rsyslog, although I have heard that the kernel ring size also plays a role. In any case, content bigger than that will be trimmed.