UPDATE: Apache Logging Via ZeroMQ Part 2

When handling a large number of websites via Apache, perhaps spread across a number of servers, it can often be nice to gather these logs in one place.

At the moment we are partly using rsyslog for this. But lately I’ve been examining a different way of logging via network using the awesome ZeroMQ (or ØMQ).

In short, ZeroMQ is a lightweight, yet very powerful and intelligent, messaging queue. It was built for financial transaction and open-sourced.

So using ZeroMQ we can do something like this Ruby script:

               require 'zmq'
               require 'mysql2'
               
               z = ZMQ::Context.new 
               s = z.socket(ZMQ::UPSTREAM)
               s.bind("tcp://172.24.73.184:4010")
               
               client = Mysql2::Client.new(:host => "localhost", :username => "root", :database => "apache_access_logs")
               
               while true
                 escaped = client.escape(s.recv)
                 client.query("INSERT INTO entries (msg) VALUES ('#{escaped}');")
               end
             

What it does is, it listens to a socket (s.recv will block until a message is received) and then when it receives a message insert it into the DB.

Why not insert directly to the MySQL server? I’ll get to that in a little later.

So, now we have something listening for messages, we will of course also need something that sends messages:

               #!/usr/bin/ruby
               require 'rubygems'
               require 'zmq'
               
               z = ZMQ::Context.new
               s = z.socket(ZMQ::DOWNSTREAM)
               s.connect "tcp://172.24.73.184:4010"
               while(line = STDIN.gets)
               	s.send(line)
               end
             

This Ruby script read from STDIN in order to allow us to read piped logs from Apache.

Finally we need a little Apache magic:

                LogFormat "%P %v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" my_custom_log	
                CustomLog "|/usr/local/bin/zeromqlogger.rb" "my_custom_log"
            	

I have defined a custom logging format here, but you can of course just use vhost_other or some of the standard logformats in Apache.

Now, why wouldn’t we just make the Ruby script so that piped logs are inserted directly to the database?

First of all, what happens if the MySQL server crashes? Our script would have to cache the logs locally, keep track of the state of the server and so on and so fourth. What a headache!

Secondly, imagine we have several servers piping logs via our script. The load most likely will not be distributed evenly through out the day. Say a site gets slashdotted or whatever. Suddenly we might see problems where our logging script is bogging down the webservers because the MySQL server cannot keep up. Again, we would need the queries to be cached until we can deal with them. More coding. Or.. just use ZeroMQ :)

Add stuff like load balancing, low overhead etc. and I think you get the idea why ZeroMQ sounds like an interesting idea for this kind of stuff.