The future is messaging

Sam Ruby and Tim Bray have each picked messaging systems as a trend to watch in the future. In particular, they are naming Jabber/XMPP as something to watch. While I think XMPP is a great way to connect people to systems or other people, I think there are better ways to connect systems to systems.

Enter AMQP, a protocol designed by the financial industry to replace their existing proprietary message queues with something built on open standards. The working group is made up of seasoned messaging profesionals who are used to having billions of dollars worth of transactions moving through their systems without mysterious delays or drops.

There are two things that stand out in my mind about AMQP.

  1. Payload Fidelity: Any stream of bytes you place in the payload will come out the other end the same way. At the end of this post, I show an example of this. If you can do this with Jabber, please say so. I certainly haven't seen a way to do it.
  2. Queues instead of Buddy Lists: Instead of having human-centric buddy lists like with Jabber, you have accounts who have access to a namespace called a 'virtual host'. Inside of the 'virtual host' are named 'queue's and 'exchange's. Shove messages in one end and read them from the other.

RabbitMQ is an AMQP implementation built in Erlang and using Mnesia. Building clustered, high performance messaging systems in Erlang really plays to that language's strengths, I feel. Erlang seems great at shoveling bits from one network interface, doing some pattern matching on them, changing some bits, and blasting them out another interface.

In my experience, you can have a clustered rabbitmq setup running at home in under 20 minutes. It's all in the admin guide.

Plus that question about transporting binary data over your messaging layer? Here's a hack I wrote serializing Scala objects into RabbitMQ and operating on them on the other side. It will work with your default RabbitMQ installation:

Multiplication class definition

Anecdotally, I've heard that RabbitMQ can saturate a 100Mbps ethernet connection with a single 3Ghz Pentium 4. I have a couple of quad core opteron's (2x2 core) with 16G of RAM and multiple gigabit ethernet interfaces in my basement; Seeing how much processing power is needed to saturate a gigabit ethernet interface is definitely on my ToDo list!

# — 20 August, 2007