Drawing graphs with graphite

Work continues a-pace ... well kinda. I've added a new reporting plugin for a new monitoring system: Graphite [see launch-pad and wiki sites]. If you've not heard, is a funky new monitoring system that does away with the traditional RRDTool and does everything in python.

There's two main components to Graphite: carbon and graphite.

Carbon is a recording daemon (in fact, a set of three daemons) that stores information efficiently on disk (using a custom format) and maintain a fast in-memory cache. Sending new metric values to the carbon agent is very simply.

Graphite is a python web front-end that uses the Django framework and the ExtJS AJAX toolkit. Graph rendering is achieved using cairo (via python's cairo bindings). It's possible to run Graphite (Django) in stand-alone mode, but I guess most people will use mod_python and apache. Although there's a simple drag-and-drop compositor, the real power comes when using the CLI interface. There, each logged-in user can create their own custom graphs (multiple can be opened concurrently). These can be arranged on the screen and the resulting view saved for later recall.

It's a bit of a faff to setup (although better with v0.9.3) and there's a few rough edges (again, better with v0.9.3). That said, it's already usable and the AJAX interface is pretty nice. It's early days, so I'm not sure where it will fit within the monitoring eco-system compared to established projects (e.g., ganglia, munin, cacti). I guess time will tell.

Because of the way Graphite (and Carbon in particular) is designed, adding the MonAMI plugin to send it data is very easy. The code is now in CVS, ready for the next release. I've included a few screen shots that show the graph compositor.

New output plugin: grmonitor

Ladies and Gentlemen, MonAMI now has a new output plugin: grmonitor. This allows the latest version of gr_monitor (available from the project's home page) to connect to MonAMI and fetch the data it then plots.

gr_monitor plots data in 3D using an OpenGL library (e.g. the open-source Mesa). This allows you to pan around and see the live data from different points of view. On the right is a screen snapshot showing several Torque metrics.

gr_monitor expects data in a series of regular n-by-m grids. This is quite different to how MonAMI sees data (a tree structure) so the configuration has to map between the two. This makes it slightly verbose, but I'm hoping to add a few tricks to improve this.

Storing monitoring data in a database? no problem!

Ganglia is a monitoring system that uses RRDTool for its storage and graphs. This provides an excellent solution for monitoring, but suffers from data becoming less detailed ("averaged out") when you look further back in time. This is deliberate, but does make later analysis of the data difficult.

If you wanted to keep detailed records of monitoring data with MonAMI that don't degrade over time, now you can, I've committed changes to the mysql plugin in CVS. In addition to monitoring a MySQL database, the plugin can now store information. You tell it which table and how to map the information into that table and it does the rest, it'll even create the table if it doesn't exist.