Kibana is a visualising tool for Logstash project, a generic log management tool. Logstash is fed from different logging sources, and indexes them with Elastic Search. Kibana queries Elastic Search and displays results, with a nice and simple user interface. The third version of Kibana is now an Elastic Search sub project, an Angularjs application (pure javascript) using ajax for querying its index.
Kibana seems to be a killer application for spreading elastic search beyond full text search domain. Logstash seems to be a nice tool, too. Eating two good cakes at the same time ? Let's be abstemious and try Kibana alone.
Kibana is a simple static html website, dead simple to install. Elastic Search is easy to install, too. Using Elastic search from a web page is simple, but a bit ugly: same domain, port 9200. Please never do that on a public website. Elastic Search is naked, anybody can do anything, reading and writing. Kibana documentation explains how to use nginx as a proxy with authentication.
Parsing logs is a bit boring, and Logstash provides powerful tools for that. Lets try reading mails, Kibana needs lots of items with a date. Python mail parsing tools are broken, but Lamson provides ultimate mail reading tool, and my Thunderbird use classical mbox format. Don't even try to use Thunderbird mork index, they are doomed.
Kibana document model is simple. Few keys beginning with a @, tags and free style fields values. Here is a simple prototype with python, using two libraries :
pip install lamson pyelasticsearch
The script is simple : iterate over emails, build documents and index them. Two arguments are used : Elastic Search url and path to a mailbox.
Use a "since date" in the UI, your mails should receive less events than your syslog.
Kibana 3 is a nice tool, still young, easy to deploy, easy to hack, but the UX needs some love.
