Indexed Logging Output with ElasticSearch

Intro

Bro’s default ASCII log format is not exactly the most efficient way for searching large volumes of data. ElasticSearch is a new data storage technology for dealing with tons of data. It’s also a search engine built on top of Apache’s Lucene project. It scales very well, both for distributed indexing and distributed searching.

Contents

• Indexed Logging Output with ElasticSearch
  • Intro
  • Warning
  • Installing ElasticSearch
  • Installing the ElasticSearch Plugin
  • Activating ElasticSearch
  • Tuning
  • TODO

Warning

This writer plugin only supports ElasticSearch 1; it will not work with ElasticSearch version 2 and above. This writer plugin is orphaned and is not being maintained. This is experimental and is not recommended for production use; it is for example missing error handling and may loose messages.

Installing ElasticSearch

Download the latest version from: http://www.elasticsearch.org/download/. Once extracted, start ElasticSearch with:

# ./bin/elasticsearch

For more detailed information, refer to the ElasticSearch installation documentation: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html

Installing the ElasticSearch Plugin

First, ensure that you have libcurl (headers and library) installed. Then the following will compile and install the plugin alongside Bro:

# ./configure && make && make install

See the output of ./configure --help for additional options if it can’t find any of the prerequisites.

If everything built and installed correctly, you should see this:

# bro -N Bro::ElasticSearch
Bro::ElasticSearch - ElasticSearch log writer (dynamic, version 1.0)

Activating ElasticSearch

The easiest way to enable ElasticSearch output is to load the plugin’s logs-to-elasticsearch.bro script. If you are using BroControl, the following line in local.bro will enable it:

@load Bro/ElasticSearch/logs-to-elasticsearch.bro

With that, Bro will now write most of its logs into ElasticSearch in addition to maintaining the Ascii logs like it would do by default. That script has some tunable options for choosing which logs to send to ElasticSearch, refer to the autogenerated script documentation for those options.

There is an interface being written specifically to integrate with the data that Bro outputs into ElasticSearch named Brownian. It can be found here:

https://github.com/grigorescu/Brownian

Tuning

A common problem encountered with ElasticSearch is too many files being held open. The ElasticSearch documentation has some suggestions on this and other issues.

• http://www.elastic.co/guide/en/elasticsearch/reference/1.3/setup-configuration.html

TODO

Lots.

• Perform multicast discovery for server.
• Better error detection.
• Better defaults (don’t index loaded-plugins, for instance).