Splunk

Splunk is a flexible data aggregation system. In laymens' words, Splunk is a system that combs through log files (and anything else that contains structured information that you want to throw at it) and presents the results in a summarized format. It is really a pretty neat thing. See the splunk website.

Splunk at UNH

We are running the free 3.0beta3 on our system Jalapeno. Splunk is resource hungry. It requires at least 600MB of memory and quite a bit of CPU. Although it is possible to run a splunkd server deamon on each node and have these pass the information to the master node, this is not how I chose to set it up. Our splunbk setup is as follows:

• Splunk runs on Jalapeno. It is installed in /data/splunk, with a link to /opt/splunk.
• Jalapeno mounts the /var/log directories from einstein and roentgen so that it can be accessed by splunk for aggregation.
• The free version of splunk does not allow for login. We should restrict access to jalapeno to sysadmins.
• This can be extended to do many different tasks!

Connecting to Splunk

jalapeno blocks all port 80 and port 8000 connections in the iptables, so it's not possible to access the interface to Splunk by simply opening a web browser and going to the appropriate port. This is a safety issue, so it is not going to change. There is a fairly simple workaround, you can open an ssh tunnel:

1. ssh -L8001:localhost:8000 jalapeno. It doesn't necessarily have to be 8001, but some non-priviledged available port on your machine.
2. Open a web browser with good Javascript support (and optionally Flash, for some fancy graphing features), and go to localhost:8001 (or whatever port you chose). On Linux and OS X only Firefox is compatible. On Window IE is compatible as well.

Sophisticated stuff for Splunk

You can use the admin button on the splunk web interface to do administration, add a user account (licensed version only), add new input streams. This is pretty simple. More sophisticated use is documented here: Splunk.com then go to documentation and click on the version used.