Splunk
Splunk is a flexible data aggregation system. In laymens' words, Splunk is a system that combs through log files (and anything else that contains structured information that you want to throw at it) and presents the results in a summarized format. It is really a pretty neat thing. See the splunk website.
Splunk at UNH
We are running the free 3.0beta3 on our system Jalapeno. Splunk is resource hungry. It requires at least 600MB of memory and quite a bit of CPU. Although it is possible to run a splunkd server deamon on each node and have these pass the information to the master node, this is not how I chose to set it up. Our splunbk setup is as follows:
- Splunk runs on Jalapeno. It is installed in /data/splunk, with a link to /opt/splunk.
- Jalapeno mounts the /var/log directories from einstein and roentgen so that it can be accessed by splunk for aggregation.
- The free version of splunk does not allow for login. We should restrict access to jalapeno to sysadmins.
- This can be extended to do many different tasks!
Connecting to Splunk
jalapeno isn't an HTTP server, so it's not possible to access the interface to Splunk by simply opening a web browser and going to the appropriate port. There is a fairly simple workaround, though:
ssh -L8001:localhost:8000 jalapeno. It doesn't necessarily have to be 8001, but some available port on your machine.- Open a web browser with good Javascript support (and optionally Flash, for some fancy graphing features), and go to localhost:8001 (or whatever port you chose)
