www.it-ebooks.info
Implementing Splunk: Big Data
Reporting and Development for
Operational Intelligence
Learn to transform your machine data into valuable
IT and business insights with this comprehensive
and practical tutorial
Vincent Bumgarner
BIRMINGHAM - MUMBAI
www.it-ebooks.info
Implementing Splunk: Big Data Reporting and
Development for Operational Intelligence
Copyright © 2013 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval
system, or transmitted in any form or by any means, without the prior written
permission of the publisher, except in the case of brief quotations embedded in
critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy
of the information presented. However, the information contained in this book is
sold without warranty, either express or implied. Neither the author, nor Packt
Publishing, and its dealers and distributors will be held liable for any damages
caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.
First published: January 2013
Production Reference: 1140113
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.

Graphics
Aditi Gajjar
Production Coordinator
Nitesh Thakur
Cover Work
Nitesh Thakur
www.it-ebooks.info
About the Author
Vincent Bumgarner has been designing software for nearly 20 years, working in
many languages on nearly as many platforms. He started using Splunk in 2007 and
has enjoyed watching the product evolve over the years.
While working for Splunk, he helped many companies, training dozens of users to
drive, extend, and administer this extremely exible product. At least one person at
every company he worked with asked for a book on Splunk, and he hopes his effort
helps ll their shelves.
I would like to thank my wife and kids as this book could not
have happened without their support. A big thank you to all of
the reviewers for contributing their time and expertise, and special
thanks to SplunkNinja for the recommendation.
www.it-ebooks.info
About the Reviewers
Mathieu Dessus is a security consultant for Verizon in France and acts as the
SIEM leader for EMEA. With more than 12 years of experience in the security
area, he has acquired a deep technical background in the management, design,
assessment, and systems integration of information security technologies. He
specializes in web security, Unix, SIEM, and security architecture design.
Cindy McCririe is a client architect at Splunk. In this role, she has worked with
several of Splunk's enterprise customers, ensuring successful deployment of the
technology. Many of these customers are using Splunk in unique ways. Sample
use cases include PCI compliance, security, operations management, business

digital book library. Here, you can access, read and search across Packt's entire
library of books.
Why Subscribe?
• Fully searchable across every book published by Packt
• Copy and paste, print and bookmark content
• On demand and accessible via web browser
Free Access for Packt account holders
If you have an account with Packt at www.PacktPub.com, you can use this to access
PacktLib today and view nine entirely free books. Simply use your login credentials
for immediate access.
www.it-ebooks.info
Table of Contents
Preface 1
Chapter 1: The Splunk Interface 7
Logging in to Splunk 7
The Home app 8
The top bar 11
Search app 13
Data generator 13
The Summary view 14
Search 16
Actions 17
Timeline 18
The eld picker 19
Fields 19
Search results 21
Options 22
Events viewer 23
Using the time picker 25
Using the eld picker 26

Schedule 49
Actions 51
Summary 52
Chapter 3: Tables, Charts, and Fields 53
About the pipe symbol 53
Using top to show common eld values 54
Controlling the output of top 56
Using stats to aggregate values 57
Using chart to turn data 61
Using timechart to show values over time 63
timechart options 65
Working with elds 66
A regular expression primer 66
Commands that create elds 68
eval 68
rex 69
Extracting loglevel 70
Using the Extract Fields interface 70
Using rex to prototype a eld 73
Using the admin interface to build a eld 75
Indexed elds versus extracted elds 77
Summary 80
www.it-ebooks.info
Table of Contents
[ iii ]
Chapter 4: Simple XML Dashboards 81
The purpose of dashboards 81
Using wizards to build dashboards 82
Scheduling the generation of dashboards 91
Editing the XML directly 91

Chapter 6: Extending Search 143
Using tags to simplify search 143
Using event types to categorize results 146
Using lookups to enrich data 150
Dening a lookup table le 150
www.it-ebooks.info
Table of Contents
[ iv ]
Dening a lookup denition 152
Dening an automatic lookup 154
Troubleshooting lookups 157
Using macros to reuse logic 157
Creating a simple macro 158
Creating a macro with arguments 159
Using eval to build a macro 160
Creating workow actions 160
Running a new search using values from an event 161
Linking to an external site 163
Building a workow action to show eld context 165
Building the context workow action 165
Building the context macro 167
Using external commands 170
Extracting values from XML 170
xmlkv 170
XPath 171
Using Google to generate results 172
Summary 172
Chapter 7: Working with Apps 173
Dening an app 173
Included apps 174

Reasons for not working with advanced XML 202
Development process 202
Advanced XML structure 203
Converting simple XML to advanced XML 205
Module logic ow 210
Understanding layoutPanel 213
Panel placement 214
Reusing a query 215
Using intentions 217
stringreplace 217
addterm 218
Creating a custom drilldown 219
Building a drilldown to a custom query 219
Building a drilldown to another panel 222
Building a drilldown to multiple panels using HiddenPostProcess 224
Third-party add-ons 228
Google Maps 228
Sideview Utils 230
The Sideview Search module 231
Linking views with Sideview 232
Sideview URLLoader 232
Sideview forms 235
Summary 241
Chapter 9: Summary Indexes and CSV Files 243
Understanding summary indexes 243
Creating a summary index 244
When to use a summary index 245
When to not use a summary index 246
Populating summary indexes with saved searches 247
Using summary index events in a query 249

Using btool 290
An overview of Splunk .conf les 292
props.conf 292
Common attributes 292
Stanza types 296
Priorities inside a type 298
Attributes with class 299
inputs.conf 300
Common input attributes 300
Files as inputs 301
Network inputs 306
Native Windows inputs 308
Scripts as inputs 309
transforms.conf 310
Creating indexed elds 310
Modifying metadata elds 312
Lookup denitions 315
Using REPORT 318
www.it-ebooks.info
Table of Contents
[ vii ]
Chaining transforms 320
Dropping events 321
elds.conf 322
outputs.conf 323
indexes.conf 323
authorize.conf 325
savedsearches.conf 326
times.conf 326
commands.conf 326

Differing longevity 351
Differing permissions 352
Using more indexes to increase performance 353
www.it-ebooks.info
Table of Contents
[ viii ]
The lifecycle of a bucket 354
Sizing an index 355
Using volumes to manage multiple indexes 356
Deploying the Splunk binary 358
Deploying from a tar le 359
Deploying using msiexec 359
Adding a base conguration 360
Conguring Splunk to launch at boot 360
Using apps to organize conguration 361
Separate congurations by purpose 361
Conguration distribution 366
Using your own deployment system 366
Using Splunk deployment server 367
Step 1 – Deciding where your deployment server will run 367
Step 2 – Dening your deploymentclient.conf conguration 368
Step 3 – Dening our machine types and locations 368
Step 4 – Normalizing our congurations into apps appropriately 369
Step 5 – Mapping these apps to deployment clients in serverclass.conf 369
Step 6 – Restarting the deployment server 373
Step 7 – Installing deploymentclient.conf 373
Using LDAP for authentication 374
Using Single Sign On 375
Load balancers and Splunk 376
web 376

www.it-ebooks.info
www.it-ebooks.info
Preface
Splunk is a powerful tool for collecting, storing, alerting, reporting, and studying
machine data. This machine data usually comes from server logs, but it could also be
collected from other sources. Splunk is by far the most exible and scalable solution
available to tackle the huge problem of making machine data useful.
The goal of this book is to serve as an organized and curated guide to Splunk 4.3. As
the documentation and community resources available for Splunk are vast, nding
the important pieces of knowledge can be daunting at times. My goal is to present
what is needed for an effective implementation of Splunk in as concise and useful a
manner as possible.
What this book covers
Chapter 1, The Splunk Interface, walks the reader through the user interface elements.
Chapter 2, Understanding Search, covers the basics of the search language,
paying particular attention to writing efcient queries.
Chapter 3, Tables, Charts, and Fields, shows how to use elds for reporting,
then covers the process of building our own elds.
Chapter 4, Simple XML Dashboards, rst uses the Splunk web interface to build our
rst dashboards. It then examines how to build forms and more efcient dashboards.
Chapter 5, Advanced Search Examples, walks the reader through examples of using
Splunk's powerful search language in interesting ways.
Chapter 6, Extending Search, exposes a number of features in Splunk to help you
categorize events and act upon search results in powerful ways.
www.it-ebooks.info
Preface
[ 2 ]
Chapter 7, Working with Apps, covers the concepts of an app, helps you install a couple
of popular apps, and then helps you build your own app.
Chapter 8, Building Advanced Dashboards, explains the concepts of advanced XML

system administrators alike. This book does not try to act as a replacement for the
ofcial Splunk documentation, but should serve as a shortcut for many concepts.
For some sections, a good understanding of regular expressions would be helpful.
For some sections, the ability to read Python would be helpful.
Conventions
In this book, you will nd a number of styles of text that distinguish between
different kinds of information. Here are some examples of these styles, and an
explanation of their meaning.
Code words in text are shown as follows: "If a eld value looks like
key=value
in the text of an event, you will want to use one of the eld widgets."
A block of code is set as follows:
index=myapplicationindex
(
sourcetype=security
AND
(
(bob NOT error)
OR
(mary AND warn)
)
)
When we wish to draw your attention to a particular part of a code block, the
relevant lines or items are set in bold:
<searchPostProcess>
timechart span=1h sum(count) as "Error count" by network
</searchPostProcess>
<title>Dashboard - Errors - errors by network timechart</title>
Any command-line input or output is written as follows:
ERROR LogoutClass error, ERROR, Error! [user=mary, ip=3.2.4.5]

[ 5 ]
Errata
Although we have taken every care to ensure the accuracy of our content, mistakes
do happen. If you nd a mistake in one of our books—maybe a mistake in the text or
the code—we would be grateful if you would report this to us. By doing so, you can
save other readers from frustration and help us improve subsequent versions of this
book. If you nd any errata, please report them by visiting ktpub.
com/support
, selecting your book, clicking on the errata submission form link, and
entering the details of your errata. Once your errata are veried, your submission
will be accepted and the errata will be uploaded on our website, or added to any
list of existing errata, under the Errata section of that title. Any existing errata can
be viewed by selecting your title from />Piracy
Piracy of copyright material on the Internet is an ongoing problem across all media.
At Packt, we take the protection of our copyright and licenses very seriously. If you
come across any illegal copies of our works, in any form, on the Internet, please
provide us with the location address or website name immediately so that we
can pursue a remedy.
Please contact us at
with a link to the suspected
pirated material.
We appreciate your help in protecting our authors, and our ability to bring
you valuable content.
Questions
You can contact us at if you are having a problem with
any aspect of the book, and we will do our best to address it.
www.it-ebooks.info
www.it-ebooks.info
The Splunk Interface
This chapter will walk you through the most common elements in the Splunk

www.it-ebooks.info