www.it-ebooks.info
Nagios
www.it-ebooks.info
www.it-ebooks.info
Wolfgang Barth
Nagios
System andNetwork Monitoring
Munich SanFrancisco
www.it-ebooks.info
NAGIOS. Copyright
c
 2006 OpenSource PressGmbH
Allrightsreserved. No part of this work maybereproducedortransmittedinany form or by anymeans, electronic or
mechanical, includingphotocopying,recording,orbyany informationstorageorretrievalsystem, withoutthe prior
writtenpermission of thecopyright ownerand thepublisher.
Printedonrecycledpaper in theUnitedStatesofAmerica.
12345678910—09 08 07 06
No Starch Pressand theNoStarch Presslogo areregisteredtrademarksofNoStarch Press, Inc. Otherproduct and
companynames mentionedhereinmay be thetrademarksoftheir respective owners.Ratherthanuse atrademark
symbol with everyoccurrence of atrademarked name,weare usingthe namesonlyinaneditorialfashionand to the
benefit of thetrademark owner, with no intentionofinfringement of thetrademark.
Publisher: William Pollock
CoverDesign: Octopod Studios
U.S. editionpublishedbyNoStarch Press, Inc.
555 De HaroStreet,Suite 250, SanFrancisco, CA 94107
phone: 415.863.9900; fax: 415.863.9950; [email protected];http://www.nostarch.com
Original edition
c
 2005 OpenSource PressGmbH
PublishedbyOpenSource PressGmbH, Munich, Germany
Publisher: Dr.Markus Wirtz

1.3.2UserAuthentication 34
2Nagios Configuration 37
2.1The Main Configuration File nagios.cfg 38
2.2Objects—an Overview 41
2.3Defining theMachines to Be Monitored,with host 44
2.4GroupingComputersTogether with hostgroup 46
2.5Defining Services to Be Monitored with service 47
2.6GroupingServices Together with servicegroup 50
2.7Defining Addressees for ErrorMessages: contact 50
2.8The Message Recipient: contactgroup 52
2.9WhenNagios NeedstoDoSomething: the command Object 53
2.10 DefiningaTime Period with timeperiod 54
5
www.it-ebooks.info
Contents
2.11 Templates 54
2.12 Configuration Aids for ThoseToo Lazy to Type 56
2.12.1Defining servicesfor severalcomputers. 56
2.12.2One host groupfor allcomputers 57
2.12.3Other configuration aids 57
2.13 CGIConfiguration in cgi.cfg 57
2.14 TheResources File resource.cfg 59
3Startup61
3.1Checking theConfiguration 61
3.2GettingMonitoringStarted 63
3.2.1Manualstart 63
3.2.2Automatic start. 64
3.2.3Making configuration changescomeintoeffect 64
3.3Overviewofthe WebInterface 64
In More Detail 69

6.7.1Testing TCPports 110
6.7.2MonitoringUDP ports 112
6.8MonitoringDatabases 114
6.8.1PostgreSQL 115
6.8.2MySQL 119
6.9MonitoringLDAP DirectoryServices 121
6.10 Checking aDHCPServer. 124
6.11 MonitoringUPS withthe Network UPS Tools 126
7Testing LocalResources 133
7.1FreeHardDrive Capacity 134
7.2Utilizationofthe SwapSpace. 136
7.3Testing theSystem Load 137
7.4MonitoringProcesses 138
7.5Checking LogFiles 141
7.5.1The standardplugin check_log 142
7.5.2The modern variation: check_logs.pl 143
7.6KeepingTabsonthe Number of Logged-in Users 144
7.7Checking theSystem Time 145
7.7.1Checking thesystem time via NTP. 145
7
www.it-ebooks.info
Contents
7.7.2Checking system time withthe time protocol 146
7.8Regularly Checking theStatusofthe Mail Queue 147
7.9KeepinganEye on theModification Date of aFile 148
7.10 MonitoringUPSswith apcupsd 149
7.11 Nagios MonitorsItself 150
7.11.1Running thepluginmanually withascript 151
7.11.2 check_nagios as atool for CGIprograms. 152
7.12 Hardware Checks withLMSensors 152

11.1.1The Management Information Base. 179
11.1.2SNMPprotocolversions 183
11.2NET-SNMP 184
11.2.1Toolsfor SNMP requests 184
11.2.2The NET-SNMP daemon 187
11.3Nagios’s OwnSNMPPlugins 196
11.3.1The genericSNMPplugin check_snmp 196
11.3.2Checking severalinterfaces simultaneously. 201
11.3.3Testing theoperating status of individualinterfaces 203
11.4Other SNMP-basedPlugins 205
11.4.1Monitoringharddrive space andprocesseswith nagios-
snmp-plugins 205
11.4.2Observingthe load on networkinterfaces with check-
iftraffic 207
11.4.3The manubulon.com pluginsfor specialapplicationpur-
poses 209
12 TheNagios Notification System 215
12.1Who Should be InformedofWhat, When?. 216
12.2WhenDoesaMessage Occur? 217
12.3The Message Filter 217
12.3.1Switchingmessagesonand off systemwide 218
12.3.2Enablingand suppressing computer andservice-related
messages 219
12.3.3Person-related filter options 221
12.3.4Caseexamples 222
12.4ExternalNotification Programs 224
12.4.1Notification via e-mail 225
12.4.2Notification via SMS 227
9
www.it-ebooks.info

www.it-ebooks.info
Contents
15.3.1Avoidingredundancy in configuration files 269
15.3.2Defining templates 270
16 TheWeb Interface273
16.1Recognizingand Acting On Problems 275
16.1.1Commentsonproblematichosts 276
16.1.2Taking responsibilityfor problems:acknowledgements 278
16.2AnOverviewofthe IndividualCGI Programs. 279
16.2.1Variationsinstatusdisplay: status.cgi 279
16.2.2Additional information andcontrol center: extinfo.cgi 284
16.2.3Interfacefor externalcommands: cmd.cgi 288
16.2.4The most important things at aglance: tac.cgi 290
16.2.5Networkplan: thetopological mapofthe network(sta-
tusmap.cgi). 291
16.2.6Navigationin3D: statuswrl.cgi 293
16.2.7Queryingthe status withacell phone: statuswml.cgi 295
16.2.8Analyzingdisrupted partialnetworks: outages.cgi 295
16.2.9Queryingthe object definitionwith config.cgi 295
16.2.10 Availabilitystatistics: avail.cgi 296
16.2.11 What events occur, howoften? histogram.cgi 298
16.2.12 Filteringlog entriesafter specificstates: history.cgi 299
16.2.13 Whowas told what,when? notifications.cgi 300
16.2.14 Showingall logfileentries: showlog.cgi 301
16.2.15 Evaluating whatever youwant: summary.cgi 301
16.2.16 Followingstates graphically over time: trends.cgi 303
16.3Planning Downtimes 304
16.3.1Maintenance periods for hosts 305
16.3.2Downtimefor services 306
16.4Additional Information on Hostsand Services 307

18.1.1Installation 354
18.1.2The check_nt plugin 355
18.1.3Commandswhich can be runwithNSClient andNC Net .356
18.1.4AdvancedfunctionsofNC Net 363
18.2NRPEfor Windows: NRPE NT 371
18.2.1Installation andconfiguration 372
18.2.2Function test 373
18.2.3The Cygwin plugins. 373
18.2.4Perlplugins in Windows 374
12
www.it-ebooks.info
Contents
19 Monitoring Room Temperatureand Humidity 377
19.1Sensors andSoftware 378
19.1.1The PCMeasure software for Linux 378
19.1.2The queryprotocol 379
19.2The Nagios Plugin check_pcmeasure 379
20 Monitoring SAPSystems 383
20.1Checking without aLogin: sapinfo 384
20.1.1Installation 384
20.1.2First test. 384
20.1.3The plugin check_sap.sh 386
20.2MonitoringwithSAP’s OwnMonitoringSystem (CCMS) 388
20.2.1CCMS—a shortoverview 388
20.2.2Obtaining thenecessary SAPusage permissionsfor Nagios 390
20.2.3Monitorsand templates 392
20.2.4The CCMS plugins 394
20.2.5Performanceoptimization 398
Appendixes 399
ARapidlyAlternating States:Flapping 401

It’s ten o’clockonMonday morning. Thebossofthe branch officeisinarage.
He’s been waitingfor hoursfor an important e-mail, anditstill hasn’t arrived. It
can only be thefault of themailserver; it’s probably hung yetagain. Butaquick
checkofthe computer showsthatnomails have got stuckinthe queuethere,and
there’snomention either in thelog filethatamail from thesenderinquestionhas
arrived. So where’sthe problem?
Thecentral mail serverofthe companydoesn’t respondtoaping.That’sprobably
theroot of theproblem. Butthe IT departmentatthe companyheadoffice abso-
lutelyinsists that it is nottoblame.Italsocannotpingthe mail node of thebranch
office, butitmaintains that thenetworkatthe head officeisrunning smoothly,
so theproblemmustlie withthe networkatthe branch office. Thesearchfor the
errorcontinues.
Thehumiliating result:the VPNconnectiontoheadoffice wasdown, andalthough
theISDNbackup connectionwas working, no routetothe head office(andthus
to thecentral mail server) wasdefinedinthe backuprouter.Aglobally operating
IT service provider wasresponsible for thenetworkconnections (VPN andISDN)
betweenbranchand head office, for whom somethinglikethis“just doesn’t hap-
pen”.The endresult: many hoursspent searchingfor theerror,anirritated boss
(the meeting for whichthe e-mail wasurgentlyrequiredhas long sincefinished),
andasweating admin.
With aproperlyconfiguredNagios system,the adminstratorwould already have
noticedthe problemateight in themorning andbeen able to isolateits cause
withinafewminutes.Insteadoflosingvaluable time,the IT service provider would
have been informeddirectly. Thetimethenrequiredtoeliminatethe error(in this
case, half an hour)would have been sufficienttodeliverthe e-mail in time.
Asecondexample:somewhere in Germany, theharddrive on whichthe central
Oracledatabasefor ahospitalstoresits logfilesreaches full capacity.Although
this doesnot cause the“lightstogoout”inthe operating room,the database
stops workingand thereisconsiderable disruptiontoworkprocedures: patients
15

reachability—as arule, asimple ping is used.Aservice check selectively tests indi-
vidualnetworkservices such as HTTP, SMTP,DNS,etc., butalsorunning processes,
CPU load, or logfiles. Host checks areperformedbyNagios irregularlyand only
whererequired, for exampleifnoneofthe servicestobemonitored can be reached
on thehostbeing monitored. As long as oneservicecan be addressedthere,then
this is basically valid for theentirecomputer,sothatthistestcan be dropped.
Thesimplesttestfor networkservices consists of lookingtosee whether therele-
vant target portisopen, andwhether aserviceislisteningthere.But this doesnot
necessarily mean that,for example, theSSH daemonreally is running on TCPport
22. Nagios thereforeusestests for many servicesthatgoseveral stepsfurther.For
SMTP,for example, thesoftware tests whether themailserveralsoannounces itself
16
www.it-ebooks.info
Introduction
witha“220” output,the so-called SMTP greeting;and for aPostgreSQLdatabase,
it checks whether this will acceptanSQL query.
Nagios becomesespecially interesting throughthe factthatittakesintoaccount
dependencies in thenetworktopology (ifitisconfiguredtodoso).Ifthe target
system can only be reached throughaparticular router that hasjustgonedown,
then Nagios reports that thetargetsystem is “unreachable”, anddoesnot bother to
bombarditwithfurther host andservicechecks.The software puts administrators
in apositionwhere they can more quicklydetect theactualcause andrectify the
situation.
Thesuppliersofinformation
Thegreat strength of Nagios—evenincomparisonwithother networkmonitoring
tools—liesinits modularstructure:the Nagios core doesnot containone single
test. Insteaditusesexternalprogramsfor service andhostchecks,which areknown
as plugins.The basicequipment already containsanumber of standardplugins for
themostimportant applicationcases.Special requests that go beyond theseare
answered—provided that youhavebasic programming knowledge—by pluginsthat

If aspecific service is to be monitoredseven daysaweek round theclock, for exam-
ple, this doesnot mean that theadministrator in charge willnever be able to take
abreak: instead, youcan instruct Nagiostonotifythe person only from Mondays
to Fridaysbetween 8am and5pm,every twohours at themost. If theadminis-
trator in charge is notable to solve theproblemwithinaspecified period of time,
eighthours for example, then theheadofdepartmentresponsible should receive
amessage.Thisisalsoknown as escalation management.The corresponding
configuration is explainedinChapter 12.5frompage 231.
Nagios can also make useoffreelyconfigurable,externalprogramsfor notifica-
tions, so that youcan integrate anysystem youlike: from e-mail to SMStoavoice
serverthatthe administrator calls up andreceivesavoicemessage concerning the
error.
With itsWeb interface(Chapter 16 from page 273, Nagios provides theadminis-
trator withawiderange of information,clearly arranged according to theissues
involved. Whether theadmin needsasummary of theoverall situation, adisplay
of problematicservices andhosts andthe causesofnetworkoutages, or thesta-
tusofentiregroupsofhosts or services, Nagios provides an individually structured
information page for nearly everypurpose.
Throughthe Webfront end, an administrator can informcolleagues upon accepting
aparticularproblemsothattheycan concentrate on otherthingsthathavenot yet
been seen to.Information already obtained can be stored as commentsonhosts
andservices,justlikescheduled downtimes:Nagios prevents false alarms going off
in theseperiods.
By reviewingpastevents, theWeb interfacecan reveal what problems occurred in
aselected time interval, whowas informed, what thesituation wasconcerningthe
availabilityofahost and/or servicesduringaparticular time period—allthisalso
taking account of downtimes,ofcourse.
Taking in information from outside
Fortests,notifications, etc.,Nagios makesuse of externalprograms, butthe reverse
is also possible:through aseparateinterface(see13.1frompage 240),independent

which, despitegrowingcom-
petition,still enjoys greatpopularity.The relatively young,but very powerfulal-
ternative is calledCacti
4
:thishas alargerrange of applications,can be configured
via Webinterface, andavoidsthe restrictions in MRTG,which can only displaytwo
measured values at thesametimeand cannotdisplay anynegative values.
Nagios itself can also displayperformancedatagraphically,using extensions(Chap-
ter 17 from page 313).Inmanycases this is sufficient, butfor very dedicatedre-
quirements,the useofNagios in tandem withagraphic representation tool such
as MRTG or Cacti is recommended.
AboutThisBook
This book is directed at networkadministratorswho want to findout aboutthe
conditionoftheir systemsand networks usinganOpenSourcetool.Itdescribes
Nagios version2.0,which is somewhat differentfromits predecessors in itsconfig-
uration. Theplugins,onthe otherhand, lead theirown lives, aretoagreatextent
independentofNagios,and arethereforenot restricted to aparticularversion.
2
http://www.bb4.org/
3
http://www.mrtg.org/
4
http://www.cacti.net/
19
www.it-ebooks.info
Introduction
Even though this book is basedonLinux as theoperating system for theNagios
computer,thisisnot arequirement.Mostdescriptionsalsoapplytoother Unix
systems,
5

14. Theuse of this is shownintwo concrete examples—integrating syslog-ng and
processing SNMP traps.NSCAisalsoarequirementfor distributed monitoring,
discussedinChapter 15.
Even though youmay have already used theWeb interface, youmight still be
wonderingabout allthe detailedoptions that this offers. Chapter 16 triestoanswer
this question as completelyaspossible,supported by very helpfulscreenshots. It
5
Forexample,*BSD, HP-UX, AIX, andSolaris;the author doesnot know of anyNagiosversions
running under MacOSX.
20
www.it-ebooks.info
Introduction
also describesaseries of parameterswhich until nowhavenot been documented
anywhere, except in thesourcecode.
Althoughinits operation,Nagios concentrates primarily on trafficlight signals
(red-yellow-green), thereare ways of evaluating andrepresentingthe performance
dataprovided by plugins, whichare describedindetail in Chapter 17.
Networksare rarely homogeneous, that is,equippedonlywithLinux andother
Unix-based operating systems. ForthisreasonChapter 18 demonstrates what util-
itiescan be used to integrate andmonitor Windowssystems.
Chapter 19 uses theexample of alow-costhardwaresensortoshowhow room
temperatureand humidity can be monitoredsimplyyet effectively.
Nagios can also monitorproprietary commercialsoftware,aslongasmechanisms
areavailable whichcan querystates of thesystem integrated into aplugin. In
Chapter 20, this is describedusing an SAP-R/3system.
Theappendix Nagios Configuration introducesall theparametersofthe twocen-
tral configuration files nagios.cfg and cgi.cfg,while Rapidly Changing States:
Flapping and EventHandler aredevoted to some useful butsomewhatexoticfea-
tures.
Furthernotes on thebook