Introduction
Here I am describing how to test Nagios components after the latest version has been built. After downloading and compiling Nagios, testing will confirm that all is working fine. Then we can apply and re-test our customizations until we are ready for the next and final step, which is the production cutover.
Plugin Functional Test
First, we test if our new set of plugins work as expected by doing a manual check execution on the commandline. Although the example below shows the excution from the root account, it is best to try it using the Nagios user ID.
susie112:/srv/app/nagios/libexec # ./check_ping -H localhost -w 3000.0,80% -c 5000.0,100% -p 5
PING OK - Packet loss = 0%, RTA = 0.05 ms|rta=0.054000ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0
susie112:/srv/app/nagios/libexec # ./check_snmp_load.pl -H localhost -C SECro -w 60,55,50 -c 90,85,80 -T netsl
Load : 0.46 0.10 0.03 : OK
susie112:/srv/app/nagios/libexec # ./check_http -H localhost
HTTP OK: HTTP/1.1 200 OK - 11094 bytes in 0.002 second response time |time=0.001515s;;;0.000000 size=11094B;;;0Nagios Deamon Functional Test
Now we move on to prepare the test of the new Nagios version as a whole. We still need a configuration, so I install the included sample configuration.
susie112:/home/sw/source/nagios/nagios-3.3.1/nagios # make install-config
/usr/bin/install -c -m 775 -o nagios -g nagios -d /srv/app/nagios-3.3.1/etc
/usr/bin/install -c -m 775 -o nagios -g nagios -d /srv/app/nagios-3.3.1/etc/objects
/usr/bin/install -c -b -m 664 -o nagios -g nagios sample-config/nagios.cfg /srv/app/nagios-3.3.1/etc/nagios.cfg
...
/usr/bin/install -c -b -m 664 -o nagios -g nagios sample-config/template-object/switch.cfg /srv/app/nagios-3.3.1/etc/objects/switch.cfg
*** Config files installed ***
Remember, these are *SAMPLE* config files. You'll need to read
the documentation for more information on how to actually define
services, hosts, etc. to fit your particular needs.The provided example configuration comes with eight pre-defined standard service checks for localhost. I am adding one additional service for check_snmp_load.pl. Because it is not a core plugin, I need to add it into commands.cfg.
susie112:/srv/app/nagios-3.3.1/etc/objects # vi commands.cfg
define command{
command_name check_snmp_load
command_line $USER1$/check_snmp_load.pl -H $HOSTADDRESS$ -C $ARG1$ -T $ARG2$ -w $ARG3$ -c $ARG4$ $ARG5$
}Here I am creating one additional service definition, adding it into localhost.cfg to additionally test a SNMP-based plugin.
susie112:/srv/app/nagios-3.3.1/etc/objects # vi localhost.cfg
###############################################################################
# Define a servicegroup for load checks, load_check_localhost becomes a member
###############################################################################
define servicegroup{
servicegroup_name load-checks ; The name of the hostgroup
alias Performance Checks ; Long name of the group
}
###############################################################################
# Server load performance check via SNMP
###############################################################################
define service {
use local-service
host_name localhost
service_description os_cpu_load
check_command check_snmp_load!SECro!netsl!20,15,10!30,25,20
contact_groups admins
servicegroups load-checks
}Finally, I run the Nagios pre-flight check to make sure the test configuration is allright.
susie112:~ # /srv/app/nagios-3.3.1/bin/nagios -v /srv/app/nagios-3.3.1/etc/nagios.cfg
Nagios Core 3.3.1
Copyright (c) 2009-2011 Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 07-25-2011
License: GPL
Website: http://www.nagios.org
Reading configuration data...
Read main config file okay...
Processing object config file '/srv/app/nagios-3.3.1/etc/objects/commands.cfg'...
...
Checking misc settings...
Total Warnings: 0
Total Errors: 0
Things look okay - No serious problems were detected during the pre-flight checkIt looks like we are in good shape! We are now ready to manually start the new versions instance in parallel to the existing, running production Nagios. This is a critical step that could cause trouble to the existing production instance. A overlooked compilation parameter, a configuration mistake or a simple typo could cause havoc. If a parallel run with production can't be avoided, it is best to have a maintenance window for this task. Be prepared to quickly kill the test instance, and possibly re-start production if necessary.
susie112:~ # ps -ef |grep nagios
nagios 1022 1 0 Aug09 ? 00:00:13 /srv/app/nagios/bin/nagios -d /srv/app/nagios/etc/nagios.cfg
root 25725 29338 0 18:29 pts/0 00:00:00 grep nagios
susie112:~ # touch /var/lock/nagios-3.3.1.lock; chown nagios:nagios /var/lock/nagios-3.3.1.lock
susie112:~ # /srv/app/nagios-3.3.1/bin/nagios -d /srv/app/nagios-3.3.1/etc/nagios.cfg
susie112:~ # ps -ef |grep nagios
nagios 1022 1 0 Aug09 ? 00:00:13 /srv/app/nagios/bin/nagios -d /srv/app/nagios/etc/nagios.cfg
nagios 25741 1 0 18:29 ? 00:00:00 /srv/app/nagios-3.3.1/bin/nagios -d /srv/app/nagios-3.3.1/etc/nagios.cfg
root 25744 29338 0 18:29 pts/0 00:00:00 grep nagios
susie112:~ # tail /srv/app/nagios-3.3.1/var/nagios.log
[1313054992] Nagios 3.3.1 starting... (PID=25739)
[1313054992] Local time is Thu Aug 11 18:29:52 JST 2011
[1313054992] LOG VERSION: 2.0
[1313054992] Finished daemonizing... (New PID=25741)Nagios WebGUI Functional Test
After we have our new Nagios instance is running, we want to connect to it using the latest web GUI. The Nagios client part is located in two subdirectories: [nagioshome]/share has the generic web content such as the splash page and the side panel files, while [nagioshome]/sbin has the CGI files responsible for interacting with the Nagios daemon.
I generally like to move these directories under the webserver's document filesystem, while keeping a symbolic link for the original locations.
susie112:~ # mv /srv/app/nagios-3.3.1/share /srv/www/std-root/nagios.frank4dd.com/nagios-3.3.1
susie112:~ # mv /srv/app/nagios-3.3.1/sbin /srv/www/std-root/nagios.frank4dd.com/nagios-3.3.1/cgi-bin
susie112:~ # ln -s /srv/www/std-root/nagios.frank4dd.com/nagios-3.3.1 /srv/app/nagios-3.3.1/share
susie112:~ # ln -s /srv/www/std-root/nagios.frank4dd.com/nagios-3.3.1/cgi-bin /srv/app/nagios-3.3.1/sbin
susie112:~ # cd /srv/app/nagios-3.3.1
susie112:/srv/app/nagios-3.3.1 # ls -l share sbin
lrwxrwxrwx 1 root root 58 2011-08-12 10:30 sbin -> /srv/www/std-root/nagios.frank4dd.com/nagios-3.3.1/cgi-bin
lrwxrwxrwx 1 root root 50 2011-08-12 10:30 share -> /srv/www/std-root/nagios.frank4dd.com/nagios-3.3.1To test all functions of Nagios without authorization, we can allow guest access for the duration of our tests.
susie112:~ # sed 's/^#default_user_name=guest/default_user_name=guest/' -i /srv/app/nagios-3.3.1/etc/cgi.cfg
susie112:~ # sed 's/nagiosadmin/guest/' -i /srv/app/nagios-3.3.1/etc/cgi.cfg
susie112:~ # grep guest /srv/app/nagios-3.3.1/etc/cgi.cfg
default_user_name=guest
authorized_for_system_information=guest
authorized_for_configuration_information=guest
authorized_for_system_commands=guest
authorized_for_all_services=guest
authorized_for_all_hosts=guest
authorized_for_all_service_commands=guest
authorized_for_all_host_commands=guestNow is a good time to open a browser and have a first look at the new version of Nagios. Click on a image below to see an example screenshot of the main splash page, the tactical overview and the services view.
Nagios Web GUI Customization
To integrate Nagios into a web portal or a customers layout, adjusting the side panel colors and design elements is often requested, together with a customized logo and the additon of customer links. Since Nagios v.3.3.1 comes with updated CSS stylesheets, old changes cannot simply be copied but need to be manually integrated.
susie112:~ # cp /srv/www/std-root/nagios.frank4dd.com/nagios-3.3.1/stylesheets/common.css /srv/www/std-root/nagios.frank4dd.com/nagios-3.3.1/stylesheets/common.css.orig
susie112:~ # cp /srv/www/std-root/nagios.frank4dd.com/nagios-3.3.1/stylesheets/common.css /home/sw//source/nagios/nagios-3.3.1-20110812-common.css
susie112:~ # vi /home/sw//source/nagios/nagios-3.3.1-20110812-common.css
susie112:~ # cp /home/sw//source/nagios/nagios-3.3.1-20110812-common.css /srv/www/std-root/nagios.frank4dd.com/nagios-3.3.1/stylesheets/common.css
susie112:~ # vi /srv/www/std-root/nagios.frank4dd.com/nagios-3.3.1/side.phpVerification in the browser shows our applied style changes:
This concludes the second stage of the upgrade. A command scratchpad for copy-and-paste is here.
We successfully confirmed all functions of the new Nagios version. Any errors we see here should be remediated before we move on to the next phase, cutting over to production.