Introduction


Here I am describing the preparations work for upgrading Nagios to a new version. With Nagios 3.3.1 just released, I'll take the time to document and walk through the steps for upgrading a production Nagios system. To learn more about the background, go Back to the introduction. Click Next to learn more about testing and customizing Nagios before cutting over.

Download the new Version of Nagios


First, I create a new staging directory for this Nagios version and download both packages, Nagios Core and the Nagios Plugins into it.

susie112:~ # cd /home/sw/source/nagios/
susie112:/home/sw/source/nagios # mkdir nagios-3.3.1
susie112:/home/sw/source/nagios # cd nagios-3.3.1/

susie112:/home/sw/source/nagios/nagios-3.3.1 # wget -q http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.3.1.tar.gz
susie112:/home/sw/source/nagios/nagios-3.3.1 # wget -q http://prdownloads.sourceforge.net/sourceforge/nagiosplug/nagios-plugins-1.4.15.tar.gz
susie112:/home/sw/source/nagios/nagios-3.3.1 # ls -l
total 3732
-rw-r--r-- 1 root root 1714630 2011-07-26 12:32 nagios-3.3.1.tar.gz
-rw-r--r-- 1 root root 2095419 2010-07-28 05:57 nagios-plugins-1.4.15.tar.gz

We untar the archives and fix the ownership, if necessary.

susie112:/home/sw/source/nagios/nagios-3.3.1 # tar xvfz nagios-3.3.1.tar.gz
nagios/
nagios/html/
nagios/html/media/
...
nagios/contrib/traceroute.cgi
nagios/functions

susie112:/home/sw/source/nagios/nagios-3.3.1 # tar xvfz nagios-plugins-1.4.15.tar.gz
nagios-plugins-1.4.15/
nagios-plugins-1.4.15/ABOUT-NLS
nagios-plugins-1.4.15/acinclude.m4
...
nagios-plugins-1.4.15/tools/tinderbox_build

susie112:/home/sw/source/nagios/nagios-3.3.1 # chown -R root:root nagios nagios-plugins-1.4.15
susie112:/home/sw/source/nagios/nagios-3.3.1 # ls -l
total 3740
drwxr-xr-x 16 root root    4096 2011-07-26 06:20 nagios
drwxrwxrwx 16 root root    4096 2010-07-28 05:49 nagios-plugins-1.4.15

Check Nagios Core Dependencies


Before I compile the new version, I doublecheck that the required library source code is available. Especially if the graphics libraries are missing, it is easy to overlook that statusmap.cgi, trends.cgi and histogram.cgi have not been built. Below is an example how to check packages on a rpm-based Linux distribution (OpenSUSE, SLES).

susie112:/home/sw/source/nagios/nagios-3.3.1 # rpm -q -a |grep 'gd-devel\|png-devel\|jpeg-devel\|zlib-devel'
libjpeg-devel-6.2.0-882.2.i586
gd-devel-2.0.36.RC1-54.1.i586
libpng-devel-1.2.39-2.4.1.i586
zlib-devel-1.2.3-140.2.i586

Compiling Nagios Core


Before compiling Nagios, we need to define system-specific parameters to set the installation directory, the Nagios username and group and others. Running ./configure --help shows all the options available.

If we are not on a dedicated development host, from here on we have to be very careful so we do not overwrite our active production files. In particular, I define several items with the new Nagios version number 3.3.1:

This will allow me later to run a second instance of Nagios and its Web GUI while not interfering with the existing, running production system.

susie112:/home/sw/source/nagios/nagios-3.3.1/nagios # ./configure --prefix=/srv/app/nagios-3.3.1 --with-command-group=nagcmd --with-nagios-user=nagios --with-nagios-group=nagios --with-httpd-conf=/etc/apache2 --with-init-dir=/etc/init.d --with-lockfile=/var/lock/nagios-3.3.1.lock --with-checkresult-dir=/var/spool/nagios-3.3.1 --with-htmurl=/nagios-3.3.1 --with-cgiurl=/nagios-3.3.1/cgi-bin
checking for a BSD-compatible install... /usr/bin/install -c
checking build system type... i686-pc-linux-gnu
...
*** Configuration summary for nagios 3.3.1 07-25-2011 ***:

 General Options:
 -------------------------
        Nagios executable:  nagios
        Nagios user/group:  nagios,nagios
       Command user/group:  nagios,nagcmd
            Embedded Perl:  no
             Event Broker:  yes
        Install ${prefix}:  /srv/app/nagios-3.3.1
                Lock file:  /var/lock/nagios-3.3.1.lock
   Check result directory:  /var/spool/nagios-3.3.1
           Init directory:  /etc/init.d
  Apache conf.d directory:  /etc/apache2
             Mail program:  /usr/bin/mail
                  Host OS:  linux-gnu

 Web Interface Options:
 ------------------------
                 HTML URL:  http://localhost/nagios-3.3.1/
                  CGI URL:  http://localhost/nagios-3.3.1/cgi-bin/
 Traceroute (used by WAP):  /usr/sbin/traceroute


Review the options above for accuracy.  If they look okay,
type 'make all' to compile the main program and CGIs.

Nagios provides several targets for the make command:

susie112:/home/sw/source/nagios/nagios-3.3.1/nagios # make all
cd ./base && make
make[1]: Entering directory `/home/sw/source/nagios/nagios-3.3.1/nagios/base'
gcc -Wall -g -O2 -DHAVE_CONFIG_H -DNSCORE   -c -o broker.o broker.c
...
*** Compile finished ***

If the main program and CGIs compiled without any errors, you
can continue with installing Nagios as follows (type 'make'
without any arguments for a list of all possible options):

  make install
     - This installs the main program, CGIs, and HTML files

  make install-init
     - This installs the init script in /etc/init.d

  make install-commandmode
     - This installs and configures permissions on the
       directory for holding the external command file

  make install-config
     - This installs *SAMPLE* config files in /srv/app/nagios-3.3.1/etc
       You'll have to modify these sample files before you can
       use Nagios.  Read the HTML documentation for more info
       on doing this.  Pay particular attention to the docs on
       object configuration files, as they determine what/how
       things get monitored!

  make install-webconf
     - This installs the Apache config file for the Nagios
       web interface

  make install-exfoliation
     - This installs the Exfoliation theme for the Nagios
       web interface

  make install-classicui
     - This installs the classic theme for the Nagios
       web interface
...

Nagios Installation


Next, we install the Nagios Core binaries into their final destination. Again, if we are on a live PRD system, we need to take extra care not to overwrite any existing files or directories. In my example, the current production Nagios is under /srv/app/nagios-3.2.3, and it is accessed through a symbolic link via /srv/app/nagios. I'll avoid touching production files by creating a separate directory for the new Nagios version 3.3.1. Since we are working very close to production files, it is good practice to take a fresh backup before continuing. Remember, typo's and mistakes happen easily even to the best of us.

susie112:~ # mkdir /srv/app/nagios-3.3.1

susie112:~ # ls -l /srv/app |grep nagios
lrwxrwxrwx  1 root root   21 2010-10-26 14:34 nagios -> /srv/app/nagios-3.2.3
drwxr-xr-x  8 root root 4096 2010-12-30 00:31 nagios-3.2.3
drwxr-xr-x  2 root root 4096 2011-08-10 12:16 nagios-3.3.1
lrwxrwxrwx  1 root root   26 2010-06-22 14:45 nagiosgraph -> /srv/app/nagiosgraph-0.9.1
drwxr-xr-x  7 root root 4096 2011-04-07 16:22 nagiosgraph-0.9.1
susie112:/home/sw/source/nagios/nagios-3.3.1/nagios # make install
cd ./base && make install
make[1]: Entering directory `/home/sw/source/nagios/nagios-3.3.1/nagios/base'
make install-basic
make[2]: Entering directory `/home/sw/source/nagios/nagios-3.3.1/nagios/base'
/usr/bin/install -c -m 775 -o nagios -g nagios -d /srv/app/nagios-3.3.1/bin
...
*** Main program, CGIs and HTML files installed ***

You can continue with installing Nagios as follows (type 'make'
without any arguments for a list of all possible options):

  make install-init
     - This installs the init script in /etc/init.d

  make install-commandmode
     - This installs and configures permissions on the
       directory for holding the external command file

  make install-config
     - This installs sample config files in /srv/app/nagios-3.3.1/etc

make[1]: Leaving directory `/home/sw/source/nagios/nagios-3.3.1/nagios'

Before we move on here, I want to highlight my favorite make output line during this step:

if [ no = yes ]; then \
		/usr/bin/install -c -m 664 -o nagios -g nagios p1.pl /srv/app/nagios-3.3.1/bin; \
	fi;

Huh? This incomprehensible output line reminds me of female logic (ladies reading, please forgive me...), which according to research, goes like this: yes = maybe, maybe = no, and no = yes! Yes! But not always...

If you are using external scripts to submit results into Nagios, for example SNMP trap parsers, then we need to create the directory for the nagios command pipe and have the file permissions set correctly.

susie112:/home/sw/source/nagios/nagios-3.3.1/nagios # make install-commandmode
/usr/bin/install -c -m 775 -o nagios -g nagcmd -d /srv/app/nagios-3.3.1/var/rw
chmod g+s /srv/app/nagios-3.3.1/var/rw

*** External command directory configured ***

Since I work directly on a PRD system, I need to verify the installation carefully against the original PRD directory. Do I have all files in place? Do they have identical permissions?

Nagios upgrade file comparison

We only have a handful files, a quick visual check shows that one file, show.cgi is missing. This is correct, because show.cgi is part of the nagiosgraph package and I simply copy it over to complete the installation.

Check Nagios Plugin Dependencies


Similar to Nagios, before we start compiling plugins we want to confirm that dependend libraries and header files are available. This will eliminate or reduce compilation errors in the next step. Nagios plugin dependencies are described in the toplevel DEPENDENCIES text file. For the most common plugins, I list them below:

PluginDependency
check_ldaps, check_http, check_tcp, check_smtp openssl or gnutls libraries for SSL connections
check_fping the fping program from http://www.fping.com/
check_ldap the LDAP libraries libldap and liblber, http://www.openldap.org/
check_mysql, check_mysql_query the libraries libmysql and libmysqlclient, http://www.mysql.org/
check_snmp, check_hpjd the NET-SNMP package from http://net-snmp.sourceforge.net/
check_ifstatus, check_ifoperstatus the Net::SNMP perl module, http://www.perl.com/CPAN/

Many of the dependencies above are also required by add-on plugins, so it is a good idea to install them if they are not already there. On a rpm-based system, I run a quick check similar to the example below to see if required packages are installed.

susie112:/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15 # rpm -q -a | grep 'ssl-dev\|fping\|ldap2-dev\|mysqlclient-dev\|snmp-dev\|perl-Net-SNMP'
libmysqlclient-devel-5.1.49-0.1.1.i586
fping-2.4b2-140.2.i586
libopenssl-devel-0.9.8k-3.14.1.i586
perl-Net-SNMP-5.2.0-134.3.i586
net-snmp-devel-5.4.2.1-11.2.1.i586
openldap2-devel-2.4.17-5.4.1.i586

Compile the Plugins


susie112:/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15 # ./configure --prefix=/srv/app/nagios-3.3.1 --with-nagios-user=nagios --with-nagios-group=nagios --without-world-permissions --with-cgiurl=/nagios-3.3.1/cgi-bin --enable-perl-modules
checking for a BSD-compatible install... /usr/bin/install -c
...
config.status: creating po/Makefile
            --with-apt-get-command: 
              --with-ping6-command: 
               --with-ping-command: /bin/ping -n -U -w %d -c %d %s
                       --with-ipv6: yes
                      --with-mysql: /usr/bin/mysql_config
                    --with-openssl: yes
                     --with-gnutls: no
               --enable-extra-opts: no
                       --with-perl: /usr/bin/perl
             --enable-perl-modules: yes
                     --with-cgiurl: /nagios-3.3.1/cgi-bin
               --with-trusted-path: /bin:/sbin:/usr/bin:/usr/sbin
                   --enable-libtap: no
susie112:/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15 # make
...
make[2]: Nothing to be done for `all-am'.
make[2]: Leaving directory `/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15'
make[1]: Leaving directory `/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15'

Plugin Installation


Plugin installation is straightforward. Plugins install into the [nagios-home]/libexec directory.

susie112:/home/sw/source/nagios/nagios-3.3.1/nagios # cd ../nagios-plugins-1.4.15/
susie112:/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15 # make install
Making install in gl
make[1]: Entering directory `/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15/gl'
make  install-recursive
...
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15'
make[1]: Leaving directory `/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15'

Plugin Verification


It is common to add a number of additional, non-core plugins to Nagios. In order to complete the plugin installation, we need to identify these and copy them over into the newly created libexec plugin directory. I am using rsync -un --ignore-existing to only pick up the plugin files that are missing out of the current production directory. In addition to the example below, I use the -n option for doing a dry-run first to check what files will be copied. Here, the list of additional plugins is small (23), this could easily be 50 or more.

Don't forget to set set the ownership of the copied files to nagios.

susie112:~ # rsync -uv --ignore-existing /srv/app/nagios-3.2.3/libexec/* /srv/app/nagios-3.3.1/libexec
check_apachestatus.pl
check_dbversion.cfg
check_dbversion_db2.class
check_dbversion_db2.java
check_dbversion_mssql.class
check_dbversion_mssql.java
check_dbversion_mysql.class
check_dbversion_mysql.java
check_http_time.pl
check_iftraffic3.pl
check_mysql_health
check_snmp_extend.sh
check_snmp_load.pl
check_snmp_mem.pl
check_snmp_patchlevel.cfg
check_snmp_storage.pl
check_snmp_time.pl
check_tablespace_mysql.class
check_tokyo_radiation.pl
check_tokyo_radiation_v10.pl
check_tokyo_tapwater.pl
nagios_send_host_mail.pl
nagios_send_service_mail.pl

sent 398220 bytes  received 449 bytes  797338.00 bytes/sec
total size is 4603903  speedup is 11.55

susie112:~ # chown nagios:nagios /srv/app/nagios-3.3.1/libexec/*

This concludes the first stage of the upgrade. A command scratchpad for copy-and-paste is here.

We successfully built and installed the latest version of Nagios as a second instance. Any errors that happen here during compilation and installation should be remediated before we move on to the next phase of testing and customizing Nagios.

Upgrading Nagios

Upgrade Preparations

Testing and Customization

Production Migration

More Information: