Introduction
Here I am describing the preparations work for upgrading Nagios to a new version. With Nagios 3.3.1 just released, I'll take the time to document and walk through the steps for upgrading a production Nagios system. To learn more about the background, go Back to the introduction. Click Next to learn more about testing and customizing Nagios before cutting over.
Download the new Version of Nagios
First, I create a new staging directory for this Nagios version and download both packages, Nagios Core and the Nagios Plugins into it.
susie112:~ # cd /home/sw/source/nagios/
susie112:/home/sw/source/nagios # mkdir nagios-3.3.1
susie112:/home/sw/source/nagios # cd nagios-3.3.1/
susie112:/home/sw/source/nagios/nagios-3.3.1 # wget -q http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.3.1.tar.gz
susie112:/home/sw/source/nagios/nagios-3.3.1 # wget -q http://prdownloads.sourceforge.net/sourceforge/nagiosplug/nagios-plugins-1.4.15.tar.gz
susie112:/home/sw/source/nagios/nagios-3.3.1 # ls -l
total 3732
-rw-r--r-- 1 root root 1714630 2011-07-26 12:32 nagios-3.3.1.tar.gz
-rw-r--r-- 1 root root 2095419 2010-07-28 05:57 nagios-plugins-1.4.15.tar.gzWe untar the archives and fix the ownership, if necessary.
susie112:/home/sw/source/nagios/nagios-3.3.1 # tar xvfz nagios-3.3.1.tar.gz
nagios/
nagios/html/
nagios/html/media/
...
nagios/contrib/traceroute.cgi
nagios/functions
susie112:/home/sw/source/nagios/nagios-3.3.1 # tar xvfz nagios-plugins-1.4.15.tar.gz
nagios-plugins-1.4.15/
nagios-plugins-1.4.15/ABOUT-NLS
nagios-plugins-1.4.15/acinclude.m4
...
nagios-plugins-1.4.15/tools/tinderbox_build
susie112:/home/sw/source/nagios/nagios-3.3.1 # chown -R root:root nagios nagios-plugins-1.4.15
susie112:/home/sw/source/nagios/nagios-3.3.1 # ls -l
total 3740
drwxr-xr-x 16 root root 4096 2011-07-26 06:20 nagios
drwxrwxrwx 16 root root 4096 2010-07-28 05:49 nagios-plugins-1.4.15Check Nagios Core Dependencies
Before I compile the new version, I doublecheck that the required library source code is available. Especially if the graphics libraries are missing, it is easy to overlook that statusmap.cgi, trends.cgi and histogram.cgi have not been built. Below is an example how to check packages on a rpm-based Linux distribution (OpenSUSE, SLES).
susie112:/home/sw/source/nagios/nagios-3.3.1 # rpm -q -a |grep 'gd-devel\|png-devel\|jpeg-devel\|zlib-devel'
libjpeg-devel-6.2.0-882.2.i586
gd-devel-2.0.36.RC1-54.1.i586
libpng-devel-1.2.39-2.4.1.i586
zlib-devel-1.2.3-140.2.i586Compiling Nagios Core
Before compiling Nagios, we need to define system-specific parameters to set the installation directory, the Nagios username and group and others. Running ./configure --help shows all the options available.
If we are not on a dedicated development host, from here on we have to be very careful so we do not overwrite our active production files. In particular, I define several items with the new Nagios version number 3.3.1:
- Nagios lockfile
- Nagios checkresult directory
- HTML URL and CGI URL .
susie112:/home/sw/source/nagios/nagios-3.3.1/nagios # ./configure --prefix=/srv/app/nagios-3.3.1 --with-command-group=nagcmd --with-nagios-user=nagios --with-nagios-group=nagios --with-httpd-conf=/etc/apache2 --with-init-dir=/etc/init.d --with-lockfile=/var/lock/nagios-3.3.1.lock --with-checkresult-dir=/var/spool/nagios-3.3.1 --with-htmurl=/nagios-3.3.1 --with-cgiurl=/nagios-3.3.1/cgi-bin
checking for a BSD-compatible install... /usr/bin/install -c
checking build system type... i686-pc-linux-gnu
...
*** Configuration summary for nagios 3.3.1 07-25-2011 ***:
General Options:
-------------------------
Nagios executable: nagios
Nagios user/group: nagios,nagios
Command user/group: nagios,nagcmd
Embedded Perl: no
Event Broker: yes
Install ${prefix}: /srv/app/nagios-3.3.1
Lock file: /var/lock/nagios-3.3.1.lock
Check result directory: /var/spool/nagios-3.3.1
Init directory: /etc/init.d
Apache conf.d directory: /etc/apache2
Mail program: /usr/bin/mail
Host OS: linux-gnu
Web Interface Options:
------------------------
HTML URL: http://localhost/nagios-3.3.1/
CGI URL: http://localhost/nagios-3.3.1/cgi-bin/
Traceroute (used by WAP): /usr/sbin/traceroute
Review the options above for accuracy. If they look okay,
type 'make all' to compile the main program and CGIs.Nagios provides several targets for the make command:
- either 'all' or the 'nagios', 'cgis', 'contrib' and 'modules' directive can be used to build only a part of Nagios
- 'clean' is the usual method to re-compile everything from scratch
- install install-base install-cgis install-html install-exfoliation install-config install-init install-commandmode fullinstall
susie112:/home/sw/source/nagios/nagios-3.3.1/nagios # make all
cd ./base && make
make[1]: Entering directory `/home/sw/source/nagios/nagios-3.3.1/nagios/base'
gcc -Wall -g -O2 -DHAVE_CONFIG_H -DNSCORE -c -o broker.o broker.c
...
*** Compile finished ***
If the main program and CGIs compiled without any errors, you
can continue with installing Nagios as follows (type 'make'
without any arguments for a list of all possible options):
make install
- This installs the main program, CGIs, and HTML files
make install-init
- This installs the init script in /etc/init.d
make install-commandmode
- This installs and configures permissions on the
directory for holding the external command file
make install-config
- This installs *SAMPLE* config files in /srv/app/nagios-3.3.1/etc
You'll have to modify these sample files before you can
use Nagios. Read the HTML documentation for more info
on doing this. Pay particular attention to the docs on
object configuration files, as they determine what/how
things get monitored!
make install-webconf
- This installs the Apache config file for the Nagios
web interface
make install-exfoliation
- This installs the Exfoliation theme for the Nagios
web interface
make install-classicui
- This installs the classic theme for the Nagios
web interface
...Nagios Installation
Next, we install the Nagios Core binaries into their final destination. Again, if we are on a live PRD system, we need to take extra care not to overwrite any existing files or directories. In my example, the current production Nagios is under /srv/app/nagios-3.2.3, and it is accessed through a symbolic link via /srv/app/nagios. I'll avoid touching production files by creating a separate directory for the new Nagios version 3.3.1. Since we are working very close to production files, it is good practice to take a fresh backup before continuing. Remember, typo's and mistakes happen easily even to the best of us.
susie112:~ # mkdir /srv/app/nagios-3.3.1
susie112:~ # ls -l /srv/app |grep nagios
lrwxrwxrwx 1 root root 21 2010-10-26 14:34 nagios -> /srv/app/nagios-3.2.3
drwxr-xr-x 8 root root 4096 2010-12-30 00:31 nagios-3.2.3
drwxr-xr-x 2 root root 4096 2011-08-10 12:16 nagios-3.3.1
lrwxrwxrwx 1 root root 26 2010-06-22 14:45 nagiosgraph -> /srv/app/nagiosgraph-0.9.1
drwxr-xr-x 7 root root 4096 2011-04-07 16:22 nagiosgraph-0.9.1susie112:/home/sw/source/nagios/nagios-3.3.1/nagios # make install
cd ./base && make install
make[1]: Entering directory `/home/sw/source/nagios/nagios-3.3.1/nagios/base'
make install-basic
make[2]: Entering directory `/home/sw/source/nagios/nagios-3.3.1/nagios/base'
/usr/bin/install -c -m 775 -o nagios -g nagios -d /srv/app/nagios-3.3.1/bin
...
*** Main program, CGIs and HTML files installed ***
You can continue with installing Nagios as follows (type 'make'
without any arguments for a list of all possible options):
make install-init
- This installs the init script in /etc/init.d
make install-commandmode
- This installs and configures permissions on the
directory for holding the external command file
make install-config
- This installs sample config files in /srv/app/nagios-3.3.1/etc
make[1]: Leaving directory `/home/sw/source/nagios/nagios-3.3.1/nagios'Before we move on here, I want to highlight my favorite make output line during this step:
if [ no = yes ]; then \
/usr/bin/install -c -m 664 -o nagios -g nagios p1.pl /srv/app/nagios-3.3.1/bin; \
fi;Huh? This incomprehensible output line reminds me of female logic (ladies reading, please forgive me...), which according to research, goes like this: yes = maybe, maybe = no, and no = yes! Yes! But not always...
If you are using external scripts to submit results into Nagios, for example SNMP trap parsers, then we need to create the directory for the nagios command pipe and have the file permissions set correctly.
susie112:/home/sw/source/nagios/nagios-3.3.1/nagios # make install-commandmode
/usr/bin/install -c -m 775 -o nagios -g nagcmd -d /srv/app/nagios-3.3.1/var/rw
chmod g+s /srv/app/nagios-3.3.1/var/rw
*** External command directory configured ***Since I work directly on a PRD system, I need to verify the installation carefully against the original PRD directory. Do I have all files in place? Do they have identical permissions?
We only have a handful files, a quick visual check shows that one file, show.cgi is missing. This is correct, because show.cgi is part of the nagiosgraph package and I simply copy it over to complete the installation.
Check Nagios Plugin Dependencies
Similar to Nagios, before we start compiling plugins we want to confirm that dependend libraries and header files are available. This will eliminate or reduce compilation errors in the next step. Nagios plugin dependencies are described in the toplevel DEPENDENCIES text file. For the most common plugins, I list them below:
| Plugin | Dependency |
|---|---|
| check_ldaps, check_http, check_tcp, check_smtp | openssl or gnutls libraries for SSL connections |
| check_fping | the fping program from http://www.fping.com/ |
| check_ldap | the LDAP libraries libldap and liblber, http://www.openldap.org/ |
| check_mysql, check_mysql_query | the libraries libmysql and libmysqlclient, http://www.mysql.org/ |
| check_snmp, check_hpjd | the NET-SNMP package from http://net-snmp.sourceforge.net/ |
| check_ifstatus, check_ifoperstatus | the Net::SNMP perl module, http://www.perl.com/CPAN/ |
Many of the dependencies above are also required by add-on plugins, so it is a good idea to install them if they are not already there. On a rpm-based system, I run a quick check similar to the example below to see if required packages are installed.
susie112:/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15 # rpm -q -a | grep 'ssl-dev\|fping\|ldap2-dev\|mysqlclient-dev\|snmp-dev\|perl-Net-SNMP'
libmysqlclient-devel-5.1.49-0.1.1.i586
fping-2.4b2-140.2.i586
libopenssl-devel-0.9.8k-3.14.1.i586
perl-Net-SNMP-5.2.0-134.3.i586
net-snmp-devel-5.4.2.1-11.2.1.i586
openldap2-devel-2.4.17-5.4.1.i586Compile the Plugins
susie112:/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15 # ./configure --prefix=/srv/app/nagios-3.3.1 --with-nagios-user=nagios --with-nagios-group=nagios --without-world-permissions --with-cgiurl=/nagios-3.3.1/cgi-bin --enable-perl-modules
checking for a BSD-compatible install... /usr/bin/install -c
...
config.status: creating po/Makefile
--with-apt-get-command:
--with-ping6-command:
--with-ping-command: /bin/ping -n -U -w %d -c %d %s
--with-ipv6: yes
--with-mysql: /usr/bin/mysql_config
--with-openssl: yes
--with-gnutls: no
--enable-extra-opts: no
--with-perl: /usr/bin/perl
--enable-perl-modules: yes
--with-cgiurl: /nagios-3.3.1/cgi-bin
--with-trusted-path: /bin:/sbin:/usr/bin:/usr/sbin
--enable-libtap: nosusie112:/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15 # make
...
make[2]: Nothing to be done for `all-am'.
make[2]: Leaving directory `/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15'
make[1]: Leaving directory `/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15'Plugin Installation
Plugin installation is straightforward. Plugins install into the [nagios-home]/libexec directory.
susie112:/home/sw/source/nagios/nagios-3.3.1/nagios # cd ../nagios-plugins-1.4.15/
susie112:/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15 # make install
Making install in gl
make[1]: Entering directory `/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15/gl'
make install-recursive
...
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15'
make[1]: Leaving directory `/home/sw/source/nagios/nagios-3.3.1/nagios-plugins-1.4.15'Plugin Verification
It is common to add a number of additional, non-core plugins to Nagios. In order to complete the plugin installation, we need to identify these and copy them over into the newly created libexec plugin directory. I am using rsync -un --ignore-existing to only pick up the plugin files that are missing out of the current production directory. In addition to the example below, I use the -n option for doing a dry-run first to check what files will be copied. Here, the list of additional plugins is small (23), this could easily be 50 or more.
Don't forget to set set the ownership of the copied files to nagios.
susie112:~ # rsync -uv --ignore-existing /srv/app/nagios-3.2.3/libexec/* /srv/app/nagios-3.3.1/libexec
check_apachestatus.pl
check_dbversion.cfg
check_dbversion_db2.class
check_dbversion_db2.java
check_dbversion_mssql.class
check_dbversion_mssql.java
check_dbversion_mysql.class
check_dbversion_mysql.java
check_http_time.pl
check_iftraffic3.pl
check_mysql_health
check_snmp_extend.sh
check_snmp_load.pl
check_snmp_mem.pl
check_snmp_patchlevel.cfg
check_snmp_storage.pl
check_snmp_time.pl
check_tablespace_mysql.class
check_tokyo_radiation.pl
check_tokyo_radiation_v10.pl
check_tokyo_tapwater.pl
nagios_send_host_mail.pl
nagios_send_service_mail.pl
sent 398220 bytes received 449 bytes 797338.00 bytes/sec
total size is 4603903 speedup is 11.55
susie112:~ # chown nagios:nagios /srv/app/nagios-3.3.1/libexec/*This concludes the first stage of the upgrade. A command scratchpad for copy-and-paste is here.
We successfully built and installed the latest version of Nagios as a second instance. Any errors that happen here during compilation and installation should be remediated before we move on to the next phase of testing and customizing Nagios.