Introduction


nagios-logo

One of the core functions in a availability monitoring system is the ability to send out status alerts and notifications. Nagios only comes with a a set of 2 default notifications through e-mails: notify-host-by-email and notify-service-by-email. These notifications provide a minimalistic, barebone structure for sending a simple e-mail alert to defined contacts when problems occur (example screenshot below).

Nagios original notification example: host down

Notification Requirements


However, this minimal setup falls short and the Nagios-builtin format is not presentable to customers. Notification requirements in large companies are diverse and complex: Below are the critical ones I have received:

The reasons behind these requirements are as varied as todays IT organisations are:

Thanks to Nagios flexibility and documentation, we are able to meet all requirements with a little extra work (click example screenshots below to enlarge).

Nagios HTML notification example: host down  Nagios HTML notification example: host recovery

Review of the initial Notification Setup


The initial Nagios e-mail notification definition is in /etc/commands.cfg. There, a minimum set of Nagios macros is combined into a message string and then piped into the standard UNIX/Linux mail client program mail or mailx. Because the mailx program expects to get the e-mail text in Unix format with lines separated by newline (\n) characters, the shell command printf uses the formatting option %b to generate these newlines by interpreting the given (\n) escape characters. Below is the original definition, with the original command_line definition being separated into mutliple lines for better readability. Be careful with the backslash, a single missing space can ruin the whole command and prevent notifications from being send out.

# 'notify-host-by-email' command definition
define command{
    command_name    notify-host-by-email
    command_line    /usr/bin/printf "%b" "***** Nagios *****\n\n\
Notification Type: $NOTIFICATIONTYPE$\n\
Host: $HOSTNAME$\n\
State: $HOSTSTATE$\n\
Address: $HOSTADDRESS$\n\
Info: $HOSTOUTPUT$\n\n\
Date/Time: $LONGDATETIME$\n" \
| /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" \
$CONTACTEMAIL$
}
# 'notify-service-by-email' command definition
define command{
    command_name    notify-service-by-email
    command_line    /usr/bin/printf "%b" "***** Nagios *****\n\n\
Notification Type: $NOTIFICATIONTYPE$\n\n\
Service: $SERVICEDESC$\n\
Host: $HOSTALIAS$\n\
Address: $HOSTADDRESS$\n\
State: $SERVICESTATE$\n\n\
Date/Time: $LONGDATETIME$\n\n\
Additional Info:\n\n\
$SERVICEOUTPUT$" \
| /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" \
$CONTACTEMAIL$
}

First modifications


Before we look at tweaking the e-mail notification style, I make 3 basic changes to the overall setup.

  1. I move the notification definition from the original commands.cfg file into its own dedicated configuration file called notifications.cfg.
  2. I create a additional configuration file no-notification.cfg with identical definition names, but a command_line containing only a sleep 1 command (example definition below). This file is commented out in nagios.cfg and will be enabled a few times a year when the Nagios system is patched, rebooted or otherwise in maintenance. This simple measure avoids flooding our customers with false notifications when Nagios is brought up. After a longer downtime, it could take a while until services are re-checked and we learned to always do a sanity health check on the web GUI before we re-enable notifications.
  3. # 'notify-service-by-email' command definition
    define command{
      command_name	notify-host-by-email
      command_line  /bin/sleep 1
    }

  4. Lastly, I created 2 external Perl scripts responsible for sending out mail in place of the original printf/mailx definition

With these 2 newly developed scripts (nagios_send_host_mail.pl and nagios_send_service_mail.pl, one for host notifications and a separate for service notifications), it is possible to control through flags which format the email should be sent in, which language should be used and what information should be handed off for mailing out. Below I describe example setups that addressed my clients requirements.

Implementing carbon copy cc: notifications


One customer requested to receive Nagios notifications for some contacts to be defined as a carbon copy (cc:), instead of the standard e-mail recipient list in the (to:) field. The normal way of e-mail address hand-off is through the Nagios $CONTACTEMAIL$ macro definition. Since this macro is already used for the (to:) field, We used the first of the six additionally available contact macros addressx. I defined address1 to hold the (cc:) e-mail address list.

Lets look at an example: We want to send alerts in plain text format to a mailing list support@frank4dd.com using English, and keep two others, acct-mgr@frank4dd.com and public@frank4dd.com on (cc:).

define contact{
  contact_name      frank4dd-support                     ; Short name of user
  use               generic-contact                      ; use generic template defaults
  alias             Frank4DD support team                ; Full name of user
  email             support@frank4dd.com                 ; expands into the (to:) field
  address1          acctmgr@frank4dd.com, public@frank4dd.com  ; sending alerts as (cc:)
  host_notification_commands   host-email-text-en        ; host notification definition
}
define contactgroup{
  contactgroup_name frank4dd-hosting
  alias             Web hosting team for Frank4DD.com
  members           frank4dd-support
}

Now we can add the $CONTACTADDRESS1$ macro into the notification definition located in notification.cfg to receive the (cc:) address list we placed in contacts.cfg.

# 'host-email-text-en' command definition
# sends plaintext e-mails in English
define command{
        command_name    host-email-text-en
        command_line    /srv/app/nagios/bin/nagios_send_host_mail.pl -c "$CONTACTADDRESS1$"
}

Example:

Nagios notification example using cc: host down


Here is a service notification example that takes advantage of the scripts ability to determine if comments have been entered in Nagios, and sending the notification in HTML format using Japanese language for the alert field descriptions. First, we set the definition format in the users contact settings to be HTML, use URL's to Nagios GUI for intranet access, and set the language to be Japanese.

define contact
  contact_name                 frank4dd-admin
  use                          generic-contact
  alias                        Frank4DD support team
  email                        public@frank4dd.com
  host_notification_commands   host-email-html-int-jp
  service_notification_commands service-email-html-int-jp
}

Now we look at the corresponding notification definition in notification.cfg. The definition uses the format argument -f html, -u (include URL's) and the language argument -l jp. For more information about the arguments, see the scripts manual page for nagios_send_service_mail.pl.

# 'service-email-html-int-jp' command definition
# sends HTML e-mails in Japanese and includes Nagios URL for Intranet access
define command{
        command_name    service-email-html-int-jp
        command_line    /srv/app/nagios/etc/objects/nagios_send_service_mail.pl \
-c "$CONTACTADDRESS1$" \
-f html -u -l jp
}

Example:

Nagios notification example using HTML with Japanese language


Here is a service notification example that sends the e-mail in multipart S/MIME with a logo image displayed inline within the HTML code. The example also adds in the company name using the -p argument. Currently, the logo image is defined in the script as it does not change for me. It can be changed by creating a uuencoded version of a different logo image, instructions are inside the script. In a future version, it could be implemented to be picked up from a filename using a script argument if necessary.

define contact
  contact_name                 frank4dd-admin
  use                          generic-contact
  alias                        Frank4DD support team
  email                        public@frank4dd.com
  host_notification_commands   host-email-multi-int-en
  service_notification_commands service-email-multi-int-en
}

Now we look at the corresponding notification definition in notification.cfg. The definition uses the format argument -f multi, -u (include URL's) and ommiting language argument -l sets the language to English. For more information about the arguments, see the scripts manual page for nagios_send_service_mail.pl.

# 'service-email-multi-int-en' command definition
# sends HTML e-mails in English, includes Nagios URL and inline logo
define command{
        command_name    service-email-multi-int-en
        command_line    /srv/app/nagios/etc/objects/nagios_send_service_mail.pl \
-p "ACME Corporation, London Branch" \
-c "$CONTACTADDRESS1$" \
-f multi -u
}

Example:

Nagios notification example using multipart S/MIME HTML, English language, and a logo


Since version v1.4, the notification scripts are able to include the related performance graph image into the message. This greatly enhances the information provided inside the alert e-mail. However, this graph generation is dependend on the graph generation package used. It has been developed with Nagiosgraph v0.9.1, which I am using. The scripts work fine even with Nagiosgraph not being available, simply using the notification options 'text', 'html' and 'multi', leaving out the 'graph' option.

# 'service-email-graph-int-en' command definition, sends
# HTML e-mails in English, includes Nagios URL, inline logo and graph
define command{
        command_name    service-email-graph-int-en
        command_line    /srv/app/nagios/etc/objects/nagios_send_service_mail.pl \
-p "ACME Corporation, London Branch" \
-c "$CONTACTADDRESS1$" \
-f graph -u
}

Example:

Nagios notification example using multipart S/MIME HTML with English language, logo and graph

Downloads and Links


Nagios host and service notifications are handled through a dedicated script for each case, resulting in two scripts to download and install. The set below can provide embedded graph images from the popular nagiosgraph package (e.g. nagiosgraph 0.91).

nagios_send_service_mail.pl

nagios_send_service_mail.pl Latest version 1.8.1, 48814 Bytes, for further documentation see also the manual page

nagios_send_service_mail-v180.pl Obsolete version 1.8.0, 46712 Bytes

nagios_send_service_mail-v173.pl Obsolete version 1.7.3, 45162 Bytes

nagios_send_service_mail-v14.pl Obsolete version 1.4.0, 41368 Bytes

nagios_send_service_mail-v13.pl Obsolete version 1.3.0, 31664 Bytes

nagios_send_host_mail.pl

nagios_send_host_mail.pl Latest version 1.8.1, 47328 Bytes, for further documentation see also the manual page

nagios_send_host_mail-v180.pl Obsolete version 1.8.0, 45490 Bytes

nagios_send_host_mail-v173.pl Obsolete version 1.7.3, 43891 Bytes

nagios_send_host_mail-v14.pl Obsolete version 1.4.0, 38330 Bytes

nagios_send_host_mail-v13.pl Obsolete version 1.3.0, 28477 Bytes

For an alternate download location, see also the corresponding github repository.

Notification Script Contributions

NameVer.Description
nagios_send_host_mail_rb.pl1.4The original host notification script v1.4, with the addition of the extra option '-g' to use the Nagios contact groups.
nagios_send_service_mail_rb.pl1.4The original service notification script v1.4, with the extra option '-g' to use the Nagios contact groups.
nagios_send_service_mail_rb.pl1.6The service notification script v1.6 (March 2012) implements various enhancements, such as improved multi-language support, the addition of German and French, enhanced debugging, etc.
Many Thanks to Robert Becht for the contribution.

Below are two example notification definition files:

Similar Solutions, Links and Credits:

If you made cool changes to the scripts, fixed bugs or expanded the functions, feel free to send them in, together with a screenshot, for sharing.

Known Bugs


NOTE: The notification scripts fail on Nagios v4.0 due to a Nagios bug (BugID 0000498). The Nagios environment variables are not exported, and as a result the data handover to the scripts does not work. No notifications go out since no destination e-mail address comes from Nagios. This problem is fixed in Nagios 4.0.3

Support for PNP4Nagios


After initial release of these scripts they became very popular. However many of you did not use the Nagiosgraph package, but the more popular choice PNP4Nagios. The question about adapting these scripts for PNP4Nagios was raised many times, and a lively discussion started. I relented to public pressure :-) I installed PNP4Nagios, built a parallel datafeed into both graphing systems and started to spend quite a few nights over code and testing, receiving invaluable help from Robert Becht. The result is a pair of notification scripts dedicated to PNP4Nagios: pnp4nagios_send host_mail.pl and pnp4nagios_send_service_mail.pl.

nagios_send_service_mail.pl

pnp4n_send_service_mail.pl Latest version 1.8.1, 45474 Bytes, for further documentation see also the manual page

pnp4n_send_service_mail-v180.pl Obsolete version 1.8.0, 43369 Bytes

pnp4n_send_service_mail-v173.pl Obsolete version 1.7.3, 41850 Bytes

pnp4n_send_service_mail-v15.pl Obsolete version 1.5.0, 41533 Bytes

pnp4n_send_service_mail-v14.pl Obsolete version 1.4.0, 40968 Bytes

pnp4n_send_host_mail.pl

pnp4n_send_host_mail.pl Latest version 1.8.1, 44172 Bytes

pnp4n_send_host_mail-v180.pl Obsolete version 1.8.0, 42333 Bytes

pnp4n_send_host_mail-v173.pl Obsolete version 1.7.3, 40778 Bytes

pnp4n_send_host_mail-v15.pl Obsolete version 1.5.0, 38019 Bytes

pnp4n_send_host_mail-v14.pl Obsolete version 1.4.0, 37926 Bytes

Topics:

More Information: