Introduction
One of the core functions in a availability monitoring system is the ability to send out status alerts and notifications. Nagios only comes with a a set of 2 default notifications through e-mails: notify-host-by-email and notify-service-by-email. These notifications provide a minimalistic, barebone structure for sending a simple e-mail alert to defined contacts when problems occur (example screenshot below).
Notification Requirements
However, this minimal setup falls short and the Nagios-builtin format is not presentable to customers. Notification requirements in large companies are diverse and complex: Below are the critical ones I have received:
- Notifications should support the use of the (cc:) field.
- Notifications should include Nagios comments and author information, if given
- Notifications should support different languages, based on the contact
- Notifications should provide URL's back to Nagios, color-highlighting and the integration of logos
The reasons behind these requirements are as varied as todays IT organisations are:
- Some notifications that go to customers or are escalated to management need to look "presentable and professional" - means pretty
- The inclusion of Nagios GUI URL's linking back to the Nagios event speeds up the troubleshooting and follow-up by the IT support teams
- Some notifications go out to outsourced IT service providers and should support their native language, the inclusion of additional information (such as contract number) and the exclusion of other details (such as device IP)
- Some notifications are handled by endusers e-mail processing rules and must be plaintext or addressed through other address fields then (to:)
Thanks to Nagios flexibility and documentation, we are able to meet all requirements with a little extra work (click example screenshots below to enlarge).
Review of the initial Notification Setup
The initial Nagios e-mail notification definition is in
# 'notify-host-by-email' command definition
define command{
command_name notify-host-by-email
command_line /usr/bin/printf "%b" "***** Nagios *****\n\n\
Notification Type: $NOTIFICATIONTYPE$\n\
Host: $HOSTNAME$\n\
State: $HOSTSTATE$\n\
Address: $HOSTADDRESS$\n\
Info: $HOSTOUTPUT$\n\n\
Date/Time: $LONGDATETIME$\n" \
| /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" \
$CONTACTEMAIL$
}
# 'notify-service-by-email' command definition
define command{
command_name notify-service-by-email
command_line /usr/bin/printf "%b" "***** Nagios *****\n\n\
Notification Type: $NOTIFICATIONTYPE$\n\n\
Service: $SERVICEDESC$\n\
Host: $HOSTALIAS$\n\
Address: $HOSTADDRESS$\n\
State: $SERVICESTATE$\n\n\
Date/Time: $LONGDATETIME$\n\n\
Additional Info:\n\n\
$SERVICEOUTPUT$" \
| /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" \
$CONTACTEMAIL$
}
First modifications
Before we look at tweaking the e-mail notification style, I make 3 basic changes to the overall setup.
- I move the notification definition from the original commands.cfg file into its own dedicated configuration file called notifications.cfg.
- I create a additional configuration file no-notification.cfg with identical definition names, but a command_line containing only a sleep 1 command (example definition below). This file is commented out in nagios.cfg and will be enabled a few times a year when the Nagios system is patched, rebooted or otherwise in maintenance. This simple measure avoids flooding our customers with false notifications when Nagios is brought up. After a longer downtime, it could take a while until services are re-checked and we learned to always do a sanity health check on the web GUI before we re-enable notifications.
- Lastly, I created 2 external Perl scripts responsible for sending out mail in place of the original printf/mailx definition
# 'notify-service-by-email' command definition
define command{
command_name notify-host-by-email
command_line /bin/sleep 1
}
With these 2 newly developed scripts (nagios_send_host_mail.pl and nagios_send_service_mail.pl, one for host notifications and a separate for service notifications), it is possible to control through flags which format the email should be sent in, which language should be used and what information should be handed off for mailing out. Below I describe example setups that addressed my clients requirements.
Implementing carbon copy cc: notifications
One customer requested to receive Nagios notifications for some contacts to be defined as a carbon copy (cc:), instead of the standard e-mail recipient list in the (to:) field. The normal way of e-mail address hand-off is through the Nagios $CONTACTEMAIL$ macro definition. Since this macro is already used for the (to:) field, We used the first of the six additionally available contact macros addressx. I defined address1 to hold the (cc:) e-mail address list.
Lets look at an example: We want to send alerts in plain text format to a mailing list support@frank4dd.com using English, and keep two others, acct-mgr@frank4dd.com and public@frank4dd.com on (cc:).
define contact{
contact_name frank4dd-support ; Short name of user
use generic-contact ; use generic template defaults
alias Frank4DD support team ; Full name of user
email support@frank4dd.com ; expands into the (to:) field
address1 acctmgr@frank4dd.com, public@frank4dd.com ; sending alerts as (cc:)
host_notification_commands host-email-text-en ; host notification definition
}
define contactgroup{
contactgroup_name frank4dd-hosting
alias Web hosting team for Frank4DD.com
members frank4dd-support
}
Now we can add the $CONTACTADDRESS1$ macro into the notification definition located in notification.cfg to receive the (cc:) address list we placed in contacts.cfg.
# 'host-email-text-en' command definition
# sends plaintext e-mails in English
define command{
command_name host-email-text-en
command_line /srv/app/nagios/bin/nagios_send_host_mail.pl -c "$CONTACTADDRESS1$"
}
Example:
Implementing HTML, links, comments and a second language
Here is a service notification example that takes advantage of the scripts ability to determine if comments have been entered in Nagios, and sending the notification in HTML format using Japanese language for the alert field descriptions. First, we set the definition format in the users contact settings to be HTML, use URL's to Nagios GUI for intranet access, and set the language to be Japanese.
define contact
contact_name frank4dd-admin
use generic-contact
alias Frank4DD support team
email public@frank4dd.com
host_notification_commands host-email-html-int-jp
service_notification_commands service-email-html-int-jp
}
Now we look at the corresponding notification definition in notification.cfg. The definition uses the format argument -f html, -u (include URL's) and the language argument -l jp. For more information about the arguments, see the scripts manual page for nagios_send_service_mail.pl.
# 'service-email-html-int-jp' command definition
# sends HTML e-mails in Japanese and includes Nagios URL for Intranet access
define command{
command_name service-email-html-int-jp
command_line /srv/app/nagios/etc/objects/nagios_send_service_mail.pl \
-c "$CONTACTADDRESS1$" \
-f html -u -l jp
}
Example:
Adding images by using the multipart S/MIME e-mail format
Here is a service notification example that sends the e-mail in multipart S/MIME with a logo image displayed inline within the HTML code. The example also adds in the company name using the -p argument. Currently, the logo image is defined in the script as it does not change for me. It can be changed by creating a uuencoded version of a different logo image, instructions are inside the script. In a future version, it could be implemented to be picked up from a filename using a script argument if necessary.
define contact
contact_name frank4dd-admin
use generic-contact
alias Frank4DD support team
email public@frank4dd.com
host_notification_commands host-email-multi-int-en
service_notification_commands service-email-multi-int-en
}
Now we look at the corresponding notification definition in notification.cfg. The definition uses the format argument -f multi, -u (include URL's) and ommiting language argument -l sets the language to English. For more information about the arguments, see the scripts manual page for nagios_send_service_mail.pl.
# 'service-email-multi-int-en' command definition
# sends HTML e-mails in English, includes Nagios URL and inline logo
define command{
command_name service-email-multi-int-en
command_line /srv/app/nagios/etc/objects/nagios_send_service_mail.pl \
-p "ACME Corporation, London Branch" \
-c "$CONTACTADDRESS1$" \
-f multi -u
}
Example:
Adding the matching Nagiosgraph performance image, if available
Since version v1.4, the notification scripts are able to include the related performance graph image into the message. This greatly enhances the information provided inside the alert e-mail. However, this graph generation is dependend on the graph generation package used. It has been developed with Nagiosgraph v0.9.1, which I am using. The scripts work fine even with Nagiosgraph not being available, simply using the notification options 'text', 'html' and 'multi', leaving out the 'graph' option.
# 'service-email-graph-int-en' command definition, sends
# HTML e-mails in English, includes Nagios URL, inline logo and graph
define command{
command_name service-email-graph-int-en
command_line /srv/app/nagios/etc/objects/nagios_send_service_mail.pl \
-p "ACME Corporation, London Branch" \
-c "$CONTACTADDRESS1$" \
-f graph -u
}
Example:
Downloads and Links
Nagios host and service notifications are handled through a dedicated script for each case, resulting in two scripts to download and install. The set below can provide embedded graph images from the popular nagiosgraph package (e.g. nagiosgraph 0.91).
nagios_send_service_mail.pl
nagios_send_service_mail.pl Latest version 1.8.1, 48814 Bytes, for further documentation see also the manual page
nagios_send_service_mail-v180.pl Obsolete version 1.8.0, 46712 Bytes
nagios_send_service_mail-v173.pl Obsolete version 1.7.3, 45162 Bytes
nagios_send_service_mail-v14.pl Obsolete version 1.4.0, 41368 Bytes
nagios_send_service_mail-v13.pl Obsolete version 1.3.0, 31664 Bytes
nagios_send_host_mail.pl
nagios_send_host_mail.pl Latest version 1.8.1, 47328 Bytes, for further documentation see also the manual page
nagios_send_host_mail-v180.pl Obsolete version 1.8.0, 45490 Bytes
nagios_send_host_mail-v173.pl Obsolete version 1.7.3, 43891 Bytes
nagios_send_host_mail-v14.pl Obsolete version 1.4.0, 38330 Bytes
nagios_send_host_mail-v13.pl Obsolete version 1.3.0, 28477 Bytes
For an alternate download location, see also the corresponding github repository.
Notification Script Contributions
Name | Ver. | Description |
---|---|---|
nagios_send_host_mail_rb.pl | 1.4 | The original host notification script v1.4, with the addition of the extra option '-g' to use the Nagios contact groups. |
nagios_send_service_mail_rb.pl | 1.4 | The original service notification script v1.4, with the extra option '-g' to use the Nagios contact groups. |
nagios_send_service_mail_rb.pl | 1.6 | The service notification script v1.6 (March 2012) implements various enhancements, such as improved multi-language support, the addition of German and French, enhanced debugging, etc. |
Many Thanks to Robert Becht for the contribution. |
Below are two example notification definition files:
- An example notification definition file is here notification.cfg
- An example no-notification definition file is here no-notification.cfg
Similar Solutions, Links and Credits:
- Further Nagios documentation is available here http://nagios.fm4dd.com/docs/en/
- Nagios and the Nagios community can be found here http://www.nagios.org
- A similar solution, written in PHP is here http://www.bergercity.de/linux/nagios-notification-mail/
- And another one written as a shell script here http://www.monitoringexchange.org/
If you made cool changes to the scripts, fixed bugs or expanded the functions, feel free to send them in, together with a screenshot, for sharing.
Known Bugs
NOTE: The notification scripts fail on Nagios v4.0 due to a Nagios bug (BugID 0000498). The Nagios environment variables are not exported, and as a result the data handover to the scripts does not work. No notifications go out since no destination e-mail address comes from Nagios. This problem is fixed in Nagios 4.0.3
Support for PNP4Nagios
After initial release of these scripts they became very popular. However many of you did not use the Nagiosgraph package, but the more popular choice PNP4Nagios. The question about adapting these scripts for PNP4Nagios was raised many times, and a lively discussion started. I relented to public pressure :-) I installed PNP4Nagios, built a parallel datafeed into both graphing systems and started to spend quite a few nights over code and testing, receiving invaluable help from Robert Becht. The result is a pair of notification scripts dedicated to PNP4Nagios: pnp4nagios_send host_mail.pl and pnp4nagios_send_service_mail.pl.
nagios_send_service_mail.pl
pnp4n_send_service_mail.pl Latest version 1.8.1, 45474 Bytes, for further documentation see also the manual page
pnp4n_send_service_mail-v180.pl Obsolete version 1.8.0, 43369 Bytes
pnp4n_send_service_mail-v173.pl Obsolete version 1.7.3, 41850 Bytes
pnp4n_send_service_mail-v15.pl Obsolete version 1.5.0, 41533 Bytes
pnp4n_send_service_mail-v14.pl Obsolete version 1.4.0, 40968 Bytes
pnp4n_send_host_mail.pl
pnp4n_send_host_mail.pl Latest version 1.8.1, 44172 Bytes
pnp4n_send_host_mail-v180.pl Obsolete version 1.8.0, 42333 Bytes
pnp4n_send_host_mail-v173.pl Obsolete version 1.7.3, 40778 Bytes
pnp4n_send_host_mail-v15.pl Obsolete version 1.5.0, 38019 Bytes
pnp4n_send_host_mail-v14.pl Obsolete version 1.4.0, 37926 Bytes