Introduction

cisco-logo

As in many companies, our main network vendor is Cisco, Inc. and we have a wide device and model variety of their switches, routers and security appliances. Cisco network devices are a large part of our device inventory. Their 'health' has a substantial impact on our network security and availability. Without extra tools, there is no single view on their patch and OS version status. Tools like CiscoWorks can help but are significant work to implement and operate. More importantly, their integration into a 'enterprise', vendor-independend patch and OS version monitoring framework requires extra effort. Here is a description on how to achieve the OS version and patch monitoring by leveraging the existing monitoring tool Nagios. Although it currently works for Cisco, the method is expandable for other vendors.

Plugin Design

This tutorial describes one approach of monitoring the patch compliance of network devices. The method is simple: Our network devices are already configured for SNMP monitoring. We are getting the usual performance and availability indicators using ping, load, memory and environment checks (power, fans OK). By leveraging the existing SNMP configuration of Nagios, we can receive the Cisco OS version (IOS, ASA, PIX) through the SNMPv2 MIB's "sysDescr" parameter. This parameter is meant to be:

"A textual description of the entity. This value should include the full name and version identification of the system's hardware type, software operating-system, and networking software." (quote SNMPv2-MIB.txt).

Not many vendors follow this standard closely by implementing a meaningful version string. Fortunately Cisco is doing it; and for their IOS software it has been done in a consistent way that can be parsed. Thank you Cisco for paying attention to such details! By writing a Nagios plugin we can extract and display the OS version in the monitoring console. Now we have a nice, complete and grouped view on what version is deployed. This mode of the plugin is called the "discovery mode".

OS Baseline

Going a step further, we want to take action on OS versions that are vulnerable, obsolete, or otherwise not following our standards for deployment. For that, the plugin implements a blacklist/whitelist mechanism through a configuration file listing the OS versions in 4 pre-defined categories:

Category Description Nagios Response
approved Versions marked 'approved' are versions that are confirmed to be recent,without known vulnerabilities (yet) or otherwise desired by IT networks/management for standardization. OK
obsolete Versions marked 'obsolete' are "End of Life", "End of Maintenance" or otherwise old versions we desire to upgrade. It is marked 'obsolete' if there are no confirmed vulnerabilities (yet). WARNING
med-vuln Versions marked 'med-vuln' are versions who have confirmed vulnerabilities that are either currently not applicable, or rated low to medium with compensations (i.e. ACL's) in place. WARNING
crit-vuln Versions marked 'crit-vuln' are versions who have confirmed vulnerabilities with a high risk for immediate impact such as device down or compromised. Devices should be upgraded ASAP. CRITICAL
unknown Versions not listed as 'approved', 'obsolete' or 'vulnerable' will return as 'unknown'. This is meant as a note to check if this version is OK to run and update the version list accordingly. UNKNOWN

By categorizing our IOS versions in the plugin configuration file, we demonstrate that we "reviewed" our deployment, hopefully by checking the IOS versions against reported vulnerabilities and Cisco best practises. Depending on version standardization and the size of your network environment, there are likely between 10 - 30 versions to check up on. This risk evaluation is a more or less time-consuming, but a necessary task. If completed, we can leverage the exisiting contact and notification setup of Nagios to alert for any changes in our OS version risk evaluation. For example, a new network device gets deployed but it was forgotten to update the IOS to a approved version. Nagios would report it as a unknown version and could send a notification to the networks group. If a new IOS vulnerability is discovered, it can be evaluated against the currently deployed versions and a version could be re-classified from 'approved' to 'vulnerable', immediately highlighting how many devices are impacted and notifying the networks group for remediation. So, here are the implementation details:

Set up and test the device SNMP access from the Nagios host

Although the expectation is that SNMP is already set up on the network devices, here is a quick info on how it might look like for completeness. 

Configure the router side:
--------------------------
Router# conf t
Router(config)# no access-list 12
Router(config)# access-list 12 permit 192.168.103.32
Router(config)# access-list 12 permit 192.168.103.34
Router(config)# snmp-server community SECro ro 12
Router(config)# snmp-server host 192.168.103.34 SECtrap
Router(config)# snmp-server enable traps snmp linkdown linkup coldstart warmstart

Check the nagios side:
----------------------
~ # snmpget -v 1 -c SECro 192.168.203.4 SNMPv2-MIB::sysDescr.0
SNMPv2-MIB::sysDescr.0 = STRING: Cisco Internetwork Operating System Software
IOS (tm) C2950 Software (C2950-I6Q4L2-M), Version 12.1(22)EA9, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2006 by cisco Systems, Inc.
Compiled Fri 01-Dec-06 18:02 by weiliu

Download and install the Nagios plugin

Now we add the new Nagios plugin check_snmp_patchlevel.pl to the plugins in our [nagioshome]/libexec directory. We check (and adjust) the scripts user and access rights, don't forget to update the 'use lib' path for utils.pm. Ready to test it by hand, using the 'discovery' mode: 

susie3:/srv/app/nagios/libexec # ./check_snmp_patchlevel.pl -H 192.168.203.4 -g ios 
-C SECro
IOS Version: 12.1(22)EA9 | Cisco Internetwork Operating System Software
IOS (tm) C2950 Software (C2950-I6Q4L2-M), Version 12.1(22)EA9, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2006 by cisco Systems, Inc.
Compiled Fri 01-Dec-06 18:02 by weiliu

Configure the plugin and service in Nagios ('discovery mode')

susie3:/srv/app/nagios/etc/objects # vi commands.cfg

define command{
  command_name    check_snmp_cisco_ios
  command_line    $USER1$/check_snmp_patchlevel.pl -H $HOSTADDRESS$ -g ios
-C $ARG1$
}

Configuring the Nagios Version Monitoring Service

susie3:/srv/app/nagios/etc/objects # vi patch-services-nw.cfg

##############################################################################
# Define a servicegroup for patch service checks
# All patch service checks will be members of this group
##############################################################################
define servicegroup{
  servicegroup_name        patch-checks-nw     ; The name of the servicegroup
  alias                    NW OS Update Checks ; Long name of the group
}
##############################################################################
# Define the database check template service
##############################################################################
define service{
  name                          generic-patch-nw
  active_checks_enabled         1
  passive_checks_enabled        1
  parallelize_check             1
  obsess_over_service           1
  check_freshness               0
  notifications_enabled         1
  event_handler_enabled         1
  flap_detection_enabled        0
  failure_prediction_enabled    1
  process_perf_data             1
  retain_status_information     1
  retain_nonstatus_information  1
  is_volatile                   0
  check_period                  24x7
  max_check_attempts            4
  normal_check_interval         240            ; check only once every 4 hours
  retry_check_interval          10
  contact_groups                cisco-admins, security-office
  notification_options          u,w,c,r
  notification_interval         1440           ; notify only once per day
  notification_period           24x7
  register                      0
  servicegroups                 patch-checks-nw
}
#############################################################################
# Check Cisco Router IOS version against a config file
##############################################################################
define service {
  use                           generic-patch-nw
  hostgroup			cisco-routers
  name                          check_snmp_ios_router
  service_description           check_snmp_ios_router
  check_command                 check_snmp_cisco_ios!SECro
  notification_period           none
}

susie3:/srv/app/nagios/etc/objects # echo "cfg_file=/srv/app/nagios/etc/
objects/patch-services-nw.cfg" >> /srv/app/nagios/etc/nagios.cfg

susie3:/srv/app/nagios/etc/objects # /etc/init.d/nagios restart
Running configuration check...done.
Stopping nagios: .done.
Starting nagios: done.

Switching from 'discovery mode' to 'compliance mode'

Here we need to configure the check_snmp_patchlevel.cfg IOS version configuration file on the Nagios server. We give th plugin a check by hand before we enable the version lookup with the '-f' option in the Nagios plugin, changing the plugin definition in commands.cfg. 
susie3:/srv/app/nagios/libexec # ./check_snmp_patchlevel.pl -H 192.168.203.4 -g ios
 -C SECro -f  ./check_snmp_patchlevel.cfg
IOS Version: 12.1(22)EA9 obsolete | Remarks: end-of-maintenance date 2008-03-15 Data:
 Cisco Internetwork Operating System Software
IOS (tm) C2950 Software (C2950-I6Q4L2-M), Version 12.1(22)EA9, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2006 by cisco Systems, Inc.
Compiled Fri 01-Dec-06 18:02 by weiliu

Update the Plugin command definition for compliance mode

susie3:/srv/app/nagios/etc/objects # vi commands.cfg

define command{
  command_name    check_snmp_cisco_ios
  command_line    $USER1$/check_snmp_patchlevel.pl -H $HOSTADDRESS$ -g ios
-C $ARG1$ -f $USER1$/check_snmp_patchlevel.cfg
}

susie3:/srv/app/nagios/etc/objects # /etc/init.d/nagios restart
Running configuration check...done.
Stopping nagios: .done.
Starting nagios: done.

Comments

Cisco discontinued all of their PIX series, PIX OS goes out of maintenance on July 28th, 2009. All their OS should be marked obsolete by now, highlighting the need for migration.

Credits, copyrights original scripts etc

Topics:

More Information:

Screenshots

Cisco update check service group Cisco update check service detail Cisco update check service line Cisco update e-mail notification