Cisco ASR 9000 Series Aggregation Services Router System Monitoring Configuration Guide, Release 6.1.x First Published: 2016-11-14 Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 527-0883 THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California. NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental. All printed copies and duplicate soft copies of this document are considered uncontrolled. See the current online version for the latest version. Cisco has more than 200 offices worldwide. Addresses and phone numbers are listed on the Cisco website at www.cisco.com/go/offices. Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com go trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1721R) © 2016 Cisco Systems, Inc. All rights reserved. CONTENTS PREFACE Preface xv Changes to This Document xv Communications, Services, and Additional Information xv CHAPTER 1 New and Changed System Monitoring Features 1 New and Changed Information 1 CHAPTER 2 Implementing and Monitoring Alarms and Alarm Log Correlation 3 Prerequisites for Implementing and Monitoring Alarms and Alarm Log Correlation 4 Information About Implementing Alarms and Alarm Log Correlation 4 Alarm Logging and Debugging Event Management System 4 Correlator 5 System Logging Process 5 Alarm Logger 6 Logging Correlation 6 Correlation Rules 6 Types of Correlation 7 Application of Rules and Rule Sets 7 Root Message and Correlated Messages 7 Alarm Severity Level and Filtering 7 Bistate Alarms 8 Capacity Threshold Setting for Alarms 8 Hierarchical Correlation 8 Context Correlation Flag 9 Duration Timeout Flags 10 Reparent Flag 10 Cisco ASR 9000 Series Aggregation Services Router System Monitoring Configuration Guide, Release 6.1.x iii Contents Reissue Nonbistate Flag 10 Internal Rules 10 SNMP Alarm Correlation 10 How to Implement and Monitor Alarm Management and Logging Correlation 11 Configuring Logging Correlation Rules 11 Configuring Logging Correlation Rule Sets 12 Configuring Root-cause and Non-root-cause Alarms 13 Configuring Hierarchical Correlation Rule Flags 14 Applying Logging Correlation Rules 15 Applying Logging Correlation Rule Sets 17 Modifying Logging Events Buffer Settings 18 Modifying Logging Correlator Buffer Settings 20 Displaying Alarms by Severity and Severity Range 21 Displaying Alarms According to a Time Stamp Range 23 Displaying Alarms According to Message Group and Message Code 24 Displaying Alarms According to a First and Last Range 25 Displaying Alarms by Location 25 Displaying Alarms by Event Record ID 26 Displaying the Logging Correlation Buffer Size, Messages, and Rules 27 Clearing Alarm Event Records and Resetting Bistate Alarms 28 Defining SNMP Correlation Buffer Size 30 Defining SNMP Rulesets 31 Configuring SNMP Correlation Rules 31 Applying SNMP Correlation Rules 32 Applying SNMP Correlation Ruleset 33 Asynchronous Syslog Communication 33 Configuration Examples for Alarm Management and Logging Correlation 34 Increasing the Severity Level for Alarm Filtering to Display Fewer Events and Modifying the Alarm Buffer Size and Capacity Threshold: Example 34 Configuring a Nonstateful Correlation Rule to Permanently Suppress Node Status Messages: Example 34 Configuring a Stateful Correlation Rule for LINK UPDOWN and SONET ALARM Alarms: Example 36 Additional References 38 Cisco ASR 9000 Series Aggregation Services Router System Monitoring Configuration Guide, Release 6.1.x iv Contents CHAPTER 3 Configuring and Managing Embedded Event Manager Policies 41 Prerequisites for Configuring and Managing Embedded Event Manager Policies 42 Information About Configuring and Managing Embedded Event Manager Policies 42 Event Management 42 System Event Detection 42 System Event Processing 43 Embedded Event Manager Management Policies 43 Embedded Event Manager Scripts and the Scripting Interface (Tcl) 43 Script Language 45 Regular Embedded Event Manager Scripts 45 Embedded Event Manager Callback Scripts 45 Embedded Event Manager Policy Tcl Command Extension Categories 46 Cisco File Naming Convention for Embedded Event Manager 46 Embedded Event Manager Built-in Actions 47 Application-specific Embedded Event Management 48 Event Detection and Recovery 48 General Flow of EEM Event Detection and Recovery 49 System Manager Event Detector 49 Timer Services Event Detector 50 Syslog Event Detector 50 None Event Detector 51 Watchdog System Monitor Event Detector 51 Distributed Event Detectors 52 Embedded Event Manager Event Scheduling and Notification 52 Reliability Statistics 53 Hardware Card Reliability Metric Data 53 Process Reliability Metric Data 53 How to Configure and Manage Embedded Event Manager Policies 54 Configuring Environmental Variables 54 Environment Variables 54 Registering Embedded Event Manager Policies 55 Embedded Event Manager Policies 56 How to Write Embedded Event Manager Policies Using Tcl 58 Cisco ASR 9000 Series Aggregation Services Router System Monitoring Configuration Guide, Release 6.1.x v Contents Registering and Defining an EEM Tcl Script 58 Displaying EEM Registered Policies 60 Unregistering EEM Policies 60 Suspending EEM Policy Execution 61 Managing EEM Policies 62 Displaying Software Modularity Process Reliability Metrics Using EEM 62 Sample EEM Policies 63 Programming EEM Policies with Tcl 64 Creating an EEM User Tcl Library Index 72 Creating an EEM User Tcl Package Index 75 Configuration Examples for Event Management Policies 78 Environmental Variables Configuration: Example 78 User-Defined Embedded Event Manager Policy Registration: Example 78 Display Available Policies: Example 78 Display Embedded Event Manager Process: Example 78 Configuration Examples for Writing Embedded Event Manager Policies Using Tcl 79 EEM Event Detector Demo: Example 79 EEM Sample Policy Descriptions 80 Event Manager Environment Variables for the Sample Policies 80 Registration of Some EEM Policies 82 Basic Configuration Details for All Sample Policies 82 Using the Sample Policies 82 Programming Policies with Tcl: Sample Scripts Example 84 tm_cli_cmd.tcl Sample Policy 85 sl_intf_down.tcl Sample Policy 87 Tracing Tcl set Command Operations: Example 89 Additional References 89 Embedded Event Manager Policy Tcl Command Extension Reference 91 Embedded Event Manager Event Registration Tcl Command Extensions 91 event_register_appl 91 event_register_cli 92 event_register_config 93 event_register_counter 94 event_register_hardware 95 Cisco ASR 9000 Series Aggregation Services Router System Monitoring Configuration Guide, Release 6.1.x vi Contents event_register_none 97 event_register_oir 97 event_register_process 98 event_register_snmp 99 event_register_snmp_notification 101 event_register_stat 102 event_register_syslog 104 event_register_timer 106 event_register_timer_subscriber 109 event_register_track 110 event_register_wdsysmon 111 Embedded Event Manager Event Information Tcl Command Extension 115 event_reqinfo 115 event_reqinfo_multi 129 Embedded Event Manager Event Publish Tcl Command Extension 130 event_publish appl 130 Embedded Event Manager Multiple Event Support Tcl Command Extensions 132 Attribute 132 Correlate 133 Trigger 133 Embedded Event Manager Action Tcl Command Extensions 134 action_process 134 action_program 135 action_script 136 action_setver_prior 137 action_setnode