Event Management for Zenoss Core 4 January 2013 Jane Curry Skills 1st Ltd www.skills-1st.co.uk Jane Curry Skills 1st Ltd 2 Cedar Chase Taplow Maidenhead SL6 0EU 01628 782565 [email protected] www.skills-1st.co.uk Synopsis This paper is intended as an intermediate-level discussion of the Zenoss event system in Zenoss Core 4. The event architecture has changed dramatically in Zenoss 4 from previous versions. It is assumed that the reader is already familiar with the Zenoss Event Console and with basic navigation around the Zenoss Graphical User Interface (GUI). It looks in some detail at the architecture behind the Zenoss event system ± the daemons and how they are inter-related ± and it looks at the structure of a Zenoss event and the event life cycle. Zenoss can receive events from many sources in addition to Zenoss itself. Events from Windows, Unix syslogs and Simple Networks Management Protocol (SNMP) TRAPs are all examined in detail. The process by which an incoming event is converted into a particular Zenoss event is known as event mapping and there are a number of different possible techniques for performing that conversion. These will all be explored along with the creation of new event classes. Once an event has been received, classified and stored by Zenoss, automation may be required. Alerting to users by email and page is discussed, as are background actions to run commands or generate TRAPs. Logging and debugging techniques are discussed in some details as is the JSON API for extracting data out of Zenoss. This paper was written using Zenoss Core 4.2.3 The paper is a companion text to the Zenoss 4 Event Management Workshop. Notations Throughout this paper, text to by typed, file names and menu options to be selected, are highlighted by italics; important points to take note of are shown in bold. Points of particular note are highlighted by an icon. 2 Event Management for Zenoss Core 4 © Skills 1st Ltd 1 February 2013 Table of Contents 1 Introduction..........................................................................................................................6 2 Zenoss event architecture....................................................................................................6 2.1 Event Console...............................................................................................................6 2.2 Event Manager settings.............................................................................................10 2.3 Event database tables ...............................................................................................11 2.3.1 Zenoss 2.x and 3.x...............................................................................................11 2.3.2 Zenoss 4................................................................................................................14 2.4 New event daemons....................................................................................................20 2.4.1 RabbitMQ.............................................................................................................20 2.4.2 zeneventserver.....................................................................................................22 2.4.3 zeneventd.............................................................................................................22 2.4.4 zenactiond ...........................................................................................................23 2.4.5 memcached...........................................................................................................23 2.5 Other database-related changes in Zenoss 4............................................................24 2.6 Event life cycle............................................................................................................25 2.6.1 Event generation.................................................................................................27 2.6.2 Application of device context..............................................................................29 2.6.3 Event class mapping...........................................................................................29 2.6.4 Application of event context...............................................................................30 2.6.5 Event transforms.................................................................................................30 2.6.6 Database insertions and de-duplication............................................................31 2.6.7 Resolution............................................................................................................32 2.6.8 Ageing and archiving..........................................................................................34 3 Events generated by Zenoss..............................................................................................34 3.1 zenping........................................................................................................................35 3.2 zenstatus.....................................................................................................................36 3.3 zenprocess...................................................................................................................36 3.4 zenwin.........................................................................................................................37 3.5 zenwinperf...................................................................................................................37 3.6 zenperfsnmp................................................................................................................37 3.7 zencommand...............................................................................................................38 4 Syslog events......................................................................................................................38 4.1 Configuring syslog.conf .............................................................................................39 4.2 Zenoss processing of syslog messages.......................................................................40 5 Zenoss processing of Windows event logs.........................................................................48 5.1 Management using the WMI protocol.......................................................................48 5.2 Management of Windows systems using syslog.......................................................51 6 Event Mapping...................................................................................................................51 6.1 Working with event classes and event mappings....................................................52 6.1.1 Generating test events........................................................................................54 6.2 Regex in event mappings...........................................................................................55 1 February 2013 Event Management for Zenoss Core 4 © Skills 1st Ltd 3 6.3 Rules in event mappings............................................................................................57 6.4 Other elements of event mappings ...........................................................................58 7 Event transforms...............................................................................................................58 7.1 Different ways to apply transforms...........................................................................59 7.2 Understanding fields available for event processing...............................................60 7.2.1 Event Proxies.......................................................................................................63 7.2.2 Event Details.......................................................................................................66 7.3 Transform examples...................................................................................................68 7.3.1 Combining user defined fields from Regex with transform.............................68 7.3.2 Applying event and device context in relation to transforms..........................69 8 Testing and debugging aids..............................................................................................71 8.1 Log files.......................................................................................................................71 8.1.1 zeneventd.log.......................................................................................................71 8.1.2 zeneventserver.log...............................................................................................72 8.1.3 Other log files......................................................................................................75 8.2 Using zendmd to run Python commands..................................................................75 8.2.1 Referencing an existing Zenoss event for use in zendmd.................................75 8.2.2 Using zendmd to understand attributes for an EventSummaryProxy...........79 8.3 Using the Python debugger in transforms................................................................83 9 Zenoss and SNMP..............................................................................................................87 9.1 SNMP introduction.....................................................................................................87 9.2 SNMP on Linux systems............................................................................................88