Bringing ZFS information into SNMP Thomas Stibor GSI Helmholtz Centre for Heavy Ion Research, HPC 27. Januar 2014 What is SNMP? • Simple Network Management Protocol (SNMP) is protocol for network management. • It allows collecting information from switches, printers, linux-boxes, . and also to configure (write access) those. thomas@lxdv65:~>snmpget -v 1 -c public localhost 1.3.6.1.2.1.1.1.0 iso.3.6.1.2.1.1.1.0 = STRING: "Linux lxdv65 3.8.13-tstibor-lxdv65-rev1 #1 SMP Wed May 15 12:32:59 CEST 2013 x86_64" thomas@lxdv65:~>snmpget -v 1 -c public localhost 1.3.6.1.2.1.25.1.4.0 iso.3.6.1.2.1.25.1.4.0 = STRING: "BOOT_IMAGE=/boot/vmlinuz-3.8.13-tstibor-lxdv65-rev1 root=/dev/mapper/vg0-debian ro quiet" What are these strange looking numbers, e.g. 1.3.6.1.2.1.25.1.4.0? • Each Object Identifier (short OID) identifies a variable that can be read or set via SNMP. • OID(s) are organized hierarchically. OID(s) as a Tree (snmpwalk) thomas@lxdv65:~> snmpwalk -c public -v 2c localhost 1.3.6.1.4.1.2021.9.1 iso.3.6.1.4.1.2021.9.1.1.1 = INTEGER: 1 iso.3.6.1.4.1.2021.9.1.1.2 = INTEGER: 2 ... iso.3.6.1.4.1.2021.9.1.1.12 = INTEGER: 12 iso.3.6.1.4.1.2021.9.1.2.1 = STRING: "/" iso.3.6.1.4.1.2021.9.1.2.2 = STRING: "/sys" ... iso.3.6.1.4.1.2021.9.1.2.12 = STRING: "/zfs/pools-deduplication" iso.3.6.1.4.1.2021.9.1.3.1 = STRING: "rootfs" iso.3.6.1.4.1.2021.9.1.3.2 = STRING: "sysfs" ... iso.3.6.1.4.1.2021.9.1.3.5 = STRING: "devpts" iso.3.6.1.4.1.2021.9.1.3.6 = STRING: "tmpfs" 1.3.6.1.4.1.2021.9.1 1 2 3 1 2 . 12 1 2 . 12 1 2 . 6 Human Readable OID(s) Given an OID • What is the semantic meaning of e.g. thomas@lxdv65:~>snmpget -v 1 -c public localhost 1.3.6.1.2.1.25.1.6.0 iso.3.6.1.2.1.25.1.6.0 = Gauge32: 597 • Is there a description giving us more information? thomas@lxdv65:~>snmptranslate -m SNMPv2-MIB 1.3.6.1.2.1.1.1 SNMPv2-MIB::sysDescr thomas@lxdv65:~>snmptranslate -m SNMPv2-MIB -On -Td 1.3.6.1.2.1.1.1 .1.3.6.1.2.1.1.1 sysDescr OBJECT-TYPE -- FROM SNMPv2-MIB -- TEXTUAL CONVENTION DisplayString SYNTAX OCTET STRING (0..255) DISPLAY-HINT "255a" MAX-ACCESS read-only STATUS current DESCRIPTION "A textual description of the entity. This value should include the full name and version identification of the system’s hardware type, software operating-system, and networking software." ::= { iso(1) org(3) dod(6) internet(1) mgmt(2) mib-2(1) system(1) 1 } These information are provided in Management Information Base (short MIB) file(s). Desired ZFS Information bringing into SNMP thomas@lxdv65:~>sudo zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT domov-0 178M 244K 178M 0% 1.00x DEGRADED - domov-1 178M 235K 178M 0% 1.00x ONLINE - domov-2 178M 235K 178M 0% 1.00x ONLINE - domov-3 178M 28.5M 150M 15% 1.00x ONLINE - domov-4 178M 235K 178M 0% 1.00x ONLINE - domov-5 178M 235K 178M 0% 1.00x ONLINE - domov-6 178M 235K 178M 0% 1.00x ONLINE - domov-7 178M 599K 177M 0% 2048.00x ONLINE - thomas@lxdv65:~>sudo zfs get all | grep "avail\|used " domov-0used 163K - Specify MIB file to bring domov-0available 86.4M - domov-1used 157K - the ZFS information in- domov-1available 86.4M - domov-2used 157K - to SNMP. domov-2available 86.4M - domov-3used 18.9M - domov-3available 67.6M - domov-4used 157K - domov-4available 86.4M - domov-5used 157K - domov-5available 86.4M - domov-6used 157K - domov-6available 86.4M - domov-7used 256M - domov-7available 86.2M - ZFS-MIB File ZFS-MIB.txt ZFS-MIB DEFINITIONS ::= BEGIN IMPORTS OBJECT-TYPE, MODULE-IDENTITY, enterprises, Counter64, Integer32 FROM SNMPv2-SMI ... -- -- A brief description and update information about the ZFS-MIB. -- zfs MODULE-IDENTITY LAST-UPDATED "201312190000Z" ORGANIZATION "GSI" CONTACT-INFO "[email protected]" DESCRIPTION "This MIB module describes read-only ZFS information gathered through libzfs. This encompasses the health status, available used and total space, as well as compression and deduplication ratio of pools." REVISION "201312190000Z" DESCRIPTION "Initial revision." ::= { hpc 1 } ZFS-MIB File (cont.) ZFS-MIB.txt ... ZFSUnsigned64 ::= TEXTUAL-CONVENTION DISPLAY-HINT "d" STATUS current DESCRIPTION "A 64 bits unsigned (which doesn’t exist in SMIv2) containing any unsigned 64 bits integer number. It is defined as a Counter64 but doesn’t carry the counter semantic" SYNTAX Counter64 -- We are hosted under GSI OID (2021). gsi OBJECTIDENTIFIER::={enterprises2021} hpc OBJECTIDENTIFIER::={gsi255} poolTable OBJECT-TYPE SYNTAX SEQUENCE OF PoolEntry MAX-ACCESS not-accessible STATUS current DESCRIPTION "ZFS Pool watching information." ::= { zfs 1 } poolEntry OBJECT-TYPE SYNTAX PoolEntry MAX-ACCESS not-accessible STATUS current DESCRIPTION "An entry containing information on a ZFS pool." INDEX { poolIndex } ::= { poolTable 1 } ... ZFS-MIB File (cont.) ZFS-MIB.txt ... PoolEntry ::= SEQUENCE { poolIndex Integer32, --1 poolName DisplayString, -- 2 poolHealth DisplayString, -- 3 poolAvail ZFSUnsigned64, -- 4 poolUsed ZFSUnsigned64, -- 5 poolTotal ZFSUnsigned64, -- 6 poolCompressRatio DisplayString -- 7 poolDedupRatio DisplayString -- 8 } poolIndex OBJECT-TYPE SYNTAX Integer32 (0..255) MAX-ACCESS read-only STATUS current DESCRIPTION "Reference Index for each observed ZFS pool." ::= { poolEntry 1 } poolName OBJECT-TYPE SYNTAX DisplayString (SIZE (0..255)) MAX-ACCESS read-only STATUS current DESCRIPTION "Name of ZFS pool." ::= { poolEntry 2 } ... Inspect our ZFS-MIB File thomas@lxdv65:~>snmptranslate -Tp -IR ZFS-MIB::poolTable +--poolTable(1) | +--poolEntry(1) | Index: poolIndex | +-- -R-- Integer32 poolIndex(1) | Range:0..255 +-- -R-- String poolName(2) | Textual Convention: DisplayString | Size:0..255 +-- -R-- String poolHealth(3) | Textual Convention: DisplayString | Size:0..15 +-- -R-- Counter64 poolAvail(4) | Textual Convention: ZFSUnsigned64 +-- -R-- Counter64 poolUsed(5) | Textual Convention: ZFSUnsigned64 +-- -R-- Counter64 poolTotal(6) | Textual Convention: ZFSUnsigned64 +-- -R-- String poolCompressRatio(7) | Textual Convention: DisplayString | Size:0..15 +-- -R-- String poolDedupRatio(8) Textual Convention: DisplayString Size: 0..15 Inspect our ZFS-MIB File (cont.) thomas@lxdv65:~>snmptranslate -On -Td ZFS-MIB::poolHealth .1.3.6.1.4.1.2021.255.1.1.1.3 poolHealth OBJECT-TYPE -- FROM ZFS-MIB -- TEXTUAL CONVENTION DisplayString SYNTAX OCTET STRING (0..15) DISPLAY-HINT "255a" MAX-ACCESS read-only STATUS current DESCRIPTION "Health status of ZFS pool." ::= { iso(1) org(3) dod(6) internet(1) private(4) enterprises(1) gsi(2021) hpc(255) zfs(1) poolTable(1) poolEntry(1) 3 } • Howto implement a SNMP sub-agent daemon, once we specified our MIB file? Excellent starting point: http://www.net-snmp.org/wiki/index.php/Tutorials From MIB ⇒ C thomas@lxdv65:~>env MIBS="+ZFS-MIB" mib2c -c mib2c.iterate.conf poolTable # poolTable.h poolTable.c /* * Note: this file originally auto-generated by mib2c using * : mib2c.iterate.conf 17821 2009-11-11 09:00:00Z dts12 */ #ifndef POOLTABLE_H #define POOLTABLE_H /* function declarations */ void init_poolTable(void); void initialize_table_poolTable(void); Netsnmp_Node_Handler poolTable_handler; Netsnmp_First_Data_Point poolTable_get_first_data_point; Netsnmp_Next_Data_Point poolTable_get_next_data_point; /* column number definitions for table poolTable */ #define COLUMN_POOLINDEX 1 #define COLUMN_POOLNAME 2 #define COLUMN_POOLHEALTH 3 #define COLUMN_POOLAVAIL 4 #define COLUMN_POOLUSED 5 #define COLUMN_POOLTOTAL 6 #define COLUMN_POOLCOMPRESSRATIO 7 #define COLUMN_POOLDEDUPRATIO 8 #endif /* POOLTABLE_H */ From MIB ⇒ C (cont.) /** Handles requests for the poolTable entries. */ int poolTable_handler(netsnmp_mib_handler *handler, netsnmp_handler_registration *reginfo, netsnmp_agent_request_info *reqinfo, netsnmp_request_info *requests) { netsnmp_request_info *request; netsnmp_table_request_info *table_info; struct poolTable_entry *table_entry; char result[ZFS_MAXPROPLEN]; switch (reqinfo->mode) { case MODE_GET: for (request=requests; request; request=request->next) { table_entry = (struct poolTable_entry *) netsnmp_extract_iterator_context(request); table_info = netsnmp_extract_table_info(request); switch (table_info->colnum) { ... case COLUMN_POOLHEALTH: if ( !table_entry ) { netsnmp_set_request_error(reqinfo, request, SNMP_NOSUCHINSTANCE); continue; } /* Pool health. */ if (get_zpool_prop(libzfs_handle, table_entry->poolName, ZPOOL_PROP_HEALTH, result) == ERROR) netsnmp_set_request_error(reqinfo, request, SNMP_NOSUCHINSTANCE); else { strcpy(table_entry->poolHealth, result); table_entry->poolHealth_len = strlen(result); snmp_set_var_typed_value(request->requestvb, ASN_OCTET_STR, (u_char*)table_entry->poolHealth, table_entry->poolHealth_len); } Ask libzfs (/dev/zfs) for ZFS Information First approach, don’t do that! #define COMMAND_ARCSTATS "/bin/cat /proc/spl/kstat/zfs/arcstats" #define COMMAND_ZPOOL_HEALTH "/usr/local/sbin/zpool list -H -o name,health" #define COMMAND_ZGET_AVAIL_USED "/usr/local/sbin/zfs get -Hpo value used,available" /* Open the command for reading. */ fp = popen(COMMAND_ZPOOL_HEALTH, "r"); if (fp == NULL) { perror("popen" ); return ERROR; } i = 0; while (fgets(line, sizeof(line)-1, fp) != NULL) { line_dup = strdup(line); while ((tok_str = strsep(&line_dup, "\t"))) { if (n_token == 0) { //printf("poolname: %s\n", tok_str); name_temp = strdup(tok_str); } else if (n_token == 1) { tok_str[strlen(tok_str)-1] = ’\0’; /* Remove CR */ health_temp = strdup(tok_str); } else { free_pool_info(pool_info); return ERROR; } n_token = (n_token + 1) % 2; } pool_info[i] = malloc(sizeof(pool_info_t)); strcpy(pool_info[i]->name, name_temp); strcpy(pool_info[i]->health, health_temp); ... Ask libzfs (/dev/zfs) for