#CLMel Coding 1003 – Useful Python Libraries, Frameworks, and Features to Master Kareem Iskander– Developer Advocate DevNet @Kareem_Isk DEVNET-2895 Cisco Webex Teams Questions? Use Cisco Webex Teams (formerly Cisco Spark) to chat with the speaker after the session How 1 Open the Cisco Events Mobile App 2 Find your desired session in the “Session Scheduler” 3 Click “Join the Discussion” 4 Install Webex Teams or go directly to the team space 5 Enter messages/questions in the team space cs.co/ciscolivebot#DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3 Agenda • Introduction • Manipulating XML in Python • Working JSON in Python • Dealing with YAML in Python • Parsing CSV in Python • HTTP Calls with Requests • YANG Model Data with NETCONF and ncclient • Conclusion DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4 Manipulating Data of All Formats • XML - xmltodict • CSV • pip install xmltodict • import csv import xmltodict • JSON • import json • YAML - PyYAML • pip install PyYAML import yaml #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5 Demystify XML XML- eXtensible Markup Language. <?xml version="1.0" encoding="UTF-8" ?> <interface xmlns="ietf-interfaces"> <name>GigabitEthernet2</name> A human readable data structure that <description> Wide Area Network applications use to store, </description> transfer, and read data. <enabled>true</enabled> <ipv4> <address> <ip>172.16.0.2</ip> <netmask>255.255.255.0</netmask> </address> </ipv4> </interface> #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 7 XML • Designed for the Internet <?xml version="1.0" encoding="UTF-8" ?> • Schema or namespace defines data <interface xmlns="ietf-interfaces"> model <name>GigabitEthernet2</name> <description> • <tags></tags> surround elements Wide Area Network for structure and layout </description> <enabled>true</enabled> • Key/Value representation <ipv4> • <key>value</key> <address> <ip>172.16.0.2</ip> • Whitespace not significant <netmask>255.255.255.0</netmask> </address> </ipv4> </interface> #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 8 XML Object • A related set of data surrounded by <?xml version="1.0" encoding="UTF-8" <tags></tags> ?> <interface xmlns="ietf-interfaces"> •An object can contain other <name>GigabitEthernet2</name> <description> objects or data entries Wide Area Network </description> •<key>value</key> <enabled>true</enabled> contained within the object <ipv4> <address> tags <ip>172.16.0.2</ip> <netmask>255.255.255.0</netmask> </address> </ipv4> </interface> #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 9 XML List . List of data <?xml version="1.0" encoding="UTF-8" ?> <addresses> . Can be composed of XML objects <ip>172.16.0.2</ip> . Repeated instances of <tags></tags> <netmask>255.255.255.0</netmask> </addresses> for each element <addresses> <ip>172.16.0.3</ip> <netmask>255.255.255.0</netmask> </addresses> <addresses> <ip>172.16.0.4</ip> <netmask>255.255.255.0</netmask> </addresses> #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10 Treat XML like Python Dictionaries with xmltodict • Easily work with XML data • Convert from XML -> Dict* and back • xmltodict.parse(xml_data) • xmltodict.unparse(dict) • Python includes a native Markup (html/xml) interfaces as well • More powerful, but also more complex * Technically to an OrderedDict https://pypi.python.org/pypi/xmltodict netprog_basics/programming_fundamentals/python_part_3/xml_example.xml #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 11 Breakdown JSON JSON - JavaScript Object Notation { "ietf-interfaces:interface": { "name": "GigabitEthernet2", "description": "Wide Area Network", A human readable data "enabled": true, "ietf-ip:ipv4": { structure that applications "address": [ { use to store, transfer, and read "ip": "172.16.0.2", data. "netmask": "255.255.255.0" } ] } } } #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 13 JSON { • A data-interchange text format "ietf-interfaces:interface": { "name": "GigabitEthernet2", • Notated with {} for objects, [] "description": "Wide Area for arrays Network", "enabled": true, • Key/Value representation "ietf-ip:ipv4": { "address": [ • "key": value { "ip": "172.16.0.2", • Whitespace not significant "netmask": "255.255.255.0" } ] } } } #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 14 JSON Object { • Data surrounded by { } "ietf-interfaces:interface": { "name": "GigabitEthernet2", • An object can contain other "description": "Wide Area objects or data entries Network", "enabled": true, • Key/Value set separated by "ietf-ip:ipv4": { comma "address": [ { • No comma at the end! "ip": "172.16.0.2", "netmask": "255.255.255.0" } ] } } } #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 15 JSON List { . List of data "addresses": [ . Can be composed of JSON { "ip": "172.16.0.2", objects "netmask": "255.255.255.0" }, . Notated with brackets { "ip": "172.16.0.3", . Comma Separated "netmask": "255.255.255.0" }, { "ip": "172.16.0.4", "netmask": "255.255.255.0" } ] } #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 16 To JSON and back again with json • JSON and Python go together like peanut butter and jelly • json.loads(json_data) • json.dumps(object) • JSON Objects convert to Dictionaries • JSON Arrays convert to Lists https://docs.python.org/3/library/json.html netprog_basics/programming_fundamentals/python_part_3/json_example.json #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 17 Simplify YAML YAML - “YAML Ain’t Markup Language” --- ietf-interfaces:interface: name: GigabitEthernet2 description: Wide Area Network enabled: true A human readable data ietf-ip:ipv4: address: structure that applications - ip: 172.16.0.2 netmask: 255.255.255.0 use to store, transfer, and read data. #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 19 YAML --- • Minimalist format commonly ietf-interfaces:interface: used for configuration files name: GigabitEthernet2 description: Wide Area Network • Whitespace indentation defines enabled: true structure ietf-ip:ipv4: address: • No commas - ip: 172.16.0.2 netmask: 255.255.255.0 • Key/Value representation • key: value #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 20 YAML Object --- • Related set of data at the ietf-interfaces:interface: common indentation level name: GigabitEthernet2 under name description: Wide Area Network enabled: true ietf-ip:ipv4: • An object can contain other address: objects or data entries - ip: 172.16.0.2 netmask: 255.255.255.0 • key: value pairs left aligned #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 21 YAML List --- . List of data addresses: . Can be composed of YAML - ip: 172.16.0.2 netmask: 255.255.255.0 objects - ip: 172.16.0.3 netmask: 255.255.255.0 . Uses “-” character to indicate - ip: 172.16.0.4 a list element netmask: 255.255.255.0 #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 22 YAML? Yep, Python Can Do That Too! • Easily convert a YAML file to a Python Object • yaml.load(yaml_data) • yaml.dump(object) • YAML Objects become Dictionaries • YAML Lists become Lists https://pypi.python.org/pypi/PyYAML/3.12 netprog_basics/programming_fundamentals/python_part_3/yaml_example.yaml #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 23 CSV, just for fun Import Spreadsheets and Data with csv • Treat CSV data as lists • csv.reader(file_object) • Efficiently processes large files without memory issues • Options for header rows and different formats https://docs.python.org/3/library/csv.html netprog_basics/programming_fundamentals/python_part_3/csv_example.csv #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 25 Access Different APIs Easily • REST APIs – requests • pip install requests import requests • NETCONF – ncclient • pip install ncclient import ncclient • Network CLI – netmiko • pip install netmiko import netmiko #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 26 HTTP Calls Make HTTP Calls with Ease using requests • Full HTTP Client • Simplifies authentication, headers, and response tracking • Great for REST API calls, or any HTTP request http://docs.python-requests.org #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 28 YANG Model Data YANG Model Data with NETCONF and ncclient • Full NETCONF Manager (ie client) implementation in Python • See later presentation on NETCONF details • Handles all details including authentication, RPC, and operations • Deals in raw XML https://ncclient.readthedocs.io netprog_basics/programming_fundamentals/python_part_3/api_ncclient_example.py #CLMel DEVNET-2895 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 30 CLI access For When CLI is the Only Option – netmiko • If no other API is available… • Builds on paramiko library for SSH connectivity • Support for a range of vendors network devices and operating systems • Send and receive clear text • Post processing of data will be key https://github.com/ktbyers/netmiko