ZenPack Development without a running Zenoss
Prerequisites Setup pip and common pip packages Clone common github repo dependencies Make changes to certain git repos zenoss-protocols Zope zenoss-prodbin Virtual Environment (Optional) Create virtual environment Install python packages Zenoss is dependent on Modify requirements.txt Developing a ZenPack Creating the ZenPack files and folders Running the ZenPack with a basic unit test Building the ZenPack Moving these test and build steps into scripts Running the ZenPack with advanced unit tests Standalone unit test Integration unit test
The following are instructions on how to setup an environment that allows you to develop a ZenPack locally
Prerequisites
The following documentation assumes the following is already setup / available
OS Linux (like Cent OS 7 which is what was used in developing these instructions) a user with sudo privs python 2.7 git
Here's an example of checking your system for the prerequisites
[zdeveloper@localhost ~]$ python --version Python 2.7.5 [zdeveloper@localhost ~]$ git --version git version 1.8.3.1
Install binaries
Install the following binaries (if not already installed). We install a subset (we avoid mariadb-devel-5.5.37 and openldap-devel ) from what is listed in Dockerfile.in at https://github.com/zenoss/zenoss-py-deps
python-devel python-pip (NOTE: Pip is part of Extra Packages for Enterprise Linux (EPEL), which is a community repository of non-standard packages for the RHEL distribution. You may have to set that up in your package manager before installing pip) gcc make libxml2-devel libxslt-devel libffi-devel protobuf-compiler
NOTE: On centos we used the package manager yum (modified output showing these are already installed) Example listing of installed binaries
[zdeveloper@localhost ~]$ sudo yum install python-devel ... Package python-devel-2.7.5-69.el7_5.x86_64 already installed and latest version [zdeveloper@localhost ~]$ sudo yum install python-pip ... Package python2-pip-8.1.2-6.el7.noarch already installed and latest version [zdeveloper@localhost ~]$ sudo yum install gcc ... Package gcc-4.8.5-28.el7_5.1.x86_64 already installed and latest version [zdeveloper@localhost ~]$ sudo yum install make ... Package 1:make-3.82-23.el7.x86_64 already installed and latest version [zdeveloper@localhost ~]$ sudo yum install libxml2-devel ... Package libxml2-devel-2.9.1-6.el7_2.3.x86_64 already installed and latest version [zdeveloper@localhost ~]$ sudo yum install libxslt-devel ... Package libxslt-devel-1.1.28-5.el7.x86_64 already installed and latest version [zdeveloper@localhost ~]$ sudo yum install libffi-devel ... Package libffi-devel-3.0.13-18.el7.x86_64 already installed and latest version
Setup pip and common pip packages
Still using Dockerfile.in at https://github.com/zenoss/zenoss-py-deps as a reference
Change the version of pip to 9.0.1 (ignore that it's an old version)
[zdeveloper@localhost ~]$ pip install --upgrade pip==9.0.1 Requirement already up-to-date: pip==9.0.1 in /usr/lib/python2.7/site-packages You are using pip version 9.0.1, however version 18.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.
Install the following common python packages using pip
wheel virtualenv setuptools [zdeveloper@localhost ~]$ pip install wheel Requirement already satisfied: wheel in /usr/lib/python2.7/site-packages [zdeveloper@localhost ~]$ pip install virtualenv Requirement already satisfied: virtualenv in /usr/lib/python2.7/site-packages [zdeveloper@localhost ~]$ pip install setuptools Requirement already satisfied: setuptools in /usr/lib/python2.7/site-packages
Clone common github repo dependencies
For the below git hub repos we cloned from within within parent folder "/home/zdeveloper/Documents/commongithubrepos"
https://github.com/zenoss/zenoss-py-deps https://github.com/zenoss/zenoss-prodbin https://github.com/zenoss/zenoss-protocols https://github.com/control-center/service-migration https://github.com/zenoss/ZenPacks.zenoss.ZenPackLib https://github.com/zenoss/ZenPacks.zenoss.PythonCollector https://github.com/zenoss/pynetsnmp https://github.com/zopefoundation/Zope
For example
git clone https://github.com/zenoss/zenoss-py-deps
Make changes to certain git repos zenoss-protocols
The zenoss-protocols project is missing some files and requires you to build some python package(s) e.g. python/zenoss/protocols/protobufs.
Navigate to zenoss-protocols/python and compile
make GNUmakefile build
Zope
The Zope branch used by Zenoss is 2.13
Navigate to Zope and change the git branch to 2.13
git checkout 2.13
zenoss-prodbin
The Zenoss Product has many different versions. You can choose which version to test against by changing the branch. Let's use a known stable branch like "support/5.3.x" NOTE: You can use github to browse what other branches are available.
git checkout support/5.3.x
Virtual Environment (Optional)
Before we pip install a lot of Python packages Zenoss uses into your general python, we recommend you create a virtual environment to prevent interference with other programs using python and visa-versa
Create virtual environment
In the below steps/examples we start within parent folder "/home/zdeveloper/Documents/virtualenvs"
1. Create the virtual environment "testlatestzenossenv"
[zdeveloper@localhost virtualenvs]$ virtualenv testlatestzenossenv New python executable in /home/zdeveloper/Documents/virtualenvs/testlatestzenossenv/bin/pyth on2 Also creating executable in /home/zdeveloper/Documents/virtualenvs/testlatestzenossenv/bin/pyth on Installing setuptools, pip, wheel... done.
2. Activate the virtual environment (the beginning of your command prompt should now indicate you are in that virtual environment)
[zdeveloper@localhost virtualenvs]$ source testlatestzenossenv/bin/activate (testlatestzenossenv) [zdeveloper@localhost virtualenvs]$
3. You can exit the virtual environment with "deactivate"
(testlatestzenossenv) [zdeveloper@localhost virtualenvs]$ deactivate [zdeveloper@localhost virtualenvs]$
Install python packages Zenoss is dependent on
Install what is listed in requirements.txt https://github.com/zenoss/zenoss-py-deps
IF YOU ARE USING VIRTUAL ENVIRONMENT MAKE SURE IT IS ACTIVE!!!!!
You can do this one at a time or can feed the file into pip like so (this assumes you have navigated to the cloned git repo zenoss-py-deps where requirements.txt resides)
NOTE: This capture of the failing is on purpose, read on (testlatestzenossenv) [zdeveloper@localhost zenoss-py-deps]$ pip install -r requirements.txt Collecting AccessControl==2.13.7 (from -r requirements.txt (line 1)) Collecting Acquisition==2.13.8.1 (from -r requirements.txt (line 2)) Could not find a version that satisfies the requirement Acquisition==2.13.8.1 (from -r requirements.txt (line 2)) (from versions: 2.11.0b1, 2.11.1, 2.11.2, 2.11.3, 2.12.0a1, 2.12.1, 2.12.2, 2.12.3, 2.12.4, 2.13.0, 2.13.1, 2.13.2, 2.13.3, 2.13.4, 2.13.5, 2.13.6, 2.13.7, 2.13.8, 2.13.9, 2.13.10, 2.13.11, 2.13.12, 4.0a1, 4.0, 4.0.1, 4.0.2, 4.0.3, 4.1, 4.2, 4.2.1, 4.2.2, 4.3.0, 4.4.0, 4.4.1, 4.4.2, 4.4.3, 4.4.4, 4.5) No matching distribution found for Acquisition==2.13.8.1 (from -r requirements.txt (line 2)) (testlatestzenossenv) [zdeveloper@localhost zenoss-py-deps]$
Modify requirements.txt
The default requirements.txt should have failed in a similar way for you, so here’s changes that were made (may need to be tweaked as requirements,txt is tweaked for the Zenoss Product)
Acquisition==2.13.8.1 changed to Acquisition==2.13.8 MySQL-python==1.2.3 Removed python-ldap==2.4.6 Removed PyXML==0.8.4 Removed RelStorage==2.0.0.1.dev14 changed to RelStorage==2.0.0 Yapps==2.1.1 changed to Yapps==2.2 zenpacksupport==1.0 Removed zope-app-compat==1.0 Removed
Developing a ZenPack
The previous steps covered setting up an environment to develop a ZenPack locally. Those steps should only have to be done once (assuming you continue to work in the same envrionment)
These next steps are repeatable steps to create a ZenPack from scratch and develop it in an iterative cycle
We use the Zenoss SDK WeatherUnderGround zenpack example (https://zenpack-sdk.zenoss.com/en/latest/tutorial-http-api/wunderground-api.h tml) to demonstrate local development.
Creating the ZenPack files and folders
Since this is a local env, we cannot create a base zenpack using the command "zenpacklib --create" provided in a running Zenoss
The 3 files in the ZenPacks.zenoss.ZenPackLib git repo, containing code that is run by "zenpacklib --create" are...
ZenPacks/zenoss/ZenPackLib/bin/zenpacklib ZenPacks/zenoss/ZenPackLib/zenpacklib ZenPacks/zenoss/ZenPackLib/lib/libexec/ZPLCommand.py
Using that python code, you can construct a script to mimic the boiler plate build out of files and folders for a ZPL based ZenPack.
With the example script called build_zenpack.py as a starting point, you can customize it further to automate repetitive steps you encounter during zenpack development
As an example we can create the Zenoss SDK WeatherUnderGround zenpack called ZenPacks.training.WeatherUnderground
NOTE: In the below execution the script was located in "/home/zdeveloper/Documents/scripts" but run from the projects folder "/home/zdeveloper/Documents/projects" (testlatestzenossenv) [zdeveloper@localhost projects]$ python ../scripts/build_zenpack.py ZenPacks.training.WeatherUnderground - making directory: ZenPacks.training.WeatherUnderground/ZenPacks/training/WeatherUndergroun d - creating file: ZenPacks.training.WeatherUnderground/setup.py - making directory: ZenPacks.training.WeatherUnderground/tests - making init file: ZenPacks.training.WeatherUnderground/tests/__init__.py - making directory: ZenPacks.training.WeatherUnderground/tests/standalone - making init file: ZenPacks.training.WeatherUnderground/tests/standalone/__init__.py - making directory: ZenPacks.training.WeatherUnderground/tests/integration - making init file: ZenPacks.training.WeatherUnderground/tests/integration/__init__.py - creating file: ZenPacks.training.WeatherUnderground/MANIFEST.in - creating file: ZenPacks.training.WeatherUnderground/ZenPacks/__init__.py - creating file: ZenPacks.training.WeatherUnderground/ZenPacks/training/__init__.py - creating file: ZenPacks.training.WeatherUnderground/ZenPacks/training/WeatherUndergroun d/modeler/__init__.py - creating file: ZenPacks.training.WeatherUnderground/ZenPacks/training/WeatherUndergroun d/modeler/plugins/__init__.py - creating file: ZenPacks.training.WeatherUnderground/ZenPacks/training/WeatherUndergroun d/__init__.py - creating file: ZenPacks.training.WeatherUnderground/ZenPacks/training/WeatherUndergroun d/zenpack.yaml - creating file: ZenPacks.training.WeatherUnderground/ZenPacks/training/WeatherUndergroun d/README.rst
Running the ZenPack with a basic unit test
The whole goal of developing locally is to have working code before you deploy to a running Zenoss. You accomplish this through unit tests running your code.
The example script build_zenpack.py created a very basic unit test file "tests/standalone/testYaml.py" that verifies Zenoss can load the zenpack.yaml in the ZenPack
To run that unit test file we give setup.py a testsuite "tests/testsuite.py" and the function that returns the TestLoader
NOTE: The unitest failing is on purpose, read on for an explanation on how to get unit tests working... (testlatestzenossenv) [zdeveloper@localhost ZenPacks.training.WeatherUnderground]$ python setup.py test --test-suite="tests.testsuite.standalone_test_suite" /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'compatZenossVers' warnings.warn(msg) /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'prevZenPackName' warnings.warn(msg) /home/zdeveloper/Documents/virtualenvs/testlatestzenossenv/lib/python2.7 /site-packages/setuptools/dist.py:470: UserWarning: Normalizing '1.0.0dev' to '1.0.0.dev0' normalized_version, running test Searching for ZenPacks.zenoss.ZenPackLib>=2.0.0 Best match: ZenPacks.zenoss.ZenPackLib 2.1.1 Processing ZenPacks.zenoss.ZenPackLib-2.1.1-py2.7.egg
Using /home/zdeveloper/Documents/projects/ZenPacks.training.WeatherUnderground /.eggs/ZenPacks.zenoss.ZenPackLib-2.1.1-py2.7.egg running egg_info writing requirements to ZenPacks.training.WeatherUnderground.egg-info/requires.txt writing ZenPacks.training.WeatherUnderground.egg-info/PKG-INFO writing namespace_packages to ZenPacks.training.WeatherUnderground.egg-info/namespace_packages.txt writing top-level names to ZenPacks.training.WeatherUnderground.egg-info/top_level.txt writing dependency_links to ZenPacks.training.WeatherUnderground.egg-info/dependency_links.txt writing entry points to ZenPacks.training.WeatherUnderground.egg-info/entry_points.txt reading manifest file 'ZenPacks.training.WeatherUnderground.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'ZenPacks.training.WeatherUnderground.egg-info/SOURCES.txt' running build_ext testYaml (unittest.loader.ModuleImportFailure) ... ERROR
======ERROR: testYaml (unittest.loader.ModuleImportFailure) ------ImportError: Failed to import test module: testYaml Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/loader.py", line 252, in _find_tests module = self._get_module_from_name(name) File "/usr/lib64/python2.7/unittest/loader.py", line 230, in _get_module_from_name __import__(name) File "/home/zdeveloper/Documents/projects/ZenPacks.training.WeatherUndergroun d/tests/standalone/testYaml.py", line 2, in
------Ran 1 test in 0.000s
FAILED (errors=1) Test failed:
NOTE: From this point forward we remove the command prompt and text in screen output that is distracting
The ZenPack didn't load properly because Python couldn't find Products.ZenModel.ZenPack e.g. "No module named ZenModel.ZenPack"
To resolve this, define environment variable "PYTHONPATH" and set it to have the path to the "zenoss-prodbin" github repo that we cloned earlier. Then run the test again.
export PYTHONPATH="/home/zdeveloper/Documents/commongithubrepos/zenoss-prodbin" python setup.py test --test-suite="tests.testsuite.standalone_test_suite" testYaml (unittest.loader.ModuleImportFailure) ... ERROR
======ERROR: testYaml (unittest.loader.ModuleImportFailure) ------ImportError: Failed to import test module: testYaml Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/loader.py", line 252, in _find_tests module = self._get_module_from_name(name) File "/usr/lib64/python2.7/unittest/loader.py", line 230, in _get_module_from_name __import__(name) File "/home/zdeveloper/Documents/projects/ZenPacks.training.WeatherUndergroun d/tests/standalone/testYaml.py", line 2, in
------Ran 1 test in 0.000s
FAILED (errors=1) Test failed:
We now have a new error "zenoss.protocols.services". This one is solved by adding the "zenoss-protocols" git repo to the PYTHONPATH (put a colon after the zenoss-prodbin path to separate the 2).
Also, the "zenoss-protocols" git repo doesn't have the python code at the root of the folder so you have to append the python directory to that path
export PYTHONPATH="/home/zdeveloper/Documents/commongithubrepos/zenoss-prodbin: /home/zdeveloper/Documents/commongithubrepos/zenoss-protocols/python" python setup.py test --test-suite="tests.testsuite.standalone_test_suite" testYaml (unittest.loader.ModuleImportFailure) ... ERROR
======ERROR: testYaml (unittest.loader.ModuleImportFailure) ------ImportError: Failed to import test module: testYaml Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/loader.py", line 252, in _find_tests module = self._get_module_from_name(name) File "/usr/lib64/python2.7/unittest/loader.py", line 230, in _get_module_from_name __import__(name) File "/home/zdeveloper/Documents/projects/ZenPacks.training.WeatherUndergroun d/tests/standalone/testYaml.py", line 2, in
------Ran 1 test in 0.000s
FAILED (errors=1) Test failed:
The new error "etc/global.conf" has nothing to do with PYTHONPATH, it is the environment variable "ZENHOME"
You have 2 options to solve this
1. Create a new directory (in this example we created /home/zdeveloper/Documents/environment) that contains 2 more directories "etc" and "log", then create an empty file "global.conf" in the "etc" directory 2. Create a "log" directory in the cloned zenoss-prodbin git repo
Set ZENHOME to the appropriate path depending on which option you go with export ZENHOME="/home/zdeveloper/Documents/environment" python setup.py test --test-suite="tests.testsuite.standalone_test_suite" testYaml (unittest.loader.ModuleImportFailure) ... ERROR
======ERROR: testYaml (unittest.loader.ModuleImportFailure) ------ImportError: Failed to import test module: testYaml Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/loader.py", line 252, in _find_tests module = self._get_module_from_name(name) File "/usr/lib64/python2.7/unittest/loader.py", line 230, in _get_module_from_name __import__(name) File "/home/zdeveloper/Documents/projects/ZenPacks.training.WeatherUndergroun d/tests/standalone/testYaml.py", line 2, in
------Ran 1 test in 0.000s
FAILED (errors=1) Test failed:
The last error is solved by again updating the PYTHONPATH with the path to the service-migration cloned git repo export PYTHONPATH="/home/zdeveloper/Documents/commongithubrepos/zenoss-prodbin: /home/zdeveloper/Documents/commongithubrepos/zenoss-protocols/python:/ho me/zdeveloper/Documents/commongithubrepos/service-migration" python setup.py test --test-suite="tests.testsuite.standalone_test_suite" testYamlName (testYaml.Test) ... ok
------Ran 1 test in 0.000s
OK
Now that we have the unittest running, there are 2 things to note
1. This zenpack is dependent on the ZenPackLib ZenPack being installed. But we didn't add the cloned ZenPackLib git repo to the PYTHONPATH. The reason for this is the setup.py in this zenpack was written to pull in the latest ZenPacks.zenoss.ZenPackLib becuase it is a required dependency. 2. The the tests were run while in the virtual environment "testlatestzenossenv" created earlier. If we were to deactivate the virtual environment you would get something like the following error
(testlatestzenossenv) [zdeveloper@localhost ZenPacks.training.WeatherUnderground]$ deactivate [zdeveloper@localhost ZenPacks.training.WeatherUnderground]$ python setup.py test --test-suite="tests.testsuite.standalone_test_suite" /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'compatZenossVers' warnings.warn(msg) /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'prevZenPackName' warnings.warn(msg) /usr/lib/python2.7/site-packages/setuptools/dist.py:470: UserWarning: Normalizing '1.0.0dev' to '1.0.0.dev0' normalized_version, running test Searching for ZenPacks.zenoss.ZenPackLib>=2.0.0 Best match: ZenPacks.zenoss.ZenPackLib 2.1.1 Processing ZenPacks.zenoss.ZenPackLib-2.1.1-py2.7.egg
Using /home/zdeveloper/Documents/projects/ZenPacks.training.WeatherUnderground /.eggs/ZenPacks.zenoss.ZenPackLib-2.1.1-py2.7.egg running egg_info writing requirements to ZenPacks.training.WeatherUnderground.egg-info/requires.txt writing ZenPacks.training.WeatherUnderground.egg-info/PKG-INFO writing namespace_packages to ZenPacks.training.WeatherUnderground.egg-info/namespace_packages.txt writing top-level names to ZenPacks.training.WeatherUnderground.egg-info/top_level.txt writing dependency_links to ZenPacks.training.WeatherUnderground.egg-info/dependency_links.txt writing entry points to ZenPacks.training.WeatherUnderground.egg-info/entry_points.txt reading manifest file 'ZenPacks.training.WeatherUnderground.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'ZenPacks.training.WeatherUnderground.egg-info/SOURCES.txt' running build_ext testYaml (unittest.loader.ModuleImportFailure) ... ERROR
======ERROR: testYaml (unittest.loader.ModuleImportFailure) ------ImportError: Failed to import test module: testYaml Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/loader.py", line 252, in _find_tests module = self._get_module_from_name(name) File "/usr/lib64/python2.7/unittest/loader.py", line 230, in _get_module_from_name __import__(name) File "/home/zdeveloper/Documents/projects/ZenPacks.training.WeatherUndergroun d/tests/standalone/testYaml.py", line 2, in
------Ran 1 test in 0.000s
FAILED (errors=1) Test failed:
Building the ZenPack
The environment should be configured to build the ZenPack into an egg with one line: "python setup.py bdist_egg"
(testlatestzenossenv) [zdeveloper@localhost ZenPacks.training.WeatherUnderground]$ python setup.py bdist_egg /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'compatZenossVers' warnings.warn(msg) /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'prevZenPackName' warnings.warn(msg) /home/zdeveloper/Documents/virtualenvs/testlatestzenossenv/lib/python2.7 /site-packages/setuptools/dist.py:470: UserWarning: Normalizing '1.0.0dev' to '1.0.0.dev0' normalized_version, running bdist_egg running egg_info creating ZenPacks.training.WeatherUnderground.egg-info writing requirements to ZenPacks.training.WeatherUnderground.egg-info/requires.txt writing ZenPacks.training.WeatherUnderground.egg-info/PKG-INFO writing namespace_packages to ZenPacks.training.WeatherUnderground.egg-info/namespace_packages.txt writing top-level names to ZenPacks.training.WeatherUnderground.egg-info/top_level.txt writing dependency_links to ZenPacks.training.WeatherUnderground.egg-info/dependency_links.txt writing entry points to ZenPacks.training.WeatherUnderground.egg-info/entry_points.txt writing manifest file 'ZenPacks.training.WeatherUnderground.egg-info/SOURCES.txt' reading manifest file 'ZenPacks.training.WeatherUnderground.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'ZenPacks.training.WeatherUnderground.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib creating build/lib/ZenPacks copying ZenPacks/__init__.py -> build/lib/ZenPacks creating build/lib/tests copying tests/__init__.py -> build/lib/tests copying tests/testsuite.py -> build/lib/tests creating build/lib/ZenPacks/training copying ZenPacks/training/__init__.py -> build/lib/ZenPacks/training creating build/lib/ZenPacks/training/WeatherUnderground copying ZenPacks/training/WeatherUnderground/__init__.py -> build/lib/ZenPacks/training/WeatherUnderground creating build/lib/ZenPacks/training/WeatherUnderground/modeler copying ZenPacks/training/WeatherUnderground/modeler/__init__.py -> build/lib/ZenPacks/training/WeatherUnderground/modeler creating build/lib/ZenPacks/training/WeatherUnderground/modeler/plugins copying ZenPacks/training/WeatherUnderground/modeler/plugins/__init__.py -> build/lib/ZenPacks/training/WeatherUnderground/modeler/plugins creating build/lib/tests/standalone copying tests/standalone/__init__.py -> build/lib/tests/standalone copying tests/standalone/testYaml.py -> build/lib/tests/standalone creating build/lib/tests/integration copying tests/integration/__init__.py -> build/lib/tests/integration copying ZenPacks/training/WeatherUnderground/README.rst -> build/lib/ZenPacks/training/WeatherUnderground copying ZenPacks/training/WeatherUnderground/zenpack.yaml -> build/lib/ZenPacks/training/WeatherUnderground creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/ZenPacks copying build/lib/ZenPacks/__init__.py -> build/bdist.linux-x86_64/egg/ZenPacks creating build/bdist.linux-x86_64/egg/ZenPacks/training copying build/lib/ZenPacks/training/__init__.py -> build/bdist.linux-x86_64/egg/ZenPacks/training creating build/bdist.linux-x86_64/egg/ZenPacks/training/WeatherUnderground copying build/lib/ZenPacks/training/WeatherUnderground/__init__.py -> build/bdist.linux-x86_64/egg/ZenPacks/training/WeatherUnderground creating build/bdist.linux-x86_64/egg/ZenPacks/training/WeatherUnderground/modele r copying build/lib/ZenPacks/training/WeatherUnderground/modeler/__init__.py -> build/bdist.linux-x86_64/egg/ZenPacks/training/WeatherUnderground/modele r creating build/bdist.linux-x86_64/egg/ZenPacks/training/WeatherUnderground/modele r/plugins copying build/lib/ZenPacks/training/WeatherUnderground/modeler/plugins/__init__. py -> build/bdist.linux-x86_64/egg/ZenPacks/training/WeatherUnderground/modele r/plugins copying build/lib/ZenPacks/training/WeatherUnderground/README.rst -> build/bdist.linux-x86_64/egg/ZenPacks/training/WeatherUnderground copying build/lib/ZenPacks/training/WeatherUnderground/zenpack.yaml -> build/bdist.linux-x86_64/egg/ZenPacks/training/WeatherUnderground creating build/bdist.linux-x86_64/egg/tests copying build/lib/tests/__init__.py -> build/bdist.linux-x86_64/egg/tests copying build/lib/tests/testsuite.py -> build/bdist.linux-x86_64/egg/tests creating build/bdist.linux-x86_64/egg/tests/standalone copying build/lib/tests/standalone/__init__.py -> build/bdist.linux-x86_64/egg/tests/standalone copying build/lib/tests/standalone/testYaml.py -> build/bdist.linux-x86_64/egg/tests/standalone creating build/bdist.linux-x86_64/egg/tests/integration copying build/lib/tests/integration/__init__.py -> build/bdist.linux-x86_64/egg/tests/integration byte-compiling build/bdist.linux-x86_64/egg/ZenPacks/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/ZenPacks/training/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/ZenPacks/training/WeatherUnderground/__init __.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/ZenPacks/training/WeatherUnderground/modele r/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/ZenPacks/training/WeatherUnderground/modele r/plugins/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/tests/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/tests/testsuite.py to testsuite.pyc byte-compiling build/bdist.linux-x86_64/egg/tests/standalone/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/tests/standalone/testYaml.py to testYaml.pyc byte-compiling build/bdist.linux-x86_64/egg/tests/integration/__init__.py to __init__.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying ZenPacks.training.WeatherUnderground.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying ZenPacks.training.WeatherUnderground.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying ZenPacks.training.WeatherUnderground.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying ZenPacks.training.WeatherUnderground.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying ZenPacks.training.WeatherUnderground.egg-info/namespace_packages.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying ZenPacks.training.WeatherUnderground.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO copying ZenPacks.training.WeatherUnderground.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying ZenPacks.training.WeatherUnderground.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO creating dist creating 'dist/ZenPacks.training.WeatherUnderground-1.0.0.dev0-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it)
The resulting egg can be found under the "dist" directory
(testlatestzenossenv) [zdeveloper@localhost ZenPacks.training.WeatherUnderground]$ ls dist ZenPacks.training.WeatherUnderground-1.0.0.dev0-py2.7.egg
NOTE: There's something wrong about what's included in this ZenPack egg, notice that the "tests" directory is being added to the egg (you can double check by cracking the egg open with any zip file utility). The tests offer no value inside a running Zenoss so should be left out of the egg to avoid it causing problems.
The "MANIFEST.in" should be the proper file to use in order to filter that folder out but it's been our observation that doesn't work. The work around is to remove the __init__.py file in the "tests" directory before building the egg and put back afterwards
Moving these test and build steps into scripts
You can move the declaring of the environment variables and need to be in the virtual environment into a reusable shell script like so
scripts/test_zenpack.sh
#!/bin/bash
export ZENHOME="/home/zdeveloper/Documents/environment" export PYTHONPATH="/home/zdeveloper/Documents/commongithubrepos/zenoss-prodbin: /home/zdeveloper/Documents/commongithubrepos/zenoss-protocols/python:/ho me/zdeveloper/Documents/commongithubrepos/service-migration"
source /home/zdeveloper/Documents/virtualenvs/testzenossenv/testzvenv/bin/activ ate
python setup.py test --test-suite="tests.testsuite.standalone_test_suite" # python setup.py test --test-suite="tests.testsuite.integration_test_suite"
deactivate [zdeveloper@localhost ZenPacks.training.WeatherUnderground]$ ../../scripts/test_zenpack.sh /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'compatZenossVers' warnings.warn(msg) /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'prevZenPackName' warnings.warn(msg) /home/zdeveloper/Documents/virtualenvs/testzenossenv/testzvenv/lib/pytho n2.7/site-packages/setuptools/dist.py:470: UserWarning: Normalizing '1.0.0dev' to '1.0.0.dev0' normalized_version, running test Searching for ZenPacks.zenoss.ZenPackLib>=2.0.0 Best match: ZenPacks.zenoss.ZenPackLib 2.1.1 Processing ZenPacks.zenoss.ZenPackLib-2.1.1-py2.7.egg
Using /home/zdeveloper/Documents/projects/ZenPacks.training.WeatherUnderground /.eggs/ZenPacks.zenoss.ZenPackLib-2.1.1-py2.7.egg running egg_info writing requirements to ZenPacks.training.WeatherUnderground.egg-info/requires.txt writing ZenPacks.training.WeatherUnderground.egg-info/PKG-INFO writing namespace_packages to ZenPacks.training.WeatherUnderground.egg-info/namespace_packages.txt writing top-level names to ZenPacks.training.WeatherUnderground.egg-info/top_level.txt writing dependency_links to ZenPacks.training.WeatherUnderground.egg-info/dependency_links.txt writing entry points to ZenPacks.training.WeatherUnderground.egg-info/entry_points.txt reading manifest file 'ZenPacks.training.WeatherUnderground.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'ZenPacks.training.WeatherUnderground.egg-info/SOURCES.txt' running build_ext testYamlName (testYaml.Test) ... ok
------Ran 1 test in 0.000s
OK
Running the ZenPack with advanced unit tests The ZenPack SDK Tutorial for WeatherUnderground adds some new code in "Create a Modeler Plugin". So, how would we go about developing and testing this code in our local environment?
The PythonPlugin modeler has 2 new functions that have different dependencies
1. collect - requires a running twisted reactor setup by Zenoss and usually depends on an external site to be up and running (WeatherUnderground in this example) 2. process - Depends on nothing but the parameters passed into it and possibly the instance variables of the modeler
NOTE: The WeatherUnderground Tutorial does not use the process function, but ideally the "collect" function would only collect the result and then return that to the "process" function where the relationship map etc... would be created (rather than everything being done in collect)
Because these function's dependencies are different they need to be tested differently. The "tests" directory is composed of 2 sub-directories "standalone" and "integration". The "process" function would be better tested by a unittest in standalone and the "collect" would be better tested by a unittest in integration. We elaborate on what those unittests would look like below. (you can choose to not make the separation but the recommendation is that you do)
Standalone unit test
Since the process function does not offer much to test in this tutorial we test other aspects of the modeler. Namely we test that the zProperties the modeler expects to be present are in the ZenPacks yaml or are zProperties defined else where and can be ignored
tests/standalone/testLocationsModeler.py
import unittest from ZenPacks.training.WeatherUnderground import CFG from ZenPacks.training.WeatherUnderground.modeler.plugins.WeatherUnderground import Locations
class Test(unittest.TestCase):
def _modeler(self): return Locations.Locations()
def testRequiredProperties(self): modeler = self._modeler() zIgnoreProps = {'zCollectorClientTimeout',} for p in modeler.deviceProperties: if p.startswith('z') and p not in zIgnoreProps: self.assertTrue(p in CFG.zProperties, p)
Integration unit test
The WeatherUnderground modeler code in the collect function listed in the Tutorial was modified to return a dict where the key = location and the value is the response for that location. This change better demonstrates what collect should so and how we would go about testing that our code successfully collects from WeatherUnderground API
We test 2 things below
1. Does a missing zWundergroundAPIKey cause no response to be collected 2. Do we get a good response when a valid location and key is used
Notice that
1. there is a different TestCase class we extend from "twisted.trial.unittest.TestCase" 2. 2. we have to manipulate the pseudo Zenoss environment in a tearDown 3. The unit tests themselves use defer.inlineCallbacks like the "collect" function we're testing
tests/integration/testLocationsModeler.py
import logging
import twisted.trial.unittest from twisted.internet import defer from Products.ZenModel.Device import Device from Products import ZenCollector
from ZenPacks.training.WeatherUnderground.modeler.plugins.WeatherUnderground import Locations
class Test(twisted.trial.unittest.TestCase): """NOTE: These are integration tests and assume an external url is reachable""" @defer.inlineCallbacks def deferTearDown(self, ignored, result): # Prevent the following error caused by import of IEventService and queryUtility in the modeler # twisted.trial.util.DirtyReactorAggregateError: Reactor was unclean. #
def _modeler(self): return Locations.Locations()
@defer.inlineCallbacks def testCollectWithMissingApiKey(self): modeler = self._modeler() device = Device('fakeid', False) device.manageIp = '127.0.0.1' device._setPropValue('zWundergroundLocations', ['Austin, TX']) log = logging.getLogger('zen.{}'.format(__name__))
results = yield modeler.collect(device, log)
self.assertIsNone(results) @defer.inlineCallbacks def testCollectWithOneLocation(self): modeler = self._modeler() device = Device('fakeid', False) device.manageIp = '127.0.0.1' device._setPropValue('zWundergroundAPIKey','D0WzaluNpOYBNAWzEJOQ') device._setPropValue('zWundergroundLocations', ['Austin, TX'])
log = logging.getLogger('zen.{}'.format(__name__))
results = yield modeler.collect(device, log)
self.assertEqual(len(results), 1) response = results.get('Austin, TX') responseresults = response.get('RESULTS', []) self.assertEqual(len(responseresults), 1) self.assertEqual(responseresults[0].get('c'), 'US')