ckan Documentation Release
2009, Open Knowledge Foundation
July 29, 2011
CONTENTS
i ii ckan Documentation, Release
This Developer Documentation provides more advanced help for CKAN development and administration. The latest released version of this documentation is found at: http://packages.python.org/ckan/ Contents:
CONTENTS 1 ckan Documentation, Release
2 CONTENTS CHAPTER ONE
README
1.1 Introduction
Comprehensive Knowledge Archive Network (CKAN) Software. See ckan.__long_description__ for more information.
1.2 Developer Installation
These are instructions to get developing with CKAN. Instructions for deploying CKAN to a server are at: Production Deployment (doc/deployment.rst). Before you start it may be worth checking CKAN has passed the auto build and tests. See: http://buildbot.okfn.org/waterfall 1. Ensure these packages are installed: Package Description mercurial Source control python-dev Python interpreter v2.5 - v2.7 and dev headers postgresql PostgreSQL database libpq-dev PostgreSQL library python-psycopg2 PostgreSQL python module libxml2-dev XML library development files libxslt-dev XSLT library development files python-virtualenv Python virtual environments wget Command line tool for downloading from the web build-essential Tools for building source code git-core Git source control (for getting MarkupSafe src) subversion Subversion source control (for pyutilib) For Ubuntu you can install these like so: sudo apt-get install build-essential libxml2-dev libxslt-dev sudo apt-get install wget mercurial postgresql libpq-dev git-core sudo apt-get install python-dev python-psycopg2 python-virtualenv sudo apt-get install subversion
2. Create a python virtual environment In your home directory run the command below. It is currently important to call your virtual environment pyenv so that the automated deployment tools work correctly.
3 ckan Documentation, Release
cd ~ virtualenv pyenv
Tip: If you don’t have a python-virtualenv package in your distribution you can get a virtualenv.py script from within the virtualenv source distribution and then run python virtualenv.py pyenv instead. To help with automatically installing CKAN dependencies we use a tool called pip. Make sure you have activated your environment (see step 3) and then install it from an activated shell like this: easy_install pip
3. Activate your virtual environment To work with CKAN it is best to adjust your shell settings so that your shell uses the virtual environment you just created. You can do this like so: . pyenv/bin/activate
When your shell is activated you will see the prompt change to something like this: (pyenv)[ckan@host ~/]$
An activated shell looks in your virtual environment first when choosing which commands to run. If you enter python now it will actually run ~/pyenv/bin/python which is what you want. 4. Install CKAN code and required Python packages into the new environment First you’ll need to install CKAN. For the latest version run: pip install --ignore-installed -e hg+http://bitbucket.org/okfn/ckan#egg=ckan
CKAN has a set of dependencies it requires which you should install too: pip install --ignore-installed -r pyenv/src/ckan/requires/lucid_missing.txt -r pyenv/src/ckan/requires/lucid_conflict.txt
The --ignore-installed option ensures pip installs software into this virtual environment even if it is already present on the system. If you are using Ubuntu Lucid you can install the rest of the dependencies from the system versions like this: sudo apt-get install python-psycopg2 python-lxml python-sphinx sudo apt-get install python-pylons python-formalchemy python-repoze.who sudo apt-get install python-repoze.who-plugins python-tempita python-zope.interface
If you are not using Ubuntu Lucid you’ll still need to install all the dependencies that would have been met in the apt-get install command at the start. You can do so like this: pip install --ignore-installed -r pyenv/src/ckan/requires/lucid_present.txt
This will take a long time. Particularly the install of the lxml package. At this point you will need to deactivate and then re-activate your virtual environment to ensure that all the scripts point to the correct locations: deactivate . pyenv/bin/activate
5. Setup a PostgreSQL database List existing databases:
4 Chapter 1. README ckan Documentation, Release
psql-l
It is advisable to ensure that the encoding of databases is ‘UTF8’, or internationalisation may be a problem. Since changing the encoding of PostgreSQL may mean deleting existing databases, it is suggested that this is fixed before continuing with the CKAN install. Next you’ll need to create a database user if one doesn’t already exist.
Tip: If you choose a database name, user or password which are different from those suggested below then you’ll need to update the configuration file you’ll create in the next step.
Here we choose ckantest as the database and ckanuser as the user: sudo -u postgres createuser -S -D -R -P ckantest
It should prompt you for a new password for the CKAN data in the database. It is suggested you enter pass for the password. Now create the database, which we’ll call ckantest (the last argument): sudo -u postgres createdb -O ckantest ckantest
6. Create a CKAN config file Make sure you are in an activated environment (see step 3) so that Python Paste and other modules are put on the python path (your command prompt will start with (pyenv) if you have) then change into the ckan directory which will have been created when you installed CKAN in step 4 and create the config file development.ini using Paste: cd pyenv/src/ckan paster make-config ckan development.ini
You can give your config file a different name but the tests will expect you to have used development.ini so it is strongly recommended you use this name, at least to start with. If you used a different database name or password when creating the database in step 5 you’ll need to now edit development.ini and change the sqlalchemy.url line, filling in the database name, user and password you used. sqlalchemy.url = postgresql://ckantest:pass@localhost/ckantest
Other configuration, such as setting the language of the site or editing the visual theme are described in CKAN Configuration (doc/configuration.rst)
Caution: Advanced users: If you are using CKAN’s fab file capability you currently need to create your config file as pyenv/ckan.net.ini so you will probably have ignored the advice about creating a development.ini file in the pyenv/src/ckan directory. This is fine but CKAN probably won’t be able to find your who.ini file. To fix this edit pyenv/ckan.net.ini, search for the line who.config_file = %(here)s/who.ini and change it to who.config_file = who.ini. We are moving to a new deployment system where this incompatibility will be fixed.
7. Create database tables Now that you have a configuration file that has the correct settings for your database, you’ll need to create the tables. Make sure you are still in an activated environment with (pyenv) at the front of the command prompt and then from the pyenv/src/ckan directory run this command:
1.2. Developer Installation 5 ckan Documentation, Release
paster db init
You should see Initialising DB: SUCCESS. If you are not in the pyenv/src/ckan directory or you don’t have an activated shell, the command will not work. If the command prompts for a password it is likely you haven’t set up the database configuration correctly in step 6. 8. Create the cache directory You need to create the Pylon’s cache directory specified by ‘cache_dir’ in the config file. (from the pyenv/src/ckan directory): mkdir data
9. Run the CKAN webserver NB If you’ve started a new shell, you’ll have to activate the environment again first - see step 3. (from the pyenv/src/ckan directory): paster serve development.ini
10. Point your web browser at: http://127.0.0.1:5000/ The CKAN homepage should load without problem. If you ever want to upgrade to a more recent version of CKAN, read the UPGRADE.txt file in pyenv/src/ckan/.
1.3 Test
1.3.1 Setting up to test
Make sure you’ve created a config file: pyenv/ckan/development.ini Ensure you have activated the environment: . pyenv/bin/activate
Install nose into your virtual environment if you haven’t already: pip install --ignore-installed nose
At this point you will need to deactivate and then re-activate your virtual environment to ensure that all the scripts point to the correct locations: deactivate . pyenv/bin/activate
1.3.2 Running developer tests
Here’s how you start the quick development tests: cd pyenv/src/ckan nosetests ckan/tests --ckan
6 Chapter 1. README ckan Documentation, Release
You must run the tests from the CKAN directory as shown above, otherwise the --ckan plugin won’t work correctly.
Caution: By default, the test run is ‘quick and dirty’ - only good enough as a check before committing code. See the next section for improved ways of running tests.
1.3.3 Test configurations
The default way to run tests is defined in test.ini (which is the default config file for nose - change it with option “–with-pylons”). This specifies to use Sqlite and sets faster_db_test_hacks, which are compromises. cd pyenv/src/ckan nosetests ckan/tests --ckan
Although Sqlite is useful for testing a large proportion of CKAN, actually in deployment, CKAN must run with PostgreSQL. Running the tests against PosgreSQL is slower but more thorough for two reasons:
1. You test subtleties of PostgreSQL 2. CKAN’s default search relies on PostgreSQL’s custom Full Text Search, so these (100 or so) tests are skipped when running against Sqlite. So when making changes to anything involved with search or closely related to the database, it is wise to test against PostgreSQL. To test against PosgreSQL: 1. Edit your local development.ini to specify a PostgreSQL database with the sqlalchemy.url parameter. 2. Tell nose to use test-core.ini (which imports settings from development.ini) nosetests ckan/tests --ckan --with-pylons=test-core.ini
The test suite takes a long time to run against standard PostgreSQL (approx. 15 minutes, or close to an hour on Ubuntu/10.04 Lucid). This can be improved to between 5 and 15 minutes by running PostgreSQL in memory and turning off durability, as described at
By default, tests are run using the model defined in ckan/model, but by using the --ckan-migrate option the tests will run using a database that has been created using the migration scripts, which is the way the database is created and upgraded in production. These tests are the most thorough and will take around 20 minutes.
Caution: Ordinarily, you should set development.ini to specify a PostgreSQL database so these also get used when running test-core.ini, since test-core.ini inherits from development.ini. If you were to change the sqlalchemy.url option in your development.ini file to use SQLite, the command above would actually test SQLite rather than PostgreSQL so always check the setting in development.ini to ensure you are running the full tests.
Note: A common error when wanting to run tests against a particular database is to change the sqlalchemy.url in test.ini or test-core.ini. The problem is that these are versioned files and people have checked in these by mistake, creating problems for all other developers and the buildbot. This is easily avoided by only changing the sqlalchemy.url in your local development.ini and testing –with-pylons=test-core.ini.
1.3. Test 7 ckan Documentation, Release
1.3.4 Testing extensions
CKAN extensions ordinarily have their own test.ini that refers to the ckan test.ini, so you can run them in exactly the same way. For example: cd ckanext-dgu nosetests ckanext/dgu/tests --ckan nosetests ckanext/dgu/tests --ckan --with-pylons=test-core.ini
1.4 Development
CKAN is an open source project and contributions are welcome! There are a number of stakeholders in the direction of the project, so we discuss large changes and new features on the ckan-discuss list: http://lists.okfn.org/mailman/listinfo/ckan-discuss New developers should aquaint themselves with the documentation (see below). Proposed changes should be made on a personal CKAN fork (on BitBucket for example). Request merging with the mainline via the ckan-discuss list. We have policies for check-ins that ensure the build doesn’t break etc. on http://ckan.org/#ProjectProcessesandPolicies which should be followed unless someone builds concensus to change it.
1.5 Documentation
The home page for the CKAN project is: http://ckan.org This README file is part of the Developer Documentation, viewable at: http://packages.python.org/ckan/ and stored in the CKAN repo at ckan/doc. The Developer Docs are built using Sphinx: python setup.py build_sphinx (An admin might upload the resulting html to packages.python.org/ckan/ by doing: easy_install sphinx-pypi-upload and python setup.py upload_sphinx) (The docs are also uploaded via dav to http://knowledgeforge.net/ckan/doc/ckan/for backwards compatability).
1.6 Contributors
• Rufus Pollock
8 Chapter 1. README ckan Documentation, Release
1.7 Copying and License
This material is copyright (c) 2006-2011 Open Knowledge Foundation. It is open and licensed under the GNU Affero General Public License (AGPL) v3.0 whose full text may be found at:
1.7. Copying and License 9 ckan Documentation, Release
10 Chapter 1. README CHAPTER TWO
CKAN DESIGN
2.1 Overview
As a system CKAN functions as a synthesis of several different services:
CKAN is part of a larger set of tools and services designed to enable automated use and reuse of content and data:
11 ckan Documentation, Release
2.2 Architecture
The key entity in CKAN is a Package. The data model for a package is pretty much what you see on: http://www.ckan.net/package/new This in turn is heavily based on the kind of packaging information provided for software but with some modifications. One of our aims is to keep things simple and as generic as possible as we have data from a lot of different domains. Thus we’ve tried to keep the core metadata pretty restricted but allow for additional info either via tags or via “extra” arbitrary key/value fields. So very roughly: • unique name • title • url + download url • author/maintainer info • license • notes • tags • [extendable with “extra” fields] All, the code is open-source and if you want to see the actual model definitions in code see here (probably start with core.py):
12 Chapter 2. CKAN Design ckan Documentation, Release
One thing to note here is that all changes to package data are versioned in a wiki-like manner. This gives us a lot of flexibility in how we manage access to the system (as well as providing features like change logging for free!). The rdfizing code can be found here:
2.2. Architecture 13 ckan Documentation, Release
14 Chapter 2. CKAN Design CHAPTER THREE
PRODUCTION DEPLOYMENT
Here’s an example for deploying CKAN to http://demo.ckan.net/ via Apache. 1. Ideally setup the server with Ubuntu. 2. Ensure these packages are installed: (e.g. sudo apt-get install
Now you can then do the deployment with something like:
15 ckan Documentation, Release
$ fab config_0:demo.ckan.net,hosts_str=someserver.net,db_pass=my_password deploy
3. Setup a PostgreSQL database List existing databases: $ sudo -u postgres psql -l
It is advisable to ensure that the encoding of databases is ‘UTF8’, or internationalisation may be a problem. Since changing the encoding of Postgres may mean deleting existing databases, it is suggested that this is fixed before continuing with the CKAN install. Create a database user if one doesn’t already exist: $ sudo -u postgres createuser -S -D -R -P
Replace
4. Create a python virtual environment In a general user’s home directory: $ mkdir -p ~/var/srvc/demo.ckan.net $ cd ~/var/srvc/demo.ckan.net $ virtualenv pyenv $ . pyenv/bin/activate
5. Create the Pylons WSGI script Create a file ~/var/srvc/demo.ckan.net/pyenv/bin/demo.ckan.net.py as follows (editing the first couple of variables as necessary): import os instance_dir=’/home/USER/var/srvc/demo.ckan.net’ config_file=’demo.ckan.net.ini’ pyenv_bin_dir= os.path.join(instance_dir,’pyenv’,’bin’) activate_this= os.path.join(pyenv_bin_dir,’activate_this.py’) execfile(activate_this, dict(__file__=activate_this)) from paste.deploy import loadapp config_filepath= os.path.join(instance_dir, config_file) from paste.script.util.logging_config import fileConfig fileConfig(config_filepath) application= loadapp(’config: %s’% config_filepath)
6. Install code and dependent packages into the environment Decide which release of CKAN you want to install. The CHANGELOG.txt has details on the releases. You’ll need the exact tag name, and these are listed on the bitbucket page: https://bitbucket.org/okfn/ckan/src and hover over tags to see the options, e.g. ckan-1.4. $ wget https://bitbucket.org/okfn/ckan/raw/ckan-1.4/pip-requirements.txt Or for the bleeding edge use: $ wget https://bitbucket.org/okfn/ckan/raw/default/pip-requirements.txt
And now install:
16 Chapter 3. Production Deployment ckan Documentation, Release
$ pip -E pyenv install -r pip-requirements.txt
If everything goes correctly then you’ll finally see: Successfully installed. 7. Create CKAN config file $ paster make-config ckan demo.ckan.net.ini
8. Configure CKAN Edit ‘demo.ckan.net.ini’ and change the default values as follows: 8.1. sqlalchemy.url Set the sqlalchemy.url database connection information using values from step 3. 8.2. licenses_group_url Set the licenses_group_url to point to a licenses service. Options include: http://licenses.opendefinition.org/2.0/ckan_original http://licenses.opendefinition.org/2.0/all_alphabetical
For information about creating your own licenses services, please refer to the Python package called ‘licenses’ (http://pypi.python.org/pypi/licenses). 8.3. loggers CKAN can make a log file if you change the [loggers] section to this: [loggers] keys = root, ckan
[handlers] keys = file
[formatters] keys = generic
[logger_root] level = INFO handlers = file
[logger_ckan] level = DEBUG handlers = file qualname = ckan
[handler_file] class = handlers.RotatingFileHandler formatter = generic level = NOTSET args = (’/var/log/ckan/demo.ckan.log’, ’a’, 2048, 3)
[formatter_generic] format = %(asctime)s %(levelname)-5.5s [%(name)s] %(message)s
9. Initialise database $ . pyenv/bin/activate $ paster --plugin ckan db init --config demo.ckan.net.ini
17 ckan Documentation, Release
10. Set some permissions for Pylons Whilst still in the ~/var/srvc/demo.ckan.net directory: $ mkdir data sstore $ chmod g+w -R data sstore $ sudo chgrp -R www-data data sstore $ ln -s pyenv/src/ckan/who.ini ./
Also edit the who.ini configuration file to set a secret for the auth_tkt plugin. 11. Setup Apache with Ckan Create file /etc/apache2/sites-available/demo.ckan.net as follows:
WSGIScriptAlias / /home/USER/var/srvc/demo.ckan.net/pyenv/bin/demo.ckan.net.py # pass authorization info on (needed for rest api) WSGIPassAuthorization On
ErrorLog /var/log/apache2/demo.ckan.net.error.log CustomLog /var/log/apache2/demo.ckan.net.custom.log combined
12. Enable site in Apache $ sudo a2ensite demo.ckan.net
13. Restart Apache $ sudo /etc/init.d/apache2 restart
14. Browse CKAN website at http://demo.ckan.net/ (assuming you have the DNS setup for this server). Should you have problems, take a look at the log files specified in your apache config and ckan oconfig. e.g. /var/log/apache2/demo.ckan.net.error.log and /var/log/ckan/demo.ckan.log.
18 Chapter 3. Production Deployment CHAPTER FOUR
UPGRADE
Ideally production deployments are upgraded with fabric, but here are the manual instructions. 1. Activate the virtual environment for your install: $ cd ~/var/srvc/demo.ckan.net $ . pyenv/bin/activate
2. It’s probably wise to backup your database: $ paster --plugin=ckan db dump demo_ckan_backup.pg_dump --config=demo.ckan.net.ini
3. Get a version of pip-requirements.txt for the new version you want to install (see info on finding a suitable tag name above): $ wget https://bitbucket.org/okfn/ckan/raw/ckan-1.4/pip-requirements.txt
4. Update all the modules: $ pip -E pyenv install -r pip-requirements.txt
5. Upgrade the database: $ paster --plugin ckan db upgrade --config {config.ini}
6. Restart apache (so modpython has the latest code): $ sudo /etc/init.d/apache2 restart
7. You could manually try CKAN works in a browser, or better still run the smoke tests found in ckanext/blackbox. To do this, install ckanext and run ckanext from another machine - see ckanext README.txt for instructions: https://bitbucket.org/okfn/ckanext and then run: $ python blackbox/smoke.py blackbox/ckan.net.profile.json
19 ckan Documentation, Release
20 Chapter 4. Upgrade CHAPTER FIVE
CKAN CONFIGURATION
A CKAN instance is configured with the .ini file in the root ckan directory. The most important parameter to set is the sqlalchemy.url giving the database connection. But there are also several additional options to change the way the CKAN site operates. On a production machine the file will be probably be named after the site name. e.g. ca.ckan.net.ini On a development machine it is probably development.ini On a new installation of ckan, you have to create a new config file from the template, something like this: paster --plugin ckan make-config ckan ca.ckan.net.ini
This creates ca.ckan.net.ini based on the template for this file in ckan/config/deployment.ini_tmpl There are several general Pylons options, and all the CKAN-specific ones are in the [app:main] section. Once the config file is changed, Apache needs to be restarted to read in the new changes.
5.1 sqlalchemy.url
Example: sqlalchemy.url = postgres://tester:pass@localhost/ckantest3
This defines the database that CKAN is to use. The format is: sqlalchemy.url = postgres://USERNAME:PASSWORD@HOST/DBNAME
5.2 package_form
Example: package_form= ca
Default value: standard This sets the name of the form to use when editing a package. This can be a form defined in the core CKAN code or in another setuputils-managed python module. The only requirement is that the setup.py has an entrypoint for the form defined in the ckan.forms section. See Forms Using Templates
21 ckan Documentation, Release
5.3 package_hide_extras
Example: package_hide_extras = my_private_field other_field
Default value: (empty) This sets a space-seperated list of extra field key values which will not be shown on the package read page. While this is useful to create internal notes etc., it is not a security measure in any way. The keys will still be available via the API and in revision diffs.
5.4 rdf_packages
Example: rdf_packages = http://semantic.ckan.net/record/
Configure this if you have an RDF store of the same packages as are in your CKAN instance. It will provide three sorts of links from each package page to the equivalent RDF URL given in rdf_packages: 1. 303 redirects for clients that content-negotiate rdf-xml or turtle. e.g. client GETs http://ckan.net/package/pollution-2008 with accept header application/rdf+xml curl -H "Accept: application/rdf+xml" http://ckan.net/package/pollution-2008. CKAN’s response is a 303 redirect to http://semantic.ckan.net/package/pollution-2008 which can be obtained with: curl -L -H "Accept: application/rdf+xml" http://ckan.net/package/pollution-2008 2. Embedded links for browsers that are aware. e.g. 3. A visible RDF link on the page. e.g.
5.5 cache_validation_enabled
Example: ckan.cache_validation_enabled= False
Default value: True This option determines whether browsers (or other caching services running between the browser and CKAN) are helped to cache particular CKAN pages, by validating when the page content hasn’t changed. This is achieved using ETags headers provided by CKAN, which is a hash that changes when the content has changed. Developers editing the templates should set this to False, since Etags hashes don’t look for template changes.
5.6 cache_enabled
Example: ckan.cache_enabled= True
22 Chapter 5. CKAN Configuration ckan Documentation, Release
Default value: False Setting this option to True turns on several server-side caches. When the caching is on, caching can be further configured as follows. (The key has been renamed from cache_enabled, which is deprecated, but still works ckan.cache_enabled for now.) To set the type of Beaker storage: beaker.cache.type= file
To set the expiry times (in seconds) for specific controllers (which use the proxy_cache) specifiy the methods like this: ckan.controllers.package.list.expires= 600 ckan.controllers.tag.read.expires= 600 ckan.controllers.apiv1.package.list.expires= 600 ckan.controllers.apiv1.package.show.expires= 600 ckan.controllers.apiv2.package.list.expires= 600 ckan.controllers.apiv2.package.show.expires= 600
There is also en option to set the max-age value of static files delivered by paster: ckan.static_max_age= 3600
5.7 openid_enabled
Example: openid_enabled= False
Default value: True Setting this option to Fase turns off openid for login.
5.8 licenses_group_url
A url pointing to a JSON file containing a list of license objects. This list determines the licenses offered by the system to users, for example when creating or editing a package. This is entirely optional – by default the system will use the ckan list of licenses available in the Licenses package. More details about the license objects including the license format and some example license lists can be found on the open license service at http://licenses.opendefinition.org/. Examples: licenses_group_url = file:///path/to/my/local/json-list-of-licenses.js licenses_group_url = http://licenses.opendefinition.org/2.0/ckan_original
5.9 lang
Example: lang=de
5.7. openid_enabled 23 ckan Documentation, Release
Default value: en (English) Use this to specify the language of the text displayed in the CKAN web UI. This requires a suitable mo file installed for the language. For more information on internationalization, see: http://wiki.okfn.org/ckan/i18n#DeployingaTranslation
5.10 extra_template_paths
Example: extra_template_paths=/home/okfn/brazil_ckan_config/templates
To customise the display of CKAN you can supply replacements for the Genshi template files. Use this option to specify where CKAN should look for them, before reverting to the ‘ckan/templates’ folder. You can supply more than one folder, separating the paths with a comma (,). The example value for the extra_template_paths option could, for example, be used to override CKAN templates with these ones: • /home/okfn/brazil_ckan_config/templates/layout.html • /home/okfn/brazil_ckan_config/templates/package/edit.html More details about this feature are found at: http://wiki.okfn.org/ckan/doc/theme
5.11 extra_public_paths
Example: extra_public_paths = /home/okfn/brazil_ckan_config/public
To customise the display of CKAN you can supply replacements for staticly served files such as HTML, CSS, script and PNG files. Use this option to specify where CKAN should look for them, before reverting to the ‘ckan/public’ folder. You can supply more than one folder, separating the paths with a comma (,). The example value for the extra_public_paths option could, for example, be used to provide an image and stylesheet: • /home/okfn/brazil_ckan_config/public/images/brazil.png • /home/okfn/brazil_ckan_config/public/css/extra.css More details about this feature are found at: http://wiki.okfn.org/ckan/doc/theme
5.12 package_new_return_url & package_edit_return_url
Example: package_new_return_url = http://datadotgc.ca/new_dataset_complete?name=
To allow the Edit Package and New Package forms to be integrated into a third party interface, setting these options allows you to set a the return address. So when the user has completed the form and presses ‘commit’, the user is redirected to the URL specified. The ‘
24 Chapter 5. CKAN Configuration ckan Documentation, Release
5.13 carrot_messaging_library
Example: carrot_messaging_library=pyamqplib
This is the messaging library backend to use. Options:
* ‘‘pyamqplib‘‘ - AMQP (e.g. for RabbitMQ)
* ‘‘pika‘‘ - alternative AMQP
* ‘‘stomp‘‘ - python-stomp
* ‘‘queue‘‘ - native Python Queue (default) - NB this doesn’t work inter-process
See carrot documentation for details.
5.14 amqp_hostname, amqp_port, amqp_user_id, amqp_password
Example: amqp_hostname=localhost amqp_port=5672 amqp_user_id=guest amqp_password=guest
These are the setup parameters for AMQP messaging. These only apply if the messageing library has been set to use AMQP (see carrot_messaging_library). The values given in the example are the default values.
5.15 build_search_index_synchronously
Example: ckan.build_search_index_synchronously=
Default (if you don’t define it):: indexing is on This controls the operation of the CKAN Postgres full text search indexing. If you don’t define this option then indexing is on. You will want to turn this off if you want to use a different search engine for CKAN (e.g. SOLR). In this case you need to define the option equal to blank (as in the given example).
5.16 search_backend
Example: search_backend= solr
Default value: sql This controls the type of search backend. Currently valid values are sql (meaning Postgres full text search) and solr. If you specify sql then ensure indexing is on (build_search_index_synchronously is not defined). If you specify solr then ensure you specify a solr_url.
5.13. carrot_messaging_library 25 ckan Documentation, Release
5.17 solr_url
Example: solr_url = http://solr.okfn.org/solr/test.ckan.net
This configures SOLR search, (if selected with ‘search_backend’_). Running solr will require a schema.xml file, such as the one in the ckan-solr-index repository. Optionally, solr_user and solr_password can also be passed along to specify HTTP Basic authentication details for all solr requests.
5.18 site_title
Example: ckan.site_title=Open Data Scotland
Default value: CKAN This sets the name of the site, as displayed in the CKAN web interface.
5.19 site_description
Example: ckan.site_description=
Default value: (none) This is for a description, or tag line for the site, as displayed in the header of the CKAN web interface.
5.20 site_logo
Example: ckan.site_logo=/images/ckan_logo_fullname_long.png
Default value: (none) This sets the logo used in the title bar.
5.21 site_url
Example: ckan.site_url=http://scotdata.ckan.net
Default value: (none) The primary URL used by this site. Uses:
26 Chapter 5. CKAN Configuration ckan Documentation, Release
* in the API to provide packages with links to themselves in the web UI.
5.22 api_url
Example: ckan.api_url=http://scotdata.ckan.net/api
Default value: /api The URL which resolves to the CKAN API part of the site. This is useful if the API is hosted on a different domain, for example when a third party site uses the forms API.
5.23 default_roles
This allows you to set the default authorization roles (i.e. permissions) for new objects. Currently this extends to new packages, groups, authorization groups and the ‘system’ object. For full details of these, see CKAN Authorization and Access Control. The value is a strict JSON dictionary of user names “visitor” and “logged_in” with lists of their roles. Example: ckan.default_roles.Package={"visitor":["editor"],"logged_in":["editor"]} ckan.default_roles.Group={"visitor":["reader"],"logged_in":["reader"]}
With this example setting, visitors (any user who is not logged in) and logged in users can only read packages that get created (only sysadmins can edit). Defaults: see in ckan/model/authz.py for: default_default_user_roles
5.24 plugins
Example: ckan.plugins = disqus synchronous_search datapreview googleanalytics stats storage admin follower
Specify which CKAN extensions are to be enabled. If you specify an extension but have not installed the code then CKAN will not start. Format in a space separated list of the extension names. The extension name is the key in the [ckan.plugins] section of the extension’s setup.py.
5.25 dumps_url & dumps_format
Example: ckan.dumps_url = http://ckan.net/dump/ ckan.dumps_format = CSV/JSON
If there is a page which allows you to download a dump of the entire catalogue then specify the URL and the format here, so that it can be advertised in the web interface. The dumps_format is just a string for display.
5.22. api_url 27 ckan Documentation, Release
5.26 log_dir
Example: ckan.log_dir = /var/log/ckan/
This is a directory where CKAN cron scripts (if there are any installed) should write log files to. Note: this setting is nothing to do with the main CKAN log file, whose filepath is set in the [handler_file] args.
5.27 dump_dir
Example: ckan.dump_dir = /var/lib/ckan/dump/
This is a directory where JSON or CSV dumps of the database are to be written, assuming a script has been installed to do this. Note it is usual to setup the apache config to serve this directory.
5.28 backup_dir
Example: ckan.backup_dir = /var/backup/ckan/
This is a directory where SQL database backups are to be written, assuming a script has been installed to do this.
28 Chapter 5. CKAN Configuration CHAPTER SIX
PASTER COMMANDS
Paster commands provide a number of useful ways to administer a CKAN installation. These are run on the command line on the server running CKAN.
6.1 Commands
changes Distribute changes create-test-data Create test data in the database. db Perform various tasks on the database. notify Send out modification notifications. ratings Manage the ratings stored in the db rights Commands relating to per-object and system-wide access rights. roles Commands relating to roles and actions. search-index Creates a search index for all packages sysadmin Gives sysadmin rights to a named user user Manage users
6.2 Running paster commands
Basically the format is: paster --plugin=ckan
To get a list of paster commands (i.e. including CKAN commands): paster--plugin=ckan--help
And you can get help for each command. e.g.: paster --plugin=ckan --help db
6.2.1 --plugin parameter
Paster is a system-wide command, so you need to tell it to find the commands defined in the ckan code. So first parameter to paster is normally --plugin=ckan.
29 ckan Documentation, Release
NB: If your current directory is where CKAN’s setup.py is, you don’t need to specify this parameter. If you want to run a “paster shell” then the plugin is pylons. e.g. paster --plugin=pylons shell. Often you will want to run this as the same user as the web application, to ensure log files are written as the same user. And you’ll also want to specify a config file. e.g.: sudo -u www-data paster --plugin=pylons shell myckan.ini
6.2.2 --config parameter
Each server can have multiple CKAN instances running, so use this parameter to specify the CKAN config file for the instance you want to operate on. e.g. --config=myckan.ini NB: Developers tend to use development.ini, and this is the default value (looking in the current directory), so in this situation you don’t need to specify this parameter.
6.2.3 Position of paster parameters
--plugin is a paster parameter, so needs to come before the CKAN command, but –config is a ckan parameter so needs to come after the CKAN command. Here are three ways to express the same thing: paster db init paster --plugin=ckan db --config=development.ini init paster --plugin=ckan db init --config=development.ini
30 Chapter 6. Paster commands CHAPTER SEVEN
CKAN API
The CKAN API presents the catalog of metadata behind CKAN.
7.1 Overview
The CKAN data catalog is not only available in a web browser, but also via its Application Programming Interface (API). The API can be used to view and change the catalog. This document describes the resource locations, data formats, and status codes which comprise the CKAN API, so that anyone can create software applications that use the API service. The CKAN API follows the RESTful (Representational State Transfer) style. Resource locations are separated both from the methods supported by the resources, and from the data formats and status codes used by the methods.
7.2 Versions
The CKAN API is versioned, so that backwards incompatible changes can be introduced without removing existing support. A particular version of the API can be used by including its version number after the API location and before the resource location. If the API version is not included in the request, then the API will use the oldest supported version of the API to handle that request. The oldest supported version is currently 1.
7.2.1 Version 1
The api/version1 is the original API. At first, package and group names were invariant, and were used in persis- tent references.
7.2.2 Version 2
The api/version2 is the latest API. Since edting the names of packages and groups is now possible, packages and groups are identified by their UUID in entity representations.
31 ckan Documentation, Release
7.3 Code Modules for Client Applications
There are also some code modules (Python, PHP, Drupal, Perl etc.) that provide convenient wrappers around much of the CKAN API. For full details of these, please consult: http://trac.ckan.org/wiki/API
7.4 Example of Usage
Let’s say you’re using ckan.net and want a list of all the packages. If you GET http://ckan.net/api/rest/package then it will return the list of the package names in JSON format: ["2000-us-census-rdf","32000-naples-florida-businesses-kml","aaoe-87","acawiki","adb-sdbs","addgene","advances-in-dental-research",...]
There are several ways you might access this URL: • simply put this URL into your web browser and save the resulting text file • you could use a command-line program such as curl or wget • you could write a program that uses an http library • use the Python library CKAN Client • use command-line tool Datapkg. It manages datasets in a similar way as getting software packages with apt-get or CPAN. You could search for packages to do with ‘open street map’ like this: http://ckan.net/api/search/package?q=open+street+map returning: {"count":4,"results":["uk-naptan-osm","osm-uk","osm","naptan"]}
You can see the full record for the osm package in JSON format with this: http://ckan.net/api/rest/package/osm which returns: {"id": "a3dd8f64-9078-4f04-845c-e3f047125028", "name": "osm", "title": "Open Street Map", "version": null, "url": "http://www.openstreetmap.org/", ... }
You might add a tag by a PUT to http://ckan.net/api/rest/package/osm with this data: {"tags":["navigation","openstreetmap","map","geo","geodata","xml","publicdomain","osm","my-new-tag"]}
Here we use curl to create a new package (you’ll need your own API Key to try this): $ curl http://test.ckan.net/api/rest/package -d ’{"name":"test", "title":"Test package"}’ -H "Authorization: 474d34e4-b710-4b77-b89f-2e909c336b91"
And we create an example package with some government data:
$ curl http://test.ckan.net/api/rest/package -d ’{"name": "abstract_of_scottish_agricultural_statistics", "title": "Abstract of Scottish Agricultural Statistics", "version": null, "url": null, "author": "Scottish Government", "author_email": null, "maintainer": null, "maintainer_email": null, "notes": "A long term series of all the main agriculture census items collected in the June census.\n\nSource agency: Scottish Government\n\nDesignation: National Statistics\n\nLanguage: English\n\nAlternative title: Abstract of Scottish Agricultural Statistics", "license_id": "ukcrown-withrights", "tags": ["farm-outputs", "environment", "agriculture", "farming", "agriculture-and-environment"], "extras": {"geographic_coverage": "010000: Scotland", "geographical_granularity": "Country", "external_reference": "ONSHUB", "temporal_granularity": "", "date_updated": "", "agency": "", "precision": "", "temporal_coverage_to": "", "temporal_coverage_from": "", "national_statistic": "yes", "import_source": "ONS-ons_data_50_days_to_2010-05-04", "department": "Scottish Government", "update_frequency": "", "date_released": "2010-03-17", "categories": "Agriculture and Environment"}, "resources": [{"url": "http://www.scotland.gov.uk/Topics/Statistics/Browse/Agriculture-Fisheries", "description": "1982-2008 | hub/id/119-33192", "format": ""}, {"url": "http://www.scotland.gov.uk/Topics/Statistics/Browse/Agriculture-Fisheries", "description": "1982-2007 | hub/id/119-34917", "format": ""}, {"url": "http://www.scotland.gov.uk/Topics/Statistics/Browse/Agriculture-Fisheries", "description": "1983-2009 | hub/id/119-44310", "format": ""}]}’ -H "Authorization: d6c3349a-6ccf-45ef-88d4-a8e59a574bf2"
32 Chapter 7. CKAN API CHAPTER EIGHT
CKAN EXTENSIONS, PLUGINS INTERFACES AND WORKERS
Note: the terms “extension”, “plugin interface” and “worker” now have very precise meanings and that the use of the generic word “plugin” to describe any way in which CKAN might be extended is deprecated.
Contents • CKAN Extensions, Plugins Interfaces and Workers – Background – CKAN Extensions * Installing an extension * Creating your own extension – Plugins * Summary * An Example * Setting the entry point * Writing a database plugin * Authorization Group Plugins * Publishing Your Extension * Writing a plugin interface * Testing plugins * Ordering of extensions * Plugin API documentation – The Queue Extension * Installing RabbitMQ · CentOS · Ubuntu * Working Directly with Carrot * Working with CKANext Queue · Writing a Publisher · Writing a worker * Configuring CKAN to use the queue · Logging
33 ckan Documentation, Release
8.1 Background
The CKAN codebase is open source so that different organisations can customise it for their own needs. As CKAN has grown in popularity it has become especially important to organise the code in a way that can accommodate the customisations which different organisations require without those changes interfering with customizations required by other organisations. To meet this need we have introduced the concepts of CKAN extensions, plugin interfaces and workers. These work together to provide a simple mechanism to extend core CKAN functionality. Let’s start by looking at extensions.
8.2 CKAN Extensions
CKAN currently has the following extensions: • disqus extension for user comments • Solr extension for full text search and result faceting • Asyncronous queue extension based on AMQP for model update, harvesting and other operations • Custom form extensions for different governments
Note: The form extension does not currently behave quite the same way as the other extensions, it will do soon.
All CKAN extensions have package names of the form ckanext-name so the queue extension code is stored in the package ckanext-queue for example. Extensions are implemented as namespace packages under the ckanext package which means that they can be imported like this: $ python >>> import ckanext.queue
Extensions are used for any code that is not required for the core CKAN code to operate but which are nethertheless and important part of one or more CKAN instances. Individual CKAN extensions may implement one or more plugin interfaces or workers to provide their func- tionality. You’ll learn about these later on. All CKAN extensions are described on the CKAN public wiki at http://wiki.okfn.org/ckan/extensions. If you write an extension then do share details of it there but also document the plugin interfaces the extension implements so that when people install it, they will know which plugin interfaces they need to set up in their config file.
8.2.1 Installing an extension
To install an extension on a CKAN instance: 1. Install the extension package code using pip. The -E parameter is for your CKAN python environment (e.g. ~/var/srvc/ckan.net/pyenv). Prefix the source url with the repo type (hg+ for Mercurial, git+ for Git). For example: $ pip install -E ~/var/srvc/ckan.net/pyenv hg+http://bitbucket.org/okfn/ckanext-disqus
2. Add the names of any plugin implementations the extension uses to the CKAN config. The config file may have a filepath something like: ~/var/srvc/ckan.net/ckan.net.ini. The plugins variable is in the ‘[app:main]’ section under ‘ckan.plugins’. e.g.:
34 Chapter 8. CKAN Extensions, Plugins Interfaces and Workers ckan Documentation, Release
[app:main] ckan.plugins = disqus
If your extension implemented multiple different plugin interfaces, separate them with spaces::
ckan.plugins = disqus amqp myplugin
3. Restart WSGI, which usually means restarting Apache: $ sudo /etc/init.d/apache2 restart
8.2.2 Creating your own extension
All CKAN extensions must start with the name ckanext-. You can create your own CKAN extension like this: (pyenv)$ paster create -t ckanext ckanext-myname
You’ll get prompted to complete a number of variables which will be used in your package. You change these later by editing the generated setup.py file. Here’s some example output: Selected and implied templates: ckan#ckanext CKAN extension project template
Variables: egg: ckanext_myname package: ckanextmyname project: ckanext-myname Enter version (Version (like 0.1)) [’’]: 0.4 Enter description (One-line description of the package) [’’]: Great extension package Enter author (Author name) [’’]: James Gardner Enter author_email (Author email) [’’]: [email protected] Enter url (URL of homepage) [’’]: http://jimmyg.org Enter license_name (License name) [’’]: GPL Creating template ckanext Creating directory ./ckanext-myname Directory ./ckanext-myname exists Skipping hidden file pyenv/src/ckan/ckan/pastertemplates/template/.setup.py_tmpl.swp Recursing into ckanext Creating ./ckanext-myname/ckanext/ .svn/ does not exist; cannot add directory Recursing into +project+ Creating ./ckanext-myname/ckanext/myname/ .svn/ does not exist; cannot add directory Copying __init__.py to ./ckanext-myname/ckanext/myname/__init__.py .svn/ does not exist; cannot add file Copying __init__.py to ./ckanext-myname/ckanext/__init__.py .svn/ does not exist; cannot add file Copying setup.py_tmpl to ./ckanext-myname/setup.py .svn/ does not exist; cannot add file Running pyenv/bin/python setup.py egg_info
Once you’ve run this you should find your extension is already set up in your virtual environment so you can import it: (pyenv)$ python Python 2.6.6 (r266:84292, Oct 6 2010, 16:19:55) [GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2
8.2. CKAN Extensions 35 ckan Documentation, Release
Type "help", "copyright", "credits" or "license" for more information. >>> import ckanext.myname >>>
To build useful extensions you need to be able to “hook into” different parts of CKAN in order to extend its function- ality. You do this using CKAN’s plugin architeture. We’ll look at this in the next section. If you do write a CKAN extension you may well want to publish it publicly so others can use it too. See the Publishing your extension section below to find out how.
8.3 Plugins
Plugin interfaces provide a specification which extensions can implement in order to “hook into” core CKAN func- tionality.
8.3.1 Summary
The CKAN plugin implementation is based on the PyUtilib component architecture (PCA). Here’s a quick summary, we’ll go through all this in much more detail in a minute: 1. The CKAN core contains various plugin interfaces, each specifying a set of methods where plugins may hook into the software. For example a plugin wanting to hook into the SQLAlchemy mapping layer would need to implement the IMapperExtension interface. 2. A plugin is a class that derives from ckan.plugins.Plugin or more commonly SingletonPlugin. It must also implement one of the plugin interfaces exposed in ckan.plugins.interfaces. The choice interface determines the functionality the plugin is expected to provide. 3. Plugin objects must be registered as setuptools entry points. The ckan.plugins configuration directive is searched for names of plugin entry points to load and activate. Here’s a list of some of the more commonly used plugin interfaces: IMapper Listens and react to every database change IRoutes and IController Provide an implementation to handle a particular URL IGenshiStreamFilter Intercept template rendering to modify the output ISession IDomainObjectModification IGroupController Plugins for in the groups controller. These will usually be called just before committing or returning the respective object, i.e. all validation, synchronization and authorization setup are complete. IConfigurable Pass configuration to plugins and extensions IAuthorizer Allows customisation of the default Authorization behaviour If you look in ckan/plugins/interfaces.py you will see the latest plugin interfaces. Alternativlly see the Plugin API documentation below.
36 Chapter 8. CKAN Extensions, Plugins Interfaces and Workers ckan Documentation, Release
Note: The existing ‘IRoutesExtension’, ‘IMapperExtension’ and ‘ISessionExtension’ should be renamed in the code to not have the word ‘Extension’ in their names.
8.3.2 An Example
Plugin interfaces are basically just Python classes where each method is a hook which allows a plugin that uses the interface to be notified when it is called. As an example, let’s look at a plugin which gets configuration options from a config file and is called each time a template is rendered in order to add some HTML to the page.
Tip: This example is based on real code used to implement the ckanext-discus plugin to add commenting to packages. You can see the latest version of this code at http://bitbucket.org/okfn/ckanext- disqus/src/tip/ckanext/plugins/disqus/__init__.py.
First we set up logging and some helpers we’ll need from Genshi to transfor the stream: import logging log= logging.getLogger(__name__) import html from genshi.core import TEXT from genshi.input import HTML from genshi.filters import Transformer
Then we import the CKAN plugin code: from ckan.plugins.core import SingletonPlugin, implements from ckan.plugins.interfaces import IConfigurable, IGenshiStreamFilter
In this case we are implementing both the IConfigurable and IGenshiStreamFilter plugin interfaces in out plugin class. The IConfigurable plugin interface defines a configure() method which will be is called on out plugin to let it know about configuration options. The IGenshiStreamFilter plugin interface defines a filter() method which will be called on the plugin to give it the oppurtunity to change the template before the HTML is returned to the browser. Let’s have a look at the code: class Disqus(SingletonPlugin): """ Insert javascript fragments into package pages and the home page to allow users to view and create comments on any package. """
implements(IConfigurable) implements(IGenshiStreamFilter)
def configure(self, config): """ Called upon CKAN setup, will pass current configuration dict to the plugin to read custom options. """ self.disqus_name= config.get(’disqus.name’, None) if self.disqus_name is None: log.warn("No disqus forum name is set. Please set \ ’disqus.name’ in your .ini!")
8.3. Plugins 37 ckan Documentation, Release
self.disqus_name=’ckan’
def filter(self, stream): """ Required to implement IGenshiStreamFilter; will apply some HTML transformations to the page currently rendered. Depends on Pylons global objects, how can this be fixed without obscuring the inteface? """
from pylons import request, tmpl_context as c routes= request.environ.get(’pylons.routes_dict’)
if routes.get(’controller’) ==’package’ and \ routes.get(’action’) ==’read’ and c.pkg.id: data={’name’: self.disqus_name, ’identifier’:’pkg-’+c.pkg.id} stream= stream| Transformer(’body’)\ .append(HTML(html.BOTTOM_CODE% data)) stream= stream| Transformer(’body//div[@id="comments"]’)\ .append(HTML(html.COMMENT_CODE% data))
if routes.get(’controller’) ==’home’ and \ routes.get(’action’) ==’index’: data={’name’: self.disqus_name} stream= stream| Transformer(’body// \ div[@id="main"]//ul[@class="xoxo"]’)\ .append(HTML(html.LATEST_CODE% data))
return stream
Notice that the Disqus class explicitly states that it implements IConfigurable and IGenshiStreamFilter with these two lines: implements(IConfigurable) implements(IGenshiStreamFilter)
Also notice that Disqus inherits from SingletonPlugin. This means that only one instance of the plugin is needed to provide the service. There is also a Plugin class for occasions where you need multiple instances. By carefully choosing the plugin interfaces your plugin uses you can hook into lots of parts of CKAN. Later on you’ll see how to write your own plugin interfaces to define your own “hooks”. Before we can use the Disqus plugin there is one more thing to do: add it to the extension and set an entry point.
8.3.3 Setting the entry point
Imagine the code above was saved into a file named disqus.py in the ckanext-myname/ckanext/myname directory of the extension that was created earlier by the paster create -t ckanext ckanext-myname command. At this point CKAN still doesn’t know where to find your plugin, even though the module is installed. To find the plugin it looks up an entry point. An entry point is just a feature of setuptools that links a string in the form package_name.entry_point_name to a particular object in Python code.
Tip: If you are interested in reading a tutorial about entry points see: • http://reinout.vanrees.org/weblog/2010/01/06/zest-releaser-entry-points.html
38 Chapter 8. CKAN Extensions, Plugins Interfaces and Workers ckan Documentation, Release
• http://jimmyg.org/blog/2010/python-setuptools-egg-plugins.html
CKAN finds plugins by searching for entry points in the group ckan.plugin. Entry points are defined in a package’s setup.py file. If you look in the setup.py file for the ckanext-myname extension you’ll see these lines commented out towards the end: entry_points=\ """ [ckan.plugins] # Add plugins here, eg # myplugin=ckanext.myname:PluginClass """,
The entry point will be called without any parameters and must return an instance of ckan.plugins.Plugin. To enable the Disqus plugin uncomment the bottom line and change it to this: disqus_example=ckanext.myname:Disqus
Any time you change the setup.py file you will need to run one of these two commands again before the change will take effect: python setup.py develop python setup.py egg_info
With your entry point in place and installed you can now add the extension to your CKAN config as described earlier. To add our example Disqus plugin you would change your ~/var/srvc/ckan.net/ckan.net.ini config file like this: [app:main] ckan.plugins = disqus_example
Note that the name of the plugin implementation that you give to ckan.plugins is always the same as the name of the entry point you’ve defined in the setup.py file. It is therefore important that you don’t choose an entry point name that is being used by in any existing extension to refer to its plugins. The same extension can have multiple different plugins, all implementing different interfaces to provide useful func- tionality. For each new plugin your extension implements you need to add another entry point (remembering to re-run python setup.py develop if needed). Your users will then need to add the new entry point name to their ckan.plugins config option too.
8.3.4 Writing a database plugin
You’ve seen how to use IConfigurable and IGenshiStreamFilter, here’s another example which imple- ments IMapperExtension to log messages after any record is inserted into the database. from logging import getLogger from ckan.plugins import implements, SingletonPlugin from ckan.plugins import IMapperExtension
log= getLogger(__name__)
class InsertLoggerPlugin(SingletonPlugin): """ Emit a log line when objects are inserted into the database """
8.3. Plugins 39 ckan Documentation, Release
implements(IMapperExtension, inherit=True)
def after_insert(mapper, connection, instance): log.info(’Object %r was inserted’, instance)
8.3.5 Authorization Group Plugins
If you are writing an authorization group plugin you might like to use the code at this URL as a basis: http://bitbucket.org/okfn/ckanextiati/src/tip/ckanext/iati/authz.py
8.3.6 Publishing Your Extension
At this point you might want to share your extension with the public. First check you have chosen an open source license (MIT license is fine for example) and then update the long_description variable in setup.py to explain what the extension does and which entry point names a user of the extension will need to add to their ckan.plugins configuration. Once you are happy, run the following commands to register your extension on the Python Package Index: python setup.py register python setup.py sdist upload
You’ll then see your extension at http://pypi.python.org/pypi. Others will then be able to install your plugin with pip. You should also add a summary of your extension and its entry points to http://wiki.okfn.org/ckan/extensions. You can create a new account if you need to.
8.3.7 Writing a plugin interface
This describes how to add a plugin interface to make core CKAN code pluggable. Suppose you have a class such as this: class DataInput(object):
def accept_new_data(self, data): self.data= data
And you want plugins to hook into accept_new_data to modify the data. You would start by declaring an interface specifying the methods that plugin classes must provide. You would add the following code to ckan/plugins/interfaces.py: class IDataMunger(Interface):
def munge(self, data): return data
Now you can tell this class that its plugins are anything that implements IDataMunger like this: from ckan.plugins import PluginImplementations, IDataMunger class DataInput(object):
plugins= PluginImplementations(IDataMunger)
40 Chapter 8. CKAN Extensions, Plugins Interfaces and Workers ckan Documentation, Release
def accept_new_data(self, data): for plugin in self.plugins: data= plugin.munge(data) self.data= data
Any registered plugins that implement IDataMunger will then be available in your class via self.plugin. See the pyutilib documentation for more information on creating interfaces and plugins. Be aware though that pyutilib uses slightly different terminology. It calls PluginImplementations ExtensionPoint and it calls instances of a plugin object a service.
8.3.8 Testing plugins
When writing tests for your plugin code you will need setup and teardown code similar to the following to ensure that your plugin is loaded while testing: from ckan import plugins
class TestMyPlugin(TestCase):
@classmethod def setup_class(cls): # Use the entry point name of your plugin as declared # in your package’s setup.py plugins.load(’my_plugin’)
@classmethod def teardown_class(cls): plugins.reset()
The exception to using plugins.load() is for when your plug-in is for routes. In this case, the plugin must be configured before the WSGI app is started. Here is an example test set-up: from paste.deploy import appconfig import paste.fixture from ckan.config.middleware import make_app from ckan.tests import conf_dir
class TestMyRoutesPlugin(TestCase):
@classmethod def setup_class(cls): config= appconfig(’config:test.ini’, relative_to=conf_dir) config.local_conf[’ckan.plugins’]=’my_routes_plugin’ wsgiapp= make_app(config.global_conf, **config.local_conf) cls.app= paste.fixture.TestApp(wsgiapp)
At this point you should be able to write your own plugins and extensions together with their tests. Over time we hope to move more functionality out into CKAN extensions.
8.3.9 Ordering of extensions
Caution: The order that extensions are initially loaded is different to the order that their plugins are run.
The order that extensions are initially loaded is as follows:
8.3. Plugins 41 ckan Documentation, Release
1. System plugins (in setup.py under ckan.system_plugins). 2. In order of the plugins specified in the config file: plugins =. 3. If more than one module has a plug-in with the same name specified in the config, then all those are loaded, in the order the modules appear in sys.path. The order that a plugins are run in, for example the order that IRoutes extensions have their before_map method run, is alphabetical by the plugin class. e.g. here is the order for these four extensions:
8.3.10 Plugin API documentation
8.4 The Queue Extension
** Note: the queue extension currently isn’t working correctly. These docs may not work for you** Certain tasks that CKAN performs lend themselves to the use of a queue system. Queue systems can be very simple. At their heart is the idea that you have two separate processes, a publisher and a consumer. The publisher publishes a message of some description to the queue and at another time, the consumer takes that message off the queue and processes it. By writing code that puts things on the queue and then writing workers to take things off, you can build lots of useful functionality. At the moment we are writing facilities to check for broken links and harvest documents from geodata servers. To use the queue in CKAN you need the ckanext-queue package. To install the latest version of ckanext-queue in editable more so that you can look at the source code run: pip install -e hg+http://bitbucket.org/okfn/ckanext-queue#egg=ckanext-queue
You will then see the source code in /pyenv/src/ckanext-queue/ckanext/queue and README file in /pyenv/src/ckanext-queue/README.md. Installing ckanext-queue will also install a worker command you will use in a minute to run workers against the queue. Internally the queue extension uses the carrot library so that we could potentially use different queue backends at some point in the future. For the moment only the AMQP backend is supported so let’s install an AMQP server called RabbitMQ.
8.4.1 Installing RabbitMQ
CentOS
First you need to install Erlang. To do that first install its dependencies: yum install ncurses-devel flex.x86_64 m4.x86_64 openssl-devel.x86_64 unixODBC-devel.x86_64
Install erlang like this:
42 Chapter 8. CKAN Extensions, Plugins Interfaces and Workers ckan Documentation, Release
wget http://www.erlang.org/download/otp_src_R14B.tar.gz tar zxfv otp_src_R14B.tar.gz cd otp_src_R14B LANG=C; export LANG ./configure --prefix=/opt/erlang_R14B make make install
Next download and install RabbitMQ: wget http://www.rabbitmq.com/releases/rabbitmq-server/v2.2.0/rabbitmq-server-2.2.0-1.noarch.rpm rpm -Uhv --no-deps rabbitmq-server-2.2.0-1.noarch.rpm
Finally edit the /etc/init.d/rabbitmq-server script so that it uses the correct path for your erlang install. Change this line PATH=/sbin:/usr/sbin:/bin:/usr/bin to: PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/erlang_R14B/bin:/opt/erlang_R14B/lib
You can start it like this: /etc/init.d/rabbitmq-server start
Ubuntu
Just run: sudo apt-get install rabbitmq-server
8.4.2 Working Directly with Carrot
As you learned earlier, CKAN uses carrot with the pyamqplib backend. Carrot is well documented at this URL: http://ask.github.com/carrot/introduction.html Before you learn how to use the tools that ckanext-queue uses to work with the queue, it is instructive to see a simple example that uses carrot directly. Save this as publisher.py: from carrot.connection import BrokerConnection conn= BrokerConnection( hostname="localhost", port=5672, userid="guest", password="guest", virtual_host="/", ) from carrot.messaging import Publisher publisher= Publisher( connection=conn, exchange=’local’, routing_key=’*’, )
8.4. The Queue Extension 43 ckan Documentation, Release
publisher.send({"import_feed":"http://cnn.com/rss/edition.rss"}) publisher.close()
Now save this as consumer.py: from carrot.connection import BrokerConnection conn= BrokerConnection( hostname="localhost", port=5672, userid="guest", password="guest", virtual_host="/", )
from carrot.messaging import Consumer consumer= Consumer( connection=conn, queue=’local.workerchain’, exchange=’local’, routing_key=’*’, ) def import_feed_callback(message_data, message): feed_url= message_data["import_feed"] print("Got message for: %s"% feed_url) # something importing this feed url # import_feed(feed_url) message.ack() consumer.register_callback(import_feed_callback) # Go into the consumer loop. consumer.wait()
You’ll notice that both examples set up a connection to the same AMQP server with the same settings, in this case running on localhost. These also happen to be the settings that (at the time of writing) ckanext-queue uses by default if you don’t specify other configuration settings. Make sure you have ckanext-queue installed (so that carrot and its dependencies are installed too) then start the consumer: python consumer.py
In a different console run the publisher: python publisher.py
The publisher will quickly exit but if you now switch back to the consumer you’ll see the message was sent to the queue and the consumer recieved it printing this message: Got message for: http://cnn.com/rss/edition.rss
8.4.3 Working with CKANext Queue
Rather than working with carrot publishers and consumers directly, ckanext-queue provides two useful Python objects to help you: ckanext.queue.connection.get_publisher(config) This returns a Publisher instance which has a send() method for adding an item to the queue.
44 Chapter 8. CKAN Extensions, Plugins Interfaces and Workers ckan Documentation, Release
The config object is the same as pylons.config. If you are writing a standalone script, you can obtain a config object from a config file with code similar to this, adjusting the relative_to option as necessary: from paste.deploy import appconfig config= appconfig(’config:development.ini’, relative_to=’pyenv/src/ckan’)
ckanext.queue.worker.Worker This is a base class which you can inherit from. You can override its consume() method to asyn- cronously pull items from the queue to do useful things
Note: To use the queue extension you don’t need to implenent any new plugin interfaces, you just need to use the get_publisher(config).send() method and the Worker class. Of course your own extension might use plugins to hook into other parts of CKAN to get information to put or retireve from the queue.
The worker implementation runs outside the CKAN server process, interacting directly with both the AMQP queue and the CKAN API (to get CKAN data). The Worker class therefore subclasses both the carrot Consumer class and the ckanclient CkanClient class so that your workers can make calls to the running CKAN server via its API.
Writing a Publisher
Here’s a simple publisher. Save it as publish_on_queue.py: from ckanext.queue import connection from paste.deploy import appconfig
import logging logging.basicConfig(level=logging.DEBUG)
config= appconfig(’config:ckan.ini’, relative_to=’.’) publisher= connection.get_publisher(config) publisher.send({"import_feed":"http://cnn.com/rss/edition.rss"}) publisher.close() print "Sent!"
Note that this requires a ckan.ini file relative to the current working directory to run. Here’s what a sample file will look like: [app:main] use = egg:ckan ckan.site_id = local queue.port = 5672 queue.user_id = guest queue.password = guest queue.hostnane = localhost queue.virtual_host = /
The idea here is that publishers you write will be able to use the same settings as CKAN itself would. In the next section you’ll see how these same options and more to a standard CKAN install config file to enable CKAN to use the RabbitMQ queue. With the consumer.py script still running, execute the new script: python publish_on_queue.py
You’ll see that once again the consumer picks up the message.
8.4. The Queue Extension 45 ckan Documentation, Release
Writing a worker
Now let’s replace the consumer with a worker. Each worker should have its own config file. There is an example you can use named worker.cfg in the ckanext-queue source code. If you don’t specify a config file, the defaults will be used.
Tip: Since the worker.cfg and CKAN configuration file are both in INI file format you can also set these variables directly in your CKAN config file and point the worker directly to your CKAN config instead. It will just ignore the CKAN options.
In particular it is worth setting queue.name which will be used internally by RabbitMQ. Here’s a suitable configuration for a worker: [worker] ckan.site_id = local_test ckan.site_url = http://localhost:5000 ckan.api_key = XXX
# queue.name = # queue.routing_key = *
# queue.port = # queue.user_id = # queue.password = # queue.hostname = # queue.virtual_host =
You can run it like this: worker echo -d -c worker.cfg
The echo example comes from ckanext.queue.echo:EchoWorker. It looks like this: from worker import Worker from pprint import pprint
class EchoWorker(Worker):
def consume(self, routing_key, operation, payload): print "Route %s, op %s"% (routing_key, operation) pprint(payload)
The EchoWorker has an entry point registered in ckanext-queue‘s setup.py so that the worker script in pyenv/bin/worker can find it: [ckan.workers] echo = ckanext.queue.echo:EchoWorker
When you run the worker command with the echo worker it looks up this entry point to run the correct code. With the worker still running, try to run the publisher again: python publish_on_queue.py
Once again the message will be output on the command line, this time via the worker. Internally the worker script you just used uses the ckanext.queue.worker.WorkerChain class to run all workers you specify on the command line. You can get a list of all available workers by executing worker with no
46 Chapter 8. CKAN Extensions, Plugins Interfaces and Workers ckan Documentation, Release arguments. # worker WARNING:root:No config file specified, using worker.cfg ERROR:ckanext.queue.worker:No workers. Aborting.
8.4.4 Configuring CKAN to use the queue
Once you have installed RabbitMQ and have it running you need to enable the CKAN queue functionality within CKAN by adding this to your CKAN config file ckan.plugins= queue
You don’t need to specify configuration options to connect to RabbitMQ because the defaults are fine. At this point if you edit a package it should be using the queue. If you have the echo worker running you’ll see the message added to the queue.
Logging
When using the queue with CKAN it is also useful to have logging set up. To get logging working you need to modify your CKAN config file to also include the queue logger. Here’s an example: [loggers] keys= root, queue
[handlers] keys= console
[formatters] keys= generic
[logger_root] level= INFO handlers= console
[logger_queue] level= DEBUG handlers= console qualname= ckanext
You will also need to set this in your CKAN configuration and ensure any workers and producers also set their ckan.site_id to the same value. ckan.site_id= local_test
Now that you know about extensions, plugins and workers you should be able to extend CKAN in lots of new and interesting ways.
8.4. The Queue Extension 47 ckan Documentation, Release
48 Chapter 8. CKAN Extensions, Plugins Interfaces and Workers CHAPTER NINE
CKAN FEEDS
9.1 Introduction
Access to the CKAN metadata is provided in the RESTful API, but to help users in touch with changes to the data, CKAN also provides feeds. The main feed provides a list of recent changes to all packages, and there are also individual feeds for each package.
9.2 Requests
The main feed is at: /revision/list?format=atom&days=1
For example, to see all the changes made to packages on ckan.net in the past day, browse address: http://ckan.net/revision/list?format=atom&days=1 In addition, each package has a feed at: /package/history/PACKAGENAME?format=atom&days=7
You might want to keep track of developments of the Ordnance Survey open data seen here http://ckan.net/package/ordnance_survey, for example. Here is the corresponding URL you’d add to your RSS/Atom Reader: http://ckan.net/package/history/ordnance_survey?format=atom&days=7
9.3 Format
The feeds are in Atom format. Each ‘entry’ in the feed corresponds to a Revision, which is a set of changes. The Package feed is a subset of the Main Feed, with only the Revisions that reflect a change in that particular package. The details given for each Revision are: Field Description updated / Date & time of the change, given in ISO format. published author ‘name’ is login name or IP address. link Link to Revision in the web interface. summary Log message for the revision. Also, for the main feed, details of packages changes, created and deleted. See below.
49 ckan Documentation, Release
Example main feed:
Example package feed for package ‘water_voles’:
9.4 Main feed summary
The main field provides a little more detail to the edits made to packages. In each entry summary, there is a list of packages affected, and the change to each is described as ‘created’, ‘updated’ or ‘deleted’. For example: For this change to have an effect, you need to recreate the egg information, so run: $ python setup.py egg_info 3. Change an option in your CKAN pylons config file to switch to using the new form. For example, your pylons config file will probably be ‘development.ini’ during development, when you ‘paster serve’ your CKAN app for testing. You need to change the ‘package_form’ setting in the ‘[app:main]’ section to the name defined int he entry point. For example: [app:main] ... package_form = my_form group_form = my_group_form package_group_form = my_package_group_form For this to have an effect you may need to restart the pylons (either by restarting the ‘serve’ command or the Apache host). Now go and edit a package and try out the new form! You can also override the config file setting with a URL parameter in your browser. For example you might browse: http://eco.ckan.net/package/edit/water-voles?package_form=my_form 62 Chapter 13. Forms using FormAlchemy (depreciated) CHAPTER FOURTEEN FORM INTEGRATION CKAN provides facility for integrating the package editing forms into another front end 14.1 Form completion redirect It is simple enough for the external front end to link to CKAN’s package creation or edit pages, but once the form is submitted the user needs to be redirected back to the external front end, instead of CKAN’s package read page. This is achieved with a parameter to the CKAN URL. The ‘return URL’ can be specified in two places: 1. passed as a URL encoded value with the parameter “return_to” in the link to CKAN’s form page. 2. specified in the CKAN config key ‘package_new_return_url’ and ‘package_edit_return_url’. (If the ‘return URL’ is given in both ways then the first takes precedence.) Since the ‘return URL’ may need to include the package name, which could be set in the form, CKAN replaces a known placeholder “ 14.1.1 Example An external front end displays a package ‘ontariolandcoverv100’ here: http://datadotgc.ca/dataset/ontariolandcoverv100 It displays a link to edit this package using CKAN’s form, which without the redirect would be: http://ca.ckan.net/package/edit/ontariolandoverv100 On first thought, the return link is: http://datadotgc.ca/dataset/ontariolandcoverv100 But when the user edit’s this package, the name may change. So the return link needs to be: http://datadotgc.ca/dataset/ This is URL encoded to be: 63 ckan Documentation, Release http%3A%2F%2Fdatadotgc.ca%2Fdataset%2F%3CNAME%3E So the edit link becomes: http://ca.ckan.net/package/edit/ontariolandoverv100?return_to=http%3A%2F%2Fdatadotgc.ca%2Fdataset%2F%3CNAME%3E During editing the package, the user changes the name to canadalandcover, presses ‘preview’ and finally ‘commit’. The user is now redirected back to the external front end at: http://datadotgc.ca/dataset/canadalandcover This same functionality could be achieved by this line in the config (ca.ckan.net.ini): ... [app:main] package_edit_return_url = http://datadotgc.ca/dataset/ ... 64 Chapter 14. Form integration CHAPTER FIFTEEN MODEL The structure of the CKAN data is described in the ‘model’. This is in the code at: ckan/model Many of the domain objects are Revisioned and some are Stateful. These are concepts introduced by vdm. 15.1 Migration When edits are made to the model code, then before the code can be used on a CKAN instance with existing data, the existing data has to be migrated. This is achieved with a migration script. CKAN currently uses SqlAlchemy Migrate to manage these scripts. When you deploy new code to a CKAN instance, as part of the process you run any required migration scripts with: paster db upgrade The scripts give their model version numbers in their filenames and are stored here: ckan/migration/versions/ The current version the database is migrated to is also stored in the database. When you run the upgrade, as each migration script is run it prints to the console something like 11->12. If no upgrade is required because it is up to date, then nothing is printed. 15.2 Creating a new migration script A migration script should be checked into CKAN at the same time as the model changes it is related to. Be- fore pushing the changes, ensure the tests pass when running against the migrated model, which requires the --ckan-migration setting - see README.html#migrationtesting. To create a new migration script, create a python file in ckan/migration/versions/ and name it with a prefix numbered one higher than the previous one and some words describing the change. You need to use the special engine provided by the SqlAlchemy Migrate. Here is the standard header for your migrate script: from sqlalchemy import * from migrate import * The migration operations go in the upgrade function: 65 ckan Documentation, Release def upgrade(migrate_engine): metadata= MetaData() metadata.bind= migrate_engine The following process should be followed when doing a migration. This process is here to make the process easier and to validate if any mistakes have been made. 1. Get a dump of the database schema before you add your new migrate scripts. paster db clean paster db upgrade pg_dump -h host -s -f old.sql dbname 2. Get a dump of the database as you have specified it in the model. paster db clean paster db create-test #this makes the database as defined in the model pg_dump -h host -s -f new.sql dbname 3. Get agpdiff (apt-get it). It produces sql it thinks that you need to run on the database in order to get it to the updated schema. apgdiff old.sql new.sql > upgrade.diff (or if you don’t want to install java use http://apgdiff.startnet.biz/diff_online.php) 4. The upgrade.diff file created will have all the changes needed in sql. Delete the drop index lines as they are not created in the model. 5. Put the resulting sql in your migrate script. eg migrate_engine.execute(‘’‘update table ...... ; update table ....’‘’) 6. Do a dump again, then a diff again to see if the the only thing left are drop index statements. 7. run nosetests with –ckan-migration flag. Its that simple. Well almost.. • If you are doing any table/field renaming adding that to your new migrate script first and use this as a base for your diff (i.e add a migrate script with these renaming before 1). This way the resulting sql wont try and drop and recreate the field/table!! • It sometimes drops the foreign key constraints in the wrong order causing an error so you may need to rearrange the order in the resulting upgrade.diff. • If you need to do any data transfer in the migrations then do it between the dropping of the constraints and adding of new ones. • May need to add some tests if you are doing data migrations. An example of a script doing it this way is 034_resource_group_table.py. This script copies the definitions of the original tables in order to do the renaming the tables/fields. In order to do some basic data migration testing extra assertions should be added to the migration script. Examples of this can also be found in 034_resource_group_table.py for example. This statement is run at the top of the migration script to get the count of rows: package_count= migrate_engine.execute(’’’select count( *) from package’’’).first()[0] And the following is run after to make sure that row count is the same: resource_group_after= migrate_engine.execute(’’’select count( *) from resource_group’’’).first()[0] assert resource_group_after == package_count 66 Chapter 15. Model CHAPTER SIXTEEN CKAN AUTHENTICATION AND IDENTIFICATION CKAN operates a delegated authentication model based on openid. Integration of authentication into CKAN is pro- vided by repoze.who and the repoze.who.openid plugin. 67 ckan Documentation, Release 68 Chapter 16. CKAN Authentication and Identification CHAPTER SEVENTEEN CKAN AUTHORIZATION AND ACCESS CONTROL This document gives an overview of CKAN’s authorization capabilities and model in relation to access control. The authentication/identification aspects of access control are dealt with separately in CKAN Authentication and Identifi- cation. 17.1 Overview CKAN implements a fine-grained role-based access control system. In a nutshell: For a particular package (or other protected object) a user can be assigned a role which specifies permitted actions (edit, delete, change permissions, etc.). 17.1.1 Protected Objects and Authorization There are variety of protected objects to which access can be controlled, for example the System, Packages, Package Groups and Authorization Groups. Access control is fine-grained in that it can be set for each individual package, group or authorization group instance. For each protected object there are a set of relevant actions such as create’, ‘admin’, ‘edit’ etc. To facilitate mapping Users and Objects with Actions, Actions are aggregated into a set of roles (e.g. an ‘editor’ role would have ‘edit’ and ‘read’ action). A special role is taken by the System object, which serves as an authorization object for any assignments which do not relate to a specific object. For example, the creation of a package cannot be linked to a specific package instance and is therefore a system operation. To gain further flexibility, users can be assigned to authorization groups. Authz groups are both the object of autho- rization (i.e. one can have several roles with regards to an authz group) and the subject of authorization (i.e. they can be assigned roles on other objects which will apply to their members. The assignment of users and authorization groups to roles on a given protected object (such as a package) can be done by ‘admins’ via the ‘authorization’ tab of the web interface (or by system admins via that interface or the system admin interface). There is also a command-line based authorization manager, detailed below. 17.1.2 Command-line authorization management Although the Admin Extension provides a Web interface for managing authorization, there is a set of more powerful Paster commands commands for fine-grained control. 69 ckan Documentation, Release The roles command will list and modify the assignment of actions to roles: $ paster --plugin=ckan roles -c my.ini list $ paster --plugin=ckan roles -c my.ini deny editor create-package $ paster --plugin=ckan roles -c my.ini allow editor create-package This would first list all role action assignments, then remove the ‘create-package’ action from the ‘editor’ role and finally re-assign that role. Similarly, the rights command will set the authorization roles of a specific user on a given object within the system: $ paster --plugin=ckan rights -c my.ini list Will list all assigned rights within the system. It is recommended to then grep over the resulting output to search for specific object roles. Rights assignment follows a similar pattern: $ paster --plugin=ckan rights -c my.ini make bar admin package:foo This would assign the user named bar the admin role on the package foo. Instead of user names and package names, a variety of different entities can be the subject or object of a role assignment. Some of those include authorization groups, package groups and the system as a whole (system:‘): # make ’chef’ a system-wide admin: $ paster --plugin=ckan rights -c my.ini make chef admin system # allow all members of authz group ’foo’ to edit group ’bar’ $ paster --plugin=ckan rights -c my.ini make agroup:foo edit \ group:bar To revoke one of the roles assigned using make, the remove command is available: $ paster --plugin=ckan rights -c my.ini remove bar admin package:foo For more help on either of these commands, also refer to the paster help. 17.1.3 Roles Each role has a list of permitted actions appropriate for a protected object. Currently there are three basic roles (although you can add others if these defaults do not suit): • reader: can read the object • anon_editor: (anonymous i.e. not logged in) can edit and read the object • editor: can edit, read and create new objects • admin: admin can do anything including: edit, read, delete, update-permissions (change authorizations for that object) When you install a new CKAN extension or upgrade your version of CKAN then new actions may be created, and permissions may given to these basic roles, according to the broad intention of the name of the roles. It is suggested that if the broad idea of these basic roles and their actions are not suitable for your CKAN instance then you create new roles and assign them actions of your own choosing, rather than edit the roles. If the definition of the roles drift from their name then it can be confusing for admins and cause problems for CKAN upgrades and new extensions. 70 Chapter 17. CKAN Authorization and Access Control ckan Documentation, Release 17.1.4 Actions Actions are defined in the Action enumeration in ckan/model/authz.py and currently include: edit, change-state, read, purge, edit-permissions, create-package, create-group, create-authorization-group, read-site, read-user, create-user. Obviously, some of these (e.g. read) have meaning for any type of Domain Object, and some (e.g. create-package) can not be associated with any particular Domain Object, so the Context for Roles with these Actions is system. The read-site action (with System context) is designed to provide/deny access to pages not associated with Domain Objects. This currently includes: • Package Search • Group index • Tags index • Authorization Group index • all requests to the API (on top of any other authorization requirements) There are also some shortcuts that are provided directly by the authorization system (rather than being expressed as subject-object-role tuples): • A user given the admin right for the System object is a ‘System Admin’ and can do any action on any object. (A shortcut for creating a System Admin is by using the paster sysadmin command.) • A user given the admin right for a particular object can do any action to that object. 17.1.5 Publisher mode setup Although ckan.net is forging ahead with the Wikipedia model of allowing anyone to add and improve metadata, some CKAN instances prefer to operate in ‘Publisher mode’ which allows edits only from authorized users. To operate in this mode: 1. Remove the rights for general public to edit existing packages and create new ones.: paster rights remove visitor anon_editor package:all paster rights remove logged_in editor package:all paster rights remove visitor anon_editor system paster rights remove logged_in editor system 2. If logged-in users have already created packages in your system then you may also wish to remove admin rights. e.g.: paster rights remove bob admin package:all 3. Change the default rights for newly created packages. Do this by using these values in your config (.ini file): ckan.default_roles.Package = {“visitor”: [”reader”], “logged_in”: [”reader”]} ckan.default_roles.Group = {“visitor”: [”reader”], “logged_in”: [”reader”]} ckan.default_roles.System = {“visitor”: [”reader”], “logged_in”: [”reader”]} ckan.default_roles.AuthorizationGroup = {“visitor”: [”reader”], “logged_in”: [”reader”]} Note there is also the possibility to restrict package edits by a user’s authorization group. See http://wiki.ckan.net/Publisher_Mode 17.1. Overview 71 ckan Documentation, Release 17.1.6 Examples Example 1: Package ‘paper-industry-stats’: • David Brent is an ‘admin’ • Gareth Keenan is an ‘editor’ • Logged-in is a ‘reader’ (This is a special user, meaning ‘anyone who is logged in’) • Visitor is a ‘reader’ (Another special user, meaning ‘anyone’) That is, Gareth and David can edit this package, but only Gareth can assign roles (privileges) to new team members. Anyone can see (read) the package. Example 2: The current default for new packages is: • the user who creates it is an ‘admin’ • Visitor and Logged-in are both an ‘editor’ and ‘reader’ NB: “Visitor” and “Logged-in” are special “pseudo-users” used as a way of concretely referring to the special sets of users, namely those that are a) not logged-in (“visitor”) and b) logged-in (“Logged-in”) 17.1.7 User Notes When a new package is created, its creator automatically become admin for it. This user can then change permissions for other users. NB: by default any user (including someone who is not logged-in) will be able to read and write. This default can be changed in the CKAN configuration - see default_roles in CKAN Configuration. 17.2 Developer Notes We record tuples of the form: user authorized_group role object levin editor package::warandpeace 17.2.1 Requirements and Use Cases • A user means someone who is logged in. • A visitor means someone who is not logged in. • An protected object is the subject of a permission (either a user or a pseudo-user) • There are roles named: Admin, Reader, Writer 1. A visitor visits a package page and reads the content 2. A visitor visits a package page and edits the package 3. Ditto 1 for a user 4. Ditto 2 for a user 5. On package creation if done by a user and not a visitor then user is made the ‘admin’ 6. An admin of a package adds a user as an admin 72 Chapter 17. CKAN Authorization and Access Control ckan Documentation, Release 7. An admin of a package removes a user as an admin 8. Ditto for admin re. editor 9. Ditto for admin re. reader 10. We wish to be able assign roles to 2 specific entire groups in addition to specific users: ‘visitor’, ‘users’. These will be termed pseudo-users as we do not have AC ‘groups’ as such. 11. The sysadmin alters the assignment of entities to roles for any package 12. A visitor goes to a package where the editor role does not include ‘visitor’ pseudo-user. They are unable to edit the package. 13. Ditto for user where users pseudo-user does not have editor role and user is not an editor for the package 14. Ditto 12 re reader role. 15. Ditto 13 re reader role. 16. Try to edit over REST interface a package for which ‘visitor’ has Editor role, but no API is supplied. Not allowed. Not Yet Implemented • Support for access-related groups • Support for blacklisting 17.2.2 Conceptual Overview Warning: not all of what is described in this conceptual overview is yet fully implemented. • There are Users and (User) Authorization Groups • There are actions which may be performed on “protected objects” such as Package, Group, System • Roles aggregate actions • UserObjectRole which assign users (or Authorization groups) a role on an object (user, role, object). We will often refer to these informally as “permissions”. NB: there is no object explicitly named “Permission”. This is to avoid confusion: a ‘normal’ “Permission” (as in e.g. repoze.what) would correspond to an action-object tuple. This works for the case where protected objects are limited e.g. a few core subsystems like email, admin panel etc). However, we have many protected objects (e.g. one for each package) and we use roles so this ‘normal’ model does not work well. Question: do we require for both Users and UserAuthorizationGroup to be subject of Role or not? Ans: Yes. Why? Consider, situation where I just want to give an individual user permission on a given object (e.g. assigning authz permission for a package)? If I just have UserAuthorizationGroups one would need to create a group just for that individual. This isn’t impossible but consider next how to assign permissions to edit the Authorization Groups? One would need create another group for this but then we have recursion ad infinitum (unless this were especially encompassed in some system level permission or one has some group which is uneditable ...) Thus, one requires both Users and UserAuthorizationGroups to be subject of “permissions”. To summarize the ap- proximate structure we have is: 17.2. Developer Notes 73 ckan Documentation, Release class SubjectOfAuthorization class User class UserAuthorizationGroup class ObjectOfAuthorization class Package class Group class UserAuthorizationGroup ... class SubjectRoleObject subject_of_authorization object_of_authorization role 17.2.3 Determining permissions See ckan.authz.Authorizer.is_authorized 17.3 Comparison with other frameworks and approaches 17.3.1 repoze.what Demo example model: User Group Permission * Users are assigned to groups * Groups are assigned permissions 17.3.2 Capabilities Each possible action-object tuple receive an identifier which we term the “capability”. We would then list tuples (capability_subject, capability). 74 Chapter 17. CKAN Authorization and Access Control CHAPTER EIGHTEEN LOAD TESTING CKAN 18.1 What we did 1. Put db in some standard state and start server: paster db clean && paster db create && paster create-test-data search # alternatively we could use a dump of the live db # psql -h localhost --user tester ckantest < .... # may want to create a production config paster serve development.ini 2. Do tests: # 5 results for this query ab -n 100 -c 10 -e myresults.csv "http://localhost:5000/api/search/package?q=government&all_fields=1" # remote test host # ab -n 100 -c 10 -e myresults.csv "http://test.ckan.net/api/search/package?q=geo&all_fields=1" 3. Examine results, fix, and repeat! 18.2 Research 18.2.1 Testing Tools Apache Benchmarking Tool • http://httpd.apache.org/docs/2.0/programs/ab.html • Well known and standard. • Can run against local or remote but local preferred • ab -n 100 -c 10 -e myresults.csv http://www.okfn.org/ funkload • Mature and python based • http://funkload.nuxeo.org/ • Functional testing primarily but with load testing support 75 ckan Documentation, Release Individual scripts • http://code.google.com/appengine/articles/load_test.html High Performance Servers and Caching Utilities 18.2.2 memcached • http://memcached.org/ • In memory caching with access via a port • Very heavily used 18.2.3 Tornado • http://www.tornadoweb.org/ • Built by friendfeed and open-sourced • Asynchronous 76 Chapter 18. Load Testing CKAN CHAPTER NINETEEN DATABASE DUMPS It’s often useful to allow users to download the complete CKAN database in a dumpfile. For example, you can download ckan.net daily dump at: http://ckan.net/dump/ in JSON format: ckan.net-daily.json.gz 19.1 Creating a dump Your dump script needs to enable the python environment and run the paster command. For example you could create /home/okfn/var/srvc/ckan.net/dump.sh: . /home/okfn/var/srvc/ckan.net/pyenv/bin/activate || exit 1 paster --plugin=ckan db simple-dump-json /home/okfn/var/srvc/ckan.net/dumps/ckan.net-daily.json --config=/home/okfn/var/srvc/ckan.net/ckan.net.ini gzip /home/okfn/var/srvc/ckan.net/dumps/ckan.net-daily.json You could change simple-dump-json to simple-dump-csv if you want CSV format instead of JSON. Or you could run both! These dump functions dump the entire database as it is stored in CKAN, omitting user account details. 19.2 Daily dumps You can set the dump to be created daily with a cron job. Edit your user’s cron config: $ crontab -e Now add a line such as this: 0 21 *** /home/okfn/var/srvc/ckan.net/dump.sh 19.3 Serving the files Some simple additions to the Apache config can serve the files to users in a directory listing. To do this, add these lines to the virtual host config (e.g. /etc/apache2/sites-enabled/ckan.net): Alias /dump/ /home/okfn/var/srvc/ckan.net/dumps/ # Disable the mod_python handler for static files 77 ckan Documentation, Release 78 Chapter 19. Database dumps CHAPTER TWENTY CKAN’S APPROACH TO DEPENDENCIES Warning: This document is still under development, use only if you are a member of the CKAN team who wishes to be an early adopter and are interested in experimenting with virtual machines. 79 ckan Documentation, Release Contents • CKAN’s Approach to Dependencies – Abstract – Overview of CKAN’s Structure from a Packaging Point of View * Understanding The Dependency Difficulty – The Three Different Requires Files for CKAN * Understanding the lucid_present.txt File * Packaging Dependencies Listed in lucid_missing.txt * Packaging Conflicting Python dependencies from lucid_conflicts.txt – Semi-Manual Python Packaging – Understanding .deb files * Broad Structure * Naming Conventions * The postinst and postrm files * The control file * Extra scripts and permissions * Packaging Python libraries – Packaging CKAN Extensions – Understanding How a User Installs from an apt repository – Setting up a CKAN Apt Repository * From Scratch * The Easy Way – Automatic Packaging – Adding a Package to a Repository – Packaging CKAN Itself – Why We use pip rather than install_requires – Packaging CKAN and its dependencies for a production install – Installing a Packaged CKAN-based Site – Testing Your Packaging in a VM – The Release Process – Creating the CKAN Command – Setting up the Repositories – Automatic Packaging – Next Steps – Proposed Changes to CKAN 20.1 Abstract A typical CKAN install can have many dependencies, in the form of required system software such as PostgreSQL, Python libraries such as SQLAlchemy and other CKAN extensions such as ckanext-qa or ckanext-harvest. As such we have to deal with lots of interdependencies which are often different depending on the combinations of features a particular CKAN install requires. There are three audiences we are primarily targetting that our dependency approach is designed to support: • Interested parties who just want to get a CKAN instance installed quickly and easily to test it or to begin contributing • CKAN developers who want to have editable versions of all of the dependant libraries so that they can improve them • System administrators who want to be able to deploy and upgrade CKAN instances quickly, easily and reliably 80 Chapter 20. CKAN’s Approach to Dependencies ckan Documentation, Release • Deployment and test managers who want to be confident that the live system is running off exactly the libraries they have tested and configured in exactly the same way In order to support these three groups, we allow installation of CKAN in two ways: • Development install via pip as described in the README.rst file • Production install on Ubuntu Lucid 10.04 LTS via apt-get install The old instructions for a production deployment can also be followed but these will be deprecated over time as the .deb packaging approach becomes better documented and understood. Virutally all CKAN instances currently run on Ubuntu Lucid 10.04 LTS so we have decided that this is the only officially supported production deployment platform. Of course CKAN will also run on any modern Mac or Linux distribution but if you go down this route you’ll need to do a development install. In this document I’ll explain in detail how and why we package CKAN the way we do. 20.2 Overview of CKAN’s Structure from a Packaging Point of View There are conceptually two main parts to any CKAN installation: • Python libraries such as CKAN itself, SQLAlchemy, owslib and all the CKAN extensions libraries that a par- ticular CKAN or customized CKAN rely on • The configuration and deployment scripts that lead to a running CKAN application (which may be a vanilla CKAN or a client-specific version (eg CKAN, CKAN-DGU, CKAN-DATANL etc) • Third party servers that CKAN relies on (eg Apache, PostgreSQL etc) Luckily all third party servers that CKAN relies on are all ready packaged in Ubuntu Lucid for us so we don’t need to worry about packaging them. We do need to worry about configuring them though. Let’s look more at the other two. Python Libraries The Python libraries CKAN relies on all have their own interdependencies. If the libraries have already been packaged as .deb files we don’t need to worry about them because their dependencies will already be specified in the package. Other libraries usually have their dependencies specified as the install_requires line in their setup.py files. For the sorts of libraries that are already available Configuration and Deployment 20.2.1 Understanding The Dependency Difficulty In the past 20.3 The Three Different Requires Files for CKAN In order to support both a development install and a package-based install it is important that we package the same versions of libraries that we develop against. There are three categories of dependant Python libraries: present Those that already exist as packages in Ubuntu Lucid missing Those that don’t exist as packages in Ubuntu Lucid conflict Those that have a version which is different from the version in Ubuntu Lucid For each of these categories we have a file in the ckan source tree’s requires directory which you can view here. 20.2. Overview of CKAN’s Structure from a Packaging Point of View 81 ckan Documentation, Release 20.3.1 Understanding the lucid_present.txt File The Python dependencies listed in the lucid_present.txt file are pip installable links to the source tree holding the exact versions of the Python dependencies that Ubuntu uses. By running the command below you get development copies of the same software that Ubuntu has packaged: pip install --ignore-installed -r lucid_present.txt We never need to package software in the lucid_present.txt file because it already exists so most of the time you would just install it directly rather than running the command above to get source versions. You can see the packages you would need to install by looking at the comment at the top of the file. At the time of writing it reads: # The CKAN dependencies are already in Lucid and should be installed via # apt-get if you are on that platform. If you are using a different platform # you can install these dependencies via pip instead. # # sudo apt-get install python-psycopg2 python-lxml python-sphinx # sudo apt-get install python-pylons python-formalchemy python-repoze.who # sudo apt-get install python-repoze.who-plugins python-tempita python-zope.interface 20.3.2 Packaging Dependencies Listed in lucid_missing.txt Note: These are already packaged, so you don’t need to package them yourself, this section just describes how you could do if you wanted to. Python dependencies listed in the lucid_missing.txt file are pip installable links to the source tree holding the exact versions of the Python dependencies that CKAN requries. We have an automatic build process which can take these entries and automatically generate Ubuntu packages for them. The resulting packages are then published to our CKAN apt repository so that they can be automatically installed in production environments. To follow the automatic build process to build the missing packages you can do this: sudo apt-get install -y python wget dh-make devscripts build-essential fakeroot cdbs mercurial git-core subversion python-virtualenv virtualenv missing bin/pip install --ignore-installed -r lucid_missing.txt bin/pip install Buildkit BuildKit script will build and place Debian packages in your missing directory. Make sure there is nothing in there that shouldn’t be overwritten by this script. Now run the BuildKit command like this: cd missing bin/python -m buildkit.update_all . For each package you’ll be loaded into vim to edit the changelog. Save and quit when you are done. Names, version numbers and dependencies are automatically generated. Caution: Most of the time you will never use the automatic process above for lazy batch packaging. You’ll more likely generate a single package with explicit version numbers using the buildkit.deb command or build your package manually. Both approaches are described later. 20.3.3 Packaging Conflicting Python dependencies from lucid_conflicts.txt 82 Chapter 20. CKAN’s Approach to Dependencies ckan Documentation, Release Note: These are already packaged, so you don’t need to package them yourself, this section just describes how you could do if you wanted to. Python packages where CKAN depends on a version that is different from the one in the Ubuntu Lucid repositories are handled slightly differently. If we were to simply package them up and make them available the same way we do with missing packages there is a slim chance that any existing software which used the other version of the library would stop working. To avoid the risk of interfering with other software on the system we take the following approach: • Create a python-ckan-deps package with copies of all the libraries we need • Change the python-ckan library to automatically try to import ckan_deps if it can and then adjust the Python’s sys.path just for this instance to use the versions of the libraries in python-ckan-deps in preference to any other versions installed. In this way we can use any arbitrary versions, without introducing conflicts. Caution: The repoze.who sets of libraries are nigh-on impossible to package in this way so we don’t actually package repoze.who.openid at all, even though we need a slightly more recent version. This is such an edge case though that you should just install it manually into the system Python and not worry too much for the time being. To actually build the python-ckan-deps package we follow the semi-manual Python packaging ap- proach described next. (The example in the next section is actually for a CKAN Python extension called python-ckanext-qa but the same process applies). hg clone ckan-deps python -m buildkit.deb /path/to/ckan-deps/.. python-ckan-deps 1.3~01+lucid http://ckan.org 20.4 Semi-Manual Python Packaging The easiest way to package a Python library is with a tool called BuildKit I wrote specfically for the purpose. This section describes how to use it, but even if you don’t want to use BuildKit and prefer to understand the complexities yourself by reading the Understanding .deb files section, please read this section too so you at least understand the naming conventions we are using. pip install buildkit For each Python package you wish to build a .deb file for you run the buildkit.deb command. Here’s an example: python -m buildkit.deb /path/to/virtualenv ckanext-qa 1.3~01+lucid http://ckan.org python-owslib python-ckanext-csw Let’s break this down. python -m buildkit.deb This is just how you invoke the command from the command line /path/to/virtualenv I think this can just be the path to the directory containing the installed source code directory you wish to package, it doesn’t have to be a virtualenv does it? ckanext-qa The lowercase Python package name of the .deb file to be created. 1.3~01+lucid The version number of the package. There are three parts to this: x 1.3 This should always match exactly the version number specified in the setup.py file for the library being packaged. 20.4. Semi-Manual Python Packaging 83 ckan Documentation, Release ~01 This is an incrementing number (starting at 01 each time the version number above changes) which you change every time you re-package the same version of the code to force apt to recognise your new package as being more recent than the old one, even if the underlying code hasn’t changed. +lucid This is a string representing the Debian/Ubuntu distribution that the package targets. The apt reposi- tory doesn’t assign any meaning to it, it is just that in order to eventually support more than one flavour of Debian or Ubuntu, the packages for each must have different filenames in addition to being in a separate part of the apt repo so we begin this convention now. http://ckan.org The homepage for the package, usually ckan.org for ckan extensions. python-owslib python-ckanext-csw ... etc Any extra arguments are treated as the Debian names of dependencies. These always begin python- for Python libraries and would usually follow ckanext- for all CKAN extensions. Tip: You can also specify any other Debian packages here that are dependcies of the software you are packaging but as you’ll see later it is usually best to add such dependencies to the packaged application. See “Packaging CKAN Extensions” for more information. When you run the command you will get your .deb file created. To release an upgrade of a package it must have a higher version number. There is a chance you may want to release a more recent version of a package despite the fact the underlying version number hasn’t changed. For this reason, we always add a ~ character followed by a two digit number to the end of the actual version number as specified in setup.py for the package. For example, if the version number for the ckanext-qa package in the example above is 1.3~01, a package named python-ckanext-qa_1.3~01_amd64.deb would be produced by the command we’ve looked at. Note: All packages that CKAN itself depends on are already packaged according to the settings in the three requires files that from part of the ckan source distribution so you shouldn’t need to use the approach above to package any of them, you should only need to do this for your own extensions or libraries they rely on which aren’t already CKAN dependencies. See “The Three Different Requires Files” for more information on how packaging of the core CKAN dependencies is managed. 20.5 Understanding .deb files 20.5.1 Broad Structure 20.5.2 Naming Conventions The base naming conventions we use for packages are as follows: ckan Uninstalls CKAN, PostgreSQL, Apache etc. It adds the ckan-instance-create command which is then the only thing you need to create a new instance. python-ckan The CKAN Python library packaged from code at http://bitbucket.org/okfn/ckan python-ckanext-* Any CKAN extensions (can be application extensions or library extensions) ckan-* Installs a client specific CKAN application 84 Chapter 20. CKAN’s Approach to Dependencies ckan Documentation, Release 20.5.3 The postinst and postrm files 20.5.4 The control file 20.5.5 Extra scripts and permissions 20.5.6 Packaging Python libraries 20.6 Packaging CKAN Extensions There are two types of CKAN extension: • Client Applications (eg ckanext-dgu, ckanext-datanl etc) • Helpful libraries (eg ckanext-qa, ckanext-harvest, ckanext-queue etc) All CKAN extensions (whether client applications or helpful libraries) are Python libraries and therefore need packag- ing. Their .deb filenames are the same as the Python package names but are always prefixed with python- so that ckanext-dgu becomes python-ckanext-dgu when packaged as a .deb and ckanext-harvest becomes python-ckanext-harvest etc. CKAN extensions which are also client applications generally need to be deployed and therefore need require Apache and PostgreSQL to be installed and configured correctly too. In addition to the python package we therefore also create an application package for the extension which is named ckan- followed by the last part of the extension name. So for ckanext-dgu two packages are created named python-ckanext-dgu and ckan-dgu. This naming may sound slightly inconsistent but it allows a user who wishes to install a DGU CKAN instance to just type the command below: sudo apt-get install ckan-dgu Usually the ckan package will be a dependency of the your client application CKAN extension. When the ckan package is installed it installs python-ckan as a dependency as well as a series of scripts in /usr/bin such as: ckan-create-instance create a new CKAN instance ckan-maintenance-mode put a CKAN intance into or out of maintenence mode (prevent POSTs from the web user interface) In the simple cases, these scripts can then be used in your client application CKAN extension’s postinst script to set up the custom instance. In more complex cases you may write a postinst script from scratch. The postinst script then forms part of the package and is run by the apt system as part of the package installation or upgrade process to configure your CKAN instance. Before we look at how to actually create an apt repository for your packages and how to publish your packages to it, let’s understand what a user of your package will do to install it. 20.7 Understanding How a User Installs from an apt repository A user will follow the following process: First create the file /etc/apt/sources.list.d/okfn.list using this command, replacing ubuntu_ckan_dev with the correct repo you want to use: echo "deb http://apt.okfn.org/ubuntu_ckan_dev lucid universe" | sudo tee /etc/apt/sources.list.d/okfn.list Then add the package key to say you trust packages from this repository: 20.6. Packaging CKAN Extensions 85 ckan Documentation, Release sudo apt-get install wget wget -qO- http://apt.okfn.org/packages.okfn.key | sudo apt-key add - sudo apt-get update Now you can not install a CKAN extension application, just like any other Debian package: sudo apt-get install ckan-std At this point you should have a running instance. You may need to copy across an existing database if you need your instance pre-populated with data. 20.8 Setting up a CKAN Apt Repository Now you’ve seen what a user expects to be able to do, let’s set up the infrastructure to make to make it happen. 20.8.1 From Scratch Our set up is based on Joseph Ruscio’s set up and will allow us to support multiple operating systems if we want to as well as multiple architectures. At the moment we only support Ubuntu Lucid amd64. To help with repository management we use the reprepro tool. Despite the fact that the repositories could be set up for different OSs and versions (eg lenny, lucid etc) we need to make sure that the package names are still unique. This means that we always add the distribution to the version number when we package. The most important detail that AFAIK isn’t covered in any of the tutorials had to do with package naming conventions. The naive assumption (at least on my part) is that you’ll have a different build of your package for each distro/arch combination, and import them into your repository as such. In other words reprepro should track the distro/arch of each import. In actuality, each build’s 20.8.2 The Easy Way Log into the existing CKAN apt repository server and copy an existing directory that already contains the packages you need. For example, to create a repository for a new ckanext-dgu instance you might do: cd /var/packages/ cp -pr lucid dgu-new At this point you have a brand new repo, you can add new packages to it like this: cd dgu-new sudo reprepro includedeb lucid ~/*.deb You can remove them like this from the same directory: sudo reprepro remove lucid python-ckan Any time a change is made you will need to enter the passphrase for the key. 20.9 Automatic Packaging The BuildKit script will build and place Debian packages in your missing directory. Make sure there is nothing in there that shouldn’t be overwritten by this script. 86 Chapter 20. CKAN’s Approach to Dependencies ckan Documentation, Release 20.10 Adding a Package to a Repository 20.11 Packaging CKAN Itself 20.12 Why We use pip rather than install_requires 20.13 Packaging CKAN and its dependencies for a production install 20.14 Installing a Packaged CKAN-based Site 20.15 Testing Your Packaging in a VM 20.16 The Release Process 20.17 Creating the CKAN Command Create a directory named ckan. Then within it create a DEBIAN directory with three files: control: Package: ckan Version: 1.3.2~09 Architecture: amd64 Maintainer: James Gardner postinst: #!/bin/sh set-e # Any commands that happen after install or upgrade go here postrm #!/bin/sh set-e # Any commands that happen after removal or before upgrade go here Then in the ckan directory you add any files you want copied. In this case we want a /usr/bin/ckan-create-instance script so we create the usr directory in the ckan directory at the same level as the DEBIAN directory, then create the bin directory within it and add the script in there. Finally we want to package up the .deb file. From within the ckan directory run this: 20.10. Adding a Package to a Repository 87 ckan Documentation, Release dpkg-deb -b . .. This will create the ../ckan_1.3.2~09_amd64.deb package ready for you to upload to the repo. The ckan package is already created so in reality you will usually be packaging ckan- 20.18 Setting up the Repositories Build individual dependencies like this: python -m buildkit.deb . ckanext-importlib 0.1~02 http://ckan.org python-ckan python -m buildkit.deb . owslib 0.3.2beta~03 http://ckan.org python-lxml python -m buildkit.deb . ckanext-inspire 0.1~03 http://ckan.org python-ckan python -m buildkit.deb . ckanext-spatial 0.1~04 http://ckan.org python-ckan python -m buildkit.deb . ckanext-harvest 0.1~15 http://ckan.org python-ckan python-ckanext-spatial python-carrot python -m buildkit.deb . ckanext-csw 0.3~10 http://ckan.org python-ckanext-harvest python-owslib python-ckan python -m buildkit.deb . ckanext-dgu 0.2~11 http://ckan.org python-ckan python-ckanext-importlib python-ckanext-dgu python-ckanext-csw python-ckan python-ckanext-spatial python-ckanext-inspire python -m buildkit.deb . ckanext-qa 0.1~19 http://ckan.org python-ckan python -m buildkit.deb . ckan 1.4~01 http://ckan.org python-routes python-vdm python-pylons python-genshi python-sqlalchemy python-repoze.who python-repoze.who-plugins python-pyutilib.component.core python-migrate python-formalchemy python-sphinx python-markupsafe python-setuptools python-psycopg2 python-licenses python-ckan-deps There’s a dependency on postfix. Choose internet site and the default hostname unless you know better. Once you have packages you’ll want to put them in a repo. You can do that as described here: • http://joseph.ruscio.org/blog/2010/08/19/setting-up-an-apt-repository/ Then add them like this: cd /var/packages/lucid/ sudo reprepro includedeb lucid ~/*.deb You can remove them like this from the same directory: sudo reprepro remove lucid python-ckan 20.19 Automatic Packaging The BuildKit script will build and place Debian packages in your missing directory. Make sure there is nothing in there that shouldn’t be overwritten by this script. To package everything automatically, run it like this: cd missing bin/python -m buildkit.update_all . For each package you’ll be loaded into vim to edit the changelog. Save and quit when you are done. Names, version numbers and dependencies are automatically generated. You should find all your packages nicely created now. 88 Chapter 20. CKAN’s Approach to Dependencies ckan Documentation, Release 20.20 Next Steps • Delayed updates 20.21 Proposed Changes to CKAN • Change the config file to support file based logging by default • Move who.ini into the config • Add a ckan/wsgi.py for standard DGU deployment • Modify __init__.py to change • No __init__.py in test directory ubuntu@ip-10-226-226-132:/var/packages/dgu-uat$ sudo reprepro includedeb lucid /home/ubuntu/release/2011- 04-18_01/.deb /home/ubuntu/release/2011-04-18_01/ckan_1.3.2~10_amd64.deb: component guessed as ‘uni- verse’ Skipping inclusion of ‘ckan’ ‘1.3.2~10’ in ‘lucid|universe|amd64’, as it has already ‘1.3.4~03’. /home/ubuntu/release/2011-04-18_01/ckan_1.3.2~11_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan’ ‘1.3.2~11’ in ‘lucid|universe|amd64’, as it has already ‘1.3.4~03’. /home/ubuntu/release/2011-04- 18_01/ckan_1.3.4~01_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan’ ‘1.3.4~01’ in ‘lu- cid|universe|amd64’, as it has already ‘1.3.4~03’. /home/ubuntu/release/2011-04-18_01/ckan_1.3.4~02_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan’ ‘1.3.4~02’ in ‘lucid|universe|amd64’, as it has already ‘1.3.4~03’. /home/ubuntu/release/2011-04-18_01/ckan_1.3.4~03_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan’ ‘1.3.4~03’ in ‘lucid|universe|amd64’, as it has already ‘1.3.4~03’. /home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~06_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~06’ in ‘lucid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011- 04-18_01/ckan-dgu_0.2~07_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~07’ in ‘lucid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011-04-18_01/ckan- dgu_0.2~08_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~08’ in ‘lu- cid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~09_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~09’ in ‘lucid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~11_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~11’ in ‘lucid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~12_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~12’ in ‘lucid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011- 04-18_01/ckan-dgu_0.2~13_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~13’ in ‘lucid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011-04-18_01/ckan- dgu_0.2~14_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~14’ in ‘lu- cid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~15_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~15’ in ‘lucid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011-04-18_01/python-ckan_1.3.4~01-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckan’ ‘1.3.4~01-1’ in ‘lucid|universe|amd64’, as it has already ‘1.3.4~02-1’. /home/ubuntu/release/2011-04-18_01/python-ckan_1.3.4~02-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckan’ ‘1.3.4~02-1’ in ‘lucid|universe|amd64’, as it has already ‘1.3.4~02-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-csw_0.3~10-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-csw’ ‘0.3~10-1’ in ‘lucid|universe|amd64’, as it has already ‘0.3~10-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~08-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-dgu’ ‘0.2~08-1’ in ‘lucid|universe|amd64’, as it has already ‘0.2~11-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~09-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-dgu’ ‘0.2~09-1’ in ‘lucid|universe|amd64’, as it has already ‘0.2~11-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~10-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-dgu’ ‘0.2~10-1’ in ‘lucid|universe|amd64’, as it has already ‘0.2~11-1’. 20.20. Next Steps 89 ckan Documentation, Release /home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~11-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-dgu’ ‘0.2~11-1’ in ‘lucid|universe|amd64’, as it has already ‘0.2~11-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~13-1_amd64.deb: component guessed as ‘uni- verse’ Skipping inclusion of ‘python-ckanext-harvest’ ‘0.1~13-1’ in ‘lucid|universe|amd64’, as it has already ‘0.1~15-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~14-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-harvest’ ‘0.1~14-1’ in ‘lucid|universe|amd64’, as it has al- ready ‘0.1~15-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~15-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-harvest’ ‘0.1~15-1’ in ‘lucid|universe|amd64’, as it has already ‘0.1~15-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~01-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-inspire’ ‘0.1~01-1’ in ‘lucid|universe|amd64’, as it has already ‘0.1~03-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~02-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-inspire’ ‘0.1~02-1’ in ‘lucid|universe|amd64’, as it has already ‘0.1~03-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~03-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-inspire’ ‘0.1~03-1’ in ‘lucid|universe|amd64’, as it has already ‘0.1~03-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-qa_0.1~19-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-qa’ ‘0.1~19-1’ in ‘lucid|universe|amd64’, as it has already ‘0.1~19-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-spatial_0.1~01-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-spatial’ ‘0.1~01-1’ in ‘lucid|universe|amd64’, as it has already ‘0.1~04-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-spatial_0.1~03-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-spatial’ ‘0.1~03-1’ in ‘lu- cid|universe|amd64’, as it has already ‘0.1~04-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext- spatial_0.1~04-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-spatial’ ‘0.1~04-1’ in ‘lucid|universe|amd64’, as it has already ‘0.1~04-1’. /home/ubuntu/release/2011-04-18_01/python- owslib_0.3.2beta~03-1_amd64.deb: component guessed as ‘universe’ ERROR: ‘/home/ubuntu/release/2011-04- 18_01/python-owslib_0.3.2beta~03-1_amd64.deb’ cannot be included as ‘pool/universe/p/python-owslib/python- owslib_0.3.2beta~03-1_amd64.deb’. Already existing files can only be included again, if they are the same, but: md5 expected: 3f38d2e844c8d6ec15da6ba51910f3e2, got: ee48427eb11f8152f50f6dc93aeb70d4 sha1 ex- pected: 87cd7724d8d8f0aaeaa24633abd86e02297771d7, got: 8476b1b0e022892ceb8a35f1848818c31d7441bf sha256 expected: 4c9937c78be05dfa5b9dfc85f3a26a51ca4ec0a2d44e8bca530a0c85f12ef400, got: ad3f7458d069a9dd268d144577a7932735643056e45d0a30b7460c38e64057d7 size expected: 57658, got: 57656 /home/ubuntu/release/2011-04-18_01/python-owslib_0.3.2beta~04-1_amd64.deb: component guessed as ‘uni- verse’ Exporting indices... 19A05DDEB16777A2 James Gardner (thejimmyg) 90 Chapter 20. CKAN’s Approach to Dependencies ckan Documentation, Release lists such files, use deleteunreferenced to delete them.) There have been errors! (reverse-i-search)‘delete’: cat src/pip-^Clete-this-directory.txt ubuntu@ip-10-226-226-132:/var/packages/dgu-uat$ sudo reprepro deleteunrefer- enced –help Error: Too many arguments for command ‘deleteunreferenced’! Syntax: reprepro deleteunreferenced There have been errors! ubuntu@ip-10-226-226-132:/var/packages/dgu-uat$ sudo reprepro deleteunreferenced deleting and forgetting pool/universe/p/python-owslib/python-owslib_0.3.2beta~03-1_amd64.deb ubuntu@ip-10- 226-226-132:/var/packages/dgu-uat$ ubuntu@ip-10-226-226-132:/var/packages/dgu-uat$ sudo reprepro includedeb lucid /home/ubuntu/release/2011-04-18_01/.deb /home/ubuntu/release/2011-04-18_01/ckan_1.3.2~10_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan’ ‘1.3.2~10’ in ‘lucid|universe|amd64’, as it has already ‘1.3.4~03’. /home/ubuntu/release/2011-04-18_01/ckan_1.3.2~11_amd64.deb: component guessed as ‘universe’ Skip- ping inclusion of ‘ckan’ ‘1.3.2~11’ in ‘lucid|universe|amd64’, as it has already ‘1.3.4~03’. /home/ubuntu/release/2011- 04-18_01/ckan_1.3.4~01_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan’ ‘1.3.4~01’ in ‘lu- cid|universe|amd64’, as it has already ‘1.3.4~03’. /home/ubuntu/release/2011-04-18_01/ckan_1.3.4~02_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan’ ‘1.3.4~02’ in ‘lucid|universe|amd64’, as it has already ‘1.3.4~03’. /home/ubuntu/release/2011-04-18_01/ckan_1.3.4~03_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan’ ‘1.3.4~03’ in ‘lucid|universe|amd64’, as it has already ‘1.3.4~03’. /home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~06_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~06’ in ‘lucid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011- 04-18_01/ckan-dgu_0.2~07_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~07’ in ‘lucid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011-04-18_01/ckan- dgu_0.2~08_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~08’ in ‘lu- cid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~09_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~09’ in ‘lucid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~11_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~11’ in ‘lucid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~12_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~12’ in ‘lucid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011- 04-18_01/ckan-dgu_0.2~13_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~13’ in ‘lucid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011-04-18_01/ckan- dgu_0.2~14_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~14’ in ‘lu- cid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~15_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘ckan-dgu’ ‘0.2~15’ in ‘lucid|universe|amd64’, as it has already ‘0.2~15’. /home/ubuntu/release/2011-04-18_01/python-ckan_1.3.4~01-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckan’ ‘1.3.4~01-1’ in ‘lucid|universe|amd64’, as it has already ‘1.3.4~02-1’. /home/ubuntu/release/2011-04-18_01/python-ckan_1.3.4~02-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckan’ ‘1.3.4~02-1’ in ‘lucid|universe|amd64’, as it has al- ready ‘1.3.4~02-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-csw_0.3~10-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-csw’ ‘0.3~10-1’ in ‘lucid|universe|amd64’, as it has already ‘0.3~10-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~08-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-dgu’ ‘0.2~08-1’ in ‘lucid|universe|amd64’, as it has already ‘0.2~11-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~09-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-dgu’ ‘0.2~09-1’ in ‘lucid|universe|amd64’, as it has already ‘0.2~11-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~10-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-dgu’ ‘0.2~10-1’ in ‘lucid|universe|amd64’, as it has already ‘0.2~11-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~11-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-dgu’ ‘0.2~11-1’ in ‘lucid|universe|amd64’, as it has already ‘0.2~11-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~13-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-harvest’ ‘0.1~13-1’ in ‘lucid|universe|amd64’, as it has already ‘0.1~15-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~14-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-harvest’ ‘0.1~14-1’ in ‘lucid|universe|amd64’, as it has already ‘0.1~15-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~15-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-harvest’ ‘0.1~15-1’ in ‘lucid|universe|amd64’, as it has already ‘0.1~15-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~01-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-inspire’ ‘0.1~01-1’ in ‘lucid|universe|amd64’, as it has already ‘0.1~03-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~02-1_amd64.deb: component 20.21. Proposed Changes to CKAN 91 ckan Documentation, Release guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-inspire’ ‘0.1~02-1’ in ‘lucid|universe|amd64’, as it has already ‘0.1~03-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~03-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-inspire’ ‘0.1~03-1’ in ‘lucid|universe|amd64’, as it has already ‘0.1~03-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-qa_0.1~19-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-qa’ ‘0.1~19-1’ in ‘lucid|universe|amd64’, as it has already ‘0.1~19-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-spatial_0.1~01-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-spatial’ ‘0.1~01-1’ in ‘lucid|universe|amd64’, as it has already ‘0.1~04-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext-spatial_0.1~03-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-spatial’ ‘0.1~03-1’ in ‘lu- cid|universe|amd64’, as it has already ‘0.1~04-1’. /home/ubuntu/release/2011-04-18_01/python-ckanext- spatial_0.1~04-1_amd64.deb: component guessed as ‘universe’ Skipping inclusion of ‘python-ckanext-spatial’ ‘0.1~04-1’ in ‘lucid|universe|amd64’, as it has already ‘0.1~04-1’. /home/ubuntu/release/2011-04-18_01/python- owslib_0.3.2beta~03-1_amd64.deb: component guessed as ‘universe’ ERROR: ‘/home/ubuntu/release/2011-04- 18_01/python-owslib_0.3.2beta~03-1_amd64.deb’ cannot be included as ‘pool/universe/p/python-owslib/python- owslib_0.3.2beta~03-1_amd64.deb’. Already existing files can only be included again, if they are the same, but: md5 expected: 3f38d2e844c8d6ec15da6ba51910f3e2, got: ee48427eb11f8152f50f6dc93aeb70d4 sha1 ex- pected: 87cd7724d8d8f0aaeaa24633abd86e02297771d7, got: 8476b1b0e022892ceb8a35f1848818c31d7441bf sha256 expected: 4c9937c78be05dfa5b9dfc85f3a26a51ca4ec0a2d44e8bca530a0c85f12ef400, got: ad3f7458d069a9dd268d144577a7932735643056e45d0a30b7460c38e64057d7 size expected: 57658, got: 57656 /home/ubuntu/release/2011-04-18_01/python-owslib_0.3.2beta~04-1_amd64.deb: component guessed as ‘uni- verse’ Exporting indices... 19A05DDEB16777A2 James Gardner (thejimmyg) 92 Chapter 20. CKAN’s Approach to Dependencies CHAPTER TWENTYONE TESTING CKAN IN A VM Warning: This document is still under development, use only if you are a member of the CKAN team who wishes to be an early adopter and are interested in experimenting with virtual machines. If you aren’t running Lucid, you may need to test in a VM. 21.1 Repository cache First set up a cache of the repositories so that you don’t need to fetch packages each time you build a VM: $ sudo apt-get install apt-proxy $ sudo apt-get install uml-utilities Now find your local host’s ethernet port name, IP address, netmask: $ ifconfig eth0 Link encap:Ethernet HWaddr 1c:6f:65:8a:0a:d8 inet addr:50.56.101.208 Bcast:50.56.101.255 Mask:255.255.255.0 i.e.: ethernet port name: eth0 IP address: 50.56.101.208 netmask: 255.255.255.0 These instructions assume these values, so replace them with yours. Now edit your apt-get config: $ sudo vim /etc/apt-proxy/apt-proxy.conf And set these two lines. The first is your host’s IP address: ;; Server IP to listen on address = 50.56.101.208 ;; Server port to listen on port = 9998 Now restart it: $ sudo /etc/init.d/apt-proxy restart 93 ckan Documentation, Release Once this is complete, your (empty) proxy is ready for use on http:// 21.2 VM creation Install the VM program: sudo apt-get install python-vm-builder sudo apt-get install kvm-pxe Now create a directory ~/Vms for your virtual machines. mkdir ~/Vms We’ll use manual bridging and networking rather than relying on the magic provided by libvirt. Out virtual network for the VMs will be 192.168.100.xxx. You can use any number from 2-253 inclusive for the last bit of the IP. This first machine will have the IP address 192.168.100.2. Each virtual machine afterwards must have a unique IP address. First set some variables (use your own IP address for HOST_IP): export THIS_IP="4" export HOST_IP="50.56.101.208" Check your CPU has native VM support: $ kvm-ok If your CPU doesn’t support KVM extensions: 1. change kvm to qemu in the vmbuilder step coming up next 2. use /usr/bin/qemu instead of /usr/bin/kvm Now create the VM: cd ${HOME}/Vms export BASE_IP="192.168.100" sudo vmbuilder kvm ubuntu \ --mem 512 \ --cpus 6 \ --domain ckan_${THIS_IP} \ --dest ckan_${THIS_IP} \ --flavour virtual \ --suite lucid \ --arch amd64 \ --hostname ckan${THIS_IP} \ --user ubuntu \ --pass ubuntu \ --rootpass ubuntu \ --debug -v \ --ip ${BASE_IP}.${THIS_IP} \ --mask 255.255.255.0 \ --net ${BASE_IP}.0 \ --bcast ${BASE_IP}.255 \ --gw ${BASE_IP}.254 \ 94 Chapter 21. Testing CKAN in a VM ckan Documentation, Release --dns ${BASE_IP}.254 \ --proxy http://${HOST_IP}:9998/ubuntu \ --components main,universe \ --addpkg vim \ --addpkg openssh-server \ --addpkg wget This assumes you already have an apt mirror set up on port 9998 as described above and that you are putting everything in ~/Vms. Now for the networking. First check you have forwarding enabled on the host: sudo -s echo "1" > /proc/sys/net/ipv4/ip_forward exit Now save this as ~/Vms/start.sh: #!/bin/bash if [ "X$1" = "X" ] || [ "X$2" = "X" ] || [ "X$3" = "X" ] || [ "X$4" = "X" ] || [ "X$5" = "X" ]; then echo "ERROR: call this script with network device name, tunnel name, amount of memory, number of CPUs and path to the image e.g." echo " $0 eth0 qtap0 512M 4 /home/Vms/ckan_2/tmpKfAdeU.qcow2 [extra args to KVM]" exit 1 fi NETWORK_DEVICE=$1 TUNNEL=$2 MEM=$3 CPUS=$4 IMAGE=$5 EXTRA=$6 MACADDR="52:54:$(dd if=/dev/urandom count=1 2>/dev/null | md5sum | sed ’s/^\(..\)\(..\)\(..\)\(..\).*$/\1:\2:\3:\4/’)"; echo "Creating bridge..." sudo iptables -t nat -A POSTROUTING -o ${NETWORK_DEVICE} -j MASQUERADE sudo brctl addbr br0 sudo ifconfig br0 192.168.100.254 netmask 255.255.255.0 up echo "done." echo "Creating tunnel..." sudo modprobe tun sudo tunctl -b -u root -t ${TUNNEL} sudo brctl addif br0 ${TUNNEL} sudo ifconfig ${TUNNEL} up 0.0.0.0 promisc echo "done." echo "Starting VM ${IMAGE} on ${TUNNEL} via ${NETWORK_DEVICE} with MAC ${MACADDR}..." sudo /usr/bin/kvm -M pc-0.12 -enable-kvm -m ${MEM} -smp ${CPUS} -name dev -monitor pty -boot c -drive file=${IMAGE},if=ide,index=0,boot=on -net nic,macaddr=${MACADDR} -net tap,ifname=${TUNNEL},script=no,downscript=no -serial none -parallel none -usb ${EXTRA} Make it executable: chmod a+x ~/Vms/start.sh Now you can start it along the lines of this: ./start.sh eth0 qtap0 512M 1 /home/james/Vms/ckan_4/tmpuNIv2h.qcow2 Now login: 21.2. VM creation 95 ckan Documentation, Release ssh ubuntu@${BASE_IP}.${THIS_IP} Once in you’ll need some more configuration. Edit /etc/resolv.conf to contain just this (the Google DNS servers, handy to used a fixed IP so that you don’t have to update your resolve.conf each time you move to a different network): nameserver 8.8.8.8 Then change /etc/apt/apt.conf to comment out the proxy line, you may as well get updates directly now. Finally, run this (swapping the repository name for the one you want to test) to allow yourself to install CKAN: sudo apt-get install wget wget -qO- http://apt-alpha.ckan.org/packages.okfn.key | sudo apt-key add - echo "deb http://apt-alpha.ckan.org/debian lucid universe" | sudo tee /etc/apt/sources.list.d/okfn.list sudo apt-get update Now that you have the repo added you can install and test CKAN as normal. Here’s how mine look: ubuntu@ckan4:~$ cat /etc/network/interfaces # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet static address 192.168.100.4 netmask 255.255.255.0 network 192.168.100.0 broadcast 192.168.100.255 gateway 192.168.100.254 dns 192.168.100.254 ubuntu@ckan4:~$ cat /etc/resolv.conf nameserver 8.8.8.8 21.3 VNC access To add ability to VNC into the VM as it runs (useful for debug), add this line to the start.sh command: -vnc :1 Then you can vnc to it when running using a VNC Client. 96 Chapter 21. Testing CKAN in a VM CHAPTER TWENTYTWO SETTING UP BUILDBOT These directions provide some rough information for setting up a build bot on a Lucid machine. 22.1 Apt Installs Install CKAN core dependencies from Lucid distribution (as per README): sudo apt-get install build-essential libxml2-dev libxslt-dev sudo apt-get install wget mercurial postgresql libpq-dev git-core sudo apt-get install python-dev python-psycopg2 python-virtualenv sudo apt-get install subversion Maybe need this too: sudo apt-get install python-include Buildbot software: sudo apt-get install buildbot Deb building software (as per CKAN’s Approach to Dependencies): sudo apt-get install -y dh-make devscripts fakeroot cdbs Fabric: sudo apt-get install -y fabric If you get errors with postgres and locales you might need to do these: sudo apt-get install language-pack-en-base sudo dpkg-reconfigure locales 22.2 Postgres setup If installation before failed to create a cluster, do this after fixing errors: sudo pg_createcluster 8.4 main --start Create users and databases: 97 ckan Documentation, Release sudo -u postgres createuser -S -D -R -P buildslave # set this password (matches buildbot scripts): biomaik15 sudo -u postgres createdb -O buildslave ckan1 sudo -u postgres createdb -O buildslave ckanext 22.3 Buildslave setup Rough commands: sudo useradd -m -s /bin/bash buildslave sudo chown buildslave:buildslave /home/buildslave sudo su buildslave cd ~ hg clone https://[email protected]/okfn/buildbot-scripts . ssh-keygen -t rsa cp /home/buildslave/.ssh/id_rsa.pub ~/.ssh/authorized_keys mkdir -p ckan/build cd ckan/build python ~/ckan-default.py buildbot create-slave ~ localhost:9989 okfn 22.4 Buildmaster setup Rough commands: mkdir ~/buildmaster buildbot create-master ~/buildmaster ln -s /home/buildslave/master/master.cfg ~/buildmaster/master.cfg cd ~/buildmaster buildbot checkconfig 22.5 Startup Setup the daemons for master and slave: sudo vim /etc/default/buildbot This file should be edited to be like this: BB_NUMBER[0]=0 # index for the other values; negative disables the bot BB_NAME[0]="okfn" # short name printed on startup / stop BB_USER[0]="okfn" # user to run as BB_BASEDIR[0]="/home/okfn/buildmaster" # basedir argument to buildbot (absolute path) BB_OPTIONS[0]="" # buildbot options BB_PREFIXCMD[0]="" # prefix command, i.e. nice, linux32, dchroot BB_NUMBER[1]=1 # index for the other values; negative disables the bot 98 Chapter 22. Setting up buildbot ckan Documentation, Release BB_NAME[1]="okfn" # short name printed on startup / stop BB_USER[1]="buildslave" # user to run as BB_BASEDIR[1]="/home/buildslave" # basedir argument to buildbot (absolute path) BB_OPTIONS[1]="" # buildbot options BB_PREFIXCMD[1]="" # prefix command, i.e. nice, linux32, dchroot Start master and slave (according to /etc/default/buildbot): sudo /etc/init.d/buildbot start Now check you can view buildbot at: http://localhost:8010/ 22.6 Connect ports It’s preferable to view the buildbot site at port 80 rather than 8010. If there is no other web service on this machine, you might connect up the addresses using iptables: sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8010 Otherwise it is best to do a reverse proxy. Using apache, edit this file: sudo vim /etc/apache2/sites-available/buildbot.okfn.org to be like this: ProxyPassReverse ts Off Then: sudo apt-get install libapache2-mod-proxy-html sudo a2enmod proxy_http sudo a2ensite buildbot.okfn.org sudo /etc/init.d/apache2 reload 22.6. Connect ports 99 ckan Documentation, Release 100 Chapter 22. Setting up buildbot CHAPTER TWENTYTHREE OTHER MATERIAL 23.1 CKAN CHANGELOG 23.1.1 v1.4.1 2011-06-27 Major: • Refactor Web interface to use logic layer rather than model objects directly (#1078) Minor: • Links in user-supplied text made less attractive to spammers (nofollow) #1181 • Package change notifications - remove duplicates (#1149) • Metadata dump linked to (#1169) • Refactor authorization code to be common across Package, Group and Authorization Group (#1074) Bug fixes • Duplicate authorization roles were difficult to delete (#1083) 23.1.2 v1.4 2011-05-19 Major: • Authorization forms now in grid format (#1074) • Links to RDF, N3 and Turtle metadata formats provided by semantic.ckan.net (#1088) • Refactor internal logic to all use packages in one format - a dictionary (#1046) • A new button for administrators to change revisions to/from a deleted state (#1076) Minor: • Etags caching can now be disabled in config (#840) • Command-line tool to check search index covers all packages (#1073) • Command-line tool to load/dump postgres database (#1067) Bug fixes: • Visitor can’t create packages on new CKAN install - since v1.3.3 (#1090) • OpenID user pages couldn’t be accessed - since v1.3.2 (#1056) 101 ckan Documentation, Release • Default site_url configured to ckan.net, so pages obtains CSS from ckan.net- since v1.3 (#1085) 23.1.3 v1.3.3 2011-04-08 Major: • Authorization checks added to editing Groups and PackageRelationships (#1052) • API: Added package revision history (#1012, #1071) Minor: • API can take auth credentials from cookie (#1001) • Theming: Ability to set custom favicon (#1051) • Importer code moved out into ckanext-importlib repo (#1042) • API: Group can be referred to by ID (in addition to name) (#1045) • Command line tool: rights listing can now be filtered (#1072) Bug fixes: • SITE_READ role setting couldn’t be overridden by sysadmins (#1044) • Default ‘reader’ role too permissive (#1066) • Resource ordering went wrong when editing and adding at same time (#1054) • GET followed by PUTting a package stored an incorrect license value (#662) • Sibling package relationships were shown for deleted packages (#664) • Tags were displayed when they only apply to deleted packages (#920) • API: ‘Last modified’ time was localised - now UTC (#1068) 23.1.4 v1.3.2 2011-03-15 Major: • User list in the Web interface (#1010) • CKAN packaged as .deb for install on Ubuntu • Resources can have extra fields (although not in web interface yet) (#826) • CSW Harvesting - numerous of fixes & improvements. Ready for deployment. (#738 etc) Minor: • Wordpress integration refactored as a Middleware plugin (#1013) • Unauthorized actions lead to a flash message (#366) • Resources Groups to group Resources in Packages (#956) • Plugin interface for authorization (#1011) • Database migrations tested better and corrected (#805, #998) • Government form moved out into ckanext-dgu repo (#1018) • Command-line user authorization tools extended (#1038, #1026) • Default user roles read from config file (#1039) 102 Chapter 23. Other material ckan Documentation, Release Bug fixes: • Mounting of filesystem (affected versions since 1.0.1) (#1040) • Resubmitting a package via the API (affected versions since 0.6?) (#662) • Open redirect (affected v1.3) (#1026) 23.1.5 v1.3 2011-02-18 http://ckan.org/milestone/ckan-v1.3 Highlights of changes: • Package edit form improved: – field instructions (#679) – name autofilled from title (#778) • Group-based access control - Authorization Groups (#647) • Metadata harvest job management (#739, #884, #771) • CSW harvesting now uses owslib (#885) • Package creation authorization is configurable (#648) • Read-only maintenance mode (#777) • Stats page (#832) and importer (#950) moved out into CKAN extensions Minor: • site_title and site_description config variables (#974) • Package creation/edit timestamps (#806) • Caching configuration centralised (#828) • Command-line tools - sysadmin management (#782) • Group now versioned (#231) 23.1.6 v1.2 2010-11-25 http://ckan.org/milestone/ckan-v1.2 Highlights of changes: • Package edit form: attach package to groups (#652) & revealable help • Form API - Package/Harvester Create/New (#545) • Authorization extended: user groups (#647) and creation of packages (#648) • Plug-in interface classes (#741) • WordPress twentyten compatible theming (#797) • Caching support (ETag) (#693) • Harvesting GEMINI2 metadata records from OGC CSW servers (#566) Minor: • New API key header (#466) 23.1. CKAN CHANGELOG 103 ckan Documentation, Release • Group metadata now revisioned (#231) 23.1.7 v1.1 2010-08-10 http://ckan.org/milestone/v1.1 Highlights of changes: • Changes to the database cause notifications via AMQP for clients (#325) • Pluggable search engines (#317), including SOLR (#353) • API is versioned and packages & groups can be referred to by invariant ID (#313) • Resource search in API (#336) • Visual theming of CKAN now easy (#340, #320) • Greater integration with external Web UIs (#335, #347, #348) • Plug-ins can be configured to handle web requests from specified URIs and insert HTML into pages. Minor: • Search engine optimisations e.g. alphabetical browsing (#350) • CSV and JSON dumps improved (#315) 23.1.8 v1.0.2 2010-08-27 • Bugfix: API returns error when creating package (#432) 23.1.9 v1.0.1 2010-06-23 Functionality: • API: Revision search ‘since id’ and revision model in API • API: Basic API versioning - packages specified by ID (#313) • Pluggable search - initial hooks • Customisable templates (#340) and external UI hooks (#335) Bugfixes: • Revision primary key lost in migrating data (#311) • Local authority license correction in migration (#319) • I18n formatting issues • Web i/f searches foreign characters (#319) • Data importer timezone issue (#330) 104 Chapter 23. Other material ckan Documentation, Release 23.1.10 v1.0 2010-05-11 CKAN comes of age, having been used successfully in several deployments around the world. 56 tickets covered in this release. See: http://ckan.org/milestone/v1.0 Highlights of changes: • Package edit form: new pluggable architecture for custom forms (#281, #286) • Package revisions: diffs now include tag, license and resource changes (#303) • Web interface: visual overhaul (#182, #206, #214-#227, #260) including a tag cloud (#89) • i18n: completion in Web UI - now covers package edit form (#248) • API extended: revisions (#251, #265), feeds per package (#266) • Developer documentation expanded (#289, #290) • Performance improved and CKAN stress-tested (#201) • Package relationships (Read-Write in API, Read-Only in Web UI) (#253-257) • Statistics page (#184) • Group edit: add multiple packages at once (#295) • Package view: RDF and JSON formatted metadata linked to from package page (#247) Bugfixes: • Resources were losing their history (#292) • Extra fields now work with spaces in the name (#278, #280) and international characters (#288) • Updating resources in the REST API (#293) Infrastructural: • Licenses: now uses external License Service (‘licenses’ Python module) • Changesets introduced to support distributed revisioning of CKAN data - see doc/distributed.rst for more infor- mation. 23.1.11 v0.11 2010-01-25 Our biggest release so far (55 tickets) with lots of new features and improvements. This release also saw a major new production deployment with the CKAN software powering http://data.gov.uk/ which had its public launch on Jan 21st! For a full listing of tickets see: 23.1. CKAN CHANGELOG 105 ckan Documentation, Release • Minor: – Package undelete (#21, #126) – Automated CKAN deployment via Fabric (#213) – Listings are sorted alphabetically (#195) – Add extras to rest api and to ckanclient (#158 #166) • Infrastructural: – Change to UUIDs for revisions and all domain objects – Improved search performance and better pagination – Significantly improved performance in API and WUI via judicious caching 23.1.12 v0.10 2009-09-30 • Switch to repoze.who for authentication (#64) • Explicit User object and improved user account UI with recent edits etc (#111, #66, #67) • Generic Attributes for Packages (#43) • Use sqlalchemy-migrate to handle db/model upgrades (#94) • “Groups” of packages (#105, #110, #130, #121, #123, #131) • Package search in the REST API (#108) • Full role-based access control for Packages and Groups (#93, #116, #114, #115, #117, #122, #120) • New CKAN logo (#72) • Infrastructural: – Upgrade to Pylons 0.9.7 (#71) – Convert to use formalchemy for all forms (#76) – Use paginate in webhelpers (#118) • Minor: – Add author and maintainer attributes to package (#91) – Change package state in the WUI (delete and undelete) (#126) – Ensure non-active packages don’t show up (#119) – Change tags to contain any character (other than space) (#62) – Add Is It Open links to package pages (#74) 23.1.13 v0.9 2009-07-31 • (DM!) Add version attribute for package • Fix purge to use new version of vdm (0.4) • Link to changed packages when listing revision • Show most recently registered or updated packages on front page • Bookmarklet to enable easy package registration on CKAN 106 Chapter 23. Other material ckan Documentation, Release • Usability improvements (package search and creation on front page) • Use external list of licenses from license repository • Convert from py.test to nosetests 23.1.14 v0.8 2009-04-10 • View information about package history (ticket:53) • Basic datapkg integration (ticket:57) • Show information about package openness using icons (ticket:56) • One-stage package create/registration (r437) • Reinstate package attribute validation (r437) • Upgrade to vdm 0.4 23.1.15 v0.7 2008-10-31 • Convert to use SQLAlchemy and vdm v0.3 (v. major) • Atom/RSS feed for Recent Changes • Package search via name and title • Tag lists show number of associated packages 23.1.16 v0.6 2008-07-08 • Autocompletion (+ suggestion) of tags when adding tags to a package. • Paginated lists for packages, tags, and revisions. • RESTful machine API for package access, update, listing and creation. • API Keys for users who wish to modify information via the REST API. • Update to vdm v0.2 (SQLObject) which fixes ordering of lists. • Better immunity to SQL injection attacks. 23.1.17 v0.5 2008-01-22 • Purging of a Revision and associated changes from cli and wui (ticket:37) • Make data available in machine-usable form via sql dump (ticket:38) • Upgrade to Pylons 0.9.6.* and deploy (ticket:41) • List and search tags (ticket:33) • (bugfix) Manage reserved html characters in urls (ticket:40) • New spam management utilities including (partial) blacklist support 23.1. CKAN CHANGELOG 107 ckan Documentation, Release 23.1.18 v0.4 2007-07-04 • Preview support when editing a package (ticket:36). • Correctly list IP address of of not logged in users (ticket:35). • Improve read action for revision to list details of changed items (r179). • Sort out deployment using modpython. 23.1.19 v0.3 2007-04-12 • System now in a suitable state for production deployment as a beta • Domain model versioning via the vdm package (currently released separately) • Basic Recent Changes listing log messages • User authentication (login/logout) via open ID • License page • Myriad of small fixes and improvements 23.1.20 v0.2 2007-02 • Complete rewrite of ckan to use pylons web framework • Support for full CRUD on packages and tags • No support for users (authentication) • No versioning of domain model objects 23.1.21 v0.1 2006-05 NB: not an official release • Almost functional system with support for persons, packages • Tag support only half-functional (tags are per package not global) • Limited release and file support 108 Chapter 23. Other material CHAPTER TWENTYFOUR INDICES AND TABLES • genindex • modindex • search 109