Rose Documentation Release 2019.01.0
Total Page:16
File Type:pdf, Size:1020Kb
Rose Documentation Release 2019.01.0 Metomi Jan 25, 2019 USER GUIDE 1 Installation 3 2 Getting Started 11 3 Cylc Tutorial 13 4 Rose Tutorial 97 5 Cheat Sheet 173 6 Glossary 177 7 Rose Configuration 191 8 Command Reference 243 9 Rose Environment Variables 273 10 Rose Bash Library 279 11 Rose Built-In Applications 281 12 Rose GTK library 293 13 Rose Macro API 301 14 Rose Upgrade Macro API 307 15 Rosie Web API 311 16 Other 315 HTTP Routing Table 317 Python Module Index 319 Index 321 i ii Rose Documentation, Release 2019.01.0 Rose is a toolkit for writing, editing and running application configurations. What Is Rose (page 97)? Rose uses the Cylc (http://cylc.github.io/cylc/) workflow engine for running suites of inter-dependent applications. What Is Cylc? (page 13) (http://cylc.github.io/cylc/) USER GUIDE 1 Rose Documentation, Release 2019.01.0 2 USER GUIDE CHAPTER ONE INSTALLATION The source code for Rose is available via GitHub (https://github.com/metomi/rose), code releases are available in zip and tar.gz archives (https://github.com/metomi/rose/tags). 1. Un-pack the archive file into an appropriate location on your system. 2. Add the rose/bin/ directory into your PATH environment variable. 3. Check system compatibility by running rose check-software (page 245). 1.1 System Requirements Rose runs on Unix/Linux systems and is known to run on RHEL6 and a number of systems including those documented under metomi-vms (https://github.com/metomi/metomi-vms). System compatibility can be tested using the rose check-software (page 245) command. Required Software • python2 - 2.6.6+, <3 • cylc • py:pygtk - 2.12+ • py:jinja2 Rose Bush • py:cherrypy Rosie • py:cherrypy • py:requests - 2.2.1+ • py:sqlalchemy - 0.6.9+ • svn - 1.8+ • fcm • perl - 5.10.1+ Documentation Builder • python2 - 2.7+, <3 • graphviz-dot • py:sphinx - 1.5.3+ • py:sphinx_rtd_theme - 0.2.4+ • py:sphinxcontrib.httpdomain 3 Rose Documentation, Release 2019.01.0 • py:hieroglyph Documentation Builder - Recommended Extras • rsvg • py:sphinxcontrib.svg2pdfconverter Documentation Builder - PDF Dependencies • tex • latexmk • pdflatex Documentation Builder - Linkcheck Dependencies • py:requests 1.2 Host Requirements Whilst you can install and use Rose & Cylc on a single host (i.e. machine), most installations are likely to be more complex. There are five types of installation: 1.2.1 Hosts For Running Cylc Suites Each user can just run their suites on the host where they issue the rose suite-run (page 261) command. However, the local host may not meet the necessary requirements for connectivity (to the hosts running the tasks) or for availability (the host needs to remain up for the duration of the suite). Therefore, Rose can be configured to run the suites on remote hosts. If multiple hosts are available, by default the host with the lowest system load average will be used. Installation requirements: • Rose, Cylc, Bash, Python, jinja2. • Subversion & FCM (only if you want rose suite-run (page 261) to install files from Subversion using FCM keywords). Connectivity requirements: • Must be able to submit tasks to the hosts which run the suite tasks, either directly or via SSH access to another host. 1.2.2 Hosts For Running Tasks In Suites Installation requirements: • Rose, Cylc, Bash, Python. • Subversion & FCM only if you want to use the Rose install utility to install files from Subversion using FCM keywords. Connectivity requirements: • Must be able to communicate back to the hosts running the Cylc suites via HTTPS (preferred), HTTP or SSH. 4 Chapter 1. Installation Rose Documentation, Release 2019.01.0 1.2.3 Hosts For Running User Interactive Tools Installation requirements: • Rose, Cylc, Bash, Python, requests, Subversion, FCM, Pygraphviz (+ graphviz), PyGTK (+ GTK). Connectivity requirements: • Must have HTTP access to the hosts running the Rosie web service. • Must have access to the Rosie Subversion repositories via the appropriate protocol. • Must have HTTP and SSH access to the hosts running the Cylc suites. • Must share user accounts and $HOME directories with the hosts running the Cylc suites. 1.2.4 Hosts For Running Rose Bush Installation requirements: • Rose, Bash, Python, cherrypy, jinja2. Connectivity requirements: • Must be able to access the home directories of users’ Cylc run directories. 1.2.5 Hosts For Rosie Subversion Repositories And The Rosie Web Services Typically you will only need a single host but you can have multiple repositories on different hosts if you require this. Installation requirements: • Rose, Bash, Python, cherrypy, jinja2, sqlalchemy, Subversion. Note: This section has assumed that you wish to use Rose & Cylc (including their respective GUIs) and use Rosie to store your suites. However, there are many ways of using Rose. For instance: • You can use Rose without using cylc. • You can use Rose & Cylc but not use Rosie. • You can use Rose & Cylc from the command line without using their GUIs. 1.3 Configuring Rose lib/bash/rose_init_site If you do not have root access to install the required Python libraries, you may need to edit this file to ensure Python libraries are included in the PYTHONPATH environment variable. For example, if you have installed some essential Python libraries in your HOME directory, you can add a lib/bash/rose_init_site file with the following contents: # Essential Python libraries installed under # "/home/daisy/usr/lib/python2.6/site-packages/" if [[ -n"${PYTHONPATH:-}" ]]; then PYTHONPATH="/home/daisy/usr/lib/python2.6/site-packages:${PYTHONPATH}" else PYTHONPATH="/home/daisy/usr/lib/python2.6/site-packages" fi export PYTHONPATH 1.3. Configuring Rose 5 Rose Documentation, Release 2019.01.0 etc/rose.conf You should add/edit this file to meet the requirement of your site. Examples can be found at the etc/rose.conf.example file in your Rose distribution. See also rose.conf (page 196) in the API reference. 1.4 Configuring Rosie Client Rosie is an optional suite storage and discovery system. Rosie stores suites using Subversion repositories, with databases behind a web interface for suite discovery and lookup. If users at your site are able to access Rosie services on the Internet or if someone else has already configured Rosie services at your site, all you need to do is configure the client to talk to the servers. Refer to the Configuring a Rosie Server (page 8) section if you need to configure a Rosie server for your site. To set up the Rosie client for the site, add/modify the rose.conf[rosie-id] (page 203) E.g.: [rosie-id] prefix-default=x prefixes-ws-default=xmyorg prefix-location.x=https://somehost.on.the.internet/svn/roses-x prefix-web.x=https://somehost.on.the.internet/trac/roses-x/intertrac/source: prefix-ws.x=https://somehost.on.the.internet/rosie/x prefix-location.myorg=svn://myhost.myorg/roses-myorg prefix-web.myorg=http://myhost.myorg/trac/roses-myorg/intertrac/source: prefix-ws.myorg=http://myhost.myorg/rosie/myorg Check the following: 1. You can access the Rosie Subversion repository without being prompted for a username and a password. This may require configuring Subversion to cache your authentication information with a keyring. (See Subversion Book > Advanced Topics > Network Model > Client Credentials for a discussion on how to do this.) 2. The Rosie web service is up and running and you can access the Rosie web service from your computer. E.g. if the Rosie web service is hosted at https://somehost.on.the.internet/rosie/x, you can check that you have access by typing the following on the command line: curl -I https://somehost.on.the.internet/rosie/x It should return a HTTP code 200. If you are prompted for a username and a password, you may need to have access to a keyring to cache the authentication information. 3. You can access the Rosie web service using the Rosie client. E.g. using the above configuration for the prefix x, type the following on the command line: rosie hello --prefix=x It should return a greeting, e.g. Hello user. 1.5 Deploying Configuration Metadata You may want to deploy Configuration Metadata (page 204) for projects using Rose in a globally readable loca- tion at your site, so that they can be easily accessed by users when using Rose utilities such as rose config-edit (page 248) or rose macro (page 252). 6 Chapter 1. Installation Rose Documentation, Release 2019.01.0 If the source tree of a project is version controlled under a trusted Subversion repository, it is possible to automat- ically deploy their configuration metadata. Assuming that the projects follow our recommendation and store Rose configuration metadata under the rose-meta/ directory of their source tree, you can: • Check out a working copy for each sub-directory under the rose-meta/ directory. • Set up a crontab job to regularly update the working copies. For example, suppose you want to deploy Rose Configuration Metadata (page 204) under /etc/rose-meta/ at your site. You can do: # Deployment location DEST='/etc/rose-meta' cd "${DEST}" # Assume only Rose metadata configuration directories under "rose-meta/" URL1='https://somehost/foo/main/trunk/rose-meta' URL2='https://anotherhost/bar/main/trunk/rose-meta' # ... # Checkout a working copy for each metadata configuration directory for URL in "${URL1}" "${URL2}"; do for NAME in $(svn ls "${URL}"); do svn checkout -q "${URL}/${NAME}" done done # Set up a crontab job to update the working copies, e.g. every 10 minutes crontab -l || true >'crontab.tmp' { echo '# Update Rose configuration metadata every 10 minutes' echo "*/10 **** svn update -q ${DEST}/*" } >>'crontab.tmp' crontab 'crontab.tmp' rm 'crontab.tmp' # Finally add the root level "meta-path" setting to site's "rose.conf" # E.g.