asynciojobs Documentation Release 0.13.2 Thierry Parmentelat Oct 10, 2018 Contents 1 README 3 1.1 A simple orchestration engine for asyncio ..............................3 1.2 Full documentation............................................4 1.3 Prerequisites...............................................4 1.4 Installing.................................................4 1.5 Examples.................................................4 1.6 Nesting schedulers............................................ 16 1.7 Other useful features on the Scheduler class............................. 19 1.8 Troubleshooting............................................. 21 1.9 Customizing jobs............................................. 22 2 The asynciojobs API 25 2.1 The Scheduler classes.......................................... 25 2.2 Job-like classes.............................................. 33 2.3 The Sequence class............................................ 37 2.4 Notes on ordering............................................ 38 2.5 Convenience classes........................................... 38 3 ChangeLog 41 3.1 0.13.2 - 2018 Oct 10........................................... 41 3.2 0.13.1 - 2018 Sep 20........................................... 41 3.3 0.13.0 - 2018 Aug 30........................................... 41 3.4 0.12.11 - 2018 Aug 23.......................................... 41 3.5 0.12.10 - 2018 Jul 5........................................... 41 3.6 0.12.9 - 2018 Jul 5............................................ 42 3.7 0.12.7 - 2018 Jul 4............................................ 42 3.8 0.12.2 - 2018 Jun 14........................................... 42 3.9 0.11.4 - 2018 Jun 12........................................... 42 3.10 0.11.3 - 2018 Jun 12........................................... 42 3.11 0.11.2 - 2018 May 15.......................................... 42 3.12 0.11.1 - 2018 May 10.......................................... 42 3.13 0.10.2 - 2018 May 2........................................... 43 3.14 0.10.1 - 2018 Apr 30........................................... 43 3.15 0.9.1 - 2018 Apr 25............................................ 44 3.16 0.8.2 - 2018 Apr 20............................................ 44 3.17 0.8.1 - 2018 Apr 19............................................ 44 3.18 0.7.1 - 2018 Apr 17............................................ 44 i 3.19 0.6.1 - 2018 Mar 12........................................... 44 3.20 0.6.0 - 2018 Feb 25............................................ 45 3.21 0.5.8 - 2018 Jan 16............................................ 45 3.22 0.5.7 - 2017 Dec 19........................................... 45 3.23 0.5.6 - 2017 Dec 18........................................... 45 3.24 0.5.5 - 2017 Nov 2............................................ 45 3.25 0.5.4 - 2016 Dec 15........................................... 45 3.26 0.5.2 - 2016 Dec 8............................................ 45 3.27 0.5.0 - 2016 Dec 6............................................ 46 3.28 0.4.6 - 2016 Dec 5............................................ 46 3.29 0.4.4 - 2016 Dec 5............................................ 46 3.30 0.4.3 - 2016 Dec 2............................................ 46 3.31 0.4.2 - 2016 Dec 1............................................ 46 3.32 0.4.1 - 2016 Nov 30........................................... 46 3.33 0.4.0 - 2016 Nov 21........................................... 46 3.34 0.3.4 - 2016 Nov 20........................................... 46 3.35 0.3.3 - 2016 Nov 17........................................... 47 3.36 0.3.2 - 2016 Nov 17........................................... 47 3.37 0.3.1 - 2016 Nov 15........................................... 47 3.38 0.2.3 - 2016 Oct 23............................................ 47 3.39 0.2.2 - 2016 Oct 20............................................ 47 3.40 0.2.1 - 2016 Oct 7............................................ 47 3.41 0.2.0 - 2016 Oct 4............................................ 47 3.42 0.1.2 - 2016 Oct 2............................................ 47 3.43 0.1.1 - 2016 Sep 28............................................ 48 3.44 0.1.0 - 2016 Sep 27............................................ 48 3.45 0.0.6 - 2016 Sep 21............................................ 48 3.46 0.0.5 - 2016 Sep 21............................................ 48 3.47 0.0.4 - 2016 Sep 20............................................ 48 3.48 0.0.3 - 2016 Sep 19............................................ 48 3.49 0.0.2 - 2016 Sep 15............................................ 48 3.50 0.0.1 - 2016 Sep 15............................................ 48 4 Indices and tables 49 Python Module Index 51 ii asynciojobs Documentation, Release 0.13.2 Contents: Contents 1 asynciojobs Documentation, Release 0.13.2 2 Contents CHAPTER 1 README 1.1 A simple orchestration engine for asyncio The main and single purpose of this library is to allow for the static description of scenarii involving asyncio- compliant jobs, that have dependencies in the sense that a given job cannot start until its requirements have not completed. So in a nutshell you would: • define a set of Job objects, • together with their requires relationship; that is to say, for each of them, which other jobs need to have completed before this one can be triggered, • and run this logic through an Scheduler object, that will orchestrate the whole scenario. Further features allow to • define a job as critical or not; a critical job that raises an exception causes the orchestration to terminate abruptly; • define a job as running forever, in which case the scheduler of course won’t wait for it, but instead will terminate it when all other jobs are done; • define a global timeout for the whole scheduler; • define a window in terms of a maximal number of simultaneous jobs that are allowed to run; • define nested schedulers: a Scheduler instance being also a job, a scheduler can be inserted in another scheduler just as if it were a re gular job; nested schedulers allow for reusability, since workflow pieces can be for example returned by regular python functions. A job object can be created: • either as a Job instance from a regular asyncio coroutine; • or by specializing the AbstractJob class, and defining its co_run() method; this is for example the case for the SshJob in the apssh library. 3 asynciojobs Documentation, Release 0.13.2 As a convenience, the Sequence class is a helper class that can free you from manually managing the requires deps in long strings of jobs that must run sequentially. 1.2 Full documentation This document, along with asynciojobs’s API reference documentation, and changelog, is available at http:// asynciojobs.readthedocs.io Contact author: thierry dot parmentelat at inria dot fr Licence: CC BY-NC-ND 1.3 Prerequisites asynciojobs requires asyncio and python-3.5 or more recent. import sys major, minor= sys.version_info[:2] if (major, minor)<(3,5): print("asynciojobs won't work in this environment") 1.4 Installing asynciojobs requires python-3.5, and can be installed from the pypi repository: pip3 install asynciojobs 1.4.1 Extra dependency to graphviz This installation method will not try to install the graphviz python package, that can be cumbersome to install, and that is not strictly necessary at run-time for orchestrating a scheduler. We recommend to install it for developping scenarios though since, as we will see shortly, asynciojobs provides a graphical representation of schedulers, that is very convenient for debugging. 1.5 Examples import asyncio In all our examples, we will use the Watch class, it is a helper class that works like a stopwatch; instead of printing the current time, we prefer to display running time from the beginning, which corresponds to the time where the watch instance is created or reset: import time from asynciojobs import Watch watch= Watch() (continues on next page) 4 Chapter 1. README asynciojobs Documentation, Release 0.13.2 (continued from previous page) time.sleep(0.5) watch.print_elapsed('some') 000.000 000.505 some We can now write a simple coroutine for illustrating schedulers through small examples: import time watch= Watch() # just print a message when entering and exiting, and sleep in the middle async def in_out(timeout): global watch watch.print_elapsed("-> in_out({})\n".format(timeout)) await asyncio.sleep(timeout) watch.print_elapsed("<- in_out({})\n".format(timeout)) # return something easy to recognize: the number of milliseconds return 1000 * timeout 000.000 1.5.1 Example A : running in parallel Running a series of coroutines in parallel - a la gather - can be done like this: from asynciojobs import Job, Scheduler a1, a2, a3= Job(in_out(0.1)), Job(in_out(0.2)), Job(in_out(0.25)), What we’re saying here is that we have three jobs, that have no relationships between them. So when we run them, we would start all 3 coroutines at once, and return once they are all done: # this is required in our case because our coroutines # use watch to show time elapsed since reset() watch.reset() sa= Scheduler(a1, a2, a3) sa Scheduler with 0 done+0 ongoing+3 idle=3 job(s) sa.run() 000.016-> in_out(0.1) 000.016-> in_out(0.2) 000.016-> in_out(0.25) 000.119<- in_out(0.1) 000.219<- in_out(0.2) 000.269<- in_out(0.25) (continues on next page) 1.5. Examples 5 asynciojobs Documentation, Release 0.13.2 (continued from previous page) True Note: the run() method is a regular python function, which is easier to illustrate in this README, but in practical terms it is only a wrapper around the