Gruvi Documentation Release 0.10.3

Geert Jansen

Jun 14, 2017

Contents

1 Documentation 3

2 Project ﬁles 45

Python Module Index 49

i ii Gruvi Documentation, Release 0.10.3

Contents 1 Gruvi Documentation, Release 0.10.3

2 Contents CHAPTER 1

Documentation

Rationale

Gruvi is an asynchronous I/O library for Python, just like asyncio, gevent and eventlet. It is therefore a fair to ask why I decided to create a new library. The short and simple answer is that I didn’t agree with some of the decision decision that went into these other projects, and thought it would be good to have a library based on different assumptions. These are the design requirements that I had when creating Gruvi: • I wanted to use green threads. Compared to Posix threads, green threads are very light weight and scalable. Compared to a generator-based yield from approach, they provide better ergonomics where there is just one type of function, rather than two that cannot easily call into each other. • I wanted to use libuv, which is a very high performance asynchronous I/O library famous from node.js, and has excellent cross-platform support including Windows. • I wanted to use a PEP 3156 style internal API based on transports and protocols. This model matches very well with libuv’s completion based model. Compared to a socket based interface, a transport/completion based interface forces a strict separation between “network facing” and “user facing” code. In a socket based interface it is easy to make mistakes such as putting a socket read deep in some protocol code, and it incentives hacks such as monkey patching.

Comparison to other frameworks

The table below compares some of the design decisions and features of Gruvi against asyncio, gevent and eventlet. Note: tables like these necessarily compress a more complex reality into a much more limited set of answers, and they also a snapshot in time. Please assume good faith. If you spot an error in the table, please let me know and I will change it.

3 Gruvi Documentation, Release 0.10.3

Feature Gruvi Asyncio Gevent Eventlet IO library libuv stdlib libev stdlib / libevent IO abstraction Transports / Transports / Protocols Green sockets Green sockets Protocols Threading ﬁbers yield from greenlet greenlet Resolver threadpool threadpool threadpool / blocking / c-ares dnspython Python: 2.x YES (2.7) YES (2.6+, via YES YES Trollius) Python: 3.x YES (3.3+) YES NO NO Python: PyPy NO NO YES YES Platform: Linux FAST FAST FAST FAST Platform: Mac FAST FAST FAST FAST OSX Platform: FAST (IOCP) FAST (IOCP) SLOW (select) SLOW (select) Windows SSL: Posix FAST FAST FAST FAST SSL: Windows FAST (IOCP) FAST (IOCP 3.5+) SLOW (select) SLOW (select) SSL: Contexts YES (also Py2.7) YES (also Py2.6+) NO NO HTTP FAST (via NO (external) SLOW (stdlib) SLOW (stdlib) http-parser) Monkey Patching NO NO YES YES

Motivations for choosing green threads

Green threads have a very low memory overhead, and can therefore support a high level of concurrently. But more importantly, green threads are cooperatively scheduled. This means that a thread switch happens only when speciﬁcally instructed by a switch() call, and with a bit of care, we can write concurrent code that does not require locking. When combining a thread implementation with an IO framework, one of the key design decisions is whether to implement explicit or implicit switching. Green thread switching in Gruvi is implicit. This means that whenever a function would block, for example to wait for data from the network, it will switch to a central scheduler called the hub. The hub will then switch to another green thread if one is ready, or it will run the event loop. A function that can block is just like any other function. It is called as a regular function, and can call other (blocking and not) functions. A common criticism of the implicit switching approach is that the locations where these switches happen, the so-called switch points, are not clearly marked. As a programmer you could call into a function that 3 levels down in the call stack, causes a thread switch. The drawback of this, according to the criticism, is that the switch points could happen essentially anywhere, and that therefore it’s like pre-emptive multi-threading where you need full and careful locking. The alternative to implicit switching is explicit switching. This is the approach taken for example by the asyncio framework. In this approach, every switch point is made explicit, in the case of asyncio because it is not called as a normal function but instead used with the yield from construct. In my view, a big disadvantages of the explicit approach is that the explicit behavior needs to percolate all the way up the call chain: any function that calls a switch point also needs to be called as a switch point. In my view this requires too much up-front planning, and it reduces composability. Rather than implementing explicit switching, Gruvi sticks with implicit switching but tries to address the “unknown switch points” problem, in the following way: • First, it makes explicit that a switch can only happen when you call a function. If you need to change some global state atomically, it is sufﬁcient to do this without making function calls, or by doing it in a leaf function. In this case, it is guaranteed that no switch will happen.

4 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

• Secondly, in case you do need to call out to multiple functions to change a shared global state, then as a programmer you need to make sure, by reading the documentation of the functions, that they do not cause a switch. Gruvi assists you here by marking all switch points with a special decorator. This puts a clear notice in the function’s docstring. In addition, Gruvi also provides a gruvi.assert_no_switchpoints context manager that will trigger an assertion if a switch point does get called within its body. You can use this to be sure that no switch will happen inside a block. • Thirdly, the “traditional” option, Gruvi provides a full set of synchronization primitives including locks that you can use if the two other approaches don’t work. In the end, the difference between implicit and explicit switching is a trade-off. In my view, with the safeguards of Gruvi e.g. the marking of switch points and the gruvi.assert_no_switchpoints context manager, the balance tips in favor of implicit switching. Some people have come to the same conclusion, others to a different one. Both approached are valid and as an programmer you should pick the approach you like most.

Motivations for not doing Monkey patching

One other important design decision in Gruvi that I decided early on is not to implement monkey patching. Monkey patching is an approach employed by e.g. gevent and eventlet where they make the Python standard library cooperative by replacing blocking functions with cooperative functions using runtime patching. In my experience, monkey patching is error prone and fragile. You end up distributing parts of the standard library yourself, bugs included. This is a maintenance burden that I’m not willing to take on. Also the approach is very susceptible to dependency loading order problems, and it only works for code that calls into the blocking functions via Python. Extension modules using e.g. the C-API don’t work, as well as extension modules that use an external library for IO (e.g. psycopg). Finally, monkey patching does not work well with libuv because libuv provides a completion based interface while the standard library assumes a ready-based interface. The solution that Gruvi offers is two-fold: • Either, use Gruvi’s own API if available. For example, Gruvi includes classes to work with streams and processes, and it also provides an excellent HTTP client and server implementation. This is the preferred option. • When integrating with third-party blocking code, run it in the Gruvi maintained thread pool. The easiest way is to call this code via the gruvi.blocking() function.

Installation

Gruvi uses the setuptools, so installation is relatively straightforward. The following Python / OS combinations are supported: OS Python versions Notes Posix 2.7, 3.3+ Only Linux is regularly tested Mac OSX 2.7, 3.3+ PPC is not tested Windows 2.7, 3.3+ No SSL backports on 2.7 Gruvi and some of its dependencies contain C extensions. This means that you need to have a C compiler, and also the Python development ﬁles. Gruvi also uses CFFI so you need to have that installed as well.

Installation using pip

If you have the “pip” package manager, then you can install Gruvi directly from the Python Package Index:

1.2. Installation 5 Gruvi Documentation, Release 0.10.3

$ pip install cffi $ pip install gruvi

You need to install CFFI ﬁrst because the Gruvi setup script depends on it.

Installation from source

The following instructions install Gruvi in a virtualenv development environment:

$ pyvenv gruvi-dev # "virtualenv gruvi-dev" with Python <= 3.3 $ . gruvi-dev/bin/activate $ git clone https://github.com/geertj/gruvi $ cd gruvi $ pip install -r requirements.txt $ python setup.py develop

Linux Speciﬁc Notes

Most mainstream Linux versions have a supported version of Python installed by default, with the notable exception of RHEL/CentOS version 6. For this OS you can use Software Collections to install a supported Python version. To install the required dependencies on Debian/Ubuntu systems:

$ sudo apt-get update $ sudo apt-get install -y gcc python-dev python-virtualenv libffi-dev

On Red Hat/Fedora type systems:

$ sudo yum install -y gcc python-devel python-virtualenv libffi-devel

Mac OSX Speciﬁc Notes

Recent versions of OSX have a supported Python version available. The easiest way to install a C compiler is to install Xcode from the Mac App Store. This is a free download. Since OSX Mavericks, the required libfﬁ dependency has become part of OSX itself, so there’s no need to install it separately. If you are on an earlier version of OSX, it’s recommended to install libfﬁ via Homebrew.

Windows Speciﬁc Notes

Windows does not come with a system provided version of Python, so you have to install one from python.org. It is recommended to use version 3.3 or 3.4. For the C compiler, I recommend to use Microsoft Visual C++ 2010 Express. It is a free download, and it is the same compiler used to create the ofﬁcial Python builds for Windows since version 3.3. You can also use MinGW. In that case make sure you use either Python 2.7.6, 3.3.3, 3.4 or later. These version contain a ﬁx for issue12641.

6 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

Reference

Front Matter

Package Namespace

The API of Gruvi is exposed through the gruvi package. The individual pieces that make up the API are actually deﬁned in submodules, but those are pulled into the package scope by the package’s __init__.py. The symbol names in the gruvi package are designed to be unique and descriptive within the scope of Gruvi, but are not globally unique. Therefore you should use Gruvi as follows:

# Import the package import gruvi gruvi.sleep(1)

# Or.. import individual symbols from gruvi import Process proc= Process()

But never like this:

# Don't use this. It won't work because __all__ is set to [] from gruvi import *

Gruvi includes a few important protocol implementations that are provided under its “batteries includes” philosophy. These are exposed as submodules under the Gruvi namespace. For example:

from gruvi.http import HttpClient client= HttpClient()

External protocols for Gruvi are recommended to install themselves into the gruvi.ext namespace package.

Exceptions

Errors are reported through exceptions. The following base exceptions are defined: exception Error Base class for Gruvi exceptions. exception Timeout A timeout has occurred. exception Cancelled A fiber or calback was cancelled. More exceptions are defined by subsystems and they are documented with their respective subsystem. Functions that are a direct mapping of a pyuv function can also raise a pyuv exception. Gruvi does not try to capture these and transform them into a Gruvi exception, so in these cases you might want to capture the pyuv exception separately. The relevant pyuv base exception is: exception pyuv.error.UVError Base exception for all pyuv errors.

1.3. Reference 7 Gruvi Documentation, Release 0.10.3

Timeouts

Many parts of the Gruvi API use timeouts. Timeouts follow the Python convention that they are expressed in seconds, and can be either passed as an integer or a ﬂoating point argument. The granularity of timeouts in Gruvi is approxi- mately 1 millisecond due to the fact that the libuv API represents timeouts as an integer number of milliseconds. In addition the above, there are two other values for timeout arguments: • None means “no timeout”. • -1 means a default timeout. This arguments is accepted for methods of objects where the constructor allows you to set a default timeout. When a timeout occurs in a function or method, a Timeout exception is raised.

Reference Counting

Memory management and the freeing of unused resources in Gruvi requires a little cooperation from the programmer. In summary, if a object has a close() method, you should call it before you are done with the object. You should never expect it to be called automatically for you by a destructor. If you do not call a close() method, then the memory associated with the object will likely not be fully reclaimed. The reason for this behavior is as follows. Gruvi, as an event-based IO library, uses a central event loop. The event loop contains a reference to all objects for which Gruvi interested in getting events. These events are registered as callbacks back into Gruvi objects. The event loop is provided by libuv, through the pyuv bindings. The libuv framework calls these objects “handles”. Gruvi provides many objects that wrap a pyuv handle. For example, there are objects for a TCP socket or a process. These objects contain a reference to a pyuv handle, and while the handle is active, it has a callback back into the Gruvi object itself. This forms a cyclic reference in which destructors are not called. This means destructors cannot be used to call close() methods automatically. Some magic could be done by using weak references. However, the fact remains that a libuv handle will not be released until it is deactivated (and there’s good reasons for that behavior). So even with weak references, you would still be required to call close(), making the whole point of them moot. Don’t be too concerned though, you will see that in practice it is not a big deal.

Fibers and the Hub

Fibers are light weight execution contexts that are cooperatively scheduled. They form the basis for concurrency in Gruvi. Comparing fibers to threads: • Like threads, fibers represent an independent flow of control in a program. • Fibers are more light weight than threads, meaning you can run more of them. • Unlike threads, there can only ever be one active fiber (per thread, see below). This means that fibers give you concurrency but not parallelism. • Unlike threads, fibers are cooperatively scheduled. They are never preempted and have to yield control other fibers explicitly (called switching). Gruvi uses the python-fibers package as its fiber library. Fibers logically exist within threads. Each fiber has its own Python and C-level stack. When a Python program starts, there will be one thread called 'Main', and this thread will have one fiber called 'Root'. Both are created automatically; the main thread by Python and the root fiber by the python-fibers package.

8 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

Fibers are organized in a tree. Each fiber except the root fiber has a parent. When a fiber exits, either because its main function exits or because of an uncaught exception, control is passed to its parent. If the root fiber exits, the thread will exit.

The hub and ﬁber scheduling

The Gruvi framework maintains a special fiber called the Hub. The hub runs an event loop, and acts as a central fiber scheduler. The event loop in Gruvi is provided by libuv through the pyuv bindings. From a high-level, the flow of a Gruvi program is as follows: 1. Run the current fiber (initially the root fiber). 2. When the fiber needs to wait for something, for example network data, the event to wait for is registered with the event loop, together with a switch back callback. 3. The fiber switches to the hub, which run the event loop. 4. When the event loop detects an event for which a fiber registered interest, it will will call the callback. This causes a switch back to the fiber that installed the event. The hub instance is automatically created when in is first needed. It can be retrieved using the following function: get_hub() Return the instance of the hub. The event loop is available as the .loop property: Hub.loop The event loop used by this hub instance. This is an instance of pyuv.Loop.

Creating ﬁbers and switching

To create a new fiber, instantiate the Fiber class and pass it a main function. The Fiber class is a thin wrapper on top of fibers.Fiber, to which it adds a few behaviors: • The fiber is created as a child of the hub. • Only the hub is allowed to switch to this fiber. This prevent complex interactions where any fiber can switch to any other fiber. In other words, the hub is a real hub. class Fiber(target, args=(), kwargs={}, name=None, hub=None) An cooperatively scheduled execution context aka green thread aka co-routine. The target argument is the main function of the fiber. It must be a Python callable. The args and kwargs specify its arguments and keyword arguments, respectively. The name argument specifies the fiber name. This is purely a diagnositic tool is used e.g. in log messages. The hub argument can be used to override the hub that will be used to schedule this fiber. This argument is used by the unit tests and should not by needed. name The fiber’s name. alive Whether the fiber is alive. start() Schedule the fiber to be started in the next iteration of the event loop.

1.3. Reference 9 Gruvi Documentation, Release 0.10.3

cancel(message=None) Schedule the fiber to be cancelled in the next iteration of the event loop. Cancellation works by throwing a Cancelled exception into the fiber. If message is provided, it will be set as the value of the exception. switchpoint join(timeout=None) Wait until the fiber completes. The only two fibers in a Gruvi program that use fibers.Fiber directly are the hub and the root fiber. All other fibers should be created as instances of Fiber. When a fiber is created it doesn’t run yet. To switch to the fiber, call its start() method and call a function that will switch to the hub:

def say_hello(): print('Hello, there!')

fiber= Fiber(say_hello) fiber.start()

print('Starting fiber') sleep(0) print('Back in root fiber')

The output of this will be:

Starting fiber Hello, there! Back in root fiber

Working with the event loop

To register interest in a certain event, you need to create the appropriate pyuv.Handle instance and add it to the loop. The callback to the handle should cause a switch back to the current fiber. You also want to make sure you implement a timeout on the event, and that you clean up the handle in case it times out. Because this logic can be relatively tricky to get right, Gruvi provides the switch_back context manager for this: class switch_back(timeout=None, hub=None, lock=None) A context manager to facilitate switching back to the current fiber. Instances of this class are callable, and are intended to be used as the callback argument for an asynchronous operation. When called, the switchback object causes Hub.switch() to return in the origin fiber (the fiber that created the switchback object). The return value in the origin fiber will be an (args, kwargs) tuple containing positional and keyword arguments passed to the callback. When the context manager exits it will be deactivated. If it is called after that then no switch will happen. Also the cleanup callbacks are run when the context manager exits. In the example below, a switchback object is used to wait for at most 10 seconds for a SIGHUP signal:

hub= get_hub() with switch_back(timeout=10) as switcher: sigh= pyuv.Signal(hub.loop) sigh.start(switcher, signal.SIGHUP) switcher.add_cleanup(sigh.close) hub.switch()

10 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

The timeout argument can be used to force a timeout after this many seconds. It can be an int or a float. If a timeout happens, Hub.switch() will raise a Timeout exception in the origin fiber. The default is None, meaning there is no timeout. The hub argument can be used to specify an alternate hub to use. This argument is used by the unit tests and should normally not be needed. fiber The origin fiber. timeout The Timeout exception if a timeout has occurred. Otherwise the timeout parameter provided to the constructor. active Whether the switchback object is currently active. switch(value=None) Switch back to the origin fiber. The fiber is switch in next time the event loop runs. throw(typ, val=None, tb=None) Throw an exception into the origin fiber. The exception is thrown the next time the event loop runs. add_cleanup(callback, *args) Add a cleanup action. The callback is run with the provided positional arguments when the context manager exists.

Lockless operation and switchpoints

Functions that may cause a switch to happen are called switch points. In Gruvi these functions are marked by a special decorator: switchpoint(func) Mark func as a switchpoint. In Gruvi, all methods and functions that call Hub.switch() directly, and all public APIs that can cause an indirect switch, are marked as a switchpoint. It is recommended that you mark your own methods and functions in the same way. Example:

@switchpoint def myfunc(): # may call Hub.switch() here

Knowing where switches may happen is important if you need to modify global state in a non-atomic way. If you can be sure that during the modification no switch points are called, then you don’t need any locks. This lockless operation is one of the main benefits of green threads. Gruvi offers a context manager that can help with this: class assert_no_switchpoints(hub=None) Context manager that defines a block in which no switches may happen, and in which no switchpoints may be called. Use it as follows:

with assert_no_switchpoints(): do_something() do_something_else()

If the context manager detects a switch or a call into a switchpoint it raises an AssertionError.

1.3. Reference 11 Gruvi Documentation, Release 0.10.3

The assert_no_switchpoints context manager should not be overused. Instead it is recommended to try and conﬁne non-atomic changes to a global state to single functions.

Utility functions current_fiber() Return the current fiber. Note: The root and hub fiber are “bare” fibers.Fiber instances. Calling this method there returns the bare instance, not a gruvi.Fiber instance. spawn(func, *args, **kwargs) Spawn a new fiber. A new Fiber is created with main function func and positional arguments args. The keyword arguments are passed to the Fiber constructor, not to the main function. The fiber is then scheduled to start by calling its start() method. The fiber instance is returned. switchpoint sleep(secs) Sleep for secs seconds. The secs argument can be an int or a float.

Fiber local data class local Fiber-local data. To manage ﬁber-local data, instantiate this class and store attributes on it:

mydata= local() mydata.x= 10

Attributes have a value or are unset independently for each ﬁber.

Hub reference class Hub The central fiber scheduler and event loop manager. ignore_interrupt = False By default the hub will raise a KeyboardInterrupt in the root fiber when a SIGINT (CTRL-C) is received. Set this to True to ignore SIGINT instead. name The name of the Hub, which is 'Hub'. loop The event loop used by this hub instance. This is an instance of pyuv.Loop. data A per-hub dictionary that can be used by applications to store data. Keys starting with 'gruvi:' are reserved for internal use. poll A centrally managed poller that can be used install callbacks for file descriptor readiness events.

12 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

switchpoint close() Close the hub and wait for it to be closed. This may only be called in the root fiber. After this call returned, Gruvi cannot be used anymore in the current thread. The main use case for calling this method is to clean up resources in a multi-threaded program where you want to exit a thead but not yet the entire process. switch() Switch to the hub. This method pauses the current fiber and runs the event loop. The caller should ensure that it has set up appropriate callbacks so that it will get scheduled again, preferably using switch_back. In this case then return value of this method will be an (args, kwargs) tuple containing the arguments passed to the switch back instance. If this method is called from the root fiber then there are two additional cases. If the hub exited due to a call to close(), then this method returns None. And if the hub exited due to a exception, that exception is re-raised here. run_callback(callback, *args) Queue a callback. The callback will be called with positional arguments args in the next iteration of the event loop. If you add multiple callbacks, they will be called in the order that you added them. The callback will run in the Hub’s fiber. This method is thread-safe: it is allowed to queue a callback from a different thread than the one running the Hub.

Mixing threads and ﬁbers

There are two common situations where you might want to mix threads and fibers: • When running CPU intensive code. In this case, you should run the code the CPU thread pool. • When running third party code that performs blocking IO. In this case, run the code in the IO thread pool. In both cases, running the code in a thread pool allows the hub to continue servicing IO for fibers. All other cases of mixing threads and fibers are generally a bad idea. The following code is Gruvi is thread safe: • The Hub.run_callback() method. • All synchronization primitives in Synchronization primitives. All other code is not thread safe. If, for whatever reason, you must access this code from multiple threads, use locks to mediate access.

Synchronization primitives

Gruvi contains a set of primitives that can be used for synchronization between multiple ﬁbers and threads. All primitives documented below are thread safe. They are modeled after the primitives in the Python threading and queue modules. class Lock A lock. The lock can be locked and unlocked explicitly using acquire() and release(), and it can also be used as a context manager.

1.3. Reference 13 Gruvi Documentation, Release 0.10.3

switchpoint acquire(blocking=True, timeout=None) Acquire the lock. If blocking is true (the default), then this will block until the lock can be acquired. The timeout parameter specifies an optional timeout in seconds. The return value is a boolean indicating whether the lock was acquired. locked() Whether the lock is currently locked. release() Release the lock. class RLock A reentrant lock. A reentrant lock has the notion of a “lock owner” and a “lock count”. If a reentrant lock is acquired, and it was already acquired by the current fiber, then the lock count is increased and the acquire call will be successful. Unlocking a reentrant lock may only be done by the lock owner. The lock becomes unlocked only after it is released as many times as it was acquired. switchpoint acquire(blocking=True, timeout=None) Acquire the lock. If blocking is true (the default), then this will block until the lock can be acquired. The timeout parameter specifies an optional timeout in seconds. The return value is a boolean indicating whether the lock was acquired. locked() Whether the lock is currently locked. release() Release the lock. class Event An event. An event contains an internal flag that is initially False. The flag can be set using the set() method and cleared using the clear() method. Fibers can wait for the flag to become set using wait(). Events are level triggered, meaning that the condition set by set() is “sticky”. Setting the event will unblock all current waiters and will cause future calls to wait() not to block, until clear() is called again. set() Set the internal flag, and wake up any fibers blocked on wait(). clear() Clear the internal flag. switchpoint wait(timeout=None) If the internal flag is set, return immediately. Otherwise block until the flag gets set by another fiber calling set(). class Condition(lock=None) A condition. A condition is always associated with a lock. The state of the condition may only change when the caller has acquired the lock. While the lock is held, a condition can be waited for using wait(). The wait method will release the lock just before blocking itself, so that another fiber can call notify() to notify the condition. The difference between a condition and an Event is that a condition is edge-trigerred. This means that when a condition is notified, only fibers that are waiting at the time of notification are unblocked. Any fiber that calls

14 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

wait() after the notification, will block until the next notification. This also explains why a lock is needed. Without the lock there would be a race condition between notification and waiting. The lock argument can be used to share a lock between multiple conditions. It must be a Lock or RLock instance. If no lock is provided, a RLock is allocated. notify(n=1) Raise the condition and wake up fibers waiting on it. The optional n parameter specifies how many fibers will be notified. By default, one fiber is notified. notify_all() Raise the condition and wake up all fibers waiting on it. switchpoint wait(timeout=None) Wait for the condition to be notified. The return value is True, unless a timeout occurred in which case the return value is False. The lock must be held before calling this method. This method will release the lock just before blocking itself, and it will re-acquire it before returning. switchpoint wait_for(predicate, timeout=None) Like wait() but additionally for predicate to be true. The predicate argument must be a callable that takes no arguments. Its result is interpreted as a boolean value. exception QueueEmpty Queue is empty. exception QueueFull Queue is full. class Queue(maxsize=0) A synchronized FIFO queue. The maxsize argument specifies the maximum queue size. If it is less than or equal to zero, the queue size is infinite. qsize() Return the size of the queue, which is the sum of the size of all its elements. switchpoint put(item, block=True, timeout=None, size=None) Put item into the queue. If the queue is currently full and block is True (the default), then wait up to timeout seconds for space to become available. If no timeout is specified, then wait indefinitely. If the queue is full and block is False or a timeout occurs, then raise a QueueFull exception. The optional size argument may be used to specify a custom size for the item. The total qsize() of the queue is the sum of the sizes of all the items. The default size for an item is 1. put_nowait(item, size=None) “Equivalent of put(item, False). switchpoint get(block=True, timeout=None) Pop an item from the queue. If the queue is not empty, an item is returned immediately. Otherwise, if block is True (the default), wait up to timeout seconds for an item to become available. If not timeout is provided, then wait indefinitely. If the queue is empty and block is false or a timeout occurs, then raise a QueueEmpty exception.

1.3. Reference 15 Gruvi Documentation, Release 0.10.3

get_nowait() “Equivalent of get(False). task_done() Mark a task as done. join() Wait until all tasks are done. class LifoQueue(maxsize=0) A queue with LIFO behavior. See Queue for a description of the API. The maxsize argument specifies the maximum queue size. If it is less than or equal to zero, the queue size is infinite. class PriorityQueue(maxsize=0) A priority queue. Items that are added via put() are typically (priority, item) tuples. Lower values for priority indicate a higher priority. See Queue for a description of the API. The maxsize argument specifies the maximum queue size. If it is less than or equal to zero, the queue size is infinite.

Asynchronous function calls

Gruvi provides functionality to execute functions asynchronously outside the current ﬂow of control. The API is a futures based interface, modeled after the concurrent.futures and asyncio packages in the Python standard library. class Future The state of an asynchronous function call. A future captures the state and the future result of an asynchronous function call. Futures are not instantiated directly but are created by a PoolBase implementation. The pool accepts work in the form of a Python function via its submit() method. This method returns a future representing the state and the result of the function that will be executed asynchronously in the pool. A future progresses through different states during its lifecyle: •Initially the future is in the pending state. •Once the pool has capacity to run the function, it moves to the running state. In this state, running() will return True. •Once the function completes or raises an exception, the future moves to the done state. In this state, done() will return True. The future and the asynchronous function it represents are two distinct entities but for brevity the terms are often used interchangeably, including in this manual. For example, if the asynchronous function completes it is said that the future has completed, and cancelling the future really means cancelling the asynchronous function it is capturing the state of. running() Return whether this future is running. cancelled() Return whether this future was successfully cancelled.

16 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

done() Return whether this future is done. cancel() Cancel the execution of the async function, if possible. This method marks the future as done and sets the Cancelled exception. A future that is not running can always be cancelled. However when a future is running, the ability to cancel it depends on the pool implementation. For example, a fiber pool can cancel running fibers but a thread pool cannot. Return True if the future could be cancelled, False otherwise. switchpoint result(timeout=None) Wait for the future to complete and return its result. If the function returned normally, its return value is returned here. If the function raised an exception, the exception is re-raised here. switchpoint exception(timeout=None) Wait for the async function to complete and return its exception. If the function did not raise an exception this returns None. add_done_callback(callback, *args) Add a callback that gets called when the future completes. The callback will be called in the context of the fiber that sets the future’s result. The callback is called with the positional arguments args provided to this method. The return value is an opaque handle that can be used with remove_done_callback() to remove the callback. If the future has already completed, then the callback is called immediately from this method and the return value will be None. remove_done_callback(handle) Remove a callback that was added by add_done_callback(). It is not an error to remove a callback that was already removed. class PoolBase(maxsize=None, minsize=0, name=None) Base class for pools. A pool contains a set of workers that can execute function calls asynchronously. The maxsize argument specifies the maximum numbers of workers that will be created in the pool. If maxsize is None then the pool can grow without bound. Normally the pool starts with zero workers and grows up to maxsize on demand. The minsize parameter can be used to change this behavior an make sure that there will always be at least this many workers in the pool. The name parameter gives a name for this pool. The pool name will show up in log messages related to the pool.

maxsize The maximum size of this pool. minsize The minimum size of this pool. name The pool name.

1.3. Reference 17 Gruvi Documentation, Release 0.10.3

submit(func, *args) Run func asynchronously. The function is run in the pool which will run it asynchrously. The function is called with positional argument args. The return value is a Future that captures the state and the future result of the asynchronous function call. switchpoint map(func, *iterables, **kwargs) Apply func to the elements of the sequences in iterables. All invocations of func are run in the pool. If multiple iterables are provided, then func must take this many arguments, and is applied with one element from each iterable. All iterables must yield the same number of elements. An optional timeout keyword argument may be provided to specify a timeout. This returns a generator yielding the results. switchpoint join() Wait until all jobs in the pool have completed. New submissions are not blocked. This means that if you continue adding work via submit() or map() then this method might never finish. switchpoint close() Close the pool and wait for all workers to exit. New submissions will be blocked. Workers will exit once their current job is finished. This method will return after all workers have exited. class FiberPool(*args, **kwargs) A pool that uses fiber workers. class ThreadPool(*args, **kwargs) A pool that uses thread workers. get_io_pool() Return the thread pool for IO tasks. By default there is one IO thread pool per application, which is shared with all threads. get_cpu_pool() Return the thread pool for CPU intenstive tasks. By default there is one CPU thread pool per application, which it is shared with all threads. blocking(func, *args, **kwargs) Run a function that uses blocking IO. The function is run in the IO thread pool. switchpoint wait(objects, count=None, timeout=None) Wait for one or more waitable objects. This method waits until count elements from the sequence of waitable objects objects have become ready. If count is None (the default), then wait for all objects to become ready. What “ready” is means depends on the object type. A waitable object is a objects that implements the add_done_callback() and remove_done_callback methods. This currently includes: •Event - an event is ready when its internal flag is set. •Future - a future is ready when its result is set.

18 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

•Fiber - a ﬁber is ready when has terminated. •Process - a process is ready when the child has exited. switchpoint as_completed(objects, count=None, timeout=None) Wait for one or more waitable objects, yielding them as they become ready. This is the iterator/generator version of wait().

Addresses

Addresses in Gruvi are a location to connect to or listen on. Addresses can be local (e.g. when using a pipe), or remote (when using TCP or UDP). The following convention is used throughput Gruvi: • Pipe addresses are represented as str or bytes instances. On Unix these represent paths in the ﬁle system but on Windows there is no such correspondence. On Linux there is a special class of abstract socket addresses that start with a null byte ('\x00'). • IP addresses are represented as (node, service) tuples. The format is identical for IPv4 and IPv6, and both protocols are supported transparently. The host component of the tuple is either DNS name or a string format numerical IPv4/IPv6 address, while the port component is either an integer port number or a service name. In addition, some APIs also support connecting to a ﬁle descriptor or to a pyuv.Stream instance.

Name Resolution

IP (node, service) tuples are resolved to numerical host addresses and port numbers using the getaddrinfo() function. The resulting, resolved address is called a socket address, and should be distinguished from a non-resolved addressed which we simply call “address”. switchpoint getaddrinfo(node, service=0, family=0, socktype=0, protocol=0, flags=0, timeout=30) Resolve an Internet node name and service into a socket address. The family, socktype and protocol are optional arguments that specify the address family, socket type and protocol, respectively. The flags argument allows you to pass flags to further modify the resolution process. See the socket.getaddrinfo() function for a detailed description of these arguments. The return value is a list of (family, socktype, proto, canonname, sockaddr) tuples. The fifth element (sockaddr) is the socket address. It will be a 2-tuple (addr, port) for an IPv4 address, and a 4-tuple (addr, port, flowinfo, scopeid) for an IPv6 address. The address resolution is performed in the libuv thread pool. Note that the distinction between addresses and socket addresses is only present for IP addresses. Pipes don’t use address resolution and their addresses and socket addresses are the same. A socket address can be resolved back to an address using getnameinfo(): switchpoint getnameinfo(sockaddr, flags=0, timeout=30) Resolve a socket address sockaddr back to a (node, service) tuple. The flags argument can be used to modify the resolution process. See the socket.getnameinfo() function for more information. The address resolution is performed in the libuv thread pool.

1.3. Reference 19 Gruvi Documentation, Release 0.10.3

Gruvi API function always accept addresses, and address resolution is performed automatically. The only place where you will work with raw socket addresses is when you query the address of an existing socket, using e.g. the get_extra_info() method of a transport. Note that the ﬁrst two elements of an IP socket address are always the numerical-as-string address and the port number (for both IPv4 and IPv6). Because getaddrinfo() also accepts numerical-as-string addresses and port numbers, they also form a valid address and could be passed back to getaddrinfo() again. Some operating systems support a special convention to specify the scope ID for an IPv6 addressing using the 'host%ifname' notation where 'ifname' is the name of a network interface. Support for this is OS speciﬁc and neither libuv nor Gruvi try to present a portable interface for this.

String Addresses

Gruvi contains two utility functions to transform addresses to and from a string representation. This is useful when logging or when accepting addresses from the command line. saddr(address) Return a string representation for an address. The address paramater can be a pipe name, an IP address tuple, or a socket address. The return value is always a str instance. paddr(address) Parse a string representation of an address. This function is the inverse of saddr().

Transports and protocols (low-level API)

Gruvi is built around the transport/protocol abstraction layers that are documented in PEP 3156 and implemented by asyncio. A transport is a standard interface to a communications channel. Different types of channels implement different interfaces. Since Gruvi uses libuv/ pyuv, its transports are mostly wrappers around the various pyuv.Handle classes. Transport methods are always non-blocking. A protocol is a callback based interface that is connected to a transport. The transport calls speciﬁc callbacks on the protocol when speciﬁc events occur. The callbacks are always non-blocking, but a protocol may expose certain protocol operations are part of their API that are switch points. As a programmer you will use transports and protocols occasionally, but you will mostly use the higher-level interface provided by Gruvi. The exception is when adding support for a new protocol.

Transports

The following transport classes are available: exception TransportError A transport error. class BaseTransport Base class for pyuv based transports. There is no public constructor. start(protocol) Bind to protocol and start calling callbacks on it.

20 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

get_write_buffer_size() Return the total number of bytes in the write buffer. get_write_buffer_limits() Return the write buffer limits as a (low, high) tuple. set_write_buffer_limits(high=None, low=None) Set the low and high watermark for the write buffer. resume_reading() Resume calling callbacks on the protocol. As a relaxation from the requirements in Python asyncio, this method may be called even if reading was already paused, and also if a close is pending. This simplifies protocol design, especially in combination with SSL where reading may need to be enabled by the transport itself, without knowledge of the protocol, to complete handshakes. pause_reading() Pause calling callbacks on the protocol. This method may be called even if reading was already paused or a close is pending. See the note in resume_reading(). close() Close the transport after all oustanding data has been written. abort() Close the transport immediately. get_extra_info(name, default=None) Get transport specific data. The following information is available for all transports: Name Description 'handle' The pyuv handle that is being wrapped. class Transport(handle, server_hostname=None, mode=’rw’) A connection oriented transport. The handle argument is the pyuv handle for which to create the transport. It must be a pyuv.Stream instance, so either a pyuv.TCP, pyuv.Pipe or a pyuv.TTY. The server_hostname argument specifies the host name of the remote peer, if available and applicable for the handle. The mode argument specifies if this is transport is read-only ('r'), write-only ('w') or read-write ('rw'). get_write_buffer_size() Return the total number of bytes in the write buffer. resume_reading() Resume calling callbacks on the protocol. As a relaxation from the requirements in Python asyncio, this method may be called even if reading was already paused, and also if a close is pending. This simplifies protocol design, especially in combination with SSL where reading may need to be enabled by the transport itself, without knowledge of the protocol, to complete handshakes. pause_reading() Pause calling callbacks on the protocol. This method may be called even if reading was already paused or a close is pending. See the note in resume_reading().

1.3. Reference 21 Gruvi Documentation, Release 0.10.3

write(data, handle=None) Write data to the transport. writelines(seq) Write all elements from seq to the transport. write_eof() Shut down the write direction of the transport. can_write_eof() Whether this transport can close the write direction. get_extra_info(name, default=None) Get transport specific data. In addition to the fields from BaseTransport.get_extra_info(), the following information is also available: Name Description 'sockname' The socket name i.e. the result of the getsockname() system call. 'peername' The peer name i.e. the result of the getpeername() system call. 'winsize' The terminal window size as a (cols, rows) tuple. Only available for pyuv.TTY handles. 'unix_creds' The Unix credentials of the peer as a (pid, uid, gid) tuple. Only available for pyuv.Pipe handles on Unix. 'server_hostname'The host name of the remote peer prior to address resolution, if applicable. class DatagramTransport(handle, mode=’rw’) A datagram transport. The handle argument is the pyuv handle for which to create the transport. It must be a pyuv.UDP instance. The mode argument specifies if this is transport is read-only ('r'), write-only ('w') or read-write ('rw'). get_write_buffer_size() Return the total number of bytes in the write buffer. resume_reading() Resume calling callbacks on the protocol. As a relaxation from the requirements in Python asyncio, this method may be called even if reading was already paused, and also if a close is pending. This simplifies protocol design, especially in combination with SSL where reading may need to be enabled by the transport itself, without knowledge of the protocol, to complete handshakes. pause_reading() Pause calling callbacks on the protocol. This method may be called even if reading was already paused or a close is pending. See the note in resume_reading(). sendto(data, addr=None) Send a datagram containing data to addr. The addr argument may be omitted only if the handle was bound to a default remote address.

SSL/TLS support

SSL and TLS support is available by means of a special SslTransport transport:

22 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3 class SslTransport(handle, context, server_side, server_hostname=None, do_handshake_on_connect=True, close_on_unwrap=True) An SSL/TLS transport. The handle argument is the pyuv handle on top of which to layer the SSL transport. It must be a pyuv.Stream instance, so either a pyuv.TCP, pyuv.Pipe or a pyuv.TTY. SSL transports are always read-write, so the handle provided needs to support that. The context argument specifies the ssl.SSLContext to use. You can use create_default_context() to create a new context which also works on Python 2.x where ssl. SSLContext does not exist. The server_side argument specifies whether this is a server side or a client side transport. The optional server_hostname argument can be used to specify the hostname you are connecting to. You may only specify this parameter if your Python version supports SNI. The optional do_handshake_on_connect argument specifies whether to to start the SSL handshake immediately. If False, then the connection will be unencrypted until do_handshake() is called. The default is to start the handshake immediately. The optional close_on_unwrap argument specifies whether you want the ability to continue using the connection after you call unwrap() of when an SSL “close_notify” is received from the remote peer. The default is to close the connection. get_extra_info(name, default=None) Return transport specific data. The following fields are available, in addition to the information exposed by Transport. get_extra_info(). Name Description 'ssl' The internal ssl.SSLObject instance used by this transport. 'sslctx' The ssl.SSLContext instance used to create the SSL object. pause_reading() Stop reading data. Flow control for SSL is a little bit more complicated than for a regular Transport because SSL handshakes can occur at any time during a connection. These handshakes require reading to be enabled, even if the application called pause_reading() before. The approach taken by Gruvi is that when a handshake occurs, reading is always enabled even if pause_reading() was called before. I believe this is the best way to prevent complex read side buffering that could also result in read buffers of arbitrary size. The consequence is that if you are implementing your own protocol and you want to support SSL, then your protocol should be able to handle a callback even if it called pause_reading() before. resume_reading() Resume reading data. See the note in pause_reading() for special considerations on flow control with SSL. do_handshake() Start the SSL handshake. This method only needs to be called if this transport was created with do_handshake_on_connect set to False (the default is True). The handshake needs to be synchronized between the both endpoints, so that SSL record level data is not incidentially interpreted as plaintext. Usually this is done by starting the handshake directly after a connection is established, but you can also use an application level protocol.

1.3. Reference 23 Gruvi Documentation, Release 0.10.3

unwrap() Remove the security layer. Use this method only if you want to send plaintext data on the connection after the security layer has been removed. In all other cases, use close(). If the unwrap is initiated by us, then any data sent after it will be buffered until the corresponding close_notify response is received from our peer. If the unwrap is initiated by the remote peer, then this method will acknowledge it. You need an application level protocol to determine when to do this because the receipt of a close_notify is not communicated to the application. close() Cleanly shut down the SSL protocol and close the transport. create_default_context(purpose=None, **kwargs) Create a new SSL context in the most secure way available on the current Python version. See ssl. create_default_context().

Protocols

A Protocols is a collection of named callbacks. A protocol is are attached to a transport, and its callbacks are called by the transport when certain events happen. The following protocol classes are available: exception ProtocolError A protocol error. class BaseProtocol(timeout=None) Base class for all protocols. The timeout argument speciﬁes a default timeout for various protocol operations. connection_made(transport) Called when a connection is made. connection_lost(exc) Called when a connection is lost. pause_writing() Called when the write buffer in the transport has exceeded the high water mark. The protocol should stop writing new data. resume_writing() Called when the write buffer in the transport has fallen below the low water mark. The protocol can start writing data again. class Protocol(timeout=None) Base class for connection oriented protocols. The timeout argument speciﬁes a default timeout for various protocol operations. data_received(data) Called when a new chunk of data is received. eof_received() Called when an EOF is received. class DatagramProtocol(timeout=None) Base classs for datagram oriented protocols.

24 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

The timeout argument speciﬁes a default timeout for various protocol operations. datagram_received(data, addr) Called when a new datagram is received. error_received(exc) Called when an error has occurred. The following class does not exist in PEP 3156 but is a useful base class for most protocols: class MessageProtocol(message_handler=None, timeout=None) Base class for message oriented protocols.

Creating transports and protocols

Transports and protocols operate in pairs; there is always exactly one protocol for each transport. A new transport/protocol pair can be created using the factory functions below: switchpoint create_connection(protocol_factory, address, ssl=False, server_hostname=None, lo- cal_address=None, family=0, flags=0, ipc=False, timeout=None, mode=’rw’) Create a new client connection. This method creates a new pyuv.Handle, connects it to address, and then waits for the connection to be established. When the connection is established, the handle is wrapped in a transport, and a new protocol instance is created by calling protocol_factory. The protocol is then connected to the transport by calling the transport’s start() method which in turn calls connection_made() on the protocol. Finally the results are returned as a (transport, protocol) tuple. The address may be either be a string, a (host, port) tuple, a pyuv.Stream handle or a file descriptor: •If the address is a string, this method connects to a named pipe using a pyuv.Pipe handle. The address specifies the pipe name. •If the address is a tuple, this method connects to a TCP/IP service using a pyuv.TCP handle. The first element of the tuple specifies the IP address or DNS name, and the second element specifies the port number or service name. •If the address is a pyuv.Stream instance, it must be an already connected stream. •If the address is a file descriptor, then it is attached to a pyuv.TTY stream if os.isatty() returns true, or to a pyuv.Pipe instance otherwise. The ssl parameter indicates whether an SSL/TLS connection is desired. If so then an ssl.SSLContext instance is used to wrap the connection using the ssl module’s asynchronous SSL support. The context is created as follows. If ssl is an SSLContext instance, it is used directly. If it is a function, it is called with the connection handle as an argument and it must return a context . If it is True then a default context is created using gruvi.create_default_context(). To disable SSL (the default), pass False. If SSL is active, the return transport will be an SslTransport instance, otherwise it will be a Transport instance. The server_hostname parameter is only relevant for SSL connections, and specifies the server hostname to use with SNI (Server Name Indication). If no server hostname is provided, the hostname specified in address is used, if available. The local_address keyword argument is relevant only for TCP transports. If provided, it specifies the local address to bind to. The family and flags keyword arguments are used to customize address resolution for TCP handles as described in socket.getaddrinfo().

1.3. Reference 25 Gruvi Documentation, Release 0.10.3

The mode parameter specifies if the transport should be put in read-only ('r'), write-only ('w') or read-write ('rw') mode. For TTY transports, the mode must be either read-only or write-only. For all other transport the mode should usually be read-write. switchpoint create_server(protocol_factory, address=None, ssl=False, family=0, flags=0, ipc=False, backlog=128) Create a new network server. This creates one or more pyuv.Handle instances bound to address, puts them in listen mode and starts accepting new connections. For each accepted connection, a new transport is created which is connected to a new protocol instance obtained by calling protocol_factory. The address argument may be either be a string, a (host, port) tuple, or a pyuv.Stream handle: •If the address is a string, this method creates a new pyuv.Pipe instance and binds it to address. •If the address is a tuple, this method creates one or more pyuv.TCP handles. The first element of the tuple specifies the IP address or DNS name, and the second element specifies the port number or service name. A transport is created for each resolved address. •If the address is a pyuv.Stream handle, it must already be bound to an address. The ssl parameter indicates whether SSL should be used for accepted connections. See create_connection() for a description. The family and flags keyword arguments are used to customize address resolution for TCP handles as described in socket.getaddrinfo(). The ipc parameter indicates whether this server will accept new connections via file descriptor passing. This works for pyuv.Pipe handles only, and the user is required to call Server.accept_connection() whenever a new connection is pending. The backlog parameter specifies the listen backlog i.e the maximum number of not yet accepted active opens to queue. To disable listening for new connections (useful when ipc was set), set the backlog to None. The return value is a Server instance.

Endpoints

Endpoints wrap transports and protocols for client and server side connections, and provide a more object-oriented way of working with them. class Endpoint(protocol_factory, timeout=None) A communications endpoint. The protocol_factory argument constructs a new protocol instance. Normally you would pass a Protocol subclass. The timeout argument speciﬁes a timeout for various network operations. timeout The network timeout. close() Close the endpoint. class Client(protocol_factory, timeout=None) A client endpoint. transport Return the transport, or None if not connected.

26 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

protocol Return the protocol, or None if not connected. switchpoint connect(address, **kwargs) Connect to address and wait for the connection to be established. See create_connection() for a description of address and the supported keyword arguments. switchpoint close() Close the connection. class Server(protocol_factory, timeout=None) A server endpoint. addresses The addresses this server is listening on. connections An iterator yielding the (transport, protocol) pairs for each connection. accept_connection(handle, ssl=False) Accept a new connection on handle. This method needs to be called when a connection was passed via ﬁle descriptor passing. handle_connection(client, ssl) Handle a new connection with handle client. This method exists so that it can be overridden in subclass. It is not intended to be called directly. connection_made(transport, protocol) Called when a new connection is made. connection_lost(transport, protocol, exc=None) Called when a connection is lost. switchpoint listen(address, ssl=False, family=0, ﬂags=0, ipc=False, backlog=128) Create a new transport, bind it to address, and start listening for new connections. See create_server() for a description of address and the supported keyword arguments. switchpoint close() Close the listening sockets and all accepted connections. switchpoint run() Run the event loop and start serving requests. This method stops serving when a CTRL-C/SIGINT is received. It is useful for top-level scripts that run only one server instance. In more complicated application you would call Hub.switch() directly, or wait on some kind of “request to shutdown” event.

Streams (generic high-level API)

Streams provide a generic, high-level API to work with connection-oriented transports using ﬁle-like, blocking and ﬂow-controlled API.

Utility classes class StreamBuffer(transport=None, timeout=None) A stream buffer. This is a utility class that is used to by Stream to implement the read side buffering.

1.3. Reference 27 Gruvi Documentation, Release 0.10.3

get_buffer_size() Return the size of the buffer. set_buffer_limits(high=None, low=None) Set the low and high watermarks for the read buffer. feed(data) Add data to the buffer. feed_eof() Set the EOF condition. feed_error(exc) Set the error condition to exc. class Stream(transport, mode=’rw’, autoclose=False, timeout=None) A byte stream. This class implements buffered, flow controlled and blocking read/write access on top of a transport. A stream works with bytes instances. To create a stream that works with unicode strings, you can wrap it with a io.TextIOWrapper. The transport argument specifies the transport on top of which to create the stream. The mode argument can be 'r' for a read-only stream, 'w' for a write-only stream, or 'rw' for a read-write stream. The autoclose argument controls whether the underlying transport will be closed in the close() method. Be careful with this as the close method is called from the io.BufferedIOBase destructor, which may lead to unexpected closing of the transport when the stream goes out of scope. transport Return the transport wrapped by this stream. wrap(encoding, **textio_args) Return a io.TextIOWrapper that wraps the stream. The wrapper provides text IO on top of the byte stream, using the specified encoding. The textio_args keyword arguments are additional keyword arguments passed to the TextIOWrapper constructor. Unless another buffering scheme is specified, the write_through option is enabled. readable() Return whether the stream is readable. writable() Return whether the stream is writable. closed Return whether the stream is closed. buffer Return the StreamBuffer for this stream. switchpoint read(size=-1) Read up to size bytes. This function reads from the buffer multiple times until the requested number of bytes can be satisfied. This means that this function may block to wait for more data, even if some data is available. The only time a short read is returned, is on EOF or error. If size is not specified or negative, read until EOF.

28 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

switchpoint read1(size=-1) Read up to size bytes. This function reads from the buffer only once. It is useful in case you need to read a large input, and want to do so efficiently. If size is big enough, then this method will return the chunks passed into the memory buffer verbatim without any copying or slicing. switchpoint readline(limit=-1, delim=b’\n’) Read a single line. If EOF is reached before a full line can be read, a partial line is returned. If limit is specified, at most this many bytes will be read. switchpoint readlines(hint=-1) Read lines until EOF, and return them as a list. If hint is specified, then stop reading lines as soon as the total size of all lines exceeds hint. switchpoint write(data) Write data to the transport. This method will block if the transport’s write buffer is at capacity. switchpoint writelines(seq) Write the elements of the sequence seq to the transport. This method will block if the transport’s write buffer is at capacity. switchpoint write_eof() Close the write direction of the transport. This method will block if the transport’s write buffer is at capacity. switchpoint close() Close the stream. If autoclose was passed to the constructor then the underlying transport will be closed as well.

Stream Protocol class StreamProtocol(timeout=None) Byte stream protocol. The timeout argument speciﬁes a default timeout for protocol operations. stream A Stream instance, providing blocking, ﬂow controlled read and write access to the underlying transport.

Stream Client and Server class StreamClient(timeout=None) A stream client. stream An alias for self.protocol.reader switchpoint read(size=-1) A shorthand for self.stream.read(). switchpoint read1(size=-1) A shorthand for self.stream.read1().

1.3. Reference 29 Gruvi Documentation, Release 0.10.3

switchpoint readline(limit=-1, delim=b’\n’) A shorthand for self.stream.readline(). switchpoint readlines(hint=-1) A shorthand for self.stream.readlines(). switchpoint write(data) A shorthand for self.stream.write(). switchpoint write_eof() A shorthand for self.stream.write_eof(). switchpoint writelines(seq) A shorthand for self.stream.writelines(). class StreamServer(stream_handler, timeout=None) A stream server. The stream_handler argument is a handler function to handle client connections. The handler will be called as stream_handler(stream, transport, protocol). The handler for each connection will run in a separate ﬁber so it can use blocking I/O on the stream. When the handler returns, the stream is closed. See Example 2: echo server, using a StreamServer for an example.

Working with Processes gruvi.PIPE = ... gruvi.DEVNULL = ... class Process(encoding=None, timeout=None) A child process. This class allows you to start up a child process, communicate with it and control it. The API is modeled after the subprocess.Popen class. The encoding argument specifies the encoding to use for communications with the child. If an encoding is specified, the standard input and output handles provide text-based IO, otherwise they provide bytes-based IO. stdin The child’s standard input, or None. stdout The child’s standard output, or None. stderr The child’s standard error, or None. returncode The child’s exit status, or None if it has not exited yet. On Unix, if the child was terminated by a signal, return -SIGNUM. pid The child’s process ID, or None if there is no child. spawn(args, executable=None, stdin=None, stdout=None, stderr=None, shell=False, cwd=None, env=None, flags=0, extra_handles=None) Spawn a new child process. The executable to spawn and its arguments are determined by args, executable and shell.

30 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

When shell is set to False (the default), args is normally a sequence and it contains both the program to execute (at index 0), and its arguments. When shell is set to True, then args is normally a string and it indicates the command to execute through the shell. The executable argument can be used to override the executable to execute. If shell is False, it overrides args[0]. This is sometimes used on Unix to implement “fat” executables that behave differently based on argv[0]. If shell is True, it overrides the shell to use. The default shell is '/bin/sh' on Unix, and the value of $COMSPEC (or 'cmd.exe' if it is unset) on Windows. The stdin, stdout and stderr arguments specify how to handle standard input, output, and error, respectively. If set to None, then the child will inherit our respective stdio handle. If set to the special constant PIPE then a pipe is created. The pipe will be connected to a gruvi.StreamProtocol which you can use to read or write from it. The stream protocol instance is available under either stdin, stdout or stderr. All 3 stdio arguments can also be a file descriptor, a file-like object, or a pyuv Stream instance. The extra_handles specifies any extra handles to pass to the client. It must be a sequence where each element is either a file descriptor, a file-like objects, or a pyuv.Stream instance. The position in the sequence determines the file descriptor in the client. The first position corresponds to FD 3, the second to 4, etc. This places these file descriptors directly after the stdio handles. The cwd argument specifies the directory to change to before executing the child. If not provided, the current directory is used. The env argument specifies the environment to use when executing the child. If provided, it must be a dictionary. By default, the current environment is used. The flags argument can be used to specify optional libuv uv_process_flags. The only relevant flags are pyuv.UV_PROCESS_DETACHED and pyuv.UV_PROCESS_WINDOWS_HIDE. Both are Windows specific and are silently ignored on Unix. child_exited(exit_status, term_signal) Callback that is called when the child has exited. switchpoint close() Close the process and frees its associated resources. This method waits for the resources to be freed by the event loop. send_signal(signum) Send the signal signum to the child. On Windows, SIGTERM, SIGKILL and SIGINT are emulated using TerminateProcess(). This will cause the child to exit unconditionally with status 1. No other signals can be sent on Windows. terminate() Terminate the child process. It is not an error to call this method when the child has already exited. switchpoint wait(timeout=-1) Wait for the child to exit. Wait for at most timeout seconds, or indefinitely if timeout is None. Return the value of the returncode attribute. switchpoint communicate(input=None, timeout=-1) Communicate with the child and return its output. If input is provided, it is sent to the client. Concurrent with sending the input, the child’s standard output and standard error are read, until the child exits.

1.3. Reference 31 Gruvi Documentation, Release 0.10.3

The return value is a tuple (stdout_data, stderr_data) containing the data read from standard output and standard error.

Included Protocols (speciﬁc high-level API)

gruvi.http – HTTP Client and Server

The gruvi.http module implements a HTTP client and server. The client and server are relatively complete implementations of the HTTP protocol. Some of the supported features are keepalive, pipelining, chunked transfers and trailers. This implementation supports both HTTP/1.0 and HTTP/1.1. The default for the client is 1.1, and the server will respond with the same version as the client. Connections are kept alive by default. This means that you need to make sure you close connections when they are no longer needed, by calling the appropriate close() method. It is important to clarify how the API exposed by this module uses text and binary data. Data that is read from or written to the HTTP header, such as the version string, method, and headers, are text strings (str on Python 3, str or unicode on Python 2). However, if the string type is unicode aware (str on Python 3, unicode on Python 2), you must make sure that it only contains code points that are part of ISO-8859-1, which is the default encoding specified in RFC 2606. Data that is read from or written to HTTP bodies is always binary. This is done in alignment with the WSGI spec that requires this. This module provides a number of APIs. Client-side there is one: •A gruvi.Client based API. You will use connect() to connect to a server, and then use request() and getresponse() to interact with it. The following server-side APIs are available: •A gruvi.Server based API. Incoming HTTP messages are passed to a message handler that needs to take care of all aspects of HTTP other than parsing. • A WSGI API, as described in PEP 333. The server-side API is selected through the adapter argument to HttpServer constructor. The default adapter is WsgiAdapter, which implements the WSGI protocol. To use the raw server interface, pass the identity function (lambda x: x). REQUEST Constant indicating a HTTP request. RESPONSE Constant indicating a HTTP response. exception HttpError Exception that is raised in case of HTTP protocol errors. class ParsedUrl A namedtuple() with the following fields: scheme, host, port, path, query, fragment and userinfo. In addition to the tuple fields the following properties are defined: addr Address tuple that can be used with create_connection(). ssl Whether the scheme requires SSL/TLS.

32 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

target The “target” i.e. local part of the URL, consisting of the path and query. parse_url(url, default_scheme=’http’, is_connect=False) Parse an URL and return its components. The default_scheme argument speciﬁes the scheme in case URL is an otherwise valid absolute URL but with a missing scheme. The is_connect argument must be set to True if the URL was requested with the HTTP CONNECT method. These URLs have a different form and need to be parsed differently. The result is a ParsedUrl containing the URL components. class HttpMessage HTTP message. Instances of this class are returned by HttpClient.getresponse() and passed as an argument to HttpServer message handlers. message_type The message type, either REQUEST or RESPONSE. version The HTTP version as a string, either '1.0' or '1.1'. status_code The HTTP status code as an integer. Only for response messages. method The HTTP method as a string. Only for request messages. url The URL as a string. Only for request messages. parsed_url The parsed URL as a ParsedUrl instance. headers The headers as a list of (name, value) tuples. charset The character set as parsed from the “Content-Type” header, if available. body The message body, as a Stream instance. get_header(headers, name, default=None) A shorthand for get_header(headers, ...). class HttpRequest(protocol) HTTP client request. Usually you do not instantiate this class directly, but use the instance returned by HttpProtocol. request(). You can however start new request yourself by instantiating this class and passing it a protocol instance. switchpoint start_request(method, url, headers=None, bodylen=None) Start a new HTTP request. The optional headers argument contains the headers to send. It must be a sequence of (name, value) tuples.

1.3. Reference 33 Gruvi Documentation, Release 0.10.3

The optional bodylen parameter is a hint that specifies the length of the body that will follow. A length of -1 indicates no body, 0 means an empty body, and a positive number indicates the body size in bytes. This parameter helps determine whether to use the chunked transfer encoding. Normally when the body size is known chunked encoding is not used. switchpoint write(buf ) Write buf to the request body. switchpoint end_request() End the request body. class WsgiAdapter(application) WSGI Adapter This class adapts the WSGI callable application so that instances of this class can be used as a message handler in HttpProtocol. class HttpProtocol(handler=None, server_side=False, server_name=None, version=None, timeout=None) HTTP protocol implementation. The handler argument specifies a message handler to handle incoming HTTP requests. It must be a callable with the signature handler(message, transport, protocol). The server_side argument specifies whether this is a client or server side protocol. For client-side protocols, the server_name argument specifies the remote server name, which is used as the Host: header. It is normally not required to set this as it is taken from the unresolved hostname passed to connect(). For server-side protocols, server_name will be used as the value for the SERVER_NAME WSGI environment variable. default_version = ‘1.1’ Default HTTP version. max_header_size = 65536 Max header size. The parser keeps the header in memory during parsing. max_buffer_size = 65536 Max number of body bytes to buffer. Bodies larger than this will cause the transport to be paused until the buffer is below the threshold again. max_pipeline_size = 10 Max number of pipelined requests to keep before pausing the transport. writer A Stream instance for writing directly to the underlying transport. switchpoint request(method, url, headers=None, body=None) Make a new HTTP request. The method argument is the HTTP method as a string, for example 'GET' or 'POST'. The url argument specifies the URL. The optional headers argument specifies extra HTTP headers to use in the request. It must be a sequence of (name, value) tuples. The optional body argument may be used to include a body in the request. It must be a bytes instance, a file-like object opened in binary mode, or an iterable producing bytes instances. To send potentially large bodies, use the file or iterator interfaces. This has the benefit that only a single chunk is kept in memory at a time.

34 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

The response to the request can be obtained by calling the getresponse() method. You may make multiple requests before reading a response. For every request that you make however, you must call getresponse() exactly once. The remote HTTP implementation will send by the responses in the same order as the requests. This method will use the “chunked” transfer encoding if here is a body and the body size is unknown ahead of time. This happens when the file or interator interface is used in the abence of a “Content-Length” header. switchpoint getresponse() Wait for and return a HTTP response. The return value will be a HttpMessage. When this method returns only the response header has been read. The response body can be read using read() and similar methods on the message body. Note that if you use persistent connections (the default), it is required that you read the entire body of each response. If you don’t then deadlocks may occur. class HttpClient(version=None, timeout=None) HTTP client. The optional version argument specifies the HTTP version to use. The default is HttpProtocol. default_version. The optional timeout argument specifies the timeout for various network and protocol operations. protocol Return the protocol, or None if not connected. switchpoint getresponse() A shorthand for self.protocol.getresponse(). switchpoint request(method, url, headers=None, body=None) A shorthand for self.protocol.request(). class HttpServer(application, server_name=None, adapter=None, timeout=None) HTTP server. The application argument is the web application to expose on this server. The application is wrapped in adapter to create a message handler as required by HttpProtocol. By default the adapter in default_adapter is used. The optional server_name argument specifies the server name. The server name is made available to WSGI applications as the $SERVER_NAME environment variable. The optional timeout argument specifies the timeout for various network and protocol operations. default_adapter The default adapter to use. alias of WsgiAdapter

Example

# HTTP client and server example.

from gruvi.http import HttpClient, HttpServer

def handler(env, start_response): headers=[('Content-Type','text/plain; charset=UTF-8')] status='200 OK'

1.3. Reference 35 Gruvi Documentation, Release 0.10.3

body='pong' start_response(status, headers) yield body.encode('utf-8') server= HttpServer(handler) server.listen(('localhost',0)) addr= server.addresses[0] client= HttpClient() client.connect(addr) client.request('GET','/ping') resp= client.getresponse() assert resp.status_code == 200 body= resp.body.read() print('result = {}'.format(body.decode(resp.charset))) gruvi.jsonrpc – JSON-RPC Client and Server

The gruvi.jsonrpc module implements a JSON-RPC client and server. There are two version of JSON-RPC: version 1.0 and version 2.0. While these versions are similar, they are not mutually compatible. The classes exposed by this module implement both versions. The default version to use is stored in JsonRpcProtocol.default_version, and most constructors take an argument that can be used to override this. The “batch” feature of version 2.0 is not supported. It more relevant for JSON-RPC over HTTP rather for that clients and servers that operate directly on top of a connection. The two main classes in this module are JsonRpcClient and JsonRpcServer. The difference is only who initiates the connection at the transport level. The JSON-RPC protocol itself does not distinguish between clients and servers. Both the client and the server can receive and respond to incoming messages. These may be method calls (more common for servers), or notifications (common for both client and servers). These incoming messages are handled by a message handler, which is mandatory for a server and optional for a client. Note that for simple blocking method callls from client to a server no message handler is needed. Message handlers run in a separate dispatcher fiber, one per connection. This means that a client will have at most one dispatcher fiber, while a server will have exactly one fiber per connection. Because they run in a separate fiber, message handlers can call into switchpoints themselves. REQUEST Constant indicating a JSON-RPC request message. RESPONSE Constant indicating a JSON-RPC response message. errorcode {...} Dictionary mapping error codes to error names. strerror(code) Return an error message for error code. The error code is the error.code field of a JSON-RPC response message. The JSON-RPC version 2 spec contains a list of standard error codes.

36 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3 exception JsonRpcError Exception that is raised in case of JSON-RPC errors. error The error field from the JSON-RPC response, if available. class JsonRpcVersion(name) Class that encapsulates the differences between JSON-RPC vesion 1.0 and 2.0. name Return the JSON-RPC version as a string, e.g. ‘1.0’. next_id() Return a unique message ID. static create(version) Return a new instance for version, which can be either ‘1.0’ or ‘2.0’. check_message(message) Check message and return the message type. Raise a ValueError in case of an invalid message. create_request(method, args=None, notification=False) Create a new request message for method with arguments args. The optional notification argument, if nonzero, creates a notification and no response to this message is expected. create_response(request, result=None, error=None) Create a new response message, as a response to request. A successful response is created when the optional error argument is not provided. In this case result specifies the result, which may be any value including None. If error is provided, an error response is created, and error should be a dict as specified by the JSON-RPC specifications. The request argument may be explicitly set to None, in which case this is an error response that is not specific to any one request. class JsonRpcProtocol(handler=None, version=None, timeout=None) A JSON-RPC client and server Protocol implementation. default_timeout = 30 Class level attribute that specifies the default timeout. default_version = ‘2.0’ Class level attribute that specifies the default JSON-RPC version. version Return the JsonRpcVersion instance for this protocol. switchpoint send_message(message) Send a raw JSON-RPC message. The message argument must be a dictionary containing a valid JSON-RPC message according to the version passed into the constructor. switchpoint call_method(method, *args) Call a JSON-RPC method and wait for its result. The method is called with positional arguments args. On success, the result field from the JSON-RPC response is returned. On error, a JsonRpcError is raised, which you can use to access the error field of the JSON-RPC response.

1.3. Reference 37 Gruvi Documentation, Release 0.10.3

switchpoint send_notification(method, *args) Send a JSON-RPC notification. The notification method is sent with positional arguments args. switchpoint send_response(request, result=None, error=None) Respond to a JSON-RPC method call. This is a response to the message in request. If error is not provided, then this is a succesful response, and the value in result, which may be None, is passed back to the client. if error is provided and not None then an error is sent back. In this case error must be a dictionary as specified by the JSON-RPC spec. class JsonRpcClient(handler=None, version=None, timeout=None) A JSON-RPC Client. The handler argument specifies an optional JSON-RPC message handler. You need to supply a message handler if you want to listen to notifications or you want to respond to server-to-client method calls. If provided, the message handler it must be a callable with signature handler(message, transport, protocol). The version and timeout argument can be used to override the default protocol version and timeout, respectively.

protocol Return the protocol, or None if not connected. switchpoint call_method(method, *args) A shorthand for self.protocol.call_method(). switchpoint send_message(message) A shorthand for self.protocol.send_message(). switchpoint send_notification(method, *args) A shorthand for self.protocol.send_notification(). class JsonRpcServer(handler, version=None, timeout=None) A JSON-RPC Server. The handler argument speciﬁes the JSON-RPC message handler. It must be a callable with signature handler(message, transport, protocol). The message handler is called in a separate dispatcher ﬁber (one per connection). The version and timeout argument can be used to override the default protocol version and timeout, respectively.

Example

# Ping-pong between JSON-RPC client and server. from gruvi.jsonrpc import JsonRpcClient, JsonRpcServer def handler(message, transport, protocol): method= message.get('method') if method =='ping': protocol.send_response(message,'pong') server= JsonRpcServer(handler) server.listen(('localhost',0)) addr= server.addresses[0] client= JsonRpcClient() client.connect(addr)

38 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

result= client.call_method('ping')

print('result = {}'.format(result))

gruvi.dbus – D-BUS Client and Server

The gruvi.dbus module implements a D-BUS client and server. The implementation uses parts of the txdbus project. A cut down copy of txdbus, containing only those parts needed by Gruvi, is available as gruvi.txdbus. You need this if you are providing a message handler (see below). Both a client and a server/bus-side implementation are provided. The bus-side implementation is very bare bones and apart from the “Hello” message it does not implement any of the “org.freedestkop.DBus” interface. It also does not implement any message routing. The server side is provided mostly for testing purposes (but it could serve as the basis for a real D-BUS server). The client side of a D-BUS connection is implemented by DbusClient and the server/bus-side by DbusServer. Both implement a procedural interface. Messages can be send using e.g. DbusClient.send_message() or DbusClient.call_method(). An object-oriented interface that represents D-BUS objects as Python objects, like the one txdbus provides, is currently not available. The procedural interface can be used as a basis for your own object-oriented interface though. To receive notifications or to respond to method calls, you need to provide a message handler to the client or the server constructor. The signature of the message handler is: message_handler(message, protocol). Here, the message argument is an instance of gruvi.txdbus.DbusMessages, and the protocol will be the DbusProtocol instance for the current connection. Message handlers runs in their own fiber, which allows them to call into switchpoints. There is one fiber for every connection. Usage example:

client= gruvi.DbusClient() client.connect('session') result= client.call_method('org.freedesktop.DBus','/org/freedesktop/DBus', 'org.freedesktop.DBus','ListNames') for name in result[0]: print('Name: {}'.format(name))

exception DbusError Exception that is raised in case of D-BUS protocol errors. exception DbusMethodCallError(method, reply) Exception that is raised when a error reply is received for a D-BUS method call. class DbusProtocol(message_handler=None, server_side=False, server_guid=None, timeout=None) D-BUS Protocol. switchpoint get_unique_name() Return the unique name of the D-BUS connection. switchpoint send_message(message) Send a D-BUS message. The message argument must be gruvi.txdbus.DbusMessage instance. switchpoint call_method(service, path, interface, method, signature=None, args=None, no_reply=False, auto_start=False, timeout=-1) Call a D-BUS method and wait for its reply.

1.3. Reference 39 Gruvi Documentation, Release 0.10.3

This method calls the D-BUS method with name method that resides on the object at bus address service, at path path, on interface interface. The signature and args are optional arguments that can be used to add parameters to the method call. The signature is a D-BUS signature string, while args must be a sequence of python types that can be converted into the types specified by the signature. See the D-BUS specification for a reference on signature strings. The flags no_reply and auto_start control the NO_REPLY_EXPECTED and NO_AUTO_START flags on the D-BUS message. The return value is the result of the D-BUS method call. This will be a possibly empty sequence of values. class DbusClient(message_handler=None, timeout=30) A D-BUS client. The message_handler argument specifies an optional message handler. The optional timeout argument specifies a default timeout for protocol operations in seconds. switchpoint connect(address=’session’) Connect to address and wait until the connection is established. The address argument must be a D-BUS server address, in the format described in the D-BUS specification. It may also be one of the special addresses 'session' or 'system', to connect to the D-BUS session and system bus, respectively. protocol Return the protocol, or None if not connected. switchpoint call_method(service, path, interface, method, signature=None, args=None, no_reply=False, auto_start=False, timeout=-1) A shorthand for self.protocol.call_method(). switchpoint get_unique_name() A shorthand for self.protocol.get_unique_name(). switchpoint send_message(message) A shorthand for self.protocol.send_message(). class DbusServer(message_handler, timeout=30) A D-BUS server. The message_handler argument specifies the message handler. The optional timeout argument specifies a default timeout for protocol operations in seconds. switchpoint listen(address=’session’) Start listening on address for new connection. The address argument must be a D-BUS server address, in the format described in the D-BUS specification. It may also be one of the special addresses 'session' or 'system', to connect to the D-BUS session and system bus, respectively.

Examples

Example 1: curl like URL downloader

1 # Gruvi example program: a cURL like URL downloader

3 import sys

40 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

4 import argparse

5 from gruvi.http import HttpClient, parse_url

7 parser= argparse.ArgumentParser()

8 parser.add_argument('url')

9 args= parser.parse_args()

11 url= parse_url(args.url)

13 client= HttpClient()

14 client.connect(url.addr, ssl=url.ssl)

15 client.request('GET', url.target)

17 resp= client.getresponse()

18 if not 200<= resp.status_code<= 399:

19 sys.stderr.write('Error: got status {}\n'.format(resp.status_code))

20 sys.exit(1)

22 stdout= getattr(sys.stdout,'buffer', sys.stdout)

24 while True:

25 buf= resp.body.read(4096)

26 if not buf:

27 break

28 stdout.write(buf)

Example 2: echo server, using a StreamServer

1 # Gruvi example program: echo server, using StreamServer

3 import gruvi

5 def echo_handler(stream, transport, protocol):

6 peer= transport.get_extra_info('peername')

7 print('New connection from {0}'.format(gruvi.saddr(peer)))

8 while True:

9 buf= stream.read1()

10 if not buf:

11 break

12 stream.write(buf)

13 print('Connection lost')

15 server= gruvi.StreamServer(echo_handler)

16 server.listen(('localhost',0))

17 for addr in server.addresses:

18 print('Listen on {0}'.format(gruvi.saddr(addr)))

20 server.run()

Example 3: echo server, using a custom Protocol

1 # Gruvi example program: an echo server, using a Protocol

3 import gruvi

1.4. Examples 41 Gruvi Documentation, Release 0.10.3

5 class EchoProtocol(gruvi.Protocol):

7 def connection_made(self, transport):

8 self._transport= transport

9 peer= transport.get_extra_info('peername')

10 print('New connection from {0}'.format(gruvi.saddr(peer)))

12 def data_received(self, data):

13 self._transport.write(data)

15 def eof_received(self):

16 print('Connection lost')

18 server= gruvi.create_server(EchoProtocol, ('localhost',0))

19 for addr in server.addresses:

20 print('Listen on {0}'.format(gruvi.saddr(addr)))

22 server.run()

Example 4: netcat client

1 # Gruvi example program: a nc(1) like client, using StreamClient

3 import sys

4 import argparse

5 import gruvi

7 parser= argparse.ArgumentParser()

8 parser.add_argument('hostname')

9 parser.add_argument('port', type=int)

10 parser.add_argument('--ssl', action='store_true')

11 args= parser.parse_args()

13 remote= gruvi.StreamClient()

14 remote.connect((args.hostname, args.port), ssl=args.ssl)

16 stdin= gruvi.StreamClient()

17 stdin.connect(sys.stdin.fileno(), mode='r')

18 stdout= gruvi.StreamClient()

19 stdout.connect(sys.stdout.fileno(), mode='w')

21 done= gruvi.Event()

23 def stdin_reader():

24 while True:

25 buf= stdin.read1()

26 if not buf:

27 print('Got EOF on stdin')

28 break

29 remote.write(buf)

30 done.set()

32 def remote_reader():

33 while True:

34 buf= remote.read1()

42 Chapter 1. Documentation Gruvi Documentation, Release 0.10.3

35 if not buf:

36 print('Got EOF from remote')

37 break

38 stdout.write(buf)

39 done.set()

41 gruvi.spawn(stdin_reader)

42 gruvi.spawn(remote_reader)

44 done.wait()

Example 5: fortune web service

1 # Gruvi example program: a "fortune" web service

3 import locale

4 import gruvi

5 import contextlib

7 def fortune_app(environ, start_response):

8 print('New connection from {0}'.format(environ['REMOTE_ADDR']))

9 proc= gruvi.Process(encoding=locale.getpreferredencoding())

10 proc.spawn('fortune', stdout=gruvi.PIPE)

11 with contextlib.closing(proc):

12 fortune= proc.stdout.read()

13 proc.wait()

14 start_response('200 OK', [('Content-Type','text/plain; charset=utf-8')])

15 return [fortune.encode('utf-8')]

17 server= gruvi.HttpServer(fortune_app)

18 server.listen(('localhost',0))

19 for addr in server.addresses:

20 print('Listen on {0}'.format(gruvi.saddr(addr)))

22 server.run()

1.4. Examples 43 Gruvi Documentation, Release 0.10.3

44 Chapter 1. Documentation CHAPTER 2

Project ﬁles

README

Improved ergonomics for Python programmers wanting to use asynchronous IO. Gruvi is an asynchronous IO library for Python. It focuses on the following desirable properties: • Simple. Async IO code should look just like normal code, with simple, sequential control flow and regular functions. • Efficient. An IO library should have a very low memory and CPU overhead, so that it can scale to small systems or to many concurrent connections. • Powerful. Common protocols like SSL/TLS and HTTP should be part of every IO library. Gruvi uses libuv (via pyuv) as the underlying high-performance event-based I/O layer, and coroutines based on fibers to create a traditional sequential programming model on top of the libuv completion/callback model. Gruvi is similar in concept existing async IO frameworks like asyncio, gevent, and eventlet. For a comparison, see Rationale.

Features

Gruvi has the following features: • Excellent platform support (mostly thanks to libuv). Linux, Mac OSX and Windows are all ﬁrst-class citizens. • PEP-3156 compatible transport/protocol interface. • A traditional, sequential programming model based on green threads, where there is no distinction between asynchronous and normal functions. • Great SSL/TLS support, also on Windows. The asynchronous SSL support in the Python standard library came from Gruvi before it was included into the stdlib. • Small core and focus on low memory usage and fast performance. This makes Gruvi very suitable for mobile applications and embedded web servers.

45 Gruvi Documentation, Release 0.10.3

• A full suite of synchronization primitives including locks, conditions and queues. • Thread and ﬁber pools with a concurrent.futures interface. • Batteries includes: built-in client/server support for HTTP, JSON-RPC and D-BUS. • Support for Python 2.7.x and 3.3+.

Example

An simple echo server, using a StreamServer: import gruvi def echo_handler(stream, transport, protocol): while True: buf= stream.read1() if not buf: break stream.write(buf) server= gruvi.StreamServer(echo_handler) server.listen(('localhost', 7777)) server.run()

Requirements

You need Python 2.7 or 3.3+. The following operating systems are currently supported: • Linux (might work on other Posix OSs) • macOS (not currently tested via CI) • Windows (not currently tested via CI)

Installation

Development install in a virtualenv:

$ git clone https://github.com/geertj/gruvi $ cd gruvi $ pip install -r requirements.txt $ python setup.py build $ python setup.py install

To run the test suite:

$ python runtests.py unit

For other installation options, see the Installation section in the manual.

Documentation

The documentation is available on readthedocs.

46 Chapter 2. Project ﬁles Gruvi Documentation, Release 0.10.3

License

Gruvi is free software, provided under the MIT license.

Contact

Feel free to contact the author at [email protected]. You can also submit tickets or suggestions for improvements on Github.

CHANGELOG

Changes since latest version: • Verify SSL certificates by default. • New function: sslcompat.create_default_context() • Limit SSL support for Python <= 2.7.8. Changes in version 0.10.3: • Refactoring and cleanups of many places in the code. • Refactored and simplified flow control. • Made @switchpoint return a closure. • Documentation improvements. Changes in version 0.10.2: • New infrastructure for polling FDs. • Add Process.close() to explicitly close the process and stdio handles. • Remove redundant Process.kill() and Process.poll() • Use Memory BIO for faster and more compatible SSL implementation. • Add support for Python 3.5. • Move to new pyuv upstream (1.0.0-dev1, using libuv 1.0.0-rc2). • Improve test coverage. • Remove non-standard method Queue.clear(). • Drop support for Python 2.6. • Make futures thread-safe and add .cancel() method. • Make Event() follow threading API for timeouts. • New functions: wait(), as_completed() • Store callbacks in a linked list for fast removal. • Fix threading issues with timeouts for sync primitves. • Add test coverage via coveralls. • Add .alive property to Fiber. • http: communicate connection_lost() to waiters.

2.2. CHANGELOG 47 Gruvi Documentation, Release 0.10.3

• create_connection(): raise on callback error. • Workaround for Windows build failure (link error 1104). Changes in version 0.10.1: • Re-release of 0.10.0 with correct README file. Changes in version 0.10.0: • PEP-3156 style transports and protocols. • Incorporate SSL backports from Bluepass. • Replace Signal with stdlib-like Event and Condition. • Add support for working with child processes. • Add support for connecting protocols to a file descriptor. • Lots of small bug fixes and improvements. • Improved documentation. Changes in version 0.9.2: • Add support for JSON-RPC v2.0 • Logger supports Python 2.7+ style docstrings on 2.6 • Lots of small bug fixes and some cleanups Changes in version 0.9.1: • Add thread-safe synchronization primitives. • Add gruvi.futures. • Test suite now uses unittest. • Lots of smaller refactorings to make the API cleaner. • Remove dependency on six. • Switch from greenlet to python-fibers. • Fixed Windows support. Changes in version 0.9.0: • Almost complete rewrite. • First release of beta quality.

AUTHORS

Main author: • Geert Jansen Gruvi includes some third party code: • The gruvi/txdbus package contains code from the “txdbus” project by Tom Cocagne ([email protected]). MIT licensed. Source code is at http://github.com/cocagne/txdbus.

48 Chapter 2. Project ﬁles Python Module Index

g gruvi,7 gruvi.dbus, 39 gruvi.http, 32 gruvi.jsonrpc, 36

49 Gruvi Documentation, Release 0.10.3

50 Python Module Index Index

A create() (JsonRpcVersion static method), 37 abort() (BaseTransport method), 21 create_default_context() (in module gruvi), 24 accept_connection() (Server method), 27 create_request() (JsonRpcVersion method), 37 active (switch_back attribute), 11 create_response() (JsonRpcVersion method), 37 add_cleanup() (switch_back method), 11 current_ﬁber() (in module gruvi), 12 add_done_callback() (Future method), 17 addr (ParsedUrl attribute), 32 D addresses (Server attribute), 27 data (Hub attribute), 12 alive (Fiber attribute),9 data_received() (Protocol method), 24 assert_no_switchpoints (class in gruvi), 11 datagram_received() (DatagramProtocol method), 25 DatagramProtocol (class in gruvi), 24 B DatagramTransport (class in gruvi), 22 BaseProtocol (class in gruvi), 24 DbusClient (class in gruvi.dbus), 40 BaseTransport (class in gruvi), 20 DbusError, 39 blocking() (in module gruvi), 18 DbusMethodCallError, 39 body (HttpMessage attribute), 33 DbusProtocol (class in gruvi.dbus), 39 buffer (Stream attribute), 28 DbusServer (class in gruvi.dbus), 40 default_adapter (HttpServer attribute), 35 C default_timeout (JsonRpcProtocol attribute), 37 default_version (HttpProtocol attribute), 34 can_write_eof() (Transport method), 22 default_version (JsonRpcProtocol attribute), 37 cancel() (Fiber method),9 do_handshake() (SslTransport method), 23 cancel() (Future method), 17 done() (Future method), 16 Cancelled,7 cancelled() (Future method), 16 E charset (HttpMessage attribute), 33 check_message() (JsonRpcVersion method), 37 Endpoint (class in gruvi), 26 child_exited() (Process method), 31 eof_received() (Protocol method), 24 clear() (Event method), 14 Error,7 Client (class in gruvi), 26 error (JsonRpcError attribute), 37 close() (BaseTransport method), 21 error_received() (DatagramProtocol method), 25 close() (Endpoint method), 26 errorcode (in module gruvi.jsonrpc), 36 close() (SslTransport method), 24 Event (class in gruvi), 14 closed (Stream attribute), 28 Condition (class in gruvi), 14 F connection_lost() (BaseProtocol method), 24 feed() (StreamBuffer method), 28 connection_lost() (Server method), 27 feed_eof() (StreamBuffer method), 28 connection_made() (BaseProtocol method), 24 feed_error() (StreamBuffer method), 28 connection_made() (Server method), 27 Fiber (class in gruvi),9 connections (Server attribute), 27 ﬁber (switch_back attribute), 11

51 Gruvi Documentation, Release 0.10.3

FiberPool (class in gruvi), 18 max_header_size (HttpProtocol attribute), 34 Future (class in gruvi), 16 max_pipeline_size (HttpProtocol attribute), 34 maxsize (PoolBase attribute), 17 G message_type (HttpMessage attribute), 33 get_buffer_size() (StreamBuffer method), 27 MessageProtocol (class in gruvi), 25 get_cpu_pool() (in module gruvi), 18 method (HttpMessage attribute), 33 get_extra_info() (BaseTransport method), 21 minsize (PoolBase attribute), 17 get_extra_info() (SslTransport method), 23 get_extra_info() (Transport method), 22 N get_header() (HttpMessage method), 33 name (Fiber attribute),9 get_hub() (in module gruvi),9 name (Hub attribute), 12 get_io_pool() (in module gruvi), 18 name (JsonRpcVersion attribute), 37 get_nowait() (Queue method), 15 name (PoolBase attribute), 17 get_write_buffer_limits() (BaseTransport method), 21 next_id() (JsonRpcVersion method), 37 get_write_buffer_size() (BaseTransport method), 20 notify() (Condition method), 15 get_write_buffer_size() (DatagramTransport method), 22 notify_all() (Condition method), 15 get_write_buffer_size() (Transport method), 21 gruvi (module),7 P gruvi.dbus (module), 39 paddr() (in module gruvi), 20 gruvi.http (module), 32 parse_url() (in module gruvi.http), 33 gruvi.jsonrpc (module), 36 parsed_url (HttpMessage attribute), 33 ParsedUrl (class in gruvi.http), 32 H pause_reading() (BaseTransport method), 21 handle_connection() (Server method), 27 pause_reading() (DatagramTransport method), 22 headers (HttpMessage attribute), 33 pause_reading() (SslTransport method), 23 HttpClient (class in gruvi.http), 35 pause_reading() (Transport method), 21 HttpError, 32 pause_writing() (BaseProtocol method), 24 HttpMessage (class in gruvi.http), 33 pid (Process attribute), 30 HttpProtocol (class in gruvi.http), 34 poll (Hub attribute), 12 HttpRequest (class in gruvi.http), 33 PoolBase (class in gruvi), 17 HttpServer (class in gruvi.http), 35 PriorityQueue (class in gruvi), 16 Hub (class in gruvi), 12 Process (class in gruvi), 30 Protocol (class in gruvi), 24 I protocol (Client attribute), 26 ignore_interrupt (Hub attribute), 12 protocol (DbusClient attribute), 40 protocol (HttpClient attribute), 35 J protocol (JsonRpcClient attribute), 38 join() (Queue method), 16 ProtocolError, 24 JsonRpcClient (class in gruvi.jsonrpc), 38 put_nowait() (Queue method), 15 JsonRpcError, 36 Python Enhancement Proposals JsonRpcProtocol (class in gruvi.jsonrpc), 37 PEP 3156,3, 20, 25 JsonRpcServer (class in gruvi.jsonrpc), 38 PEP 333, 32 JsonRpcVersion (class in gruvi.jsonrpc), 37 Q L qsize() (Queue method), 15 LifoQueue (class in gruvi), 16 Queue (class in gruvi), 15 local (class in gruvi), 12 QueueEmpty, 15 Lock (class in gruvi), 13 QueueFull, 15 locked() (Lock method), 14 locked() (RLock method), 14 R loop (Hub attribute),9, 12 readable() (Stream method), 28 M release() (Lock method), 14 release() (RLock method), 14 max_buffer_size (HttpProtocol attribute), 34

52 Index Gruvi Documentation, Release 0.10.3 remove_done_callback() (Future method), 17 throw() (switch_back method), 11 REQUEST (in module gruvi.http), 32 Timeout,7 REQUEST (in module gruvi.jsonrpc), 36 timeout (Endpoint attribute), 26 RESPONSE (in module gruvi.http), 32 timeout (switch_back attribute), 11 RESPONSE (in module gruvi.jsonrpc), 36 Transport (class in gruvi), 21 resume_reading() (BaseTransport method), 21 transport (Client attribute), 26 resume_reading() (DatagramTransport method), 22 transport (Stream attribute), 28 resume_reading() (SslTransport method), 23 TransportError, 20 resume_reading() (Transport method), 21 resume_writing() (BaseProtocol method), 24 U returncode (Process attribute), 30 unwrap() (SslTransport method), 24 RFC url (HttpMessage attribute), 33 RFC 2606, 32 RLock (class in gruvi), 14 V run_callback() (Hub method), 13 version (HttpMessage attribute), 33 running() (Future method), 16 version (JsonRpcProtocol attribute), 37 S W saddr() (in module gruvi), 20 wrap() (Stream method), 28 send_signal() (Process method), 31 writable() (Stream method), 28 sendto() (DatagramTransport method), 22 write() (Transport method), 21 Server (class in gruvi), 27 write_eof() (Transport method), 22 set() (Event method), 14 writelines() (Transport method), 22 set_buffer_limits() (StreamBuffer method), 28 writer (HttpProtocol attribute), 34 set_write_buffer_limits() (BaseTransport method), 21 WsgiAdapter (class in gruvi.http), 34 spawn() (in module gruvi), 12 spawn() (Process method), 30 ssl (ParsedUrl attribute), 32 SslTransport (class in gruvi), 22 start() (BaseTransport method), 20 start() (Fiber method),9 status_code (HttpMessage attribute), 33 stderr (Process attribute), 30 stdin (Process attribute), 30 stdout (Process attribute), 30 Stream (class in gruvi), 28 stream (StreamClient attribute), 29 stream (StreamProtocol attribute), 29 StreamBuffer (class in gruvi), 27 StreamClient (class in gruvi), 29 StreamProtocol (class in gruvi), 29 StreamServer (class in gruvi), 30 strerror() (in module gruvi.jsonrpc), 36 submit() (PoolBase method), 17 switch() (Hub method), 13 switch() (switch_back method), 11 switch_back (class in gruvi), 10 switchpoint() (in module gruvi), 11 T target (ParsedUrl attribute), 32 task_done() (Queue method), 16 terminate() (Process method), 31 ThreadPool (class in gruvi), 18

Index 53