The Zeromq Guide - for Python Developers the Zeromq Guide - for Python Developers Dedication
Total Page:16
File Type:pdf, Size:1020Kb
The ZeroMQ Guide - for Python Developers The ZeroMQ Guide - for Python Developers Dedication By Pieter Hintjens With thanks to the hundred or so people who contributed examples in two dozen programming languages, who helped with suggestions and fixes, and who kept pushing for more examples of how to connect your code. Thanks to Bill Desmarais, Brian Dorsey, Daniel Lin, Eric Desgranges, Gonzalo Diethelm, Guido Goldstein, Hunter Ford, Kamil Shakirov, Martin Sustrik, Mike Castleman, Naveen Chawla, Nicola Peduzzi, Oliver Smith, Olivier Chamoux, Peter Alexander, Pierre Rouleau, Randy Dryburgh, John Unwin, Alex Thomas, Mihail Minkov, Jeremy Avnet, Michael Compton, Kamil Kisiel, Mark Kharitonov, Guillaume Aubert, Ian Barber, Mike Sheridan, Faruk Akgul, Oleg Sidorov, Lev Givon, Allister MacLeod, Alexander D’Archangel, Andreas Hoelzlwimmer, Han Holl, Robert G. Jakabosky, Felipe Cruz, Marcus McCurdy, Mikhail Kulemin, Dr. Gergö Érdi, Pavel Zhukov, Alexander Else, Giovanni Ruggiero, Rick "Technoweenie", Daniel Lundin, Dave Hoover, Simon Jefford, Benjamin Peterson, Justin Case, Devon Weller, Richard Smith, Alexander Morland, Wadim Grasza, Michael Jakl, Uwe Dauernheim, Sebastian Nowicki, Simone Deponti, Aaron Raddon, Dan Colish, Markus Schirp, Benoit Larroque, Jonathan Palardy, Isaiah Peng, Arkadiusz Orzechowski, Umut Aydin, Matthew Horsfall, Jeremy W. Sherman, Eric Pugh, Tyler Sellon, John E. Vincent, Pavel Mitin, Min RK, Igor Wiedler, Olof Åkesson, Patrick Lucas, Heow Goodman, Senthil Palanisami, John Gallagher, Tomas Roos, Stephen McQuay, Erik Allik, Arnaud Cogoluègnes, Rob Gagnon, Dan Williams, Edward Smith, James Tucker, Kristian Kristensen, Vadim Shalts, Martin Trojer, Tom van Leeuwen, Pandya Hiten, Harm Aarts, Marc Harter, Iskren Ivov Chernev, Jay Han, Sonia Hamilton, and Zed Shaw. Thanks to Stathis Sideris for Ditaa (http://www.ditaa.org), which I used for the diagrams. Please use the issue tracker (https://github.com/imatix/zguide2/issues) for all comments and errata. This version covers the latest stable release of ØMQ (3.2) and was published on Tue 30 October, 2012. If you are using older versions of ØMQ then some of the examples and explanations won’t be accurate. The Guide is originally in C (http://zguide.zeromq.org/page:all), but also in PHP (http://zguide.zeromq.org/php:all), Python (http://zguide.zeromq.org/py:all), Lua (http://zguide.zeromq.org/lua:all), and Haxe (http://zguide.zeromq.org/hx:all). We’ve also translated most of the examples into C++, C#, CL, Erlang, F#, Felix, Haskell, Java, Objective-C, Ruby, Ada, Basic, Clojure, Go, Haxe, Node.js, ooc, Perl, and Scala. Table of Contents 1. Basic Stuff...............................................................................................................................................1 1.1. Fixing the World .........................................................................................................................1 1.2. ØMQ in a Hundred Words ..........................................................................................................2 1.3. Some Assumptions......................................................................................................................2 1.4. Getting the Examples..................................................................................................................2 1.5. Ask and Ye Shall Receive ...........................................................................................................3 1.6. A Minor Note on Strings.............................................................................................................7 1.7. Version Reporting .......................................................................................................................9 1.8. Getting the Message Out.............................................................................................................9 1.9. Divide and Conquer ..................................................................................................................14 1.10. Programming with ØMQ ........................................................................................................19 1.11. Getting the Context Right .......................................................................................................21 1.12. Making a Clean Exit ...............................................................................................................22 1.13. Why We Needed ØMQ ...........................................................................................................23 1.14. Socket Scalability....................................................................................................................27 1.15. Missing Message Problem Solver...........................................................................................28 1.16. Upgrading from ØMQ/2.2 to ØMQ/3.2..................................................................................30 1.17. Warning - Unstable Paradigms! ..............................................................................................30 2. Intermediate Stuff................................................................................................................................32 2.1. The Zen of Zero ........................................................................................................................32 2.2. The Socket API .........................................................................................................................32 2.3. Plugging Sockets Into the Topology .........................................................................................34 2.4. Using Sockets to Carry Data.....................................................................................................35 2.5. Unicast Transports.....................................................................................................................37 2.6. ØMQ is Not a Neutral Carrier...................................................................................................38 2.7. I/O Threads ...............................................................................................................................39 2.8. Limiting Socket Use..................................................................................................................40 2.9. Core Messaging Patterns...........................................................................................................40 2.10. High-level Messaging Patterns................................................................................................41 2.11. Working with Messages ..........................................................................................................42 2.12. Handling Multiple Sockets......................................................................................................43 2.13. Handling Errors and ETERM .................................................................................................46 2.14. Handling Interrupt Signals ......................................................................................................52 2.15. Detecting Memory Leaks........................................................................................................53 2.16. Multi-part Messages................................................................................................................54 2.17. Intermediaries and Proxies......................................................................................................55 2.17.1. The Dynamic Discovery Problem ..............................................................................55 2.17.2. The Shared Queue Problem........................................................................................59 2.17.3. ØMQ’s Built-in Proxy Function.................................................................................66 2.17.4. The Transport Bridging Problem................................................................................67 2.18. Multithreading with ØMQ ......................................................................................................68 2.19. Signaling between Threads .....................................................................................................73 2.20. Node Coordination..................................................................................................................77 2.21. Zero Copy................................................................................................................................81 2.22. Pub-Sub Message Envelopes ..................................................................................................81 iv 2.23. High Water Marks...................................................................................................................84 2.24. A Bare Necessity.....................................................................................................................85 3. Advanced Request-Reply Patterns.....................................................................................................87 3.1. Request-Reply Envelopes .........................................................................................................87 3.2. Custom Request-Reply Routing................................................................................................91 3.3. ROUTER-to-DEALER Routing ...............................................................................................92