The Zeromq Guide - for Python Developers

Total Page:16

File Type:pdf, Size:1020Kb

The Zeromq Guide - for Python Developers The ZeroMQ Guide - for Python Developers Pieter Hintjens The ZeroMQ Guide - for Python Developers by Pieter Hintjens Dedication This book is dedicated to the ØMQ community. Table of Contents Preface.......................................................................................................................................................ix 1. ØMQ in a Hundred Words............................................................................................................ix 2. How It Began................................................................................................................................ix 3. The Zen of Zero............................................................................................................................ix 4. How This Book Came To Be.........................................................................................................x 5. Audience.......................................................................................................................................xi I. Learning ØMQ.................................................................................................................................... xii 1. Basics.............................................................................................................................................1 1.1. Fixing the World................................................................................................................1 1.2. Starting Assumptions.........................................................................................................2 1.3. Getting the Examples........................................................................................................2 1.4. Ask and Ye Shall Receive..................................................................................................2 1.5. A Minor Note on Strings...................................................................................................7 1.6. Version Reporting..............................................................................................................9 1.7. Getting the Message Out...................................................................................................9 1.8. Divide and Conquer.........................................................................................................13 1.9. Programming with ØMQ.................................................................................................18 1.10. Why We Needed ØMQ..................................................................................................20 1.11. Socket Scalability..........................................................................................................24 1.12. Upgrading from ØMQ v2.2 to ØMQ v3.2....................................................................25 1.13. Warning: Unstable Paradigms!......................................................................................26 2. Sockets and Patterns.....................................................................................................................28 2.1. The Socket API................................................................................................................28 2.2. Messaging Patterns..........................................................................................................33 2.3. Handling Errors and ETERM..........................................................................................52 2.4. Handling Interrupt Signals..............................................................................................57 2.5. Detecting Memory Leaks................................................................................................58 2.6. Multithreading with ØMQ...............................................................................................59 2.7. Signaling Between Threads (PAIR Sockets)...................................................................64 2.8. Node Coordination..........................................................................................................67 2.9. Zero-Copy........................................................................................................................70 2.10. Pub-Sub Message Envelopes.........................................................................................71 2.11. High-Water Marks.........................................................................................................73 2.12. Missing Message Problem Solver.................................................................................74 3. Advanced Request-Reply Patterns...............................................................................................78 3.1. The Request-Reply Mechanisms.....................................................................................78 3.2. Request-Reply Combinations..........................................................................................83 3.3. Exploring ROUTER Sockets...........................................................................................86 3.4. The Load Balancing Pattern............................................................................................88 3.5. A High-Level API for ØMQ.........................................................................................101 3.6. The Asynchronous Client/Server Pattern......................................................................108 3.7. Worked Example: Inter-Broker Routing.......................................................................114 4. Reliable Request-Reply Patterns................................................................................................140 4.1. What is "Reliability"?....................................................................................................140 4.2. Designing Reliability.....................................................................................................141 iv 4.3. Client-Side Reliability (Lazy Pirate Pattern).................................................................142 4.4. Basic Reliable Queuing (Simple Pirate Pattern)............................................................146 4.5. Robust Reliable Queuing (Paranoid Pirate Pattern)......................................................150 4.6. Heartbeating..................................................................................................................158 4.7. Contracts and Protocols.................................................................................................161 4.8. Service-Oriented Reliable Queuing (Majordomo Pattern)............................................162 4.9. Asynchronous Majordomo Pattern................................................................................178 4.10. Service Discovery........................................................................................................186 4.11. Idempotent Services....................................................................................................187 4.12. Disconnected Reliability (Titanic Pattern)..................................................................188 4.13. High-Availability Pair (Binary Star Pattern)...............................................................193 4.14. Brokerless Reliability (Freelance Pattern)...................................................................200 4.15. Conclusion...................................................................................................................208 5. Advanced Pub-Sub Patterns.......................................................................................................209 5.1. Pros and Cons of Pub-Sub.............................................................................................209 5.2. Pub-Sub Tracing (Espresso Pattern)..............................................................................211 5.3. Last Value Caching........................................................................................................211 5.4. Slow Subscriber Detection (Suicidal Snail Pattern)......................................................213 5.5. High-Speed Subscribers (Black Box Pattern)...............................................................216 5.6. Reliable Pub-Sub (Clone Pattern).................................................................................219 II. Advanced ØMQ................................................................................................................................239 6. The ØMQ Community...............................................................................................................240 6.1. Architecture of the ØMQ Community..........................................................................241 6.2. How to Make Really Large Architectures.....................................................................242 6.3. The ØMQ Process: C4...................................................................................................251 6.4. A Real-Life Example.....................................................................................................263 6.5. Git Branches Considered Harmful................................................................................266 6.6. Designing for Innovation...............................................................................................270 6.7. Burnout..........................................................................................................................277
Recommended publications
  • Evaluating DDS, MQTT, and Zeromq Under Different Iot Traffic Conditions
    Evaluating DDS, MQTT, and ZeroMQ Under Different IoT Traffic Conditions Zhuangwei Kang Robert Canady Abhishek Dubey Vanderbilt University Vanderbilt University Vanderbilt University Nashville, Tennessee Nashville, Tennessee Nashville, Tennessee [email protected] [email protected] [email protected] Aniruddha Gokhale Shashank Shekhar Matous Sedlacek Vanderbilt University Siemens Technology Siemens Technology Nashville, Tennessee Princeton, New Jersey Munich, Germany [email protected] [email protected] [email protected] Abstract Keywords: Publish/Subscribe Middleware, Benchmark- ing, MQTT, DDS, ZeroMQ, Performance Evaluation Publish/Subscribe (pub/sub) semantics are critical for IoT applications due to their loosely coupled nature. Although OMG DDS, MQTT, and ZeroMQ are mature pub/sub solutions used for IoT, prior studies show that their performance varies significantly under different 1 Introduction load conditions and QoS configurations, which makes Distributed deployment of real-time applications and middleware selection and configuration decisions hard. high-speed dissemination of massive data have been hall- Moreover, the load conditions and role of QoS settings in marks of the Internet of Things (IoT) platforms. IoT prior comparison studies are not comprehensive and well- applications typically adopt publish/subscribe (pub/- documented. To address these limitations, we (1) propose sub) middleware for asynchronous and cross-platform a set of performance-related properties for pub/sub mid- communication. OMG Data Distribution Service (DDS), dleware and investigate their support in DDS, MQTT, ZeroMQ, and MQTT are three representative pub/sub and ZeroMQ; (2) perform systematic experiments under technologies that have entirely different architectures (de- three representative, lab-based real-world IoT use cases; centralized data-centric, decentralized message-centric, and (3) improve DDS performance by applying three and centralized message-centric, respectively).
    [Show full text]
  • This Paper Must Be Cited As
    Document downloaded from: http://hdl.handle.net/10251/64607 This paper must be cited as: Luzuriaga Quichimbo, JE.; Pérez, M.; Boronat, P.; Cano Escribá, JC.; Tavares De Araujo Cesariny Calafate, CM.; Manzoni, P. (2015). A comparative evaluation of AMQP and MQTT protocols over unstable and mobile networks. 12th IEEE Consumer Communications and Networking Conference (CCNC 2015). IEEE. doi:10.1109/CCNC.2015.7158101. The final publication is available at http://dx.doi.org/10.1109/CCNC.2015.7158101 Copyright IEEE Additional Information © 2015 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. A comparative evaluation of AMQP and MQTT protocols over unstable and mobile networks Jorge E. Luzuriaga∗, Miguel Perezy, Pablo Boronaty, Juan Carlos Cano∗, Carlos Calafate∗, Pietro Manzoni∗ ∗Department of Computer Engineering Universitat Politecnica` de Valencia,` Valencia, SPAIN [email protected], jucano,calafate,[email protected] yUniversitat Jaume I, Castello´ de la Plana, SPAIN [email protected], [email protected] Abstract—Message oriented middleware (MOM) refers to business application [6]. It works like instant messaging or the software infrastructure supporting sending and receiving email, and the difference towards these available
    [Show full text]
  • Mobile Telemedicine and Wireless Remote Monitoring Applications
    İSTANBUL TECHNICAL UNIVERSITY INSTITUTE OF SCIENCE AND TECHNOLOGY MOBILE TELEMEDICINE AND WIRELESS REMOTE MONITORING APPLICATIONS M.Sc. Thesis by Taner SOYUGENÇ, B.Sc. Department : Electronics and Communication Engineering Programme : Biomedical Engineering NOVEMBER 2006 PREFACE In this project, my main goal is to implement a mobile sample application by defining the related global standards for telemedicine. The work is focused on recommendations of technology associated with a feasibility study. First of all, I would like to thank Assoc. Prof. Dr. Selçuk PAKER for his valuable advice, support and encouragement to accomplish the project. Besides, I would like to thank my family who is always with me giving support at every step of my life. November 2006 Taner SOYUGENÇ iii CONTENTS ACRONYMS vi LIST OF TABLES viii LIST OF FIGURES ix SUMMARY xi ÖZET xii 1. INTRODUCTION 1 1.1. Technology Overview 2 1.1.1. Communication Infrastructure 5 1.1.2. Overview of GSM-GPRS 6 1.1.2.1. Brief History of GSM 8 1.1.2.2. GPRS 12 1.1.3. Mobile Solutions 14 1.1.4. Wireless Medical Sensors 15 1.2. Aim of the Project 16 2. WORLDWIDE APPLICATIONS, VENDORS AND STANDARDS 18 2.1. Available Products 19 2.1.1. ECG 19 2.1.2. Pulse Oximeter 20 2.1.3. Blood Pressure Sensor 23 2.1.4. Various Sensor Brands 24 2.1.5. Advanced Research 27 2.1.6. Home Care Monitoring Systems 31 2.2. Medical Information Standards and Organizations 35 2.2.1. ASTM 39 2.2.2. CEN/TC251 Health Informatics 39 2.2.3.
    [Show full text]
  • Robust Messaging with Rabbitmq
    Spring RabbitMQ for High Load Martin Toshev Who am I Software consultant (CoffeeCupConsulting) BG JUG board member (http://jug.bg) OpenJDK and Oracle RDBMS enthusiast Twitter: @martin_fmi 2 3 Agenda • Messaging Basics • RabbitMQ Overview • Spring RabbitMQ 4 Messaging Basics 5 Messaging • Messaging provides a mechanism for loosely-coupled integration of systems • The central unit of processing in a message is a message which typically contains a body and a header 6 Use cases • Offloading long-running tasks to worker nodes • Distributing and processing high loads of data • Aggregating logs and propagating events between systems • Many others … 7 Messaging protocols • Messaging solutions implement different protocols for transferring of messages such as AMQP, XMPP, MQTT, STOMP, Kafka binary protocol and others • The variety of protocols imply vendor lock-in 8 Messaging protocols comparison AMQP MQTT XMPP STOMP Kafka goal replacement of messaging for instant messaging, message-oriented processing of large proprietary protocols resource-constrained adopted for wider middleware real-time data feeds devices use format binary binary XML-based text-based binary API divided into classes simple (5 basic different XML items ~ 10 basic commands 42 request types in (> 40 methods in operations with 2-3 with multiple types latest version (Kafka RabbitMQ) packet types for each) 2.0.0) reliability publisher/subscriber acknowledgements Acknowledgments Subscriber Acknowledgements, acknowledgements, and resumptions acknowledgements transactional transactions
    [Show full text]
  • Rocketbufs: a Framework for Building Efficient, In-Memory, Message-Oriented Middleware
    RocketBufs: A Framework for Building Efficient, In-Memory, Message-Oriented Middleware Huy Hoang, Benjamin Cassell, Tim Brecht, Samer Al-Kiswany Cheriton School of Computer Science, University of Waterloo ABSTRACT ACM Reference Format: As companies increasingly deploy message-oriented middleware Huy Hoang, Benjamin Cassell, Tim Brecht, Samer Al-Kiswany. 2020. Rock- (MOM) systems in mission-critical components of their infrastruc- etBufs: A Framework for Building Efficient, In-Memory, Message-Oriented Middleware. In The 14th ACM International Conference on Distributed and tures and services, the demand for improved performance and func- Event-based Systems (DEBS ’20), July 13–17, 2020, Virtual Event, QC, Canada. tionality has accelerated the rate at which new systems are being ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/3401025.3401744 developed. Unfortunately, existing MOM systems are not designed to take advantages of techniques for high-performance data center communication (e.g., RDMA). In this paper, we describe the design 1 INTRODUCTION and implementation of RocketBufs, a framework which provides Message-Oriented Middleware (MOM) systems, also often referred infrastructure for building high-performance, in-memory Message- to as publish/subscribe, message-queuing, or event-based systems, Oriented Middleware (MOM) applications. RocketBufs provides are a popular class of software designed to support loosely-coupled memory-based buffer abstractions and APIs, which are designed messaging in modern distributed applications. Examples of appli- to work efficiently with different transport protocols. Applications cations and services that utilize MOM systems include IBM’s cloud implemented using RocketBufs manage buffer data using input functions [29], the Apache OpenWhisk serverless framework [10], (rIn) and output (rOut) classes, while the framework is responsible the Hyperledger blockchain framework [7] and streaming media for transmitting, receiving and synchronizing buffer access.
    [Show full text]
  • Analysis of Notification Methods with Respect to Mobile System Characteristics
    Proceedings of the Federated Conference on DOI: 10.15439/2015F6 Computer Science and Information Systems pp. 1183–1189 ACSIS, Vol. 5 Analysis of notification methods with respect to mobile system characteristics Piotr Nawrocki ∗, Mikołaj Jakubowski † and Tomasz Godzik ‡ ∗AGH University of Science and Technology, al. A. Mickiewicza 30, 30-059 Krakow, Poland e-mail:[email protected] †e-mail:[email protected] ‡e-mail:[email protected] Abstract—Recently, there has been an increasing need for most promise and therefore the purpose is to discern their secure, efficient and simple notification methods for mobile usefulness in the best way possible. systems. Such systems are meant to provide users with precise In addition to the protocols and methods above, we inves- tools best suited for work or leisure environments and a lot of effort has been put into creating a multitude of mobile tigated other solutions, such as the Apple push notification applications. However, not much research has been put at the or Line application which, for various reasons, were not same time into determining which of the available protocols considered further. The Apple push notification technology is a are best suited for individual tasks. Here a number of basic good solution, but it is proprietary, i.e. limited to Apple devices notification methods are presented and tests are performed for and that is why we decided to test more universal solutions the most promising ones. An attempt is made to determine which methods have the best throughput, latency, security and other first. There are also solutions (applications) that use their own characteristics.
    [Show full text]
  • SEED™ Assessment Guide for Family Planning Programming
    SEED™ Assessment Guide for Family Planning Programming SEED™ Assessment Guide for Family Planning Programming © 2011 EngenderHealth EngenderHealth 440 Ninth Avenue New York, NY 10001 U.S.A. Telephone: 212-561-8000 Fax: 212-561-8067 e-mail: [email protected] www.engenderhealth.org This publication was made possible through support provided by the F.M. Kirby Foundation. The opinions expressed herein are those of the publisher and do not necessarily reflect the views of the foundation. Cover design, graphic design, and typesetting: Weronika Murray and Tor de Vries Printing: Automated Graphic Systems ISBN 978-1-885063-97-7 Printed in the United States of America. Printed on recycled paper. Suggested citation: EngenderHealth. 2011. The SEED assessment guide for family planning programming. New York. Photo credits: M. Tuschman/EngenderHealth, A. Fiorente/EngenderHealth, C. Svingen/EngenderHealth. ii ContEntS Acknowledgments .................................................................................................................................. iv Acronyms and Abbreviations ................................................................................................................... v Introduction ................................................................................................................................. 2 EngenderHealth’s SEED Programming Model .......................................................................................... 3 How to Use This Assessment Guide........................................................................................................
    [Show full text]
  • Next Generation Web Scanning Presentation
    Next generation web scanning New Zealand: A case study First presented at KIWICON III 2009 By Andrew Horton aka urbanadventurer NZ Web Recon Goal: To scan all of New Zealand's web-space to see what's there. Requirements: – Targets – Scanning – Analysis Sounds easy, right? urbanadventurer (Andrew Horton) www.morningstarsecurity.com Targets urbanadventurer (Andrew Horton) www.morningstarsecurity.com Targets What does 'NZ web-space' mean? It could mean: •Geographically within NZ regardless of the TLD •The .nz TLD hosted anywhere •All of the above For this scan it means, IPs geographically within NZ urbanadventurer (Andrew Horton) www.morningstarsecurity.com Finding Targets We need creative methods to find targets urbanadventurer (Andrew Horton) www.morningstarsecurity.com DNS Zone Transfer urbanadventurer (Andrew Horton) www.morningstarsecurity.com Find IP addresses on IRC and by resolving lots of NZ websites 58.*.*.* 60.*.*.* 65.*.*.* 91.*.*.* 110.*.*.* 111.*.*.* 113.*.*.* 114.*.*.* 115.*.*.* 116.*.*.* 117.*.*.* 118.*.*.* 119.*.*.* 120.*.*.* 121.*.*.* 122.*.*.* 123.*.*.* 124.*.*.* 125.*.*.* 130.*.*.* 131.*.*.* 132.*.*.* 138.*.*.* 139.*.*.* 143.*.*.* 144.*.*.* 146.*.*.* 150.*.*.* 153.*.*.* 156.*.*.* 161.*.*.* 162.*.*.* 163.*.*.* 165.*.*.* 166.*.*.* 167.*.*.* 192.*.*.* 198.*.*.* 202.*.*.* 203.*.*.* 210.*.*.* 218.*.*.* 219.*.*.* 222.*.*.* 729,580,500 IPs. More than we want to try. urbanadventurer (Andrew Horton) www.morningstarsecurity.com IP address blocks in the IANA IPv4 Address Space Registry Prefix Designation Date Whois Status [1] -----
    [Show full text]
  • Advanced Architecture for Java Universal Message Passing (AA-JUMP)
    The International Arab Journal of Information Technology, Vol. 15, No. 3, May 2018 429 Advanced Architecture for Java Universal Message Passing (AA-JUMP) Adeel-ur-Rehman1 and Naveed Riaz2 1National Centre for Physics, Pakistan 2School of Electrical Engineering and Computer Science, National University of Science and Technology, Pakistan Abstract: The Architecture for Java Universal Message Passing (A-JUMP) is a Java based message passing framework. A- JUMP offers flexibility for programmers in order to write parallel applications making use of multiple programming languages. There is also a provision to use various network protocols for message communication. The results for standard benchmarks like ping-pong latency, Embarrassingly Parallel (EP) code execution, Java Grande Forum (JGF) Crypt etc. gave us the conclusion that for the cases where the data size is smaller than 256K bytes, the numbers are comparative with some of its predecessor models like Message Passing Interface CHameleon version 2 (MPICH2), Message Passing interface for Java (MPJ) Express etc. But, in case, the packet size exceeds 256K bytes, the performance of the A-JUMP model seems to be severely hampered. Hence, taking that peculiar behaviour into account, this paper talks about a strategy devised to cope up with the performance limitation observed under the base A-JUMP implementation, giving birth to an Advanced A-JUMP (AA- JUMP) methodology while keeping the basic workflow of the original model intact. AA-JUMP addresses to improve performance of A-JUMP by preserving its various traits like portability, simplicity, scalability etc. which are the key features offered by flourishing High Performance Computing (HPC) oriented frameworks of now-a-days.
    [Show full text]
  • Dcamp: Distributed Common Api for Measuring
    DCAMP: DISTRIBUTED COMMON API FOR MEASURING PERFORMANCE A Thesis presented to the Faculty of California Polytechnic State University San Luis Obispo In Partial Fulfillment of the Requirements for the Degree Master of Science in Computer Science by Alexander Paul Sideropoulos December 2014 c 2014 Alexander Paul Sideropoulos ALL RIGHTS RESERVED ii COMMITTEE MEMBERSHIP TITLE: dCAMP: Distributed Common API for Measuring Performance AUTHOR: Alexander Paul Sideropoulos DATE SUBMITTED: December 2014 COMMITTEE CHAIR: Michael Haungs, Ph.D. Associate Professor of Computer Science COMMITTEE MEMBER: Aaron Keen, Ph.D. Assistant Professor of Computer Science COMMITTEE MEMBER: John Bellardo, Ph.D. Associate Professor of Computer Science iii ABSTRACT dCAMP: Distributed Common API for Measuring Performance Alexander Paul Sideropoulos Although the nearing end of Moore's Law has been predicted numerous times in the past [22], it will eventually come to pass. In forethought of this, many modern computing systems have become increasingly complex, distributed, and parallel. As software is developed on and for these complex systems, a common API is necessary for gathering vital performance related metrics while remaining transparent to the user, both in terms of system impact and ease of use. Several distributed performance monitoring and testing systems have been proposed and implemented by both research and commercial institutions. How- ever, most of these systems do not meet several fundamental criterion for a truly useful distributed performance monitoring system: 1) variable data delivery mod- els, 2) security, 3) scalability, 4) transparency, 5) completeness, 6) validity, and 7) portability [30]. This work presents dCAMP: Distributed Common API for Measuring Per- formance, a distributed performance framework built on top of Mark Gabel and Michael Haungs' work with CAMP.
    [Show full text]
  • Seed Modules Reference Manual
    Seed Modules Reference Manual May 20, 2009 Seed Modules Reference Manual ii Contents 1 readline module. 1 1.1 API Reference . .1 1.2 Examples . .1 2 SQLite module. 3 2.1 API Reference . .3 2.2 Examples . .3 3 GtkBuilder module. 5 3.1 API Reference . .5 3.2 Examples . .5 4 Sandbox module. 7 4.1 API Reference . .7 4.2 Examples . .7 iii Chapter 1 readline module. Robert Carr 1.1 API Reference The readline module allows for basic usage of the GNU readline library, in Seed. More advanced fea- tures may be added a a later time. In order to use the readline module it must be first imported. readline= imports.readline; readline.readline (prompt) Prompts for one line of input on standard input using prompt as the prompt. prompt A string to use as the readline prompt Returns A string entered on standard input. readline.bind (key, function) Binds key to function causing the function to be invokved when- ever key is pressed key A string specifying the key to bind function The function to invoke when key is pressed readline.done () Indicates that readline should finish the current line, and return from readline- .readline. Can be used in callbacks to implement features like multiline editing readline.buffer() Retrieve the current readline buffer Returns The current readline buffer readline.insert (string) Inserts string in to the current readline buffer string The string to insert 1.2 Examples Below are several examples of using the Seed readline module. For additional resources, consult the examples/ folder of the Seed source 1 CHAPTER 1.
    [Show full text]
  • (12) United States Patent (10) Patent No.: US 7,203,956 B2 Thomas Et Al
    USOO7203956B2 (12) United States Patent (10) Patent No.: US 7,203,956 B2 Thomas et al. (45) Date of Patent: Apr. 10, 2007 (54) SYSTEM AND METHOD FOR THE SECURE 5,408,465. A 4, 1995 Gusella et al. ................ 37O/17 ENROLLMENT OF DEVICES WITH A 5.434,848 A 7/1995 Chimento, Jr. et al. CLEARNGHOUSE SERVER FOR INTERNET 5,473,630 A 12/1995 Penzias et al. TELEPHONY AND MULTIMEDIA 5,563,939 A 10, 1996 La Porta et al. COMMUNICATIONS 5,570,417 A 10/1996 Byers et al. 5,581,544 A 12/1996 Hamada et al. ............. 370,253 (75) Inventors: Stephen Thomas, Marietta, GA (US); 5,600,794. A 3. R - - - - - - - i - - - - - - - - - - - 395.200.01 Rodney Scott McManus, Atlanta, GA 5,606,602 A 2, 1997 Johnson et al. (US); Rick Vaughn, Roswell, GA (US) 5,633,919 A 5/1997 Hogan et al. s s s 5,638.433 A 6/1997 Bubien, Jr. et al. 5,668,955 A 9, 1997 deCiutiis et al. (73) Assignee: TransNexus, Inc., Atlanta, GA (US) 5,675,636 A 10/1997 Gray c - r 5,712.907 A 1/1998 Wegner et al. (*) Notice: Subject to any disclaimer, the term of this 5,740,361 A 4, 1998 Brown .................. 395,187.01 patent is extended or adjusted under 35 5,790,642 A 8, 1998 Tavlor et all U.S.C. 154(b) by 984 days. - W y (21) Appl. No.: 09/747,365 (Continued) (22) Filed: Dec. 22, 2000 FOREIGN PATENT DOCUMENTS e Afaf 9 EP O 781.
    [Show full text]