Advanced Messaging with AQMP 1.0 Dejan Bosanac WHO AM I?
DEJAN BOSANAC
• Senior Software Engineer at Red Hat • Messaging and Integration background (ActiveMQ, Camel, Fabric8) • Focused more on messaging and backend for IoT lately AGENDA
• Messaging protocols history • Introduction to AMQP 1.0 • AMQP messaging broker • Apache ActiveMQ Artemis • AMQP Messaging scaling • Vertical and horizontal • Qpid Dispatch Router • Scalable deployments MESSAGING PROTOCOLS
• Messaging • Moving data between systems
• Messaging protocol • Defines contract between endpoints • Defines quality of service • Defines data format
5 INSERT DESIGNATOR, IF NEEDED JMS
• Addresses majority of requirements • 1-1, 1-Many,Grouping, • Queues, Topics, Message Selectors • Push • Req/Resp • Reply to queues • Inter-operable • JVM • Connection failures • Durable Messages and Various ACK Modes JMS
• Inter-operability • Java Runtime only • Protocol vs API • Vendor specific • Rich feature set • Complexity on the client • Large footprint • Vendor protocols • High network overhead STOMP
• http://stomp.github.com • Simple Text Orientated Messaging Protocol • HTTP for the messaging realm • Very simple, so it’s easy to write clients and servers in practically any language • A lot of client APIs in C, Java, Ruby, Pyhton, JS, PHP • Implemented by ActiveMQ, Apollo, HornetQ, RabbitMQ • Not standardized • Performance penalties
8 INSERT DESIGNATOR, IF NEEDED MQTT
• OASIS standard (v3.1.1) • Created by IBM and Eurotech • Lightweight • Small network message • Simple protocol • Pub / Sub • Quality of Service • Connection failures • Very popular in IoT scenarios AMQP 1.0
• International Standard (ISO/IEC ISO 19464) • Binary Protocol • Rich feature set: • conversation multiplexing • advanced flow control • Type system • QoS Guarantees • Symmetrical message exchange • No Broker require QOS GUARANTEES
• Quality of Service guarantees • At most once • Fire and Forget • At least once • Retry • Exactly once • 3 Way Ack TYPE SYSTEM
• Highly Interoperable • Rich Type Set • Primitives • integer, long • lists, maps • Descriptive types • Allow application defined types • Mappings in most languages AMQP API
Message( properties: { correlation-id: 1, to: "$management", reply-to: "/myaddress" }, application-properties: { "name" -> "newQueue", "operation" -> "CREATE", "type" -> "org.example.queue" }, application-data: AmqpValue( Map( "max_size" -> "2000Mb" ) ) )
13 INSERT DESIGNATOR, IF NEEDED SYMMETRICAL PROTOCOL
• Client -> Broker • Broker less message • Client -> Client • Client -> Router • Interesting features • Smart routing FLOW CONTROL
• Limit producer and consumer flow • Each channel (link) can be controlled individually • Limit telemetry flow for command messages AMQP MESSAGING STACK
• Clients • Broker • Router
16 INSERT DESIGNATOR, IF NEEDED AMQP CLIENTS
• Java JMS 1.1 client (Apache Qpid JMS based on Qpid Proton) • Reactive C++ client (Apache Qpid Proton) • Reactive Python client (Apache Qpid Proton) • Reactive pure JavaScript client with support for Node.js (GitHub – Rhea) • Fully-featured .NET library (GitHub – AMQP .NET Lite)
17 INSERT DESIGNATOR, IF NEEDED ACTIVEMQ ARTEMIS
• Multi Protocol Broker • AMQP, MQTT, STOMP, OpenWire, Artemis Core • JMS (API) • Started as HornetQ JBoss project in 2009 • Embedded WildFly (JBoss AS) JMS messaging service • In 2014 donated to Apache ActiveMQ • Sub project ActiveMQ Artemis. • Latest Release 1.4.0 ACTIVEMQ ARTEMIS CORE
• Contains Broker Business logic • Core maintains a tight scope • Message Routing • Persistence • Protocol utility API • HA and Scaling • Highly Performance • Protocols are pluggable ACITVEMQ ARTEMIS
• Great performance due to • Reactive Architecture • Efficient Append only Journal
• Multi - Protocol Broker • AMQP, MQTT, STOMP, CORE, OpenWire • JMS
• HA and Scalability built in QPID DISPATCH ROUTER
• Lightweight AMQP 1.0 message router written in C • http://qpid.apache.org/components/dispatch-router/ • Provides flexible and scalable interconnect between AMQP endpoints QPID DISPATCH ROUTER
• It is not a broker • It never owns a message • It propagates AMQP transfer, settlement and disposition frames between endpoints • Message based or link based routing
/device1 Router
/device2 QPID DISPATCH ROUTER
• It can be deployed in multiple router-broker-endpoint topology • Redundant paths
Broker Router
Router QPID DISPATCH ROUTER
Cost-based route computation
Broker Router
Router QPID DISPATCH ROUTER
Automatic re-routing on failure
Broker Router
Router QPID DISPATCH ROUTER
• Better scaling due to more focused tasks • Smart routing can be used to partition the traffic • Ideal candidate for gateway into the system MESSAGING SCALING
• Vertical and Horizontal scaling of Brokers • Qpid Dispatch Router • Scalable Deployment BROKER – HORIZONTAL SCALING
• One broker can only do so much • Horizontal scaling by using networks of brokers • Load balance connections • Limitations • All destinations on all brokers • Broker network is the bottleneck SCALABLE DEPLOYMENTS
• Combination of brokers and routers provides powerful tool box • Brokers should focus on storing messages • Routers should do the rest • Allows for better horizontal scaling topologies SCALABLE DEPLOYMENTS
• Connections concentration • Destinations concentration • Destination sharding • Smart routing CONNECTION CONCENTRATION
Challenge: Reduce the number of connections on the broker
Broker
Router DESTINATION CONCENTRATION
Challenge: Reduce the number of destinations on the broker
/building1/room1
Broker /building1/room2 Router
/building1
/building1/room2 DESTINATION SHARDING
Challenge: Distribute destinations across brokers
Broker
Queue.A
Router Broker
Queue.B DESTINATION PARTITIONING
Challenge: Distribute ONE destination across brokers
Broker
Queue.A
Router Broker
Queue.A DEPLOYMENTS – SMART ROUTING
Challenge: Decrease the load of messages on the broker
QoS 0 Broker
Router DEPLOYMENTS – SMART ROUTING
QoS 1 Broker
Router CONCLUSIONS
• AMQP 1.0 is powerful messaging protocol • Provides options for new messaging patterns like message routing • Enables us to achieve next level of scalability • and messaging-as-service cloud deployments Thank you!