Python Microservices Development
Total Page:16
File Type:pdf, Size:1020Kb
Python Microservices Development Build, test, deploy, and scale microservices in Python Tarek Ziadé BIRMINGHAM - MUMBAI Python Microservices Development Copyright © 2017 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. First published: July 2017 Production reference: 1210717 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78588-111-4 www.packtpub.com Credits Author Copy Editor Tarek Ziadé Sonia Mathur Reviewer Project Coordinator William Kahn-Greene Vaidehi Sawant Commissioning Editor Proofreader Aaron Lazar Safis Editing Acquisition Editor Indexer Chaitanya Nair Mariammal Chettiyar Content Development Editor Graphics Rohit Kumar Singh Jason Monteiro Technical Editor Production Coordinator Pavan Ramchandani Nilesh Mohite About the Author Tarek Ziadé is a Python developer, located in the countryside near Dijon, France. He works at Mozilla in the services team. He founded a French Python user group called Afpy, and he has written several books about Python in French and English. When he is not hacking on his computer or hanging out with his family, he's spending time between his two passions, running and playing the trumpet. You can visit his personal blog (Fetchez le Python) and follow him on Twitter (@tarek_ziade). You can also take a look at one of his books on Amazon, Expert Python Programming, published by Packt. I would like to thank the Packt team for their help, and the following hackers who helped me: Stéfane Fermigier, William Kahn-Greene, Chris Kolosiwsky, Julien Vehent, and Ryan Kelly. I would also like to thank Amina, Milo, Suki, and Freya for their love and patience. I hope you will enjoy this book as much as I’ve enjoyed writing it! About the Reviewer William Kahn-Greene has been writing Python and building applications on the web since the late 90s. He works in the crash-stats group on the crash ingestion pipeline at Mozilla and also maintains a variety of Python libraries, including bleach. When he’s waiting for CI to test his code changes, he’s building things with wood, tending to his tomato plant, and cooking for four. www.PacktPub.com For support files and downloads related to your book, please visit www.PacktPub.com. Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details. At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBook. h t t p s ://w w w . p a c k t p u b . c o m /m a p t Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career. Why subscribe? Fully searchable across every book published by Packt Copy and paste, print, and bookmark content On demand and accessible via a web browser Customer Feedback Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at h t t p s ://w w w . a m a z o n . c o m /d p /1785881116. If you'd like to join our team of regular reviewers, you can e-mail us at [email protected]. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products! Table of Contents Preface 1 Chapter 1: Understanding Microservices 8 Origins of Service-Oriented Architecture 9 The monolithic approach 10 The microservice approach 14 Microservice benefits 16 Separation of concerns 16 Smaller projects 16 Scaling and deployment 17 Microservices pitfalls 18 Illogical splitting 18 More network interactions 19 Data storing and sharing 19 Compatibility issues 20 Testing 20 Implementing microservices with Python 21 The WSGI standard 22 Greenlet and Gevent 23 Twisted and Tornado 25 asyncio 26 Language performances 29 Summary 31 Chapter 2: Discovering Flask 33 Which Python? 35 How Flask handles requests 35 Routing 39 Variables and converters 40 The url_for function 43 Request 44 Response 45 Flask built-in features 47 The session object 47 Globals 48 Signals 49 Extensions and middlewares 51 Templates 52 Configuration 54 Blueprints 56 Error handling and debugging 57 Custom error handler 58 The debug mode 60 A microservice skeleton 61 Summary 64 Chapter 3: Coding, Testing, and Documenting - the Virtuous Cycle 65 Different kinds of tests 67 Unit tests 68 Functional tests 71 Integration tests 73 Load tests 74 End-to-end tests 77 Using WebTest 79 Using pytest and Tox 80 Developer documentation 83 Continuous Integration 88 Travis-CI 89 ReadTheDocs 90 Coveralls 91 Summary 94 Chapter 4: Designing Runnerly 95 The Runnerly application 96 User stories 96 Monolithic design 98 Model 98 View and Template 99 Background tasks 103 Strava token 106 Authentication and authorization 107 Putting together the monolithic design 111 Splitting the monolith 112 Data Service 114 Using Open API 2.0 115 More splitting 118 Summary 119 [ ii ] Chapter 5: Interacting with Other Services 121 Synchronous calls 122 Using Session in a Flask app 123 Connection pooling 127 HTTP cache headers 129 Improving data transfer 132 GZIP compression 132 Binary payloads 134 Putting it together 137 Asynchronous calls 137 Task queues 138 Topic queues 139 Publish/subscribe 144 RPC over AMQP 144 Putting it together 145 Testing 145 Mocking synchronous calls 145 Mocking asynchronous calls 147 Mocking Celery 147 Mocking other asynchronous calls 149 Summary 150 Chapter 6: Monitoring Your Services 151 Centralizing logs 152 Setting up Graylog 154 Sending logs to Graylog 157 Adding extra fields 159 Performance metrics 161 System metrics 161 Code metrics 164 Web server metrics 166 Summary 168 Chapter 7: Securing Your Services 169 The OAuth2 protocol 170 Token-based authentication 172 The JWT standard 173 PyJWT 175 X.509 certificate-based authentication 176 The TokenDealer microservice 179 The POST/oauth/token implementation 180 [ iii ] Using TokenDealer 184 Web application firewall 186 OpenResty - Lua and nginx 188 Rate and concurrency limiting 191 Other OpenResty features 193 Securing your code 194 Asserting incoming data 194 Limiting your application scope 198 Using Bandit linter 199 Summary 202 Chapter 8: Bringing It All Together 204 Building a ReactJS dashboard 205 The JSX syntax 206 React components 207 ReactJS and Flask 210 Using Bower, npm, and Babel 211 Cross-origin resource sharing 215 Authentication and authorization 218 Interacting with Data Service 218 Getting the Strava token 219 JavaScript authentication 221 Summary 223 Chapter 9: Packaging and Running Runnerly 225 The packaging toolchain 226 A few definitions 227 Packaging 228 The setup.py file 228 The requirements.txt file 233 The MANIFEST.in file 235 Versioning 236 Releasing 239 Distributing 241 Running all microservices 244 Process management 246 Summary 250 Chapter 10: Containerized Services 251 What is Docker? 252 Docker 101 254 Running Flask in Docker 256 [ iv ] The full stack - OpenResty, Circus and Flask 258 OpenResty 259 Circus 261 Docker-based deployments 264 Docker Compose 265 Introduction to Clustering and Provisioning 267 Summary 270 Chapter 11: Deploying on AWS 271 AWS overview 272 Routing - Route53, ELB, and AutoScaling 274 Execution - EC2 and Lambda 274 Storage - EBS, S3, RDS, ElasticCache, and CloudFront 276 Messaging - SES, SQS, and SNS 277 Simple Email Service (SES) 277 Simple Queue Service (SQS) 278 Simple Notification Service (SNS) 278 Provisioning and deployment - CloudFormation and ECS 279 Deploying on AWS - the basics 280 Setting up your AWS account 280 Deploying on EC2 with CoreOS 284 Deploying with ECS 288 Route53 294 Summary 296 Chapter 12: What Next? 297 Iterators and generators 298 Coroutines 301 The asyncio library 303 The aiohttp framework 304 Sanic 305 Asynchronous versus synchronous 306 Summary 309 Index 310 [ v ] Preface If we try to deploy our web applications into the cloud, it requires our code to interact with many third-party services. Using microservice architectures, you can build applications that will allow you to manage these interactions. However, this comes with its own set of challenges, since each set has its own complexity, and getting their interaction right isn't easy. This easy-to-follow guide covers techniques to help you overcome these challenges. You will learn how to best design, write, test, and deploy your microservices. The real- world examples will help Python developers create their own Python microservices using the most efficient methods. By the end of this book, you will have acquired the skills to craft applications that are built as small standard units, using all the proven best practices and avoiding the usual traps.