Pro Couchbase Server

Second Edition

David Ostrovsky Mohammed Haji Yaniv Rodenski Pro Couchbase Server, Second Edition Copyright © 2015 by David Ostrovsky, Mohammed Haji, and Yaniv Rodenski This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’s location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law. ISBN-13 (pbk): 978-1-4842-1186-1 ISBN-13 (electronic): 978-1-4842-1185-4 Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image, we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Managing Director: Welmoed Spahr Lead Editor: Jonathan Gennick Technical Reviewer: Kenneth Williams Editorial Board: Steve Anglin, Louise Corrigan, Jim DeWolf, Jonathan Gennick, Robert Hutchinson, Michelle Lowman, James Markham, Susan McDermott, Matthew Moodie, Jeffrey Pepper, Douglas Pundick, Ben Renow-Clarke, Gwenan Spearing Coordinating Editor: Jill Balzano Copy Editor: Michael G. Laraque Compositor: SPi Global Indexer: SPi Global Artist: SPi Global Cover Designer: Anna Ishchenko Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail [email protected], or visit www.springer.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please e-mail [email protected], or visit www.apress.com. Apress and friends of ED books may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Special Bulk Sales–eBook Licensing web page at www.apress.com/bulk-sales. Any source code or other supplementary material referenced by the author in this text is available to readers at www.apress.com/. For detailed information about how to locate your book’s source code, go to www.apress.com/source-code/. To my wonderful wife, Dina, for not divorcing me despite my working weekends on this book; my son, Adam, for being awesome in general; and my daughter, Abigail, for sleeping well while I worked. —David Ostrovsky

Thanks to all my close family. This is especially to my mum, sisters, my nephew Hamzah, and Talal for being patient and supportive. —Mohammed Haji

For my wife, Morin, and my daughters, Romy and Ella—thanks for letting Daddy do this. —Yaniv Rodenski

Contents at a Glance

About the Authors ��xvii About the Technical Reviewer ��xix Acknowledgments ��xxi Foreword ��xxiii

■■Part I: Getting Started �� 1 ■■Chapter 1: Getting Started with Couchbase Server �� 3 ■■Chapter 2: Designing Document-Oriented Databases with Couchbase �� 19

■■Part II: Development �� 31 ■■Chapter 3: The Couchbase Client Libraries �� 33 ■■Chapter 4: CRUD and Key-Based Operations �� 61 ■■Chapter 5: Working with Views �� 83 ■■Chapter 6: The N1QL Query Language �� 107 ■■Chapter 7: Advanced Couchbase Techniques �� 135 ■■Chapter 8: ElasticSearch Integration �� 159 ■■Part III: Under the Hood �� 179 ■■Chapter 9: Sizing and Deployment Considerations �� 181 ■■Chapter 10: Basic Administration �� 197

v ■ Contents at a Glance

■■Part IV: Couchbase at Scale �� 229 ■■Chapter 11: Monitoring and Best Practices �� 231 ■■Chapter 12: Couchbase Server in the Cloud �� 267 ■■Chapter 13: Cross-Datacenter Replication (XDCR) �� 291 ■■Chapter 14: Couchbase Lite on Android �� 307 ■■Chapter 15: Couchbase Lite on iOS �� 319

■■Part V: Mobile Development with Couchbase �� 329 ■■Chapter 16: Synchronizing Data with the Couchbase Sync Gateway �� 331

Index �� 343

vi Contents

About the Authors ��xvii About the Technical Reviewer ��xix Acknowledgments ��xxi Foreword ��xxiii

■■Part I: Getting Started �� 1 ■■Chapter 1: Getting Started with Couchbase Server �� 3 From SQL to NoSQL �� 3 The CAP Theorem ��4 NoSQL and Couchbase Server ��4 Couchbase as Key-Value Store vs. Document Database �� 5 Couchbase Server Architecture �� 6 Data Storage �� 7 Installing Couchbase Server �� 8 Selecting a Couchbase Server Edition ��9 Installing Couchbase on Different Operating Systems ��9 Configuring Couchbase Server �� 10 Creating a Bucket ��16 Summary �� 18

vii ■ Contents

■■Chapter 2: Designing Document-Oriented Databases with Couchbase �� 19 RanteR: The Antisocial Network �� 19 Mapping Application Entities �� 20 Using a Relational Database ��22 Using a Document-Oriented Approach with Couchbase ��23 Designing Keys �� 24 Natural Keys ��25 Creating Your First Document ��25 Counters ��27 Universally Unique Identifiers (UUID) ��27 Metadata �� 28 Document Versioning �� 29 Versioning Considerations When Using the Couchbase SDKs ��29 Versioning Considerations in Couchbase Views ��29 Versioning Consideration with N1QL Queries ��29 Summary �� 30 ■■Part II: Development �� 31 ■■Chapter 3: The Couchbase Client Libraries �� 33 The Java SDK �� 34 Adding the Java SDK Manually ��34 Using Maven ��34 Using the Client ��35 The .NET SDK �� 37 Manually Referencing the Assemblies ��38 Using NuGet Package Manager ��38 Using the Client ��38 The C SDK �� 41 Linux ��42 Windows ��42

viii ■ Contents

Mac OS X ��42 Using libcouchbase ��42 The Node.js SDK �� 45 The PHP SDK �� 47 Linux ��47 Windows ��48 Mac OS X ��48 Using the PHP SDK ��49 The Python SDK �� 50 Linux ��51 Windows ��51 Mac OS X ��51 Using the Python SDK ��51 Asynchronous Framework Integration ��53 The Ruby SDK �� 54 A Simple Ruby Script ��54 Building a Rails Application ��55 Inside the Couchbase Client Libraries �� 57 Thread Safety ��58 Couchbase Client Initialization ��58 Performing Operations Using the Couchbase Client ��59 Summary �� 60 ■■Chapter 4: CRUD and Key-Based Operations �� 61 Persistence and Replication �� 61 Concurrency �� 62 Document Expiration �� 63 Database Operations �� 64 Storing Data ��65 Retrieving Data ��67

ix ■ Contents

Retrieving and Updating Data Concurrently ��69 Pessimistic Concurrency Through Locking ��72 Appending and Prepending Binary Data ��75 Numeric Data ��77 Deleting Data ��79 Managing the Cluster �� 80 Summary �� 81 ■■Chapter 5: Working with Views �� 83 MapReduce 101 �� 83 Creating a View in Couchbase �� 84 The map Function ��86 Creating Views from Code ��88 Querying Views ��90 The Reduce Function ��94 Views and Persistence �� 99 Updating Indexes ��100 Views and Expiration ��102 Multidimensional and Geospatial Views �� 102 Creating Geospatial Views ��102 Summary �� 105 ■■Chapter 6: The N1QL Query Language �� 107 Executing N1QL Queries �� 107 The N1QL Language �� 112 The FROM Clause ��116 Query Conditions and Expressions ��121 Sorting and Grouping ��123 Indexing Properties ��126 Exploring the Query Execution Details ��128 Index Equivalence ��130

x ■ Contents

Building Indexes Asynchronously ��130 Data Manipulation with N1QL ��131 Summary �� 133 ■■Chapter 7: Advanced Couchbase Techniques �� 135 Software Error Handling �� 135 Reading Replica Documents �� 136 Handling Temporary Out-of-Memory Errors �� 137 Implementing Transactions with Two-Phase Commit �� 138 Maintaining Sets of Items �� 149 Using CAS As an ETag �� 152 Using Couchbase As an Out-of-Process Session Store �� 155 Summary �� 158 ■■Chapter 8: ElasticSearch Integration �� 159 Setting Up ElasticSearch Clusters �� 160 Forming a Cluster ��161 Setting Up Replication Between Couchbase and ElasticSearch ��162 Querying with ElasticSearch �� 164 The ElasticSearch Query DSL ��166 Summary �� 178

■■Part III: Under the Hood �� 179 ■■Chapter 9: Sizing and Deployment Considerations �� 181 Multidimensional Scaling �� 181 Planning the Couchbase Cluster �� 183 RAM ��183 Storage ��187 Storage Performance ��188 CPU ��189 Nodes ��191 Network Topology ��191

xi ■ Contents

Replication ��194 Swap Space ��195 Summary �� 196 ■■Chapter 10: Basic Administration �� 197 Building the Cluster �� 197 Adding a Node to the Couchbase Cluster ��198 Removing a Node from the Cluster ��200 Rebalancing ��201 Swap Rebalancing ��203 Changing the Data and Index Path ��204 Backing Up and Restoring �� 204 Using cbbackup to Back Up Data ��204 Using cbrestore to Restore Backed-Up Data ��209 Using cbtransfer to Transfer Data ��210 Backing Up and Restoring Using Data Files ��211 Database and View Compaction �� 212 Dealing with Compaction ��212 Auto-Compaction ��215 Triggering Compaction Externally ��218 Compaction in Action ��220 Failover �� 223 Automatic Failover ��223 Manual or Monitored Failover ��224 Dealing with Failed Nodes ��227 Summary �� 227

■■Part IV: Couchbase at Scale �� 229 ■■Chapter 11: Monitoring and Best Practices �� 231 Less Obvious Couchbase Mechanisms �� 231 Replication and Queues ��231 Working-Set Management and Ejection ��233 xii ■ Contents

Controlling the Purge Interval �� 234 Monitoring �� 234 Monitoring Couchbase at the System Level ��237 Detailed Per-Bucket Monitoring ��241 Server Resources ��242 Summary (the Category) ��244 vBucket Resources ��248 Disk Queues ��250 TAP Queues ��252 DCP Queues �� 254 Views �� 255 Index �� 255 Query ��256 XDCR ��257 Memcached Buckets ��257 Statistics and Cluster Health �� 258 Using cbstats ��259 Retrieving Statistics from the HTTP API ��260 Server Warm-up �� 261 Logs �� 262 Alerts �� 263 Optimizing Disk Throughput �� 265 Summary �� 266 ■■Chapter 12: Couchbase Server in the Cloud �� 267 Couchbase Server on Amazon Web Services �� 267 Considerations for Deploying Couchbase Server on AWS ��268 Setting Up Couchbase Server on AWS ��270

xiii ■ Contents

Couchbase Server on Microsoft Azure �� 281 Considerations for Deploying Couchbase on Microsoft Azure ��282 Setting Up Couchbase Server on Microsoft Azure ��285 Best Practices for Running Couchbase in Microsoft Azure ��289 Summary �� 290 ■■Chapter 13: Cross-Datacenter Replication (XDCR) �� 291 Prerequisites �� 291 Setting Up Unidirectional XDCR �� 292 Advanced XDCR Settings �� 297 Conflict Resolution �� 299 Bidirectional Replication and Advanced Topologies �� 300 Monitoring XDCR �� 301 Outbound XDCR Operations ��302 Incoming XDCR Operations ��303 Recovering Data from a Remote Cluster �� 303 Summary �� 306 ■■Chapter 14: Couchbase Lite on Android �� 307 Getting Started with Couchbase Lite �� 308 Adding Couchbase Lite to an Android Project ��308 Creating a Couchbase Lite Manager ��310 Creating a Couchbase Lite Database ��311 Using Couchbase Lite �� 312 CRUD Operations ��312 Attachments ��314 Views �� 315 Defining Views ��315 Querying Views ��316 Summary �� 317

xiv ■ Contents

■■Chapter 15: Couchbase Lite on iOS �� 319 Getting Started with Couchbase Lite on iOS �� 319 Adding Couchbase Lite to an iOS Project ��319 Creating a Couchbase Lite Manager ��321 Creating a Couchbase Lite Database ��322 Using Couchbase Lite �� 322 CRUD Operations ��323 Attachments ��325 Views �� 326 Defining Views ��327 Querying Views ��328 Summary �� 328

■■Part V: Mobile Development with Couchbase �� 329 ■■Chapter 16: Synchronizing Data with the Couchbase Sync Gateway �� 331 Couchbase Sync Gateway �� 331 Installing the Sync Gateway ��332 Running the Sync Gateway ��332 Working with the Sync Gateway ��334 Channels ��336 The Sync Function ��337 Sync Gateway Administration �� 339 Authenticating Users ��339 Access Control ��340 Shadowing Couchbase Server Buckets ��341 Deploying the Sync Gateway ��341 Summary �� 342

Index �� 343

About the Authors

When he was nine years old, little David Ostrovsky found a book in Russian entitled (in English) Electronic Computational Machines at the local library, and after reading it cover-to-cover in a single weekend, he decided what he was going to do with his life. Three years later, he finally got to touch a computer for the first time and discovered that it was totally worth the wait. One thing led to another, and now he’s a senior solutions architect at Couchbase. David is a software developer with more than 16 years of industry experience as a speaker, trainer, blogger, and coauthor of the book Pro Couchbase Server, published by Apress Media, and author of the “Essentials of Developing Windows Store Apps Using C#” and “Advanced Windows Store App Development Using C#” Microsoft official courses (20484, 20485). In addition, he has written numerous articles. He specializes in big data architecture and large-scale distributed systems

Mohammed Haji is a senior solution architect within Couchbase EMEA. He is an expert in Grid computing (high-performance computing) and big data, with a Ph.D. in this area. He has published in numerous international conference papers (such as IEEE), Oxford University Press journals, book chapters, and industry magazines. He has worked extensively on large distributed, low-latency, high-throughput, mission-critical applications in such fields as civil aviation, instant credit card fraud detection, investment banking, and online gaming. He has helped many institutions facing chronic downtime and performance degradation to become more efficient and was able to handle such major events as the Grand National, Cheltenham, Cyber Monday, and Black Friday.

Yaniv Rodenski is a principal consultant at UXC Professional Solutions, a speaker, an author, and an all-around nerd. Yaniv has been developing software as a hobby since he was young and professionally since 1997. He is the coauthor of the “Developing Windows Azure and Web Services” Microsoft official course (20487) and, over the years, has been a contributor to official Microsoft training materials on Windows Azure, Windows 2008 R2 HPC Server, and HDInsight (Hadoop).

xvii

About the Technical Reviewer

Kenneth Williams is a senior architect at Burstly, a mobile analytics and advertising company based in beautiful Santa Monica, California. He is an accomplished big data architect, with extensive work experience on large-scale databases. He was an early adopter of key-value stores, specifically Couchbase, and has spoken publicly about his implementation and scaling experience, most recently at CouchConf San Francisco.

xix

Acknowledgments

We would like to thank all the people who helped us throughout the writing of both editions of this book: from Couchbase, Owen Hughes, Lior Tagor, Dipti Borkar, Cihan Biyikoglu, and many others who answered technical questions and supported us. From Sela Group, in particular, Dudu Bassa, Caro Sega, Ishai Ram, Roy Rachmany, and all our colleagues who provided support and enthusiasm. We also want to thank Sasha Goldshtein, Yuval Mazor, and Roi Katz for their feedback about the book’s content. Next, we would like to thank the Apress team, as this book wouldn’t have come about without Jonathan Gennick, Jill Balzano, and the rest of the great people at Apress. You guys do tremendous work. A special thanks to Gwenan Spearing, for putting us in touch with Jonathan. Of course, we’d like to thank the other fine folks from Couchbase for making Couchbase Server in the first place, thus giving us an awesome database to write about. And last, but absolutely not least, we would like to thank our families, for putting up with us, for feeding us as we huddled with our laptops in the kitchen, and carefully stepping over power cables whenever we took over the living room.

xxi

Foreword

I have worked in the database world for many years, a forty plus year old, well known technology. I started like most studying relational databases, first at university and then in my professional career. About a decade ago I started hearing murmurs of a new type of database emerging that were in an new category called NoSQL. Like many at the time I was too busy to pay it much attention but over time the noise grew to a point where this new movement demanded my attention. I started to look at this genre of data management technology and in this search stumbled across Couchbase. Information on this exciting database was scarce, particularly to someone new to the field. I came across the first edition of the book you now have in your hands and the path to demystifying Couchbase and NoSQL became very clear. This great work became my guide and handbook for my daily work, for now I was privileged to be managing the technologists in Europe, Middle East and Africa for Couchbase. As luck would have it, one of the book’s authors David joined my team, you cannot imagine how delighted I was. It was at this time I hatched a plan to ask David and Mohammed, also on my team and a well published industry and PhD academic, to write the successor to the original great work. The depth of knowledge these two great personalities had was amazing, not just of Couchbase, or the database space, but of distributed computing in general. David and Mohammed agreed that the time was right for a new edition of Pro Couchbase Server as Couchbase Server 4 was being readied for general availability. This new release is a huge leap forward in the NoSQL data management space and warranted a great book to accompany it. Couchbase Server 4 pushes Couchbase to the lead position technologically in the NoSQL space with a huge amount of new features that expand use cases beyond what NoSQL traditionally catered for and into the realms of traditional relational databases. These new features like N1SQL, Multi-Dimensional Scaling, Global Secondary Indexes and Spatial to name but a few will empower you to build enterprise application for the new digital economy we are all now part of. I hope this book will become your guide just as the original was mine. The authors have devoted many hours of work, research and sacrifice to deliver a body of work that will be appreciated by developers, DBA’s and operations people in the Couchbase world and beyond. They have given up evenings, weekends and holidays to create a book of exceptional detail and quality. They have had inside access to the engineers, architects and product managers who have built Couchbase. Let this book be your definitive guide to managing and developing applications in a distributed NoSQL environment using the best database available. It is a book for people new to Couchbase and seasoned veterans alike. I thank the authors for lighting the way and hope you enjoy the journey that this definitive guide takes you on. —Owen Hughes Solutions Engineering Manger Couchbase EMEA

xxiii