The S3 Cookbook Get cooking with Amazon’s Simple Storage Service Scott Patten This book is for sale at http://leanpub.com/thes3cookbook This version was published on 2015-01-15 This is a Leanpub book. Leanpub empowers authors and publishers with the Lean Publishing process. Lean Publishing is the act of publishing an in-progress ebook using lightweight tools and many iterations to get reader feedback, pivot until you have the right book and build traction once you do. ©2010 - 2015 Scott Patten Tweet This Book! Please help Scott Patten by spreading the word about this book on Twitter! The suggested hashtag for this book is #thes3cookbook. Find out what other people are saying about the book by clicking on this link to search for this hashtag on Twitter: https://twitter.com/search?q=#thes3cookbook Contents Preface ............................................... 1 Conventions Used in This Book ................................ 1 Using Code Examples ...................................... 1 Getting the Code ........................................ 1 How to Contact Me ....................................... 2 Why Ruby ............................................ 2 Chapter 1. What is S3, and what can I use it for? ....................... 4 Backups ............................................. 4 Serving Data ........................................... 4 Use Cases ............................................ 4 NASDAQ Market Replay .................................... 5 Jason Kester ........................................... 7 Chapter 2. S3’s Architecture ................................... 9 A Quick, Tounge-in-cheek, Overview ............................. 9 Amazon S3 and REST ...................................... 9 Buckets ............................................. 12 S3 Objects ............................................ 15 Access Control Policies ..................................... 16 Logging Object Access ..................................... 16 S3 Recipes ............................................. 17 Signing up for Amazon S3 ................................... 17 Installing Ruby and the AWS/S3 Gem ............................. 19 Setting up the S3SH command line tool ............................ 22 Installing the S3Lib library ................................... 24 Making a request using s3Lib .................................. 26 Getting the response with AWS/S3 ............................... 28 Installing The FireFox S3 Organizer .............................. 30 Working with multiple s3 accounts .............................. 31 Accessing your buckets through virtual hosting ........................ 33 Creating a bucket ........................................ 34 Creating a European bucket .................................. 37 CONTENTS Synchronizing two buckets ................................... 40 Listing All Of Your Buckets ................................... 43 Listing only objects with keys starting with some prefix ................... 44 Downloading a File From S3 .................................. 46 Understanding access control policies ............................. 48 Setting a canned access control policy ............................. 52 Keeping the Current ACL When You Change an Object ................... 54 Making sure that all objects in a bucket are publicly readable . 57 Detecting if a File on S3 is the Same as a Local File ...................... 60 Chapter 4. Authenticating S3 Requests ............................. 62 Authenticating S3 Requests ................................... 62 Writing an S3 Authentication Library ............................. 66 The HTTP Verb ......................................... 69 The Canonicalized Positional Headers ............................. 70 The Canonicalized Amazon Headers .............................. 78 Date Stamping Requests .................................... 83 The Canonicalized Resource .................................. 85 The Full Signature ........................................ 88 Signing the Request ....................................... 93 Making the Request ....................................... 98 Error Handling .........................................101 Preface Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Note This icon signifies a tip, suggestion, or general note. Warning This icon indicates a warning or caution. Using Code Examples All of the code in this book is released under an MIT license, and can be used pretty much anywhere and anyhow you please. Getting the Code All of the code in the book is available on github¹ at http://github.com/spatten/thes3cookbook². You can checkout the code using Git with the following command: ¹http://github.com ²http://github.com/spatten/thes3cookbook Preface 2 1 $> git clone git://github.com/spatten/thes3cookbook.git You can get a .zip or .tar archive of the code by going to http://github.com/spatten/thes3cookbook³ and clicking on the ‘download’ button. How to Contact Me I can be reached via e-mail at [email protected]. Please contact me if you have any questions, comments, kudos or criticism on the book. Constructive criticism is definitely appreciated; I want this book to get better through your feedback. Why Ruby You might be asking yourself why I wrote the examples in this book in Ruby. Here’s a quick set of reasons: It’s concise The last thing you want if you’re reading (or writing!) a book with lots of code in it is lots of repetitive, boilerplate code. Ruby keeps this to a minimum. irb irb is an interactive Ruby shell which you can use to play around with Ruby. In this book, two other programs (s3sh and s3lib) extend irb to allow you to play around with S3 on the command prompt. It’s available Ruby comes pre-installed on most Unixes, including OS X. If you are on Windows, the Ruby One Click Installer will get you up and running quickly. See “Installing Ruby and the AWS/S3 Gem” for instructions on installing Ruby AWS/S3 Marcel Molina’s AWS/S3 library is an elegant interface to Amazon S3. It’s used in most of the examples in the book. RubyGems The RubyGems package library allows you to easily install the additional libraries needed to communicate with S3. Ruby makes me happy Yukihiro Matsumoto, the creator of Ruby, often says that “… Ruby is designed to make programmers happy.” He also says “…I designed Ruby to minimize my surprise.” This works for the me: Ruby is a pleasure to program in and (once you get used to it) easy to read. ³http://github.com/spatten/thes3cookbook Preface 3 This is not to say that Ruby is the best and only language out there. It just happens to be one that is well suited to this book, so I went with it. I’ve written a quick intro to Ruby in Appendix A, A Short Introduction to Ruby. Chapter 1. What is S3, and what can I use it for? The purpose of this section of the book is to show you what S3 can be used for. To that end, I’ve talked to a couple of companies who are doing interesting things with S3. I’ll also talk about some of the more common use cases. Backups The first thing you think of when you hear about S3 is backups. It’s quite a nice solution for this: you can easily backup any type of file and the storage is pretty cheap. Personally, I use it to back up all of my pictures. I used to back them up onto a rewriteable CD, but that was flaky and time consuming. Then I used various solutions such as synchronizing multiple computers on my home network. This is fine for most uses, but pictures of my kids are irreplaceable, so I prefer an online backup. That way, even if my house burns down, the big quake hits Vancouver or all of my computers are stolen, I know that my pictures are safe. Yes, it’s a little paranoid, but that’s what backups are all about! The big win for S3, however, is how easy it is to back up just about anything. There’s no GUI or Web interface to work around - it’s designed for people like you and me: people who can code. You can use it to back up your SVN repositories, your databases or user generated content on your website. Not only that, you can also share your backed up files with others in creative ways. Serving Data The second thing you might want to do with S3 is serve data to your users. This data might be static data for your site (“Using S3 as an asset host”), user generated data (“Serving user generated data from s3”), a Bit Torrent for a large media file (“Seeding a bit torrent”) or a file that only authenticated people can access (“Giving access to a bucket or object with a special URL” and “Giving another user access to an object or bucket using S3SH”). When you are serving data, you will probably want to keep track of what is being viewed and by who. This is discussed in “Determining logging status for a bucket” to “Accessing your logs using S3stat”. Use Cases There are a whole bunch of people out there using S3 in lots of different ways. I wanted to get in touch with some of them to find out how they were using S3 and why they decided to use Chapter 1. What is S3, and what can I use it for? 5 S3. So, I semi-randomly sent out two e-mails to some people whose work I had found interesting. Amazingly enough, both were kind enough to take the time to talk to me. Claude Courbois works for the NASDAQ Stock