Mit hotfixes von Carsten Bormann 2011-03-05 + 2013-02-28

Deploying and Monitoring A practical guide

Mathias Meyer and Jonathan Weiss, 25.05.2009 Danke! Peritor GmbH Mashed up with... 2011-03-05 Deployment/Scaling/ Caching/Debugging

jan krutisch mindmatters gmbh&co. kg [email protected] http://mindmatters.de Deployment Real artists ship! Komponenten client (Browser)

rails

database client (Browser) Heute: rails ➔ thinmongrel ) (∨database ist schnell apache ist schneller client

webserver rails

Dateisystem public/ database rails ist nicht threadsafe client rails

webserver rails database

Dateisystem rails public/ webserver rails webserver rails oldschool: CGI / SCGI / FastCGI newschool: http mongrel versus... mod_proxy_balancer (in Apache//) oder ein richtiger proxy oder ein richtiger balancer Varnish HAProxy, Squid, etc. true school: passenger Apache-Modul client

rails webserver rails database

rails

Dateisystem public/ mod_passenger automagisch. it just works. http://modrails.com Infrastructure Infrastructure

5 Simple Rails Setup

One Rails instance handles all requests Rails is single-threaded: There is only one concurrent request

6 Rails Setup

7 Rails Setup

8 Typical Rails Setup

•! A load-balancer distributes the incoming requests •! Some load-balancers will deliver static requests themselves •! Several Rails instances handle all requests •! Number of concurrent requests equals number of Rails instances

9 Rails Setup Options

10 Deployment Questions

Apache? Pound? mod_proxy_balancer?

FastCGI? Mongrel?

Proxy? Load-balancer?

Nginx? mod_rails? Ebb? HA-Proxy? ? Swiftiply? ?

Thin? Pen? Rails ? Lighttpd? 11 What we are going to cover today

Rails Application Server Proxy/ •! FastCGI •! Apache2 •! Mongrel •! Nginx •! mod_rails / Phussion Passenger •! Lighttpd •! JRuby + Glassfish & Co. •! HA-Proxy

12 FastCGI

13 FastCGI

•! Protocol to communicate with long-running CGI applications •! Usage of either mod_fcgi with Apache 1.3 or mod_fcgi with Lighttpd •! Proxy local and remote FastCGI instances •! Oldest way of deploying Rails •! Deprecated and unstable •! Hard to debug (FastCGI protocol)

14 FastCGI

15 FastCGI

16 17 Mongrel

18 Mongrel

•! Developed by Zed Shaw as an alternative to FastCGI •! Complete HTTP-Server that can load arbitrary Ruby-servlets •! Built-in Rails support

19 Mongrel

20 Mongrel

Apache 2.2 – mod_proxy_balancer Lighttpd Nginx HA-Proxy Pound

21 Mongrel Cluster

Utility to manage several Mongrel instances

22 Mongrel Cluster

Control Mongrels

23 Mongrel and Apache 2.2

Define Mongrel Cluster in Apache

24 Simple Mongrel and Apache 2.2

Redirect all traffic to the Mongel cluster

25 A more complex example

•! Redirect dynamic requests •! Serve static content •! Support cached pages •! Support maintenance page •! Enable client-side caching of images, stylesheets, and JavaScript •! Compress output if supported

26 Mongrel

•! Very robust •! Strict HTTP parser •! Easy to debug (HTTP!) •! Used to be defacto deployment setup with Apache 2.2 and mod_proxy_balancer •! Can be a bit difficult to setup (mongrel_cluster, ports, Apache) •! Not so easy on mass/

27 mod_rails

28 mod_rails a.k.a Phusion Passenger

•! Module for Apache 2.2 (and Nginx) •! Allows Apache to control Rails instances •! Apache starts and stops application instances depending on the application load •! Able to run any Rack-compatible Ruby application (, & Co.) •! Only manages Rails on one host - no remote instances •! Combine with HTTP-Proxy / balancing solution

29 Install Phusion Passenger

Install Apache module

Load and activate in Apache

30 Install Phusion Passenger

Install Passenger with nginx (installs custom nginx)

Configure nginx

31 Customized Phusion Passenger

Control Rails instance number

32 Control Phusion Passenger

Restart after deployment:

33 Phusion Passenger

One machine

34 Phusion Passenger

One machine Multiple machines

35 Phusion Passenger

•! Ready for production •! Makes setup easier – on the single machine level •! Multiple servers still require load balancer •! Suitable for mass-hosting •! upcoming standard way of deploying Rails

36 JRuby

37 •! Ruby Runtime on the •! Implemented in Java and Ruby •! Compiles Ruby into Java-bytecode •! Integrates with Java code and libraries •! Java’s promises of native threads and JIT •! Allows for Ruby/Rails applications to be packaged as WAR files •! WAR files deployable on any J2EE-container: Glassfish, JBoss, Tomcat, , …

38 JRuby on Rails

39 JRuby on Glassfish

One machine

40 JRuby on Glassfish

One machine Multiple machines

41 Setup JRuby on Glassfish

1.! Download JRuby and Glassfish 2.! From http://blog.headius.com/2008/08/zero-to-production-in-15-minutes.

42 Warble Configuration

Define min/max Rails runtimes

43 Glassfish for development

Shrunk-down version of Glassfish v3 - packaged as a gem Can run any Rack-based application

Installation: •! $ -S gem install glassfish •! $ glassfish •! There is no Step 3

44 Rails Setup

45 Proxy Options

46 Proxy Requirements

•! Hide cluster backend from the user •! Load-balancer backend instances •! Recognize down •! Fair scheduler •! (Deliver static content)

47 Apache 2.2

•! Apache 2.2 introduced mod_proxy_balancer •! mod_proxy_balancer can speak to multiple backends and balance requests •! Apache can acts as a pure proxy or can also serve static files

48 Apache 2.2

•! Apache 2.2 introduced mod_proxy_balancer •! mod_proxy_balancer can speak to multiple backends and balance requests •! Apache can acts as a pure proxy or can also serve static files

49 Apache 2.2

•! Apache 2.2 introduced mod_proxy_balancer •! mod_proxy_balancer can speak to multiple backends and balance requests •! Apache can acts as a pure proxy or can also serve static files

50 Apache 2.2

Pro •! Stable, robust, and mature •! Many people know how to work with Apache •! Integrates well with other modules (SVN, DAV, Auth, …)

Con •! Apache can be complicated to configure •! The stock Apache is quite resource-hungry compared to pure proxy solutions

51 Nginx – From Russia with love

•! Nginx - popular Russian webserver with good proxy support •! Can load-balance multiple backends and deliver static content •! Quite popular with Mongrel as the Rails backend •! Recent Passenger support

52 Nginx Configuration

Simple proxy example

Get complete version here:

http://brainspl.at/nginx.conf.txt

53 Nginx

Pro •! Stable, robust, and fast •! Uses fewer resources (CPU and RAM) than Apache for proxy-mode and static files •! Simpler configuration file •! Can directly talk to memcached - SSI

Con •! More documentation would be nice •! No equivalent for many Apache modules

54 Lighttpd

•! Lightweight and fast webserver •! Balancing proxy support •! Good FastCGI support •! Used to be popular – until Mongrel came around

55 Lighttpd Configuration

Simple proxy example

56 Lighttpd

Pro •! Fast and lightweight •! Uses fewer resources (CPU and RAM) than Apache for proxy-mode and static files •! Simpler configuration file

Con •! Unstable for some people •! Slow development cycle •! More documentation would be nice •! Configuration file can be too simple (virtual host aliasing) •! No equivalent for many Apache modules •! (No Passenger support)

57 HA-Proxy

•! HAProxy – reliable, high performance TCP/HTTP load balancer •! Proxying and content inspection •! No content serving, just a proxy •! Mature proxy module (fair scheduler) •! ACL support

See also similar Pound and Pen

58 HAProxy

Simple proxy example

59 HAProxy

Pro •! Mature, stable, robust, and fast •! TCP and HTTP balancing

Con •! Few Rails examples •! Usually not needed in a Rails setup

60 Recommended Setups

61 Small Site

Recommendation Apache 2.2 with mod_rails / Phusion Passenger

62 Medium Site

Recommendation •! Apache/Nginx as the frontend proxy •! Passenger/mod_rails as the backend •! Deliver static files with Apache

63 Large Rails Setup

Recommendation •! Redundant load-balancer •! Redundant proxy / web •! Passenger/mod_rails

64 Heavy Static Files

Recommendation •! Deliver static assets through separate web server farm •! Passenger/mod_rails

65 Java Shop

Recommendation •! Deliver a Rails-WAR file and you are done •! Integrate with existing Java landscape and infrastructure

66 67 Application Server Handler

68 Application Server Handler

69 Application Server Handler

70 Rack

71 Rack Handler

72 Rack

73 Rack Middleware

74 Rack Middleware

•! Authentication (HTTP, OpenID) •! Sessions •! Routing •! Caching •! Logging •! Debugging

75 Support

Frameworks Web Server •! Rails •! CGI •! Merb •! FastCGI •! •! Mongrel •! Sinatra •! Phusion Passenger •! Mack •! JRuby Servlets •! Maveric •! WEBrick •! Halcyon •! Ebb •! Ramaze •! Thin •! Rum •! LiteSpeed •! Vintage •! Swiftcore •! Waves •! Unicorn

76 Remarks

77 Ruby Enterprise Edition

•! Copy-On-Write patches to Ruby 1.8 •! Saves memory when spawning several Rails instances •! Used by Phusion PassengerDead: if available ➔ 1.9.3 ➔ 2.0.0

78 Thin, Ebb, Evented Mongrel & Co.

•! Alternatives to Mongrel •! Claim to be faster, lighter, and what have you •! Rendering “Hello World” is usually not your bottleneck

Stick with stable and robust Mongrel, or better yet, with Passenger Might be2011-03-05 somewhat dated

79 deploy-prozess prozesse (mongrels) müssen neugestartet werden. Bei Passenger: touch tmp/restart.txt > cap deploy:migrations konfiguration in ruby- dateien. Yay! set :application, "name" set :repository, "http://kenn.i.net/svn/trunk/name" role :app, "mog.i.net" role :web, "mog.i.net" role :db, "mog.i.net", :primary => true require 'lib/cap/backgroundrb_recipes' after 'deploy:update_code', :set_symlinks task :set_symlinks do run "ln -s #{shared_path}/config/database.yml #{release_path}/config/ database.yml" run "ln -s #{shared_path}/data #{release_path}/public/data" end AWE13danke! Gruppe/04 AWE13danke! Gruppe/04

Erweitern um eigene "Recipes" Kommandos werden parallel auf allen Servern ausgeführt per SSH Unterstützung diverser SCM-Systeme Erlaubt (relativ) einfache rollbacks, jeder deploy- stand wird abgelegt. Quasi-Standard Nicht auf Rails beschränkt! Infrastructure Deployment

80 Deployment Options

82 Deployment Options

83 Deployment Options

84 Deployment Options

85 What does Capistrano do?

86 Capistrano Deployment Cycle

87 Requirements

88 What doesn’t Capistrano do?

•! Plan your initial server setup •! Configure basic services

89 Basic Ingredients

•! The cap command •! Variables •! Roles •! Tasks •! Namespaces

90 Basic Ingredients - cap

Your one-stop deployment shop

91 Basic Ingredients - Variables

•! Configure basic project information •! Override Capistrano’s default assumptions •! Once set, variables are available globally •! Defined using the set method

92 Basic Ingredients - Roles

•! Define types of servers •! Default roles •! :www •! :app •! :db •! All can point to the same server •! But all three must be defined

•! At least one database server needs to be primary

93 Basic Ingredients - Roles

Define custom roles as you please

Can be reused when defining tasks

94 Basic Ingredients - Tasks

•! Define an atomic set of actions for Capistrano •! Can be called from the command line •! Or other tasks

95 Basic Ingredients - Tasks

To find all the tasks available in your project, use

96 Basic Ingredients - Namespaces

Group tasks together logically

Namespaces and tasks are separated with “:”

97 Get Your Capistrano On

98 Get Your Capistrano On

Capfile, the place to include more recipes

99 Get Your Capistrano On

config/deploy.rb, application specific configuration

100 Capistrano’s Defaults

•! Your SCM is Subversion •! Deployment directory is /u/apps/#{application_name} •! User for SCM and SSH is the currently logged-in user •! Commands are run with sudo

101 Get Your Capistrano On

102 Get Your Capistrano On

•! Capistrano expects a directory structure •! Can be created with cap deploy:setup

103 The Deployment Lifecycle

104 The Deployment Lifecycle

Check the prerequisites:

105 The Deployment Lifecycle

Set up your application for the first time

106 The Deployment Lifecycle

The initial deployment

1.! Checks the revision from the local machine 2.! Checks out the code on the remote machines 3.! Sets a link called current pointing to the lates release 4.! Runs the migrations 5.! Fires up application servers

107 The Deployment Lifecycle

108 The Deployment Lifecycle

Subsequent deployments

1.! Checks the revision from the local machine 2.! Checks out the code on the remote machines 3.! Updates current link 4.! Restarts application servers

109 The Deployment Lifecycle

110 Common Capistrano Tasks

Deploy and run migrations

Run only the migrations

Restart application servers

Rollback to the previous release

111 Have Your Shell, and Eat It Too

112 Invoking any Command

113 Deployment Strategies

114 Deployment Strategies

Direct checkout (from scratch) on the servers

115 Deployment Strategies

Keep a cached copy of the current SCM head

116 Deployment Strategies

Check out locally and transfer

117 Customizing Capistrano

119 Write your own Tasks

120 Write your own Tasks

121 Namespace your Tasks

122 Callbacks

Execute a task before another runs

Execute a task after another has finished

Callbacks are run in the order they’re defined

123 Useful Variables

124 Transactions

125 Transactions

126 Transactions

127 Transactions

128 The Rest

•! Gem dependencies •! Support for deploying through gateway servers •! Parallel execution on multiple servers •! Server setup with deprec gem

129 Vlad the Deployer

•! Simple, -based deployment solution •! Comes with defaults for most popular SCM, app and web servers

131 Vlad the Deployer

Load

Configure

132 Vlad the Deployer

Deployment

133 Vlad the Deployer

•! Simple, sometimes too simple •! No callback mechanism, instead the Rake syntax must be used

•! Very non-verbose output, using --trace recommended

134

•! System Integration Framework •! Written in Ruby – inspired by •! Manage multiple nodes •! Configuration •! Packages •! Custom scripts and commands

135 Architecture

136 Recipes

Install and upgrade packages

137 Recipes

Create and update remote files

138 Recipes

Create and update remote files

139 chef-deploy

Enable

Use in recipe

140 Chef and chef-deploy

•! In heavy development: sometimes unstable and hard to debug •! Be prepared to get your hands dirty •! Better suited for configuration than deployment •! Community recipes at http://github.com/opscode/cookbooks

141 Infrastructure Monitoring

154 The two questions of monitoring

155 1. Is everything still running?

156 2. What are the trends?

157

•! Process-level monitoring •! Checks PID-files, ports, and permissions •! Reacts by executing a script and/or alerting •! Has a web GUI for administering nodes

158 Monit

MySQL

159 Monit

Apache

160 Monit

Mongrel

161 God

•! Ruby-based configuration •! Easily extendible •! Can work with event- and poll-based conditions •! Comes with less recipes than Monit

162 God

Mongrel

163 Munin

•! Host-level monitoring •! Master periodically asks nodes for local data •! Checks system resources and records historical data •! Allows to recognize trends and make predictions •! Alerting support

164 Munin

165 Munin

166 Munin

167 Exception Monitoring

•! Monitor errors raised by your application •! Usually notifying you by email •! Good old exception_notification Plugin •! Exceptions as a Service

168 Hoptoad

169 Exceptional

170 New Relic

•! Rails Performance monitoring •! Live view into your application

171 Other Tools

•! Nagios •! Big Brother •! New Relic RPM •! FiveRuns •! JMX

172 Infrastructure Advanced Deployment

173 Search

174 Search

Full text search

Can become very slow on big data sets

175 Full Text Search Engine

Separate Service •! Creates full text index •! Application queries search daemon •! Index update through application or database

Possible Engines •! Ferret •! Sphinx •! Solr •! Lucene •! …

176 Search Slave

Database replication slave •! Has complete dataset •! Migrates slow search queries from master •! Can use different database table engine

177 Database Index

PostgreSQL Tsearch2 •! Core since 8.3 •! Allows to create full text index on multiple columns or arbitrary SQL expressions

MySQL MyISAM FULLTEXT index •! Only works with MySQL <= 5.0 and MyISAM tables •! Full text index on multiple columns

178 What to use?

Different characteristics •! Real-time updates and stale data •! Lost updates •! Performance •! Document content and format •! Complexity

179 Skalierung does rails scale? what's scaling? Was kann passieren? Server-CPU-Last zu hoch (Bad!) DB-Server-Last zu hoch (Bad!) Zu wenige Connections (Luxusproblem...) Verstopfte Leitungen (eher selten) Volle Festplatten (kill -9 administrator) Festplattendurchsatz! (yeah, right!) viele Probleme, viele Lösungen 3 Grundregeln: 1. Do not optimize without measurement. 2. Do not optimize without measurement. 3. Do not optimize without measurement. Serverlast: Throw Hardware At It Zwei oder mehr App- Server. (Gern auch: VMs) rails client rails rails

rails webserver rails database rails

rails Dateisystem rails public/ rails Share nothing! Dinge, die üblicherweise shared sind: Sessions (nur bei Rails <2) Caches Hochgeladene Dateien Lösungen: Datenbanken Shared Filesystems memcached Folgefehler: Zu wenige DB-Connections, bzw. zu hohe DB-Last rails client rails rails webserver rails rails databasedatabase rails database

rails Dateisystem rails public/ rails Zusätzliche Komplexität: DB-Replikation Back to the lab. 'cause I can fix it in the mix Caching! 1. Page caching class BlogController < ApplicationController caches_page :list

def list Post.find(:all, :order => "created_on desc", :limit => 10) end end 2. Action caching caches_action :list 3. Fragment caching My Blog Posts <% cache do %>

    <% for post in @posts %>
  • <%= link_to post.title, :controller => 'blog', :action => 'show', :id => post %>
    • <% end %>
<% end %> def list unless read_fragment({}) @post = Post.find(:all, !!:order => 'created_on desc', :limit => 10) end end def list unless read_fragment({}) @post = Post.find(:all, !!:order => 'created_on desc', :limit => 10) end Race condition! end 4. Model caching Cache invalidation expire_page(:controller => 'blog', :action => 'list') expire_action(:controller => 'blog', :action => 'list') expire_fragment(:controller => 'blog', :action => 'list', ! :page => 1) class BlogSweeper < ActionController::Caching::Sweeper observe Post def after_create(post) expire_cache_for(post) end # [...after_update...after_destroy...] private def expire_cache_for(record) expire_page(:controller => 'blog', :action => 'list') expire_page(:controller => 'blog', :action => 'show', :id => record.id) end end Caches müssen auch skalieren Cache-store DB-Server-Last Queries reduzieren Datenbank- Optimierungen Know your Database. Und viele, viele, viele, viele weitere kleine Dinge. You have been warned. Scheduling simpel: cron jobs > crontab -e Über ./script/runnerRails 3: kann ein komplettes rake Rails geladen➔ werden. task :foo => :environment ∨ rails runner #!/usr/bin/env ruby ./script/runner

@game_plan = GamePlan.find_by_id($ARGV[0]) if @game_plan.nil? puts "game_plan not found" exit(5) end Achtung: Gute Cronjobs sind auch aufwändig. DJ (delayed_job) plugin class NewsletterJob < Struct.new(:text, :emails) def perform emails.each do |e| NewsletterMailer.deliver_text_to_email(text, e) end end end Delayed::Job.enqueue NewsletterJob.new( 'lorem ipsum...', Customers.find(:all).collect( &:email ) ) Background Processing

180 Problem

Long running tasks •! Resizing uploaded images •! Mailing •! Computing an expensive operation •! Accessing slow back-ends

When running inside request-response-cycle •! Blocks user •! Blocks Rails instance •! Hard to monitor and debug

181 Solution

Asynchronous processing in the background

Message/Queue Scheduler

182 Background Processing

183 Options

Options for message bus: Options for background process: •! Database •! (Ruby) Daemon •! Amazon SQS •! Cron job with script/runner •! Drb •! Forked process •! Memcache •! Delayed Job / BJ / (Backgroundrb) •! ActiveMQ •! run_later •! … •! ….

184 Database/Ruby daemon example

185 Cloud Infrastructure

186 Cloud Infrastructure

Servers come and go •! You do not know your servers before deploying •! Restarting is the same as introducing a new machine

You can’t hardcode IPs database.yml

187 Solution #1

Query and manually adjust •! Servers do not change that often •! New nodes probably need manual intervention •! Use AWS ElasticIPs to ease the pain

Set servers dynamically AWS Elastic IP

188 Solution #2

Use a central directory service •! A central place to manage your running instances •! Instances query the directory and react

189 Solution #2

Use a central directory service •! A central place to manage your running instances •! Instances query the directory and react

190 Central Directory

Different Implementations •! File on S3 •! SimpleDB •! A complete service, capable of monitoring and controlling your instances

191 Q & A

192

Peritor GmbH

Teutonenstraße 16 14129 Berlin Telefon: +49 (0)30 69 20 09 84 0 Telefax: +49 (0)30 69 20 09 84 9

Internet: www.peritor.com E-Mail: [email protected]

193

!Peritor GmbH - Alle Rechte vorbehalten 193