Platform Structure

for scalable SNS platform

날짜

2011.01.11 박혜웅 1 written by BAGE Twitter vs Facebook • Twitter (2010.06) – 2006 ~ 2009 – 190,000,000 users – 750 tweets/second, 3,000 tweets/second peek – 600 TPS/second, 1000 TPS/sec peak – 600,000,000 searches/day

• Facebook (2010.07) – 500,000,000 users – 200,000,000 users on mobile – new 90 contents per month

2 written by BAGE Twitter Architecture

• Front-end • Logging – Ruby on Rails – Scribe • Caching • Monitoring for system & – Varnish (for static) network – Memcached (for dynamic) – Munin & Nagios – CacheMoney • Statistics • Message Queue – Google Anlyics(?) – Kestrel • Storage – Hadoop, Cassandra(?) • Searching – Lucene(?), Hbase • Analysis – Pig on Hadoop • Graphing – FlockDB

3 Twitter Architecture

invalidated each time a tweet was coming

read-through page per user (50% hit rate) read-through cache from cache (95% hit rate) write-through real data of tweet and user (95% hit rate) write-through an array of tweet IDs (99% hit rate)

from MySQL to Cassandra

4 written by BAGE Front-end • Ruby on Rails – Ruby: script language – Rails: web framework • Mongrel – web server

5 written by BAGE Caching • from MySQL to Memcached, CacheMoney, Varnish • make user’s timeline using Fanout. • Memcached – “There are only two hard things in CS: cache invalidation and naming things.” Phil Karlton, via Tim Bray • CacheMoney – write-through caching – developed by twitter • Varnish – for web pages cache. (Reverse Proxy) – uses virtual memory (mapping memory to file) – better than Squid

6 written by BAGE Message Queue • from DRB, Rinda, Erlang, to Kestrel • Kestrel – Simplest Message Queue – based on Starling • Starling uses eventmachine in a single-thread single-process from. • Kestrel uses eventmachine in a multi I/O thread and a pool of worker thread. – written in Scala – developed by twitter – with Apache Mina • a network application framework • Features of Kestrel – fast • queues are stored in memory. • runs on the JVM, good garbage collector. – small • only 2K lines os Scala. – durable • logged into a journal on disk. – reliable • if fail, item is automatically handed to another client.

7 written by BAGE Message Queue • Anti-Features of Kestrel – strongly ordered • each queue is ordered on each machine. • a queue cluster will apear “loosely ordered”. – transactional • item ownership is transferred with acknowledgement. • Use messages to invalidate cache. • Simplest Message Queue – Gives up constraints for scalability. – No strict ordering of jobs. – No shared state among servers. – Uses memcached protocol.

8 written by BAGE Searching • from Memcached to Lucene... – low hit rates problem. – term_id, doc_id scheme + partioning by time

9 written by BAGE Storage • from MySQL to Cassandra – for Huge Data and free Licence • only for Backup • Table Scheme for partioning and replicatioin – Commutative (가환) • 연산의 순서가 바뀌어도 됨, sum(x,y) – Idempotent (멱등) • 같은 연산을 여러 번 수행해도 결과가 같음. ex: max(x,y) – Denormalization – make duplicated table with different primary key. • Multi functional masters. • Database Table Partioning – partioning by time for tweets. – partioning by user for timeline. – “The Next Application I Build is Going to Be Easily Partitionable” - S. Butterfield • Index Everything – Do not use “like %keyword%” • Lightweight Locking

10 written by BAGE Storage • Use explain to how your queries are running. • Avoid complex joins. • Avoid scanning large sets of data

11 written by BAGE Lessons Learned • Talk to the community. • Treat your scaling plan like a business plan. • Build it yourself. – Don’t rely on memcache and database. • Build in user limits. • Don’t make the databse the central bottleneck of doom. – Not everything needs to require a gigantic join. • Make your application easily partitionable from the start. • Realize your site is slow. – Instrument everything. – Start graphing early • Prepare a full test suite – Test everything. • Don’t make services dependent. • Process asynchronously when possible • Cache as much as possible – Databases not always the best store. • Start working on scaling early.

12 written by BAGE Lessons Learned • Long running processes should be abstracted to daemons. • Most performance comes from application design. – not from the language. • Turn your website into an open service by creating an API.

13 written by BAGE