Apache Zookeeper in the Wild

Security Level:

Slide title :40-47pt Slide subtitle :26-30pt Color::white Corporate Font : FrutigerNext LT Medium Font to be used by customers and ZooKeeper In The Wild partners : Arial

ApacheCon Budapest 2014

www.huawei.com

Rakesh Radhakrishnan [email protected] HUAWEI TECHNOLOGIES CO., LTD. Top right corner for field-mark, customer or $whoami partner logotypes.

------

• Technical Lead Engineer, Huawei India R & D The following Slide title :32-35pt nine groups of Color: R153 G0 B0 • Apache ZooKeeper committer colors are an Corporate Font : example of how FrutigerNext LT Medium • Apache BookKeeper committer our design colors can be used, Font to be used by customers and please take note partners : • Hadoop user that you should Arial only use one design color [email protected] group per slide. For specific

https://www.linkedin.com/in/rakeshadr usage details, Slide text :20-22pt refer to the Bullets level 2-5: https://twitter.com/rakeshadr “Typesetting 18pt Standard”. Color:Black Corporate Font : FrutigerNext LT Medium Font to be used by customers and partners : Arial

HUAWEI TECHNOLOGIES CO., LTD. 2 Agenda

Content Page Title  Introduction 35-40pt Color: R153 G0 B0  ZooKeeper in Hadoop Corporate Font: FrutigerNext LT Medium Font to be used by customers and  Pitfalls partners: Arial . Deploy ZooKeeper cluster

. Manage ZooKeeper cluster – Maintain & Monitor Content Page Text : . Use ZooKeeper Client 28-30pt Bullets level 2-5 20-30pt Color:Black Corporate Font: FrutigerNext LT Medium Font to be used by customers and partners: Arial Top right corner for field-mark, customer or What is ZooKeeper ? partner logotypes.

------

 A high performance centralized coordination service for distributed applications. The following Slide title :32-35pt nine groups of Color: R153 G0 B0  Simplifies the implementation of many advanced patterns in distributed systems like, colors are an Corporate Font : example of how FrutigerNext LT Medium configuration store, our design colors can be used, Font to be used by customers and distributed lock, please take note partners : that you should Arial queueing, only use one design color group per slide. leader election, For specific

usage details, Slide text :20-22pt coordination and many more.. refer to the Bullets level 2-5: “Typesetting 18pt Standard”. Color:Black Corporate Font : FrutigerNext LT Medium Font to be used by customers and partners : Arial

HUAWEI TECHNOLOGIES CO., LTD. 4 Top right corner for field-mark, customer or ZooKeeper Architecture partner logotypes.

------

The following Slide title :32-35pt nine groups of Color: R153 G0 B0 colors are an Corporate Font : example of how FrutigerNext LT Medium our design colors can be used, Font to be used by customers and please take note partners : that you should Arial only use one design color group per slide. For specific

usage details, Slide text :20-22pt refer to the Bullets level 2-5: “Typesetting 18pt  distributed over a set of machines and replicated. Standard”. Color:Black  all servers store a copy of the data (in memory as well as local file system) Corporate Font : FrutigerNext LT Medium  a LEADER is elected at the startup Font to be used by customers and partners :  LEADER will do atomic broadcast to all other servers (ZooKeeperAtomicBroadcast) Arial  strong ordering guarantees  no partial read/writes

HUAWEI TECHNOLOGIES CO., LTD. 5 Top right corner for field-mark, customer or ZooKeeper Data Model partner logotypes.

------

usage details, Slide text :20-22pt refer to the Bullets level 2-5:  hierarchical namespace “Typesetting 18pt Standard”. Color:Black  each node in the namespace is called as a ‘zNode’ Corporate Font : FrutigerNext LT Medium  every zNode in the name space is identified by a path (for example, /app1). Font to be used by customers and  zNode types – persistent and ephemeral partners : Arial  each zNode has data and optionally can have children  no rename of zNode  add/remove watchers to the zNode

HUAWEI TECHNOLOGIES CO., LTD. 6 Top right corner for field-mark, customer or ZooKeeper Session partner logotypes.

------(1) Connection request The following Slide title :32-35pt Client App nine groups of Color: R153 G0 B0 (2) SyncConnected event ZK colors are an Corporate Font : ZK Client Ensemble example of how FrutigerNext LT Medium our design colors can be used, Font to be used by customers and (3) SaslAuthenticated event please take note partners : that you should Arial only use one (4) send periodic heartbeats design color group per slide. For specific

usage details, Slide text :20-22pt refer to the Bullets level 2-5: “Typesetting 18pt  client establishes a session with a single server Connection Events are: Standard”. Color:Black • SyncConnected Corporate Font :  session has uniqueid and timeout FrutigerNext LT Medium  auto keep-alive heartbeats • SaslAuthenticated Font to be used by customers and partners :  automatic failover • Disconnected Arial  sends request to server in FIFO order • AuthFailed  no operation retries in case of connection loss • Expired

HUAWEI TECHNOLOGIES CO., LTD. 7 Top right corner for field-mark, customer or Writes partner logotypes.

Cli - 2 Cli - 1 ------

The following Slide title :32-35pt (1.1) create znode (2.1) create znode nine groups of “/appnode” (1.6) Color: R153 G0 B0 “/mynode” colors are an send Corporate Font : response example of how FrutigerNext LT Medium Proposal Queue our design colors can be used, Font to be used by customers and R R 2 1 please take note partners : (2.2) (1.2) that you should forward Arial forward only use one Follower req to leader Leader Follower req to leader design color

group per slide. Disk Disk Disk For specific

usage details, Slide text :20-22pt refer to the Bullets level 2-5: (1.3) PROPOSAL (1.3) PROPOSAL “Typesetting 18pt Standard”. (1.4) ACK (1.4) ACK Color:Black (1.5) COMMIT Corporate Font : (1.5) COMMIT FrutigerNext LT Medium Font to be used by customers and partners :  all updates routed through leader Arial  every update has a unique transaction id (zxid)  needs majority consensus  persists the change on disk before sending response to client

HUAWEI TECHNOLOGIES CO., LTD. 8 Top right corner for field-mark, customer or Reads partner logotypes.

------

The following Slide title :32-35pt Cli - 1 Cli - 2 nine groups of Color: R153 G0 B0 colors are an Corporate Font : (1.1) getData (2.1) getData example of how “/appnode” “/appnode” our design colors FrutigerNext LT Medium (2.2) local read (1.2) local read can be used, Font to be used by customers and and send data and send data please take note partners : that you should Arial only use one design color group per slide. Follower Leader Follower For specific

usage details, Slide text :20-22pt refer to the Disk Disk Bullets level 2-5: Disk “Typesetting

18pt Standard”. Color:Black Corporate Font : FrutigerNext LT Medium Font to be used by customers and partners :  read from the connected server memory Arial

HUAWEI TECHNOLOGIES CO., LTD. 9 Top right corner for field-mark, customer or Watches partner logotypes.

------Cli - 1 Cli - 2 The following Slide title :32-35pt 1.1) nine groups of Color: R153 G0 B0 exists(/appnode,watcher) 1.2) not 2.1) create colors are an “/appnode” Corporate Font : exists example of how our design colors FrutigerNext LT Medium 2.4) notify /appnode 2.4) created OK can be used, Font to be used by customers and “NodeCreated” please take note partners : that you should Arial only use one Follower Leader Follower design color 2.2) voting 2.2) voting group per slide. For specific Disk Disk Disk usage details,

Slide text :20-22pt refer to the Bullets level 2-5: “Typesetting 2.3) COMMIT 2.3) COMMIT Standard”. 18pt Color:Black Corporate Font :  one time trigger FrutigerNext LT Medium Font to be used by customers and  will sent notifications asynchronously to the watchers(callback object) partners :  watches will be triggered after committing the changes to disk Arial  two types of watches – data & child watches  client can remove watches

HUAWEI TECHNOLOGIES CO., LTD. 10 Top right corner for field-mark, customer or ZooKeeper APIs partner logotypes.

------ simple APIs Operation Type The following Slide title :32-35pt nine groups of Color: R153 G0 B0  sync & async versions create Write colors are an Corporate Font : example of how getData Read FrutigerNext LT Medium our design colors can be used, Font to be used by customers and setData Write please take note partners : that you should Arial exists Read only use one design color getChildren Read group per slide. For specific

usage details, Slide text :20-22pt multi Read/Write refer to the Bullets level 2-5: “Typesetting 18pt reconfig Write Standard”. Color:Black Corporate Font : setACL Write FrutigerNext LT Medium sync Write Font to be used by customers and partners : setWatches Read Arial removeWatches Read

delete Write

HUAWEI TECHNOLOGIES CO., LTD. 11 Top right corner for field-mark, ZooKeeper in customer or partner logotypes.

------

Leader Follower The following Slide title :32-35pt nine groups of Color: R153 G0 B0 colors are an Corporate Font : example of how FrutigerNext LT Medium Follower our design colors can be used, Font to be used by customers and please take note partners : that you should Arial only use one • Config service •Config service design color •Distributed •Distributed group per slide. For specific transaction Lock usage details, Slide text :20-22pt refer to the Bullets level 2-5: “Typesetting 18pt Standard”. Cli Cli Color:Black Cli Corporate Font : FrutigerNext LT Medium Colocation HBase Font to be used by customers and partners : Arial Cli Cli Cli Cli Cli Cli BKJM HDFS BookKeeper YARN

HUAWEI TECHNOLOGIES CO., LTD. 12 Top right corner for field-mark, How to… customer or partner logotypes.

------

HUAWEI TECHNOLOGIES CO., LTD. 13 Top right corner for field-mark, Pitfalls… customer or partner logotypes.

------

The following Slide title :32-35pt nine groups of Color: R153 G0 B0 Deployment Management Use Client colors are an Corporate Font : example of how FrutigerNext LT Medium our design colors 1) Ideal number of servers 1) Disk full 1) Operation after Connection loss can be used, Font to be used by customers and please take note partners : that you should Arial 2) Design topology 2) JMX MBeans are missing 2) Herd Effect only use one design color 3) Choose right server role 3) How to reconfigure ZK cluster 3) Number of direct children group per slide. For specific

usage details, Slide text :20-22pt 4) Configuration is the key 4) Expecting things to fail - A lot 4) High latency with many writes refer to the Bullets level 2-5: “Typesetting 18pt Standard”. 5) Multi-tenancy at session level Color:Black Corporate Font : FrutigerNext LT Medium 6) Careful about Watchers Font to be used by customers and partners : Arial

HUAWEI TECHNOLOGIES CO., LTD. 14 Top right corner for field-mark, Deployment customer or partner logotypes.

------

The following Slide title :32-35pt nine groups of Color: R153 G0 B0 colors are an Corporate Font : example of how FrutigerNext LT Medium Cases: our design colors can be used, Font to be used by customers and please take note partners : 1) Ideal number of servers that you should Arial only use one design color 2) Design topology group per slide. For specific

3) Choose right server role usage details, Slide text :20-22pt refer to the Bullets level 2-5: “Typesetting 18pt 4) Configuration is the key Standard”. Color:Black Corporate Font : FrutigerNext LT Medium Font to be used by customers and partners : Arial

HUAWEI TECHNOLOGIES CO., LTD. 15 Top right corner for field-mark, Case#1: Ideal number of servers ? customer or partner logotypes.

Scenario ------

The following Slide title :32-35pt nine groups of Highly Available – Color: R153 G0 B0 colors are an tolerated number of failures Corporate Font : example of how

FrutigerNext LT Medium our design colors can be used, Font to be used by customers and please take note partners : that you should Arial only use one design color group per slide. For specific Dead usage details, Slide text :20-22pt refer to the Bullets level 2-5: “Typesetting 18pt Standard”. Color:Black Corporate Font : FrutigerNext LT Medium Solution Font to be used by customers and partners : Quorum = Leader + Followers, Arial (2n+1) nodes can tolerate failure of ‘n’ nodes.

HUAWEI TECHNOLOGIES CO., LTD. 16 Top right corner for field-mark, Case#2: Design topology customer or partner logotypes.

Scenario ------

The following Slide title :32-35pt Hadoop & other processes in a single machine: nine groups of Color: R153 G0 B0 colors are an Corporate Font : • Datanode example of how FrutigerNext LT Medium our design colors • RegionServer can be used, Font to be used by customers and please take note partners : • NodeManager Disk that you should Arial • ZooKeeper only use one design color • Other user application(s) group per slide. For specific

usage details, Slide text :20-22pt refer to the Bullets level 2-5: “Typesetting 18pt Solution Standard”. Color:Black Corporate Font :  Don't colocate ZooKeeper server with I/O intense processes. FrutigerNext LT Medium Font to be used by customers and  Dedicated disk for dataLogDir to reduce disk contention partners : Arial  Datatree resides in memory, should have enough JVM heap size. For example, if you have 4G of RAM, do not set the Java max heap size to 6G or even 4G. Avoid situation in which ZooKeeper swaps to disk.

HUAWEI TECHNOLOGIES CO., LTD. 17 Top right corner for field-mark, Case#3: Choose right server role customer or partner logotypes.

Scenario ------

The following Slide title :32-35pt Adding new Servers will affect – nine groups of Color: R153 G0 B0 colors are an Corporate Font :  write performance example of how FrutigerNext LT Medium our design colors can be used, Font to be used by customers and  the availability please take note partners : that you should Arial only use one design color group per slide. For specific

usage details, Slide text :20-22pt Solution refer to the Bullets level 2-5: “Typesetting Standard”. 18pt ZK Server can be either Participant (a voting member) or Observer (a non-voting member) Color:Black Corporate Font : Voting member: FrutigerNext LT Medium Font to be used by customers and member of an ensemble which will vote on proposals and can become Leader partners : Non-voting member: Arial member of an ensemble which only hear the results of votes

 Choose server role as OBSERVER

HUAWEI TECHNOLOGIES CO., LTD. 18 Top right corner for field-mark, Case#3: Choose right server role customer or partner logotypes. voting members non-voting members ------Follower Follower Observer Leader Observer The following Slide title :32-35pt nine groups of Color: R153 G0 B0 colors are an Corporate Font : example of how ACK FrutigerNext LT Medium our design colors ZK Server as Observer: packet can be used, Font to be used by customers and INFORM please take note partners : packet  leader will inform the changes of zNodes PROPOSAL that you should Arial packet only use one  can easily add by simple changes in configuration design color group per slide. For specific

usage details, Slide text :20-22pt refer to the Bullets level 2-5: “Typesetting 18pt Standard”. Color:Black Benefits: Corporate Font :  big-performance improvement for read-heavy workloads FrutigerNext LT Medium  allows to scale without hurting write performance Font to be used by customers and partners :  failure of Observer server will not affect the availability Arial Example usecase:  As a datacenter bridge - ensemble runs entirely in one datacenter, and the second datacenter runs only Observers. Therefore voting protocol doesn’t takesplace across a high-latency intra-datacenter link, and improves performance.

HUAWEI TECHNOLOGIES CO., LTD. 19 Top right corner for field-mark, Case#4: Configuration is the Key customer or partner logotypes.

Scenario ------

The following Slide title :32-35pt Changing Host configurations like vm.swappiness nine groups of Color: R153 G0 B0 colors are an Corporate Font : example of how FrutigerNext LT Medium our design colors can be used, Font to be used by customers and High value of swappiness will please take note partners : make the kernel page out, can that you should Arial cause ZooKeeper times out only use one design color group per slide. For specific

 It can be set to a value between 0-100; the higher the value, the more aggressive the usage details, Slide text :20-22pt refer to the Bullets level 2-5: kernel is in seeking out inactive memory pages and swapping them to disk. “Typesetting 18pt Standard”. Color:Black  Swapping to disk can cause operations to timeout and potentially fail if the disk is Corporate Font : performing other I/O operations. FrutigerNext LT Medium Font to be used by customers and partners : Solution Arial  High value may cause problems such as lengthy garbage collection pauses  It is important to avoid swapping, which will seriously degrade ZK performance  Recommends to set the value to 0

HUAWEI TECHNOLOGIES CO., LTD. 20 Top right corner for field-mark, Case#4: Configuration is the Key customer or partner logotypes.

Scenario ------

The following Slide title :32-35pt ZooKeeper cluster is unstable and not forming quorum nine groups of Color: R153 G0 B0 colors are an Corporate Font : example of how FrutigerNext LT Medium Ensemble detail is not our design colors can be used, Font to be used by customers and consistent in all the please take note partners : machines that you should Arial only use one design color group per slide. For specific

usage details, Slide text :20-22pt refer to the Bullets level 2-5: “Typesetting 18pt Standard”. Color:Black Corporate Font : FrutigerNext LT Medium Font to be used by customers and Solution partners : Arial  ZooKeeper configurations - Ensemble details should be same in every machine

HUAWEI TECHNOLOGIES CO., LTD. 21 Top right corner for field-mark, Management: Maintain & Monitor customer or partner logotypes.

------

The following Slide title :32-35pt nine groups of Color: R153 G0 B0 colors are an Corporate Font : example of how FrutigerNext LT Medium Cases: our design colors can be used, Font to be used by customers and please take note partners : 1) Disk full that you should Arial only use one 2) JMX MBeans are missing design color group per slide. For specific

3) How to reconfigure ZK cluster usage details, Slide text :20-22pt refer to the Bullets level 2-5: “Typesetting 18pt 4) Expecting things to fail - A lot Standard”. Color:Black Corporate Font : FrutigerNext LT Medium Font to be used by customers and partners : Arial

HUAWEI TECHNOLOGIES CO., LTD. 22 Top right corner for field-mark, Case#1: Disk full customer or partner logotypes.

Scenario ------

The following Slide title :32-35pt Disk nine groups of Color: R153 G0 B0 Due to old snapshots & colors are an

Corporate Font : transaction logs example of how FrutigerNext LT Medium our design colors can be used, Font to be used by customers and please take note partners : that you should Arial only use one design color group per slide. For specific

usage details, Slide text :20-22pt refer to the Bullets level 2-5: “Typesetting 18pt Standard”. Color:Black Corporate Font : Solution FrutigerNext LT Medium Font to be used by customers and partners :  cleanup dataDir & dataLogDir by scheduling purge task Arial

Reference: http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#Ongoing+Data+Directory+Cleanup

HUAWEI TECHNOLOGIES CO., LTD. 23 Top right corner for field-mark, Case#2: JMX MBeans are missing ? customer or partner logotypes.

Scenario ------

LeaderElection MBean is The following Slide title :32-35pt nine groups of unregistered and not available Color: R153 G0 B0 colors are an

Corporate Font : example of how FrutigerNext LT Medium our design colors can be used, Font to be used by customers and  has hierarchy of MBeans please take note partners : that you should Arial only use one  need to enable remote JMX design color group per slide. For specific  MBeans will be registered and de-registered usage details, Slide text :20-22pt dynamically based on the server state. refer to the Bullets level 2-5: “Typesetting 18pt Server states are: Standard”. Color:Black • LOOKING (forming quorum) Corporate Font : • FOLLOWER FrutigerNext LT Medium • LEADER Font to be used by customers and • OBSERVER partners : Arial Solution

 Expect MBeans based on server state.

HUAWEI TECHNOLOGIES CO., LTD. 24 Top right corner for field-mark, Case#3: How to reconfigure ZK cluster ? customer or partner logotypes.

Scenario ------

The following Slide title :32-35pt • machine is slow nine groups of Color: R153 G0 B0 colors are an Corporate Font : • machine is dead example of how FrutigerNext LT Medium our design colors • wants to modify address, port, server role can be used, Font to be used by customers and please take note partners : • add a new server that you should Arial only use one design color group per slide. For specific

Solution usage details, Slide text :20-22pt  dynamic reconfiguration of ZK cluster feature is available since 3.5.0-alpha release. refer to the Bullets level 2-5: “Typesetting 18pt  use reconfig() client APIs or shell admin commands to, Standard”. Color:Black Corporate Font : - add new member FrutigerNext LT Medium - remove an existing member Font to be used by customers and partners : - update address & ports, server roles Arial  updateServerList() client API -a probabilistic client re-balancing algorithm (move the client to a new server)

Reference: https://issues.apache.org/jira/browse/ZOOKEEPER-107

HUAWEI TECHNOLOGIES CO., LTD. 25 Top right corner for field-mark, Case#4: Expecting things to fail – A lot customer or partner logotypes.

------

Remember, The following Slide title :32-35pt nine groups of Color: R153 G0 B0  Hardware - colors are an Corporate Font : power supplies, hard drives, network outages etc. example of how FrutigerNext LT Medium our design colors can be used, Font to be used by customers and please take note partners : that you should Arial  Operating System - only use one Kernel panics, zombie processes, dropped packets etc. design color group per slide. For specific

HUAWEI TECHNOLOGIES CO., LTD. 26 Top right corner for field-mark, Use ZooKeeper Client customer or partner logotypes.

------

The following Slide title :32-35pt nine groups of Color: R153 G0 B0 colors are an Corporate Font : example of how FrutigerNext LT Medium Cases: our design colors can be used, Font to be used by customers and please take note partners : 1) Operation after Connection loss that you should Arial only use one 2) Herd Effect design color group per slide. For specific

3) Number of direct children usage details, Slide text :20-22pt refer to the Bullets level 2-5: 4) High latency with many writes “Typesetting 18pt Standard”. Color:Black 5) Multi-tenancy at session level Corporate Font : FrutigerNext LT Medium 6) Careful about Watchers Font to be used by customers and partners : Arial

HUAWEI TECHNOLOGIES CO., LTD. 27 Top right corner for field-mark, Case#1: Manage operation after connection loss customer or partner logotypes.

Scenario (1) Successfully send create “/app” request ------

The following Slide title :32-35pt nine groups of Color: R153 G0 B0 (2) Disconnected Event ZK colors are an ZK client Corporate Font : Ensemble example of how FrutigerNext LT Medium our design colors (3) SyncConnected Event can be used, Font to be used by customers and please take note partners : that you should (4) Client do retries create “/app” request Arial only use one ConnectionLossException doesn’t design color mean the request is failed at Server group per slide. For specific

usage details, Slide text :20-22pt refer to the Bullets level 2-5: “Typesetting 18pt Standard”. Color:Black Corporate Font : Solution FrutigerNext LT Medium Font to be used by customers and Since the request successfully reaches quorum, quorum will continue with voting and partners : update the changes based on the ACK responses. Arial

Retrying ZooKeeper Client can either read the update before next retries or handle KeeperExceptions properly

HUAWEI TECHNOLOGIES CO., LTD. 28 Top right corner for field-mark, Case#2: Herd Effect customer or partner logotypes.

Scenario too many watches on a  cause large spike on n/w traffic. ------

single zNode…  one such example is distributed lock impl The following Slide title :32-35pt nine groups of Color: R153 G0 B0 colors are an Corporate Font : example of how our design colors FrutigerNext LT Medium Approach-1 /applock Approach-2 /applock can be used, Font to be used by customers and please take note partners : that you should Arial Cli - 1 /lock001 /lock(n-1) /lock n only use one Cli - 1 /lock001 /lock002 /lock n design color group per slide. For specific

watch ‘lock001’ usage details, watch ‘lock001’ watch ‘lock001’ watch ‘lock(n-1)’ Slide text :20-22pt deletion refer to the deletion deletion deletion Bullets level 2-5: “Typesetting Cli - 2 Cli - 2 Cli - n 18pt Cli - n Standard”. Color:Black Figure-1: Watch on “lock” zNode deletion Figure-2: Watch on “predecessor” zNode deletion Corporate Font : FrutigerNext LT Medium Font to be used by customers and Solution partners : Distributed Lock Recipe: Arial step-1: client who created the zNode with least sequence number will always get the lock step-2: others will watch for zNode deletion and not interested on every master lock changes

Approach-2 : has significant benefits of less number of watch notifications.

HUAWEI TECHNOLOGIES CO., LTD. 29 Top right corner for field-mark, Case#3: Number of children on a single znode customer or partner logotypes.

Scenario ------

The following Slide title :32-35pt For example, can't grow the list of children to more nine groups of Color: R153 G0 B0 than 1 MB ‘jute.maxbuffer’ (the sum of colors are an Corporate Font : Assume ‘zNode name’ is less than 100 the names of all of the children) example of how FrutigerNext LT Medium our design colors bytes, you probably want to limit the can be used, Font to be used by customers and please take note partners : maximum number of children to 10,000. that you should Arial only use one design color / group per slide. For specific

Solution usage details, Slide text :20-22pt /ledger refer to the Bullets level 2-5: Recommended hierarchical structure. “Typesetting 18pt Standard”. Color:Black Reference: Apache BookKeeper stores /ledger/01 /ledger/00 Corporate Font : ledger metadata hierarchically. FrutigerNext LT Medium /ledger/00/0000 Font to be used by customers and partners : Arial /ledger/00/0000/L0001

Example, L0000000001 is split into 3 parts (level1/level2/L(level3), which is stored in ledger/00/0000/L0001.

Here BookKeeper can store upto 9999999999 number of ledgers.

HUAWEI TECHNOLOGIES CO., LTD. 30 Top right corner for field-mark, Case#4: High latency with many writes customer or partner logotypes.

Scenario Cli - 1 Cli - n ------

The following Slide title :32-35pt (1.1) create znode (n.1) create nine groups of Color: R153 G0 B0 “/mynode” “/app” data colors are an

Corporate Font : overwhelming Proposal Queue example of how our design colors FrutigerNext LT Medium with writes. R R R R R 5 4 3 2 1 can be used, Font to be used by customers and Bottleneck at (1.2) forward please take note partners : disk fsync () Follower req to leader Leader Follower that you should Arial only use one design color Disk Disk Disk group per slide.

For specific

usage details, Slide text :20-22pt refer to the Bullets level 2-5: network packets “Typesetting 18pt Standard”. Color:Black Corporate Font : high network traffic FrutigerNext LT Medium Font to be used by customers and partners : Solution Arial  meant for 80% read and 20% write,  should be careful on the number of simultaneous write operations

HUAWEI TECHNOLOGIES CO., LTD. 31 Top right corner for field-mark, Case#5: Multi-tenancy at session level customer or partner logotypes.

Scenario ------

 Multiple applications share the same Zookeeper ensemble. The following Slide title :32-35pt nine groups of Color: R153 G0 B0  User application need to make sure that znodes are unique. colors are an Corporate Font : example of how FrutigerNext LT Medium / our design colors can be used, Font to be used by customers and please take note partners : that you should Arial /app only use one design color chroot =/app/a group per slide. For specific create “/etc” /app/a Cli - 1 usage details, Slide text :20-22pt refer to the Bullets level 2-5: “Typesetting 18pt chroot =/app/b Standard”. create “/etc” Color:Black /app/b Cli - 2 Corporate Font : FrutigerNext LT Medium Font to be used by customers and Solution partners :  An optional "chroot" suffix also be appended to the connection string. Arial For example, "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002/app/a"

 client can code his/her application as if it were rooted at "/", while actual location (say /app/a) and all paths would be relative to this

HUAWEI TECHNOLOGIES CO., LTD. 32 Top right corner for field-mark, Case#6: Careful about Watchers customer or partner logotypes.

Scenario ------

The following Slide title :32-35pt nine groups of Color: R153 G0 B0 Sometimes watch notification is missing colors are an Corporate Font : example of how FrutigerNext LT Medium our design colors can be used, Font to be used by customers and please take note partners : that you should Arial only use one design color group per slide. For specific

usage details, Slide text :20-22pt refer to the Bullets level 2-5: “Typesetting 18pt Standard”. Color:Black Corporate Font : Solution FrutigerNext LT Medium Font to be used by customers and  If a znode changes multiple times between getting the event and setting the partners : Arial watch again, it will miss the changes. Carefully handle this.

 Recommends to process the watch notifications in separate application thread

HUAWEI TECHNOLOGIES CO., LTD. 33 Top right corner for field-mark, A few points to remember customer or partner logotypes.

------

The following Slide title :32-35pt nine groups of Color: R153 G0 B0 colors are an Corporate Font : • Highly available – decide tolerated failures example of how FrutigerNext LT Medium Deployment • Choose right server role our design colors can be used, Font to be used by customers and • Don’t colocate with IO intense processes please take note partners : that you should Arial only use one design color group per slide. For specific

• Cleanup datadir & transaction logs periodically usage details, Slide text :20-22pt Manage • Dynamic reconfiguration APIs to change topology refer to the Bullets level 2-5: “Typesetting 18pt Standard”. Color:Black Corporate Font : FrutigerNext LT Medium • Connection loss doesn’t mean operation failed at Server Font to be used by customers and • ZK meant for 80% read and 20% write Use Client partners : • Careful when adding many children under single zNode Arial • Careful about Watchers

HUAWEI TECHNOLOGIES CO., LTD. 34 Top right corner for field-mark, Questions ? customer or partner logotypes.

------

HUAWEI TECHNOLOGIES CO., LTD. 35 Top right corner for field-mark, Who uses ? customer or partner logotypes.

------

Yahoo The following Slide title :32-35pt nine groups of Color: R153 G0 B0 Twitter colors are an Corporate Font : example of how FrutigerNext LT Medium Facebook our design colors can be used, Font to be used by customers and Huawei please take note partners : that you should Arial Netflix only use one design color Rackspace group per slide. For specific

Apache Projects like, Hadooop, BookKeeper, Solr, Kafka etc. usage details, Slide text :20-22pt refer to the Bullets level 2-5: and many more… “Typesetting 18pt Standard”. Color:Black Corporate Font : https://cwiki.apache.org/confluence/display/ZOOKEEPER/PoweredBy FrutigerNext LT Medium Font to be used by customers and partners : Arial

HUAWEI TECHNOLOGIES CO., LTD. 36 Top right corner for field-mark, Community customer or partner logotypes.

------

usage details, Slide text :20-22pt refer to the Bullets level 2-5: “Typesetting 18pt Standard”. Color:Black Corporate Font : FrutigerNext LT Medium Font to be used by customers and http://apache.markmail.org/search/?q=zookeeper partners : Arial

HUAWEI TECHNOLOGIES CO., LTD. 37 Top right corner for field-mark, Client libraries customer or partner logotypes.

------

 Java and C clients available with ZooKeeper release The following Slide title :32-35pt nine groups of Color: R153 G0 B0 colors are an Corporate Font : example of how FrutigerNext LT Medium our design colors can be used, Font to be used by customers and please take note partners : that you should Arial only use one design color group per slide. For specific

HUAWEI TECHNOLOGIES CO., LTD. 38 Top right corner for field-mark, Where are we ? customer or partner logotypes.

------

Released The following Slide title :32-35pt nine groups of Color: R153 G0 B0 colors are an 3.5.0-alpha on 6 August, 2014 Corporate Font :  example of how FrutigerNext LT Medium our design colors can be used, Font to be used by customers and  3.4.6 (stable version) on 10 March, 2014 please take note partners : that you should Arial only use one design color group per slide. For specific

HUAWEI TECHNOLOGIES CO., LTD. 39 Top right corner for field-mark, Future Directions customer or partner logotypes.

------

 Wire encryption - https://issues.apache.org/jira/browse/ZOOKEEPER-2063 The following Slide title :32-35pt nine groups of Color: R153 G0 B0 colors are an Corporate Font :  Stabilize dynamic reconfiguration feature example of how FrutigerNext LT Medium our design colors can be used, Font to be used by customers and  Improve usability, reliability please take note partners : that you should Arial only use one design color group per slide. For specific

HUAWEI TECHNOLOGIES CO., LTD. 40 Top right corner for field-mark, References customer or partner logotypes.

------

 http://zookeeper.apache.org/ The following Slide title :32-35pt nine groups of Color: R153 G0 B0 colors are an Corporate Font :  https://wiki.apache.org/hadoop/ZooKeeper example of how FrutigerNext LT Medium our design colors can be used, Font to be used by customers and  http://hadoop.apache.org/ please take note partners : that you should Arial only use one  http://zookeeper.apache.org/bookkeeper/ design color group per slide. For specific

 http://web.stanford.edu/class/cs347/reading/zab.pdf usage details, Slide text :20-22pt refer to the Bullets level 2-5: “Typesetting 18pt Standard”. Color:Black Corporate Font : FrutigerNext LT Medium Font to be used by customers and partners : Arial

HUAWEI TECHNOLOGIES CO., LTD. 41 Top right corner for field-mark, Photo Attributions customer or partner logotypes.

------https://zookeeper.apache.org/doc/r3.4.6/zookeeperOver.html The following Slide title :32-35pt https://peterskastner.files.wordpress.com/2011/02/it-guy-0013.png nine groups of Color: R153 G0 B0 colors are an Corporate Font : http://www.learnhowtoorap.com/images/how-to-write-pic.jpg example of how FrutigerNext LT Medium our design colors can be used, Font to be used by customers and http://en.wikipedia.org/wiki/Kernel_panic#mediaviewer/File:MacOSX_kernel_panic.png please take note partners : that you should Arial http://www.pd4pic.com/images/tower-computer-server-cpu-processor-box-cartoon.png only use one design color http://png-2.findicons.com/files/icons/1676/primo/128/lock.png group per slide. For specific

https://www.terilogy.com/product/sevone-eng/images/img_feature01.jpg usage details, Slide text :20-22pt refer to the Bullets level 2-5: http://www.inode.gr/img/administration_small.jpg “Typesetting 18pt Standard”. Color:Black http://marketingmotivator.net/wp-content/uploads/2010/09/Handshake_business-man-and- Corporate Font : FrutigerNext LT Medium woman-correct_iStock_000000306702Small.jpg Font to be used by customers and partners : http://hadoop.apache.org/images/hadoop-logo.jpg Arial http://etc-mysitemyway.s3.amazonaws.com/icons/legacy-previews/icons/simple-red-square- icons-alphanumeric/128241-simple-red-square-icon-alphanumeric-question-mark3.png

HUAWEI TECHNOLOGIES CO., LTD. 42 Thank you www.huawei.com

Copyright©2011 Huawei Technologies Co., Ltd. All Rights Reserved. The information in this document may contain predictive statements including, without limitation, statements regarding the future financial and operating results, future product portfolio, new technology, etc. There are a number of factors that could cause actual results and developments to differ materially from those expressed or implied in the predictive statements. Therefore, such information is provided for reference purpose only and constitutes neither an offer nor an acceptance. Huawei may change the information at any time without notice.