THE NOTION of DESIGN EMERGENCE EVOLVING YOUR SOFTWARE Who Are We

THE NOTION OF DESIGN EMERGENCE EVOLVING YOUR SOFTWARE Who Are we

陈永胜 [email protected] @yeongsheng yeongsheng-tan

2 EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE

“THE BEST ARCHITECTURES, REQUIREMENTS, AND DESIGNS EMERGE FROM SELF-ORGANIZING TEAMS.” – PRINCIPLES BEHIND THE AGILE MANIFESTO AGENDA ‣ EMERGENT DESIGN ‣ CHOOSE YOUR POISON ‣ SWITCHING OUT AN ARCHITECTURAL LAYER

‣ WHEN RUBBER HITS THE ROAD ‣ SHOW ME THE CODE ‣ Q&A LISTEN, LEARN, UNDERSTAND, ADAPT EMERGENT DESIGN EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE KNOWLEDGE CREATION & PRESERVATION

KNOWLEDGE CREATION

KNOWLEDGE PRESERVATION EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE WHEN DO YOU HAVE BEST KNOWLEDGE? KNOWLEDGE VS TIME 100

Knowledge 25

0 AUG SEP OCT NOV DEC Time EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE HYPOTHETICAL KNOWLEDGE VS TIME 100

Knowledge 25

0 AUG SEP OCT NOV DEC Time EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE REALITY KNOWLEDGE VS TIME 100

Knowledge 25

0 AUG SEP OCT NOV DEC Time EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE WHAT WE SHOULD STRIVE FOR KNOWLEDGE VS TIME 100

Knowledge 25

0 AUG SEP OCT NOV DEC Time EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE TRADITIONAL VS EMERGENT

Design Implement the initial design (try to make all decisions)

TRADITIONAL

EMERGENT Design Reﬂect and (decide an Design/Code Decide Design/Code initial Direction direction) Reﬂect and Design/Code Decide Direction EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE MAKE IT WORK #| MAKE IT RIGHT #| MAKE IT FAST THE PRIMITIVE WHOLE ISN'T JUST A BUNCH OF UNASSEMBLED PARTS, IT'S A "UNDER-DEVELOPED WHOLE" AND SOMETHING THAT DOES WORK

"BIG DESIGN UP FRONT IS DUMB, BUT DOING NO DESIGN UP FRONT IS EVEN DUMBER" EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE MOTIVATION #- SKILLS #- TIMING EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE WHAT UPFRONT DECISIONS ARE HARDEST TO CHANGE? ▸ Programming Language

▸ Operating System/Processor Architecture

▸ Database CHOICE - NOW OR LATER? CHOOSE YOUR POISON EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE HERE BE DRAGONS… EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE DEFER CHOICE, MAINTAIN FLEXIBILITY Technical Requirement Tech Stack A Tech Stack B

Nginx/Apache2 + Phusion HTTP Server Erlang - Cowboy Passenger Erlang - BEAM Request Processing Scala + Akka / Ruby on Rails scheduler/Phoenix/raxx Erlang - OTP/Task/ Long Running Requests Go Agent

Server-side State Aerospike/Redis Erlang - OTP/ets

Data Persistence Aerospike + MongoDB/MySQL Erlang - Mnesia/dets

Cron, Bash scripts, Java Quartz / Background Jobs Erlang - Task/Agent Ruby + Resque/Sidekiq

Service crash recovery systemd/upstart Erlang - Supervisor EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE HOMOGENOUS #- SIMPLICITY ▸ Erlang/Elixir gives you options to implement initial solution exclusively homogenous Erlang stack. ▸ Switch to alternatives more full blown solution when enough knowledge has been acquired (e.g. through usage or after data characterisation has been conducted, etc). ▸ Homogeneous stack, single repo systems are easier to develop, test and maintain. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE RESPONSIVE INTERACTIVITY W/O JS ▸ Phoenix.LiveView

▸ https://github.com/yeongsheng-tan/live_view_todos

▸ Branches

▸ master

▸ distributed_phx_pubsub EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE WHAT IS NEEDED?

▸ Production code !-

▸ Phoenix 1.4.9 and above

▸ hex package: phoenix.liveview

▸ Look Ma, No Javascript!!! D.A.S.H + NEWSQL SWITCHING OUT AN ARCHITECTURAL LAYER EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE YOUR WEAKEST LINK

▸ Traditional RDBMS

▸ NoSQL

▸ NewSQL EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE D.A.S.H ▸ Beyond 12-Factor-App

▸ 4 properties of CloudNative Ready DB

D : DISPOSABILITY A : API SYMMETRY S : SHARED NOTHING H : HORIZONTAL SCALING EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE ENABLES GLOBAL, SCALABLE, RESILIENT SERVICES

▸ Low-touch and highly automated distributed DB for operators; Simple to reason about for developers.

▸ Industry-leading consistency, even on massively scaled deployments. Distributed transactions without the pain of eventual consistency issues.

▸ Always-on database that accepts reads and writes on all nodes without generating conﬂicts.

▸ Flexible deployment in any environment, without tying you to any platform or vendor.

▸ Familiar tooling for working with relational data (i.e., SQL). EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE PERILS OF INCONSISTENCIES

▸ Emin Gün Sirer, an Associate Professor at Cornell University, wrote a blog post blaming eventually consistent data stores for the lost bitcoins. He mentions MongoDB, Cassandra and Riak among the NoSQL solutions that are vulnerable to banking thefts because: EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE

Architecture CONSENSUS REPLICATION SQL API ▸ Raft • Topics ▸ Commit when quorum has Distributed, transactional KV written data (nodes !> 3) Node 2 ○ Data distribution▸ 1 consensus group per ○ Consensus replicationdata range protocol (Raft)

○ Distributed transactions Gossip / ○ Deployment, repair, rebalance Node 1 Raft Node 3

@cockroachdb ConsensusEVOLVING YOUR Replication SOFTWARE - THE NOTION OF DESIGN EMERGENCE

Put “cherry” Leader

apricot banana RAFT Follower Follower blueberry apricot apricot cherry CONSENSUS banana banana grape blueberry blueberry

grape grape PROTOCOL Consensus Replication

@cockroachdb Put “cherry” Leader Put “cherry” apricot

banana Follower Follower blueberry apricot apricot cherry banana banana grape blueberry blueberry

grape grape

@cockroachdb Consensus Replication EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE

Put “cherry” Leader Put “cherry” apricot

banana Follower Follower blueberry apricot apricot cherry banana banana grape blueberry blueberry

cherry grape Write committed when 2 out of 3 nodes have written data grape

@cockroachdb ConsensusEVOLVING YOUR Replication SOFTWARE - THE NOTION OF DESIGN EMERGENCE

Put “cherry” Leader Put “cherry” apricot

banana Follower Follower blueberry apricot apricot cherry banana banana Ack grape Ack blueberry blueberry

cherry grape

Consensus Replicationgrape

@cockroachdb Put “cherry” Leader Put “cherry” apricot

banana Follower Follower blueberry apricot apricot cherry Ack banana banana grape blueberry blueberry

cherry cherry

grape grape

@cockroachdb CAN IT BE TRUSTED? WHEN RUBBER HITS THE ROAD EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE

COCKROACHDB PASSES JEPSEN TESTING ▸ CockroachDB Beta passes Jepsen testing

▸ Jepsen tests lessons 2+ years on...

▸ Register, Bank, Monotonic & Sets

▸ G2, Sequential, Comments & Nemesis modes (clock skews, network failures) EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE DEMO EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE MONOLITHIC TO MULTI-MASTER DB ▸ Elixir Ecto

▸ https://github.com/yeongsheng-tan/lunchbox_api.git

▸ Branches

▸ master

▸ cockroachdb_postgrex_cdb_api_jwt_auth_e2e_tests EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE WHAT IS NEEDED?

▸ Production code !- hex package: postgrex_cdb

▸ Test code !- hex package: ecto_replay_sandbox

▸ No prod code changes!!!

▸ Test code changes. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE

“FROM A PRACTICAL POINT OF VIEW, CHANGEABILITY IS THE ONLY DESIGN METRIC THAT MATTERS; CODE THAT’S EASY TO CHANGE IS WELL-DESIGNED.” – SANDI METZ APPLYING CAP TO BUILD DISTRIBUTED ROBUST PHOENIX API APP EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE

CREDITS ▸ Saša Jurić

▸ Nate Stewart ▸ Emin Gün Sirer ▸ Ben Darnell

▸ Diana Hsieh ▸ Aphyr Jepsen