THE NOTION OF DESIGN EMERGENCE EVOLVING YOUR SOFTWARE Who Are we 陈永胜 [email protected] @yeongsheng yeongsheng-tan !2 EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE “THE BEST ARCHITECTURES, REQUIREMENTS, AND DESIGNS EMERGE FROM SELF-ORGANIZING TEAMS.” – PRINCIPLES BEHIND THE AGILE MANIFESTO AGENDA ‣ EMERGENT DESIGN ‣ CHOOSE YOUR POISON ‣ SWITCHING OUT AN ARCHITECTURAL LAYER ‣ WHEN RUBBER HITS THE ROAD ‣ SHOW ME THE CODE ‣ Q&A LISTEN, LEARN, UNDERSTAND, ADAPT EMERGENT DESIGN EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE KNOWLEDGE CREATION & PRESERVATION KNOWLEDGE CREATION KNOWLEDGE PRESERVATION EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE WHEN DO YOU HAVE BEST KNOWLEDGE? KNOWLEDGE VS TIME 100 75 50 Knowledge 25 0 AUG SEP OCT NOV DEC Time EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE HYPOTHETICAL KNOWLEDGE VS TIME 100 75 50 Knowledge 25 0 AUG SEP OCT NOV DEC Time EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE REALITY KNOWLEDGE VS TIME 100 75 50 Knowledge 25 0 AUG SEP OCT NOV DEC Time EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE WHAT WE SHOULD STRIVE FOR KNOWLEDGE VS TIME 100 75 50 Knowledge 25 0 AUG SEP OCT NOV DEC Time EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE TRADITIONAL VS EMERGENT Design Implement the initial design (try to make all decisions) TRADITIONAL EMERGENT Design Reflect and (decide an Design/Code Decide Design/Code initial Direction direction) Reflect and Design/Code Decide Direction EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE MAKE IT WORK #| MAKE IT RIGHT #| MAKE IT FAST THE PRIMITIVE WHOLE ISN'T JUST A BUNCH OF UNASSEMBLED PARTS, IT'S A "UNDER-DEVELOPED WHOLE" AND SOMETHING THAT DOES WORK "BIG DESIGN UP FRONT IS DUMB, BUT DOING NO DESIGN UP FRONT IS EVEN DUMBER" EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE MOTIVATION #- SKILLS #- TIMING EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE WHAT UPFRONT DECISIONS ARE HARDEST TO CHANGE? ▸ Programming Language ▸ Operating System/Processor Architecture ▸ Database CHOICE - NOW OR LATER? CHOOSE YOUR POISON EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE HERE BE DRAGONS… EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE DEFER CHOICE, MAINTAIN FLEXIBILITY Technical Requirement Tech Stack A Tech Stack B Nginx/Apache2 + Phusion HTTP Server Erlang - Cowboy Passenger Erlang - BEAM Request Processing Scala + Akka / Ruby on Rails scheduler/Phoenix/raxx Erlang - OTP/Task/ Long Running Requests Go Agent Server-side State Aerospike/Redis Erlang - OTP/ets Data Persistence Aerospike + MongoDB/MySQL Erlang - Mnesia/dets Cron, Bash scripts, Java Quartz / Background Jobs Erlang - Task/Agent Ruby + Resque/Sidekiq Service crash recovery systemd/upstart Erlang - Supervisor EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE HOMOGENOUS #- SIMPLICITY ▸ Erlang/Elixir gives you options to implement initial solution exclusively homogenous Erlang stack. ▸ Switch to alternatives more full blown solution when enough knowledge has been acquired (e.g. through usage or after data characterisation has been conducted, etc). ▸ Homogeneous stack, single repo systems are easier to develop, test and maintain. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE RESPONSIVE INTERACTIVITY W/O JS ▸ Phoenix.LiveView ▸ https://github.com/yeongsheng-tan/live_view_todos ▸ Branches ▸ master ▸ distributed_phx_pubsub EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE WHAT IS NEEDED? ▸ Production code !- ▸ Phoenix 1.4.9 and above ▸ hex package: phoenix.liveview ▸ Look Ma, No Javascript!!! D.A.S.H + NEWSQL SWITCHING OUT AN ARCHITECTURAL LAYER EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE YOUR WEAKEST LINK ▸ Traditional RDBMS ▸ NoSQL ▸ NewSQL EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE D.A.S.H ▸ Beyond 12-Factor-App ▸ 4 properties of CloudNative Ready DB D : DISPOSABILITY A : API SYMMETRY S : SHARED NOTHING H : HORIZONTAL SCALING EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE ENABLES GLOBAL, SCALABLE, RESILIENT SERVICES ▸ Low-touch and highly automated distributed DB for operators; Simple to reason about for developers. ▸ Industry-leading consistency, even on massively scaled deployments. Distributed transactions without the pain of eventual consistency issues. ▸ Always-on database that accepts reads and writes on all nodes without generating conflicts. ▸ Flexible deployment in any environment, without tying you to any platform or vendor. ▸ Familiar tooling for working with relational data (i.e., SQL). EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE PERILS OF INCONSISTENCIES ▸ Emin Gün Sirer, an Associate Professor at Cornell University, wrote a blog post blaming eventually consistent data stores for the lost bitcoins. He mentions MongoDB, Cassandra and Riak among the NoSQL solutions that are vulnerable to banking thefts because: EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE Architecture CONSENSUS REPLICATION SQL API ▸ Raft • Topics ▸ Commit when quorum has Distributed, transactional KV written data (nodes !> 3) Node 2 ○ Data distribution▸ 1 consensus group per ○ Consensus replicationdata range protocol (Raft) ○ Distributed transactions Gossip / ○ Deployment, repair, rebalance Node 1 Raft Node 3 @cockroachdb ConsensusEVOLVING YOUR Replication SOFTWARE - THE NOTION OF DESIGN EMERGENCE Put “cherry” Leader apricot banana RAFT Follower Follower blueberry apricot apricot cherry CONSENSUS banana banana grape blueberry blueberry grape grape PROTOCOL Consensus Replication @cockroachdb Put “cherry” Leader Put “cherry” apricot banana Follower Follower blueberry apricot apricot cherry banana banana grape blueberry blueberry grape grape @cockroachdb Consensus Replication EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE Put “cherry” Leader Put “cherry” apricot banana Follower Follower blueberry apricot apricot cherry banana banana grape blueberry blueberry cherry grape Write committed when 2 out of 3 nodes have written data grape @cockroachdb ConsensusEVOLVING YOUR Replication SOFTWARE - THE NOTION OF DESIGN EMERGENCE Put “cherry” Leader Put “cherry” apricot banana Follower Follower blueberry apricot apricot cherry banana banana Ack grape Ack blueberry blueberry cherry grape Consensus Replicationgrape @cockroachdb Put “cherry” Leader Put “cherry” apricot banana Follower Follower blueberry apricot apricot cherry Ack banana banana grape blueberry blueberry cherry cherry grape grape @cockroachdb CAN IT BE TRUSTED? WHEN RUBBER HITS THE ROAD EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE COCKROACHDB PASSES JEPSEN TESTING ▸ CockroachDB Beta passes Jepsen testing ▸ Jepsen tests lessons 2+ years on... ▸ Register, Bank, Monotonic & Sets ▸ G2, Sequential, Comments & Nemesis modes (clock skews, network failures) EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE DEMO EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE MONOLITHIC TO MULTI-MASTER DB ▸ Elixir Ecto ▸ https://github.com/yeongsheng-tan/lunchbox_api.git ▸ Branches ▸ master ▸ cockroachdb_postgrex_cdb_api_jwt_auth_e2e_tests EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE WHAT IS NEEDED? ▸ Production code !- hex package: postgrex_cdb ▸ Test code !- hex package: ecto_replay_sandbox ▸ No prod code changes!!! ▸ Test code changes. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE “FROM A PRACTICAL POINT OF VIEW, CHANGEABILITY IS THE ONLY DESIGN METRIC THAT MATTERS; CODE THAT’S EASY TO CHANGE IS WELL-DESIGNED.” – SANDI METZ APPLYING CAP TO BUILD DISTRIBUTED ROBUST PHOENIX API APP EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE CREDITS ▸ Saša Jurić ▸ Nate Stewart ▸ Emin Gün Sirer ▸ Ben Darnell ▸ Diana Hsieh ▸ Aphyr Jepsen.