Software Architecture and Design Patterns

This full-semester course introduces important architectural patterns for making software easier to build, test and maintain. Architecture: carefully designed structure of something. Required textbooks: None. Recommended textbooks:

• Harry Percival and Bob Gregory. Architecture Patterns with Python. O’Reilly Media; 1st edition (March 31, 2020).

A signiﬁcant portion of the lecture notes is built on the above two books.

Recommended reference books:

• Eric Evans. Domain-Driven Design: Tackling Complexity in the Heart of Software. Addison Wesley, 2003.

• Martin Fowler. Patterns of Enterprise Application Architecture. Addison Wesley, 2002.

• Python 3.9.2 interpreter: https://www.python.org/downloads/release/python-392 [Scroll to the bottom to ﬁnd an installer for your operating

• Wing IDE 101: http://wingware.com/downloads/wing-101

• Flask: https://pypi.org/project/Flask

• SQLAlchemy: https://pypi.org/project/SQLAlchemy

Big ball of mud. Big ball of yarn. Figure 1. A real-life dependency diagram Both refer to a software that lacks a perceivable architecture. Caution: with a circular layout, how messy the graph looks depends on how the nodes are arranged. So you cannot just say the software lacks architecture if it looks like a big ball of mud with the circle layout.

Analogy: the garden runs wild without maintenance. Chaotic software application – everything is coupled to everything else and changing any part is dangerous. A common problem: business logic spread over all places. A common practice: build a database schema ﬁrst. Object

MANAGE COMPLEXITY. Structure the application to facilitate testing (unit, integration, end-to-end). It needs eﬀorts. Layered design/archictecture (presentation layer – business logic – database layer). See Figure 5-2 Typical Application Layers. Do not cross layers. Like our government structure? Levels of abstractions. (Note: this abstraction is diﬀerent

Diﬀerent names: Hexagonal Architecture, Ports-and- Adapters, Onion Architecture, Clean Architecture.

Figure 5-7 Onion view of clean architecture.

Figure 3. Onion architecture in Chapter 2 of the textbook.

The OAPS Class Diagram from LiuSiyuan et al.

• Complexity.

• Big project size.

• Evolving requirements.

• Regression.

• High coupling.

• Slow speed while testing the UI layer or the real data layer.

• Requiring a test database.

Encapsulating behavior using abstractions. Think on the level of behavior, not on the level of data or algorithm. BDD: behavior should come ﬁrst and drive our storage requirements.. How to understand abstraction? Public API. Interface. Function. Method. Responsibility: search sausages from duckduckgo.com. A low-level abstraction

Copyright ©2021 Hui Lan 12 import j s o n from urllib .request import u r l o p e n from urllib .parse import u r l e n c o d e params = d i c t (q=’Sausages ’ , format=’ j s o n ’ ) handle = urlopen(’http://api.duckduckgo.com’ + ’?’ + urlencode(params)) r a w text = handle.read().decode(’utf8’) parsed = json.loads(raw t e x t ) results = parsed[’RelatedTopics’] f o r r in r e s u l t s : i f ’ Text ’ in r : p r i n t (r[’FirstURL’] + ’ = ’ + r[’Text’])

A medium-level abstraction import r e q u e s t s

Copyright ©2021 Hui Lan 13 params = d i c t (q=’Sausages ’ , format=’ j s o n ’ ) parsed = requests.get(’http://api.duckduckgo.com/’, params=params).json() results = parsed[’RelatedTopics’] f o r r in r e s u l t s : i f ’ Text ’ in r : p r i n t (r[’FirstURL’] + ’ = ’ + r[’Text’])

A high-level abstraction import duckduckpy f o r r in duckduckpy.query( ’Sausages ’). related t o p i c s : p r i n t ( r . f i r s t u r l , ’ = ’ , r . t e x t )

Which one is easier to understand?

When do we say one depends on the other?

Depends: uses, needs, knows, calls, imports.

Dependency graph/network.

Figure 2. Layered architecture

Things in Presentation Layer should not use things in Database Layer directly.

For business logic. Seperate business logic from infrastructure and from user-interface logic. Domain model: a mental map that business owners have of their business, expressed in domain-speciﬁc terminologies. Describe the business core (most valuable and likely to change). Domain model: another name for business logic layer. Why? Keep the model decoupled from technical concerns. Figure 1. A component diagram for our app at the end of

• Value Object: uniquely identiﬁed by the data it holds (not by a long-lived ID). @dataclass(frozen=True) c l a s s OrderLine : orderid: OrderReference sku: ProductReference qty: Quantity

• Entity: uniquely identiﬁed by a unique ID.

• Aggregate.

What is domain? The problem (we are going to solve).

Example: an online furniture retailer (purchasing, product design, logistics, delivery, etc).

Business jargons (domain language): source, goods, SKU (stock-keeping unit), supplier, stock, out of stock, warehouse, inventory, lead time, ETA, batch, order, order reference, order line, quantity, allocate, deliver, ship, shipment tracking, dispatch, etc. Check MADE.com.

What is model? A map of a process that captures useful

Data model: database, database schema.

Decouple it (the model) from technical concerns (or infrastructure concerns).

Related topic: DDD.

Tips:

1. Ask the domain expert for glossary and concrete examples.

2. Start with a minimal version of the domain model.

4. Add new classes instead of changing existing ones. Reason: won’t aﬀect existing code.

5. Do not share a database between two applications.

** Talk over Some Notes on Allocation.

Task: allocate order lines to batches.

Our domain model expressed in Python: from dataclasses import d a t a c l a s s from t y p i n g import O p t i o n a l

@dataclass(frozen=True) #(1) (2) c l a s s OrderLine : o r d e r i d : s t r sku : s t r qty : i n t c l a s s Batch : def i n i t ( s e l f , r e f : str , sku : str , qty : int , eta: Optional[date]): #(2) self.reference = ref self.sku = sku self.eta = eta self.available quantity = qty

def allocate(self , line: OrderLine):

def c a n allocate(self , line: OrderLine) => bool : return self.sku == line.sku and self.available q u a n t i t y >= l i n e . qty

Our model in a graphic form:

Real-world business has lots of edge cases:

• Deliver on a speciﬁc date.

The EnglishPal story: the web UI was built after the domain model had been built. In fact, I never thought I would build a web application for EnglishPal.

Figure 2. Context diagram for the allocation service

- A global supply chain.

- Freight partners.

- Manufacturers.

- Minimum storage. Treat goods on the road as our stock.

Use unit test to describe the behavior of the system.

Think about how to use it before implementing it.

The test describes the behavior of the system from an interface point of view. def t e s t a l l o c a t i n g t o a b a t c h r e d u c e s t h e a v a i l a b l e quantity ():

batch = Batch(”batch =001” , ”SMALL=TABLE”, qty=20, eta=date.today()) line = OrderLine(”order =r e f ” , ”SMALL=TABLE” , 2)

batch.allocate(line)

### More tests ### def m a k e b a t c h a n d line(sku, batch qty , l i n e q t y ) : return ( Batch(”batch =001”, sku, batch qty , eta=date.today()), OrderLine(”order =123”, sku, line q t y ) , ) def t e s t c a n a l l o c a t e i f a v a i l a b l e g r e a t e r t h a n required (): l a r g e batch , small l i n e = m a k e b a t c h a n d l i n e ( ”ELEGANT=LAMP” , 20 , 2) assert large b a t c h . c a n allocate(small l i n e ) def t e s t c a n n o t a l l o c a t e i f a v a i l a b l e s m a l l e r t h a n required (): s m a l l batch, large l i n e = m a k e b a t c h a n d l i n e ( ”ELEGANT=LAMP” , 2 , 20) assert small b a t c h . c a n allocate(large l i n e ) i s F a l s e

Copyright ©2021 Hui Lan 26 def t e s t c a n a l l o c a t e i f a v a i l a b l e e q u a l t o required (): batch, line = make b a t c h a n d l i n e ( ”ELEGANT=LAMP” , 2 , 2) assert batch.can allocate(line) def t e s t c a n n o t a l l o c a t e i f s k u s d o n o t m a t c h ( ) : batch = Batch(”batch =001” , ”UNCOMFORTABLE=CHAIR”, 100, eta=None) d i f f e r e n t s k u line = OrderLine(”order =123” , ”EXPENSIVE=TOASTER” , 10) assert batch.can allocate(different s k u l i n e ) i s F a l s e

• allocate

• can allocate

• deallocate

Copyright ©2021 Hui Lan 27 def t e s t c a n o n l y d e a l l o c a t e a l l o c a t e d l i n e s ( ) : batch, unallocated l i n e = m a k e b a t c h a n d l i n e ( ”DECORATIVE=TRINKET” , 20 , 2) batch.deallocate(unallocated l i n e ) assert batch.available quantity == 20

Updated model: c l a s s Batch : def i n i t ( s e l f , r e f : str , sku : str , qty : int , eta: Optional[date]): self.reference = ref self.sku = sku self.eta = eta s e l f . p u r c h a s e d quantity = qty s e l f . allocations = set () # type: Set[OrderLine]

def allocate(self , line: OrderLine): i f s e l f . c a n allocate(line ): s e l f . allocations.add(line)

@property def a l l o c a t e d quantity(self) => i n t : return sum( l i n e . qty f o r l i n e in s e l f . allocations)

@property def a v a i l a b l e quantity(self) => i n t : return s e l f . p u r c h a s e d q u a n t i t y = self.allocated q u a n t i t y

def c a n allocate(self , line: OrderLine) => bool : return self.sku == line.sku and self.available q u a n t i t y >= l i n e . qty

Can the above model pass the following test?

Copyright ©2021 Hui Lan 29 def t e s t a l l o c a t i o n i s idempotent (): batch, line = make b a t c h a n d l i n e ( ”ANGULAR=DESK”, 20, 2) batch.allocate(line) batch.allocate(line) assert batch.available quantity == 18

Easy to test as no database is involved.

For something that has data but no ID. For example, order line, name, and money.

Value Object pattern. Value objects have value equality.

Orer line: @dataclass(frozen=True) c l a s s OrderLine : orderid: OrderReference sku: ProductReference qty: Quantity

@dataclass(frozen=True) c l a s s Name : f i r s t n a m e : s t r surname : s t r c l a s s Money(NamedTuple ): c u r r e n c y : s t r v a l u e : i n t

Line = namedtuple(’Line’, [’sku’, ’qty’]) def t e s t equality (): assert Money(’gbp’, 10) == Money(’gbp’, 10)

Copyright ©2021 Hui Lan 32 assert Name(’Harry’, ’Percival ’) != Name(’Bob’, ’Gregory’) assert Line(’RED=CHAIR’, 5) == Line(’RED=CHAIR ’ , 5) def t e s t n a m e equality (): assert Name(”Harry”, ”Percival”) != Name(”Barry”, ”Percival”)

Math for value objects: fiver = Money(’gbp’, 5) tenner = Money(’gbp’, 10) def c a n a d d m o n e y v a l u e s f o r t h e s a m e currency (): assert fiver + fiver == tenner def c a n s u b t r a c t m o n e y v a l u e s ( ) : assert tenner = fiver == fiver def a d d i n g d i f f e r e n t c u r r e n c i e s f a i l s ( ) :

Copyright ©2021 Hui Lan 33 with pytest.raises(ValueError): Money(’usd’, 10) + Money(’gbp’, 10) def c a n m u l t i p l y m o n e y b y a n u m b e r ( ) : assert fiver * 5 == Money(’gbp’, 25) def multiplying t w o m o n e y v a l u e s i s a n e r r o r ( ) : with pytest.raises(TypeError): t e n n e r * f i v e r

For something with a permanent ID (that is identiﬁed by a reference). For example, a person (who has a permanent identity). c l a s s Person :

def i n i t (self , name: Name): self .name = name def t e s t b a r r y i s h a r r y ( ) : harry = Person(Name(”Harry”, ”Percival”)) barry = harry

assert harry i s b a r r y and b a r r y i s h a r r y

Entities have identity equality. Unlike value objects, an entitie’s attributes can change without making it a diﬀerent entity (as long as its identity keeps the same).

Deﬁne equality operator ( eq ) on entities: c l a s s Batch : ...

def e q (self , other): if not isinstance (other , Batch): return F a l s e

def h a s h ( s e l f ) : return hash (self.reference)

It is just a function.

allocate(): A service that allocates an order line, given a list of batches.

Testing script: def t e s t p r e f e r s c u r r e n t s t o c k b a t c h e s t o shipments (): i n s t o c k batch = Batch(”in =stock =batch ” , ”RETRO=CLOCK”, 100, eta=None) s h i p me n t batch = Batch(”shipment =batch ” , ”RETRO=CLOCK”, 100, eta=tomorrow) line = OrderLine(”oref”, ”RETRO=CLOCK” , 10)

allocate(line , [in s t o c k batch , shipment b a t c h ] )

Copyright ©2021 Hui Lan 38 a s s e r t i n s t o c k batch.available quantity == 90 assert shipment batch.available quantity == 100 def t e s t p r e f e r s e a r l i e r b a t c h e s ( ) : earliest = Batch(”speedy =batch” , ”MINIMALIST=SPOON”, 100, eta=today) medium = Batch(”normal=batch” , ”MINIMALIST=SPOON”, 100, eta=tomorrow) latest = Batch(”slow =batch” , ”MINIMALIST=SPOON”, 100, eta=later) line = OrderLine(”order1”, ”MINIMALIST=SPOON” , 10)

allocate(line , [medium, earliest , latest])

assert earliest.available quantity == 90 assert medium.available quantity == 100 assert latest.available quantity == 100

Copyright ©2021 Hui Lan 39 def t e s t r e t u r n s a l l o c a t e d b a t c h r e f ( ) : i n s t o c k batch = Batch(”in =stock =batch=r e f ” , ”HIGHBROW=POSTER”, 100, eta=None) s h i p me n t batch = Batch(”shipment =batch=r e f ” , ”HIGHBROW=POSTER”, 100, eta=tomorrow) line = OrderLine(”oref”, ”HIGHBROW=POSTER” , 10) allocation = allocate(line , [in s t o c k batch , shipment b a t c h ] ) assert allocation == in s t o c k batch.reference

Model: def allocate(line: OrderLine , batches: List[Batch]) => s t r : batch = next ( b f o r b in sorted ( b a t c h e s ) i f b . c a n allocate(line)) batch.allocate(line) return batch. reference

For the sorted() method to work, we need to deﬁne an order function gt : c l a s s Batch :

def g t (self , other): i f s e l f . eta i s None : return F a l s e i f o t h e r . eta i s None : return True return s e l f . eta > o t h e r . eta

Tips:

• Instead of FooManager, use manage foo().

• Instead of BarBuilder, use build bar().

• Instead of BazFactory, use get baz().

A domain concept: out of stock. def t e s t r a i s e s o u t o f s t o c k e x c e p t i o n i f c a n n o t allocate (): batch = Batch(”batch1”, ”SMALL=FORK” , 10 , eta=today ) allocate(OrderLine(”order1”, ”SMALL=FORK” , 10) , [ batch ] )

with pytest . raises(OutOfStock , match=”SMALL=FORK” ) : allocate(OrderLine(”order2”, ”SMALL=FORK” , 1) , [ batch ] )

Deﬁne the exception class: c l a s s OutOfStock(Exception ): pass

Copyright ©2021 Hui Lan 42 def allocate(line: OrderLine , batches: List[Batch]) => s t r : t r y : batch = next ( ... except StopIteration: r a i s e OutOfStock(f”Out of stock for sku { l i n e . sku}”)

Figure 4. Our domain model at the end of the chapter

What do we lack? A database!

Ball-of-mud Problem: business logic spreads all over the place. ”Business logic” classes that perform no calculations but do perform I/O.

Customers do not care about how the data is stored (MySQL, SQLite, Redis, etc).

What do they care?

Domain model includes business logic.

Wrong: build a database schema ﬁrst for whatever software application.

Correct: behavior should come ﬁrst and drive our storage requirements.

Public contract.

• Repository pattern - for persistent storage.

• Service layer - for use cases.

• Unit of Work pattern - for atomic operations.

We don’t want our domain model to depend on infrastructure in afraid of slowing down unit tests or making changes hard.

An abstraction over persistent storage (add() and get()). A repository pattern pretends that all data is in memory. Usage example: import a l l m y d a t a def c r e a t e a b a t c h ( ) : batch = Batch(...)

Copyright ©2021 Hui Lan 48 a l l m y data.batches.add(batch) def m o d i f y a batch(batch i d , n e w q u a n t i t y ) : batch = a l l m y data.batches.get(batch i d ) batch. change i n i t i a l quantity(new q u a n t i t y )

Abstract Repository (allowing us to use repo.get() instead of session.query()): c l a s s AbstractRepository(abc.ABC): @abc. abstractmethod #(1) def add(self , batch: model.Batch): r a i s e NotImplementedError #(2)

@abc. abstractmethod def get(self , reference) => model.Batch: r a i s e NotImplementedError

Avoid using raw SQL in the domain model.

Check Figure 1. Before and after the Repository pattern in Chapter 2.

- Retrieve batch info from database.

- Save batch info to database.

Check Figure 3 (onion architecture) in Chapter 2.

Dependency Inversion Principle (DIP): high-level

Why? Decouple core logic from infrastructural concerns.

What?

• High-level modules should not depend on low-level modules. Both should depend on abstractions.

• Abstractions should not depend on details. Instead, details should depend on abstractions.

How? Depending on abstractions. Interfaces. Let infrastructure depend on domain. def allocate(line: OrderLine , repo: AbstractRepository , session) => s t r :

We can use SqlAlchemyRepository or CsvRepository. Check Figure 4-1. Direct denpendency graph and Figure 4-2. Inverted dependency graph for illustration.

Benﬁts: allowing high-level modules to change more independently of low-level modules, vice versa.

Need a database.

1. Retrieve batch information from database.

2. Instantiate domain objects.

@flask.route.gubbins # gubbins means stuff def a l l o c a t e endpoint (): # extract order line from request line = OrderLine(request.params, ...) # load all batches from the DB batches = ...

Translate the model to a relational database.

ORM - object-relational mapper

O - the world of objects and domain modeling.

R - the world of databases and relational algebra.

Beneﬁt of ORM: persistence ignorance.

Heavily depend on SQLAlchemy’s ORM now: from s q l a l c h e m y import Column, ForeignKey, Integer , String from sqlalchemy.ext. declarative import declarative b a s e from sqlalchemy .orm import relationship

Copyright ©2021 Hui Lan 56 Base = declarative b a s e ( ) c l a s s Order(Base): id = Column(Integer , primary k e y=True ) c l a s s OrderLine(Base): id = Column(Integer , primary k e y=True ) sku = Column(String(250)) qty = Integer(String(250)) o r d e r id = Column(Integer , ForeignKey(’order.id’)) order = relationship(Order) c l a s s Allocation(Base): ...

Is the above model indenpendent (ignorant) of database?

Principle: Persistence Ignorance (PI). Ignore particular database technologies. Purpose: make the domain model stay ”pure”. Explicit ORM mapping with SQLAlchemy Table objects.

Note the statement import model. # orm . py from sqlalchemy .orm import mapper, relationship import model #(1)

Copyright ©2021 Hui Lan 59 metadata = MetaData() o r d e r lines = Table( #(2) ” o r d e r l i n e s ” , metadata , Column(”id”, Integer , primary key=True , autoincrement=True), Column(”sku”, String(255)), Column(”qty”, Integer , nullable=False), Column(”orderid”, String(255)), )

... def s t a r t m a p p e r s ( ) : l i n e s mapper = mapper(model.OrderLine , order l i n e s ) #(3)

session: a SQLAlchemy database session. def t e s t o r d e r l i n e m a p p e r c a n l o a d lines(session): #(1) session.execute( ”INSERT INTO o r d e r lines (orderid , sku, qty) VALUES ” ’(”order1”, ”RED=CHAIR”, 12),’ ’(”order1”, ”RED=TABLE” , 13) , ’ ’(”order2”, ”BLUE=LIPSTICK”, 14) ’ ) e xp e ct e d = [ model.OrderLine(”order1”, ”RED=CHAIR”, 12), model.OrderLine(”order1”, ”RED=TABLE” , 13) , model.OrderLine(”order2”, ”BLUE=LIPSTICK”, 14), ]

Copyright ©2021 Hui Lan 62 assert session.query(model.OrderLine). a l l () == expected def t e s t o r d e r l i n e m a p p e r c a n s a v e lines(session): n e w line = model.OrderLine(”order1”, ”DECORATIVE=WIDGET” , 12) session.add(new l i n e ) session .commit()

rows = l i s t (session.execute(’SELECT orderid , sku, qty FROM ”order l i n e s ” ’ ) ) assert rows == [(”order1”, ”DECORATIVE=WIDGET” , 1 2 ) ]

We use a more speciﬁc repository here: SqlAlchemyRepository.

From session.query(Batch) (the SQLAlchemy language) to batches repo.get (the repository pattern).

Test add - saving an object def t e s t r e p o s i t o r y c a n s a v e a batch(session): batch = model.Batch(”batch1”, ”RUSTY=SOAPDISH” , 100 , eta=None )

repo = repository.SqlAlchemyRepository(session) repo.add(batch) #(1) session .commit() #(2)

Copyright ©2021 Hui Lan 64 rows = session.execute( #(3) ’SELECT reference , sku, p u r c h a s e d quantity , eta FROM ”batches”’ ) a s s e r t l i s t (rows) == [(”batch1”, ”RUSTY=SOAPDISH” , 100 , None ) ]

Test get - retrieving a complex object. def i n s e r t o r d e r line(session): session.execute( #(1) ”INSERT INTO o r d e r lines (orderid , sku, qty)” ’ VALUES (”order1”, ”GENERIC=SOFA” , 12) ’ ) [[ orderline id]] = session.execute( ”SELECT i d FROM o r d e r lines WHERE orderid=:orderid AND sku=:sku”, d i c t (orderid=”order1”, sku=”GENERIC=SOFA” ) , ) return o r d e r l i n e i d

Copyright ©2021 Hui Lan 65 def i n s e r t batch(session , batch i d ) : #(2) ... def t e s t r e p o s i t o r y c a n r e t r i e v e a b a t c h w i t h allocations(session): o r d e r l i n e i d = i n s e r t o r d e r line(session) b a t c h 1 i d = i n s e r t batch(session , ”batch1”) i n s e r t batch(session , ”batch2”) i n s e r t allocation(session , orderline i d , b a t c h 1 i d ) #(2)

repo = repository.SqlAlchemyRepository(session) retrieved = repo.get(”batch1”)

expected = model.Batch(”batch1”, ”GENERIC=SOFA”, 100, eta=None) assert retrieved == expected # Batch . e q only compares reference #(3) assert retrieved.sku == expected.sku #(4)

Copyright ©2021 Hui Lan 66 assert retrieved. p u r c h a s e d quantity == expected. p u r c h a s e d q u a n t i t y assert retrieved. allocations == { #(4) model.OrderLine(”order1”, ”GENERIC=SOFA” , 12) , }

SqlAlchemyRepository: c l a s s SqlAlchemyRepository(AbstractRepository ): def i n i t (self , session): self.session = session

def add(self , batch): self .session.add(batch)

def get(self , reference): return self .session.query(model.Batch). filter by(reference=reference ).one()

def l i s t ( s e l f ) :

Not using Repository Pattern: @flask.route.gubbins def a l l o c a t e endpoint (): session = start s e s s i o n ( )

# extract order line from request line = OrderLine( request.json[’orderid ’], request.json[ ’sku’], request.json[’qty’], )

# load all batches from the DB

# call our domain service allocate(line , batches)

# save the allocation back to the database session .commit()

return 201

Using Repository Pattern: @flask.route.gubbins def a l l o c a t e endpoint (): batches = SqlAlchemyRepository. l i s t () l i n e s = [ OrderLine(l[’orderid’], l[’sku’], l[’qty’]) f o r l in request.params ...

Problem: high-level code is coupled to low-level details. Tricks:

• Use simple abstractions to hide mess.

• Separate what we want to do from how to do it.

• Make interface between business logic and messy I/O.

Coupling: changing component A may break component B.

Globally, coupling is bad.

Figure 1. Lots of coupling in Chapter 3.

Figure 2. Less coupling in Chapter 3.

Example: directory sync.

Imperative Shell, Functional Core.

This layer deals with orchestration logic. What does the Service Layer pattern look like? How is it diﬀerent from entrypoint, i.e., Flask API endpoint? How is it diﬀerent from domain model, i.e., business logic? Flask API talks to the service layer; the service layers talks to the domain model. Don’t forget our task: allocate an order line to a batch.

Test code: 11.8 KB Functional code: 8.7KB Fast tests (e.g., unit tests) and slow tests (e.g., E2E tests).

What we have now:

• Domain model.

• Domain service (allocate).

• Repsitory interface.

@pytest.mark. usefixtures(”restart a p i ” ) def t e s t a p i r e t u r n s allocation(add s t o c k ) : sku, othersku = random sku(), random sku(”other”) #(1) earlybatch = random batchref(1)

Copyright ©2021 Hui Lan 76 laterbatch = random batchref(2) otherbatch = random batchref(3) a d d s t o c k ( #(2) [ (laterbatch , sku, 100, ”2011=01=02” ) , (earlybatch , sku, 100, ”2011=01=01” ) , (otherbatch , othersku , 100, None), ] ) data = {”orderid”: random orderid(), ”sku”: sku, ”qty”: 3} url = config.get a p i u r l ( ) #(3)

r = requests.post(f”{ u r l }/allocate”, json=data)

assert r.status c o d e == 201 assert r.json()[”batchref”] == earlybatch

Copyright ©2021 Hui Lan 77 @pytest.mark. usefixtures(”restart a p i ” ) def t e s t allocations a r e persisted(add s t o c k ) : sku = random sku ( ) batch1, batch2 = random batchref(1), random batchref(2) order1, order2 = random orderid(1), random o r d e r i d ( 2 ) a d d s t o c k ( [(batch1, sku, 10, ”2011=01=01”), (batch2, sku, 10, ”2011=01=02” ) , ] ) l i n e 1 = {”orderid”: order1, ”sku”: sku, ”qty”: 10} l i n e 2 = {”orderid”: order2, ”sku”: sku, ”qty”: 10} url = config.get a p i u r l ( )

# first order uses up all stock in batch 1 r = requests.post(f”{ u r l }/allocate”, json=line1) assert r.status c o d e == 201 assert r.json()[”batchref”] == batch1

@pytest.mark. usefixtures(”restart a p i ” ) def t e s t 4 0 0 m e s s a g e f o r o u t o f s t o c k ( a d d s t o c k ) : #(1) sku , smalL batch, large order = random sku(), random batchref(), random o r d e r i d ( ) a d d s t o c k ( [ ( smalL batch, sku, 10, ”2011=01=01” ) , ] ) data = {”orderid”: large order , ”sku”: sku, ”qty”: 20} url = config.get a p i u r l ( ) r = requests.post(f”{ u r l }/allocate”, json=data) assert r.status c o d e == 400

@pytest.mark. usefixtures(”restart a p i ” ) def t e s t 4 0 0 m e s s a g e f o r i n v a l i d s k u ( ) : #(2) unknown sku, orderid = random sku(), random o r d e r i d ( ) data = {”orderid”: orderid , ”sku”: unknown sku, ”qty”: 20} url = config.get a p i u r l ( ) r = requests.post(f”{ u r l }/allocate”, json=data) assert r.status c o d e == 400 assert r.json()[”message”] == f”Invalid sku {unknown sku}”

@pytest.mark.useﬁxtures(“restart api”): call restart api in conftest.py before carrying out the test. from f l a s k import Flask , request from s q l a l c h e m y import c r e a t e e n g i n e

Copyright ©2021 Hui Lan 80 from sqlalchemy .orm import sessionmaker import c o n f i g import model import orm import r e p o s i t o r y orm . s t a r t m a p p e r s ( ) g e t session = sessionmaker(bind=create engine(config.get p o s t g r e s u r i ( ) ) ) app = F l a s k ( name ) def i s v a l i d sku(sku, batches): return sku in {b . sku f o r b in b a t c h e s }

@app. route(”/allocate”, methods=[”POST”])

Copyright ©2021 Hui Lan 81 def a l l o c a t e endpoint (): session = get s e s s i o n ( ) batches = repository.SqlAlchemyRepository(session ). l i s t () line = model.OrderLine( request.json[”orderid”], request.json[”sku”], request.json[”qty”], )

i f not i s v a l i d sku(line.sku, batches): return {”message”: f”Invalid sku { l i n e . sku}” } , 400

t r y : batchref = model.allocate(line , batches) except model.OutOfStock as e: return {” message ” : s t r ( e )} , 400

session .commit() return {”batchref”: batchref } , 201

Solution: introduce a service layer and put orchestration logic in this layer.

Steps:

• Fetch some objects from the repository.

• Make some checks.

• Call a domain service.

• Save/update any state that has been changed.

Copyright ©2021 Hui Lan 83 c l a s s InvalidSku(Exception): pass def i s v a l i d sku(sku, batches): return sku in {b . sku f o r b in b a t c h e s } def allocate(line: OrderLine , repo: AbstractRepository , session) => s t r : batches = repo. l i s t () #(1) i f not i s v a l i d sku(line.sku, batches): #(2) r a i s e InvalidSku(f”Invalid sku { l i n e . sku}”) batchref = model.allocate(line , batches) #(3) session .commit() #(4) return b a t c h r e f

Two awkwardness in the service layer: (1) Coupled to

Copyright ©2021 Hui Lan 84 OrderLine. (2) Coupled to session. Depend on Abstractions: def allocate(line: OrderLine, repo: AbstractRepository, session) Let the Flask API endpoint do “web stuﬀ” only. @app. route(”/allocate”, methods=[”POST”]) def a l l o c a t e endpoint (): session = get s e s s i o n ( ) #(1) repo = repository.SqlAlchemyRepository(session) #(1) line = model.OrderLine( request.json[”orderid”], request.json[”sku”], request.json[”qty”], #(2) )

t r y : batchref = services.allocate(line , repo, session) #(2)

return {”batchref”: batchref } , 201 ( 3 )

E2E tests: happy path and unhappy path. @pytest.mark. usefixtures(”restart a p i ” ) def t e s t h a p p y p a t h r e t u r n s 2 0 1 a n d a l l o c a t e d b a t c h ( a d d s t o c k ) : sku, othersku = random sku(), random sku(”other”) earlybatch = random batchref(1) laterbatch = random batchref(2) otherbatch = random batchref(3) a d d s t o c k ( [ (laterbatch , sku, 100, ”2011=01=02” ) , (earlybatch , sku, 100, ”2011=01=01” ) , (otherbatch , othersku , 100, None),

r = requests.post(f”{ u r l }/allocate”, json=data)

assert r.status c o d e == 201 assert r.json()[”batchref”] == earlybatch

@pytest.mark. usefixtures(”restart a p i ” ) def t e s t u n h a p p y p a t h r e t u r n s 4 0 0 a n d e r r o r m e s s a g e ( ) : unknown sku, orderid = random sku(), random o r d e r i d ( ) data = {”orderid”: orderid , ”sku”: unknown sku, ”qty”: 20} url = config.get a p i u r l ( ) r = requests.post(f”{ u r l }/allocate”, json=data) assert r.status c o d e == 400

Service layer vs. domain service. For example, where should we put the responsiblity of Calculating Tax?

Check Putting Things in Folders to See Where It All Belongs.

UoW (pronounced you-wow): Unit of Work Without UoW: Check Figure 1. Without UoW: API talks directly to three layers

• The Flask API talks directly to the database layer to start a session.

• The Flask API talks to the repository layer to initialize

• The Flask API talks to the service layer to ask it to allocate.

With UoW: Check Figure 2. With UoW: UoW now manages database state Beneﬁt: Decouple service layer from the data layer.

• The Flask API initializes a unit of work.

• The Flask API invokes a service.

service layer/services.py: def a l l o c a t e ( o r d e r i d : str , sku : str , qty : int , uow : u n i t o f work .AbstractUnitOfWork , ) => s t r : line = OrderLine(orderid , sku, qty) with uow : #(1) batches = uow.batches. l i s t () #(2) ... batchref = model.allocate(line , batches) uow.commit() #(3)

Use uow as a context manager (see this tutorial for more explanation on Context Managers).

integration/test uow.py def t e s t u o w c a n r e t r i e v e a b a t c h a n d a l l o c a t e t o i t ( s e s s i o n f a c t o r y ) : session = session f a c t o r y ( ) i n s e r t batch(session , ”batch1”, ”HIPSTER=WORKBENCH” , 100 , None ) session .commit()

uow = u n i t o f work .SqlAlchemyUnitOfWork(session f a c t o r y ) #(1) with uow : batch = uow.batches.get(reference=”batch1”) #(2) line = model.OrderLine(”o1”, ”HIPSTER=WORKBENCH” , 10) batch.allocate(line) uow.commit() #(3)

batchref = get a l l o c a t e d b a t c h ref(session , ”o1”, ”HIPSTER=WORKBENCH” ) assert batchref == ”batch1”

def e x i t ( s e l f , * a r g s ) : #(2) self.rollback() #(4)

@abc. abstractmethod def commit(self ): #(3) r a i s e NotImplementedError

@abc. abstractmethod def rollback(self ): #(4) r a i s e NotImplementedError

Copyright ©2021 Hui Lan 94 DEFAULT SESSION FACTORY = sessionmaker( #(1) bind=create e n g i n e ( c o n f i g . g e t p o s t g r e s u r i ( ) , ) ) c l a s s SqlAlchemyUnitOfWork(AbstractUnitOfWork ): def i n i t (self , session factory=DEFAULT SESSION FACTORY ) : self.session factory = session f a c t o r y #(1)

def e n t e r ( s e l f ) : self.session = self.session f a c t o r y ( ) # type: Session #(2) self .batches = repository.SqlAlchemyRepository(self .session) #(2) return super (). e n t e r ()

def commit(self ): #(4) self .session.commit()

def rollback(self ): #(4) self.session.rollback()

Use the UoW in the service layer def add batch ( r e f : str , sku : str , qty : int , eta: Optional[date], uow : u n i t o f work .AbstractUnitOfWork , #(1) ): with uow :

Copyright ©2021 Hui Lan 96 uow.batches.add(model.Batch(ref , sku, qty, eta)) uow.commit() def a l l o c a t e ( o r d e r i d : str , sku : str , qty : int , uow : u n i t o f work .AbstractUnitOfWork , #(1) ) => s t r : line = OrderLine(orderid , sku, qty) with uow : batches = uow.batches. l i s t () i f not i s v a l i d sku(line.sku, batches): r a i s e InvalidSku(f”Invalid sku { l i n e . sku}”) batchref = model.allocate(line , batches) uow.commit() return b a t c h r e f

A collection of entities (domain objects) treated a unit for data changes. Example: a shopping cart (encapsulating many items). An aggregate provides a single Consistency Boundary. A shopping cart provides a consistency bounary. We do not change the shopping carts of several customers at the same time. Cart is the accesible root entity, while other entities are not directly accessible from outside. Example: Aggregates with multiple or single entities

Invariants – things that must be true after we ﬁnish an operation.

Constraints – double-booking is not allowed.

Business rules for order line allocation: 1 and 2.

Concurrency and locks: allocating 360,000 order lines per hour. How many order lines per second? Too slow if we lock the batches table while allocating each order line. Not OK: allocate multiple order lines for the same product

OK: allocate multiple order lines for diﬀerent products at the same time.

Bounded Context:

• Product(sku, batches) for allocation service.

• Product(sku, description, price, image url) for ecommerce.

Deﬁne the aggregate Product:

def allocate(self , line: OrderLine) => s t r : #(3) t r y : batch = next ( b f o r b in sorted (self .batches) i f b . c a n allocate(line)) batch.allocate(line) return batch. reference except StopIteration: r a i s e OutOfStock(f”Out of stock for sku { l i n e . sku}”)

What has changed in the repository? The get method. c l a s s SqlAlchemyRepository(AbstractRepository ): def i n i t (self , session): self.session = session

def get(self , sku): return self .session.query(model.Product). filter by(sku=sku). first ()

How does unit of work look like now? c l a s s SqlAlchemyUnitOfWork(AbstractUnitOfWork ): def i n i t (self , session factory=DEFAULT SESSION FACTORY ) : self.session factory = session f a c t o r y

def e n t e r ( s e l f ) : self.session = self.session f a c t o r y ( ) # type: Session self .products = repository.SqlAlchemyRepository(self .session) return super (). e n t e r ()

def commit(self ): self .session.commit()

def rollback(self ): self.session.rollback()

How does service allocate look like now? def add batch ( r e f : str , sku : str , qty : int , eta: Optional[date], uow : u n i t o f work .AbstractUnitOfWork , ): with uow : product = uow.products.get(sku=sku)

Copyright ©2021 Hui Lan 103 i f product i s None : product = model.Product(sku, batches=[]) uow.products.add(product) product.batches.append(model.Batch(ref , sku, qty, eta)) uow.commit() def a l l o c a t e ( o r d e r i d : str , sku : str , qty : int , uow : u n i t o f work .AbstractUnitOfWork , ) => s t r : line = OrderLine(orderid , sku, qty) with uow : product = uow.products.get(sku=line .sku) i f product i s None : r a i s e InvalidSku(f”Invalid sku { l i n e . sku}”) batchref = product.allocate(line) uow.commit()

Task: send out an email alert when allocation is failed due to out of stock. Possible solutions:

• Dump the email sending stuﬀ in flask app.py, the HTTP layer (the endpoint). Quick and dirty. See function send mail. @app. route(”/allocate”, methods=[”POST”])

Copyright ©2021 Hui Lan 106 def a l l o c a t e endpoint (): line = model.OrderLine( request.json[”orderid”], request.json[”sku”], request.json[”qty”], ) t r y : uow = u n i t o f work .SqlAlchemyUnitOfWork() batchref = services.allocate(line , uow) except (model.OutOfStock, services.InvalidSku) as e: s e n d m a i l ( ”out of stock”, ” stock [email protected]” , f ”{ line.orderid } = { l i n e . sku}” ) return {” message ” : s t r ( e )} , 400

return {”batchref”: batchref } , 201

Copyright ©2021 Hui Lan 107 Question: Should the HTTP layer be responsible for sending out alert email messages? This may violate the Thin Web Controller Principle, as well violate Single Responsibility Principle (SRP). Also, how to do unit testing?

• Move the email sending stuﬀ to the domain layer, i.e., model.py. Question: Is the domain layer for sending email, or for allocating an order line?

• Move the email sending stuﬀ to the service layer, i.e., services.py. def a l l o c a t e (

Copyright ©2021 Hui Lan 108 o r d e r i d : str , sku : str , qty : int , uow : u n i t o f work .AbstractUnitOfWork , ) => s t r : line = OrderLine(orderid , sku, qty) with uow : product = uow.products.get(sku=line .sku) i f product i s None : r a i s e InvalidSku(f”Invalid sku { l i n e . sku}”) t r y : batchref = product.allocate(line) uow.commit() return b a t c h r e f except model.OutOfStock: e m a i l . s e n d mail(”[email protected]”, f”Out of stock for { l i n e . sku}”) r a i s e

allocate or allocate and send mail if out of stock?

• Make an Event class for the Out of Stock event. Check domain/events.py. from dataclasses import d a t a c l a s s

c l a s s Event : #(1) pass

@ d a t a c l a s s c l a s s OutOfStock(Event ): #(2)

An event becomes a value object. Do you still remember that OutOfStock is an exception class in the previous chapters?

• Add an attribute called events in class Product.

• Append this event object in events – raise the event.

Write test before modifying model.py and other ﬁles. def t e s t r e c o r d s o u t o f s t o c k e v e n t i f c a n n o t allocate (): batch = Batch(”batch1”, ”SMALL=FORK” , 10 , eta=today ) product = Product(sku=”SMALL=FORK”, batches=[batch]) product. allocate(OrderLine(”order1”, ”SMALL=FORK” , 10))

Note: product.events[-1]. Now let’s update model.py. from . import e v e n t s c l a s s Product : def i n i t ( s e l f , sku : str , batches: List[Batch], version n u m b e r : i n t = 0 ) : self.sku = sku self.batches = batches self .version number = version n u m b e r self.events = [] # type: List[events.Event]

Copyright ©2021 Hui Lan 112 def allocate(self , line: OrderLine) => s t r : t r y : batch = next ( b f o r b in sorted (self .batches) i f b . c a n allocate(line)) batch.allocate(line) self .version n u m b e r += 1 return batch. reference except StopIteration: self .events.append(events.OutOfStock(line .sku)) return None

Note: we no longer raise the OutOfStock exception now.

Purpose: to provide a uniﬁed way of invoking use cases from the endpoint. Check Figure 1. Events ﬂowing through the system in Chapter 8 of the textbook. Method: mapping an event to a list of handlers (function names). It is a dictionary. A message bus has an important function called handle.

Copyright ©2021 Hui Lan 114 HANDLERS = { events.OutOfStock: [send o u t o f s t o c k notification], } # type: Dict[Type[events.Event], List[Call def handle(event: events.Event): f o r h a n d l e r in HANDLERS[ type ( event ) ] : handler(event) def s e n d o u t o f s t o c k notification(event: events.OutOfStock): e m a i l . s e n d m a i l ( ”[email protected]” , f”Out of stock for { event . sku}”, )

HANDLERS is a dictionary. Its key is an event type, and its

In which folder should we put messagebus.py?

Where will the function handle be called?

services.py: from . import messagebus ... def a l l o c a t e ( o r d e r i d : str , sku : str , qty : int , uow : u n i t o f work .AbstractUnitOfWork , ) => s t r : line = OrderLine(orderid , sku, qty) with uow : product = uow.products.get(sku=line .sku) i f product i s None :

Copyright ©2021 Hui Lan 117 r a i s e InvalidSku(f”Invalid sku { l i n e . sku}”) t r y : #(1) batchref = product.allocate(line) uow.commit() return b a t c h r e f f i n a l l y : #(1) messagebus.handle(product.events) #(2)

services.py: def a l l o c a t e ( o r d e r i d : str , sku : str , qty : int , uow : u n i t o f work .AbstractUnitOfWork , ) => s t r : line = OrderLine(orderid , sku, qty) with uow : product = uow.products.get(sku=line .sku) i f product i s None : r a i s e InvalidSku(f”Invalid sku { l i n e . sku}”) batchref = product.allocate(line) uow.commit() #(1)

... and message bus handles them. from . import messagebus c l a s s AbstractUnitOfWork(abc.ABC): products: repository.AbstractRepository

def e n t e r ( s e l f ) => AbstractUnitOfWork : return s e l f

def e x i t ( s e l f , * a r g s ) : self.rollback()

def p u b l i s h events(self ): f o r product in self .products.seen: while product.events: event = product.events.pop(0) messagebus.handle(event)

@abc. abstractmethod def commit(self ): r a i s e NotImplementedError

@abc. abstractmethod def rollback(self ): r a i s e NotImplementedError

Question: how to understand the seen in repostory.py? Magic. Most complex design as the authors said!

Treat events as ﬁrst-class citizens.

• BatchCreated

• BatchQuantityChanged

• AllocationRequired

• OutOfStock

How does the endpoint flask app.py look like now?

# add batch event = events.BatchCreated( request.json["ref"], request.json["sku"], request.json["qty"], eta )

messagebus.handle(event, unit_of_work.SqlAlchemyUnitOfWork())

# allocate order line event = events.AllocationRequired(

messagebus.handle(event, unit_of_work.SqlAlchemyUnitOfWork())

Convert use-case functions to event handlers.

services.py is replaced by messagebus.py. def handle ( event: events.Event, uow : u n i t o f work .AbstractUnitOfWork , ): r e s u l t s = [ ] queue = [event] while queue : event = queue.pop(0) f o r h a n d l e r in HANDLERS[ type ( event ) ] : results .append(handler(event , uow=uow))

HANDLERS = { events.BatchCreated: [handlers.add batch ] , events.BatchQuantityChanged: [handlers.change b a t c h q u a n t i t y ] , events.AllocationRequired: [handlers.allocate], events.OutOfStock: [handlers.send o u t o f s t o c k notification], } # type: Dict[Type[events.Event], List[Callable]]

Question: what does uow.collect new events() do? Handle a series of old and new events using handler. Use a handler to handle an incoming event. This handler could make another event, which is to be handled by another

A possibility of circular dependency?

Example: the event BatchQuantityChanged is handled by change batch quantity, which could in turn make an event called AllocationRequired. change batch quantity in model.py. class Product: def __init__(self, sku: str, batches: List[Batch], version_number: int = 0): self.sku = sku self.batches = batches self.version_number = version_number self.events = [] # type: List[events.Event]

Copyright ©2021 Hui Lan 129 def allocate(self, line: OrderLine) -> str: try: batch = next(b for b in sorted(self.batches) if b.can_allocate(line)) batch.allocate(line) self.version_number += 1 return batch.reference except StopIteration: self.events.append(events.OutOfStock(line.sku)) return None

def change_batch_quantity(self, ref: str, qty: int): batch = next(b for b in self.batches if b.reference == ref) batch._purchased_quantity = qty while batch.available_quantity < 0: line = batch.deallocate_one() self.events.append( ########################################## events.AllocationRequired(line.orderid, line.sku, line.qty) )

Now the service layer becomes an event processor.

Before: uow pushes events onto the message bus.

Now: the message bus pulls events from the uow.

BatchCreated versus CreateBatch.

BatchCreated - event (broadcast knowledge, for notiﬁcation).

CreateBatch - command (capture intent).

Examples: class Command: pass

@dataclass class Allocate(Command): #(1) orderid: str sku: str qty: int

@dataclass class ChangeBatchQuantity(Command): #(3)

Separate two kinds of handlers, EVENT HANDLERS and COMMAND HANDLERS: EVENT HANDLERS = { events.OutOfStock: [handlers.send o u t o f s t o c k notification], } # type: Dict[Type[events.Event], List[Callable]]

COMMAND HANDLERS = { commands.Allocate: handlers.allocate , commands.CreateBatch: handlers .add batch , commands.ChangeBatchQuantity: handlers . change b a t c h q u a n t i t y , } # type: Dict[Type[commands.Command], Callable]

The message bus should be adpated with the commands.

Message = Union[commands.Command, events .Event] def handle ( #(1) message: Message, uow : u n i t o f work .AbstractUnitOfWork , ): r e s u l t s = [ ] queue = [message] while queue : message = queue.pop(0) if isinstance (message, events.Event): h a n d l e event(message, queue, uow) #(2) elif isinstance (message , commands.Command): c m d result = handle command(message , queue, uow) #(2)

It forces writing tests before writing code. It helps understand requirements better. A good reading: Yamaura, Tsuneo. ”How to Design Practical Test Cases.” IEEE Software (November/December 1998): 30-36. It ensures that code continues working after we make changes. Note that changes to one part of the code can inadvertently

Related concepts: IoC (Inversion of Control), DIP (dependency inversion principle), interfaces.

Class A uses class B. Without dependency injection (DI), we create an object of class B, b, inside class A and use b’s useful methods. Class A is responsible for creating b.

With DI, instead of creating an object of class B inside class A, create b outside, and pass this object to class A’s initialization method.

Copyright ©2021 Hui Lan 139 If class A depends on object b (i.e., class A’s dependency is b), then Inject b to class A through the initialization method. Class A is no longer responsible for creating b. Why is dependency injection good? Loose coupling. We can inject any object derived from a subclass of the Abstract Base Class B to class A. More ﬂexible. Easier to test. Example: the allocate’s second argument (repo)’s type is Abstract Base Class. We can pass a repo of FakeRepository, or a repo of SqlAlchemyRepository. # from services.py in Chapter 4 def allocate(line: OrderLine , repo: AbstractRepository , session) => s t r : batches = repo. l i s t () i f not i s v a l i d sku(line.sku, batches):

Example: the allocate’s second argument (uow)’s type is Abstract Base Class. We can pass a uow of FakeUnitOfWork, or a uow of SqlAlchemyUnitOfWork. # from services.py in Chapter 8 def a l l o c a t e ( o r d e r i d : str , sku : str , qty : int , uow : u n i t o f work .AbstractUnitOfWork , ) => s t r : line = OrderLine(orderid , sku, qty) with uow : product = uow.products.get(sku=line .sku)

Allow diﬀerent implementations. See What is Dependency Injection with Java Code Example. See Figure 4-1 and 4-2 in Architectural principles.

• Constructor injection – inject through constructor.

• Setter injection – inject through setter.