A Domain-Specific Design Patterns Perspective

Total Page:16

File Type:pdf, Size:1020Kb

A Domain-Specific Design Patterns Perspective Characterizing the Impact of Context-Variables in Software Performance Factors: a Domain-Specific Design Patterns Perspective Master Thesis Document Presented in partial fulfillment to obtain the Title of Magister in Informatics and Telecommunications by Juan Manuel Ropero L. Advisor: Gabriel Tamura, PhD Department of Information and Communication Technologies Faculty of Engineering 2017 Contents 1 Introduction 2 1.1 Motivation and Background . .2 1.2 Problem Statement . .3 1.3 Challenges . .3 1.4 Research Objectives . .4 1.4.1 General . .4 1.4.2 Specific . .4 1.5 Methodology . .4 1.6 Contributions . .5 2 State-of-the-Art Background 6 2.1 Self-Adaptive Software (SAS) Systems . .6 2.1.1 The MAPE-K Loop . .7 2.1.2 Self-* Properties . .7 2.2 Service Component Architecture (SCA) . .8 2.3 The Performance Quality Attributes and Performance Factors . .8 2.3.1 Performance Factor Definitions . .9 2.4 Context-Aware Computing . .9 2.4.1 Context-Variable Definitions . 10 2.4.2 Design Patterns for Software Architecture . 14 2.5 Chapter Summary . 15 3 Problem Modeling 17 3.1 The Problem Context . 17 3.2 The Need for an Experiments Design . 18 3.3 Identification of Performance Factors . 18 3.3.1 Search and Identification Process . 18 3.3.2 Distribution Results of Performance Factors . 19 3.3.3 Throughput - Special Consideration . 19 3.4 Identification of Context Variables . 21 3.4.1 Search and Identification Process . 21 3.4.2 Context-variables Categorization . 22 3.4.3 Context Variables - Special Considerations . 22 3.5 Identification of Domain-Specific Design Patterns for Performance . 24 3.5.1 Overview of the Research Methodology . 24 i 3.6 Domain-Specific Design Patterns for Performance . 25 3.6.1 The Design Pattern Template . 26 3.6.2 Random Access Parser Design Pattern . 27 3.6.3 Reactor Design Pattern . 29 3.6.4 State-based Pipeline Design Pattern . 31 3.6.5 Thread Pool Design Pattern . 34 3.6.6 Master-Worker Design Pattern . 35 3.6.7 Separable Dependencies Design Pattern (Variant of 3.6.6) . 39 3.6.8 Fork / Join Design Pattern . 41 3.6.9 Producer-Consumer Design Pattern . 43 3.6.10 Sender Released . 46 3.6.11 Leader and Followers . 48 3.6.12 Half-Sync / Half-Async . 51 3.6.13 Sayl . 52 3.6.14 MapReduce . 55 3.7 Problem Model Instantiation . 58 3.7.1 Experiment Case Studies . 59 3.7.2 Domain-Specific Design Patterns for Performance - Selection . 61 3.7.3 Performance Factors - Selection . 63 3.7.4 Context Variables - Selection . 64 3.8 Chapter Summary . 65 4 Experiments Design 66 4.1 Experiment Environment Configuration . 66 4.1.1 Software Technologies . 66 4.1.2 The Warming Up Process . 67 4.1.3 Pilot Experiments for Special System Variables . 68 4.1.4 Hardware and Network Architecture . 69 4.1.5 Software Architecture for The Sorting Case . 69 4.1.6 Memory Architecture for The Sorting Case . 76 4.1.7 Variable Configuration: Pilot Experiments . 77 4.2 Experiments Design . 83 4.2.1 The Sorting Case - Experiments Design . 83 4.2.2 The Large XML Processing Case - Experiments Design . 90 4.3 Chapter Summary . 90 5 Analysis of Experiment Results 93 5.1 Impact of Context-Variables on Latency for the Sorting Case . 94 5.1.1 Design Patterns and Memory Structure Variations . 94 5.1.2 Network Bandwidth . 95 5.1.3 Communication Time . 97 5.1.4 RAM Memory Usage . 106 5.1.5 Buffer Size . 111 5.1.6 Batch Time Span . 114 5.1.7 File Length . 114 5.1.8 Task Granularity: Batch-Size . 121 ii 5.1.9 Number of Available Distributed Task Processors . 122 5.1.10 Number of Available Task Processors + Task Granularity . 125 5.1.11 Number of Available Task Processors + File Length . 125 5.1.12 Task Granularity + File Length . 129 5.1.13 Number of Task Processors + Task Granularity + File Length . 133 5.1.14 An Isolated Experiment for The UMA-NORMA Memory Structure . 144 5.2 Context-Variables Impact on Throughput for The Sorting Case . 144 5.2.1 Number of Service Requests . 145 5.2.2 Design Patterns and File Size . 145 5.2.3 Number of Available Distributed Task Processors . 146 5.2.4 The Impact of Two-Mergers . 150 5.3 A Comparative Analysis between Latency and Throughput for The Sorting Case . 150 5.4 A Complementary Evaluation of The XML Processing Case . 153 5.4.1 Summary of the Throughput Analysis from "Processing Large Volumes of Data Physically Stored in XML Files" . 155 5.4.2 Latency Performance Analysis . 157 5.4.3 Batch Time Span Evaluation in the XML Processing Case . 157 5.5 Best Performance Combinations . 157 5.5.1 Sorting Case . 159 5.5.2 Best Latency Results . 159 5.5.3 Best Throughput Results . 159 5.5.4 XML Processing Case . 159 5.6 Chapter Summary . 159 6 Summary and Conclusions 162 iii List of Tables 3.1 Context-Variables Categorization . 23 4.1 JVM Heap Size Experiment . 68 4.2 Hardware Technical Specifications . 69 4.3 Network Switch Hardware Specifications . 70 4.4 Latency Results for the NORMA Configuration of the Experimental Pilot for Batch Size Tests . 78 4.5 Latency Results for the UMA Configuration of the Experimental Pilot for Batch Size Tests............................................. 79 4.6 Latency Distribution Behavior Test for Master-Worker Under Different Architectures Configurations. All values are given in milliseconds . 80 4.7 Latency Distribution Behavior Test for Producer-Consumer Under Different Archi- tectures Configurations. All values are given in milliseconds . 81 4.8 Latency Distribution Behavior Test for Sayl Under Different Architectures Configu- rations. All values are given in milliseconds . 82 4.9 Producer-Consumer NORMA latency results using synchronous and asynchronous methods for triggering tasks. All latency data is presented in milliseconds. 84 4.10 Producer-Consumer UMA latency results using synchronous and asynchronous meth- ods for triggering tasks. All latency data is presented in milliseconds. 85 4.11 Sayl NORMA latency results using synchronous and asynchronous methods for trig- gering tasks. All latency data is presented in milliseconds. 86 4.12 Sayl UMA latency results using synchronous and asynchronous methods for trigger- ing tasks. All latency data is presented in milliseconds. 87 4.13 Experiments for the sorting case. Compendium of the variables variations for La- tency Experiments. 89.
Recommended publications
  • Leader/Followers
    Leader/Followers Douglas C. Schmidt, Carlos O’Ryan, Michael Kircher, Irfan Pyarali, and Frank Buschmann {schmidt, coryan}@uci.edu, {Michael.Kircher, Frank.Buschmann}@mchp.siemens.de, [email protected] University of California at Irvine, Siemens AG, and Washington University in Saint Louis The Leader/Followers architectural pattern provides an efficient concurrency model where multiple threads take turns sharing a set of event sources in order to detect, de- multiplex, dispatch, and process service requests that occur on these event sources. Example Consider the design of a multi-tier, high-volume, on-line transaction processing (OLTP) system. In this design, front-end communication servers route transaction requests from remote clients, such as travel agents, claims processing centers, or point-of-sales terminals, to back-end database servers that process the requests transactionally. After a transaction commits, the database server returns its results to the associated communication server, which then forwards the results back to the originating remote client. This multi-tier architecture is used to improve overall system throughput and reliability via load balancing and redundancy, respectively.It LAN WAN Front-End Back-End Clients Communication Servers Database Servers also relieves back-end servers from the burden of managing different communication protocols with clients. © Douglas C. Schmidt 1998 - 2000, all rights reserved, © Siemens AG 1998 - 2000, all rights reserved 19.06.2000 lf.doc 2 Front-end communication servers are actually ``hybrid'' client/server applications that perform two primary tasks: 1 They receive requests arriving simultaneously from hundreds or thousands of remote clients over wide area communication links, such as X.25 or TCP/IP.
    [Show full text]
  • A Survey of Architectural Styles.V4
    Survey of Architectural Styles Alexander Bird, Bianca Esguerra, Jack Li Liu, Vergil Marana, Jack Kha Nguyen, Neil Oluwagbeminiyi Okikiolu, Navid Pourmantaz Department of Software Engineering University of Calgary Calgary, Canada Abstract— In software engineering, an architectural style is a and implementation; the capabilities and experience of highest-level description of an accepted solution to a common developers; and the infrastructure and organizational software problem. This paper describes and compares a selection constraints [30]. These styles are not presented as out-of-the- of nine accepted architectural styles selected by the authors and box solutions, but rather a framework within which a specific applicable in various domains. Each pattern is presented in a software design may be made. If one were to say “that cannot general sense and with a domain example, and then evaluated in be called a layered architecture because the such and such a terms of benefits and drawbacks. Then, the styles are compared layer must communicate with an object other than the layer in a condensed table of advantages and disadvantages which is above and below it” then one would have missed the point of useful both as a summary of the architectural styles as well as a architectural styles and design patterns. They are not intended tool for selecting a style to apply to a particular project. The to be definitive and final, but rather suggestive and a starting paper is written to accomplish the following purposes: (1) to give readers a general sense of several architectural styles and when point, not to be used as a rule book but rather a source of and when not to apply them, (2) to facilitate software system inspiration.
    [Show full text]
  • The Need for Hardware/Software Codesign Patterns - a Toolbox for the New Digital Designer
    The need for Hardware/Software CoDesign Patterns - a toolbox for the new Digital Designer By Senior Consultant Carsten Siggaard, Danish Technological Institute May 30. 2008 Abstract Between hardware and software the level of abstraction has always differed a lot, this is not a surprise, hardware is an abstraction for software and the topics which is considered important by either a hardware developer or a software developer are quite different, making the communication between software and hardware developers (or even departments) difficult. In this report the software concept of design patterns is expanded into hardware, and the need for a new kind of design pattern, The Digital Design Pattern, is announced. Copyright © 2008 Danish Technological Institute Page 1 of 20 Contents Contents ............................................................................................................................................... 2 Background .......................................................................................................................................... 4 What is a design pattern ....................................................................................................................... 4 Three Disciplines when using Design Patterns ................................................................................ 5 Pattern Hatching ........................................................................................................................... 6 Pattern Mining.............................................................................................................................
    [Show full text]
  • Concurrency Patterns for Easier Robotic Coordination
    Concurrency Patterns for Easier Robotic Coordination Andrey Rusakov∗ Jiwon Shin∗ Bertrand Meyer∗† ∗Chair of Software Engineering, Department of Computer Science, ETH Zurich,¨ Switzerland †Software Engineering Lab, Innopolis University, Kazan, Russia fandrey.rusakov, jiwon.shin, [email protected] Abstract— Software design patterns are reusable solutions to Guarded suspension – are chosen for their concurrent nature, commonly occurring problems in software development. Grow- applicability to robotic coordination, and evidence of use ing complexity of robotics software increases the importance of in existing robotics frameworks. The paper aims to help applying proper software engineering principles and methods such as design patterns to robotics. Concurrency design patterns programmers who are not yet experts in concurrency to are particularly interesting to robotics because robots often have identify and then to apply one of the common concurrency- many components that can operate in parallel. However, there based solutions for their applications on robotic coordination. has not yet been any established set of reusable concurrency The rest of the paper is organized as follows: After design patterns for robotics. For this purpose, we choose six presenting related work in Section II, it proceeds with a known concurrency patterns – Future, Periodic timer, Invoke later, Active object, Cooperative cancellation, and Guarded sus- general description of concurrency approach for robotics pension. We demonstrate how these patterns could be used for in Section III. Then the paper presents and describes in solving common robotic coordination tasks. We also discuss detail a number of concurrency patterns related to robotic advantages and disadvantages of the patterns and how existing coordination in Section IV.
    [Show full text]
  • The Evolution of Control Architectures Towards Next Generation
    Department of Electronic and Electrical Engineering University College London University of London T h e Ev o l u t io n o f C o n t r o l A rchitectures T o w a r d s N ext G e n e r a t io n N e t w o r k s By Christos Solomonides BEng(Hons), MRes(Distinction) 7 September 2001 A thesis submitted to the University o f London for the Ph.D. in Electronic and Electrical Engineering ProQuest Number: U643054 All rights reserved INFORMATION TO ALL USERS The quality of this reproduction is dependent upon the quality of the copy submitted. In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if material had to be removed, a note will indicate the deletion. uest. ProQuest U643054 Published by ProQuest LLC(2016). Copyright of the Dissertation is held by the Author. All rights reserved. This work is protected against unauthorized copying under Title 17, United States Code. Microform Edition © ProQuest LLC. ProQuest LLC 789 East Eisenhower Parkway P.O. Box 1346 Ann Arbor, Ml 48106-1346 A b s t r a c t This thesis describes the evolution of control architectures and network intelligence towards next generation telecommunications networks. Network intelligence is a term given to the group of architectures that provide enhanced control services. Network intelligence is provided through the control plane, which is responsible for the establishment, operation and termination of calls and connections. The work focuses on examining the way in which network intelligence has been provided in the traditional telecommunications environment and in a converging environment.
    [Show full text]
  • Active Object
    Active Object an Object Behavioral Pattern for Concurrent Programming R. Greg Lavender Douglas C. Schmidt [email protected] [email protected] ISODE Consortium Inc. Department of Computer Science Austin, TX Washington University, St. Louis An earlier version of this paper appeared in a chapter in the book ªPattern Languages of Program Design 2º ISBN 0-201-89527-7, edited by John Vlissides, Jim Coplien, and Norm Kerth published by Addison-Wesley, 1996. : Output : Input Handler Handler : Routing Table : Message Abstract Queue This paper describes the Active Object pattern, which decou- 3: enqueue(msg) ples method execution from method invocation in order to : Output 2: find_route(msg) simplify synchronized access to a shared resource by meth- Handler ods invoked in different threads of control. The Active Object : Message : Input pattern allows one or more independent threads of execution Queue Handler 1: recv(msg) to interleave their access to data modeled as a single ob- ject. A broad class of producer/consumer and reader/writer OUTGOING OUTGOING MESSAGES GATEWAY problems are well-suited to this model of concurrency. This MESSAGES pattern is commonly used in distributed systems requiring INCOMING INCOMING multi-threaded servers. In addition,client applications (such MESSAGES MESSAGES as windowing systems and network browsers), are increas- DST DST ingly employing active objects to simplify concurrent, asyn- SRC SRC chronous network operations. 1 Intent Figure 1: Connection-Oriented Gateway The Active Object pattern decouples method execution from method invocation in order to simplify synchronized access to a shared resource by methods invoked in different threads [2]. Sources and destinationscommunicate with the Gateway of control.
    [Show full text]
  • Performance Analysis of the Reactor Pattern in Network Services
    Performance Analysis of the Reactor Pattern in Network Services Swapna Gokhale1, Aniruddha Gokhale2, Jeff Gray3, Paul Vandal1, Upsorn Praphamontripong1 1University of Connecticut 2Vanderbilt University Dept. of Computer Science Dept. of Electrical Engineering and Engineering and Computer Science Storrs, CT 06269 USA Nashville, TN 37235 USA [email protected] [email protected] 3University of Alabama at Birmingham Dept of Computer and Information Science Birmingham, AL USA [email protected] Abstract 1. Introduction Service oriented computing (SoC), which is made feasi- ble by middleware-based distributed systems, is an emerg- ing technology to provide the next-generation services to meet societal needs ranging from basic necessities, such as The growing reliance on services provided by software education, energy, communications and healthcare to emer- applications places a high premium on the reliable and ef- gency and disaster management. For SOC to be successful ficient operation of these applications. A number of these in meeting the demands of society, assurance on the perfor- applications follow the event-driven software architecture mance of these services is necessary. Since these services style since this style fosters evolvability by separating event are primarily built using communication middleware, the handling from event demultiplexing and dispatching func- problem reduces to the issue of performance assurance of tionality. The event demultiplexing capability, which ap- the middleware platforms. pears repeatedly across a class of event-driven applica- Middleware typically comprises a number of building tions, can be codified into a reusable pattern, such as the blocks, which are essentially patterns-based reusable soft- Reactor pattern. In order to enable performance analysis of ware frameworks.
    [Show full text]
  • The Design and Use of the ACE Reactor an Object-Oriented Framework for Event Demultiplexing
    The Design and Use of the ACE Reactor An Object-Oriented Framework for Event Demultiplexing Douglas C. Schmidt and Irfan Pyarali g fschmidt,irfan @cs.wustl.edu Department of Computer Science Washington University, St. Louis 631301 1 Introduction create concrete event handlers by inheriting from the ACE Event Handler base class. This class specifies vir- This article describes the design and implementation of the tual methods that handle various types of events, such as Reactor pattern [1] contained in the ACE framework [2]. The I/O events, timer events, signals, and synchronization events. Reactor pattern handles service requests that are delivered Applications that use the Reactor framework create concrete concurrently to an application by one or more clients. Each event handlers and register them with the ACE Reactor. service of the application is implemented by a separate event Figure 1 shows the key components in the ACE Reactor. handler that contains one or more methods responsible for This figure depicts concrete event handlers that implement processing service-specific requests. In the implementation of the Reactor pattern described in this paper, event handler dispatching is performed REGISTERED 2: accept() by an ACE Reactor.TheACE Reactor combines OBJECTS 5: recv(request) 3: make_handler() the demultiplexing of input and output (I/O) events with 6: process(request) other types of events, such as timers and signals. At Logging Logging Logging Logging Handler Acceptor LEVEL thecoreoftheACE Reactor implementation is a syn- LEVEL Handler Handler chronous event demultiplexer, such as select [3] or APPLICATION APPLICATION Event Event WaitForMultipleObjects [4]. When the demulti- Event Event Handler Handler Handler plexer indicates the occurrence of designated events, the Handler ACE Reactor automatically dispatches the method(s) of 4: handle_input() 1: handle_input() pre-registered event handlers, which perform application- specified services in response to the events.
    [Show full text]
  • ESWP3 - Embedded Software Principles, Patterns and Procedures Release 1.0
    ESWP3 - Embedded Software Principles, Patterns and Procedures Release 1.0 ESWP3 contributors September 24, 2015 Contents 1 Human Relation Patterns 3 1.1 Categorization of human relation patterns................................3 2 Principles 5 2.1 Categorization of principles.......................................5 2.2 All principles in alphabetic order....................................5 3 Build Patterns 7 3.1 Categorization of build patterns.....................................7 3.2 All build patterns in alphabetic order..................................8 4 Release Antipatterns 11 5 Requirement Patterns 13 5.1 Standardized Textual Specification Pattern............................... 13 5.2 Perform Manual Review Pattern..................................... 13 6 Design Patterns 15 6.1 Categorization of “design” patterns................................... 15 6.2 Pattern Selection Procedure....................................... 21 6.3 Legend to the design pattern sections.................................. 21 6.4 All design patterns in alphabetic order.................................. 21 7 Idioms in C 29 7.1 Classification of idioms......................................... 29 7.2 Add the name space........................................... 29 7.3 Constants to the left........................................... 29 7.4 Magic numbers as variables....................................... 30 7.5 Namend parameters........................................... 30 7.6 Sizeof to variables............................................ 30 8 Unit Test Patterns
    [Show full text]
  • Principles of Software Design Concurrency and Parallelism
    Concurrency and Parallelism Principles of Software Design Concurrency and Parallelism Robert Luko´ka [email protected] www.dcs.fmph.uniba.sk/~lukotka M-255 Robert Luko´ka Concurrency and Parallelism Concurrency and Parallelism Concurrency and Parallelism Concurrency - is the ability of dierent parts or units of a program, algorithm, or problem to be executed out-of-order or in partial order, without aecting the nal outcome. Parallelism - calculations or the execution of processes are carried out simultaneously. Concurrency allows for parallel execution. Robert Luko´ka Concurrency and Parallelism Concurrency and Parallelism Concurrency and Parallelism Concurrency is useful even without parallel computing (it makes sense to use more threads even if we have only one processor): Ecient allocation of resources - while a thread waits for something, another thread may be executed. You do not need threads to get concurrent behavior, see e.g. select system call. You do not know the order on which your code is executed. But it should not matter. Robert Luko´ka Concurrency and Parallelism Concurrency and Parallelism What could possibly go wrong? int etx_rcvd = FALSE; void WaitForInterrupt() { etx_rcvd = FALSE; while (!ext_rcvd) { counter++; } } Robert Luko´ka Concurrency and Parallelism Concurrency and Parallelism What could possibly go wrong? Compiler does obvious optimization. int etx_rcvd = FALSE; void WaitForInterrupt() { while (1) { counter++; } } Robert Luko´ka Concurrency and Parallelism Concurrency and Parallelism Race conditions OK, that was a bit silly, this is a more standard examples (Python, C++): //Example 1 if x == 5: x = x * 2 //Example2 x = x + 1 //Example3 x += 1 //Example4 (C++) x++; Robert Luko´ka Concurrency and Parallelism Concurrency and Parallelism Race conditions Each of the examples can lead to surprising behavior provided that another process can modify x concurrently.
    [Show full text]
  • Stage Scheduling for CPU-Intensive Servers
    UCAM-CL-TR-781 Technical Report ISSN 1476-2986 Number 781 Computer Laboratory Stage scheduling for CPU-intensive servers Minor E. Gordon June 2010 15 JJ Thomson Avenue Cambridge CB3 0FD United Kingdom phone +44 1223 763500 http://www.cl.cam.ac.uk/ c 2010 Minor E. Gordon This technical report is based on a dissertation submitted December 2009 by the author for the degree of Doctor of Philosophy to the University of Cambridge, Jesus College. Technical reports published by the University of Cambridge Computer Laboratory are freely available via the Internet: http://www.cl.cam.ac.uk/techreports/ ISSN 1476-2986 Stage scheduling for CPU-intensive servers Minor E. Gordon Summary The increasing prevalence of multicore, multiprocessor commodity hardware calls for server software architectures that are cycle-efficient on individual cores and can maximize concurrency across an entire machine. In order to achieve both ends this dissertation ad- vocates stage architectures that put software concurrency foremost and aggressive CPU scheduling that exploits the common structure and runtime behavior of CPU-intensive servers. For these servers user-level scheduling policies that multiplex one kernel thread per physical core can outperform those that utilize pools of worker threads per stage on CPU-intensive workloads. Boosting the hardware efficiency of servers in userspace means a single machine can handle more users without tuning, operating system modifications, or better hardware. Contents 1 Introduction 9 1.1 Servers . 9 1.2 Stages . 10 1.3 An image processing server . 10 1.4 Optimizing CPU scheduling for throughput . 11 1.5 Outline . 13 1.5.1 Results .
    [Show full text]
  • Reactive Programming
    Reactive programming Lessons learned Tomasz Nurkiewicz You must be this tall to practice reactive programming [...] a very particular set of skills, skills [...] acquired over a very long career. Skills that make me a nightmare for people like you Liam Neeson on reactive programming Who am I? I built complex reactive systems * not really proud about the “complex” part 1000+ Akka cluster nodes Tens of thousands of RPS on a single node Beat C10k problem ...and C100k I wrote a book with the word “reactive” in the title May you live in interesting times Chinese curse May you support interesting codebase me 1. Fetch user by name from a web service 2. If not yet in database, store it 3. Load shopping cart for user 4. Count total price of items 5. Make single payment 6. Send e-mail for each individual item, together with payment ID User user = ws.findUserByName(name); if (!db.contains(user.getSsn())) { db.save(user); } List<Item> cart = loadCart(user); double total = cart.stream() .mapToDouble(Item::getPrice) .sum(); UUID id = pay(total); cart.forEach(item -> sendEmail(item, id)); User user = ws.findUserByName(name) Mono<User> user = ws.findUserByName(name) boolean contains = db.contains(user.getSsn()) Mono<Boolean> contains = db.contains(user.getSsn()) if (!db.contains(user.getSsn())) { db.save(user); } user -> db .contains(user.getSsn()) //Mono<Bool>, true/false .filter(contains -> contains) //Mono<Bool>, true/empty .switchIfEmpty(db.save(user)) //if empty, //replace with db.save() User user = ws.findUserByName(name); if (!db.contains(user.getSsn())) { db.save(user); } List<Item> cart = loadCart(user); double total = cart.stream() .mapToDouble(Item::getPrice) .sum(); UUID id = pay(total); cart.forEach(item -> sendEmail(item, id)); Now, take a deep breath..
    [Show full text]