A Component-Based Approach for Adaptive Fault Tolerance Miruna Stoicescu, Jean-Charles Fabre, Matthieu Roy

Total Page:16

File Type:pdf, Size:1020Kb

A Component-Based Approach for Adaptive Fault Tolerance Miruna Stoicescu, Jean-Charles Fabre, Matthieu Roy Architecting resilient computing systems: A component-based approach for adaptive fault tolerance Miruna Stoicescu, Jean-Charles Fabre, Matthieu Roy To cite this version: Miruna Stoicescu, Jean-Charles Fabre, Matthieu Roy. Architecting resilient computing systems: A component-based approach for adaptive fault tolerance. Journal of Systems Architecture, Elsevier, 2017, 73, pp.6-16. 10.1016/j.sysarc.2016.12.005. hal-01472877 HAL Id: hal-01472877 https://hal.archives-ouvertes.fr/hal-01472877 Submitted on 21 Feb 2017 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Architecting Resilient Computing Systems: A Component-Based Approach for Adaptive Fault Tolerance Miruna Stoicescu1 Jean-Charles Fabre Matthieu Roy LAAS-CNRS, Université de Toulouse, CNRS, INP, Toulouse, France 1 EUMETSAT (European Organisation for the Exploitation of Meteorological Satellites) ABSTRACT be easily updated through transition packages. Evolution of systems during their operational life is manda- The paper is organized as follows. Section 2 presents the tory and both updates and upgrades should not impair their context and motivation of our work. Section 3.1 briefly de- dependability properties. Dependable systems must evolve scribes the resilient system architecture we consider; next, to accommodate changes, such as new threats and undesir- we describe a set of Fault Tolerance Mechanisms (FTMs) able events, application updates or variations in available and analyze transitions between them in Section 3.2. Sec- resources. A system that remains dependable when facing tion 4 presents the process of designing FTMs for subsequent changes is called resilient. In this paper, we present an inno- adaptation. In Section 4.4, we detail the practical imple- vative approach taking advantage of component-based soft- mentation of FTMs on top of a reflective component-based ware engineering technologies for tackling the on-line adap- support and in Section 5, we describe the implementation tation of fault tolerance mechanisms. We propose a develop- of on-line transitions between FTMs. Next, we evaluate our ment process that relies on two key factors: designing fault approach in Section 6. In Section 7, we discuss related work tolerance mechanisms for adaptation and leveraging a re- and in Section 8 we present the lessons learned and conclude. flective component-based middleware enabling fine-grained control and modification of the software architecture at run- 2. PROBLEM STATEMENT time. We thoroughly describe the methodology, the devel- Resilient computing refers to dependability in the pres- opment of adaptive fault tolerance mechanisms and evaluate ence of changes due to system evolution [5]. Our work fo- the approach in terms of performance and agility. cuses on Fault Tolerance Mechanisms (FTMs) that can be influenced by changes occurring in the system or in its envi- 1. INTRODUCTION ronment. Ideally, fault-tolerant applications consist of two Dependable systems are becoming increasingly complex interconnected abstraction layers. The first one (the base and their capacity to evolve in order to efficiently accommo- level) contains the business logic that implements the func- date changes is a requirement of utmost importance. Changes tional requirements . The second one (the meta level) con- have different origins, such as fluctuations in available re- tains fault tolerant mechanisms and is attached to the first sources, additional features requested by users, environmen- one through clearly identified hooks. As the development tal perturbations. All changes that may occur during ser- of a fault-tolerant application implies different roles/stake- vice life are rarely foreseeable when designing the system. holders (application developer(s), safety expert, integrator), Dependable systems must cope with changes while main- separation of concerns is a key concept. Separation of con- taining their ability to deliver trustworthy services and their cerns has some limit since fault tolerance strategies often required attributes, e.g., availability, reliability, integrity [1]. depend on the semantics of the business logic. However, A lot of effort has been put into building applications that hooks can be defined to externalize some \application de- can adapt themselves to changing conditions. A rich body of fined assertions" used to parameterize fault tolerance mech- research exists in the field of software engineering consisting anisms. In any case, the concept of separation of concerns of concepts, tools, methodologies and best practices for de- is essential to implement adaptive fault tolerance without signing and developing adaptive software [2]. For instance, impacting deeply the business logic. agile software development approaches [3] emphasize the im- Among the well-established and documented FTMs, e.g., portance of accommodating change during the lifecycle of [6, 7], the choice of an appropriate FTM for a given applica- an application at a reasonable cost, rather than striving to tion depends on the values of several parameters. We iden- anticipate an exhaustive set of requirements. Component- tified three classes of parameters: fault tolerance require- based approaches [4] separate service interfaces from their ments, application characteristics and available resources. actual implementation in order to increase flexibility, evolv- • Fault tolerance requirements (FT). This class of pa- ability and reuse. Although dependable systems could ben- rameters contains the considered fault model. A fault model efit from these advancements in order to become more flex- determines the category of FTM to be used (e.g., simple ible and adaptive, very little has been done in this direction replication or diversification). Our fault model classification for now. In this paper, we describe an innovative approach is well-known [1], dealing with crash faults, value faults and for tackling on-line adaptation of dependability mechanisms development faults. We focus on hardware faults (perma- that leverages component-based software engineering tech- nent and transient physical faults) but the approach can be nologies. Component-based fault tolerance mechanisms can extended to other FTMs. 1 • Application characteristics (A). The characteristics tem may encounter throughout its service life and making that have an impact on the choice of an FTM are applica- provisions for them is obviously impossible. tion statefulness, state accessibility and determinism. State This approach is of interest for long-lived space systems accessibility is essential for checkpointing-based fault toler- (satellites and deep-space probes) and for automotive appli- ance strategies. Determinism refers here to behavioural de- cations regarding over-the-air software updates, a very im- terminism, i.e., the same inputs produce the same outputs portant trend in the automotive industry today. in the absence of faults, mandatory for active replication. In this context, we propose an alternative to preprogram- • Resources (R). FTMs require resources in terms of CPU, med adaptation that we denote agile adaptation of FTMs battery life/energy, bandwidth, etc. A cost function can be (the term \agile" is inspired from agile software develop- associated to each FTM based on the values of such pa- ment). Agile adaptation of FTMs enables their system- rameters. For a given set of resources, several mechanisms atic evolution: new FTMs can be designed off-line at any can be used with different trade-offs (e.g., more CPU, less point during service life and integrated on-line in a flexible bandwidth). manner, with limited impact on the existing software archi- tecture. Our approach for the agile adaptation of FTMs The first two parameters FT and A correspond to assump- leverages advancements in the field of software engineer- tions to be considered for the selection of an appropriate ing such as Component-Based Software Engineering (CBSE) mechanism and to determine its validity. The resource di- technologies [4], Service Component Architecture [11] and mension R states the amount of resources needed to accept Aspect-Oriented Programming [12]. Using such concepts a given solution according to system resources availability. and technologies, we design FTMs as brick-based assemblies In practice, based on the values of (FT,A,R) set at design (similar to \Lego" constructions) that can be methodically time, an FTM is attached to an application when the system modified at runtime through fine-grained modifications af- is installed for the first time. As far as resilient computing [5] fecting a limited number of bricks. This approach maximizes is concerned, the challenge lies in maintaining consistency reuse and flexibility, contrary to monolithic replacements of between the FTM(s) and the non-functional requirements FTMs found in related work, e.g., [8, 9, 10]. It is worth despite variations of the parameters at runtime, e.g.: noting that, whatever the approach is (pre-programmed or new threats/faults and physical perturbations such as • agile), when the FTM evolution goes outside
Recommended publications
  • System-Of-Systems Viewpoint for System Architecture Documentation
    System-of-Systems Viewpoint for System Architecture Documentation John Klein∗ and Hans van Vliet† VU University, Amsterdam, Netherlands Revised 11 Nov 2017 Abstract Context: The systems comprising a system of systems (SoS) are inde- pendently acquired, operated, and managed. Frequently, the architecture documentation of these existing systems addresses only a stand-alone perspective, and must be augmented to address concerns that arise in the integrated SoS. Objective: We evaluated an architecture documentation viewpoint to address the concerns of a SoS architect about a constituent system, to support SoS design and analysis involving that constituent system. Method: We performed an expert review of documentation produced by applying the viewpoint to a system, using the active review method. Results: The expert panel was able to used a view constructed using the baseline version of the viewpoint to answer questions related to all SoS architect concerns about a constituent system, except for questions concerning the interaction of the constituent system with the platform and network infrastructure. Conclusions: We found that the expert panel was unable to answer certain questions because the baseline version of the viewpoint had a gap in coverage related to relationship of software units of execution (e.g., processes or services) to computers and networks. The viewpoint was revised to add a Deployment Model to address these concerns, and is included in an appendix. arXiv:1801.06837v1 [cs.SE] 21 Jan 2018 Keywords: architecture documentation; system of systems; viewpoint defi- nition; active review; expert panel; design cycle 1 Introduction A system of systems (SoS) is created by composing constituent systems.
    [Show full text]
  • (2008). Exploring the Impact of Systems Architecture
    Exploring the Impact of Systems Architecture and Systems Requirements on Systems Integration Complexity Dr. Rashmi Jaini, Anithashree Chandrasekaran, George Elias, Dr. Robert Cloutier Stevens Institute of Technology [email protected], [email protected], [email protected], [email protected] Abstract development projects in the government The need to perform faster systems sector. integration of complex systems require the Keywords: System Architecture, System architect and design team understand how Integration, Integration Complexity, System the selected architecture and design Requirements components will impact the system Introduction integration processes complexity. System It is necessary to determine if integration process complexity is an integration of the defined physical elements outcome of the interaction between degree and interfaces in system architecture of feasibility and level of effort required to contributes to system integration process understand, describe, implement, manage complexity. Today’s systems are usually and document the system integration process expected to function independently and in for a given system development and cooperation with the existing systems operational environment. This paper (system of systems). The cooperating analyzes the cause and effect relationships systems undergo frequent changes and between the system requirements, interoperability becomes a key factor and a architecture and the system integration major contributor to system integration processes complexity. In order to address process complexity. Interoperability is based systems integration issues upfront in the on the existence of the conceptual view design phase it is necessary to determine if [Carney and Oberndoff, 2004]. The the architecture and design of components, conceptual view can be embodied in sub-systems, processes and interfaces requirements and architecture/design and the impacts (and to what extent) system system architecture determines the level of integration process complexity.
    [Show full text]
  • Toward an Evolutionary System of Systems Architecture Scott A
    Toward an Evolutionary System of Systems Architecture Scott A. Selberg Mark A. Austin Agilent Technologies, Inc. Institute for Systems Research 1400 Fountain Grove Parkway University of Maryland Mailstop 4USE Col 26HH College Park, MD 20742 Santa Rosa, CA 95403 Copyright © 2008 by Institute for Systems Research. Published and used by INCOSE with permission. Abstract. While recent advances in computing/communications technology have enabled the development and managed evolution of large scale system of systems (SoS) applications, lessons learned from industry indicate that these projects are not always successful. A key problem is the lack of a formal framework from which the development and management of SoS architectures can be studied. This paper describes the key ingredients and approaches to formal analysis that we believe will end up in this architectural infrastructure. Introduction The world is full of systems that are being upgraded from industrial- to information-age capability. System upgrades are commonly driven by the need for enhancements, either by expanding functionality or improving performance. Often, present-day systems are limited by their ability to: (1) sense the surrounding environment in which they are operating, (2) look ahead and anticipate events, and (3) control system behavior. In an effort to relax, or even remove, these constraints, next-generation systems are incorporating advances in computing, sensing, and communications. Sensors gather data which is fed to computers for advanced processing. The enriched information leads to better decision making which in turn is fed back to automated control systems. Through this process, the aforementioned barriers are overcome, thereby providing the desired results. The first important consequence of this trend is that over time, present-day reliance on centralized management of operations will be replaced by operations that are partially or fully decentralized.
    [Show full text]
  • Systems Architecture: Product Designing and Engineering Social
    Systems Architecture: Product Designing and Social Engineering Rebecca E. Grinter Bell Labs, Lucent Technologies 263 Shuman Blvd Naperville, IL 60566 USA [email protected] http://www.bell-labs.com/-beki ABSTRACT structure - of what was being designed. The role of The production of large and complex systems usually architect evolved to fill that gap. requires the coordination and collaboration of many At the same time as the role of architect was being individuals spread among numerous divisions of a established within some corporations, software architecture corporation. However, much research examining emergedas a researcharea. Software architecture researchers coordination has focused on the subtleties of interactions have focusedon a number of topics including architectural between individuals who may work together in the same description languagesfor expressing the design, codifying department. In this paper, I present a study of systems architectural experienceinto principles, and domain-specific architects and the work that they do to coordinate design frameworks with formal methods for reasoning about acrossorganizational and institutional boundaries. I also architectures[ 13,251. describethe processesand tools that the architects use to support their work. The implications of the social While much of the research in software architecture has processesinvolved in coordinating the design of large focusedon the outcome of the process - the architecture complex systemson the product and those involved in its itself - there is an increasing concern being shown for production are discussed. understandinghow architecturework happensin practice [ 1, 131. One reason why there has been a turn to Keywords understanding practice is it is often the relationship between Systems architecture, empirical studies of design and the design and the organization that presents challenges.
    [Show full text]
  • Component-Based Software Architectures: a Framework Based on Inheritance of Behavior W.M.P
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Elsevier - Publisher Connector Science of Computer Programming 42 (2002) 129–171 www.elsevier.com/locate/scico Component-based software architectures: a framework based on inheritance of behavior W.M.P. van der Aalsta;c;d; ∗, K.M. van Heeb;c, R.A. van der Toornb;c aDepartment of Information and Technology, Faculty of Technology Management, Eindhoven University of Technology, P.O. Box513, NL-5600 MB, Eindhoven, Netherlands bDeloitte & Touche Bakkenist, P.O. Box23103, NL-1100 DP Amsterdam, Netherlands cFaculty of Mathematics and Computing Science, Eindhoven University of Technology, P.O. Box513, NL-5600 MB, Eindhoven, Netherlands dDepartment of Computer Science, University of Colorado at Boulder, Campus Box430, Boulder, CO 80309-0430, USA Received 12 April 2000; accepted 19 July 2000 Abstract Software architectures shift the focus of developers from lines-of-code to coarser-grained com- ponents and their interconnection structure. Unlike ÿne-grained objects, these components typi- cally encompass business functionality and need to be aware of the underlying business processes. Hence, the interface of a component should re4ect relevant parts of the business process and the software architecture should emphasize the coordination among components. To shed light on these issues, we provide a framework for component-based software architectures focusing on the process perspective. The interface of a component is described in terms of Petri nets and projection inheritance is used to determine whether a component “ÿts”. Compositionality and substitutability are key issues for component-based development. This paper provides new results to e8ectively deal with these issues.
    [Show full text]
  • Modeling a Software Architecture
    Modeling a Software Architecture Paolo Ciancarini Agenda n Describing software architectures n Architectural frameworks n Models based on architectural languages n Models based on UML n Main architectural views 2 Why document the architecture? In the software life cycle we: n Create an architecture n Using architectural patterns, design patterns, experience n Evaluate the architecture n Using ATAM for example (see lecture on SA evaluation) n Refine, update, and refactor the architecture along the way n Use the architecture to guide the implementation n (Try to) enforce the architecture during the implementation and throughout maintenance 3 Creating a software architecture n The architecture of a software system is closely related to its quality attributes n Architectures allow or preclude nearly all of the system’s quality attributes n Without a proper architecture the quality of a system cannot be ensured or can be highly expensive, or even impossible, to implement 4 Sw architecture and function n Qualities are attributes of the system, while the function is the purpose of the system n Functionality describes what the system does; the quality “functional suitability” describes how well the system does its function n Functional suitability: “the capability of a software product to provide functions which meet stated and implied needs when the software is used under specified conditions” (ISO 25010 Systems and software Quality Requirements and Evaluation) 5 Architecture and structure A software architecture includes multiples structures
    [Show full text]
  • Nato Architecture Framework (Naf)
    NATO ARCHITECTURE FRAMEWORK Version 4 Architecture Capability Team Consultation, Command & Control Board January 2018 Acknowledgments for NAFv4 Publication Throughout the development of version 4 of this publication numerous individual experts of NATO Nations participated, resulting in this significant achievement: The realization of the NATO Architecture Framework. This work would not have been possible without the continuous support of the Ministries of Defence of United Kingdom and France, and the NATO Science and Technology Organization. Also special thanks goes to Partner Nations and Industry Partners for their unwavering support in assigning and providing their best professional resources in the architecture domain. The NATO Architecture Framework is a substantial achievement for the Architecture Capability Team under the Consultation, Command and Control Board. Each member of the Architecture Capability Team worked determinedly over the last four years to provide extensive professional guidance and personal effort in the development of this product. The Architecture Capability Team is grateful to all for their contributions to this effort. 4 NAFv4 NAFv4 5 CONTENTS Chapter 1 - Introduction 1 GENERAL ....................................................................................................................................................................... 11 1.1 Purpose ........................................................................................................................................................................
    [Show full text]
  • Breaking Down the Silos How Systems Architecture Enables Interdisciplinary Collaboration
    WHITE PAPER Breaking Down the Silos How Systems Architecture Enables Interdisciplinary Collaboration Michael Pfenning, Senior Product Manager, Aras 2 Breaking Down the Silos: How Systems Architecture Enables Interdisciplinary Collaboration With the ever-increasing complexities of today’s systems, organizations struggle to address the interdisciplinary collaboration across a product’s entire lifecycle. This is because the collaboration must extend beyond the traditional BOM-driven and mechanically- oriented physical structures. It must embrace functional aspects of the system that are allocated and cross-optimized across multiple disciplines whose data models today reside in individual silos. Systems Engineering has a central role in enabling such interdisciplinary collaboration because it is the only authoritative source of the system’s design intent. The other disciplines rely on that source to implement or to verify that design intent. Systems Engineering creates a formal system definition during the design phase to formalize early decisions. It does that by documenting the system’s requirements, behaviors, and structures. Unfortunately, Systems Engineering today, in most cases, also exists in its own data silo and therefore design-intent driven interdisciplinary collaboration remains unresolved. Figure 1: MBSE systems model sections in SysML Source: http://www.omgsysml.org/what-is-sysml.htm Breaking Down the Silos: How Systems Architecture Enables Interdisciplinary Collaboration 3 From Documents to Models and Data What systems engineers did over decades in the form of documents now needs to be done in a more formal way. Systems engineers have been accustomed to working with Systems Engineering Management Plans, Design Specifications, Concepts of Operation, Functional Flow Block Diagrams (FFBD), and of course spreadsheets, a lot of them—documents a human being may understand but a computer does not.
    [Show full text]
  • Evaluating Enterprise Architecture Frameworks Using Essential Elements Q
    Communications of the Association for Information Systems Volume 41 Article 6 8-2017 Evaluating Enterprise Architecture Frameworks Using Essential Elements Q. Neo Bui Rochester Institute of Technology, [email protected] Follow this and additional works at: https://aisel.aisnet.org/cais Recommended Citation Bui, Q. Neo (2017) "Evaluating Enterprise Architecture Frameworks Using Essential Elements," Communications of the Association for Information Systems: Vol. 41 , Article 6. DOI: 10.17705/1CAIS.04106 Available at: https://aisel.aisnet.org/cais/vol41/iss1/6 This material is brought to you by the AIS Journals at AIS Electronic Library (AISeL). It has been accepted for inclusion in Communications of the Association for Information Systems by an authorized administrator of AIS Electronic Library (AISeL). For more information, please contact [email protected]. ommunications of the C ssociation for nformation ystems A I S Research Paper ISSN: 1529-3181 Evaluating Enterprise Architecture Frameworks Using Essential Elements Quang “Neo” Bui Rochester Institute of Technology [email protected] Abstract: Enterprise architecture (EA) frameworks offer principles, models, and guidance to help one develop an EA program. Due to EA’s flexible and abstract nature, there is a proliferation of EA frameworks in practice. Yet, comparison studies to make sense of them are far from satisfactory in that they lack a theoretical foundation for comparison criteria and do not meaningfully interpret the differences. In this paper, I propose a comparison approach using EA essential elements—the underlying key features of EA programs—to distinguish EA frameworks. Based on the extant literature, I identify eight elements, each with its own theoretical justification and empirical evidence.
    [Show full text]
  • Job Description – System Architect/Manager Title
    Job Description – System Architect/Manager Title: System Architect/Manager Immediate Supervisor: Director of Technology Services Description The System Architect/Manager’s role is to strategically design and implement in-house information systems and networked software architectures that support core County functions, and assure their high availability. This individual will gain organizational commitment for all systems and software plans, as well as evaluate and select all technologies required to complete those plans. In addition, the System Architect/Manager provides technical leadership across the County from strategic decision making down to the project planning level. They will work closely with decision makers in other departments to identify, recommend, develop, implement, and support cost-effective technology solutions for all aspects of the County. This person will also define and implement IT policies, procedures, and best practices. Responsibilities Strategy & Planning Design and implement long-term strategic goals and short-term tactical plans for managing and maintaining County systems and software. Ensure that proposed and existing systems architectures are aligned with County goals and objectives. Provide architectural expertise, direction, and assistance to Systems Analysts, Systems Engineers, other Systems Architects, and software development teams. Develop, document, and communicate plans for investing in systems architecture, including analysis of cost reduction opportunities. Conduct research on emerging technologies
    [Show full text]
  • Systems Engineering and Architecture
    Systems Engineering and Architecture Richard M. Murray Control and Dynamical Systems California Institute of Technology Design Principles in Biological Systems 21 April 2008 Product Systems Engineering Systems engineering methodology • requirements capture and analysis • systems architecture and design • functional analysis • interface design and specification • communications protocol design & specs • simulation and modeling • verification and validation fault modeling Boeing 737: first flight, April 1967 • PDP-8: debuted 1965 Banbury, May 2007 Richard M. Murray, Caltech CDS 2 Systems of Systems Engineering Little centralized control over the design • Individual systems build for specific purpose • No global requirements document + evolution Example: air operations center (think ATC) • Multiple aircraft, designed over the last 50 years (with lots of variations in capabilities) • Ground control stations + imagery analysis design to run independent of AOC • All running on COTS computers, networks Layer 4 Inter-layer interfaces Layer 3 Layer 2 Layer 1 Layer 1 Banbury, May 2007 Richard M. Murray, Caltech CDS 3 The Role of Architecture How do we define architecture? • IEEE: “The fundamental organization of a system embodied in its components, their relationships to each other, and to the environment, and the principles guiding its design and evolution.” • Doyle (following Gerhart and Kirschner): “The constraints that deconstrain” • Partha (following from building architecture): Integration of structure and function Some useful concepts • Functional decomposition: how do we break down a system into functionally independent modules • Interfaces and standards: how to we specify consistent Protocols interfaces that let us integrate functional modules • Protocols: how do we build layered abstractions that allow designers to ignore the details above and below Interfaces Banbury, May 2007 Richard M.
    [Show full text]
  • Architectural View Model for Cloud Applications
    The (5+1) Architectural View Model for Cloud Applications Mohammad Hamdaqa, Ladan Tahvildari Software Technologies Applied Research (STAR) Group Department of Electrical and Computer Engineering University of Waterloo, Canada fmhamdaqa, [email protected] Abstract Existing software architecture frameworks focus on application development, rather than the dy- namic evolution of applications at runtime. Their view models reflect design considerations, more than service operations. However, the quality of a cloud application depends on its configuration and the architecture of its service model. For this Figure 1: Architecting for Application Implemen- reason, we need a view model that is constructed tation v.s. Service Operation around deployment. This paper proposes a (5+1) architectural view model, where each view corre- sponds to a different perspective on cloud applica- and leverage portability and scalability. tion deployment. The (5+1) view model has been View modeling is a well-established prac- realized as a layered, domain specific modeling tice. Existing frameworks include the (4+1) view language, and the capabilities of this language have model [25], the Three Schema approach [23], the been illustrated using a representative domain ex- Zachman framework [34] and the Department of ample. The model was derived by investigating the Defense framework [13]. These frameworks all process of architecting cloud applications, and then clarify software design, because that is when ar- providing a set of meta-models to describe cloud chitectural decisions were traditionally finalized. applications within their ecosystem. Static deployment infrastructures were the norm. In cloud computing however, architecture evolves 1 Introduction during deployment. Runtime operation needs as much architectural modeling as design does [31].
    [Show full text]