D2.1 Unicorn Libraries, IDE Plugin, Container Packaging and Deployment Toolset Early Release Unicorn Libraries, IDE Plugin, Container Packaging and Deployment Toolset Early Release Deliverable D2.1 Editors Athanasios Tryfonos Demetris Trihinas Reviewers Sotiris Koussouris (SUITE5) John Samuelsson, Erik Robertson (REDIKOD) Date 30 March 2018 Classification Public 1 D2.1 Unicorn Libraries, IDE Plugin, Container Packaging and Deployment Toolset Early Release Contributing Author # Version History Name Partner Description Table of Contents (ToC) and partner Demetris Trihinas UCY 1 contribution assignment. Athanasios Tryfonos UCY 2 Introduction of Section 5 and subsection 5.2 Finalized state-of-the-art and key technologies Zacharias Georgiou UCY 3 challenges and updates section. Initial content for subsection 5.1 has been George Pallis UCY 4 added. Julia Vuong CAS 5 Finalized Section 1 Monitoring and Elasticity, Perimeter Security Manos Papoutsakis FORTH 6 and Vulnerability Assessment models are added Giannis Ledakis UBITECH 7 Deliverable ready for review. Addressed comments from reviewers and 8 compiled final version ready for submission 2 D2.1 Unicorn Libraries, IDE Plugin, Container Packaging and Deployment Toolset Early Release Contents CONTENTS 3 1 INTRODUCTION 9 1.1 Document Purpose and Scope 10 1.2 Document Relationship with other Project Work Packages 11 1.3 Document Structure 11 2 STATE OF THE ART AND KEY TECHNOLOGY AXES CHALLENGES AND UPDATES 12 2.1 OASIS TOSCA Specification 12 2.2 Docker Container Technology Stack 13 2.3 Service Mesh and Container Design Patterns 15 3 UNICORN DESIGN LIBRARIES 18 3.1 Overview 18 3.2 Monitoring and Elasticity 18 3.2.1 Design Library Overview 18 3.2.2 Design Library Model Description 20 3.3 Perimeter Security and Risk & Vulnerability Assessment 23 3.3.1 Design Library Overview 23 3.3.2 Design Library Model Description 26 3.4 Privacy and Authorization 28 3.4.1 Design Library Overview 28 3.4.2 Design Library Model Description 30 4 DESIGN LIBRARIES IMPLEMENTATION 32 4.1 Design Libraries Implementation Aspects 32 4.2 Unicorn Service Graph Representation 33 4.3 Unicorn Compose File 36 4.3.1 Unicorn Compose Descriptor Interpreter 37 4.4 Unicorn Enabled Container Packaging 38 4.5 Unicorn Design Libraries Features 40 4.5.1 Monitoring and Elasticity Design Libraries Features 40 4.5.2 Perimeter Security Design Library Implementation 41 4.5.3 Risk and Vulnerability Assessment Implementation 41 4.5.4 Privacy and Authorization Implementation 43 5 UNICORN DASHBOARD 43 5.1 Unicorn Management Perspective 44 5.1.1 Entities and Entity Relationship in Unicorn Dashboard 44 5.1.2 Technical Approach 46 5.1.3 Provided Functionalities 47 5.1.4 Exposed APIs 51 5.2 Unicorn Development Perspective 56 6 CONCLUSION 60 3 D2.1 Unicorn Libraries, IDE Plugin, Container Packaging and Deployment Toolset Early Release 7 REFERENCES 61 8 ANNEX 62 8.1 Unicorn Enhanced Docker-Compose Model Specification 62 8.1.1 The format and the specification of the Unicorn Docker-Compose model 62 8.1.2 Unicorn Compose File Extended Example 70 4 D2.1 Unicorn Libraries, IDE Plugin, Container Packaging and Deployment Toolset Early Release List of Figures Figure 1 Unicorn Reference Architecture 10 Figure 2 Structural Elements of a Service Template and their relationships 12 Figure 3 Abstract view of the Docker Engine 13 Figure 4 Docker Swarm can be swapped out with different container orchestration technologies such as Kubernetes or Mesos 14 Figure 5 Docker-Compose that consists of 3 services; i) redis service which is bound onto frontend network, ii) db service that mounts a volume on the host machine called db-data and bound on the backend network and iii) a vote service bound on frontend network 14 Figure 6 x-custom special field definition. This field is not in the semantics of docker-compose language specification, however with its definition it is being ignored or handled in a different manner by docker- compose. 15 Figure 7 Example of a sidecar container that syncs the file system with a git repository 15 Figure 8 Example illustrating the Ambassador design pattern 16 Figure 9 Example illustrating the Adapter design pattern 16 Figure 10 Istio Architecture 17 Figure 11: Elasticity Policy Example 19 Figure 12 The security model conforming to the abstract syntax of the security meta-model 25 Figure 13 ABAC Model for Unicorn 30 Figure 14 Maven Import of Unicorn Monitoring Library 32 Figure 15 Timeit annotation that calculates the time needed for a method to be executed 32 Figure 16 Example of Unicorn Monitoring Annotation 33 Figure 17 Simple Service Graph with properties 34 Figure 18 Representation of Service Graph as Tosca YAML 35 Figure 19 The Unicorn Descriptor Interpreter 38 Figure 20 Example Dockerfile that builds a Unicorn parent image based on Ubuntu 16.04 40 Figure 21 The Dashboard layer divided into two perspectives. 44 Figure 22. Entities in Unicorn Dashboard. 45 Figure 23. Entity Relationship in Unicorn Dashboard. 46 Figure 24 Eclipse Che Architecture 57 Figure 25 Unicorn Che IDE Plugin High-Level Architecture 57 Figure 26 Unicorn Stack Configuration for Che 59 List of Tables Table 1 Metric EBNF Definition 20 Table 2 Monitoring Agent EBNF Definition 20 Table 3 Insight EBNF Definition 21 Table 4 Elasticity Policy EBNF Definition 22 Table 5 Elasticity Trigger EBNF Definition 22 Table 6 Elasticity Action EBNF Definition 23 Table 7 Perimeter Security and Risk & Vulnerability Assessment EBNF 26 Table 8 EBNF Definition of the ABAC model used in Unicorn 30 Table 9 Docker container requirements for Unicorn enabled microservices 39 Table 10 Monitoring and Elasticity Table of Features 40 Table 11 Perimeter Security Table of Features 41 Table 12 Risk and Vulnerability Assessment Table of Features 41 Table 13 Privacy and Authorization Table of Features 43 Table 14 User Registration Functionality 47 Table 15 Login/Logout Functionality 47 Table 16 Delete User Functionality 48 5 D2.1 Unicorn Libraries, IDE Plugin, Container Packaging and Deployment Toolset Early Release Table 17 Update User Profile Functionality 48 Table 18 Change Authorization Details Functionality 48 Table 19 Register a new Usergroup Functionality 48 Table 20 Update an existing usergroup Functionality 48 Table 21 Start a Unicorn-compliant app Functionality 49 Table 22 Stop an application Functionality 49 Table 23 Delete application Functionality 49 Table 24 Deploy Unicorn Compliant Application Functionality 49 Table 25 Get Monitoring Data Functionality 50 Table 26 Update Runtime Policies 50 Table 27 Update Service Graph Functionality 50 Table 28 Cloud Offering Marketplace Functionality 50 Table 29 Generate Service Graph Functionality 51 Table 30 Update Service Graph Functionality 51 Table 31 Launch Unicorn IDE Plugin 51 Table 32 Update Usergroup API 52 Table 33 Delete Usergroup by ID API 52 Table 34 Update User API 52 Table 35 Delete User API 53 Table 36 Get all Users API 53 Table 37 Get User by ID API 53 Table 38 Enable Registered user API 54 Table 39 Disable User API 54 Table 40 Create new Usergrup 54 Table 41 Register user API 55 Table 42 Login API 55 Table 43 Update Credentials API 56 6 D2.1 Unicorn Libraries, IDE Plugin, Container Packaging and Deployment Toolset Early Release Executive Summary The purpose of this deliverable is to report the progress made for the ongoing work regarding workpackage 2 and more specifically to document the early release of the Unicorn Libraries, IDE Plugin, Container Packaging and Deployment Toolset. The deliverable begins by presenting an updated state-of-the-art and key technology axes section. We identify and analyze technologies that are introduced for the first time within Unicorn and identify challenges and potential obstacles that current technologies may impose during the implementation and integration phases. More specifically, we introduce the concepts of sidecar architecture which is a multi-container architectural pattern that extends the functionality of a main container with additional containers, without performing any changes to its structure and functionality. Also, we present leading technologies that adhere to the sidecar architecture, such as Google’s Istio and Envoy Proxy, which will be used in the implementation of Unicorn. We elaborate on the concept of the service graph by adopting the OASIS Tosca specification, since the interrelation between nodes and edges of a graph can be mapped to structural components of the Tosca Specification. The final focus on this first section of the deliverable is on Docker and its technology stack and various tools that play an important role in Unicorn. Unicorn Design Libraries, assist developers to design cloud applications as Unicorn-enabled micro-services, with the minimum code intrusion that is required to enable portable security enforcement mechanisms, monitoring and elastic scaling, and to ensure data privacy constraints. To this end, Unicorn offers four design libraries namely i) Monitoring and Elasticity, ii) Perimeter Security, iii) Risk and Vulnerability Assessment and iv) Privacy and Authorization design libraries; their meta-models and language are presented in EBNF format later on Section 3. It should be noted that due to similarities and interdependencies of the concepts of the aforementioned design libraries, some are presented and represented as a single library, i.e. Monitoring and Elasticity and Perimeter Security with Risk and Vulnerability assessment. Minimum code intrusion in the development of Unicorn-enabled microservices is achieved by annotating specific bits of code, methods or variables with decorators that provide additional functionality, without being a necessity for developer to have deep knowledge of the annotating concepts. Even though deep knowledge of the concepts is not required, developers should at least provide initial policies and feature configuration that match their requirements for the developed microservice. For this purpose, Section 4 introduces a table with the features and policies that each design library exposes as functionality and at which stage of the continuous development and integration pipeline this feature applies to (it is either at design-time, using annotations or at run-time through configurations at the dashboard).