A Modular and Open Software and Hardware Architecture for Internet of Things Sensor Networks

Total Page:16

File Type:pdf, Size:1020Kb

A Modular and Open Software and Hardware Architecture for Internet of Things Sensor Networks UNIVERSITY OF SOUTHAMPTON FACULTY OF ENGINEERING AND PHYSICAL SCIENCES A Modular and Open Software and Hardware Architecture for Internet of Things Sensor Networks by Mihaela Apetroaie-Cristea Supervisor: Prof. Simon J. Cox and Dr. Steven J.J. Ossont June 25, 2020 iii UNIVERSITY OF SOUTHAMPTON ABSTRACT FACULTY OF ENGINEERING AND PHYSICAL SCIENCES A Modular and Open Software and Hardware Architecture for Internet of Things Sensor Networks by Mihaela Apetroaie-Cristea We are experiencing a new industrial revolution, a cyber-physical systems revolution. The Internet of Things (IoT) is at the core of this development, and it is changing the way we perceive the world around us, impacting both business and technology. Smart cars, smart thermostats, home automation hubs, smart lighting, and smart weather stations are common technologies met in most cities and households. The number of these devices is going to increase even more, and it is predicted to reach billions over the next few years. Although the IoT technology has reached maturity, we are still unprepared for the large number of devices predicted to reach the world in the near future. Managing high numbers of IoT devices requires stable infrastructures for deployment and manage- ment. It also requires creating sustainable infrastructures to minimise the impact on the environment. In this thesis, we hypothesise that using flexible, open, modular, and reconfigurable hardware and software architectures at the base of smart city infrastructure can increase device longevity and minimise device management complexity at scale, while promot- ing sustainable development. The main contributions are: (1) identification of design requirements for building the next generation IoT device (2) reference architecture for flexible, modular IoT devices, (3) a novel, modular, open-source sensor board for build- ing plug-and-play smart devices, allowing for complete removal/replacement of sensors and computing module, (4) a novel, modular, open-source software architecture, includ- ing: minimal Operating System (OS), over the air (OTA) updates, containerisation and remote device management, and (5) demonstration using a real-life application of en- vironmental monitoring. The reference architecture presented in this thesis provides a robust, persistent, and reliable long-term solution for IoT deployements that addresses concerns regarding the negative impact of IoT on long term sustainable development. Contents Acknowledgements xi Declaration of Authorship xiii 1 Introduction1 1.1 Thesis structure.................................3 2 The Internet of Things in our lives7 2.1 Towards ubiquitous computing........................8 2.1.1 Sustainability..............................8 2.1.2 Internet of Things to address global goals..............8 2.1.3 Internet of Things in our cities....................9 2.2 Internet of Things hardware technologies................... 12 2.2.1 Single Board Computers........................ 12 2.2.2 Sensors................................. 12 2.3 Enabling technologies............................. 14 2.3.1 Open Sensors Platforms........................ 14 2.3.2 Scalability................................ 15 2.3.3 Networking............................... 18 2.3.4 Operating Systems and Unikernels.................. 19 2.3.5 Over the air updates.......................... 22 2.3.6 Containerisation............................ 24 2.4 Internet of Things applications........................ 26 2.4.1 Air Quality monitoring......................... 26 2.4.2 Plug and Play platforms........................ 27 2.5 Research Questions and Objectives...................... 29 3 Indoor positioning system 33 3.1 Indoor localisation system based on low-cost commodity hardware.... 34 3.1.1 Introduction............................... 34 3.1.2 Design of the Indoor Localisation System.............. 35 3.1.3 Preliminary Testing.......................... 36 3.1.3.1 Methods........................... 36 3.1.3.2 Results............................ 37 3.2 Further assessment of the system performance................ 38 3.2.1 Methods................................. 39 3.2.2 Discussion and analysis of the results................. 39 3.3 Conclusions and future work.......................... 45 v vi CONTENTS 4 Generic IoT device for an environmental monitoring use case 49 4.1 Development.................................. 49 4.2 Theoretical use case - Southampton Smart City............... 50 4.3 Air quality monitoring............................. 52 4.4 Preliminary testing............................... 53 4.4.1 Implementation............................. 53 4.4.2 Data................................... 54 4.4.3 Security and Privacy.......................... 54 4.4.4 Outreach................................ 57 4.5 Lessons learned................................ 58 5 Hardware architecture 61 5.1 Introduction................................... 61 5.2 Motivation.................................... 61 5.3 Proposed Solution............................... 62 5.4 Environmental monitoring example...................... 64 5.5 Face validation - results and conclusions................... 65 5.6 Known issues and limitations......................... 69 6 Software architecture 71 6.1 Introduction................................... 71 6.2 Requirements.................................. 73 6.3 Operating Systems............................... 73 6.4 Remote device management.......................... 75 6.5 Over the air updates.............................. 75 6.5.1 OSTree................................. 76 6.6 Containerisation................................ 78 6.6.1 Applications in containers....................... 79 6.7 Server-side requirements and considerations................. 79 6.8 Power...................................... 80 6.9 Implementation details - other technologies................. 81 6.10 Face validation - results and discussion.................... 82 6.10.1 Implementation details......................... 82 6.10.2 Experiments............................... 83 6.10.3 Security and privacy.......................... 86 6.10.4 Known issues and limitations..................... 87 7 Conclusions 91 8 Future work 95 A PiSEB v1.0 Eagle Schematics 99 B PiSEB v2.0 Eagle Schematics 109 References 119 List of Figures 1.1 Hype Cycle for Internet of Things [148]....................2 1.2 IoT continous development..........................3 1.3 Proposed hardware and software modular architecture, where S represents a sensor.....................................5 2.1 The United Nations Global Goals adopted in 2015 [54]...........8 2.2 BalenaOS Architecture............................. 21 2.3 Containarisation software architecture.................... 24 2.4 Sandboxing software architecture....................... 25 3.1 Locator node.................................. 35 3.2 Section of the office area............................ 37 3.3 Testing area layout............................... 37 3.4 Graph illustrating the error variation at 17 measurement points....... 38 3.5 The error variation when three locator nodes have been considered.... 38 3.6 Variation of the position estimation errors at the measurement points for the indoors experiment............................. 40 3.7 Variation of the position estimation errors at the measurement points for the outdoors experiment............................. 40 3.8 Variation of the error with ideal solution scoring for the indoor measurements 41 3.9 Variation of the error with ideal solution scoring for the park measurements 42 3.10 Represenation of the solution interval as the intersection between the rings to include the signal strength noise parameter ∆ .............. 43 3.11 Radiation pattern of the omnidirectional, high-gain antenna used for this project [143].................................. 44 4.1 Map of Southampton.............................. 51 4.2 Device architecture for the preliminary test................. 53 4.3 Preliminary test SO2 and NO2 data...................... 55 4.4 Preliminary test PM2.5 data.......................... 55 4.5 Preliminary test temperature data....................... 56 4.6 Preliminary test pressure data......................... 56 4.7 Example of the charts used in the preliminary testing............ 58 5.1 Reference hardware modular architecture, where S represents a sensor.. 63 5.2 PiSEB development board [12] designed as part of this research, where the continous line delimitates the microcontroller (pycom) and SBC (Rasp- berry Pi) interfaces............................... 63 5.3 The hardware used for the environmental sensor prototype......... 65 vii viii LIST OF FIGURES 5.4 Reference hardware modular architecture, where S represents a sensor.. 67 6.1 Reference modular architecture showing the main technologies used for our software implementation.......................... 73 List of Tables 2.1 Gas sensors costs and performance metrics.................. 15 3.1 Cost and accuracy of top indoor positioning systems on the market.... 35 3.2 Price of locator node............................. 35 3.3 Requirements.................................. 47 4.1 The sensors chosen for the first version of the generic IoT device..... 50 4.2 Methods for achieving the design requirements - customisation, expansion, maintenance and accessibiliy- hw represents hardware and sw represents software..................................... 60 5.1 The sensors chosen for the final version of the generic IoT device..... 66 5.2 Hardware face validation results.......................
Recommended publications
  • GNU Guix Cookbook Tutorials and Examples for Using the GNU Guix Functional Package Manager
    GNU Guix Cookbook Tutorials and examples for using the GNU Guix Functional Package Manager The GNU Guix Developers Copyright c 2019 Ricardo Wurmus Copyright c 2019 Efraim Flashner Copyright c 2019 Pierre Neidhardt Copyright c 2020 Oleg Pykhalov Copyright c 2020 Matthew Brooks Copyright c 2020 Marcin Karpezo Copyright c 2020 Brice Waegeneire Copyright c 2020 Andr´eBatista Copyright c 2020 Christine Lemmer-Webber Copyright c 2021 Joshua Branson Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled \GNU Free Documentation License". i Table of Contents GNU Guix Cookbook ::::::::::::::::::::::::::::::: 1 1 Scheme tutorials ::::::::::::::::::::::::::::::::: 2 1.1 A Scheme Crash Course :::::::::::::::::::::::::::::::::::::::: 2 2 Packaging :::::::::::::::::::::::::::::::::::::::: 5 2.1 Packaging Tutorial:::::::::::::::::::::::::::::::::::::::::::::: 5 2.1.1 A \Hello World" package :::::::::::::::::::::::::::::::::: 5 2.1.2 Setup:::::::::::::::::::::::::::::::::::::::::::::::::::::: 8 2.1.2.1 Local file ::::::::::::::::::::::::::::::::::::::::::::: 8 2.1.2.2 `GUIX_PACKAGE_PATH' ::::::::::::::::::::::::::::::::: 9 2.1.2.3 Guix channels ::::::::::::::::::::::::::::::::::::::: 10 2.1.2.4 Direct checkout hacking:::::::::::::::::::::::::::::: 10 2.1.3 Extended example ::::::::::::::::::::::::::::::::::::::::
    [Show full text]
  • Introduction to the Nix Package Manager
    Introduction Nix concepts Usage examples Conclusion Introduction to the Nix Package Manager Millian Poquet 2021-05-12 — Datamove (Inria) seminar 1 / 16 Introduction Nix concepts Usage examples Conclusion Why Nix? Control your software environment! Programs/libraries/scripts/configurations + versions Why is it important for us? Use/develop/test/distribute software Manually install many dependencies? No, just type nix-shell Shared env for whole team (tunable) and test machines Bug only on my machine? Means this is hardware or OS related Reproducible research Repeat experiment in exact same environment Introduce or test variation 2 / 16 Introduction Nix concepts Usage examples Conclusion What is Nix? Nix: package manager Download and install packages Shell into well-defined environment (like virtualenv) Transactional (rollback works) Cross-platform: Linux, macOS, Windows (WSL) Nix: programming language Define packages Define environments (set of packages) Functional, DSL NixOS: Linux distribution Declarative system configuration Uses the Nix language Transactional (rollback still works) 3 / 16 Introduction Nix concepts Usage examples Conclusion Nix in numbers Started in 2003 Nix 1: 10k commits, 28k C++ LOC Nixpkgs 2: 285k commits, 55k packages 3 1. https://github.com/NixOS/nix 2. https://github.com/NixOS/nixpkgs 3. https://repology.org/repositories/statistics 4 / 16 Introduction Nix concepts Usage examples Conclusion Presentation summary 2 Nix concepts 3 Usage examples 4 Conclusion 5 / 16 Introduction Nix concepts Usage examples Conclusion Traditional
    [Show full text]
  • CDE: Run Any Linux Application On-Demand Without Installation
    CDE: Run Any Linux Application On-Demand Without Installation Philip J. Guo Stanford University [email protected] Abstract with compiling, installing, and configuring software and their myriad of dependencies. For example, the official There is a huge ecosystem of free software for Linux, but Google Chrome help forum for “install/uninstall issues” since each Linux distribution (distro) contains a differ- has over 5800 threads. ent set of pre-installed shared libraries, filesystem layout In addition, a study of US labor statistics predicts that conventions, and other environmental state, it is difficult by 2012, 13 million American workers will do program- to create and distribute software that works without has- ming in their jobs, but amongst those, only 3 million will sle across all distros. Online forums and mailing lists be professional software developers [24]. Thus, there are are filled with discussions of users’ troubles with com- potentially millions of people who still need to get their piling, installing, and configuring Linux software and software to run on other machines but who are unlikely their myriad of dependencies. To address this ubiqui- to invest the effort to create one-click installers or wres- tous problem, we have created an open-source tool called tle with package managers, since their primary job is not CDE that automatically packages up the Code, Data, and to release production-quality software. For example: Environment required to run a set of x86-Linux pro- grams on other x86-Linux machines. Creating a CDE • System administrators often hack together ad- package is as simple as running the target application un- hoc utilities comprised of shell scripts and custom- der CDE’s monitoring, and executing a CDE package re- compiled versions of open-source software, in or- quires no installation, configuration, or root permissions.
    [Show full text]
  • Functional Package Management with Guix
    Functional Package Management with Guix Ludovic Courtès Bordeaux, France [email protected] ABSTRACT 1. INTRODUCTION We describe the design and implementation of GNU Guix, a GNU Guix1 is a purely functional package manager for the purely functional package manager designed to support a com- GNU system [20], and in particular GNU/Linux. Pack- plete GNU/Linux distribution. Guix supports transactional age management consists in all the activities that relate upgrades and roll-backs, unprivileged package management, to building packages from source, honoring the build-time per-user profiles, and garbage collection. It builds upon the and run-time dependencies on packages, installing, removing, low-level build and deployment layer of the Nix package man- and upgrading packages in user environments. In addition ager. Guix uses Scheme as its programming interface. In to these standard features, Guix supports transactional up- particular, we devise an embedded domain-specific language grades and roll-backs, unprivileged package management, (EDSL) to describe and compose packages. We demonstrate per-user profiles, and garbage collection. Guix comes with a how it allows us to benefit from the host general-purpose distribution of user-land free software packages. programming language while not compromising on expres- siveness. Second, we show the use of Scheme to write build Guix seeks to empower users in several ways: by offering the programs, leading to a \two-tier" programming system. uncommon features listed above, by providing the tools that allow users to formally correlate a binary package and the Categories and Subject Descriptors \recipes" and source code that led to it|furthering the spirit D.4.5 [Operating Systems]: Reliability; D.4.5 [Operating of the GNU General Public License|, by allowing them to Systems]: System Programs and Utilities; D.1.1 [Software]: customize the distribution, and by lowering the barrier to Applicative (Functional) Programming entry in distribution development.
    [Show full text]
  • Reproducible Builds Summit II
    Reproducible Builds Summit II December 13-15, 2016. Berlin, Germany Aspiration, 2973 16th Street, Suite 300, San Francisco, CA 94103 Phone: (415) 839-6456 • [email protected] • aspirationtech.org Table of Contents Introduction....................................................................................................................................5 Summary.......................................................................................................................................6 State of the field............................................................................................................................7 Notable outcomes following the first Reproducible Builds Summit..........................................7 Additional progress by the reproducible builds community......................................................7 Current work in progress.........................................................................................................10 Upcoming efforts, now in planning stage................................................................................10 Event overview............................................................................................................................12 Goals.......................................................................................................................................12 Event program........................................................................................................................12 Projects participating
    [Show full text]
  • Unable to Require Openssl Install Openssl
    Unable To Require Openssl Install Openssl Maurits horse-collar her wienies sloppily, she synthetising it manifestly. Cy jutes her largo smart, existentialist and cuter. Garp is uninvolved and misaddressed oversea as tinned August frightens toploftily and rewrite transcontinentally. Tell me to install, right pieces to 1525565 openssl-devel and compat-openssl10-devel are. After that requires to install and installed. A new openssl11 version was installed and about I am unable to. Something basic knowledge within a comment to openssl library. How can enjoy use ruby gem commands like bundler when ruby is installed by nix package manager? Unable to require openssl is driving me the gem 203. Watch for installing requirements for in to require openssl installed the installation will not start openssl version if he refuses to uninstall the certificate. In install with solutions and requires the installer exits, navigate to require that software into the sdk itself to rbenv solved all web. Successful exploitation could survive to a security bypass screw where an attacker could gain praise to potentially sensitive information. Also be pretty hard to distribute dpkg packages are unable to it which i edit your trusted root. Scrap the installation and world over? Installing PowerShell on macOS PowerShell Microsoft Docs. Now i expect it can you are unable to the requirements for installing for detailed explanation with a pull request may close the files from source. Any suggestion as to however this? While pride can't infer much about her yet-to-be-identified bugs you charge at. Is to install location that requires to work in this? Keys saved to disk without encryption are now secure from anyone who gets ahold of the fork may use gas unless mistake is encrypted.
    [Show full text]
  • A Deep Dive Into Nixos: from Configuration to Boot CS5250: Advanced Operating Systems
    A Deep Dive into NixOS: From Configuration To Boot CS5250: Advanced Operating Systems Chen Jingwen A0111764L National University of Singapore Abstract Mature operating systems (e.g. Windows, Fedora) are inherently stateful and imperative, adding layers of complexity by installing or upgrading software. This causes side-effects such as breaking existing software while upgrading shared libraries without maintaining backwards compatibility. NixOS is a Linux distribution designed to be purely functional, where building everything from the kernel to the web browser has no side- effects. System configuration files are written in the Nix language, a lazy functional domain specific language with a declarative syntax, and software packages are managed by the Nix package manager. A distinct feature of NixOS is the ability to declare the configuration of an entire system in one file, which is then used to build a bootable system deterministically. This report gives an overview and the motivations of NixOS, and a deep dive into how the configuration of an operating system can be derived from a single file. 1 Contents 1 Introduction 4 2 Motivation 5 2.1 Multiple versions . 5 2.2 Destructive updates . 5 2.3 Rollback difficulties . 6 2.4 Non-atomic upgrades . 6 2.5 Inability to reproduce builds . 6 3 NixOS Architecture 7 3.1 Package specifications and the Nix expression language . 7 3.1.1 Nix expression language . 8 3.1.2 Derivations . 9 3.2 Nix store . 9 3.2.1 Cryptographic hash . 9 3.2.2 Source to binary deployment . 10 3.2.3 Nix database . 10 3.3 Nix package manager .
    [Show full text]
  • Nix(OS) - Revolutionizing Packaging and Configuration Management!
    Nix(OS) - Revolutionizing packaging and configuration management! The Purely Functional Linux Distribution 1 Before we begin (FYI) Ask questions at any time Please ask lots of questions :) The slides contain some redundancy There are a few optional slides at the end Please give me feedback Louder Faster/slower More/less details Etc. 2 About me Michael Weiss aka. primeos Computer science student at the University of Tübingen I love free soware, etc. First nixpkgs commit: 2016-10-05 I maintain ~41 packages and ~3 modules (2018-06-08) I also love privacy (i.e. no more details :P) Email: [email protected] (#privacy) 3 Main components Nix (package manager) Nixpkgs (Nix packages collection) NixOS (operating system) NixOps (DevOps / cloud deployment tool) 4 Nix* ISO/OSI model NixOps NixOS Nixpkgs Nix 5 Other tools Hydra (Nix based continuous build system) Disnix (distributed services deployment) PatchELF (change dynamic linker and RPATH) {cabal,go,node,pip,python,pypi,composer,hex,bower,vim,...}2 6 History Started as a research project (with funding) First paper in 2004 (many will follow) Nix package manager developed by Eelco Dolstra as part of his PhD research (~2003) First NixOS prototype developed by Armijn Hemel as his master's thesis project Hydra developed as part of the LaQuSo Buildfarm project 7 Timeline 2003: init (research begins) 2007: NixOS becomes usable + x86_64 support 2008: Website moved to nixos.org 2009: Nix logo + Nix(OS) build on Hydra 2011: Migration from Subversion to Git(Hub) 2013: Switch from Upstart to systemd +
    [Show full text]
  • Copyrighted Material
    1 WHAT ’ S IN THIS CHAPTER? ➤ Installing and getting started with Visual Studio Code ➤ Understanding the cross-platform components that make up Visual Studio Code GETTING STARTED The choice of the editor used by any developer is an incredibly personal one. The reason to pick one over the rest depends on a collection of attributes typically related to the tasks they perform on a daily basis. Developers look for functionality, keystroke shortcuts, code snippets, colora- tions, and more that allow them to stay productive. Dislodging developers from their choice is not easy. Any change in editors is going to result in an immediate loss of productivity. After all, it takes time to become familiar with the features offered and have them become a natural part of the coding “flow.” As a result, it takes a special level of “better” for a developer to switch editors. For this reason, the success of Visual Studio Code speaks volumes for its features and function- ality. Although it has been officially released for just three years (it left public preview in April 2016), it has quickly become one of the top editors in terms of popularity, competing with Sublime Text, Atom,COPYRIGHTED and UltraEdit for the top spot. MATERIAL But that doesn ’ t matter to you, the reader. What you care about more is what Visual Studio Code can do to help you be productive. As a developer, it is frequently the small things that make the biggest difference—knowing how to add code with a single keyboard chord, being able to do client and server debugging on your Node.js project, or language-sensitive code completion.
    [Show full text]
  • Journal of Functional Programming Nixos: a Purely Functional Linux
    Journal of Functional Programming http://journals.cambridge.org/JFP Additional services for Journal of Functional Programming: Email alerts: Click here Subscriptions: Click here Commercial reprints: Click here Terms of use : Click here NixOS: A purely functional Linux distribution EELCO DOLSTRA, ANDRES LÖH and NICOLAS PIERRON Journal of Functional Programming / Volume 20 / Special Issue 5­6 / November 2010, pp 577 ­ 615 DOI: 10.1017/S0956796810000195, Published online: 15 October 2010 Link to this article: http://journals.cambridge.org/abstract_S0956796810000195 How to cite this article: EELCO DOLSTRA, ANDRES LÖH and NICOLAS PIERRON (2010). NixOS: A purely functional Linux distribution. Journal of Functional Programming,20, pp 577­615 doi:10.1017/ S0956796810000195 Request Permissions : Click here Downloaded from http://journals.cambridge.org/JFP, by Username: nrnr, IP address: 108.20.67.9 on 31 Aug 2012 JFP 20 (5 & 6): 577–615, 2011. c Cambridge University Press 2010 577 doi:10.1017/S0956796810000195 First published online 15 October 2010 NixOS: A purely functional Linux distribution EELCO DOLSTRA Department of Software Technology, Delft University of Technology, Postbus 5031, 2600 GA Delft, The Netherlands (e-mail: [email protected]) ANDRES LOH¨ Department of Information and Computing Sciences, Utrecht University, Postbus 80 . 089, 3508 TB Utrecht, The Netherlands (e-mail: [email protected]) NICOLAS PIERRON EPITA Research and Development Laboratory, 14-16 rue Voltaire, 94276 Le Kremlin-Bicetreˆ cedex, France (e-mail: [email protected]) Abstract Existing package and system configuration management tools suffer from an imperative model, where system administration actions such as package upgrades or changes to system configuration files are stateful: they destructively update the state of the system.
    [Show full text]
  • Upgrade Without Bricking
    Upgrade without Bricking Arnout Vandecappelle http://mind.be/content/Presentation_Upgrade-without-Bricking.pdf or .odp © 2012 Essensium N.V. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License You never know where your product will be used High-precision GNSS receiver You never know where your product will be used What if you install new firmware on remote systems? What if you install new firmware on remote systems? Murphy's Law What if you install new firmware on remote systems? Murphy's Law Upgrade without Bricking Arnout Vandecappelle http://mind.be/content/Presentation_Upgrade-without-Bricking.pdf or .odp © 2012 Essensium N.V. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License Overview 1 Failure mechanisms ● Power failure ● Bad firmware ● Communication errors 2 Boot loader upgrade 3 Package-based upgrade Overview 1 Failure mechanisms ● Power failure ● Bad firmware ● Communication errors 2 Boot loader upgrade 3 Package-based upgrade Power failure Power fails during upgrade ⇒ new firmware only partially written Solutions: Add fail-safe firmware Detect failed power Atomic update of firmware images Use journalling filesystem for writable data Detecting power failure: Switch to fail-safe firmware 1. Boot current firmware fail- boot current config safe loader firmware files FW Detecting power failure: Switch to fail-safe firmware 2. Switch to fail-safe fail- boot current config safe loader firmware files FW Detecting power failure: Switch to fail-safe firmware fail- boot new config safe loader firmware files FW 3. Overwrite firmware Detecting power failure: Switch to fail-safe firmware 4. Fail-safe restarts upgrade fail- boot new config safe loader firmware files FW Detecting power failure: Switch to fail-safe firmware 5.
    [Show full text]
  • Self-Scaling Clusters and Reproducible Containers to Enable Scientific Computing
    Self-Scaling Clusters and Reproducible Containers to Enable Scientific Computing Peter Z. Vaillancourt∗, J. Eric Coultery, Richard Knepperz, Brandon Barkerx ∗Center for Advanced Computing Cornell University, Ithaca, New York, USA Email: [email protected] yCyberinfrastructure Integration Research Center Indiana University, Bloomington, IN, USA Email: [email protected] zCenter for Advanced Computing Cornell University, Ithaca, New York, USA Email: [email protected] xCenter for Advanced Computing Cornell University, Ithaca, New York, USA Email: [email protected] Abstract—Container technologies such as Docker have become cyberinfrastructure at a number of campuses, enabling them a crucial component of many software industry practices espe- to provide computational resources to their faculty members cially those pertaining to reproducibility and portability. The con- [2]. As more organizations utilize cloud resources in order tainerization philosophy has influenced the scientific computing community, which has begun to adopt – and even develop – con- to provide flexible infrastructure for research purposes, CRI tainer technologies (such as Singularity). Leveraging containers has begun providing tools to harness the scalability and utility for scientific software often poses challenges distinct from those of cloud infrastructure, working together with the members encountered in industry, and requires different methodologies. of the Aristotle Cloud Federation Project [3]. Red Cloud, the This is especially true for HPC. With an increasing number Cornell component of the Aristotle Cloud Federation, provides of options for HPC in the cloud (including NSF-funded cloud projects), there is strong motivation to seek solutions that provide an OpenStack cloud services much like the Jetstream resource, flexibility to develop and deploy scientific software on a variety that is available both to Cornell faculty as well as to Aristotle of computational infrastructures in a portable and reproducible members.
    [Show full text]