DOCSLIB.ORG

Nexedi Stack: Less Is More AWS Vs. Rapid.Space

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Nexedi Stack: Less Is More AWS Vs. Rapid.Space Most cloud services could be built by combining an open source / Free Software (eg. MariaDB) together with a service lifecycle automation (SLM) platform (eg. SlapOS). With one notable exception, this goal has not yet been reached by the Open Source / Free Software community because most projects are still focusing on some kind of virtualisation (eg. virtual machines, containers) or orchestration (eg. Kuberneres) which only represent 10% to 20% of what is necessary to implement a public cloud service. This may leave SlapOS as the only open source / Free Software project that could possibly match leading public cloud services (AWS, Azure, Alicloud), as the following comparison table highlights: SlapOS OpenStack Kubernetes Jupyter NixOS AWS IaaS ✔ ✔ ✔ PaaS ✔ ✔ ✔ ✔ ✔ Service App Store ✔ ✔ ✔ Orchestration ✔ ✔ ✔ ✔ Virtualisation ✔ ✔ ✔ Network management ✔ ✔ ✔ Resilient networking ✔ ✔ Bare metal encapsulation ✔ ✔ ✔ ✔ Portability ✔ limited ? Multi-tenant services (eg. CDN) ✔ ✔ Edge computing ✔ ✔ Continuous integration ✔ ✔ Self-monitoring ✔ ✔ Autonomous convergence ✔ ? Automated DR ✔ ✔ Accounting & Billing ✔ ✔ Nexedi stack: less is more Nexedi develops and operates complex, scalable applications with less than 15 software: the Nexedi Freee Software stack. Out of those 15 software, developers actually focus on four of them: ERP5 for enterprise records management (including CRM, MRP, MES) OfficeJS for Progressive Web Application (PWA) development; SlapOS for IT infrastructure automation (on the cloud, at the edge or in the workshop); Wendelin for data collection and processing (including AI and conversion). Since both SlapOS, Wendelin and OfficeJS are just variations of ERP5, Nexedi developers actually only need to learn a single framework (ERP5) and a single language (python). By relying on less tools, Nexedi developers have more time to learn ERP5 in depth. They can reuse their ERP5 knowledge with SlapOS and Wendelin. And thanks to the huge size of python library, most problems that are not already covered by ERP5, SlapOS, Wendelin or OfficeJS can be solved quickly. AWS vs. Rapid.Space Amazon AWS provides more than 200 cloud services. The table below provides a comparison between Amazon AWS cloud services and technologies of the Nexedi Free Software stack which can be used to build simlar services deployed with SlapOS on Rapid.Space high performance, low cost cloud platform. For each AWS category and product, we provide a possible alternative in Nexedi Free Software stack either as a SlapOS profile (server based) or as Progressive Web App (browser based). We also provide open source / Free Software alternatives we are aware of. Category Product Description Nexedi or partner SlapOS PWA FLOSS Alternative Query data in S3 using Athena Wendelin ✔ SQL Managed search CloudSearch Wendelin ✔ Searx service Elasticsearch Run and scale Wendelin ✔ Elasticsearch Service Elasticsearch clusters Hosted Hadoop EMR Wendelin ✔ Hadoop framework Analyze real-time video Category KinesPirsoduct and dDateas sctrriepatmiosn WNenxdeedlii nor partner Sla✔pOS PW A FLOSS Alternative Managed Fully managed Apache Streaming for Wendelin (fluentd) ✔ Kafka Kafka service Apache Kafka Analytics Fast, simple, cost- Redshift effective data Wendelin ✔ warehousing Fast business analytics QuickSight Wendelin (iodide) ✔ Superset or Metabase service Find, subscribe to, and Data Exchange use third-party data in Wendelin (ebulk) ✔ the cloud Orchestration service Data Pipeline for periodic, data- Wendelin ✔ Airflow or Activeeon driven workflows Wendelin AWS Glue Prepare and load data ✔ Talend (CONNECT) AWS Lake Build a secure data Wendelin (ebulk) ✔ Formation lake in days AWS Step Coordination for Wendelin (activate) ✔ Airflow or Activeeon Functions distributed applications Serverless event bus EventBridge for SaaS apps & AWS Wendelin (activate) ✔ Airflow or Activeeon services Managed message MQ JIO ✔ open62541 or RabbiitMQ broker for ActiveMQ Application Simple Integration Managed message Notification JIO ✔ open62541 or DPS topics for pub/sub Service (SNS) Simple Queue Managed message Wendelin (activate) ✔ Airflow or Activeeon Service (SQS) queues Power your apps with AWS AppSync the right data from JIO ✔ many sources, at scale Build and run VR and RenderJS + VR and AR Sumerian ✔ AR applications BabylonJS AWS Cost Analyze your AWS SlapOS (UI) ✔ Explorer cost and usage Set custom cost and AWS Budgets usage budgets Access comprehensive AWS Cost & cost and usage SlapOS (Monitor) ✔ AWS Cost Usage Report information Management Reserved Dive deeper into your Instance reserved instances SlapOS (Monitor) ✔ Reporting (RIs) Save up to 72% on Savings Plans compute usage with flexible pricing Amazon Create and manage Managed scalable blockchain Blockchain networks Blockchain Amazon Quantum Ledger Fully managed ledger Database database (QLDB) Alexa for Empower your Business organization with Alexa Frustration-free Amazon Chime meetings, video calls, SlapOS (Nextalk) ✔ and chat Business Secure enterprise Applications Amazon document storage and ERP5 (DMS) ✔ SlapOS (Nextcloud) WorkDocs sharing Amazon Secure email and Category WorkPMroadil uct calendDaersincgription SNlaepxOedSi (oNre pxatcrltonuedr) Sla✔pOS PW A FLOSS Alternative Virtual servers in the Amazon EC2 SlapOS (kvm) ✔ Proxmox cloud Scale compute Amazon EC2 capacity to meet SlapOS (buildout) ✔ Auto Scaling demand Amazon Elastic Store and retrieve Container SlapOS (buildout) ✔ docker images Registry Amazon Elastic Run and manage Container SlapOS (buildout) ✔ Proxmox docker containers Service Amazon Elastic Run managed Kubernetes SlapOS (buildout) ✔ Kubernetes Kubernetes on AWS Service Launch and manage Amazon Lightsail SlapOS ✔ virtual private servers Run batch jobs at any AWS Batch Wendelin scale Compute AWS Elastic Run and manage web SlapOS (buildout) ✔ Beanstalk apps Run containers without AWS Fargate managing servers or SlapOS (buildout) clusters Run code without AWS Lambda Wendelin ✔ thinking about servers Run AWS AWS Outposts infrastructure on- Rapid.Space Node ✔ premises AWS Serverless Discover, deploy, and Application publish serverless Repository applications Deliver ultra-low AWS Wavelength latency applications for 5G devices Build a hybrid cloud VMware Cloud without custom SlapOS ✔ on AWS hardware Cloud-based contact Amazon Connect ERP5 ✔ center service Personalized user Amazon Pinpoint engagement across channels Customer Amazon Simple Engagement Email sending and Email Service receiving (SES) Contact center Contact Lens for analytics powered by ERP5/Wendelin ✔ Amazon Connect ML High performance Amazon Aurora managed relational MariaDB + Repman ✔ database Amazon Managed NoSQL NEO ✔ DynamoDB database Amazon DocumentDB Fully managed NEO ✔ (with MongoDB document database compatibility) Amazon In-memory caching ✔ Kumofs ElastiCache system Amazon Managed Managed Cassandra- MariaDB Apache compatible database (CASSANDRA) Cassandra Service Fully managed graph MariaDB Category AmaPzoron dNuecpttune databDasees csreirpvtiicoen (NOeQxGeRdiA oPrH p)artner Sla✔pOS PW A FLOSS Alternative Database Amazon Quantum Ledger Fully managed ledger NEO ✔ Database database (QLDB) Managed relational database service for Amazon RDS MySQL, PostgreSQL, MariaDB ✔ Oracle, SQL Server, and MariaDB Amazon RDS on Automate on-premises SlapOS ✔ VMware database management Fast, simple, cost- Amazon Redshift effective data Wendelin ✔ warehousing Amazon Fully managed time MariaDB ✔ InfluxDB Timestream series database (COLUMNSTORE) AWS Database Migrate databases with MariaDB ✔ Migration Service minimal downtime (CONNECT) Production-ready Amazon Corretto distribution of OpenJDK AWS Cloud Model cloud Development Kit infrastructure using WebRunner ✔ (CDK) code Write, run, and debug AWS Cloud9 WebRunner ✔ code on a cloud IDE AWS CodeBuild Build and test code WebRunner ✔ AWS Store code in private WebRunner ✔ CodeCommit Git repositories AWS Automate code WebRunner ✔ CodeDeploy deployment Developer AWS Release software using ERP5 (Test Tools ✔ CodePipeline continuous delivery Runner) Develop and deploy AWS CodeStar WebRunner ✔ AWS applications AWS Command Unified tool to manage SlapOS (console) ✔ Line Interface AWS services Test Android, iOS, and AWS Device web apps on real Farm devices in the AWS cloud AWS Tools and Tools and SDKs for SlapOS ✔ SDKs AWS Analyze and debug AWS X-Ray WebRunner ✔ your applications Amazon Virtual desktops in the SlapOS (kvm) ✔ WorkSpaces cloud Stream desktop End User Amazon applications securely to SlapOS (kvm) ✔ Computing AppStream 2.0 a browser Amazon Enable mobile access SlapOS (CDN) ✔ WorkLink to internal websites Simple, fast, cost- Amazon effective dedicated Rapid.Space ✔ GameLift game server hosting Game Tech A free cross-platform Amazon 3D game engine, with BabylonJS ✔ Lumberyard Full Source, integrated with AWS and Twitch Connect devices to the AWS IoT Core Wendelin ✔ cloud Amazon IoT operating system Nuttx FreeRTOS for microcontrollers Category AWSP Groredeuncgtrass Local Dcoemscpruipteti,on SNlaepxOedSi or partner Sla✔pOS PW A FLOSS Alternative messaging, and sync for devices One click creation of an AWS IoT 1-Click SlapOS (token) ✔ AWS Lambda trigger AWS IoT Analytics for IoT Wendelin ✔ Analytics devices Cloud programmable AWS IoT Button dash button Internet of AWS IoT Device Security management Things Defender for IoT devices Onboard, organize, AWS IoT Device and remotely manage SlapOS (Master) ✔ Mender UpKit Management IoT devices IoT event detection and AWS IoT Events Wendelin (windea) ✔ response AWS IoT IoT data collector
Load more

Recommended publications
  • Effective Virtual CPU Configuration with QEMU and Libvirt
    Effective Virtual CPU Configuration with QEMU and libvirt Kashyap Chamarthy <[email protected]> Open Source Summit Edinburgh, 2018 1 / 38 Timeline of recent CPU flaws, 2018 (a) Jan 03 • Spectre v1: Bounds Check Bypass Jan 03 • Spectre v2: Branch Target Injection Jan 03 • Meltdown: Rogue Data Cache Load May 21 • Spectre-NG: Speculative Store Bypass Jun 21 • TLBleed: Side-channel attack over shared TLBs 2 / 38 Timeline of recent CPU flaws, 2018 (b) Jun 29 • NetSpectre: Side-channel attack over local network Jul 10 • Spectre-NG: Bounds Check Bypass Store Aug 14 • L1TF: "L1 Terminal Fault" ... • ? 3 / 38 Related talks in the ‘References’ section Out of scope: Internals of various side-channel attacks How to exploit Meltdown & Spectre variants Details of performance implications What this talk is not about 4 / 38 Related talks in the ‘References’ section What this talk is not about Out of scope: Internals of various side-channel attacks How to exploit Meltdown & Spectre variants Details of performance implications 4 / 38 What this talk is not about Out of scope: Internals of various side-channel attacks How to exploit Meltdown & Spectre variants Details of performance implications Related talks in the ‘References’ section 4 / 38 OpenStack, et al. libguestfs Virt Driver (guestfish) libvirtd QMP QMP QEMU QEMU VM1 VM2 Custom Disk1 Disk2 Appliance ioctl() KVM-based virtualization components Linux with KVM 5 / 38 OpenStack, et al. libguestfs Virt Driver (guestfish) libvirtd QMP QMP Custom Appliance KVM-based virtualization components QEMU QEMU VM1 VM2 Disk1 Disk2 ioctl() Linux with KVM 5 / 38 OpenStack, et al. libguestfs Virt Driver (guestfish) Custom Appliance KVM-based virtualization components libvirtd QMP QMP QEMU QEMU VM1 VM2 Disk1 Disk2 ioctl() Linux with KVM 5 / 38 libguestfs (guestfish) Custom Appliance KVM-based virtualization components OpenStack, et al.
    [Show full text]
  • Red Hat Enterprise Linux 7 7.1 Release Notes
    Red Hat Enterprise Linux 7 7.1 Release Notes Release Notes for Red Hat Enterprise Linux 7 Red Hat Customer Content Services Red Hat Enterprise Linux 7 7.1 Release Notes Release Notes for Red Hat Enterprise Linux 7 Red Hat Customer Content Services Legal Notice Copyright © 2015 Red Hat, Inc. This document is licensed by Red Hat under the Creative Commons Attribution-ShareAlike 3.0 Unported License. If you distribute this document, or a modified version of it, you must provide attribution to Red Hat, Inc. and provide a link to the original. If the document is modified, all Red Hat trademarks must be removed. Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix, Fedora, the Infinity Logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries. Linux ® is the registered trademark of Linus Torvalds in the United States and other countries. Java ® is a registered trademark of Oracle and/or its affiliates. XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries. MySQL ® is a registered trademark of MySQL AB in the United States, the European Union and other countries. Node.js ® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
    [Show full text]
  • Red Hat Enterprise Linux 6 Developer Guide
    Red Hat Enterprise Linux 6 Developer Guide An introduction to application development tools in Red Hat Enterprise Linux 6 Dave Brolley William Cohen Roland Grunberg Aldy Hernandez Karsten Hopp Jakub Jelinek Developer Guide Jeff Johnston Benjamin Kosnik Aleksander Kurtakov Chris Moller Phil Muldoon Andrew Overholt Charley Wang Kent Sebastian Red Hat Enterprise Linux 6 Developer Guide An introduction to application development tools in Red Hat Enterprise Linux 6 Edition 0 Author Dave Brolley [email protected] Author William Cohen [email protected] Author Roland Grunberg [email protected] Author Aldy Hernandez [email protected] Author Karsten Hopp [email protected] Author Jakub Jelinek [email protected] Author Jeff Johnston [email protected] Author Benjamin Kosnik [email protected] Author Aleksander Kurtakov [email protected] Author Chris Moller [email protected] Author Phil Muldoon [email protected] Author Andrew Overholt [email protected] Author Charley Wang [email protected] Author Kent Sebastian [email protected] Editor Don Domingo [email protected] Editor Jacquelynn East [email protected] Copyright © 2010 Red Hat, Inc. and others. The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version. Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
    [Show full text]
  • Building Machine Learning Inference Pipelines at Scale
    Building Machine Learning inference pipelines at scale Julien Simon Global Evangelist, AI & Machine Learning @julsimon © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Problem statement • Real-life Machine Learning applications require more than a single model. • Data may need pre-processing: normalization, feature engineering, dimensionality reduction, etc. • Predictions may need post-processing: filtering, sorting, combining, etc. Our goal: build scalable ML pipelines with open source (Spark, Scikit-learn, XGBoost) and managed services (Amazon EMR, AWS Glue, Amazon SageMaker) © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Apache Spark https://spark.apache.org/ • Open-source, distributed processing system • In-memory caching and optimized execution for fast performance (typically 100x faster than Hadoop) • Batch processing, streaming analytics, machine learning, graph databases and ad hoc queries • API for Java, Scala, Python, R, and SQL • Available in Amazon EMR and AWS Glue © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. MLlib – Machine learning library https://spark.apache.org/docs/latest/ml-guide.html • Algorithms: classification, regression, clustering, collaborative filtering. • Featurization: feature extraction, transformation, dimensionality reduction. • Tools for constructing, evaluating and tuning pipelines • Transformer – a transform function that maps a DataFrame into a new
    [Show full text]
  • Developer Guide
    Red Hat Enterprise Linux 6 Developer Guide An introduction to application development tools in Red Hat Enterprise Linux 6 Last Updated: 2017-10-20 Red Hat Enterprise Linux 6 Developer Guide An introduction to application development tools in Red Hat Enterprise Linux 6 Robert Krátký Red Hat Customer Content Services [email protected] Don Domingo Red Hat Customer Content Services Jacquelynn East Red Hat Customer Content Services Legal Notice Copyright © 2016 Red Hat, Inc. and others. This document is licensed by Red Hat under the Creative Commons Attribution-ShareAlike 3.0 Unported License. If you distribute this document, or a modified version of it, you must provide attribution to Red Hat, Inc. and provide a link to the original. If the document is modified, all Red Hat trademarks must be removed. Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries. Linux ® is the registered trademark of Linus Torvalds in the United States and other countries. Java ® is a registered trademark of Oracle and/or its affiliates. XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries. MySQL ® is a registered trademark of MySQL AB in the United States, the European Union and other countries. Node.js ® is an official trademark of Joyent.
    [Show full text]
  • Open Source in the Enterprise
    Open Source in the Enterprise Andy Oram and Zaheda Bhorat Beijing Boston Farnham Sebastopol Tokyo Open Source in the Enterprise by Andy Oram and Zaheda Bhorat Copyright © 2018 O’Reilly Media. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online edi‐ tions are also available for most titles (http://oreilly.com/safari). For more information, contact our corporate/institutional sales department: 800-998-9938 or [email protected]. Editor: Michele Cronin Interior Designer: David Futato Production Editor: Kristen Brown Cover Designer: Karen Montgomery Copyeditor: Octal Publishing Services, Inc. July 2018: First Edition Revision History for the First Edition 2018-06-18: First Release The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Open Source in the Enterprise, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the informa‐ tion and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.
    [Show full text]
  • Virtualization Best Practices
    SUSE Linux Enterprise Server 15 SP1 Virtualization Best Practices SUSE Linux Enterprise Server 15 SP1 Publication Date: September 24, 2021 Contents 1 Virtualization Scenarios 2 2 Before You Apply Modifications 2 3 Recommendations 3 4 VM Host Server Configuration and Resource Allocation 3 5 VM Guest Images 25 6 VM Guest Configuration 36 7 VM Guest-Specific Configurations and Settings 42 8 Xen: Converting a Paravirtual (PV) Guest to a Fully Virtual (FV/HVM) Guest 45 9 External References 49 1 SLES 15 SP1 1 Virtualization Scenarios Virtualization oers a lot of capabilities to your environment. It can be used in multiple scenarios. To get more details about it, refer to the Book “Virtualization Guide” and in particular, to the following sections: Book “Virtualization Guide”, Chapter 1 “Virtualization Technology”, Section 1.2 “Virtualization Capabilities” Book “Virtualization Guide”, Chapter 1 “Virtualization Technology”, Section 1.3 “Virtualization Benefits” This best practice guide will provide advice for making the right choice in your environment. It will recommend or discourage the usage of options depending on your workload. Fixing conguration issues and performing tuning tasks will increase the performance of VM Guest's near to bare metal. 2 Before You Apply Modifications 2.1 Back Up First Changing the conguration of the VM Guest or the VM Host Server can lead to data loss or an unstable state. It is really important that you do backups of les, data, images, etc. before making any changes. Without backups you cannot restore the original state after a data loss or a misconguration. Do not perform tests or experiments on production systems.
    [Show full text]
  • Virtualization
    Virtualization Rich Jones & Mark McLoughlin 2009-06-27 @ 11:00 What? ● Guest virtual machines ● Run multiple OSes on a single host ● Support from modern hardware Why? ● Test rawhide ● Play with other distros ● Run windows ● Server consolidation ● Migration How? ● yum groupinstall Virtualization ● virt-manager Previously - Xen ● Standalone hypervisor ● Paravirtualization ● Dom0 not upstream Now - KVM ● kvm.ko ● VM is just a process ● Uses Intel VT or AMD SVM ● Paravirt too ● Simple, secure, fast Cool Features ● Save/restore ● Live migration ● PCI/USB device assignment ● Swapable guest memory ● Ballooning ● KSM libvirt ● A VM management API ● e.g. list VMs, start/stop VM ● XML format to describe VMs ● Hypervisor independent - KVM, Xen, LXC, VirtualBox, UML, ESX, ... libvirt – There's More ● Storage and networking management too – Storage pools and volumes – Virtual networks – Host network interface configuration ● Remote management ● Language bindings – C, Python, Perl, Ruby, OCaml, Java, C# Virtual Machine Manager ● Simple UI ● Easy to create a VM ● Monitoring ● Hotplug ● Remote management virt-install ● Kick off a guest install from the command line $> virt-install -n MyGuest -m 512 \ --disk pool=default,size=8 \ --network:default \ -l http://download.fedoraproject.org/.../x86_64/os Misc. Tools ● virt-clone/virt-image/virt-convert ● xenner – run Xen guests in KVM ● virt-top – 'top' for VMs ● virt-ps/virt-uname/virt-dmesg – poke VMs Get Involved ● list: [email protected] ● http://fedoraproject.org/wiki/Virtualization ● #virt on irc.oftc.net
    [Show full text]
  • Serverless Predictions at Scale
    Serverless Predictions at Scale Thomas Reske Global Solutions Architect, Amazon Web Services © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Serverless computing allows you to build and run applications and services without thinking about servers © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What are the benefits of serverless computing? No server management Flexible scaling $ Automated high availability No excess capacity © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Machine Learning Process Fetch data Operations, Clean monitoring, and and optimization format data Integration Prepare and and deployment transform data Evaluation and Model Training verificaton © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Machine Learning Process Fetch data Operations, Clean How can I use pre-trained monitoring, and and models for serving optimization format data predictions? How to scale effectively? Integration Prepare and and deployment transform data Evaluation and Model Training verificaton © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What exactly are we deploying? © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. A Pre-trained Model... • ... is simply a model which has been trained previously • Model artifacts (structure, helper files, parameters, weights, ...) are files, e.g. in JSON or binary format • Can be serialized (saved) and de-serialized (loaded) by common deep learning frameworks • Allows for fine-tuning of models („head-start“ or „freezing“), but also to apply them to similar problems and different data („transfer learning“) • Models can be made publicly available, e.g. in the MXNet Model Zoo © 2018, Amazon Web Services, Inc.
    [Show full text]
  • Analytics Lens AWS Well-Architected Framework Analytics Lens AWS Well-Architected Framework
    Analytics Lens AWS Well-Architected Framework Analytics Lens AWS Well-Architected Framework Analytics Lens: AWS Well-Architected Framework Copyright © Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. Analytics Lens AWS Well-Architected Framework Table of Contents Abstract ............................................................................................................................................ 1 Abstract .................................................................................................................................... 1 Introduction ...................................................................................................................................... 2 Definitions ................................................................................................................................. 2 Data Ingestion Layer ........................................................................................................... 2 Data Access and Security Layer ............................................................................................ 3 Catalog and Search Layer ...................................................................................................
    [Show full text]
  • Ezgi Korkmaz Outline
    KTH ROYAL INSTITUTE OF TECHNOLOGY BigDL: A Distributed Deep Learning Framework for Big Data ACM Symposium on Cloud Computing 2019 [Acceptance Rate: 24%] Jason Jinquan Dai, Yiheng Wang, Xin Qiu, Ding Ding, Yao Zhang, Yanzhang Wang, Xianyan Jia, Cherry Li Zhang, Yan Wan, Zhichao Li, Jiao Wang, Sheng- sheng Huang, Zhongyuan Wu, Yang Wang, Yuhao Yang, Bowen She, Dongjie Shi, Qi Lu, Kai Huang, Guoqiong Song. [Intel Corporation] Presenter: Ezgi Korkmaz Outline I Deep learning frameworks I BigDL applications I Motivation for end-to-end framework I Drawbacks of prior approaches I BigDL framework I Experimental Setup and Results of BigDL framework I Critique of the paper 2/16 Deep Learning Frameworks I Big demand from organizations to apply deep learning to big data I Deep learning frameworks: I Torch [2002 Collobert et al.] [ C, Lua] I Caffe [2014 Berkeley BAIR] [C++] I TensorFlow [2015 Google Brain] [C++, Python, CUDA] I Apache MXNet [2015 Apache Software Foundation] [C++] I Chainer [2015 Preferred Networks] [ Python] I Keras [2016 Francois Chollet] [Python] I PyTorch [2016 Facebook] [Python, C++, CUDA] I Apache Spark is an open-source distributed general-purpose cluster-computing framework. I Provides interface for programming clusters with data parallelism 3/16 BigDL I A library on top of Apache Spark I Provides integrated data-analytics within a unifed data analysis pipeline I Allows users to write their own deep learning applications I Running directly on big data clusters I Supports similar API to Torch and Keras I Supports both large scale distributed training and inference I Able to run across hundreds or thousands servers efficiently by uses underlying Spark framework 4/16 BigDL I Developed as an open source project I Used by I Mastercard I WorldBank I Cray I Talroo I UCSF I JD I UnionPay I GigaSpaces I Wide range of applications: transfer learning based image classification, object detection, feature extraction, sequence-to-sequence prediction, neural collaborative filtering for reccomendation etc.
    [Show full text]
  • Apache Mxnet on AWS Developer Guide Apache Mxnet on AWS Developer Guide
    Apache MXNet on AWS Developer Guide Apache MXNet on AWS Developer Guide Apache MXNet on AWS: Developer Guide Copyright © Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. Apache MXNet on AWS Developer Guide Table of Contents What Is Apache MXNet? ...................................................................................................................... 1 iii Apache MXNet on AWS Developer Guide What Is Apache MXNet? Apache MXNet (MXNet) is an open source deep learning framework that allows you to define, train, and deploy deep neural networks on a wide array of platforms, from cloud infrastructure to mobile devices. It is highly scalable, which allows for fast model training, and it supports a flexible programming model and multiple languages. The MXNet library is portable and lightweight. It scales seamlessly on multiple GPUs on multiple machines. MXNet supports programming in various languages including Python, R, Scala, Julia, and Perl. This user guide has been deprecated and is no longer available. For more information on MXNet and related material, see the topics below. MXNet MXNet is an Apache open source project. For more information about MXNet, see the following open source documentation: • Getting Started – Provides details for setting up MXNet on various platforms, such as macOS, Windows, and Linux.
    [Show full text]