DOCSLIB.ORG

What Happens When the Science DMZ Meets the Commodity Internet?

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
What Happens When the Science DMZ Meets the Commodity Internet? What happens when the Science DMZ meets the Commodity Internet? Presenter: Joe Breen Work done: Brian Haymore, Sam Liston University of Utah Center for High Performance Computing Boil and Bubble, Toil and Trouble, What do you get with a Science DMZ, That's mixed up with unlimited storage Fast and free? Image credit: http://www.zastavki.com/eng/Holidays/Halloween/wallpaper-24660.htm Start with a researcher Offer him candy (unlimited storage) Tell her its free Mix in a "frictionless" Science DMZ environment with 40G and 10G Data Transfer Nodes Image: https://fasterdata.es.net/science-dmz/science-dmz-architecture/ Mix in a well built commodity cloud service that can consume lots of data quickly Throw in an open source parallel tool that knows how to efficiently utilize a cloud provider’s API Google Drive Amazon S3 Openstack Swift / Rackspace cloud files / Memset Memstore Dropbox Google Cloud Storage Amazon Cloud Drive The local filesystem http://rclone.org/ What do you get? 14+Gig spikes, 5-8Gb/s of sustained traffic One 10G commodity pipe fills completely, traffic rolls to next available close peering point, and fills it too R&E routes yanked temporarily by Cloud vendor NOC to allow service to other commodity users and to better understand nature of congestion A call from a cloud provider NOC early in the morning asking to stop. (at least for a bit) Almost 100TB of data moved in 2.5 days Very happy researchers who want more What's an HPC center to do? Start mixing more... Multiple vendors now offering Apps/unlimited storage targeted at EDUcation *individual* users. Same vendors offer multi-tier Cloud storage for purchase for archival storage, mid-level storage, highly available storage, specific application storage Today's HPC researcher use cases for large personal cloud storage • Using as another storage bin for keeping points in time source code, input files, etc. • Using instead of a USB drive - a temporary location for: • Saving Snippets of code • Looking at single files • Moving data from a national resource that requires cleaning • Sharing data to a distributed international audience Potential other use cases for personal cloud storage or organization cloud storage • Use as large archive for different data sets for individual researcher • Use as archive for a research group or a collection of collaborating groups • "service account" with Linux extended ACLs to tar up and backup multiple collaborating research groups using the same file systems • allows a simple formula for groups to replicate data in another location -- flexibility for users at their level and convenience • Back off info from scratch drives - shuffle back and forth from Cloud provider and from HPC scratch – Accommodate over DMZ DTN box • ... The list continues … Some of these vendors have Research & Education peering directly, some of them only maintain Commodity peering Commodity peering not designed for bursty research or long term large data set file transfers Large research data set transfer Normal Commodity traffic Commodity peering points serve lots of companies and businesses Do we need additional tools in our ecosystem? How do we protect the community, the vendors, and the collaborative network environments, AND, encourage the innovation? Should we more aggressively leverage the emerging capabilities of the national Research and Education Software Defined Networking backbone? Match = Action Match flow (source IP, source port, destination IP, destination port…) => Action apply QoS or block per flow Data Transfer Nodes are generally multi-tenant, serving the transfer needs of several groups simultaneously. Image: https://fasterdata.es.net/science-dmz/science-dmz-architecture/ How might we work with one tenant AND continue to service other tenants well without null routing whole networks or even individual DTNs? Does the community need a more precision tool? OR Image: http://onsurg.com/instrument-handling-10-blade/ Summary • Science DMZs are frictionless, Commodity connections to Cloud vendors are not • Cloud vendors are coming up with new business plans that researchers like to use creatively • Do emerging technologies allow us to create new tools that might enable the innovation and protect the ecosystem? .
Load more

Recommended publications
  • Solving Top Business Challenges: Moving to the Cloud
    Solving Top Business Challenges: Moving to the Cloud How McKesson Did It with Rackspace Take every step with an expert team Business: Healthcare company McKesson needed that uses the latest methodologies and infrastructure and expertise layered with a provides you with personalized end-to- consultative approach to help its business units move complex, compliance-sensitive workloads end engagement. to public cloud. Challenge: Modernize applications; architect the Move Beyond the Data Center with Confidence right environment with a strictly defined security and The technology landscape is constantly evolving. As a result, regulatory approach; build the business case and get organizations that rely solely on traditional IT infrastructure are buy-in for public cloud; acquire an end-to-end cloud struggling to keep up with their competitors. Traditional infrastructure lifecycle strategy partner to accelerate deployment hampers engineering teams from being able to provide the agility, and to provide ongoing maintenance. flexibility and responsiveness their businesses need to thrive. Solution: Managed Public Cloud, Professional Across every industry, organizations are investing in cloud solutions. Services, Application Services and DevOps, These efforts require rethinking infrastructure, focusing on ways Microsoft Azure. to streamline and optimize operations, updating go-to-market strategies and modernizing applications to extract the most from Outcome: Reduced dependency on dedicated today’s technologies. data centers by moving dozens of applications across multiple business units to public cloud Desired Outcomes with a consultative approach to planning, assessment, execution and ongoing management, Cloud adoption has helped countless businesses drive efficiency through while maintaining stringent security and operations automation, grow revenue through innovation, and realize healthcare compliance.
    [Show full text]
  • 1) Installation 2) Configuration
    rclone 1) Installation........................................................................................................................................1 2) Configuration...................................................................................................................................1 2.1) Server setup..............................................................................................................................1 2.2) Client setup...............................................................................................................................2 2.3) Server setup - part 2..................................................................................................................2 2.4) Client verification.....................................................................................................................3 2.5) rclone - part 1............................................................................................................................3 2.6) rclone - part 2............................................................................................................................4 3) Backup configuration.......................................................................................................................5 4) Usage................................................................................................................................................5 1) Installation https://rclone.org/install/ Script installation To install rclone on Linux/macOS/BSD
    [Show full text]
  • University of Maine System Digial Migration Services - RFP# 2016-62 ADDENDUM #01
    University of Maine System Digial Migration Services - RFP# 2016-62 ADDENDUM #01 QUESTIONS 1. We do not understand this sentence: “By moving to a cloud storage system we eliminate the need to re-code the MySQL web application.” Explain cloud database services more. And, here’s a bit more info: Amazon Web Services has a variety of cloud-based database services, including both relational and NoSQL databases. Amazon Relational Database (RDS) run either MySQL, Oracle or SQL Server instances, while Amazon SimpleDB is a schema-less database meant for smaller workloads. On the NoSQL side, Amazon DynamoDB is its solid-state drive (SSD)-backed database that automatically replicates workloads across at least three availability zones. Amazon also offers a variety of auxiliary data management services, such as Redshift and Data Pipeline that helps to integrate data from multiple sources for easier management. Help us to understand what data cloud services you are referencing. Typically, moving MySQL to the cloud makes the database more redundant. Moving to the cloud does not have much impact on application coding. We agree that the database layer needs to be restructured and positioned into the cloud prior to any application coding work. In addition to Amazon, clients often use Google, Microsoft Azure, and the Rackspace cloud. ANSWER: The answer to this question can be found in section 1.1.3 of the RFP. We are looking for a two-phase solution. Phase one is migration of the website from U-Maine servers to a cloud hosted server (primarily for increased website speed and the ability to host high resolution images) and the migration from FileMaker to a new collections management database that will eliminate the MySQL web application by being able to link directly to the online display of the website with out the need of an intermediate web application.
    [Show full text]
  • Scaling a Game-Sharing Platform Introduction
    Level Up and SIMMER.io Down Scaling a Game-sharing Platform Introduction Much like gaming, starting a business means a lot of trial and error. In the beginning, you’re just trying to get your bearings and figure out which enemy to fend off first. After a few hours (or a few years on the market), it’s time to level up. SIMMER.io, a community site that makes sharing Unity WebGL games easy for indie game developers, leveled up in a big way to make their business sustainable for the long haul. When the site was founded in September 2017, the development team focused on getting the platform built and out the door, not on what egress costs would look like down the road. As it grew into a home for 80,000+ developers and 30,000+ games, though, those costs started to encroach on their ability to sustain and grow the business. We spoke with the SIMMER.io development team about their experience setting up a multi-cloud solution—including their use of the Bandwidth Alliance between Cloudflare and Backblaze B2 Cloud Storage to reduce egress to $0—to prepare the site for continued growth. © 2021 Backblaze Inc. All rights reserved. 500 Ben Franklin Ct. San Mateo, CA 94401 USA How to Employ a Multi-cloud Approach for Scaling a Web Application In 2017, sharing games online with static hosting through a service like AWS S3 was possible but certainly not easy. As one SIMMER.io team member put it, “No developer in the world would want to go through that.” The team saw a clear market opportunity.
    [Show full text]
  • Initial Definition of Protocols and Apis
    Initial definition of protocols and APIs Project acronym: CS3MESH4EOSC Deliverable D3.1: Initial Definition of Protocols and APIs Contractual delivery date 30-09-2020 Actual delivery date 16-10-2020 Grant Agreement no. 863353 Work Package WP3 Nature of Deliverable R (Report) Dissemination Level PU (Public) Lead Partner CERN Document ID CS3MESH4EOSC-20-006 Hugo Gonzalez Labrador (CERN), Guido Aben (AARNET), David Antos (CESNET), Maciej Brzezniak (PSNC), Daniel Muller (WWU), Jakub Moscicki (CERN), Alessandro Petraro (CUBBIT), Antoon Prins Authors (SURFSARA), Marcin Sieprawski (AILLERON), Ron Trompert (SURFSARA) Disclaimer: The document reflects only the authors’ view and the European Commission is not responsible for any use that may be made of the information it contains. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 863353 Table of Contents 1 Introduction ............................................................................................................. 3 2 Core APIS .................................................................................................................. 3 2.1 Open Cloud Mesh (OCM) ...................................................................................................... 3 2.1.1 Introduction .......................................................................................................................................... 3 2.1.2 Advancing OCM ....................................................................................................................................
    [Show full text]
  • Faq Cloud Sync
    FAQ CLOUD SYNC 1 What is Cloud Sync? NetApp® Data Fabric Cloud Sync is a simple replication and synchronization service. This software-as-a-service (SaaS) offering enables you to transfer and synchronize NAS data to and from cloud or on-premises object storage. The SMB/CIFS or NFS server can be the NetApp Cloud Volumes Service, a NetApp system, or a non-NetApp system. Cloud Sync supports these sources and targets: • CIFS • NFS • Amazon S3 • Amazon EFS • Azure Blob • IBM Cloud Object Storage • NetApp StorageGRID® Webscale appliance After your data is synchronized, it remains available for use in the target server and is updated during the next synchronization cycle. © 2019 NetApp, Inc. All Rights Reserved. | 1 2 Why should I use Cloud Sync? Cloud Sync enables you to perform data migration, data transformation, and data synchronization in a fast, efficient, and secure way. Key benefits of using Cloud Sync are: Fast. Cloud Sync transfers data in parallel processes. This speeds throughput to 1TB in four hours (in the default configuration), and up to 10x faster than in-house developed or traditional tools (such as rsync or Robocopy). Efficient. After the initial synchronization, only changes since the last synchronization are transferred. Data that hasn’t changed isn’t re-replicated, which makes updates faster. Cost-effective. Cloud Sync pricing is based on hourly usage, not on capacity. Compatible. Cloud Sync supports any NFS or CIFS servers, Amazon or private S3 buckets, Azure Blob, IBM Cloud Object Storage, Amazon EFS. Secure. Data is not transferred to our service domain; it remains in your protected environment.
    [Show full text]
  • Rackspace Service Blocks: Complex Cloud Operations Gain Advanced Operational Guidance and Support from Certified Cloud Specialists
    Rackspace Service Blocks: Complex Cloud Operations Gain advanced operational guidance and support from certified cloud specialists. Large organizations and businesses running highly complex cloud About Rackspace environments often require customized cloud architecture and access Rackspace is your trusted partner across cloud, to highly skilled cloud resources. Finding the right level of technical applications, security, data and infrastructure. expertise can be challenging, because talent is scarce, expensive and hard to retain. Rackspace® Complex Cloud Operations help you manage • A leader in the 2018 Gartner Magic Quadrant ongoing operational challenges and keep you informed on ever-evolving for Public Cloud Infrastructure Managed cloud features. Service Providers, Worldwide • 3,000+ cloud experts Experts Dedicated to Your Success • Hosting provider for more than half of the Rackspace is a next-generation cloud services provider that manages the Fortune 100 full cloud lifecycle. Complex Cloud Operations offers access to advanced technical resources to better manage public cloud operations and • 20+ years of hosting experience infrastructure architecture. • Customers in 150+ countries • 3,000+ AWS accreditations Key Features • 1,000+ AWS certifications Rackspace offers a range of robust features to support complex cloud operations including: • 1,300+ Microsoft certifications worldwide • 200+ Microsoft-certified professionals Expertise: Expert technical resources, including solution architects, lead engineers, container and Kubernetes experts, and DevOps professionals • 150+ Microsoft Azure certifications worldwide help optimize public cloud operations and architectural infrastructure. • 100+ Microsoft Azure-certified professionals Optimized Custom Configurations: Custom configurations include • 13 Microsoft partner certifications monitoring services, OS optimization, patch management, alert response • Five-time Microsoft Hosting Partner of the Year runbooks and deployment management.
    [Show full text]
  • Cloud Computing in Amazon Web Services, Microsoft Window Neu
    CHIMA DESMOND OPARADESMOND CHIMA CLOUD COMPUTING IN AMAZON WEB SERVICES, MICROSOFT WINDOWS AZURE, GOOGLE APP ENGINE AND IBM CLOUD PLATFORMS: A COMPARATIVE STUDY CLOUD COMPUTING IN AMAZON WEB SERVICES, MICROSOFT WINDOW NEU NEU WINDOW MICROSOFT SERVICES, WEB AMAZON IN COMPUTING CLOUD AZURE, GOOGLE APP ENGINE AND IBM CLOUD PLATFORMS: 2019 2019 PLATFORMS: CLOUD IBM AND ENGINE APP GOOGLE AZURE, A THESIS SUBMITTED TO THE GRADUATE SCHOOL OF APPLIED SCIENCES A COMPARATIVE STUDY A COMPARATIVE OF NEAR EAST UNIVERSITY By CHIMA DESMOND OPARA In Partial Fulfillment of the Requirements for the Degree of Master of Science in Computer Information Systems NICOSIA, 2019 CLOUD COMPUTING IN AMAZON WEB SERVICES, MICROSOFT WINDOWS AZURE, GOOGLE APP ENGINE AND IBM CLOUD PLATFORMS: A COMPARATIVE STUDY A THESIS SUBMITTED TO THE GRADUATE SCHOOL OF APPLIED SCIENCES OF NEAR EAST UNIVERSITY By CHIMA DESMOND OPARA In Partial Fulfillment of the Requirements for the Degree of Master of Science in Computer Information System NICOSIA, 2019 Chima Desmond OPARA: CLOUD COMPUTING IN AMAZON WEB SERVICES, MICROSOFT WINDOWS AZURE, GOOGLE APP ENGINE, AND IBM CLOUD PLATFORMS: A COMPARATIVE STUDY Approval of Director of Graduate School of Applied Sciences Prof. Dr. Nadire CAVUS We certify this thesis is satisfactory for the award of the Degree of Masters of Science in Computer Information Systems Examining Committee in Charge: Assoc.Prof.Dr. Fezile Özdamlı Committee Chairperson, Department of Computer Information Systems, NEU Prof.Dr. Nadire Çavuş Supervisor, Department of Computer Information Systems, NEU Asst.Prof.Dr. Damla Karagözlü Co-Supervisor, Department of Computer Information Systems, NEU Assoc.Prof.Dr. Hüseyin Bicen Department of Computer Education and Instructional Technologies, NEU Asst.Prof.Dr.
    [Show full text]
  • On High Performance Cloud Based File Synchronization with User Collaboration
    On High Performance Cloud Based File Synchronization with User Collaboration A THESIS SUBMITTED TO THE FACULTY OF THE GRADUATE SCHOOL OF THE UNIVERSITY OF MINNESOTA BY Mounika Chillamcherla IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE Haiyang Wang July 2016 © Mounika Chillamcherla 2016 Acknowledgements I would like to take this opportunity to sincerely thank Dr. Haiyang Wang for his enor- mous support and guidance, without whom this thesis would not have been possible. I am greatly indebted to him for his vital suggestions throughout the work. He is very kind, en- couraging, extremely supportive and always takes care of the well-being of all his students. I really feel blessed to have him as my thesis advisor. I would like to thank Dr. Ted Pederson and Dr. Yang Li for serving on my thesis com- mittee. A special thanks to Dr. Ted Pederson for teaching me the course Natural Language Processing and filling energy and enthusiasm in me, and for being my inspiration. Further, I would thank all the professors of CS Department, Lori Lucia, Clare Ford, Jim Luttinen and International Student Services for their timely help. I would like to thank all my friends in the class for all the fun, knowledge, encourage- ment, and for making my stay a memorable one. Thanks to SaiKrishna Peachara for taking the time to proofread my thesis documentation and for his indispensable support. Lastly, I would like to thank my parents who stood by me and motivated me at all times. i Dedication I would like to dedicate this thesis to my parents, Adinarayana Chillamcherla and Guru Lakshmi Chillamcherla and to my brother, Eshwar Sai Chillamcherla for their everlasting love and endless support.
    [Show full text]
  • Cloud Computing Bible
    Barrie Sosinsky Cloud Computing Bible Published by Wiley Publishing, Inc. 10475 Crosspoint Boulevard Indianapolis, IN 46256 www.wiley.com Copyright © 2011 by Wiley Publishing, Inc., Indianapolis, Indiana Published by Wiley Publishing, Inc., Indianapolis, Indiana Published simultaneously in Canada ISBN: 978-0-470-90356-8 Manufactured in the United States of America 10 9 8 7 6 5 4 3 2 1 No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, 201-748-6011, fax 201-748-6008, or online at http://www.wiley.com/go/permissions. Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation warranties of fitness for a particular purpose. No warranty may be created or extended by sales or promotional materials. The advice and strategies contained herein may not be suitable for every situation. This work is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional services.
    [Show full text]
  • Fanatical Support for Microsoft Azure
    PRODUCT GUIDE FANATICAL SUPPORT FOR MICROSOFT AZURE With Rackspace Service Blocks October 5, 2018 TABLE OF CONTENTS OVERVIEW . 3 AZURE REGION AVAILABILITY . 3 SERVICE OFFERS . 3 Cloud Foundation ............................................. 3 Architect & Deploy ............................................ 3 Manage & Operate ............................................ 3 Complex Cloud Operations .................................... 4 COMBINING SERVICE OFFERS . 4 SERVICE MATRIX . 4 HUMAN EXPERTISE . 5 Your Fanatical Support for Azure Team ........................ 5 How to Contact Support ...................................... 6 Microsoft Premier Support Escalations ........................ 6 SERVICE OPERATIONS . 6 Customer Runbooks ........................................... 6 Monitoring (Watchman) ....................................... 7 Incident Management ......................................... 7 Change Management ......................................... 9 Ongoing Management Services ............................... 9 Patching ................................................... 9 Backups ................................................... 9 Antivirus ................................................... 9 Operating System Support ................................ 10 Passport (Secure VM Access) .............................. 10 Boarding Pass (Secure Azure Portal Access) ............... .10 Service Reviews ............................................ 11 Response Time Guarantees ................................... 11 ADDITIONAL SERVICES
    [Show full text]
  • How Can Startups Make Use of Cloud Services
    California State University, San Bernardino CSUSB ScholarWorks Electronic Theses, Projects, and Dissertations Office of aduateGr Studies 4-2021 How can startups make use of cloud services Gauri Nade Gauri Nade California State University - San Bernardino Follow this and additional works at: https://scholarworks.lib.csusb.edu/etd Part of the Educational Technology Commons Recommended Citation Nade, Gauri and Nade, Gauri, "How can startups make use of cloud services" (2021). Electronic Theses, Projects, and Dissertations. 1262. https://scholarworks.lib.csusb.edu/etd/1262 This Thesis is brought to you for free and open access by the Office of aduateGr Studies at CSUSB ScholarWorks. It has been accepted for inclusion in Electronic Theses, Projects, and Dissertations by an authorized administrator of CSUSB ScholarWorks. For more information, please contact [email protected]. HOW CAN START UPS MAKE USE OF CLOUD SERVICES A Project Presented to the Faculty of California State University, San Bernardino In Partial Fulfillment of the Requirements for the Degree Master of Science in Information Systems and Technology by Gauri Rajendra Nade May 2021 HOW CAN START UPS MAKE USE OF CLOUD SERVICES A Project Presented to the Faculty of California State University, San Bernardino by Gauri Rajendra Nade May 2021 Approved by: Benjamin Becerra, PhD, Committee Chair Conrad Shayo, PhD, Reader Jay Varzandeh, PhD, Dept. Chair, Information & Decision Sciences © 2021 Gauri Rajendra Nade ABSTRACT The purpose of this project is to discuss the technical obstacles that small and medium-sized enterprises (SMEs) face, as well as how cloud computing can help to solve these issues. Cloud computing has the ability to radically change competitive environments by offering a new forum for generating and delivering business value and market development.
    [Show full text]