DOCSLIB.ORG

Machine Learning Cloud Getting Started Guide

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Machine Learning Cloud Getting Started Guide Machine Learning Cloud Getting started guide March 4th, 2020 Tübingen Outline • Who, what, how, … what again? o Our Cloud Infrastructure o Openstack, Storage, ... • General basics o SSH o DNS & TLS, … • Openstack basics: images, flavors, security groups, networks, floating IPs • Usage Terms, • VMs & Baremetal basics • How to create, connect and use • VM Hardening • Difference VM vs. Baremetal • GPUs (Driver, CUDA, within Docker) • Storage: Cinder, Quobyte, S3, WebDAV • Cloud-based services: Gitlab, Nextcloud, … • Docker • Support Who are we ... [email protected] [email protected] [email protected] [email protected] 07071 - 29 70282 Openstack – ML-Cloud Physical infrastructure ML-Cloud - TTR2 server room • 14x 1G Switches & 10x 40G Switches • 2x Provisioningmachines • 2x Loadbalancer • 3x Openstack controllers • 11x Storage servers (~1.8 PB, Quobyte software) • 12x GPU - Hypervisors with 4x Tesla V100 32GB RAM (NVLINK possible) • 7 Baremetal machines with 8x RTX 2080Ti 11GB RAM Insufficient cooling: Automatic server shutdown possibly without warning Openstack – what it is Most used open source IaaS platform o based on python, sql, messaging, http REST APIs, … o started by NASA & Rackspace, currently involved Companies: Red Hat/Ubuntu/Mirantis/... o commercially used (Telekom, …): Projects, Quotas o 2 major releases a year Multiple Services/Components o required: Keystone(Identity), Nova(Compute), Neutron(Network), Glance(Image) o optional: Horizon(Dashboard), Cinder(Blockstorage), Heat(Orchestration), Magnum(Container Orchestration), Barbican(Key Management), Swift(Objectstorage), … Some cons … o complex installation & configuration o sometimes confusing documentation o updates & long term support (-->Sardina) Openstack – How it works Openstack – ML-Cloud Storage - Quobyte LDAP Load - balancers Switches Openstack – how to use/access it • WebGUI: https://mlcloud.uni-tuebingen.de • Commandline tools: (apt|yum) install python-openstackclient openstack help • python modules (python-openstacksdk) • or the http API itself https://mlcloud.uni-tuebingen.de/identity/v3/auth/tokens POST { "auth": { "identity": { "methods": [ "password" ], "password": { "user": { "name": "admin", "domain": { "name": "Default" }, "password": "devstacker" } } } } } General Basics - SSH • secure shell • connecting to remote machine • Being able to work on remote machine as you were sitting right in front of it • Linux: $ ssh • Windows: use e.g Putty Client Generating ssh keys (Linux) • If you are using a Linux machine, open a terminal and enter the following: • $ ssh-keygen -t rsa -b 2048 • This creates a new ssh key. When you're prompted to "Enter a file in which to save the key," press Enter. • > Enter a file in which to save the key (/home/you/.ssh/id_rsa): [ Press Enter ] • At the next prompt, type a secure passphrase. • > Enter passphrase (empty for no passphrase): [Type a passphrase] • > Enter same passphrase again: [Type passphrase again] • Now you have created a public ssh key which can be found in /home/you/.ssh/id_rsa.pub • The public key is pasted into Openstack. Generating ssh keys (Windows) • If using Windows, use puttygen.exe to generate ssh keys. • Select "RSA" as type of key and enter "2048" or more for number of bits. • Click on generate. Generating ssh keys (Windows) • Enter a secure passphrase. • Click on "Save public key" to save the key on your disk. • Click on "Save private key" to save it in .ppk format on your disk. • You will use these keys with Putty to open a ssh session. • Now copy the generated public key and paste it into Openstack. General Basics - SSH You are using Windows: You are using Linux or MacOS: 1. Start Putty # generate key 2. Go to Session and type Hostname ssh-keygen -f cloud-key 3. Get IP from Openstack Dashboard 4. You can save a session and use it again # login ssh -i cloud-key centos@<IP from later. Dashboard> 5. Click Open to start ssh session 6. Type in the password for your ssh keypair when prompted Putty session [email protected] General Basics - port tunneling (OpenSSH) Route Network A Network B some host/jumphost • IP = 134.2.168.180 • Ssh connect via [email protected] VM Your workstation Machine/VM you want to login to --> localhost with Port 8444 open • Not in your network • IP = 192.168.212.46 • Connection to Website over HTTPS (Port 443) ssh –L PORT_LOCALHOST:IP_YOU_WANT_TO_BE_CONNECTED_WITH:PORT_ON_THAT_MACHINE jumphost • ssh –L 8444:192.168.212.46:443 [email protected] • Open a browser & enter: https://localhost:8444 General Basics - port tunneling (Putty) 1. Start Putty. 2. Go to Session and type Hostname/IP of the jump host. 3. You can save a session and use it again later. 4. Go to Connection > SSH > Tunnels 5. Enter a port of your choice in Source Port 6. Enter the destination address (the machine accessible from the jump host but not your local machine) along with the port on the destination server. 7. Click Add to add it to the list of tunnels. 8. Click Open to start ssh tunnel. 9. Type in the password for your ssh keypair when prompted. General Basics - Shell man <SOME_COMMAND> manual page for <SOME_COMMAND> whoami which user do I work as right now cd change directory ls list what is in present directory pwd print present working directory vim OR nano enter files and be able to edit them less enter file and just read cat concatenate files and print on the standard output sudo 'superuser do' for getting root privileges mv File or directory move cp file or directory copy rm remove a file mkdir make a new directory mount / umount mount/unmount a filesystem found on a device chmod / chown modify permissions/ownership of files or directories find find files in a directory hierarchy ip [a|r] print network adresse OR routes ... and many more Responsibilites Users deploying VMs must take care of them: VM owners must harden their VMs to avoid/prevent abuse. No password or ssh-key sharing! Errors, flaws or loopholes must be reported. All illegal activities will lead to legal measures. General Basics – DNS/TLS Domain Name Service • Translates IP addresses to domain names and vice versa • Not 1 DNS but many hierarchically ordered: *.mlcloud.uni-tuebingen.de controlled by ZDV networkers (DFN: no wildcards) • domain records only make real sense with TLS certificates Transport Layer Security • Secures network traffic • Use TLS 1.2 or 1.3: previous versions are unsafe/deprecated • TLS certificates used for HTTPS and other protocols(not ssh) • Provides authentification (and starts encryption) • Certificate are associated with one or more domains (incl. wildcards) • requireDNS entries • DFN > lets-encrypt > own CA > self-signed > no encrytion General Basics - Security SSH Keys • Use ssh keys with password, encrytion type = rsa and encryption depth ≥ 2048bit ssh-keygen -t rsa -b 4096 • Keep your private key secret! Do not share it with other people! Networks & Firewalls • Use whitelistingin every context with firewalls (security groups) --> block everything and allow only connections from and to Ports / IP-address-ranges you need Downloads • Downloads only from reliable sites, whenever possible use https • If available: check (GPG) signatures after downloadingrepos, software, etc. Regularly perform security updates Stop and deactivate services and containers you no longer need Openstack Basics Projects • Permissions • Users/Members • Ownership & Quotas Flavors Images Hardware setup of your VM Software setup of your VM • defines ressources --> • Linux distributionswe support: • CPUs • CentOS • GPUs (names) • Ubuntu LTS • RAM • we can create additional flavors on • Your own images demand. • Public/Private/Shared Openstack Basics - Networks local_network 192.168.0.0/24 We use ... • a physical 40G network: Openstack, Storage • VLANs for securing the physical network by dividing it into separate virtual networks. bmprov Openstack Networks 192.168.212.0/22 • VXLAN: self-service, storage limitations, no BM • FLAT/VLAN: provider - can be set up for groups and sensitive projects individually, fast qbstorage local (not routed) IPv4 networks 192.168.128.0/22 • 10.0.0.0/8: 10.0.0.0 – 10.255.255.255 • 172.16.0.0/12: 172.16.0.0 – 172.31.255.255 • 192.168.0.0/16: 192.168.0.0 – 192.168.255.255 osprovider public IPv4 network: osprovider 134.2.168.0/24 Openstack Basics - Access Floating IPs: "Direct" Access Jumphost: "Indirect" Access • IP of specific network: public/bmprov • Specific VM with a public IP for access • Quota: Each project has a limited amount • May have the only purpose of getting access to other VMs or baremetal machines you cannot • Can be allocated to a project reach directly for security or other reasons • Can be associated and disassociated to VMs: • Different access methods: The VM doesnt know about it 1. ssh to Jumphost via Floating IP, then ssh to internal machine • Used for direct access from remote host / 2. port forwarding different network 3. proxy internal ports onto public ip of the jumphost • Not working for baremetal machines Openstack Basics - Security Groups • Using security groups you restrict access to your VMs • A security group consists of multiple network access rules • A simplified firewall for VMs: filter source/protocols/ports • When you launch an instance, you can assign one or more security groups to it. • If you do not specify security groups, new instances are automatically assigned to the default security group. You are in charge to secure your VMs! • Create new security groups. • Add or delete rules to existing
Load more

Recommended publications
  • Légende : Montage Webdav/DAFVS2 [Tuto/Howto] [GNU/Linux] Montage
    Nebuleuse0rion Lost in the Digital Dimension Search… Quick links FAQ Register Login Index ‹ Techno ‹ Operating System ‹ Memory [Tuto/HowTo] Monter dossier distant sur Raspberry Pi & Ubuntu/Debian Post Reply Search this topic… 7 posts • Page 1 of 1 [Tuto/HowTo] Monter dossier distant sur Raspberry Pi & Ubuntu/Debian by voxdemonix » Mon Apr 25, 2016 12:45 pm P voxdemonix o s Posts: 830 t Joined: Mon Feb 02, 2015 7:28 pm Licence du tuto : - - L'ensemble de ces tutos fonctionne sur : Raspbian, (X|K|L)Ubuntu, Debian et tout ses dérivés Légende : 1. Montage Webdav/DAVFS 2. Montage Webdav/DAVFS via SSH Tunneling over Tor 3. Montage SSHFS 4. Montage SFTP 5. Monter un volume/répertoire GlusterFS 6. Montage Webdav/DAFVS2 cross-canal via SSH Tunneling Last edited by voxdemonix on Sun Dec 18, 2016 6:22 pm, edited 25 times in total. ~ Infernalis Creatorem ~ Rejoins le côté obscure, on a des cookies ! Donation Bitcoin : 1N8QGrhJGWdZNQNSspm3rSGjtXaXv9Ngat Donation Dash : XmowiBRku3tsEREp7PhpheY4TgeLLDyKdM T o Re: [Tuto/HowTo] Monter dossier distant sur Raspberry Pi & Ubuntu/Debian p by voxdemonix » Sat Apr 30, 2016 11:20 pm P o voxdemonix s Montage Webdav/DAFVS2 Posts: 830 t DAFVS2 est entre autre utilisé pour monter localement des dossiers distant hébergés sur un serveur owncloud/nextcloud. Il Joined: Mon Feb 02, 2015 7:28 pm utilise le protocole HTTP(S) se qui le rend plus lent (lag) mais compatible avec le Web. Si vous souhaitez effectuer le montage à travers internet (si vous êtes en adressage dynamique) il vous est conseillé d'utiliser un nom de domaine (ou plusieurs) et/ou un Tor Hidden Service.
    [Show full text]
  • Pipenightdreams Osgcal-Doc Mumudvb Mpg123-Alsa Tbb
    pipenightdreams osgcal-doc mumudvb mpg123-alsa tbb-examples libgammu4-dbg gcc-4.1-doc snort-rules-default davical cutmp3 libevolution5.0-cil aspell-am python-gobject-doc openoffice.org-l10n-mn libc6-xen xserver-xorg trophy-data t38modem pioneers-console libnb-platform10-java libgtkglext1-ruby libboost-wave1.39-dev drgenius bfbtester libchromexvmcpro1 isdnutils-xtools ubuntuone-client openoffice.org2-math openoffice.org-l10n-lt lsb-cxx-ia32 kdeartwork-emoticons-kde4 wmpuzzle trafshow python-plplot lx-gdb link-monitor-applet libscm-dev liblog-agent-logger-perl libccrtp-doc libclass-throwable-perl kde-i18n-csb jack-jconv hamradio-menus coinor-libvol-doc msx-emulator bitbake nabi language-pack-gnome-zh libpaperg popularity-contest xracer-tools xfont-nexus opendrim-lmp-baseserver libvorbisfile-ruby liblinebreak-doc libgfcui-2.0-0c2a-dbg libblacs-mpi-dev dict-freedict-spa-eng blender-ogrexml aspell-da x11-apps openoffice.org-l10n-lv openoffice.org-l10n-nl pnmtopng libodbcinstq1 libhsqldb-java-doc libmono-addins-gui0.2-cil sg3-utils linux-backports-modules-alsa-2.6.31-19-generic yorick-yeti-gsl python-pymssql plasma-widget-cpuload mcpp gpsim-lcd cl-csv libhtml-clean-perl asterisk-dbg apt-dater-dbg libgnome-mag1-dev language-pack-gnome-yo python-crypto svn-autoreleasedeb sugar-terminal-activity mii-diag maria-doc libplexus-component-api-java-doc libhugs-hgl-bundled libchipcard-libgwenhywfar47-plugins libghc6-random-dev freefem3d ezmlm cakephp-scripts aspell-ar ara-byte not+sparc openoffice.org-l10n-nn linux-backports-modules-karmic-generic-pae
    [Show full text]
  • Irods User Group Meeting 2016 Proceedings
    iRODS User Group Meeting 2016 Proceedings © 2016 All rights reserved. Each article remains the property of the authors. 8TH ANNUAL CONFERENCE SUMMARY The iRODS User Group Meeting of 2016 gathered together iRODS users, Consortium members, and staff to discuss iRODS-enabled applications and discoveries, technologies developed around iRODS, and future development and sustainability of iRODS and the iRODS Consortium. The two-day event was held on June 8th and 9th in Chapel Hill, North Carolina, hosted by the iRODS Consortium, with over 90 people attending. Attendees and presenters represented over 30 academic, government, and commercial institutions. Contents Listing of Presentations .......................................................................................................................................... 1 ARTICLES iRODS Audit (C++) Rule Engine Plugin and AMQP .............................................................................................. 5 Terrell Russell, Jason Coposky, Justin James - RENCI at UNC Chapel Hill Speed Research Discovery with Comprehensive Storage and Data Management .......................................... 21 HGST | PANASAS | iRODS Integrating HUBzero and iRODS ............................................................................................................................ 23 Rajesh Kalyanam, Robert A. Campbell, Samuel P. Wilson, Pascal Meunier, Lan Zhao, Elizabett A. Hillery, Carol Song - Purdue University An R Package to Access iRODS Directly ...........................................................................................................
    [Show full text]
  • The Evolution of File Systems
    The Evolution of File Systems Thomas Rivera, Hitachi Data Systems Craig Harmer, April 2011 SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA. Member companies and individuals may use this material in presentations and literature under the following conditions: Any slide or slides used must be reproduced without modification The SNIA must be acknowledged as source of any material used in the body of any document containing material from these presentations. This presentation is a project of the SNIA Education Committee. Neither the Author nor the Presenter is an attorney and nothing in this presentation is intended to be nor should be construed as legal advice or opinion. If you need legal advice or legal opinion please contact an attorney. The information presented herein represents the Author's personal opinion and current understanding of the issues involved. The Author, the Presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK. The Evolution of File Systems 2 © 2012 Storage Networking Industry Association. All Rights Reserved. 2 Abstract The File Systems Evolution Over time additional file systems appeared focusing on specialized requirements such as: data sharing, remote file access, distributed file access, parallel files access, HPC, archiving, security, etc. Due to the dramatic growth of unstructured data, files as the basic units for data containers are morphing into file objects, providing more semantics and feature- rich capabilities for content processing This presentation will: Categorize and explain the basic principles of currently available file system architectures (e.g.
    [Show full text]
  • Silenus – a Federated Service-Oriented Approach To
    SILENUS – A FEDERATED SERVICE-ORIENTED APPROACH TO DISTRIBUTED FILE SYSTEMS by MAXIMILIAN BERGER, B.S. A DISSERTATION IN COMPUTER SCIENCE Submitted to the Graduate Faculty of Texas Tech University in Partial Fulfillment of the Requirements for the Degree of DOCTOR OF PHILOSOPHY Approved Michael Sobolewski Chairperson of the Committee Noé López-Benitez Michael Shin Per Anderson Accepted John Borrelli Dean of the Graduate School December, 2006 Copyright © 2006 Maximlilian Berger ACKNOWLEDGEMENTS I would like to thank my Advisor Dr. Michael Sobolweski. This thesis could not have been done without his permanent effort to keep me on focused on the goal. I would also like to thank Dr. Noe Lopez-Benitez, Dr. Michael Shin, and Dr. Per Anderson for serving in my comittee. They could provide me with a different viewpoint. A very special thanks goes to Barbara Hartmann for being the greatest person on earth and for making our relationship work half way around the globe for three years. I would also like to thank my best friend and roommate Nathan Larson. He was always there to support, no matter how crazy the idea. ii TABLE OF CONTENTS ACKNOWLEDGMENTS ...................................................................................... ii ABSTRACT .......................................................................................................... viii I. INTRODUCTION .............................................................................................. 1 Problem Statement ........................................................................................
    [Show full text]
  • A Federated Service-Oriented Approach to Distributed File Systems
    Next Generation Concurrent Engineering - M. Sobolewski & P. Ghodous (eds) © 2005 ISPE, Inc., ISBN 0-9768246-0-4 SILENUS - A federated service-oriented approach to distributed file systems Max Berger & Michael Sobolewski SORCER Research Group, Texas Tech University A new approach for a distributed file system SILENUS based on the federated service-oriented computing environment SORCER is presented. Instead of services running on specific computers, the system is build from federating services in the network. The services are: byte-store service (Byzantium) for the storage of the actual date; metadata-store service (Midas) for information about the files; WebDAV adapter (Daphne) for support for legacy applications and operating system; and optimizer services that keep the collaborating federations in a healthy state. This system implements file-based data grid services in the SORCER environment that are needed for collaboration of distributed teams. 1 INTRODUCTION distributed services enabling for robust, secure, and Ever since the introduction of a personal computer shared vast repository of engineering data. there exists the problem of managing distributed data. Under the sponsorship of the National Institute for This data is usually stored on one host and not easily Standards and Technology (NIST) the Federated In- accessible for other users or from other hosts. telligent Product Environment (FIPER) (Sobolewski The current trend goes away from a personal com- 2002a, b, Kolonay & Sobolewski 2004) was devel- puter back to a networked one. We want to work any- oped (1999-2003) as one of the first service-to-service where, anytime, and of course always have the same (S2S) grid computing environments.
    [Show full text]
  • Technical Notes All Changes in Fedora 13
    Fedora 13 Technical Notes All changes in Fedora 13 Edited by The Fedora Docs Team Copyright © 2010 Red Hat, Inc. and others. The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. The original authors of this document, and Red Hat, designate the Fedora Project as the "Attribution Party" for purposes of CC-BY-SA. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version. Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix, Fedora, the Infinity Logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries. For guidelines on the permitted uses of the Fedora trademarks, refer to https:// fedoraproject.org/wiki/Legal:Trademark_guidelines. Linux® is the registered trademark of Linus Torvalds in the United States and other countries. Java® is a registered trademark of Oracle and/or its affiliates. XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries. All other trademarks are the property of their respective owners. Abstract This document lists all changed packages between Fedora 12 and Fedora 13.
    [Show full text]
  • 1. Why POCS.Key
    Symptoms of Complexity Prof. George Candea School of Computer & Communication Sciences Building Bridges A RTlClES A COMPUTER SCIENCE PERSPECTIVE OF BRIDGE DESIGN What kinds of lessonsdoes a classical engineering discipline like bridge design have for an emerging engineering discipline like computer systems Observation design?Case-study editors Alfred Spector and David Gifford consider the • insight and experienceof bridge designer Gerard Fox to find out how strong the parallels are. • bridges are normally on-time, on-budget, and don’t fall ALFRED SPECTORand DAVID GIFFORD • software projects rarely ship on-time, are often over- AS Gerry, let’s begin with an overview of THE DESIGN PROCESS bridges. AS What is the procedure for designing and con- GF In the United States, most highway bridges are budget, and rarely work exactly as specified structing a bridge? mandated by a government agency. The great major- GF It breaks down into three phases: the prelimi- ity are small bridges (with spans of less than 150 nay design phase, the main design phase, and the feet) and are part of the public highway system. construction phase. For larger bridges, several alter- There are fewer large bridges, having spans of 600 native designs are usually considered during the Blueprints for bridges must be approved... feet or more, that carry roads over bodies of water, preliminary design phase, whereas simple calcula- • gorges, or other large obstacles. There are also a tions or experience usually suffices in determining small number of superlarge bridges with spans ap- the appropriate design for small bridges. There are a proaching a mile, like the Verrazzano Narrows lot more factors to take into account with a large Bridge in New Yor:k.
    [Show full text]
  • Apache - Korean Usergroup Webdav : Collaboration Based on Apache
    Apache - Korean UserGroup WebDAV : Collaboration based on Apache 김규([email protected]) 하 CSA,SCNA,CCNA,CISSP Korea technical office - CollabNet,Inc Computer Eng dept. - Hansung University Apache Korean User Group Apache Korean Session Summary User Group ¾ 협업 — WebDAV 이전의 아파치 웹서버 ¾ WebDAV란? — WebDAV Benefits ¾ WebDAV Basics — mod_dav 소개 ¾ WebDAV 활용 — WebDAV softwares ¾ WebDAV 설치 및 운영 ¾ SubVersion이란? — SubVersion & CVS features ¾ SubVersion 설치 및 운영 — SubVersion Basic work scenario — SubVersion softwares ¾ Resources & References ¾ Q&A -2 - Collaborations based on Apache http://www.apache-kr.org Apache Korean 협업 ? User Group ¾ Collaboration ¾ 협업 소프트웨어들 — Collaborative groupwares — Groove — Collaborative tools — Collaboration systems ¾ 협업 소프트웨어의 기본적인 요건들 참여자들의 입장에 따라 보장되는 가시성 완벽하고 실제적인 보안성 다양한 배경의 사용자들을 위한 확장성 언제 어디서든 표준환경으로 참여할 수 있는 접근성 쉬운 사용으로 교육 없이 참여할 수 있는 사용성 -3 - Collaborations based on Apache http://www.apache-kr.org Apache Korean 협업 ? User Group -4 - Collaborations based on Apache http://www.apache-kr.org Apache Korean 협업 ? User Group Tool & Embedded Mobile Service Technology Software Phone Providers Providers Companies Manufacturers Java Libraries IDEs WAP Protocol 1 “2G” CDMA WebSphere iMODE Protocol 2 “2.5G” CDMA IDEs Libraries C# Libraries SKVM IDEs SK WITOP “3G” GSM Content Providers e.g., daum, Yahoo, ISVs Users -5 - Collaborations based on Apache http://www.apache-kr.org Apache Korean 협업 ? User Group ¾ OpenSource Community — 비영리적 오픈소스 커 뮤니티 — 기업 주도의 오픈소스 커뮤니티 -6 - Collaborations based on Apache http://www.apache-kr.org Apache Korean WebDAV 이전의 아파치 웹서버 인프라 User Group ¾ Apache : A patched web server — HTTP 프로토콜을 기반 —WEB을 전 인류에게.
    [Show full text]
  • Grid4all D4.8 User Manual
    Project no. 034567 Grid4All Specific Targeted Research Project (STREP) Thematic Priority 2: Information Society Technologies D4.8 User Manual Due date of deliverable: 01-06-2009 Actual submission date: 20-07-2009 Start date of project: 1 June 2006 Duration: 36 months Contributors : Antares, ICCS, INRIA, FT, KTH, SICS, UPC, UPRC Project co-funded by the European Commission within the Sixth Framework Programme (2002-2006) Dissemination Level PU Public PP Restricted to other programme participants (including the Commission Services) RE Restricted to a group specified by the consortium (including the Commission Services) CO Confidential, only for members of the consortium (including the Commission Services) D4.8 User Manual Grid4All-034567 Table of Contents Abbreviations used in this document ......................................................................................................... 2 Grid4All list of participants .......................................................................................................................... 3 1 Executive Summary .................................................................................................................................... 4 2 Introduction ................................................................................................................................................. 5 3 Core VO services ........................................................................................................................................ 7 3.1 Niche, a Distributed
    [Show full text]
  • EWB Denver Webdisk Manual Version 0.91
    EWB Denver Webdisk Manual version 0.91 Summary This manual is for EWB Denver members for use with the file storage capabilities that are available on our organization's website. Our current web hosting provider calls this service a webdisk. Technically speaking, it's more like a web-based file system than a disk or a folder. However, for the purposes of this manual the term webdisk will be used. This manual is broken into two main sections. First is the users perspective which covers how to use a webdisk, and explains the reasons for using a webdisk. The second section is more for the IT personnel and those tech savvy folks who can set up either the operating system interface, or install the applications that are webdisk-aware. Section 1 – User's Manual What is a Webdisk? A webdisk is a web accessible storage area. Because it uses a communications protocol related to that used for serving web pages, web browsers have built-in capabilities for traversing and reading a webdisk. Additionally, the open standard protocol used with a webdisk allows many operating systems to treat them like a local file systems. Validated users can use familiar operating system utilities, like a file manager, to manipulate a webdisk. Also, applications that are webdisk-aware will allow validated users to use the webdisk as a file storage area. Why use it at EWB Denver? The main purpose for the webdisk is as a temporary shared storage area for preparing project materials. Another is for archiving project materials for future reference.
    [Show full text]
  • A Model to Manage Shared Mutable Data in a Distributed Environment
    THE SEA OF STUFF: A MODEL TO MANAGE SHARED MUTABLE DATA IN A DISTRIBUTED ENVIRONMENT Simone Ivan Conte A Thesis Submitted for the Degree of PhD at the University of St Andrews 2018 Full metadata for this thesis is available in St Andrews Research Repository at: http://research-repository.st-andrews.ac.uk/ Please use this identifier to cite or link to this thesis: http://hdl.handle.net/10023/16827 This item is protected by original copyright This item is licensed under a Creative Commons Licence https://creativecommons.org/licenses/by-nc-nd/4.0/ The Sea of Stuff: a Model to Manage Shared Mutable Data in a Distributed Environment Simone Ivan Conte This thesis is submitted in partial fulfilment for the degree of Doctor of Philosophy (PhD) at the University of St Andrews August 2018 ii Abstract Managing data is one of the main challenges in distributed systems and computer science in general. Data is created, shared, and managed across heterogeneous distributed systems of users, services, applications, and devices without a clear and comprehensive data model. This technological fragmentation and lack of a common data model result in a poor understanding of what data is, how it evolves over time, how it should be managed in a distributed system, and how it should be protected and shared. From a user perspective, for example, backing up data over multiple devices is a hard and error-prone process, or synchronising data with a cloud storage service can result in conflicts and unpredictable behaviours. This thesis identifies three challenges in data management: (1) how to extend the current data abstractions so that content, for example, is accessible irrespective of its location, versionable, and easy to distribute; (2) how to enable transparent data storage relative to locations, users, applications, and services; and (3) how to allow data owners to protect data against malicious users and automatically control content over a distributed system.
    [Show full text]