DOCSLIB.ORG

To- #8208;Peer Software Distribu癢n in ALICE Grid Services to Enable

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
To- #8208;Peer Software Distribu癢n in ALICE Grid Services to Enable Employing peer-to-peer so0ware distribu6on in ALICE Grid Services to enable opportunis6c use of OSG resources R Jeff Porter & Iwona Sakrejda (LBNL) Cosn Grigoras, Latchezar Betev, Federico Carmina, Pablo Saiz (CERN) Outline • Introduc6on – ALICE Grid & AliEn – Tradi6onal So0ware Deployment • BitTorrent so0ware deployment AliTorrent • Tes6ng on OSG resources • Performance characteris6cs • Target mul6ple resources • Summary .R. Jeff Porter LBNL 2 AliEn & the ALICE Grid • ALICE Grid Facility – In produc6on since ~2004 – more than 80 sites – Small, central operaons team • & site admins around the world • AliEn: ‘Alice Environment’ – Central Services at CERN ALICE Compu6ng Grid • Task Queue, Job & File Catalog • So0ware management (build & deployment) this – VO box site-specific operaons talk • Job Agent submission • So0ware deployment • Resource monitoring – Grid Monitoring with MonALisa – Data Management • AliEn FileCatalog • Grid-Enabled XRootD SEs Yesterday’s talk by Pablo Saiz: “AliEn.R. Jeff Porter LBNL : ALICE Environment on the GRID” 3 Tradi6onal So0ware Deployment • Managed Central So0ware: Central Build Servers – rou6ne so0ware builds AliEn, AliRoot, … – Catalogued & stored in AliEn SLC MacOS Ubuntu • Grid Site SW Operaons – Jobs request SW from VO box service Catalogue ALICE:CERN:SE – VO box PackMan service pulls SW AliEn – SW deployed on shared area – WNs read SW from shared area ALICE Grid Site VO Shared SW area Box – resource bofleneck requests WW N W – single point of failure N WW N N – can require ac6ve repairs per site Shared SW Area N WN NFS/GPFS/AFS .R. Jeff Porter LBNL 4 Basic Torrent details File chunks contain hashes of original file to provide data integrity C.Grigoras, ALICE T1/T2 Workshop Tracker: map of seeders:files Seeders: have & serve file Leeches: pull & serve file chunks .R. Jeff Porter LBNL 5 AliTorrent So0ware Deployment • Managed Central So0ware Central Build Servers – Addi6onal AliEn torrent store AliEn, AliRoot, … – Catalogue, seeder & tracker SLC MacOS Ubuntu • Grid site SW deployment – VO Box is not involved Catalogue alitorrent.cern.ch – Seeder | Tracker Jobs pull SW from: AliEn • alitorrent.cern.ch seeder • local peers • other sites as available ALICE Grid Site – though typically behind a FW requests W VO W • Resolves: N W Box N W – Bofleneck & single point failures N W N N WN – Site level maintenance of shared area .R. Jeff Porter LBNL 6 AliTorrent Details • Torrent Features – Distributed Hash Tables • Decentralized seeder lookup – seeders are trackers – Peer Exchange • Local peer informaon is propagated by seeders – Local Peer Discovery • Mul6cast to discover peers on same network • ALICE/AliEn Features – Total so0ware download is ~300-400MB – Enabled per site (VO box) with AliEn LDAP flag .R. Jeff Porter LBNL 7 AliTorrent as an opportunity • AliTorrent use in AliEn – Reduces problems associated with SW deployment – Simplifies site operaons by removing a VO box service Does not eliminate VO box model from ALICE Grid Does eliminate site-specific VO box requirement • Eliminaon of site-specific VO box allows for remote use of other Grid resources OSG .R. Jeff Porter LBNL 8 AliEn-OSG Interface • Developed in 2009 by ALICE & OSG teams – in produc6on at ALICE-US sites since 2009 – poster @ CHEP’09 • Features added to AliEn – VO box as a Condor-G submit host to local OSG CE – Condor module in AliEn • Builds submit files to launch JobAgents • monitors Condor queue for site u6lizaon – Specificaons put into AliEn LDAP • OSG-CE endpoint • Job occupancy targets .R. Jeff Porter LBNL 9 Extending OSG-AliEn Interface for AliTorrent • Without torrent Shared SW area #!/bin/bash /project/projectdirs/alice/alicedev/alien.v2-19.168/bin/alien RunAgent • With torrent #!/bin/bash DIR=`pwd`/alien_installaon.$$ job sandbox mkdir -p $DIR cd $DIR wget hp://alien.cern.ch/alien-installer -O alien-auto-installer chmod +x alien-auto-installer ./alien-auto-installer -type workernode -batch -torrent -install-dir $DIR/alien ./alien/bin/alien RunAgent • With torrent on OSG: • replace `pwd` above with $OSG_WN_TMP DIR= /ALICE/alien_installaon.$$ OSG TMP space .R. Jeff Porter LBNL 10 First Results • Ini6al goal to target NERSC Carver system – Carver: NERSC IBM IDATAPLEX HPC system with OSG CE – ALICE obtained a small allocaon for development work – No VO Box would allowed on the system • Conferred with NERSC security about torrent use – earlier requests for use of torrent were ‘tabled’ for review – Result: torrent not banned explicitly but NERSC relies on dynamic filtering “go ahead and try” ---- log file ---- It just worked WNHost = "c1554.nersc.gov"; Geng the torrent It looks like the default IP address is 128.55.61.240 Seeding the torrent Extrac6ng the files Tes6ng /osg/carver/wn_tmp/ALICE/alien_installaon.20613/alien.v2-19.169/ AliEn workernode installaon took approximately 133 seconds Installaon finished! .R. Jeff Porter LBNL 11 Performance • Job latency from SW install: Tradi6onal vs AliTorrent – Tradi6onal install: • ‘copy’ from shared area - torrent install • lone job latency: ~1 sec - ‘cp’ from shared area • steady increase vs #-jobs – AliTorrent install: • CERN to LBNL/NERSC • lone job latency: ~360 sec • sharp decrease vs #-jobs Install 6me (seconds) increase with #-jobs • AliTorrent advantage Number of concurrent jobs – small numbers of long jobs where SW install 6me is insignificant – sites with large numbers of concurrent ALICE jobs .R. Jeff Porter LBNL 12 Expand to OSG Sites • Single VO box submission to OSG pilot job factory Gridresources \ – OSG-CE Endpoints carvergrid.nersc.gov/jobmanager-pbs:50:(jobType=single)(queue=serial) \ itbv-ce-pbs.uchicago.edu/jobmanager-pbs:5:none \ • Managed as a list glcc88.ucllnl.org/jobmanager-pbs:20:none \ osgitb1.nhn.ou.edu/jobmanager-condor:5:none \ • Endpoint:MaxJobs:RSL pdsfgrid.nersc.gov/jobmanager-sge:50:none – Test on selected sites • OSG-ITB sites • ALICE-US sites LBNL & LLNL – Close SE 700 TB SE @ NERSC • output des6naon NERSC/PDSF 10592 gt2 pdsfgrid.nersc.gov/jobmanager-sge hps://pdsfgrid4.nersc.gov:46490/25251/1337447241/ 10628 gt2 glcc88.ucllnl.org/jobmanager-pbs hfps://glcc88.ucllnl.org:40894/8704/1337454774/ LLNL 10629 gt2 glcc88.ucllnl.org/jobmanager-pbs hps://glcc88.ucllnl.org:40902/9292/1337454810/ 10645 gt5 itbv-ce-pbs.uchicago.edu/jobmanager-pbs hfps://itbv-ce-pbs.uchicago.edu:36913/3026418950974269075/ UC-ITB 10647 gt5 itbv-ce-pbs.uchicago.edu/jobmanager-pbs hfps://itbv-ce-pbs.uchicago.edu:36913/3026418950974269075/ 10664 gt2 carvergrid.nersc.gov/jobmanager-pbs hfps://carvergrid.nersc.gov:60888/2359/1337525731/ NERSC/Carver 10665 gt2 carvergrid.nersc.gov/jobmanager-pbs hfps://carvergrid.nersc.gov:60891/2358/1337525731/ 10674 gt5 osgitb1.nhn.ou.edu/jobmanager-condor hfps://osgitb1.nhn.ou.edu:64981/933117525531598993/ OU-ITB 10675 gt5 osgitb1.nhn.ou.edu/jobmanager-condor hfps://osgitb1.nhn.ou.edu:64981/933117525531598993/ .R. Jeff Porter LBNL 13 Towards produc6on operaons • Proof in principal is complete – AliEn Jobs run at sites submifed by remote VO box – successful running on mulple sites from single VO box – Plan to target OSG produc6on sites • Site policy cau6on: – limits can be broad – has hampered wider use • ALICE Experience: – ini6al distrust turns into recogni6on of approach’s merit – simplifies site operaons .R. Jeff Porter LBNL 14 Summary • So0ware deployment on shared area – Bofleneck & site-level single point failure – site-level SW corrup6on requires admin interven6on • Torrent model AliTorrent – Removes bofleneck & site-level single point of failure – Eliminates a site service & reduces site management – Performance capabili6es meets typical ALICE workflow & site requirements Eliminates requirement for site-specific VO box • We have leveraged this capability to demonstrate AliEn workflow for opportunis6c use of mul6ple OSG resources • AliTorrent is a site-friendly tool for opportunis6c (or general) use – don’t ask the site to “do” something install or manage a service – ask the site to “not do” something block torrent use .R. Jeff Porter LBNL 15 .
Load more

Recommended publications
  • ஜ Torrent Is Not Seeding ஜ Скачать: Torrent Is Not Seeding
    ▬▬▬▬▬▬▬ஜ Torrent is not seeding ஜ▬▬▬▬▬▬▬ Скачать: ➤ Torrent is not seeding Download: ➤ Torrent is not seeding ▬▬▬▬▬▬▬ஜ Torrent is not seeding ஜ▬▬▬▬▬▬▬ . Torrent is not seeding We are a thriving community dedicated to helping users old and new understand and use torrents. I think is stable enough. A peer or downloader becomes a seed when it starts uploading the already downloaded content for other peers to download from. Do not send - not taken into account. Please check the Accepted clients list. When it completes you switch to a torrent is not seeding and dedicate that stream to simply uploading. My upload speed limit should be all I need instead of limiting upload slots. Then add the port you selected in step 5. Torrent files contain information like the file list, sizes, pieces, etc. Every piece received is first checked against the hash. If yes, please state the libtorrent version used by your distro. However, whether to seed or not, or how much to seed, depends on the availability of downloaders and the choice of the peer at the seeding end. I tried raising my connection limits to higher and higher numbers currently at 1000, 500, 500, 500 I have a static port so I can forward it. For legal torrents try. I advice you to use only one active seeding torrent when capturing. Current settings: DHT: off PeX: off Local peer discovery: on Anonymous mode: off Max downloads: 5 Max uploads: 7 Max active: 12 Do not count slow torrents in these limits: on no see ratio limits set. Each seed adds 1.
    [Show full text]
  • The Hidden Locality in Swarms
    13-th IEEE International Conference on Peer-to-Peer Computing The hidden locality in swarms John S. Otto and Fabian´ E. Bustamante Northwestern University {jotto,fabianb}@eecs.northwestern.edu Abstract—People use P2P systems such as BitTorrent to share We overcome the challenges of local peer discovery by an unprecedented variety and amount of content with others leveraging diurnal patterns and applying client-side techniques around the world. The random connection pattern used by to improve overall peer discovery. BitTorrent has been shown to result in reduced performance for users and costly cross-ISP traffic. Although several client-side Through an analysis of swarm population dynamics, we systems have been proposed to improve the locality of BitTorrent show that locality is present in swarms – if one looks at the traffic, their effectiveness is limited by the availability of local right time. For popular content swarms, 50% of ISPs seen in peers. the swarm have at least five local peers online during the ISP’s We show that sufficient locality is present in swarms – if peak hour. During an ISP’s peak hour, the relative fraction of one looks at the right time. We find that 50% of ISPs have at least five local peers online during the ISP’s peak hour, local peers – and therefore the local peer discovery rate – is typically in the evening, compared to only 20% of ISPs during typically 50% higher than the daily average. the median hour. To better discover these local peers, we show We evaluate client-side techniques that boost the peer how to increase the overall peer discovery rate by over two orders discovery rate by two orders of magnitude, enabling peers of magnitude using client-side techniques: leveraging additional to quickly discover online local peers.
    [Show full text]
  • User Manual - Documentationcase - Help3:11-Cv-00651-IEG - Bittorrent - Delivering The-JMA World's Documentcontent 26-1 Filed 12/05/11 Page 2 of 4 11/30/11 3:15 PM
    Case 3:11-cv-00651-IEG -JMA Document 26-1 Filed 12/05/11 Page 1 of 4 EXHIBIT A User Manual - DocumentationCase - Help3:11-cv-00651-IEG - BitTorrent - Delivering the-JMA World's DocumentContent 26-1 Filed 12/05/11 Page 2 of 4 11/30/11 3:15 PM English Get Started Features Community Help Free Download Peers Help BitTorrent User Manual Appendix A: The BitTorrent Interface Main Window Detailed Info Pane FAQ Peers The Peers tab is where you'll find information about all the peers you are currently connected to on Videos & Guides the selected torrent. The following is a description of each column: IP displays the peer's IP address and the flag of the country they're from. If Resolve IPs is enabled, Forums the country flag and hostname will be displayed instead. % displays the approximate percent of the torrent job the peer has completed. User Manual Client displays the BitTorrent client that the peer reports it is using. If BitTorrent suspects the client Chapter 01: Introduction to of spoofing (faking) its client ID, it will note it as such. BitTorrent Debug displays information that might be useful in debugging problems. Chapter 02: Basic Guides Down Speed is the averaged speed at which you are downloading data from the peer. Chapter 03: Advanced Guides Downloaded is the amount of data you have downloaded from the peer since the torrent job was Appendix A: The BitTorrent added. Interface Flags displays various letters, each carrying a special meaning about the state of the connection: Appendix B: Advanced ?: your client unchoked the peer but
    [Show full text]
  • Guidelines for the Secure Deployment of Ipv6
    Special Publication 800-119 Guidelines for the Secure Deployment of IPv6 Recommendations of the National Institute of Standards and Technology Sheila Frankel Richard Graveman John Pearce Mark Rooks NIST Special Publication 800-119 Guidelines for the Secure Deployment of IPv6 Recommendations of the National Institute of Standards and Technology Sheila Frankel Richard Graveman John Pearce Mark Rooks C O M P U T E R S E C U R I T Y Computer Security Division Information Technology Laboratory National Institute of Standards and Technology Gaithersburg, MD 20899-8930 December 2010 U.S. Department of Commerce Gary Locke, Secretary National Institute of Standards and Technology Dr. Patrick D. Gallagher, Director GUIDELINES FOR THE SECURE DEPLOYMENT OF IPV6 Reports on Computer Systems Technology The Information Technology Laboratory (ITL) at the National Institute of Standards and Technology (NIST) promotes the U.S. economy and public welfare by providing technical leadership for the nation’s measurement and standards infrastructure. ITL develops tests, test methods, reference data, proof of concept implementations, and technical analysis to advance the development and productive use of information technology. ITL’s responsibilities include the development of technical, physical, administrative, and management standards and guidelines for the cost-effective security and privacy of sensitive unclassified information in Federal computer systems. This Special Publication 800-series reports on ITL’s research, guidance, and outreach efforts in computer security and its collaborative activities with industry, government, and academic organizations. National Institute of Standards and Technology Special Publication 800-119 Natl. Inst. Stand. Technol. Spec. Publ. 800-119, 188 pages (Dec. 2010) Certain commercial entities, equipment, or materials may be identified in this document in order to describe an experimental procedure or concept adequately.
    [Show full text]
  • Technology Stack for Decentralized Mobile Services
    Technology Stack for Decentralized Mobile Services Matouš Skála Technology Stack for Decentralized Mobile Services by Matouš Skála to obtain the degree of Master of Science at the Delft University of Technology, to be defended publicly on Monday August 31, 2020 at 3:00 PM. Student number: 4893964 Project duration: November 15, 2019 – August 31, 2020 Thesis committee: Dr.ir. J.A. Pouwelse, TU Delft, supervisor Dr. J.S. Rellermeyer, TU Delft Dr. N. Yorke-Smith, TU Delft An electronic version of this thesis is available at http://repository.tudelft.nl/. Preface When I was choosing my thesis topic, I originally came up with an idea of designing a decen- tralized social network. After realizing how ambitious that goal was, I later decided to focus on more fundamental issues first and create a library that would allow for building any de- centralized applications, running purely on an overlay network consisting of smartphones. Rather than reinventing the wheel, I took inspiration from an existing networking library de- veloped at TU Delft over the last decade and created its wire-compatible implementation in Kotlin. Interestingly, in the end, I have even implemented a trivial social network to demon- strate the usage of the library, returning back to the original idea. I would like to thank my supervisor Johan Pouwelse for an endless stream of fresh ideas and valuable feedback, and to PhD students of the Delft Blockchain Lab for numerous coffee meetings and for serving me as a walking documentation of the existing codebase. Matouš Skála Prague,
    [Show full text]
  • CSC 634: Networks Programming
    CSC 634: Networks Programming Lecture 08: Peer to Peer Computing Instructor: Haidar M. Harmanani Traditional Internet Services Model § Client-server Many clients, 1 (or more) server(s) Web servers, DNS, file downloads, video streaming § Problems Scalability: how many users can a server support? o What happens when user traffic overload servers? o Limited resources (bandwidth, CPU, storage) Reliability: if # of servers is small, what happens when they break, fail, get disconnected, are mismanaged by humans? Efficiency: if your users are spread across the entire globe, how do you make sure you answer their requests quickly? Spring 2018 Network Programming 2 The Alternative: Peer-to-Peer § A simple idea Users bring their own resources to the table A cooperative model: clients = peers = servers § The benefits Scalability: # of “servers” grows with users o BYOR: bring your own resources (storage, CPU, B/W) Reliability: load spread across many peers o Probability of them all failing is very low… Efficiency: peers are distributed o Peers can try and get service from nearby peers Spring 2018 Network Programming 3 The Peer-to-Peer Challenge § What are the key components for leveraging P2P? Communication: how do peers talk to each other Service/data location: how do peers know who to talk to § New reliability challenges Network reachability, i.e. dealing with NATs Dealing with churn, i.e. short peer uptimes § What about security? Malicious peers and cheating The Sybil attack Spring 2018 Network Programming 4 Centralized Approach
    [Show full text]
  • Maximum Number of Utorrent
    Maximum number of utorrent click here to download Sky Chan is correct on the settings location, but I think you are asking what those settings should be set to. What the maximum number of active torrents should be set to depends on the speed of your internet connection, particularly the upload speed. Spreading your upload effectively will help your download speeds. But I have 19 active torrents and 9 active downloads (status is "downloading" or "seeding"). Because uTorrent is switching from one torrent to another, I get "torrents limit reached" errors. You really should queue downloads and seeds in respect to the following options: Maximum Maximum number of active torrents overrided (uT The number of connections is the most essential part of the entire optimization process, as it controls the global maximum number of connections, the maximum number of connected peers per torrent and the number of upload slots per torrent. Power users are recommended to use a /50/4 configuration. There are certain ways to tweak your uTorrent settings and optimize your connection for maximum download speeds. BitTorrent is a P2P protocol. Unlike HTTP there is no centralized server distributing data. Instead, files and folders are stored on regular PCs just like yours. Recommended: Top 5 Torrent. This video shows how to put a limit on the maximum no. of active torrents in uTorrent/ Bittorrent. Linnet's How To _ Remember to like and subscribe See all my videoes in playlist / categories here https://www. The Maximum number of connections is the most important part to increase uTorrent download speed.
    [Show full text]
  • Bittorrent and Bittyrant CS 344G 2/9/2015 Gregory D
    BitTorrent and BitTyrant CS 344G 2/9/2015 Gregory D. Hill BitTorrent Overview Download .torrent file or use magnet URI to get file hash, required metadata Get partial list of peers from Tracker or Kademlia-based DHT Download file from peers following peer protocol (uses TCP) 10 message types described in following slides keep-alive choke/unchoke Interested/not interested have bitfield request piece cancel Basic BitTorrent Protocol Handshake with peer, after send bitfield(length, bitfield) message which has a bit for each piece of the file representing whether that peer has that piece or not have(piece index) message – tell a peer you have piece x request(piece index, byte offset, length) to a peer hopefully reciprocated by a piece(piece index, byte offset, data) which can be aborted with a cancel(piece index, byte offset, length) Send interested() to a peer that has data you want and undo this by sending them not interested() Piece Selection First piece: choose at random until a complete piece downloaded Normal: Rarest First, try to download piece fewest peers have Endgame: on last pieces, request all subpieces left from all peers, send cancels to others when data arrives BitTorrent Choking Refuse to upload to, or choke, all but a few (commonly 4) peers Upload even share to these unchoked peers Prefer to give to peers who upload most to you (tit-for-tat) Every 10 seconds decide which mutually interested peer to unchoke based on best rolling 20 second average of download rate from peers If no reciprocation from an
    [Show full text]
  • Forensic Investigation of P2P Cloud Storage Services and Backbone For
    Forensic investigation of P2P cloud storage services and backbone for IoT networks : BitTorrent Sync as a case study Yee Yang, T, Dehghantanha, A, Choo, R and Yang, LK http://dx.doi.org/10.1016/j.compeleceng.2016.08.020 Title Forensic investigation of P2P cloud storage services and backbone for IoT networks : BitTorrent Sync as a case study Authors Yee Yang, T, Dehghantanha, A, Choo, R and Yang, LK Type Article URL This version is available at: http://usir.salford.ac.uk/id/eprint/40497/ Published Date 2016 USIR is a digital collection of the research output of the University of Salford. Where copyright permits, full text material held in the repository is made freely available online and can be read, downloaded and copied for non-commercial private study or research purposes. Please check the manuscript for any further copyright restrictions. For more information, including our policy and submission procedure, please contact the Repository Team at: [email protected]. Note: This is authors accepted copy – for final article please refer to International Journal of Computers & Electrical Engineering Forensic Investigation of P2P Cloud Storage: BitTorrent Sync as a Case Study 1 2 3 1 Teing Yee Yang , Ali Dehghantanha , Kim-Kwang Raymond Choo , Zaiton Muda 1 Department of Computer Science, Faculty of Computer Science and Information Technology, Universiti Putra Malaysia, UPM Serdang, Selangor, Malaysia 2 The School of Computing, Science & Engineering, Newton Building, University of Salford, Salford, Greater Manchester, United Kingdom 3 Information Assurance Research Group, University of South Australia, Adelaide, South Australia, Australia. Abstract Cloud computing has been regarded as the technology enabler for the Internet of Things (IoT).
    [Show full text]
  • CS 3700 Networks and Distributed Systems
    CS 3700 Networks and Distributed Systems Lecture 18: Peer-to-Peer Systems Revised 3/23/13 2 Outline ❑ Peer-to-Peer Overview ❑ Example: Bittorrent ❑ µTP: Micro Transport Protocol ❑ Cheating on BitTorrent Traditional Internet Services Model 3 Client-server ! Many clients, 1 (or more) server(s) ! Web servers, DNS, file downloads, video streaming Problems ! Scalability: how many users can a server support? ■ What happens when user traffic overload servers? ■ Limited resources (bandwidth, CPU, storage) ! Reliability: if # of servers is small, what happens when they break, fail, get disconnected, are mismanaged by humans? ! Efficiency: if your users are spread across the entire globe, how do you make sure you answer their requests quickly? The Alternative: Peer-to-Peer 4 A simple idea ! Users bring their own resources to the table ! A cooperative model: clients = peers = servers The benefits ! Scalability: # of “servers” grows with users ■ BYOR: bring your own resources (storage, CPU, B/W) ! Reliability: load spread across many peers ■ Probability of them all failing is very low… ! Efficiency: peers are distributed ■ Peers can try and get service from nearby peers The Peer-to-Peer Challenge 5 What are the key components for leveraging P2P? ! Communication: how do peers talk to each other ! Service/data location: how do peers know who to talk to New reliability challenges ! Network reachability, i.e. dealing with NATs ! Dealing with churn, i.e. short peer uptimes What about security? ! Malicious peers and cheating ! The Sybil attack Centralized
    [Show full text]
  • Distributed Caching Using the Htcondor Cached Derek J
    CORE Metadata, citation and similar papers at core.ac.uk Provided by DigitalCommons@University of Nebraska University of Nebraska - Lincoln DigitalCommons@University of Nebraska - Lincoln Holland Computing Center -- Faculty Publications Holland Computing Center Summer 7-27-2015 Distributed Caching Using the HTCondor CacheD Derek J. Weitzel University of Nebraska-Lincoln, [email protected] Brian Bockelman University of Nebraska-Lincoln, [email protected] David Swanson University of Nebraska - Lincoln, [email protected] Follow this and additional works at: http://digitalcommons.unl.edu/hollandfacpub Part of the Numerical Analysis and Scientific omputC ing Commons Weitzel, Derek J.; Bockelman, Brian; and Swanson, David, "Distributed Caching Using the HTCondor CacheD" (2015). Holland Computing Center -- Faculty Publications. 4. http://digitalcommons.unl.edu/hollandfacpub/4 This Article is brought to you for free and open access by the Holland Computing Center at DigitalCommons@University of Nebraska - Lincoln. It has been accepted for inclusion in Holland Computing Center -- Faculty Publications by an authorized administrator of DigitalCommons@University of Nebraska - Lincoln. Distributed Caching Using the HTCondor CacheD Derek Weitzel, Brian Bockelman, and David Swanson Computer Science and Engineering University of Nebraska – Lincoln Lincoln, Nebraska 68588 Email: [email protected] Abstract—A batch processing job in a distributed system has Bosco allowed campus researchers to submit high throughput three clear steps, stage-in, execution, and stage-out. As data sizes jobs to high performance clusters. We extend Bosco to include have increased, the stage-in time has also increased. In order data caching and novel data transfer methods. to optimize stage-in time for shared inputs, we propose the CacheD, a caching mechanism for high throughput computing.
    [Show full text]
  • Analysis of Bandwidth Attacks in a Bittorrent Swarm
    City Research Online City, University of London Institutional Repository Citation: Adamsky, Florian (2016). Analysis of bandwidth attacks in a bittorrent swarm. (Unpublished Doctoral thesis, City, University of London) This is the accepted version of the paper. This version of the publication may differ from the final published version. Permanent repository link: https://openaccess.city.ac.uk/id/eprint/16158/ Link to published version: Copyright: City Research Online aims to make research outputs of City, University of London available to a wider audience. Copyright and Moral Rights remain with the author(s) and/or copyright holders. URLs from City Research Online may be freely distributed and linked to. Reuse: Copies of full items can be used for personal research or study, educational, or not-for-profit purposes without prior permission or charge. Provided that the authors, title and full bibliographic details are credited, a hyperlink and/or URL is given for the original metadata page and the content is not changed in any way. City Research Online: http://openaccess.city.ac.uk/ [email protected] florian e. w. adamsky ANALYSISOFBANDWIDTHATTACKSINA BITTORRENTSWARM ANALYSISOFBANDWIDTHATTACKSINABITTORRENT SWARM florian e. w. adamsky A thesis submitted for the degree of Doctor of Philosophy (Information Engineering) School of Mathematics, Computer Science & Engineering City University London August 2016 – version 1.1 Florian E. W. Adamsky: Analysis of Bandwidth Attacks in a BitTorrent Swarm, A thesis submitted for the degree of, Doctor of Philosophy (Information Engineering), c August 2016 supervisors: Prof. Dr. Muttukrishnan Rajarajan Dr. Syed Ali Khayam local supervisor: Prof. Dr. Rudolf Jäger Ohana means family. Family means nobody gets left behind, or forgotten.
    [Show full text]