An Asynchronous Collaboration and Content Management Framework with High-Performance P2P-based Data Transfer Capability for Scientific Computing Ali Kaplan Submitted to the faculty of the University Graduate School in partial fulfillment of the requirements for the degree Doctor of Philosophy in the Department of Computer Science, Indiana University July 2007 i Accepted by the Graduate Faculty, Indiana University, in partial fulfillment of the requirements for the degree of Doctor of Philosophy. Doctoral Committee _______________________________ Prof. Geoffrey C. Fox (Principal Advisor) ________________________________ Prof. Dennis Gannon ________________________________ Prof. ________________________________ Prof. 28, 2008 ii © 2008 Ali Kaplan All Rights Reserved iii Acknowledgements iv Abstract v Table of Contents Chapter 1 ‐ Introduction ..................................................................................................... 1 1.1 Motivation ......................................................................................................................... 4 1.2 Use Cases .......................................................................................................................... 5 1.3 Research Issues .................................................................................................................. 6 1.4 Contributions ..................................................................................................................... 7 1.5 Organization of the Thesis ................................................................................................. 9 Chapter 2 ‐ Literature Survey .......................................................................................... 12 2.1 Introduction ..................................................................................................................... 12 2.2 Overview ......................................................................................................................... 14 2.3 System-level Data Movement Techniques ...................................................................... 16 2.4 Application-level Data Movement Techniques ............................................................... 17 2.4.1 TCP -based Data Movement Techniques ................................................................... 17 2.4.1.1 GridFTP ............................................................................................................................................. 18 2.4.1.2 GridHTTP .......................................................................................................................................... 22 2.4.1.3 bbFTP ................................................................................................................................................ 22 2.4.1.4 The BaBar Copy Program (bbcp) ....................................................................................................... 23 2.4.2 UDP -based Data Movement Techniques ................................................................... 23 2.4.2.1 Simple Available Bandwidth Utilization Library (SABUL) .............................................................. 24 2.4.2.2 UDP-based Data Transfer Protocol (UDT) ........................................................................................ 24 2.4.2.3 Fast Object –Based data transfer System (FOBS) .............................................................................. 25 2.4.2.4 Reliable Blast UDP (RBUDP) ........................................................................................................... 26 2.4.2.5 Tsunami ............................................................................................................................................. 26 vi 2.4.2.6 UFTP ................................................................................................................................................. 27 2.5 Peer-to-Peer based Data Movement Techniques ............................................................. 27 2.5.1 BitTorrent ................................................................................................................... 30 2.6 Network-level Data Transfer Techniques ........................................................................ 34 2.7 Discussion ....................................................................................................................... 37 Chapter 3 ‐ The GridTorrent Framework Architecture ............................................... 39 3.1 The Overview of the GridTorrent Framework ................................................................ 39 3.2 Main Components of the GridTorrent Framework .......................................................... 42 3.3 Summary ......................................................................................................................... 46 Chapter 4 ‐ The GridTorrent Framework Client Architecture .................................... 48 4.1 Introduction ..................................................................................................................... 48 4.2 Overview of the GridTorrent Framework Client Architecture ........................................ 50 4.3 Torrent Data Sharing Logic ............................................................................................. 52 4.4 Core Modules Layer ........................................................................................................ 53 4.4.1 Data Transfer Modules ............................................................................................... 53 4.4.2 Management Modules ................................................................................................ 55 4.4.2.1 Task Manager .................................................................................................................................... 56 4.4.2.2 WS-Tracker Client ............................................................................................................................. 56 4.5 Security Manager ............................................................................................................. 58 4.6 Task Generation and Exchange ....................................................................................... 59 4.6.1 Tasks ........................................................................................................................... 60 4.6.1.1 Task List Request Task ...................................................................................................................... 63 4.6.1.2 Share Content Request Task .............................................................................................................. 64 4.6.1.3 Share Content Response Task ............................................................................................................ 65 4.6.1.4 Download Content Request Task ....................................................................................................... 66 4.6.1.5 Download Content Response Task .................................................................................................... 67 vii 4.6.1.6 Access Control List Request Task ..................................................................................................... 68 4.6.1.7 Access Control List Response Task ................................................................................................... 69 4.6.1.8 Update Status Task: (UPT) ................................................................................................................ 70 4.7 Summary ......................................................................................................................... 71 Chapter 5 ‐ Asynchronous Collaboration and Content Management Architecture ... 72 5.1 Introduction ..................................................................................................................... 72 5.2 The concept of Asynchronous Collaboration and Content Management ........................ 73 5.2.1 Definition of term “Asynchronous Collaboration” ..................................................... 73 5.2.2 Definition of term “Content” and “Content Management” ........................................ 74 5.2.3 Requirements .............................................................................................................. 74 5.3 Related Work ................................................................................................................... 75 5.4 Access Control Schemes ................................................................................................. 81 5.5 The Collaboration and Content Management .................................................................. 83 5.5.1 Objects of the System ................................................................................................. 85 5.5.2 Services of the System ................................................................................................ 87 5.5.3 Collaboration Management Module ........................................................................... 90 5.5.4 Access Control Mechanism of CCM .......................................................................... 92 5.5.5 Content Management Module .................................................................................... 93 5.6 Summary ........................................................................................................................