IMPROVING THE COMMUNICATION PERFORMANCE OF DISTRIBUTED ANIMATION RENDERING USING BITTORRENT FILE SYSTEM ASST. PROF. NAMFON ASSAWAMEKIN, PH.D. EKASIT KIJSIPONGSE, PH.D. THE RESEARCH WAS FINANCIALLY SUPPORTED BY THE UNIVERSITY OF THE THAI CHAMBER OF COMMERCE 2013 i Title : Improving the Communication Performance of Distributed Animation Rendering Using BitTorrent File System Main researcher : Asst. Prof. Namfon Assawamekin, Ph.D. School of Science and Technology, University of the Thai Chamber of Commerce Co-researcher : Ekasit Kijsipongse, Ph.D. National Electronics and Computer Technology Center Year of accomplishment : 2013 No. of pages : 46 Key words : animation rendering, BitTorrent, distributed file system, peer-to-peer ABSTRACT* Rendering is a crucial process in the production of computer generated animation movies. It executes a computer program to transform 3D models into series of still images, which will eventually be sequenced into a movie. Due to the size and complexity of 3D models, rendering process becomes a tedious, time-consuming and unproductive task on a single machine. As a result, animation rendering is commonly carried out in a distributed computing environment where a number of computers execute in parallel to speedup the rendering process. In accordance with the distribution of computing, the data dissemination to all computers also needs certain mechanisms which allow large 3D models to be efficiently moved to those distributed computers to ensure the reduction of time and cost in animation production. In this report, we present and evaluate the BitTorrent File System (BTFS) for improving the communication performance of distributed animation rendering. The BTFS provides an efficient, secure and transparent distributed file system which decouples the applications from the complicated communication mechanism. By having the data disseminated in a peer-to-peer manner and using local cache, the rendering time can be reduced. Its performance comparison with a production-grade 3D animation favorably shows that BTFS outperforms the traditional distributed file systems up to the factor of 4 in our test configuration. *The research was financially supported by the University of the Thai Chamber of Commerce. ii CONTENTS Page ABSTRACT i LIST OF TABLES v LIST OF FIGURES vi CHAPTER 1 INTRODUCTION 1 1.1 Introduction …………………………………………………………… 1 1.2 Research Objectives …………………………………………………… 2 1.3 Research Scope ………………………………………………………… 2 1.4 Research Methodology ………………………………………………… 3 1.5 Research Contributions ………………………………………………… 3 CHAPTER 2 RESEARCH BACKGROUND AND RELATED WORK 4 2.1 Distributed File System (DFS) ………….……………………………… 4 2.1.1 Network File System (NFS) ………….………………………… 4 2.1.2 Server Message Block (SMB) / Common Internet File System (CIFS) …………………………………………………………… 5 2.1.3 Andrew File System (AFS) / Coda ……………………………… 6 2.1.4 MogileFS ………………………………………………………… 6 2.1.5 Hadoop Distributed File System (HDFS) ……………………… 7 2.2 Peer-to-Peer File Sharing ………….…………………………………… 7 2.2.1 Napster ………….……………………………………………… 7 2.2.2 Gnutella ………….……………………………………………… 8 2.2.3 Freenet ………….………………………………………………… 8 2.2.4 OceanStore ………….…………………………………………… 8 2.2.5 KaZaA / FastTrack ………….…………………………………… 9 2.2.6 BitTorrent ………….…………………………………………… 9 iii CONTENTS (cont.) Page 2.3 BitTorrent Protocol ………….………………………………………… 9 2.3.1 .torrent ………….………………………………………………… 10 2.3.2 Tracker ………….……………………………………………… 10 2.3.3 Peer ………….…………………………………………………… 11 2.4 Related Work ………….……………………………………………… 11 CHAPTER 3 DESIGN AND IMPLEMENTATION 15 3.1 Metadata Server ………...……………………………………………… 15 3.2 Seeder ………………………...………………………………………… 17 3.3 Tracker ……………………...…………………………………………… 18 3.4 BTFS Client …………………………………………………………… 18 3.5 Mapping to POSIX Semantics ………………………………………… 19 3.6 Consistency Model ……………………………………………………… 20 3.6.1 Attribute and File Cache Management ………….……………… 21 3.6.2 Write Back Policy ………….…………………………………… 21 3.7 Security ………………………………………………………………… 22 3.7.1 Authentication, Authorization and Access Control ……………… 22 3.7.2 Data Integrity ………….………………………………………… 23 3.7.3 Confidentiality ………….……………………………………… 23 3.8 Load Balancing and Fault Tolerance …………………………………… 24 3.9 Global Configuration Management …………………………………… 25 3.10 Garbage Collection …………………………………………………… 26 3.11 Operation ……………………………………………………………… 26 iv CONTENTS (cont.) Page CHAPTER 4 EVALUATION AND EXPERIMENTS 27 4.1 Testbed System Configuration ………………………………………… 27 4.2 Render Data and Software ……………………………………………… 28 4.3 Performance Comparison of BTFS and SMB File System …………… 31 4.3.1 Small Job ………………………………………………………… 31 4.3.2 Medium Job ……………………………………………………… 32 4.3.3 Large Job ………………………………………………………… 33 4.4 Peer Contribution under BTFS File System …………………………… 34 4.5 Load Balance of BTFS with Multiple Seeders ………………………… 36 4.6 BTFS Replication Performance ………………………………………… 37 4.7 Operation Breakdown ………………………………………………… 39 CHAPTER 5 CONCLUSIONS 42 REFERENCES 43 BIOGRAPHY 46 v LIST OF TABLES Table Page 2.1 Comparison of BitTorrent-based data dissemination in distributed computing environments ……………………………………………………………………… 13 3.1 Supported file system operations ………………………………………………… 19 3.2 Mapping from BTFS to POSIX semantics ……………………………………… 20 4.1 Characteristics of the testing data ………………………………………………… 29 4.2 Number of files and disk usages on each seeder ………………………………… 37 vi LIST OF FIGURES Figure Page 3.1 BitTorrent file system architecture ……………………………………………… 15 3.2 Mapping from ZooKeeper to BTFS file system ………………………………… 17 3.3 Global configuration file ………………………………………………………… 25 4.1 Testbed system …………………………………………………………………… 28 4.2 Job status ………………………………………………………………………… 30 4.3 Node status ……………………………………………………………………… 30 4.4 Outbound network traffic from file server (seeder) for small job ………………… 32 4.5 Outbound network traffic from file server (seeder) for medium job …………… 33 4.6 Outbound network traffic from file server (seeder) for large job ………………… 34 4.7 Breakdown of inbound traffic under BTFS for small job ………………………… 35 4.8 Breakdown of inbound traffic under BTFS for medium job …………………… 35 4.9 Breakdown of inbound traffic under BTFS for large job ………………………… 36 4.10 Load of each seeder ……………………………………………………………… 37 4.11 Data transfer time of multiple replicas …………………………………………… 38 4.12 Speedup of multiple replicas ……………………………………………………… 38 4.13 Read operation breakdown ……………………………………………………… 40 4.14 Write operation breakdown ……………………………………………………… 41 CHAPTER 1 INTRODUCTION This chapter aims to provide the introduction of distributed animation rendering and the problems of the research (Section 1.1). The research objectives are defined in Section 1.2, followed by the scope of research in Section 1.3. Section 1.4 describes the research methodology. Section 1.5 summarizes the key contributions of this research. 1.1 Introduction Animation rendering is a process that transforms 3D models into hundred thousands of image frames to be composed into a movie. Rendering process is very computing intensive and time consuming. A single frame of an industrial-level animation can even take several hours in rendering on a single machine. Animation rendering is then typically carried out on a set of high performance computers where distributed rendering is taken place in which frames are distributed and rendered across many machines in a network to reduce the overall rendering time. Distributed rendering comes in many flavors such as render farm and volunteer-based rendering. For render farm, machines are dedicated for rendering task and all machines are tightly-coupled in local area network having high bandwidth. In volunteer-based rendering [1], machines are loosely connected by the public Internet and the machine owners provides the idle time of their computing resources for rendering. For example, Renderfarm.fi [2] is a large-scale, volunteer, loosely- coupled rendering service that distributes rendering process over the Internet. Originally, volunteer computing uses central servers for distributing data and computing to clients [3]. There is no notion of data exchange between clients. However, in animation rendering, the 3D models and related library files are large. These input files have to be transferred to the clients before the rendering can begin on the clients. The transfer time is significant due to the latency of the public Internet. Additionally, centralized servers can become overloaded when there are too many clients requesting for 2 the data which will slow down the rendering process. Since the same files may be used by several clients almost the same time, it is a great opportunity to coordinate the file transfer among clients in the peer-to-peer (P2P) manner to reduce the data transfer time. With the P2P file sharing model, a client who has already downloaded a whole or some parts of the file from the central servers can share the file to other clients so that others can directly download the file (or parts of it) from the former client instead of from the central servers. As there are more copies of the files on different clients, there are fewer requirements to download data from the central servers. To allow rendering applications to transparently access the shared files over the P2P model without having the applications modified, it is necessary to implement the P2P file sharing service as the file system layer in the operating system. Thus, transferring the files from several peers across the network is invisible to the applications such that the shared files can be treated the same as files on local disks.