UNIVERSITAT POLITÈCNICA DE CATALUNYA DEPARTMENT OF COMPUTER ARCHITECTURE File System Metadata Virtualization by Ernest Artiaga i Amouroux Advisor: Toni Cortés Rosselló A dissertation submitted in partial fulfilment of the requirements for the degree of Doctor per la Universitat Politècnica de Catalunya File System Metadata Virtualization Abstract: The advance of computing systems has brought new ways to use and access the stored data that push the architecture of traditional file systems to its limits, making them inadequate to handle the new needs. Current challenges affect both the performance of high-end computing systems and the usability of general-purpose personal computing systems. One of the elements that limit the usability of file systems is the exposure of their internal organization through rigid name spaces, that usually reflect how data is actually stored in the underlying media. Users must know and understand the nuts and bolts of specific storage systems in order to achieve good results, receiving the burden of taking into account the complexities and restrictions of the new environments. This burden can be alleviated by providing dynamic, flexible name spaces adapted to specific application needs, instead of representing how data is stored in the underlying storage systems. This thesis contributes to the goal above by proposing a mechanism to decouple the user view of the storage from its underlying structure. This mechanism consists in the virtualization of file system metadata and the intro- duction of a sensible layer able to take automatic decisions on where and how the files should be stored in order to take advantage of the underlying storage system features. More specifically, the first contribution of this thesis is the design of a metadata virtualization framework which is able to produce consistent and flexible views of a file system, independently from the actual underlying data organization. The second contribution consists in using this framework to alleviate the complex file system tuning requirements in high-performance environments. Finally, The third contribution of this thesis consists in applying the metadata virtualization principles to provide an ubiquity layer for cloud storage systems that extends the cloud-awareness to the whole file system, making it independent from the directory where the cloud-related files reside. Keywords: File system virtualization, file system metadata, parallel file systems, cloud storage, name spaces. Place and date: Barcelona. October, 2013 Funding: This work was partially supported by Spanish MECD under grants TIN2007-60625 and TIN2012-34557, and the Catalan Government under the 2009-SGR-980 grant. Part of the research leading to the presented results has also received funding from the EU IST program as part of the XtreemOS project (contract FP6-033576) and the European Community’s Seventh Frame- work Programme (FP7/2007-2013) under grant agreement RI-283493, as part of the PRACE-2IP project. Infrastructure: Part of the research in this thesis has been possible thanks to the high- performance computing infrastructures kindly provided by the Barcelona Supercomputing Center (BSC), the Partnership for Advanced Computing in Europe (PRACE) and the Lawrence-Livermore National Laboratory (LLNL) via the Hyperion Project; additionally, part of the works in this thesis have made use of the computing infrastructure of the Department of Computer Architecture at the Technical University of Catalonia (DAC-UPC). I sincerely acknowledge their support. Acknowledgements This thesis is part of a long path that started many years ago, and I have met many people that have supported me along the way and helped me in critical moments. All of them have contributed in some way to this work, I sincerely acknowledge their part in the effort. Specially, I would like to thank Mateo Valero for giving me the opportunity of working at BSC since its beginning and dedicating these last years to research. Thanks to that, I have had access to both the people and the technological resources that have enabled me to pursue my PhD. I would not be finishing my thesis now without his support. All journeys have a start, and I would not have reached this point without the continuous support of Pep Fuertes. He was the person that many years ago started to push me into the world of research, and trusted me even before I finished my Computer Science courses at University. I have learned many valuable things from him, and I owe many of my achievements to his advice, encouragement and friendship during all these years. The day to day efforts leading to my PhD have been assisted by Toni Cortés who, despite knowing me, accepted to be my advisor. His vision and wisdom have indeed contributed to the happy ending of my PhD, and his friendliness and optimism have created a wonderful and interesting place to work. I want to thank the current and former members of the Storage Systems Research Group at BSC for being a wonderful team to work in. I have received many useful inputs from them. In particular, Jonathan Martí, Jan Wiberg, Juan González de Benito and Thanos Makatos have had an important participation in the technical aspects related to this thesis. I must specially mention Alberto Miranda, whose abilities and curiosity make of him an amazing source of information and technical knowledge, apart from a nice colleague to work with. The use of the supercomputing infrastructures required for the present work has been greatly facilitated by the Operations team at BSC. I really appreciate the help from the system administrators and the support team to prepare the rather uncon- ventional setups required for my experiments. I specially want to thank to Javier Bartolomé, David Vicente, Sergi Moré, Jordi Valls and Albert Benet their readiness to collaborate, and the help of Ferran Sellés, Toni Espinar, Pedro Gómez and Albert Riera to solve any issues in the working environment. I want to mention the people from the Operating Systems Group at the Depart- ment of Computer Architecture who shared my first steps (and specially Xavier Martorell, whom I honestly admire for his ability to comprehend all aspects of a system at any arbitrary level of detail). Since then, a lot of people from the Depart- ment of Computer Architecture at UPC and from the BSC have made my life easier by sharing their knowledge, but also breakfasts, coffees, meals and discussions about spherical cows. I am afraid that the list is too long to mention you all, but I really appreciate your company. I specially want to thank David Carrera, Marc González vi and Anna Queralt for consistently being able to cast away any worries during lunch times. I have a very long list of reasons to say thanks to Yolanda Becerra, and excellent colleague and good friend since school time. She has that rare quality of being reliable on good and bad times, and so many times she has helped me to stay focused during trouble. Thanks for always being there! I also want to thank my friends for helping me to have a life outside the work. Specially Pedro, Oriol, Maria José, Rosa, Sandra and Nacho who, even when life makes it difficult to meet often, continue being great friends. And I cannot forget my family for being by my side at all times. Specially my uncles, my cousins (Edu, Anna and Núria) and my new family in Granada (Manolo and Ana). Above all, I want to thank my mother, Pilar, for always being caring and teaching me everything I needed to prepare for life, and to my grandmother, Carmen, who would have enjoyed seeing this work finished. Finally, I want to thank Pili, my wife, for her sustained support during all these years, for her patience and for making my life a happier place. Her energy has always inspired me to go on. Thanks for sharing your life with me! Agraïments Aquesta tesi és part d’un llarg camí que va començar fa molts anys, durant el qual he trobat a molta gent que m’ha anat ajudant, especialment en els moments crítics. Tots ells han aportat d’alguna forma el seu gra de sorra a aquest treball i desitjo agraïr-los sincerament la seva contribució. En particular, m’agradaria agrair a Mateo Valero que em donés l’oportunitat de treballar al BSC des de la seva creació i poder dedicar aquests últims anys a la recerca. Gràcies a això he pogut tenir accés tant a la gent com als recursos tecnològics que m’han permès fer el doctorat. Sense el seu suport, no estaria ara escrivint aquestes línies. Tots els camins tenen un començament, i no hauria arribat a aquest moment sense el suport continu d’en Pep Fuertes. Ell va ser la persona que, fa molts anys, em va introduir en el món de la recerca, confiant en mi fins i tot abans que acabés la carrera d’Informàtica. He tingut la sort de poder aprendre moltes coses d’ell, i moltes de les coses que he aconseguit fer les dec al seus consells, el seu suport i la seva amistat durant tots aquests anys. Els esforços del dia a dia que han portat a la conclusió del meu doctorat han estat guiats per en Toni Cortés qui, tot i que em coneixia, va acceptar dirigir-me la tesi. La seva visió de la tecnologia i la seva experiència han contribuït de forma decisiva a que el meu doctorat tingués un bon final; la seva amistat i el seu optimisme han creat un espai interessant per treballar-hi. Vull agrair als membres i ex-membres del Storage Systems Research Group del BSC per haver format un equip fantàstic. D’ells he rebut moltes aportacions. En particular, en Jonathan Martí, en Jan Wiberg, en Juan González de Benito i en Thanos Makatos han tingut una participació destacada en els aspectes tècnics relacionats amb aquesta tesi.