Numerical Solution of 3-D Electromagnetic Problems in Exploration Geophysics and its Implementation on Massively Parallel Computers Jelena Koldan Department of Computer Architecture Universitat Polit`ecnicade Catalunya Thesis submitted for the degree of Doctor of Philosophy in Computer Architecture September, 2013 Supervisor: Jos´eMar´ıaCela Abstract The growing significance, technical development and employment of electromagnetic methods in exploration geophysics have led to the increasing need for reliable and fast techniques of interpretation of 3-D electromagnetic data sets acquired in complex geological environments. The first and most important step to creating an inversion method is the development of a solver for the forward problem. In order to create an efficient, reliable and practical 3-D electromagnetic inversion, it is necessary to have a 3-D electromagnetic modelling code that is highly accurate, robust and very fast. This thesis focuses precisely on this crucial and very demanding step to building a 3-D electromagnetic interpretation method. The thesis presents as its main contribution a highly accurate, robust, very fast and extremely scalable numerical method for 3-D electromagnetic modelling in geo- physics that is based on finite elements and designed to run on massively parallel computing platforms. Thanks to the fact that the finite-element approach supports completely unstructured tetrahedral meshes as well as local mesh refinements, the presented solver is able to represent complex geometries of subsurface structures very precisely and thus improve the solution accuracy and avoid misleading arte- facts in images. Consequently, it can be successfully used in geological environments of arbitrary geometrical complexities. The parallel implementation of the method, which is based on the domain decomposition and a hybrid MPI{OpenMP scheme, has proved to be highly scalable { the achieved speed-up is close to the linear for more than a thousand processors. As a result, the code is able to deal with ex- tremely large problems, which may have hundreds of millions of degrees of freedom, in a very efficient way. The importance of having this forward-problem solver lies in the fact that it is now possible to create a 3-D electromagnetic inversion that can deal with data obtained in extremely complex geological environments in a way that is realistic for practical use in industry. So far, such imaging tools have not been proposed due to a lack of efficient, parallel finite-element solutions as well as the limitations of efficient solvers based on finite differences. In addition, the thesis discusses physical, mathematical and numerical aspects and challenges of 3-D electromagnetic modelling, which have been studied during my research in order to properly design the presented software for electromagnetic field simulations on 3-D areas of the Earth. Through this work, a physical problem for- mulation based on the secondary Coulomb-gauged electromagnetic potentials has been validated, proving that it can be successfully used with the standard nodal finite-element method to give highly accurate numerical solutions. Also, this work has shown that Krylov subspace iterative methods are the best solution for solv- ing linear systems that arise after finite-element discretisation of the problem under consideration. More precisely, it has been discovered empirically that the best it- erative method for this kind of problems is biconjugate gradient stabilised with an elaborate preconditioner. Since most commonly used preconditioners proved to be either unable to improve the convergence of the implemented solvers to the desired extent, or impractical in the parallel context, I have proposed a preconditioning technique for Krylov methods that is based on algebraic multigrid. Tests for vari- ous problems with different conductivity structures and characteristics have shown that the new preconditioner greatly improves the convergence of different Krylov subspace methods, even in the most difficult situations, which significantly reduces the total execution time of the program and improves the solution quality. Fur- thermore, the preconditioner is very practical for parallel implementation. Finally, through this work, it has been concluded that there are not any restrictions in em- ploying classical parallel programming models, MPI and OpenMP, for parallelisation of the presented finite-element solver. Moreover, these programming models have proved to be enough to provide an excellent scalability for it, as shown by different large-scale tests. Keywords: high-performance computing, parallel programming, 3-D electromag- netic modelling, finite element, preconditioning, algebraic multigrid To my parents. Acknowledgements First of all, I would like to thank my thesis advisor, Jos´eMar´ıaCela, for accepting me in his team in Barcelona Supercomputing Center and for proposing an inspir- ing and exciting research path, along which he has supported me with patience and understanding, as well as with complete readiness to share his broad knowledge. I would also like to thank all my colleagues and office mates from BSC{Repsol Research Center, who are not only great co-workers, but also wonderful people, for all the help with different aspects of my work, as well as for creating a relaxed, pleas- ant and cheerful working environment in the office 302 in Nexus I. I am especially grateful to Josep de la Puente, who appeared just in time to significantly facilitate my research, for sharing his knowledge as well as his clear and bright insights and, maybe even more, for simply being always there for an inspiring discussion. Also, he is the one of few people who patiently and thoroughly red my articles as well as this thesis and I greatly appreciate his useful comments and ideas that helped me to improve all my writings. I am also thankful to Vladimir Puzyrev for helping me a lot in the past few years by working on the same project and for sharing with me all ups and downs of the working process (and a few musical concerts, as well), F´elixRubio Dalmau and Albert Farres for always being ready to save me from my occasional struggles with computers and for being very good friends during all these years in Barcelona, as well as Mauricio Hanzich, Jean Kormann, Oscar Peredo, Miguel Fer- rer, Natalia Guti´errez,Gen´ısAguilar and Juan Esteban Rodr´ıguezfor being such a nice team. Here, I would also like to thank Repsol for financially supporting the research whose result is the presented thesis. Also, I am very thankful to all my colleagues from Computer Applications in Science and Engineering department of Barcelona Supercomputing Center that supported me during my research and shared with me their insight and expertise on different aspects of the topic. I am especially grateful to Guillaume Houzeaux for finding the time in his incredibly busy daily schedule to help me to cope with the Alya system and Xavie S´aezfor his valuable insights into parallelisation meth- ods and issues as well as for being a great friend that is always ready to explore different artistic and cultural events. I would also like to acknowledge all other members of Barcelona Supercomputing Center that contributed in different ways to my academic life in Barcelona. Here, above all, I have to thank Mateo Valero for accepting me six years ago into this wonderful, stimulating and vibrant environment and for giving me a chance to take up this exciting and life-changing academic jour- ney. However, none of this would have happened if there had not been for Professor Veljko Milutinovi´cfrom University of Belgrade, who believed in me enough to give me his recommendation which opened me the door to this new world of incredible opportunities. Also, I would like to acknowledge former and current members of the Serbian community in BSC (which is not small) that helped me to overcome numerous practical difficulties that foreign students normally face as well as to feel less homesick, thanks to which I was able to primarily focus on my research. Espe- cially, I would like to thank Nikola Markovi´c,Vesna Nowack, Ana Jokanovi´c,Maja Etinski, Srdjan Stipi´cand all Vladimirs for helping me at different points of my academic and non-academic life in Barcelona to overcome some difficult times and to find a way to carry on. Outside BSC, there are people from other academic communities that I would like to mention. First, I would like to thank Xavier Garc´ıafrom Barcelona Center for Subsurface Imaging (Barcelona-CSI) for sharing his knowledge of geophysical ex- plorations with electromagnetic methods and for introducing me to the community of electromagnetic geophysicists at the unforgettable workshop on magnetotelluric 3-D inversion, held in Dublin in March 2011, at which I have made a lot of valu- able contacts, learned a lot and found inspiration to continue with my research, In May 2012, I visited the leading scientist in the community, Gregory Newman, and his famous group at the Geophysics Department in the Earth Sciences Division of Lawrence Berkeley National Laboratory, Berkeley, CA, USA. I am extremely grate- ful to Dr. Newman for his willingness to be my host and to share his incredible knowledge of the topic. Also, I would like to express deep gratitude to Evan Um, a great scientist and a wonderful person, for agreeing to share his office with me during my visit as well as for helping me to improve my work by sharing his broad knowledge and experience with unique patience and kindness. Barcelona is a vibrant and fascinating city with many opportunities for young people. Therefore, being a PhD student in it is both exciting and challenging. Luckily, I have managed to dedicate myself to my research without losing too much of experiencing life outside the office. I would like to thank Irina Blagojevi´c,Jelena Lazarevi´c, Neda Kostandinovi´c,Ognjen Obu´cinaand Rade Stanojevi´cfor exploring this city with me and for sharing some wonderful and unforgettable moments, as well as for broadening my views by talking about other interesting topics from outside of the worlds of computer architecture and geophysics.