Parallelization of Ensemble Kalman Filter (EnKF) for Oil Reservoirs Md. Khairullah Parallelization of Ensemble Kalman Filter (EnKF) for Oil Reservoirs Master’s Thesis in Computer Simulations for Science and Engineering (COSSE) Delft Institute of Applied Mathematics Faculty of Electrical Engineering, Mathematics, and Computer Science Delft University of Technology The Netherlands Md. Khairullah Student id: 4187326 20th June 2012 Author Md. Khairullah Title Parrallelization of Ensemble Kalman Filter (EnKF) for Oil Reservoirs MSc presentation July 05, 2012 Graduation Committee Prof. dr. ir. A.W. Heemink (chair) Delft University of Technology Prof. dr. ir. C. Vuik Delft University of Technology Prof. dr. ir. H.X. Lin Delft University of Technology dr. R.G. Hanea Delft University of Technology dr. ir. Harald Kostler¨ Friedrich-Alexander University Abstract This thesis describes the design and implementation of a parallel algorithm for data assimilation with ensemble Kalman filter (EnKF) for oil reservoir manage- ment. The implemented application works on large number of observations from time-lapse seismic, which lead to a large turnaround time for the analysis step, in addition to the time consuming simulations of the realizations. Provided that par- allel resources are used for the parallel simulations of the realizations, the analysis step also deserves parallelization. Our experiments show that parallelization of the analysis step in addition to the forecast step also scales well, exploiting the same set of resources with some additional efforts. iv Preface The work lying before you is my Master of Science thesis, in which a description is given of the work done and the results achieved since September 2011 at the Delft Institute of Applied Mathematics (DIAM), Faculty of EEMCS, Delft University of Technology. Having my bachelor degree in Computer Science and Engineering, I feel lucky enough to apply my achieved programming and analytical ability to some applied and practical topic of the real world in my current studies. I was very fond of parallel computing after I learnt Socket Programming in Java during my undergrad studies. It was a nice opportunity for me to work on parallelization of data assimilation for reservoir management. Before continuing with my thesis, I would like to thank some people who have been of great help during my Masters project. First of all, I would like to thank my supervisors Dr. Lin and Dr. Hanea for giving me their valuable time for answering all my stupid and wise questions related to the project and also for their comments, suggestions and supports at different stages of the project. I want to thank the graduation committee members for their collective and individual contributions. I would like to thank Dr. J. D. Jansen for his support at the very beginning of the project to teach me some basic things of the simsim simulator. I would also thank ir. C.W.J. Lemmens for his technical supports. I would like to thank my family members for their patience and support for my life in abroad. Special thanks go to Dr. Harald Kostler,¨ FAU, Erlangen for his supports during my Erlangen life. Last but not the least, I express my deep gratitude to European Union for funding my studies through Erasmus Mundus Scholarship. Md. Khairullah Delft, The Netherlands 20th June 2012 v vi Contents Preface v 1 Introduction 1 2 Parallel Data Assimilation with EnKF 5 2.1 Problem settings . .5 2.2 Previous works . .6 2.3 Proposed parallel algorithm . .7 2.4 Parallel performance . 11 3 Basics of Reservoir Engineering 15 3.1 Definitions . 15 3.2 Recovery processes . 16 3.3 Different data sets . 19 3.3.1 Production history data . 19 3.3.2 Time-lapse seismic data . 20 3.3.3 Prior knowledge . 20 4 Data Assimilation with EnKF 21 4.1 Review of the Kalman Filter . 22 4.1.1 Variance minimizing analysis scheme . 22 4.1.2 Kalman filter . 23 4.2 Ensemble Kalman filter . 23 4.2.1 Representation of error statistics . 24 4.2.2 Analysis scheme . 24 4.3 Practical implementation . 26 4.3.1 Ensemble representation of the covariance . 26 4.3.2 Measurement perturbations . 27 4.3.3 Analysis equation . 27 4.3.4 EnKF for combined parameter and state estimation . 29 5 Parallel Computing with MPI and ScaLAPACK 33 5.1 Parallel computing . 33 5.2 MPI . 35 vii 5.3 Parallel numerical libraries for SVD . 36 5.3.1 ScaLAPACK . 37 6 Implementation 43 6.1 Reservoir model . 43 6.2 Software structure . 45 6.3 Optimization . 47 6.3.1 Load balancing . 47 6.3.2 Minimizing data communication . 48 7 Results and Discussions 53 7.1 Verification . 53 7.2 Performance of the parallel algorithm . 54 7.2.1 Difference in speedup: forecast vs analysis step . 59 7.2.2 Super-linear speedup . 64 7.2.3 Parallel scalability . 64 7.2.4 Difference in performance: 48 realizations vs 96 realizations 65 7.2.5 Difference in speedup and performance: LAN cluster vs SARA Lisa cluster . 65 7.2.6 Effect of ScaLAPACK process grid . 65 7.2.7 Effect of ScaLAPACK block size . 69 7.2.8 Effect of non blocking communication . 71 8 Conclusion and Future Work 75 8.1 Conclusion . 75 8.2 Future work . 77 viii List of Figures 1.1 The forecast of global demand of energies [2] . .2 1.2 The forecast of global production of energies [2] . .2 2.1 Typical logical domain decomposition for the forecast step (top) and for the analysis step (bottom) . .8 2.2 An example of physical domain decomposition used for the ocean circulation model in [21] . .8 2.3 Serial implementation of the EnKF for data assimilation . .9 2.4 1st level parallelization of the simulator . .9 2.5 2nd level parallelization of the simulator . 10 2.6 A multi level parallelization . 10 2.7 Proposed parallel EnKF for data assimilation . 11 3.1 Reservoir management depicted as a closed loop model-based con- trolled process [19] . 19 4.1 Schematic of how data assimilation (DA) works and adds value to observational and model information. The data shown are various representations of ozone data of a particular day [22] . 22 4.2 The ongoing discrete Kalman filter cycle. The time update projects the current state estimate ahead in time. The measurement update adjusts the projected estimate by an actual measurement at that time. 24 4.3 The procedure of data assimilation with EnKF for the ensemble member j. 26 5.1 Shared memory architecture . 34 5.2 Distributed memory architecture: MPI’s work place . 35 5.3 ScaLAPACK Software Structure . 38 5.4 Data distribution examples: column blocked (left) and column cyc- lic (right) . 40 5.5 Data distribution examples: column blocked cyclic (left) and 2D blocked cyclic . 41 5.6 An example of view of data distribution in ScaLAPACK [3] . 42 6.1 Location of the wells in the model reservoir field . 44 ix 6.2 Assumed constant porosity field for all ensemble members . 45 6.3 Software structure of the developed parallel application . 46 7.1 rms differences of log permeability over time . 55 7.2 log permeability fields with 16 realizations: initial (top), after 6 assimilation steps in 3 years (middle), the true log permeability (bottom) . 56 7.3 log permeability fields with 48 realizations: initial (top), after 6 assimilation steps in 3 years (middle), the true log permeability (bottom) . 57 7.4 log permeability fields with 96 realizations: initial (top), after 6 assimilation steps in 3 years (middle), the true log permeability (bottom) . 58 7.5 Speedup of the parallel implementation for 48 realizations in the LAN cluster for different parts . 60 7.6 Speedup of the parallel implementation for 48 realization on SARA Lisa cluster for different parts . 61 7.7 Speedup of the parallel implementation for 96 realizations in the LAN cluster for different parts . 62 7.8 Speedup of the parallel implementation for 96 realization on SARA Lisa cluster for different parts . 63 7.9 Execution time of the forecast and analysis step for 48 and 96 real- izations on the LAN cluster . 66 7.10 Execution time of the forecast and analysis step for 48 and 96 real- izations on the SARA Lisa cluster . 67 7.11 Comparison of execution time of the forecast and analysis step for 48 realizations on different clusters . 68 7.12 Execution time of the analysis step for different process grid ori- entation for 48 realizations . 70 7.13 Execution time of the analysis step for different block sizes for 48 realizations . 72 7.14 Execution time of the analysis step for blocking and non-blocking data communication on the LAN cluster (top) and SARA Lisa cluster (bottom) . 74 x List of Tables 7.1 rms differences between the true and assimilated log permeabilities at different time steps . 54 7.2 Hardware specification of the LAN cluster for initial tests and meas- urements . 55 7.3 Performance of the parallel implementation for 48 realizations in the LAN cluster . 59 7.4 Performance of the parallel implementation for 48 realizations in the SARA Lisa cluster . 59 7.5 Performance of the parallel implementation for 96 realizations in the LAN cluster . 60 7.6 Performance of the parallel implementation for 96 realizations in the SARA Lisa cluster . 61 7.7 Execution time (in minute) of the analysis step for different process grid orientation for 48 realizations . 69 7.8 Execution time (in minute) of the analysis step for different block sizes for 48 realizations . 71 7.9 Execution time (in minutes) of the analysis step for blocking and non-blocking data communication . 73 xi xii List of Algorithms 1 A single step of basic EnKF .