University of Calgary PRISM: University of Calgary's Digital Repository Graduate Studies The Vault: Electronic Theses and Dissertations 2020-01 Preconditioned Iterative Solvers on GPU and an In-Situ Combustion Simulator Yang, Bo Yang, B. (2020). Preconditioned Iterative Solvers on GPU and an In-Situ Combustion Simulator (Unpublished doctoral thesis). University of Calgary, Calgary, AB. http://hdl.handle.net/1880/111505 doctoral thesis University of Calgary graduate students retain copyright ownership and moral rights for their thesis. You may use this material in any way that is permitted by the Copyright Act or through licensing that has been assigned to the document. For uses that are not allowable under copyright legislation or licensing, you are required to seek permission. Downloaded from PRISM: https://prism.ucalgary.ca UNIVERSITY OF CALGARY Preconditioned Iterative Solvers on GPU and an In-Situ Combustion Simulator by Bo Yang A THESIS SUBMITTED TO THE FACULTY OF GRADUATE STUDIES IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY GRADUATE PROGRAM IN CHEMICAL AND PETROLEUM ENGINEERING CALGARY, ALBERTA JANUARY, 2020 c Bo Yang 2020 Abstract This thesis consists of two parts: preconditioned iterative solvers on GPU and an in-situ combus- tion simulator. The purpose of the first research is to develop a new parallel solution platform based on GPU features. An application of HPC (high-performance computing) technology to reservoir simulation has become an inevitable trend. As a platform for HPC, GPU can provide an effective solution for per- sonal computers and workstations. In this research, not only a series of special CPR (constrained pressure residual) preconditioned solvers are developed for black oil models, but also a variety of other preconditioned solvers are completed as contrast solvers. The numerical experiments verify a significant improvement in the parallel performance of the solvers on GPU. They also provide an overall comparison among the combinations of different GPUs, solvers, and preconditioners. The results demonstrate that the CPR developed has excellent advantages in both parallelism and convergence for the solution of a benchmark reservoir model. The purpose of the second research is to develop a new comprehensive ISC (in-situ combustion) simulator with the PER (pseudo equilibrium ratio) method and to compare functions with those of a benchmark simulator with the VS (variable substitution) method. ISC is considered a promising recovery method because of its low cost and less environmental impact. However, an ISC simulator is regarded as one of the most complex simulators to de- velop. The PER method can reduce the complexity of simulator development because it lowers the influence of the phase disappearance and appearance on the mathematical system of reservoir simulation. The ISC simulator in this study is developed with comprehensive typical functions of the ISC process. For the verification of the equivalence in numerical results between the PER method and the VS method, the numerical experiments are carried out in an omnidirectional range. Be- ii cause the results show a very close match, the research provides reliable experimental support for popularizing the use of the PER method to develop an in-situ combustion simulator. Acknowledgements I would like to express my most genuine gratitude to my supervisor, Dr. Zhangxing Chen. It is fortunate for me to be his student. His extensive knowledge and good personality deeply infect me. He will always be a model for me. I would like to show my sincerest appreciation to my co-supervisor, Dr. Wenyuan Liao. It is my great pleasure to receive the guidance of mathematics and career concerns from him. His advice and care will be unforgettable to me. I would like to present special recognition to Dr. Antony Ware for his valuable suggestion, tolerant attitude, and patient explanation as an indispensable part of this research. I would like to give my great gratefulness to committee members and examiners, Drs. Qingye Lu, Jalel Azaiez, Antony Ware, and Pengtao Sun for their experience, comments, and time. I am very grateful to Dr. Hui Liu for his help in the past years. The technical discussions with him enabled many difficulties to be overcome. For the ISC simulator research, I wish to thank Min Yang who cooperated in the collation of test cases and Ruijian He who participated in the coding of the dry combustion part. Thanks go to the Department of Chemical and Petroleum Engineering and the Reservoir Sim- ulation Group, Schulich School of Engineering. iv To my wife Xiaoling Zhong Table of Contents Abstract ........................................... ii Acknowledgements .................................... iv TableofContents ..................................... vi ListofTables ....................................... ix ListofFigures...................................... xi ListofSymbols ...................................... xvi 1 RESERVOIRSIMULATOROVERVIEW . 1 1.1 Petroleum....................................... 1 1.2 Reservoir Numerical Simulation . ...... 2 1.3 ReservoirSimulators .............................. ... 6 1.4 Layout......................................... 8 2 INTRODUCTION(1)................................. 12 2.1 High-Performance Computing . 12 2.2 Preconditioned Iterative Solvers on GPU . .......... 16 2.3 LiteratureReview................................. 21 2.4 ResearchObjectives............................... 24 3 GPUANDBLACKOILMODEL .......................... 26 3.1 GPU.......................................... 26 3.1.1 GPUArchitecture............................... 26 3.1.2 CUDA .................................... 29 3.2 BlackOilModel ................................... 32 3.2.1 MathematicalModel . .... .... .... .... .... ... .... 32 3.2.2 SPE10Model2................................ 36 4 PRECONDITIONEDITERATIVESOLVERS. 39 4.1 SparseMatrices.................................. 39 4.1.1 Discretization................................ 39 4.1.2 Permutation.................................. 41 4.1.3 Storage .................................... 43 4.2 IterativeSolvers................................ 49 4.2.1 LinearSolutionMethods . 49 4.2.2 KrylovSubspaceMethods . 51 4.2.3 AMGMethod................................. 53 4.3 Preconditioners ................................. 62 4.3.1 Principle ................................... 62 4.3.2 ILU...................................... 63 4.3.3 BILU ..................................... 70 4.3.4 CPR...................................... 74 4.4 ParallelImplementation. ...... 78 4.4.1 ParallelComponents . .... .... .... .... .... ... .... 78 4.4.2 Parallel Triangular Solver . 80 4.4.3 RAS...................................... 82 4.4.4 Parallel Algorithms . 84 vi 5 NUMERICALEXPERIMENTS(1) ......................... 87 5.1 Introduction.................................... 87 5.2 Environments .................................... 89 5.3 Solvers ........................................ 91 5.4 ILU .......................................... 92 5.5 BILU ......................................... 95 5.6 RAS.......................................... 97 5.7 AMG ......................................... 99 5.8 CPR..........................................102 6 CONCLUSIONS(1)..................................108 6.1 Conclusions...................................... 108 6.2 FutureWork...................................... 110 7 INTRODUCTION(2).................................112 7.1 ThermalRecovery.................................. 112 7.2 In-SituCombustionProcess . 114 7.3 LiteratureReview................................. 115 7.4 ResearchObjectives............................... 119 8 IN-SITUCOMBUSTIONSIMULATOR. .121 8.1 ModelDescription ................................ 121 8.1.1 PhasesandComponents . .121 8.1.2 ChemicalReactions .............................122 8.1.3 PhysicalProperties . 123 8.1.4 Assumptions .................................132 8.2 MathematicalSystem .............................. 132 8.2.1 MassConservation ..............................132 8.2.2 EnergyConservation . 135 8.2.3 Constraints ..................................136 8.2.4 WellEquation.................................137 8.2.5 PhaseChange.................................137 8.2.6 HeatLoss...................................139 8.2.7 PDESystem .................................141 8.3 NumericalMethods................................ 142 8.3.1 DiscretizationScheme . 142 8.3.2 SpecificTreatments. 144 8.3.3 NonlinearSystem............................... 146 8.3.4 Newton-Raphson Method . 147 8.3.5 SimulatorFlow................................150 9 NUMERICALEXPERIMENTS(2) . .153 9.1 Introduction.................................... 153 9.2 DryCombustionTube................................ 154 9.3 WetCombustionTube ................................ 175 9.4 Full-ScaleHeatLoss .............................. 184 9.5 2-DMulti-PerfWells .............................. 198 9.6 3-D Inverted Five-Spot Pattern . .......208 10 CONCLUSIONS(2)..................................217 vii 10.1Conclusions..................................... 217 10.2FutureWork..................................... 219 REFERENCES .......................................220 viii List of Tables 1.1 Total petroleum and other liquids production of major oil-producing countries. 3 1.2 Open source reservoir simulators. ........ 8 1.3 Commercial reservoir simulators. ........ 9 1.4 Reservoir simulator general classification. ............ 10 2.1 Evolution of computing architecture.