GPUs TO MARS Full Scale Simulation of SpaceX’s Mars Rocket Engine Adam Lichtl, Stephen Jones, GTC 2015 Background
. Independent space launch company
. Build and operate our own rocket & capsule
. Operations in CA, TX & FL
. Founded in 2002, first launch in 2008
. 18 successful missions to date Design Build Launch Mars
Length of day 24 hours, 40 minutes Length of year 687 days
Gravity 0.375gs Distance from Sun 1.53 AU
Mars Air CO2 (96%), N2 (2%) Earth Air O2 (21%), N2 (78%) Atmosphere 1% density / pressure of Earth Temperature -284 / +86 / -81 F (low / high / avg) Human Mars Mission Classes
Credit: CollectSpace Mass to Mars
Apollo Command & International Space Station NASA Mars Architecture Service Module . Mass: 450 tons . Mass: 300 tons . Mass: 46 tons . 36x Space Shuttle, . 3x heavy-lift . Terrestrial assembly 5x Proton launches launches . In-orbit assembly
Image Credit: NASA Image Credit: NASA Image Credit: Mark Benson & Kerbal Space Program Methane Fuel
Carbon dioxide from atmosphere
Synthesize return-journey fuel on Mars O O C O C 2H2O + CO2 -> CH4 + 2O2 O
O H H C O O H H O
H Water from below ground O O H H H Rocket Engine Design Why Simulation?
1. Investigate what cannot be measured
2. Reduce need for testing
3. Design optimisation: narrow design space
4. Proactive instead of reactionary design Time scales vary by 8 orders of magnitude
Diffusion Advection Acoustics
10-11 10-10 10-9 10-8 10-7 10-6 10-5 10-4 10-3 seconds
Reaction Chamber Residence Injection Length scales vary by 6 orders of magnitude log E(k)
Cascade
Combustion
Chamber: ~1m
Kolmogorov Scale: ~1µm Eddy Energy Eddy Viscous
dissipation
kmin kmax Eddy Wavenumber Total points: (103)6 = 1018 = 1,000,000,000,000,000,000 points @1kB per point -> Yottabytes of data
Structure in Turbulence
Credit: SpaceWeather Simulation Meshing
Structured Grid Adaptive Grid Unstructured Grid Fourier Transform: Spectral Compression Wavelets: Local Fractal Basis Wavelet Compression Wavelet Compression Wavelet Compression Wavelet Compression Wavelet Compression
δ δ
δ < ε δ > ε (ε is some tolerance) Wavelet Compression
Transform Output Wavelet Compression
Transform Output Wavelet Compression
Transform Output Wavelet Compression
Transform Output Wavelet Compression
Transform Output Wavelet Compression
Transform Output Wavelet Compression
Transform Output Wavelet Compression
Transform Output Wavelet Compression
Transform Output Wavelet Compression
Transform Output Wavelet Compression
Transform Output Wavelet Compression
Transform Output Wavelet Compression
Transform Output Wavelet Compression
Transform Output Wavelet Compression
Transform Output Wavelet Compression
Transform Output Decompression
Transform Output Decompression
Transform Output Decompression
Transform Output Decompression
+δ
Transform Output Decompression
Transform Output Decompression
Transform Output Decompression
Transform Output Decompression
Transform Output Decompression
Transform Output
Regular Grid: Direct Indexing
(x, y, z)
(x, y+1, z)
Index (x, y, z) = p
Index (x, y+1, z) = p + Nx Sparse Grid: Difficult Indexing
(x, y, z)
(x, y+1, z)
Index (x, y, z) = p Index (x, y+1, z) = ??? Sparse Grid Point Lookup
Linear search
Tree-based search
Hybrid tree search + dense indexing Grid Point Contains Reference to Data Grid Point 14
Location (x,y,z) Data Reference
(x, y, z) Grid Point Contains Reference to Data Grid Point 14 Grid Index 14 15 16 Location (x,y,z) (11,11,5) (11,12,5) (Location) Data Reference Grid Data 3 8 (Reference)
(x, y, z) Grid Point Contains Reference to Data Grid Point 14 Grid Index 14 15 16 Location (x,y,z) (11,11,5) (11,12,5) … Data Reference Grid Data 3 8 …
density ρ0 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ9
(x, y, z) X-momentum ρu0 ρu1 ρu2 ρu3 ρu4 ρu5 ρu6 ρu7 ρu8 ρu9
Y-momentum ρv0 ρv1 ρv2 ρv3 ρv4 ρv5 ρv6 ρv7 ρv8 ρv9
Z-Momentum ρw0 ρw1 ρw2 ρw3 ρw4 ρw5 ρw6 ρw7 ρw8 ρw9
Energy E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 Simulation Data Grid Point Contains Reference to Data Grid Point 14 Grid Index 14 15 16 Location (x,y,z) (11,11,5) (11,12,5) … Data Reference Grid Data 3 8 …
density ρ0 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ9
(x, y, z) X-momentum ρu0 ρu1 ρu2 ρu3 ρu4 ρu5 ρu6 ρu7 ρu8 ρu9
Y-momentum ρv0 ρv1 ρv2 ρv3 ρv4 ρv5 ρv6 ρv7 ρv8 ρv9
Z-Momentum ρw0 ρw1 ρw2 ρw3 ρw4 ρw5 ρw6 ρw7 ρw8 ρw9
Energy E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 Simulation Data Grid Adaptation
Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9
Location x0y0 x1y1 x2y2 x3y3 x4y4 x5y5 x6y6 x7y7 x8y8 x9y9 Reference 0 1 2 3 4 5 6 7 8 9 Grid Adaptation
Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9
Location x0y0 x1y1 x2y2 x3y3 x4y4 x5y5 x6y6 x7y7 x8y8 x9y9 Reference 0 1 2 3 4 5 6 7 8 9 Grid Adaptation
Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9
Location x0y0 x1y1 x2y2 x3y3 x4y4 x5y5 x6y6 x7y7 x8y8 x9y9 Reference 0 1 2 3 4 5 6 7 8 9 Grid Adaptation
Grid Index P0 P2 P4 P6 P8
Location x0y0 x2y2 x4y4 x6y6 x8y8 Reference 0 2 4 6 8 Grid Adaptation
Grid Index P0 P1 P2 P3 P4
Location x0y0 x2y2 x4y4 x6y6 x8y8 Reference 0 2 4 6 8 Grid Transformations Track Data
Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9
Location x0y0 x1y1 x2y2 x3y3 x4y4 x5y5 x6y6 x7y7 x8y8 x9y9 Reference 0 1 2 3 4 5 6 7 8 9 Grid Transformations Track Data
Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9
Location x0y0 x1y1 x2y2 x3y3 x4y4 x5y5 x6y6 x7y7 x8y8 x9y9 Reference 0 1 2 3 4 5 6 7 8 9
Delete Add Sort
Transpose Grid Transformations Track Data
Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9
Location x0y0 x1y1 x2y2 x3y3 x4y4 x5y5 x6y6 x7y7 x8y8 x9y9 Reference 0 1 2 3 4 5 6 7 8 9
Delete Add Sort
Transpose
Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9
Location x4y4 x7y7 x9y9 x0y0 x5y5 x1y1 x8y8 x3y3 x6y6 x2y2 Reference 4 7 9 0 5 1 8 3 6 2 Losing Coalesced Memory Access
Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 Reference 0 1 2 3 4 5 6 7 8 9
ρ0 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ9
ρu0 ρu1 ρu2 ρu3 ρu4 ρu5 ρu6 ρu7 ρu8 ρu9
ρv0 ρv1 ρv2 ρv3 ρv4 ρv5 ρv6 ρv7 ρv8 ρv9
ρw0 ρw1 ρw2 ρw3 ρw4 ρw5 ρw6 ρw7 ρw8 ρw9
E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 Coalesced Simulation Data Losing Coalesced Memory Access
Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 Reference 4 7 9 0 5 1 8 3 6 2
ρ0 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ9
ρu0 ρu1 ρu2 ρu3 ρu4 ρu5 ρu6 ρu7 ρu8 ρu9
ρv0 ρv1 ρv2 ρv3 ρv4 ρv5 ρv6 ρv7 ρv8 ρv9
ρw0 ρw1 ρw2 ρw3 ρw4 ρw5 ρw6 ρw7 ρw8 ρw9
E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 Uncoalesced Simulation Data Restoring Coalesced Memory Access
Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9
Location x4y4 x7y7 x9y9 x0y0 x5y5 x1y1 x8y8 x3y3 x6y6 x2y2 Reference 4 7 9 0 5 1 8 3 6 2
ρ0 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ9
ρu0 ρu1 ρu2 ρu3 ρu4 ρu5 ρu6 ρu7 ρu8 ρu9
ρv0 ρv1 ρv2 ρv3 ρv4 ρv5 ρv6 ρv7 ρv8 ρv9
ρw0 ρw1 ρw2 ρw3 ρw4 ρw5 ρw6 ρw7 ρw8 ρw9
E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 Uncoalesced Simulation Data Restoring Coalesced Memory Access
Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P0 P1 P2 P3 P4 P5 P6 P7 P8 P9
Location x4y4 x7y7 x9y9 x0y0 x5y5 x1y1 x8y8 x3y3 x6y6 x2y2 x4y4 x7y7 x9y9 x0y0 x5y5 x1y1 x8y8 x3y3 x6y6 x2y2 Reference 4 7 9 0 5 1 8 3 6 2 Reset Indices 0 1 2 3 4 5 6 7 8 9
ρ0 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ9 ρ4 ρ7 ρ9 ρ0 ρ5 ρ1 ρ8 ρ3 ρ6 ρ2
ρu0 ρu1 ρu2 ρu3 ρu4 ρu5 ρu6 ρu7 ρu8 ρu9 ρu4 ρu7 ρu9 ρu0 ρu5 ρu1 ρu8 ρu3 ρu6 ρu2 Reorder ρv0 ρv1 ρv2 ρv3 ρv4 ρv5 ρv6 ρv7 ρv8 ρv9 Simulation ρv4 ρv7 ρv9 ρv0 ρv5 ρv1 ρv8 ρv3 ρv6 ρv2 Data ρw0 ρw1 ρw2 ρw3 ρw4 ρw5 ρw6 ρw7 ρw8 ρw9 ρw4 ρw7 ρw9 ρw0 ρw5 ρw1 ρw8 ρw3 ρw6 ρw2
E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 E4 E7 E9 E0 E5 E1 E8 E3 E6 E2 Uncoalesced Simulation Data Coalesced Simulation Data
Coupled Multi-Physics
Advection
Chemistry
Thermodynamics
Diffusion Real Gas Properties
Extreme pressure requires real-gas properties, evaluated at each grid node . Viscosity Pressure . Conductivity . Diffusion constants . Per-species diffusion rates
Independent Calculations Temperature
Massively Parallelizable O2 phase diagram, Z-axis is density (colour: liquid is blue, gas is white) Strong Temperature Dependence
Arrhenius equation includes temperature in exponent Chemical Kinetic Model
H H O + O O H H O H H H H
H2 O2 H2O Chemical Kinetic Model
H H O + O O H H O X H H H H
H2 O2 H2O
O H H O O O H OH O H2O2 O H O H
HO2 H Chemical Kinetic Model
H H O + O O H H O H H H H
H2 O2 H2O
O H H O O O H OH O H2O2 O H O H
HO2 H Methane Chemistry
O O H H O C O H H + O C + O H H O O H H
CH4 O2 CO2 H2O Methane Chemistry 53 Species, 325 Reactions
O O H H O C O H H + O C + O H H O O H H
CH4 O2 CO2 H2O
(Actual simulation data, rendered with a ray tracer) Acknowledgements
Oleg Vasilyev, University of Colorado Boulder Department of Mechanical Engineering
Jonathan Regele, Iowa State University Department of Aerospace Engineering
Don Lamb The Flash Center for Computational Science
Praveen Ramaprabhu University of North Carolina at Charlotte
Joseph Oefelein, Sandia National Laboratories Combustion Research Facility
Marc Massot Ecole Centrale, Paris Want To Help?
If you have a software or physics background and you’re interested in working with us, please contact
research@spacex.com