ANL/ALCF-16/3 Camellia v1.0 Manual Part I Argonne Leadership Computing Facility About Argonne National Laboratory Argonne is a U.S. Department of Energy laboratory managed by UChicago Argonne, LLC under contract DE-AC02-06CH11357. The Laboratory’s main facility is outside Chicago, at 9700 South Cass Avenue, Argonne, Illinois 60439. For information about Argonne and its pioneering science and technology programs, see www.anl.gov. DOCUMENT AVAILABILITY Online Access: U.S. Department of Energy (DOE) reports produced after 1991 and a growing number of pre-1991 documents are available free via DOE’s SciTech Connect (http://www.osti.gov/scitech/) Reports not in digital format may be purchased by the public from the National Technical Information Service (NTIS): U.S. Department of Commerce National Technical Information Service 5301 Shawnee Rd Alexandria, VA 22312 www.ntis.gov Phone: (800) 553-NTIS (6847) or (703) 605-6000 Fax: (703) 605-6900 Email: [email protected] Reports not in digital format are available to DOE and DOE contractors from the Office of Scientific and Technical Information (OSTI): U.S. Department of Energy Office of Scientific and Technical Information P.O. Box 62 Oak Ridge, TN 37831-0062 www.osti.gov Phone: (865) 576-8401 Fax: (865) 576-5728 Email: [email protected] Disclaimer This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor UChicago Argonne, LLC, nor any of their employees or officers, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof. The views and opinions of document authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof, Argonne National Laboratory, or UChicago Argonne, LLC. ANL/ALCF-16/3 Camellia v1.0 Manual Part I prepared by Nathan V. Roberts Argonne Leadership Computing Facility, Argonne National Laboratory September 28, 2016 Camellia v1.0 Manual Part I Nathan V. Roberts September 28, 2016 Contents 1 Introduction 4 1.1 CoreFeatures ........................... 4 1.2 Structure of this Manual . 5 2 Some Preliminaries 7 2.1 CodingConventions. 7 2.1.1 Reference-Counted Pointers in Camellia . 7 2.1.2 CommandLineProcessor . 8 2.1.3 SomeUsefulC++11Features . 9 2.1.4 The auto Keyword . 9 2.2 Terminology . 10 2.3 ABriefIntroductiontoDPG . 11 2.3.1 AnExample: UltraweakPoisson. 12 2.4 Core Classes in Camellia . 13 3 Stokes Cavity Flow Using DPG 17 3.1 TheDPGUltraweakFormulation . 17 3.2 Stokes Bilinear Form Implementation . 19 3.3 TheCavityFlowProblem . 21 3.4 CavityFlowImplementation . 23 3.4.1 Defining the Mesh . 23 3.4.2 Boundary Conditions and Right-Hand Side . 24 3.4.3 TheTestSpaceInnerProduct . 26 3.4.4 SolutionandVisualization . 26 3.4.5 AdaptiveMeshRefinements . 27 3.5 StokesCavityFlowDriver . 30 1 4 Navier-Stokes Cavity Flow Using DPG 32 4.1 Ultraweak Formulation for Navier-Stokes . 32 4.2 Adjusting the RHS for Navier-Stokes Linearization . 35 4.3 Test Space Inner Product . 36 4.4 Boundary Conditions . 36 4.5 The Navier-Stokes Solve and the Newton Stopping Criterion . 37 4.6 ADynamicNewtonThreshold. 38 4.7 Navier-StokesCavityFlowDriver . 41 5 Formulations: Poisson, Stokes, and Navier-Stokes 44 5.1 The PoissonFormulation Class . 45 5.2 The StokesVGPFormulation Class . 47 5.2.1 StokesVGPFormulation: Other Notable Features . 49 5.3 The NavierStokesVGPFormulation Class . 52 5.4 Drivers . 54 5.4.1 Poisson Driver . 54 5.4.2 StokesCavityFlowDriver . 56 5.4.3 Navier-StokesCavityFlowDriver . 58 6 Global Linear Solvers: Direct and Iterative Options 60 6.1 Solver Interface . 60 6.2 Static Condensation . 61 6.3 Geometric Multigrid Preconditioned Iterative Solves . 62 6.3.1 Other GMGSolver Options . 63 A Other Features 66 A.1 SupportForTransientProblems . 67 A.2 CustomFunctionsinCamellia . 68 A.3 OtherFiniteElementMethods. 68 A.4 CustomRefinementStrategies . 69 A.5 UnitTests............................. 70 A.6 Visualization ........................... 71 A.7 CommonIssueswithMPI . 72 A.8 Distributed Algorithms using Camellia’s MPIWrapper ..... 73 A.9 The BasisReconciliation Class . 76 A.10ImportingMeshesUsingMOAB . 76 A.11Exporting Matrices for External Analysis . 77 2 Acknowledgments This work was supported by the Office of Science, U.S. Department of Energy, under Contract DE-AC02-06CH11357. This research used resources of the Argonne Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC02-06CH11357. I gratefully acknowledge the support of Argonne and the Argonne Leadership Computing Facility during the postdoctoral work that allowed much of the development documented in this manual. I thank Brendan Keith for critical feedback regarding the manuscript. I thank Paul Fischer for advice regarding geometric multigrid precondi- tioning, and Jesse Chan for collaborating on the geometric multigrid work. I thank Jesse Chan, Truman Ellis, and Brendan Keith for their contributions to Camellia development. I thank Leszek Demkowicz for ongoing collaborations regarding discon- tinuous Petrov-Galerkin methods, and for his support during my PhD. work and since. 3 Chapter 1 Introduction Camellia [14] began as an e↵ort to simplify implementation of efficient solvers for the discontinuous Petrov-Galerkin (DPG) finite element methodology of Demkowicz and Gopalakrishnan [6, 7]. Since then, the feature set has expanded, to allow implementation of traditional continuous Galerkin methods, as well as discontinuous Galerkin (DG) methods, hybridizable DG (HDG) methods [5], first-order-system least squares (FOSLS) [3], and the primal DPG method [8]. This manual serves as an introduction to using Camellia. We begin, in Section 1.1, by describing some of the core features of Camellia. In Section 1.2 we provide an outline of the manual as a whole. 1.1 Core Features Camellia provides mechanisms for rapid specification for many of the standard parts of finite element methods, including: bilinear formulations, • boundary conditions, and • material and load data, • as well as the inner product on the test space, a requirement for DPG solves. Camellia supports both field and trace variables—that is, variables defined on the mesh volume as well as those defined on the mesh skeleton. For both of these, vector and scalar variables are supported. Basis functions conforming 4 to the exact sequence—that is, H1, H(div) , H(curl), and L2—are provided on each of the supported topologies, which include lines in 1D, triangle and quadrilaterals in 2D, and hexahedra and tetrahedra in 3D. Support for regular h-refinements of all these—with the exception of tetrahedra in 3D—is provided.1 Essentially arbitrary polynomial orders are supported for hypercube topologies (some limitations are imposed on simplices). Because Camellia began as a research tool investigating DPG in the context of problems with relatively simple geometries, support for curvilinear elements is currently limited to 2D geometries specified parametrically (a somewhat homegrown approach). While some examples can be found in the source code, we do not discuss curvilinear geometry in this manual; we expect to support curvilinear geometry in arbitrary dimensions in a fashion similar to other finite element codes (likely through nodal weights on each element to specify geometry isoparametrically). This should be a relatively simple extension of the current straight-edged implementation, and we hope to have support for this in the near future. At present, Camellia supports only real-valued, double-precision variables— though it is worth noting that complex problems can be implemented by introducing a real-valued variable for each component of complex variables. We would like to extend Camellia to support templated scalar types in a future release—this will allow direct support of complex variables, as well as single-precision and quad-precision variables. Essentially all computationally-intensive features support distributed computation using MPI. Load-balancing is implemented using Zoltan. For DPG systems, a robust iterative solver using geometric multigrid is provided. Support for static condensation2 for ultraweak DPG systems is also provided. 1.2 Structure of this Manual In Chapter 2, we cover a few preliminary concepts and conventions that will be useful in the code examples. The remainder of the manual is driven by examples. The first part of the manual focuses on DPG solvers, starting 1Support for tetrahedral refinements is a relatively simple extension that we hope to add in the near future. 2Static condensation eliminates local degrees of freedom, to reduce the size of the global system at the cost of some local computation. In the present release of Camellia, we support element-local elimination of discontinuous field variables. 5 in Chapter 3, wherein we demonstrate the implementation of an adaptive Stokes solver for the steady 2D lid-driven cavity flow problem, using DPG (a brief introduction to DPG is also provided there). The third chapter extends that implementation to support Navier-Stokes (and discusses