Utah State University DigitalCommons@USU

All Graduate Theses and Dissertations Graduate Studies

12-2009

Parallelization of Performance Limiting Routines in the Computational Fluid Dynamics General Notation System Library

Kyle Horne Utah State University

Follow this and additional works at: https://digitalcommons.usu.edu/etd

Part of the Mechanical Engineering Commons

Recommended Citation Horne, Kyle, "Parallelization of Performance Limiting Routines in the Computational Fluid Dynamics General Notation System Library" (2009). All Graduate Theses and Dissertations. 522. https://digitalcommons.usu.edu/etd/522

This Thesis is brought to you for free and open access by the Graduate Studies at DigitalCommons@USU. It has been accepted for inclusion in All Graduate Theses and Dissertations by an authorized administrator of DigitalCommons@USU. For more information, please contact [email protected]. PARALLELIZATION OF PERFORMANCE LIMITING ROUTINES IN THE COMPUTATIONAL FLUID DYNAMICS GENERAL NOTATION SYSTEM LIBRARY

by

Kyle Horne

A thesis submitted in partial fulfillment of the requirements for the degree

of

MASTER OF SCIENCE

in

Mechanical Engineering

Approved:

Dr. Thomas Hauser Dr. Robert Spall Major Professor Committee Member

Dr. Heng Ban Dr. Byron Burnham Committee Member Dean of Graduate Studies

UTAH STATE UNIVERSITY Logan, Utah

2009 ii

Copyright © Kyle Horne 2009

All Rights Reserved iii

Abstract

Parallelization of Performance Limiting Routines in the

Computational Fluid Dynamics General

Notation System Library

by

Kyle Horne, Master of Science

Utah State University, 2009

Major Professor: Dr. Thomas Hauser Department: Mechanical and Aerospace Engineering

The Computational Fluid Dynamics General Notation System provides a unified way in which computational fluid dynamics data can be stored, but does not support the parallel I/O capabilities now available from version five of the Hierarchical Data Format library which serves as a back end for the standard. To resolve this deficiency, a new parallel extension library has been written and benchmarked for this work which can write files compliant with the standard using parallel

file access modes. When using this new library, the write performance shows an increase of four- fold in some cases when compared to the same hardware operating in serial. Additionally, the use of parallel I/O allows much larger cases to be written since the problem is scattered across many nodes of a cluster, whose aggregate memory is much greater than that found on a single machine.

These developments will allow computational fluid dynamics simulations to execute faster, since less time will be spent waiting for each time step to finish writing, as well prevent the need for lengthy reconstruction of data after the completion of a simulation.

(293 pages) iv

Acknowledgments

Firstly, I would like to thank my wife, Lydia, for supporing me in the completion of this project. Her efforts to make our humble apartment our home have made all the difference.

Secondly, I want to thank my parents for encouraging my curiosity and scientific interest from an early age, and giving me the freedom to become who I am.

I would also like to thank Dr. Thomas Hauser for employing me since my senior year of undergraduate studies. He has funded my research and provided me with opportunities to grow that

I had not even considered before. Additionally, I would thank the faculty for their efforts in making the undergraduate and graduate programs at Utah State University environments in which students can succeed. Much credit must also go to Bonnie Ogden for making sure that graduate students get all their paperwork done.

Additionally, I would thank Mr. Mike Kennedy, Mrs. Vanessa Liveris, Ms. Nicole Burt, and

Mr. Gussman from Neuqua Valley High School for the sound foundation in science and mathemat- ics which I received there.

Kyle Horne v

Contents

Page

Abstract ...... iii

Acknowledgments ...... iv

List of Tables ...... viii

List of Figures ...... ix

1 Introduction ...... 1 1.1 Scientific Computing ...... 1 1.1.1 Data Storage History ...... 1 1.1.2 System Level I/O ...... 2 1.1.3 Hierarchical Data Format ...... 3 1.1.4 Supercomputing ...... 3 1.1.5 Vector Computers ...... 3 1.1.6 Cluster Computing ...... 4 1.2 Computational Fluid Dynamics ...... 5 1.2.1 Problem Formulation ...... 5 1.2.2 Solution Algorithms ...... 7 1.2.3 Parallel Implementations ...... 8 1.2.4 Storage Concerns ...... 8

2 Problem Description ...... 13 2.1 Problem Introduction ...... 13 2.2 Problem Background ...... 16 2.3 Computational Fluid Dynamics General Notation System ...... 18 2.3.1 Standard Interface Data Structures ...... 18 2.3.2 Advanced Data Format ...... 18 2.3.3 Hierarchical Data Format version 5 ...... 20 vi

2.3.4 Mid-level Library ...... 20

3 Literature Review ...... 21 3.1 Previous Work ...... 21 3.2 Parallel Input/Output ...... 22 3.2.1 Message Passing Interface Standard ...... 22 3.2.2 Network Attached Storage ...... 24 3.2.3 Message Passing Interface-I/O ...... 24 3.2.4 Parallel File Systems ...... 24 3.2.5 Hierarchical Data Format version 5 ...... 27

4 Parallel Implementation for the CFD General Notation System ...... 28 4.1 Design ...... 28 4.2 Features ...... 29

5 Results ...... 31 5.1 Hardware Setup ...... 31 5.2 Benchmark Selection ...... 33 5.2.1 IOR ...... 33 5.2.2 pCGNS ...... 33 5.3 Benchmark Results ...... 34 5.3.1 IOR ...... 35 5.3.2 pCGNS ...... 38

6 Summary and Conclusion ...... 48

Bibliography ...... 51

Appendix ...... 58

A pCGNS Source Code & Documentation ...... 59 A.1 Data Structure Index ...... 59 A.1.1 Data Structures ...... 59 A.2 File Index ...... 59 A.2.1 File List ...... 59 A.3 Data Structure Documentation ...... 60 vii

A.3.1 base_s Struct Reference ...... 60 A.3.2 coords_s Struct Reference ...... 63 A.3.3 file_s Struct Reference ...... 64 A.3.4 iter_s Struct Reference ...... 69 A.3.5 section_s Struct Reference ...... 70 A.3.6 slice_s Struct Reference ...... 72 A.3.7 sol_s Struct Reference ...... 74 A.3.8 zone_s Struct Reference ...... 76 A.4 File Documentation ...... 80 A.4.1 benchmark.c File Reference ...... 80 A.4.2 benchmark.c ...... 88 A.4.3 open_close.c File Reference ...... 94 A.4.4 open_close.c ...... 96 A.4.5 pcgns_util.c File Reference ...... 97 A.4.6 pcgns_util.c ...... 119 A.4.7 pcgns_util.h File Reference ...... 140 A.4.8 pcgns_util.h ...... 163 A.4.9 pcgnslib.c File Reference ...... 168 A.4.10 pcgnslib.c ...... 186 A.4.11 pcgnslib.h File Reference ...... 228 A.4.12 pcgnslib.h ...... 249 A.4.13 test_base.c File Reference ...... 252 A.4.14 test_base.c ...... 254 A.4.15 test_queue.c File Reference ...... 256 A.4.16 test_queue.c ...... 258 A.4.17 test_unstructured.c File Reference ...... 260 A.4.18 test_unstructured.c ...... 262 A.4.19 test_zone.c File Reference ...... 265 A.4.20 test_zone.c ...... 266 A.4.21 thesis_benchmark.c File Reference ...... 268 A.4.22 thesis_benchmark.c ...... 274 viii

List of Tables

Table Page

4.1 API of the pCGNS library as implemented in pcgnslib.c and declared in pcgnslib.h. Software using the library to access CGNS files in parallel must do so using the routines listed here...... 30 ix

List of Figures

Figure Page

1.1 A generalized diagram of a cluster computer (Upper) and a shared-memory com- puter (Lower). The abbreviations used in the diagram are as follows: CPU=Central Processing Unit; NIC=Network Interface Card; HD=Hard Disk. The dashed line connecting all the components of each computer represents the primary data bus of the machine. The simplicity of the shared-memory computer architecture com- pared to that of the cluster is apparent, making shared-memory computers desirable when possible. Unfortunately, the performance required from the main data bus in a shared-memory computer limits the scalability of the platform...... 6

1.2 Mesh partitioned for parallel computation using Gmsh [1]. The mesh has been split into four partitions algorithmically, with the intent of minimizing the number of boundary edges between each partition. This minimization lowers the communica- tions overhead incurred when executing a CFD simulation on the mesh...... 9

1.3 Mesh from a CFD solution of the common driven cavity problem on an unstructured grid. The mesh was generated using Gmsh [1], and is composed of triangles and quadrilaterals on the interior, with one dimensional cells all around the boundary. . 11

1.4 Velocity field from a CFD solution of the common driven cavity problem on an unstructured grid. The solution was obtained using a code written by the author and is used only as an example of CFD results. The color corresponds to the velocity magnitude at each cell center and the lines portray both the direction and magnitude of the velocity...... 11

1.5 Pressure field from a CFD solution of the common driven cavity problem on an unstructured grid. The solution was obtained using a code written by the author and is used only as an example of CFD results. The color corresponds to the relative pressure at each cell center...... 12

1.6 General transported quantity field from a CFD solution of the common driven cavity problem on an unstructured grid. The solution was obtained using a code written by the author and is used only as an example of CFD results. The color corresponds to the magnitude of the quantity phi at each cell center...... 12 x

2.1 Storage topology of a modern computer cluster in two configurations. Configuration (a) shows the traditional CFD approach to I/O, where each node writes a separate file to disk. These files must be combined to obtain the full solution in a process which can be very time consuming. Configuration (b) shows the object of this work, being a CFD I/O method in which the partial solution from each node is written into the appropriate location within a single solution file for the whole problem. . . . . 15

2.2 Diagram depicting the hierarchical nature of a CGNS file. Data is broken into groups and sub-groups, with the location and type of a group in the hierarchy con- trolling how the data is interpreted...... 19

4.1 Software stack used by the pCGNS library. Layers are ordered from most abstract at the top to least abstract at the bottom. Each layer relies on lower layers to implement functionality. This allows higher layers to be independent of the implementation of lower layers. Thus, as long as the MPI implementation used can take advantage of the file system on a machine, HDF5 and pCGNS will also be able to use that file system...... 29

5.1 Topology of the computing clusters and network attached storage at Utah State Uni- versity. A single high speed switch connects all the storage to the clusters, and serves as the I/O backbone for Wasatch. The network connections between the Wasatch nodes and the main switch are composed of pairs of 1 gigabit Ethernet connections bonded to form a 2 gigabit connection for each node. The connection between the Panasas storage (/panfs) and the switch is composed of four 10 gigabit Ethernet connections...... 32

5.2 Data distribution scenarios for benchmarking. Part (a) shows two zones on one node. Part (b) shows one zone per node on four nodes. Part (c) shows four zones on two nodes. Part (d) shows two zones on four nodes. Part (e) shows four zones on four nodes with zones spread across multiple nodes...... 34

5.3 Performance of the MPI-IO library on Wasatch using the IOR benchmark. IRead stands for independent read, CWrite for collective write, and so on. The hints rec- ommended by Panasas are active to ensure best performance...... 35

5.4 Performance of the HDF5 library on Wasatch using the IOR benchmark. IRead stands for independent read, CWrite for collective write, and so on. The hints rec- ommended by Panasas are active to ensure best performance...... 36

5.5 Performance of HDF5 on Wasatch using collective writes only, on a 256 MB data set. The hints recommended by Panasas are inactive to give results which are com- parable to the pCGNS benchmarks which were not run with any hints activated. . . 37 xi

5.6 Results of pCGNS benchmark (a). The larger file sizes experience approximately the same bandwidth, while the smaller sizes enjoy the benefits of buffered file I/O. The speedup of all sizes of file is approximately one, which demonstrates good efficiency in the pCGNS library, and all layers beneath it...... 39

5.7 Results of pCGNS benchmark (b). Large file sizes perform better then smaller ones. The largest file size performs four times better in parallel I/O than the serial I/O, but performance nearly levels off after four nodes, possibly due to the topology of the storage at USU...... 40

5.8 Results of pCGNS benchmark (c). Two zones are written by each node while keep- ing the file size constant, causing less and less data to be written with each additional node. The performance is still acceptable, but does level off after four nodes. . . . 42

5.9 Results of pCGNS benchmark (c). Four zones are written by each node while keep- ing the file size constant, causing less and less data to be written with each additional node. The performance is no longer an improvement over the serial case with the smallest data size, although larger files still perform acceptably...... 43

5.10 Results of pCGNS benchmark (c). Eight zones are written by each node while keeping the file size constant, causing less and less data to be written with each additional node. The performance is no longer an improvement over the serial case with all but the largest data size, which barely performs better than the serial version. 44

5.11 Results of pCGNS benchmark (d). Multiple nodes share each zone, requiring HDF5 to sort out any possible data overlaps. This causes a severe slow down of the I/O operations. None of the tested file sizes outperformed the serial case...... 45

5.12 Results of pCGNS benchmark (e). The most general of the benchmarks, type-e demonstrates the performance when multiple zones are written by multiple pro- cesses. This situation results from parallel partitioning independent of the zone arrangement of the domain...... 47 1

Chapter 1

Introduction

1.1 Scientific Computing Throughout the history of scientific progress and engineering development there has been a need to compute the values of mathematical expressions to obtain results from theoretical work. In the earliest days, these computations were carried out by hand. This later developed into specialized calculating aids such as the Arabic abacus, Japanese soroban, or the Russian s’choty [2]. While faster than the unaided mind, these methods were still just an aid to the user’s efforts. Modern computing machines started to appear in the early twentieth century and have continued to develop to this day [3].

1.1.1 Data Storage History Since the dawn of the computer age, there has been the need to get information into computers and to then receive the results of the calculations. With the earliest mechanical computers, this was often done in the form of nobs or dials, which allowed the operator to set the inputs, perhaps select one of a small set of operations to be done, and then read the results. While this method worked, and was faster than doing the calculations by hand, it was still very cumbersome and dramatically slowed down the entire process. Later the development of punch cards significantly improved the situation, allowing vast amounts of data to be entered and processed; the results of the computation themselves often being punched onto cards. The fact that large amounts of data could be processed in addition to the ability to write results in a machine-readable format contributed to the punch card being a dominant method of I/O even into the 1980’s [3].

Later developments include the use of punched tape and magnetic tape storage, along with various lesser-used mediums. The real advancement toward modern computer storage came with the first hard drives, which only became affordable for common use in the 1970’s and 1980’s. 2

The hard disk offers the storage of vast amounts of data with the ability to access it at relatively high speeds [4]. The capacity of hard drives has continued to increase exponentially since their invention in the 1950’s, and it is this large capacity combined with their flexibility that has made them the primary form of data storage on modern computers, from the largest cluster to the smallest laptop [5].

1.1.2 System Level I/O On modern computer systems, programs nearly always rely on an operating system of some sort to provide support facilities to the program, often done in the form of some abstraction of the hardware into a more universal model. This allows programs to be written for one piece of computing hardware, but run on any hardware that provides the same abstract interface. The abstract interfaces used generally come in the form of a C application programing interface (API). Many operating systems provide the POSIX [6] API by which a program can interface with the system in a predefined way. Abstract interfaces are also provided by some programing languages, such as

Fortran, with the actual implementation of the abstraction layer being found within the compiler.

Using this model, when a program needs to make some sort of I/O, it makes a request of the operating system to accomplish this. Most operating systems use libraries to provided the APIs, which themselves access the operating system’s kernel directly. The kernel is the core of the operat- ing system which manages the lowest level of interaction between the hardware and software. This is done through the use of drivers, which present an interface to the computer’s hardware in a form that the kernel can use. Many kernels include vast databases of drivers for commonly encountered hardware, but if the kernel does not have native support for a device, a new driver can generally be loaded into the kernel to allow for operation of the device [7].

The exact nature of the I/O need not be storage access. In the broadest terms, I/O is any interaction between the computer and the outside world. This includes keyboard input, mouse movements, screen display, printer access, network connections and many more forms of I/O. For

CFD and other simulation codes, the most import I/O forms are storage and network access [5]. 3

1.1.3 Hierarchical Data Format Nearly all scientific computing produces data which needs to be analyzed after completion of the simulation. Often these data sets can be very large and quite complicated in structure, making storage and post processing of the data more difficult. To solve this problem there are many file formats which are specific to each discipline within scientific computing, but this solution creates a new problem of needing specialized tools to be written for every field of study. To resolve this problem, general purpose data storage file formats have evolved which can describe any sort of data.

One such file format is the hierarchical data format, or HDF [8].

HDF implements a sort of file system within a file. Data is arranged in groups, which can be layered and nested just like directories on a file system [9]. The actual data can be stored in a variety of built-in data types, or new data types can be created for the needs of a specific program. Arrays of built-in and user defined types can also be saved. This simple but flexible arrangement allows

HDF to describe many different types of data, ranging from historic stock data to CFD simulation results.

1.1.4 Supercomputing The fundamental motivation for computing hardware has always been the improvement of computation time required to obtain some result. Many advances have been made in the field of computing, especially in the last century with the development of the modern transistor-based dig- ital computer [3], but even with the significant computational power available to modern desktop computers the needs of the scientific and engineering communities continue to demand more com- putational power to solve the problems with which they work.

1.1.5 Vector Computers Early efforts to build computers which met these unusually large requests for processing power generally revolved around building a single monolithic computer with more processing power. This led researchers in the field of computing, such as Seymour Cray, to design vector computers. A vector computer is based on the execution model of single instruction multiple data, or SIMD. These computers were designed with large vector computation registers capable of holding many floating 4 point values and running the same operation on all of them simultaneously [10]. The ramifications of this hardware design are fully grasped when one considers the frequency with which one encounters vector or array operations when developing numerical analysis software for simulation of physical phenomena [11].

1.1.6 Cluster Computing By the late 1990’s the speed of commodity computers was closing the gap with special-purpose supercomputers on a per processor level. The main difference between the two types of computers largely became a matter of scale rather than processor design. This development paved the way for modern computer clusters to make their debut. Initially named after the first computer of this type, cluster supercomputers are often referred to as Beowulf clusters [10]. A cluster supercomputer is a set of “off the shelf” computers which work on a single problem as a group rather than individually

[12]. Because the components for clusters are either identical, or at least similar to standard desktop computer components, the cost of clusters is lower than that of the traditional supercomputers. The expertise required to design a cluster is also significantly less, since most of the work entailed is networking the computers together, not designing new computer hardware [13].

Computers in a cluster are connected using a network of some sort, which is generally called the interconnect of the cluster. This network is used by the various computers in the cluster, called nodes, to communicate information to each other about the solution process being executed. The kind of data that must be sent back and forth between the nodes of a cluster is highly dependent on the problem being solved. In CFD simulations, the data is generally the current flow solution along the domain boundary between sections of the mesh owned by different processes on the cluster [14, 15].

While any network can be used as the cluster interconnect, specialized network hardware and protocols have been developed to meet the needs of cluster computers. Some examples of these are Myrinet [16–19] and Infiniband [20–23]. Specialized networks are commonly used in clusters because the interconnect of the cluster is being used as an equivalent of the main data bus on a monolithic computer, and thus requires high speed performance for economical operation of the 5 machine [12, 24]. The data bus is a system on the mother board of a computer which carries infor- mation from one component of the computer to another. Any time data is accessed from memory, sent to the network, or accessed on storage, it needs to pass from one component to another via the data bus, and generally makes a stop at the CPU. All computers have a data bus, but not all buses are equal, nor are all routes through the bus of equivalent speeds.

Figure (1.1) portrays a cluster computer and a shared-memory computer. It can be seen that the cluster computer relies on its interconnect to send data from some processors to other processors, but that on each node of the cluster a data bus moves data from point to point. This is in contrast to the shared-memory computer which uses a single very high speed bus to move the data around the machine. The need to have a high speed, high bandwidth data bus greatly complicates the design of shared-memory computers and limits the scales on which they can be built. The path from the processors and memory to the storage of each machine is also quite different. The shared-memory computer’s data bus provides direct access from the CPUs and memory to the storage, whereas clusters often access a centralized storage system through a network separate from the primary interconnect.

1.2 Computational Fluid Dynamics The field of computational fluid dynamics concerns itself with the simulation of fluid flow using computers. This process is non-trivial due to the complexity of the equations which must be solved and the numerical properties of the algorithms used to approximate these equations. These difficulties are compounded by the scale of most CFD problems.

1.2.1 Problem Formulation Computational fluid dynamics codes generally work by solving some form of the differential equations which govern fluid flow. Equations (1.1) through (1.4) show those equations most often used as the governing equations by CFD codes, although there are many additional equations which may also be solved besides these. Some assumptions can be made to reduce the number of equa- tions which must be solved, but such assumptions restrict the code to certain flow regimes, such as incompressible, inviscid, etc. [25]. 6

Fig. 1.1: A generalized diagram of a cluster computer (Upper) and a shared-memory computer (Lower). The abbreviations used in the diagram are as follows: CPU=Central Processing Unit; NIC=Network Interface Card; HD=Hard Disk. The dashed line connecting all the components of each computer represents the primary data bus of the machine. The simplicity of the shared- memory computer architecture compared to that of the cluster is apparent, making shared-memory computers desirable when possible. Unfortunately, the performance required from the main data bus in a shared-memory computer limits the scalability of the platform. 7

The continuity equation, (1.1), enforces the conservation of mass during the simulation. In the case of steady-state incompressible solvers, it often also provides the coupling between velocity and pressure, since the incompressible assumption removes the physical relationship between them.

The momentum equation, (1.2), is actually a vector equation, which provides one equation for the conservation of momentum in each dimension of the simulation. These equations enforce Newton’s second law of motion, and here are presented with the Newtonian fluid shear tensor assumption already made.

Normally only found in compressible flows or when heat transfer is of interest, the energy equation, (1.3), and the accompanying equations of state, (1.4), are used to couple the pressure and temperature to the density, as well as other material properties such as viscosity and thermal conductivity.

∂ρ ∂ + (ρui)=0 (1.1) ∂t ∂xi

∂ ∂ ∂p ∂ ∂ui ∂uj ∂uk (ρui)+ (ρuiuj)= + µ + + δijµ￿ (1.2) ∂t ∂xj −∂xi ∂xi ￿ ￿∂xj ∂xi ￿ ∂xk ￿ ∂e ∂e ∂Q ∂ ∂ ρ + u = + k T +Φ (1.3) ∂t i ∂x ∂t ∂x ∂x ￿ i ￿ i i p = p (e, ρ) and T = T (e, ρ) (1.4)

1.2.2 Solution Algorithms Many different approaches have been used to solve the equations governing fluid flow, but most of them can be identified as one of three commonly encountered forms. Finite difference methods (FDM) can be used to solve the equations with very high computational efficiencies, and bring the added benefit of being easy to understand and program [26]. Unfortunately, these methods are restricted to structured computational grids [27], although these grids may be deformed in the physical coordinates. Most early investigations in CFD used finite difference methods [28].

Finite volume methods (FVM) allow for fully unstructured meshes to be used in the solution process, but they are more complex to understand and program. They do have the desirable quality that each term in the discretized form of the equations can be directly related back to some physical 8 meaning. This intuitive correlation is attractive to researchers in the field as it makes understanding the codes easier than it would otherwise be. Most current commercial CFD codes use the finite volume method [29].

Finite element methods (FEM) have recently become more popular in the field of computa- tional fluid dynamics, but suffer from very abstract derivations being required to build a full CFD solver. Like FVM, these methods can be used on arbitrarily complex meshes [27]. Unlike FDM and

FVM, finite element methods generally form the discretized equations in matrix form and solve the system directly, whereas FDM and FVM usually use an in-place iterative solution which does not require the full linear system to be expressed in matrix form [26].

1.2.3 Parallel Implementations Because of the computation cost of solving most problems in CFD, it is normal for solvers to be designed to take advantage of computing clusters or multiprocessor systems in order to ob- tain solutions more rapidly. This is generally accomplished by partitioning the physical domain of the problem into regions of approximately equal computational cost and then distributing these partitions among the computing resources available as shown in Figure (1.2). During solution, the processes which contain neighboring partitions of the domain pass solution information back and forth between each other, so that the solution of the entire domain is computed [15]. This process places large loads on the interconnect of computer clusters [30] and is one of the motivations for the development of faster interconnects such as Infiniband or Myrinet.

Beyond the desire for additional performance, CFD codes are also parallelized to allow larger problems to be solved than can be done with a single desktop computer. Since the domain of the problem is distributed among the nodes of a cluster, no single node needs to store the entire solution.

This allows problems to be solved which are too large to fit on lesser machines.

1.2.4 Storage Concerns Traditional CFD works by splitting the problem domain into smaller regions composing the mesh or grid of the problem and applying some sort of discretization scheme to the governing differential equations so as to satisfy those equations on each element or volume of the mesh. This 9

Fig. 1.2: Mesh partitioned for parallel computation using Gmsh [1]. The mesh has been split into four partitions algorithmically, with the intent of minimizing the number of boundary edges between each partition. This minimization lowers the communications overhead incurred when executing a CFD simulation on the mesh. 10 means that the field variables are stored as a finite set of sample points and interpolated between these known values. As a result, the mesh must be sufficiently refined to ensure accuracy, which causes the number of points at which the field variables must be stored to be very large.

Besides the size of the data sets used in CFD, the complexity of the data is also problematic for storage. Figures (1.3) through (1.6) show a sample CFD solution to the well-known driven cavity problem as computed by a code written by the author. The mesh is unstructured, which results in data structures needing to be stored which describe the location of each point, as well as the structure of each cell in the mesh. In this example, four field quantities are stored; those being the velocity component in the x-direction, the velocity component in the y-direction, the pressure, and an arbitrary parameter phi, which is transported by the fluid. Each of these variables must be stored and somehow correlated to the mesh in order for the data to be meaningful.

The complexity of storing CFD is only made worse by the use of computer clusters in the solution process, since none of the nodes of the cluster have a copy of the entire solution in memory.

This requires that the nodes somehow coordinate the reading and writing of solution data in such a way that it can be reassembled. Most often, this is accomplished by each process running on the cluster writing its own data in separate files. The files for each process are then recombined after completion of the simulation in order to obtain the final solution. Advancements in parallel I/O have removed the need for CFD codes to write one file per process, since multiple processes can now access the same file simultaneously [8]. 11

Fig. 1.3: Mesh from a CFD solution of the common driven cavity problem on an unstructured grid. The mesh was generated using Gmsh [1], and is composed of triangles and quadrilaterals on the interior, with one dimensional cells all around the boundary.

U 1.0000

0.75000

0.50000

0.25000

0.0000

Fig. 1.4: Velocity field from a CFD solution of the common driven cavity problem on an unstruc- tured grid. The solution was obtained using a code written by the author and is used only as an example of CFD results. The color corresponds to the velocity magnitude at each cell center and the lines portray both the direction and magnitude of the velocity. 12

P 1.6626

1.0677

0.47286

-.12202

-.71691

Fig. 1.5: Pressure field from a CFD solution of the common driven cavity problem on an unstruc- tured grid. The solution was obtained using a code written by the author and is used only as an example of CFD results. The color corresponds to the relative pressure at each cell center.

phi 1.0000

0.75000

0.50000

0.25000

0.0000

Fig. 1.6: General transported quantity field from a CFD solution of the common driven cavity problem on an unstructured grid. The solution was obtained using a code written by the author and is used only as an example of CFD results. The color corresponds to the magnitude of the quantity phi at each cell center. 13

Chapter 2

Problem Description

2.1 Problem Introduction A large portion of the work in fluid mechanics is now done in the form of computational fluid dynamics (CFD), in which a discretized form of the partial differential equations governing fluid

flow are solved on a particular domain using a computer. CFD is used in design and optimization for many different applications. Such applications are so diverse as to include both jet fighters and bleach bottles. It would be difficult to find someone who had never interacted with something de- signed using CFD. In order to attain the accuracy needed for such projects, CFD requires significant computing resources, and as such is often completed with the aid of some sort of supercomputer.

In the early days of CFD most supercomputers came in the form of monolithic vector computers which packed numerous processors into one single machine [10]. Modern supercomputers more of- ten take the form of computing clusters, in which numerous desktop-like computers work together on solving the discretized equations [3]. Utah State University possesses several such computing clusters through the Center for High Performance Computing to facilitate CFD based fluid dynamics research, as well as other studies.

CFD demands not only a great deal of processing power, but also large amounts of data stor- age. Even a simple CFD simulation can produce gigabytes of information at a tremendous rate, and all this data must be stored somewhere so that the researcher can analyze the results of the simula- tion. This situation is worsened in the case of unsteady 3d simulations. The process of reading or writing data by a program is generally termed I/O, which stands for Input/Output. On the old vector computers this was mostly a matter of providing sufficiently fast hard-drives in the computer which could keep up with the I/O demands of the CFD code. On modern computer clusters however, the problem becomes more complicated. Because of the distributed nature of computer clusters, the domain of a CFD problem must be broken up into small pieces and scattered across the nodes of the 14 cluster. This means that each node in the cluster only has a small part of the whole problem [31].

When solutions need to be written to disk for later examination, there are several options. Firstly, each node can simply write its part of the problem to some local or network storage, and the various pieces can be reassembled after completion of the simulation. While this solution has the benefit of simplicity, it does not provide desirable performance, since the reconstruction process is lengthy, and in extreme cases can take longer than the duration of the simulation itself. The second option is for the nodes of the cluster to all write to one file on the storage system simultaneously. For this to be done the nodes must cooperate on the I/O in parallel. This method is harder to program, but should be able to provide nearly the same write-time performance as the first method, without the need to reconstruct the data after completion of the simulation. Implementation of this second form of clustered CFD I/O is the purpose of this work. An overview of the two I/O methods is provided in Figure (2.1).

Since the data written by CFD codes tends to be quite complex, specialized file formats have been developed to standardize the storage and processing of results obtained from simulations. One such file format is known as the CFD General Notation System (CGNS) [32]. This file format provides a standardized method of storing the geometry, configuration and solution of a CFD simu- lation, including support for time varying flow solutions. Such standardization is very important for

CFD since it allows researchers and designers to share both the results of a simulation and the setup of the simulation in a single uniform format. Many organizations use or support the CGNS standard including Boeing, Onera, Fluent and others [33].

While any program which complies with certain specifications can write acceptable CGNS

files, there is a standard library which can be used to write CGNS files from either the Fortran or C programing languages. This library, known as the CGNS Mid-level Library (MLL), currently only supports the older method of CFD I/O. It supports two different styles of CGNS file, the first being an older database-like file format called ADF, and the second using a scientific data storage library called HDF5. The plan for the CGNS file format and library is to migrate completely from ADF to HDF5 [34]. The HDF5 library itself already supports the low-level parallel I/O features needed by the proposed parallel version of CGNS [9]. It does this through the use of the MPI-IO library. 15

Fig. 2.1: Storage topology of a modern computer cluster in two configurations. Configuration (a) shows the traditional CFD approach to I/O, where each node writes a separate file to disk. These files must be combined to obtain the full solution in a process which can be very time consuming. Configuration (b) shows the object of this work, being a CFD I/O method in which the partial solution from each node is written into the appropriate location within a single solution file for the whole problem. 16

The most recent revision of the MPI standard has added specifications for parallel I/O, and has been implemented by the ROMIO developers at Argonne National Laboratory [35].

The purpose of this work is to augment the CGNS library so as to allow for parallel writing of

HDF5 based files without the need for radical changes to the CFD codes themselves. The augmenta- tion is accomplished in the form of a new library which initially implements a sub-set consisting of the most used features from the full CGNS standard. The new library is called pCGNS, for parallel

CGNS. Unlike the current implementation of HDF5 support in the CGNS library, this new library uses the HDF5 API directly instead of accessing it through an emulation of the ADF API. This allows pCGNS to access the parallel I/O routines and abilities of the newer HDF5 library versions.

It also simplifies the library’s internal structure.

2.2 Problem Background The original CGNS library was conceived during the 1990’s to resolve some of the more im- portant difficulties in transferring NASA technologies to industrial development. Because NASA and industry used many different file formats which were incompatible with each other, data and software exchange between NASA and industry was limited. The CGNS file format and library were introduced to solve these problems, and have been widely adopted by both government agen- cies and industrial entities around the world [33]. This adoption is largely due to the ability of CGNS

files to express nearly any CFD simulation setup, and the work-flow simplification that ensues from its use [36].

Also in the 1990’s, a transition was occurring within the field of supercomputing. Large mono- lithic supercomputers such as those offered by Cray Computing were being replaced by lower cost computer clusters, often called Beowulf clusters when built out of “off the shelf” components and running a Linux distribution [10]. While this transition was driven by lower costs and higher pos- sible performance of the new cluster computers, they also presented new problems. A computer cluster requires a high speed network which allows the nodes of the cluster to communicate one with another. Such a network is often termed the cluster interconnect. Any network connection can be used in this capacity, but specialized solutions have been developed such as Myrinet [16–19] and 17

Infiniband [20–23].

Another difficulty which has arisen from the transition to clustered supercomputing is the im- plementation of high speed data storage. Since cluster computers often separate the computation nodes from the storage nodes, generally in the form of a commercial or custom network attached storage (NAS) system [37], the rate at which data can be transferred from memory on a compu- tation node to a storage node is therefore limited by the speed of the network connection between these nodes and the NAS. Additionally, the storage nodes and computation nodes are sometimes not connected by the high speed interconnect of the cluster, since many clusters may share the same storage. This is further limited by the fact that the file servers used in the early cluster computers only supported a single client writing at a time for each file. Thus multiple processes across the cluster could not write to the same file simultaneously. This caused many simulation codes to adopt the practice of each process writing to a separate file during parallel operation, with the intent that the files would be reconstructed after completion of the simulation [31].

As a first step to solving this problem, parallel file systems have been developed for use on computer clusters and network attached storage such as Lustre [38] and PVFS [39]. These file systems allow multiple clients to open and modify a single file simultaneously. This development allows for a parallel simulation running on a computer cluster to write all of its data to a single

file without the need for reconstruction. The need for such an operation is created by the way in which CFD codes distribute the computational work on a cluster, which is to partition the compu- tational domain of the problem and distribute the various physical regions in the domain among the processes running on the cluster. A partitioned example mesh is shown in Figure (1.2).

The problem which this effort works to address is the lack of a method for writing CGNS files in parallel. Since CGNS is a format specifically aimed at CFD, and many CFD codes run in parallel on computing clusters, the lack of support for parallel file systems in the main CGNS library is an obstacle to its goal of being a universal data exchange format for the field of CFD. By implementing the parallel extension to CGNS, this impediment to performance and flexibility for CFD codes will be eliminated by allowing them to take advantage of the I/O performance now available to cluster computers through the use of parallel file systems on the network attached storage. 18

2.3 Computational Fluid Dynamics General Notation System As stated previously, CGNS is a standardized method of notating the setup and results of a CFD case originally designed to facilitate collaboration between NASA and industry [40]. The CGNS standard can be separated into two components. The first component is the set of concepts that allow the case to be described in a fashion generic enough to suit many purposes, but still specific enough to include all relevant data. This is done using the standard interface data structures (SIDS). The second component of the CGNS standard governs how the SIDS are expressed in a file. This also includes a reference library capable of writing compliant files, called the Mid-level Library (MLL).

2.3.1 Standard Interface Data Structures The SIDS describe a CFD case by using a hierarchy of nodes, with each node defining some quality incident to all the nodes subordinate to it. By doing this, all the data is organized by scope automatically. Most important to the current work is the organization of the mesh description and the corresponding solution data.

Figure (2.2) shows the hierarchy of an example CGNS file. The problem domain is broken up into bases, and then zones, although most files only contain a single base and many only contain a single zone as well. The choice of how to split up the domain into bases and zones is not defined by the CGNS standard but rather left up to the user [41].

It can be seen that each zone contains its own set of coordinates and solutions. The zone node contains the number of points and cells which define its portion of the domain, so the relationship between the solution data and coordinate data in known implicitly.

2.3.2 Advanced Data Format The first file format used to express the CGNS standard was the advanced data format (ADF) which sees heavy use by NASA and the aerospace industry. ADF stores data in a hierarchical manner, making it a natural choice for the implementation of the CGNS standard. A mapping between the CGNS SIDS and ADF data structures is provided which allows any who wish to write a program which would write CGNS compliant files in the ADF container format. However, because of the complexity of this task most software supporting CGNS does so through the MLL. 19 Fig. 2.2: Diagram depicting thegroup hierarchical in nature the of hierarchy a controlling CGNS how the file. data Data is is interpreted. broken into groups and sub-groups, with the location and type of a 20

2.3.3 Hierarchical Data Format version 5 HDF5 also describes data in a hierarchical manner, and mappings have been defined for the use of HDF5 as the container format for CGNS [42]. Not only is it desirable to have multiple back ends to ensure the generality of the standard, but the newest versions of HDF5 support parallel I/O through MPI-IO, including collective I/O, which holds the potential for performance improvements and the ability to write larger data-sets directly [43].

2.3.4 Mid-level Library The Mid-level Library provides C and Fortran interfaces with which applications can read and write standards-compliant CGNS files without the need to use ADF or HDF5 directly. Unfortu- nately, since CGNS initially only supported the ADF container format, the HDF5 support in the

MLL is provided by an HDF5 to ADF compatibility layer which has been coded into the MLL. This compatibility layer ensured feature parity between the two back ends, but now prevents the parallel

I/O abilities present in HDF5 from being accessed. 21

Chapter 3

Literature Review

3.1 Previous Work Considerable research has been done on data access for scientific applications. The work has focused on data I/O performance and data management convenience. Three projects, MPI-IO,

HDF5 and parallel netCDF (PnetCDF) are closely related to this research.

MPI-IO is a parallel I/O interface specified in the MPI-2 standard. It is implemented and used on a wide range of platforms. The most popular implementation, ROMIO [44] is implemented portably on top of an abstract I/O device layer [35,45] that enables portability to new underlying I/O systems. One of the most important features in ROMIO is collective I/O operations, which adopt a two-phase I/O strategy [46–49] and improve the parallel I/O performance by significantly reducing the number of I/O requests that would otherwise result in many small, noncontiguous I/O requests.

However, MPI-IO reads and writes data in a raw format without providing any functionality to effectively manage the associated metadata, nor does it guarantee data portability, thereby making it inconvenient for scientists to organize, transfer, and share their application data.

HDF is a file format and software developed at NCSA for storing, retrieving, analyzing, vi- sualizing, and converting scientific data. The most popular versions of HDF are HDF4 [50] and

HDF5 [8]. Both versions store multidimensional arrays together with ancillary data in portable, self-describing file formats. HDF4 was designed with serial data access in mind, whereas HDF5 is a major revision in which its API is completely redesigned and now includes parallel I/O access.

The support for parallel data access in HDF5 is built on top of MPI-IO, which ensures its portabil- ity. This move undoubtedly inconvenienced users of HDF4, but it was a necessary step in providing parallel access semantics. HDF5 also adds several new features, such as a hierarchical file structure, that provide application programmers with a host of options for organizing how data is stored in

HDF5 files. Unfortunately, this high degree of flexibility can sometimes come at the cost of high 22 performance, as seen in previous studies [51, 52].

Parallel-NetCDF [53] is a library providing high-performance I/O while still maintaining file- format compatibility with Unidata’s NetCDF [54]. In the parallel implementation the serial netCDF interface was extended to facilitate parallel access. By building on top of MPI-IO, a number of interface advantages and performance optimizations were obtained. Preliminary test results show that the somewhat simpler netCDF file format coupled with a parallel API combine to provide a very high-performance solution to the problem of portable, structured data storage.

With these developments in the field of parallel I/O it is unsuprising that this is not the first time that a parallel extension to CGNS has been proposed. Hauser and Pakalapati previously worked on a possible parallel extension to the CGNS Mid-level library, but at the time the HDF5 library did not support collective I/O [43,55]. This forced their efforts to use independent I/O, which is anticipated to be slower than collective [9,56]. Their work also did not provide the same level of compatability with the main CGNS library as the current effort.

3.2 Parallel Input/Output With the transition to cluster-based supercomputing, the parallelization of all processes in- volved in the solution of scientific computational problems has become a major concern in the industry. This includes the need for high-speed parallel access to the file systems mounted by the nodes of a cluster, as well as a unified means of coordinating the work of various nodes in a cluster to solve a single problem.

3.2.1 Message Passing Interface Standard The switch from monolithic vector supercomputers to cluster computers created the need to coordinate the solution of a problem on more than one machine at a time. The problem simplifies into the need for a standardized means of transferring data back and forth from one node to another in the cluster. Traditional networking APIs can be used in this manner, and still drive the commu- nications of certain parallel programs such as DaVis from LaVision [57]. The needs of scientific parallel codes are very specific, however, and do not fit well with the more generic networking APIs available. This has led to the development of specialized means of communication between nodes 23 of a cluster [58]. One earlier development was the parallel virtual machine (PVM) library and run- time [59]. Among other things, it provides a message passing interface which allows one process running on the machine to easily send information to another in a standardized way.

The concept of message passing proved to be an effective paradigm in which to program cluster computers. This success led to the development of a standard message passing interface (MPI) de-

fined by the MPI Forum [60] and implemented by various vendors and open source groups [61,62].

Since all implementations of MPI adhere to the standard, programs can be written against the MPI standard and then compiled and run on any computer which has an MPI implementation. Because of this open nature and the effectiveness of the message passing concept for parallel programing,

MPI has risen to be the primary means by which programs are written to run on a computer cluster, or even on symmetric multiprocessor machines such as modern multi-core computers.

The MPI standard defines a library which provides the MPI API. This API supplies the pro- grammer with numerous communications subroutines which allow the processes running on a su- percomputer to pass information back and forth. Some of the routines are quite simple such as

MPI_Send, which sends the data contained in a memory buffer from one process to another. The re- ceiving process must call a matching MPI_Recv, which receives the sent data into a memory buffer.

More complex communications are also supported, such as operations which find the average, sum, or max of a set of memory buffers on various processes across the machine. Using these routines, the program can ensure that data passes through the entire machine as if it were one computer, despite the fact that it may be composed of many discrete computers scattered across large geographical distances.

MPI provides a run-time component as well as the main library. This run-time handles the spawning of all the processes for an MPI job and provides the library with critical execution infor- mation such as how many processes are running in the job and the rank of a process relative to the group. The run-time component also implements the fundamental code which actually executes the message passing. Since many cluster computers are composed of nodes which themselves are sym- metric multi-processor machines [63], the best way to send data from one process to another greatly depends on the physical path from the processor executing the send routine and the processor exe- 24 cuting the receive routine. Most implementations of MPI are intelligent about how a message is sent from one process to another. In the case of two processes which are executed on the same physical node, the message is generally passed through a shared memory interface [61]. This can be faster than the interconnect of the cluster, depending highly on the code being executed, and allows MPI to be used as the parallel programing interface for multi-processor machines as well as clusters.

3.2.2 Network Attached Storage A network attached storage (NAS) unit is a computer system attached to a network for the purpose of providing storage to other clients of the network [37]. While a NAS can be built out of commodity hardware just like the nodes in a cluster, there are commercial solutions which can be purchased pre-built. Being a complete computer system, a NAS contains a CPU, memory, and other features common to servers. What make a NAS capable of fulfilling its purpose is the operation of

file server daemons running on the machine. These allow clients on the network to access the files stored on the NAS’s internal hard drives.

3.2.3 Message Passing Interface-I/O With the rise of parallel file systems on network attached storage or even integrated into the cluster itself, the need arose for a standardized way to access these file systems, so that simulations could be written which would run on any system with a simple recompile. To this end, the MPI standard was extended in version 2 to include file access routines which allow for parallel storage

I/O. These routines are designed to be generic enough to fit any of the file systems that can be used as a back end, including the old standard serial routines if no parallel system is available. In general, implementations of MPI need to write interfaces for each parallel file system that they will support, and then the system administrator will enable the appropriate back ends when MPI is compiled for a new machine [35, 60].

3.2.4 Parallel File Systems NAS systems are commonly used on large networks to provide a central location for storing shared data. As such, the concept of a using a NAS is well adapted to cluster computing since all 25 the nodes working on a problem need access to the same sets of files. Generally a NAS provides file servers such as Server Message Block (SMB) or Network File System (NFS) to allow clients access to the stored data. These protocols do not allow for parallel access to files, however. To resolve this problem, several parallel file systems have been developed which can either run on a NAS or set of

NAS systems working together or can be run directly on the nodes of a cluster equipped with hard disks in each node and special daemons running on the nodes to provide access to the data.

3.2.4.1 Contiguous vs Dis-contiguous I/O Most common file systems, including parallel file systems, represent a file as a one dimensional array of bytes. Assuming that the data stored on a hard drive is not fragmented, this model is actually very close to the true nature of the data’s organization on the drive. A data-set which occupies every byte in a contiguous region of the file is called contiguous data. When data is contiguous, it can be read and written very efficiently by modern hardware, which takes advantage of memory-buffered reads and writes to speed up these operations.

Dis-contiguous data-sets are scattered throughout a region of a file. Even if the organization of the data is orderly, dis-contiguous data cannot be written nearly as efficiently by storage systems.

In serial computing, the difference between contiguous and dis-contiguous data is not a serious problem since ensuring that a data-set is contiguous in storage is often as simple as ensuring that it is contiguous in memory. In parallel computing, when a domain has been split between multiple processes, data which is contiguous in memory is still dis-contiguous in storage.

3.2.4.2 Collective vs Independent I/O The semantics of MPI-IO define two different types of I/O, those being collective and inde- pendent. Independent I/O is characterized by each process in an MPI communicator accessing the

file independently from the other processes in the communicator. This reduces the amount of extra communication required to manage the I/O, since each process accesses the file without regard for the activities of the other processes.

Collective I/O requires that every process in the communicator participate in the I/O operation.

While this requires greater communication between the processes to synchronize the operation, it 26 allows MPI to use the high speed interconnect and user defined data descriptions to aggregate the data as it is sent to the data storage system. This can dramatically increase the total bandwidth of the operation since the data is rearranged into a more contiguous form [43, 49].

3.2.4.3 PVFS Currently being developed by at the Parallel Architecture Research Laboratory at Clemson

University, the Mathematics and Computer Science Division at Argonne National Laboratory, and the Ohio Supercomputer Center, PVFS is a parallel file system designed to be as close to standard

Unix file systems as possible while still providing the performance and parallel access capabilities as other parallel file systems [39, 64, 65].

3.2.4.4 Lustre Originally developed at Carnegie Mellon University, the Lustre file system was designed with

Linux-based scientific computing clusters in mind from the beginning. A feature of Lustre common to all object-based distributed file systems is the separation of the file meta data from the actual

file data. The meta data of a file includes its name, path, creation time, and other properties, but is generally only needed when opening a file. This way the meta data servers can direct the clients to other servers which actually store the file’s data. Luster is highly scalable and has seen use on the Blue Gene installation at Lawrence Livermore National Laboratory, one of the most powerful computers in the world [38, 66]. Additionally, it has been demonstrated that the Lustre file system can accommodate storage systems spread across very large geographical distances [38]. Lustre has been shown to be more performant than PVFS [67].

3.2.4.5 GPFS Introduced in IBM’s own custom Unix variant AIX, the General Parallel File system is now supported on the AIX, Linux and Windows platforms. Like Lustre, GPFS splits the meta data from the file content, allowing for higher performance. The file system also provides complete POSIX compliance [68]. 27

3.2.4.6 Panasas The Panasas storage products are good examples of commercially available cluster storage solutions. Providing both the needed software and hardware components, these systems employ a proprietary file system tightly integrated with the storage nodes’ operating systems. The clients interface with the storage through kernel modules and libraries, which are supported by some im- plementations of MPI-IO [69].

3.2.5 Hierarchical Data Format version 5 With the advent of parallel I/O and other changes to the needs of a standardized data format, the HDF file format released a new revision called HDF5. The later versions of the HDF5 library, especially 1.8.x, provide support for parallel I/O by making use of MPI-IO as a back end. This allows HDF5 to support any parallel file system which the MPI implementation can handle. The

HDF5 library also simplifies the use of parallel I/O and allows the programmer to focus on the data being described and let HDF5 determine the most efficient way to access the parallel storage [43]. 28

Chapter 4

Parallel Implementation for the CFD General Notation System

The reimplementation of CGNS, directly using HDF5 as the container file format with the intent of parallel access to files is the purpose of the pCGNS library. Currently it is intended as a companion library to the Mid-level Library, since pCGNS only supports the most important and

I/O-bound of the operations needed to write an entire CGNS file. The library is written in C and complies to the C99 standard. This presents many syntactic advantages over traditional C which greatly simplify the process of writing the library [70]. At present there is no Fortran interface to the library, but the addition of such an interface would not present much difficulty, especially in light of the automatically generated interface present in the main CGNS library.

4.1 Design Figure (4.1) shows the software stack used to implement the functionality of the library. The

Figure shows the layers of software libraries, with calls only made from one adjacent layer to an- other. Thus, pCGNS is entirely dependent on HDF5 and MPI, and completely ignorant of any particular file system. This allows the lower layers of the system to be swapped out from one system to the next without the need to redesign pCGNS.

Internally, the library is composed of two primary source files. The first, pcgns_util.c contains utility functions which are used repeatedly by the main library. These functions are not intended to be called by a program but rather are to be called from other parts of the library only. The utility functions implemented in pcgns_util.c are either used to maintain the internal state of the library, which uses several global variables to keep track of open files and details about those files, or the functions exist to streamline certain commonly executed operations using HDF5. These operations include the creation of SIDS nodes within the file, and the reading and writing of simple data types.

The presence of global variables in the library precludes the possibility of calling the library from a 29

Fig. 4.1: Software stack used by the pCGNS library. Layers are ordered from most abstract at the top to least abstract at the bottom. Each layer relies on lower layers to implement functionality. This allows higher layers to be independent of the implementation of lower layers. Thus, as long as the MPI implementation used can take advantage of the file system on a machine, HDF5 and pCGNS will also be able to use that file system. multi-threaded program such as one created using p-threads or OpenMP, as this would corrupt the internal state of the library. Resolution of this issue would require a significant break from the MLL

API, which would discourage adoption of this library by existing users of CGNS.

The second primary source file, pcgnslib.c implements the public interface to the library, or in other terms, this file actually implements the library’s API. These functions are intended to be called by external programs to write CGNS files. Most of these functions rely on either HDF5 or pcgns_util.c for functionality. Calls to MPI are present in both portions of the library.

The implementation details of the library, including the full annotated source code and structure diagrams are available in Appendix (A) of this document.

4.2 Features Table (4.1) shows the functions defined in pcgnslib.h and which constitute the pCGNS API as it currently exists. The functions chosen to be implemented in the initial version of the library were selected on the basis of two criteria. The first was the need for the function to provide the most basic functionality of the library. The second criteria was the possible benefit from parallelization of the function. The result of using these priorities is that the pCGNS library can write the basic components of a CGNS file needed for the standard CGNS tools to function, but still provides 30

General File Operations cgp_open Open a new file in parallel cgp_base_read Read the details of a base in the file cgp_base_write Write a new base to the file cgp_nbases Return the number of bases in the file cgp_zone_read Read the details of a zone in the base cgp_zone_type Read the type of a zone in the base cgp_zone_write Write a zone to the base cgp_nzones Return the number of zones in the base Coordinate Data Operations cgp_coord_write Create the node and empty array to store coordinate data cgp_coord_write_data Write coordinate data to the zone in parallel Unstructured Grid Connectivity Operations cgp_section_write Create the nodes and empty array to store grid connectivity for an unstructured mesh cgp_section_write_data Write the grid connectivity to the zone in parallel for an unstructured mesh Solution Data Operations cgp_sol_write Create the node and empty array to store solution data cgp_sol_write_data Write solution data to the zone in parallel General Array Operations cgp_array_write Create the node and empty array to store general array data cgp_array_write_data Write general array data to the zone in parallel Queued I/O Operations queue_slice_write Queue a write operation to be executed later queue_flush Execute queued write operations

Table 4.1: API of the pCGNS library as implemented in pcgnslib.c and declared in pcgnslib.h. Software using the library to access CGNS files in parallel must do so using the routines listed here. routines for parallel file access on the largest of the data sets that need to be written to the file. Any extra data or properties that may need to be written to the file can still be added using the MLL after the file has been closed with pCGNS. 31

Chapter 5

Results

After implementation of the pCGNS library, several benchmarks of the library were run on

Wasatch, the newest computer cluster at Utah State University. The benchmarks run were selected to explore the performance of the library under conditions which would be typical of the file ac- cess patterns generated by CFD codes. In addition to the benchmarks of the pCGNS library, an- other benchmark program has also been run, called IOR, which provides a measurement of the dis-contiguous parallel I/O performance of a cluster when using both the MPI-IO and HDF5 inter- faces. This second set of benchmarks provides an upper limit against which the performance of pCGNS can be compared.

5.1 Hardware Setup Figure (5.1) shows the setup of the HPC computer systems at USU. The older cluster, Uinta, on which much of the developmental work and testing was done, is shown with its 1 gigabit connection to the central Procurve switch. An odd feature of Uinta is the non-uniform access to the storage systems. This not only complicates the process of benchmarking, since which nodes the job runs on affects the results of the job. The newer cluster, Wasatch, has a connection from each node directly to the Procurve switch. This connection is actually two 1 gigabit Ethernet connections bonded together to form a single connection. Not only does this put every Wasatch node at an equal position with regard to storage, but it also provides a high-speed direct link to the switch.

The storage used in these benchmarks is a parallel storage solution provided by Panasas, mounted at /panfs. All of the files used by the program during the benchmarking process are stored on /panfs, which is composed of four shelves of storage servers. Each shelf is connected to the

Procurve switch using a 10 gigabit Ethernet connection. 32

Fig. 5.1: Topology of the computing clusters and network attached storage at Utah State University. A single high speed switch connects all the storage to the clusters, and serves as the I/O backbone for Wasatch. The network connections between the Wasatch nodes and the main switch are composed of pairs of 1 gigabit Ethernet connections bonded to form a 2 gigabit connection for each node. The connection between the Panasas storage (/panfs) and the switch is composed of four 10 gigabit Ethernet connections. 33

5.2 Benchmark Selection The selection of benchmarks to be run using both the IOR benchmarking tool and the pCGNS library with its related test programs was accomplished using several criteria and goals as outlined below.

5.2.1 IOR Two sets of benchmarks were selected to be run using the IOR benchmarking tool. The first set is intended to characterize both independent and collective read and write I/O performance on

Wasatch. These benchmarks were selected to give an idea of the complete I/O capabilities of the new machine.

A second and smaller benchmark was selected to be run in collective write mode with only the

HDF5 API, as a direct comparison with pCGNS.

5.2.2 pCGNS Five categories of benchmark were selected to be run on the pCGNS library. These categories are shown in Figure (5.2). Each scenario of data distribution models a general method of splitting up data among the processes of a parallel program. The scenarios will be hereafter referred to by the designation each scenario bears in the figure. In all of the benchmarks, the file size is kept constant, since during practical use of the library the file size will generally not be a function of the number of nodes used for a simulation, but rather a function of the simulation itself.

The first of these, the type-a benchmark, explores the performance of the library when multiple zones are stored on a single node. This is the only benchmark in which multiple processes are run on a single node. Normally this is avoided since having multiple processes on a node forces those processes to share a single network connection, thus affecting the measurement of the parallel performance of the library. In the type-a benchmark, however, this limitation is desired since it allows the single process with multiple zones to have access to the same storage bandwidth as the aggregate bandwidth of multiple processes on the node.

The type-b benchmarks measure the performance of pCGNS in a configuration very close to the benchmarks run by IOR. In this scenario, each node runs a single process, and each process only 34

Fig. 5.2: Data distribution scenarios for benchmarking. Part (a) shows two zones on one node. Part (b) shows one zone per node on four nodes. Part (c) shows four zones on two nodes. Part (d) shows two zones on four nodes. Part (e) shows four zones on four nodes with zones spread across multiple nodes.

writes a single zone. Because each zone has its own data array in the file, and each zone is written by a single process, each process writes to a single data array in the file.

The type-c benchmarks test the ability of the library to write multiple zones from each node in parallel. Because the file size is kept constant, more zones per node result in smaller data arrays written to the file in each I/O operation. Since this is expected to affect performance, multiple cases are run with different numbers of zones per node.

Type-d benchmarks test the ability of the library to write a single zone from multiple nodes.

This causes multiple processes to access a single data array simultaneously, which is quite different from multiple processes accessing multiple data array. In this scenario, each node only writes to a single zone, but each zone is written to by multiple processes.

The last scenario, type-e, tests the most general access pattern, in which multiple processes access each zone and multiple zones are accessed by each process. This access pattern would result from the zone definitions and parallel partitioning processes being completely independent one from another.

5.3 Benchmark Results 35

Fig. 5.3: Performance of the MPI-IO library on Wasatch using the IOR benchmark. IRead stands for independent read, CWrite for collective write, and so on. The hints recommended by Panasas are active to ensure best performance.

5.3.1 IOR Figures (5.3) through (5.4) detail the performance of USU’s newest cluster Wasatch with re- gard to accessing a single file from multiple processes using several different APIs via the IOR benchmark software.

Figure (5.3) shows the IOR benchmark results when using the MPI-IO API to read and write

files in both collective and independent modes. It can be seen that there is not much difference between the collective and independent I/O when using the MPI-IO interface. The fact that the four-processor job enjoyed much faster write speeds is most certainly a result of buffered I/O on the part of the file system. This is accomplished by the kernel module used to interface with the storage keeping the data in memory while it waits finish writing, but allowing the benchmark program to continue, thus giving the perception of much faster I/O. It can also be seen that read speeds are significantly faster than write times, which is normal for these kinds of I/O operations.

The plot presented in Figure (5.4) displays the results of the same benchmark but uses the 36

Fig. 5.4: Performance of the HDF5 library on Wasatch using the IOR benchmark. IRead stands for independent read, CWrite for collective write, and so on. The hints recommended by Panasas are active to ensure best performance.

HDF5 interface. The most important difference between the MPI-IO and HDF5 results is the markedly improved performance provided by the HDF5 library, in both collective and independent

I/O. The HDF5 library has received considerable work to provide high performance by optimizing

I/O based on the structure of the HDF5 file to which the data is being written. Also of note is the larger difference between the collective and independent I/O modes. It would appear that the independent routines are better-optimized than the collective ones.

The last IOR benchmark was run to get a candidate for direct comparison with the performance of the pCGNS library. Since pCGNS currently only operates in collective write mode, that was the only mode tested in this benchmark. The data size was chosen to be 256 MB per process, since the pCGNS tests are run data sets of approximately this size. Figure (5.5) shows the results, which demonstrate a sharp drop in performance when the library switches into true parallel operation. 37

Fig. 5.5: Performance of HDF5 on Wasatch using collective writes only, on a 256 MB data set. The hints recommended by Panasas are inactive to give results which are comparable to the pCGNS benchmarks which were not run with any hints activated. 38

5.3.2 pCGNS All benchmarks were run five times, and the line shown in each plot is the average of the runs.

Error bars are also shown, based on the standard deviation of the bandwidths. The speedup plots are based solely on the averaged data, and are computed at the bandwidth for a particular number of processes or nodes divided by the bandwidth for a single process or node.

The results of the type-a benchmarks are summarized in Figure (5.6). All of the processes in this case are on the same node, as the purpose of this benchmark is to gage the efficiency of the library itself, with no real parallel I/O advantages gained by using more nodes. While the various sizes of file did affect the overall bandwidth of the writes, with larger files the bandwidth is relatively constant between 65 and 70 MB/s. The smaller files write faster because of the buffering inherent in the file system. The results are actually promising, since the speedup of all sizes of file are very close to one. This means that the bandwidth is being limited by the network connection between the storage and the node doing the writing, not by some inefficiency in the library.

Figure (5.7) shows the results of the type-b benchmark, which investigates the parallel per- formance of the library in optimal conditions. In this case, each node writes a single zone to the

file, with the size of the file being held constant. The plot demonstrates that the performance of the library is generally better with larger data sets. This is to be expected, since the larger data sets cause the time spent on inter-process coordination to be smaller relative to the total time of program execution. Also unsurprisingly, the performance worsens with more nodes when transferring small amounts of data, since each node is writing less and less data, which is less efficient.

From the speedup plot, it is seen that near-optimal performance is maintained using up to four nodes on the largest case. This is perhaps a result of the four shelves which compose the Panasas

NAS. Yet again, the lower performance with smaller file sizes is expected. The best performance is four times faster than the equivalent operation using serial I/O, thus validating the possibility for performance improvement using parallel I/O.

The type-c benchmarks include more data than the others, since they were run with multiple numbers of zones per node in an attempt to measure the performance resulting from aggressively breaking up the computational domain. The results of these runs are summarized in Figures (5.8) 39

(a) Bandwidth

(b) Speedup

Fig. 5.6: Results of pCGNS benchmark (a). The larger file sizes experience approximately the same bandwidth, while the smaller sizes enjoy the benefits of buffered file I/O. The speedup of all sizes of file is approximately one, which demonstrates good efficiency in the pCGNS library, and all layers beneath it. 40

(a) Bandwidth

(b) Speedup

Fig. 5.7: Results of pCGNS benchmark (b). Large file sizes perform better then smaller ones. The largest file size performs four times better in parallel I/O than the serial I/O, but performance nearly levels off after four nodes, possibly due to the topology of the storage at USU. 41 through (5.10). When compared to the performance of the library in the type-b benchmarks, the largest file size with the fewest zones compares favorably, although it is slower. The speedup peaks at just under three times the performance of the serial code, and still levels off around four nodes. As the number of zones increases, however, the performance rapidly deteriorates. This is largely due to the phenomenon of smaller data set sizes having a detrimental affect on performance as observed with the type-b results. Thus, it is advantageous to use as few zones as possible when organizing the domain of the CFD case so that the I/O requests can write more data at a time and provide better performance. In the most extreme cases, the parallel version actually performed worse than the serial version.

Figure (5.11) shows the results of the type-d benchmarks of the pCGNS library, which test the library’s ability to provide multiple processes with access to the same data array in a file simulta- neously. Unlike previous benchmarks, the rapid drop in bandwidth when switching from one to two nodes is not because of an I/O buffer advantage enjoyed by the serial job, but rather the library switching from a single process accessing the data array to multiple processes accessing the data array. While the HDF5 library which powers the parallel I/O capabilities of pCGNS supports this type of I/O, it obviously does not perform very well. This is likely due to the added complexity of dealing with possible overlaps in the data writes, since the library cannot assume that all ranges of data written will be independent from one another. As usual the bandwidth observed with larger data sizes is greater, but none of the runs achieved a speedup greater than one. These performance limitations do not invalidate the usefulness of pCGNS when operating in this mode. Even if the

I/O performance may be worse than the serial case when processes share zones, the total possible problem size is still greater than allowed by serial I/O, since that requires that the entire problem fit into the RAM of a single node.

Like the type-d benchmarks, the type-e benchmarks also demonstrate poor performance, as shown by Figure (5.12). Again, these results can be explained by multiple processes accessing a single array in the file simultaneously. In type-e, each process also writes to multiple zones, so two features are multiply associated. This is the most general case explored by the benchmarks, and demonstrates the complexity present in CFD data when the physically significant zones in the 42

(a) Bandwidth

(b) Speedup

Fig. 5.8: Results of pCGNS benchmark (c). Two zones are written by each node while keeping the file size constant, causing less and less data to be written with each additional node. The perfor- mance is still acceptable, but does level off after four nodes. 43

(a) Bandwidth

(b) Speedup

Fig. 5.9: Results of pCGNS benchmark (c). Four zones are written by each node while keeping the file size constant, causing less and less data to be written with each additional node. The perfor- mance is no longer an improvement over the serial case with the smallest data size, although larger files still perform acceptably. 44

(a) Bandwidth

(b) Speedup

Fig. 5.10: Results of pCGNS benchmark (c). Eight zones are written by each node while keeping the file size constant, causing less and less data to be written with each additional node. The per- formance is no longer an improvement over the serial case with all but the largest data size, which barely performs better than the serial version. 45

(a) Bandwidth

(b) Speedup

Fig. 5.11: Results of pCGNS benchmark (d). Multiple nodes share each zone, requiring HDF5 to sort out any possible data overlaps. This causes a severe slow down of the I/O operations. None of the tested file sizes outperformed the serial case. 46 domain do not match up well with the computationally significant partitions. Also like the type-d results, these results are not totally negative, however, since the performance is not so bad that the method could not be considered as a means of easily writing very large case files which cannot be stored in the memory of any node on the cluster.

In general the results of benchmarking the pCGNS library are not unexpected. Larger file sizes enjoy better utilization of the parallel I/O hardware available, since more time is spent transferring data than coordinating the I/O operations between the processes. This effect is seen in nearly every benchmark run on the library, but is not truly problematic since the purpose of using a computing cluster when solving a CFD case is generally to handle larger domains than otherwise possible.

Additionally, for performance reasons, it is critical that each process only access a single zone at a time. While this will reduce the total amount of data that can be transferred in each operation, the performance gains expected from this restriction far outweigh any reduction caused by smaller data sizes. In light of the poor performance when accessing a single zone from multiple processes, the queuing operations of the library should be altered in future versions to prevent this I/O mode from being used.

When considered against the small collective write benchmark done using IOR with the HDF5 interface, most of the pCGNS runs compare favorably. Even on the type-d and type-e benchmarks, the largest file sizes performed nearly as well as IOR in those same cases. This suggests that the pCGNS library successfully assumes the performance of the HDF5 library under these operating conditions. 47

(a) Bandwidth

(b) Speedup

Fig. 5.12: Results of pCGNS benchmark (e). The most general of the benchmarks, type-e demon- strates the performance when multiple zones are written by multiple processes. This situation results from parallel partitioning independent of the zone arrangement of the domain. 48

Chapter 6

Summary and Conclusion

The purpose of this work was to implement a new library which augments the CGNS Mid- level Library with parallel I/O write capabilities using the new features added in recent versions of the HDF5 library and benchmark the performance of the resulting software. This was done in hopes that the parallel extension would provide improved performance when compared with serial execution of the same code, but parallel I/O provides more benefits than just improved performance.

Having parallel I/O capabilities simplifies the process of writing solutions to disk when running a simulation on a cluster computer, since there is no longer the need to manually synchronize the writes from each process or to write a single file from each process and recombine them after the simulation completes.

The decision made early in development to implement the new library directly using HDF5 instead of the ADF to HDF5 compatibility layer in the MLL not only made coding pCGNS much easier, but most likely made it possible at all. Fortunately, since CGNS supports both the ADF and

HDF5 back ends in the main library and all the tools, there are no compatibility problems between

files generated with pCGNS and the MLL.

By benchmarking the library developed for this work the performance of the library on the clusters at Utah State University using the Panasas storage system was measured and these results were compared against benchmarks run on the same hardware using only the HDF5 library through the IOR benchmarking tool. When run using multiple processors on a single node, the pCGNS library does experience variation in performance, but overall speedup of the code as processors are added is very near to one. Since the aggregate I/O bandwidth of all the processes on a node is limited by the network connection of that node, a speedup of one indicates that the library is operating efficiently. If the speedup were less than one, this would be due to processes coordination overhead incurred by running multiple jobs. 49

When running benchmarks in which multiple processes across multiple nodes wrote to separate array, the speedup of the code is nearly ideally linear for up to four nodes running a single process each. The maximum speedup was attained when using all nodes and performed four times better than the serial version. The limitation of the linear speedup to four nodes is likely a result of having only four shelves of Panasas storage.

The only cases for which the parallel version of the library did not outperform the serial version were those in which each array was accessed by multiple processes across the cluster. These cases experienced a speedup less than one, indicating that the code actually ran slower than the single process case. The results from these case do still compare favorably with the IOR benchmarks when executing the same operations, so the inefficiencies must be in the HDF5 library. In these cases, the parallel version of the code can still write more data from memory to disk than the serial version, since the domain is scattered across the nodes of the cluster.

It is hoped that this version of the pCGNS library will serve as the alpha release of the library for general adoption by the CGNS and CFD community. It will be presented at SC09 (Supercom- puting ’09) as Utah State University’s Storage Challenge submission. USU was selected as one of four finalists in this challenge, and faces competition from such organizations as NVIDIA, Mi- crosoft, and IBM, as well as first tier universities. After presentation at the conference, the source code to the pCGNS library will be posted as a project on Sourceforge under an open source license so that those concerned may continue its development, as led by the CGNS steering committee.

This decision is in line with the licensing and use of the rest of the CGNS software. 50 51

Bibliography

[1] Geuzaine, C. and Remacle, J.-F., “Gmsh: A Three-dimensional Finite Element Mesh Gen-

erator with Built-in Pre- and Post-processing Facilities,” Gmsh Home Page http://www.

geuz.org/gmsh/, accessed September 2009.

[2] Gullberg, J., Mathematics from the Birth of Numbers, W. W. Norton & Comparny, New York,

1997.

[3] Ceruzzi, P., A History of Modern Computing, The MIT Press, Cambridge, 2003.

[4] Chen, S., “Computer Systems Architecture,” New York Times, 1998.

[5] Deitel, H. M. and Deitel, P. J., C++ How to Program, Prentice Hall, Upper Saddle River,

1998.

[6] Lewine, D., POSIX Programmer’s Guide, O’Reilly, Sebastopol, 1992.

[7] “Debian GNU/Linux device driver check page,” http://kmuto.jp/debian/hcl/, ac-

cessed August 2009.

[8] “HDF5 Home Page,” The HDF Group. http://www.hdfgroup.org/products/

hdf5/, accessed August 2009.

[9] “What is HDF5?” http://www.hdfgroup.org/HDF5/whatishdf5.html, ac-

cessed May 2009.

[10] Sterling, T., Beowulf Cluster Computing with Linux, The MIT Press, Cambridge, 2001.

[11] Cray, S., “Supercomputing ’88, Keynote Address,” 1988.

[12] Hauser, T., Mattox, T., LeBeau, R., Dietz, H., and Huang, P., “High-cost CFD on a low-cost

cluster,” Proceedings of the 2000 ACM/IEEE Conference on Supercomputing (CDROM), IEEE

Computer Society Washington, DC, USA, 2000. 52

[13] Hwang, K. and Xu, Z., Scalable Parallel Computing, McGraw-Hill, Columbus, 1998.

[14] an Mey, D. and Schmidt, S., “From a Vector Computer to an SMP-Cluster-Hybrid Paralleliza-

tion of the CFD Code PANTA,” Proceedings on the 2nd European Workshop on OpenMP

(EWOMP2000), 2000.

[15] Hauser, T., LeBeau, R., Mattox, T., Huang, P., and Dietz, H., “A Comparative Study of the

Performance of a CFD Program Across Different Linux Cluster Architectures,” Tech. rep.,

Linux Clusters Institute, 2002.

[16] Bhoedjang, R., Ruhl, T., and Bal, H., “Design Issues for User-level Network Interface Proto-

cols on Myrinet,” IEEE Computer, Vol. 31, No. 11, 1998.

[17] Bhoedjang, R., Ruhl, T., and Bal, H., “Efficient Multicast on Myrinet Using Link-level Flow

Control,” Int. Conf. on Parallel Processing, Citeseer, 1998.

[18] Boden, N., Cohen, D., Felderman, R., Kulawik, A., Seitz, C., Seizovic, J., Su, W., et al.,

“Myrinet: A Gigabit-per-second Local Area Network,” IEEE Micro, Vol. 15, No. 1, 1995.

[19] Chun, B., Mainwaring, A., and Culler, D., “Virtual Network Transport Protocols for Myrinet,”

IEEE Micro, Vol. 18, No. 1, 1998.

[20] Pelissier, J., “Providing Quality of Service over Infiniband Architecture Fabrics,” Proceedings

of the 8th Symposium on Hot Interconnects, Intel Corporation, 2000.

[21] Pfister, G., “An Introduction to the InfiniBand Architecture,” High Performance Mass Storage

and Parallel I/O, 2001.

[22] Pfister, G., “Aspects of the InfiniBand Architecture,” Proceedings of the 2001 IEEE Interna-

tional Conference on Cluster Computing, 2002.

[23] Shipman, G., Woodall, T., Graham, R., Maccabe, A., Bridges, P., Los Alamos, N., and Al-

buquerque, N., “Infiniband scalability in Open MPI,” International Parallel and Distributed

Processing Symposium (IPDPS’06), Citeseer, 2006. 53

[24] Gray, J., “Distributed Computing Economics,” Queue, Vol. 6, No. 3, 2008.

[25] Mase, G. T. and Mase, G. E., Continuum Mechanics for Engineers, CRC Press, Boca Raton,

1999.

[26] Chung, T. J., Computational Fluid Dynamics, Cambridge University Press, New York, 2002.

[27] Chapra, S. C. and Canale, R. P., Numerical Methods for Engineers, McGraw-Hill, Columbus,

2006.

[28] Tannehill, J. C., Anderson, D. A., and Pletcher, R. H., Computational Fluid Mechanics and

Heat Transfer, Taylor & Francis, London, 1997.

[29] Versteeg, H. K. and Malalasekera, W., An Introduction to Computational Fluid Dynamics: The

Finite Volume Method, Pearson Education Limited, Upper Saddle River, 2007.

[30] Wendler, J. and Schintke, F., “Executing and Observing CFD Applications on the Grid,” Future

Generation Computer Systems, Vol. 21, No. 1, 2005.

[31] “OpenFOAM User’s Guide,” OpenFOAM User’s Guide http://www.opencfd.co.uk/

openfoam/doc/user.html, accessed August 2009.

[32] “CFD General Notation System What is CGNS?” http://cgns.sourceforge.net/

WhatIsCGNS.html, accessed September 2009.

[33] Poirier, D. M. A., Bush, R. H., Cosner, R. R., Rumsey, C. L., and McCarthy, D. R., “Advances

in the CGNS Database Standard for Aerodynamics and CFD,” AIAA, 2000.

[34] “CFD General Notation System Switch to HDF5,” http://cgns.sourceforge.net/

news.html, accessed September 2009.

[35] Thakur, R., Gropp, W., and Lusk, E., “On Implementing MPI-IO Portably and with High

Performance,” Proceedings of the Sixth Workshop on I/O in Parallel and Distributed Systems,

Argonne National Laboratoty, ACM, Argonne, IL 60439, 1999. 54

[36] Poinot, M., Rumsey, C. L., and Mani, M., “Impact of CGNS on CFD Workflow,” AIAA, June

2004.

[37] Gibson, G. and Van Meter, R., “Network Attached Storage Architecture,” Communications of

the ACM, Vol. 43, No. 11, 2000.

[38] “Lustre File System Home Page,” http://wiki.lustre.org/index.php/Main_

Page, accessed August 2009.

[39] Carns, P., Ligon III, W., Ross, R., and Thakur, R., “PVFS: A Parallel File System for Linux

Clusters,” Third Extreme Linux Workshop, Atlanta, October 2000.

[40] Legensky, S. M., Edwards, D. E., Bush, R. H., Poirier, D. M. A., Rumsey, C. L., Cosner, R. R.,

and Towne, C. E., “CFD General Notation System (CGNS): Status and Future Directions,”

AIAA, 2002.

[41] CGNS, CFD General Notation System: Standard Interface Data Structures, 2nd ed., 2009.

[42] Wedan, B., “CFD General Notation System,” Slides, Feburary 2005.

[43] Yang, M., “Collective IO Support Inside HDF5,” Slides, University of Illinois at Urbana-

Champaign: National Center for Supercomputing Applications.

[44] Thakur, R., Ross, R., Lusk, E., and Gropp, W., “Users Guide for ROMIO: A High-

Performance, Portable MPI-IO Implementation,” Tech. Rep. Technical Memorandum No. 234,

Mathematics and Computer Science Division, Argonne National Laboratory, Revised January

2002.

[45] Thakur, R., Gropp, W., and Lusk, E., “An Abstract-Device Interface for Implementing Portable

Parallel-I/O Interfaces(ADIO),” Proceedings of the 6th Symposium on the Frontiers of Mas-

sively Parallel Computation, October 1996.

[46] Rosario, J., Bordawekar, R., , and Choudhary, A., “Improved Parallel I/O via a Two-Phase

Run-time Access Strategy,” IPPS ’93 Parallel I/O Workshop, February 1993. 55

[47] Thakur, R., Bordawekar, R., Choudhary, A., and Ponnusamy, R., “PASSION Runtime Library

for Parallel I/O,” Scalable Parallel Libraries Conference, October 1994.

[48] Thakur, R. and Choudhary, A., “An Extended Two-Phase Method for Accessing Sections of

Out-of-Core Arrays,” Scientific Programming, Vol. 5, No. 4, 1996.

[49] Thakur, R., Gropp, W., and Lusk, E., “Data Sieving and Collective I/O in ROMIO,” Proceeding

of the 7th Symposium on the Frontiers of Massively Parallel Computation, February 1999.

[50] “HDF4 Home Page,” The HDF Group. http://www.hdfgroup.org/products/

hdf5/, accessed August 2009.

[51] Li, J., Liao, W., Choudhary, A., and Taylor, V., “I/O Analysis and Optimization for an AMR

Cosmology Application,” Proceedings of IEEE Cluster 2002, Chicago, IL, September 2002.

[52] Ross, R., Nurmi, D., Cheng, A., and Zingale, M., “A Case Study in Application I/O on Linux

Clusters,” Proceedings of SC2001, Denver, CO, November 2001.

[53] Li, J., keng Liao, W., Choudhary, A., Ross, R., Thakur, R., Gropp, W., Latham, R., and

Siegel, A., “Parallel netCDF: A High-Performance Scientific I/O Interface,” Proceedings of

the Supercomputering 2003 Conference, Phoenix, AZ, November 2003.

[54] Rew, R. and Davis, G., “The Unidata netCDF: Software for Scientific Data Access,” Sixth International Conference on Interactive Information and Processing Systems for Meteorology,

Oceanography and Hydrology, Anaheim, CA, February 1990.

[55] Pakalapati, P. D. and Hauser, T., “Benchmarking Parallel I/O Performance for Computational

Fluid Dynamics Applications,” AIAA Aerospace Sciences Meeting and Exhibit, Vol. 43, 2005.

[56] Hauser, T., “Parallel I/O for the CGNS System,” AIAA, 2004.

[57] La Vision, DaVis User’s Manual, 7th ed., 2009.

[58] Bell, C., Bonachea, D., Cote, Y., Duell, J., Hargrove, P., Husbands, P., Iancu, C., Welcome,

M., and Yelick, K., “An Evaluation of Current High-performance Networks,” IPDPS 2003,

Citeseer, 2003. 56

[59] Al Geist, A., Dongarra, J., Jiang, W., Manchek, R., and Sunderam, V., PVM-Parallel Virtual

Machine, The MIT Press, Cambridge, Cambridge, 1994.

[60] Forum, M. P. I., “MPI-2: Extensions to the Message-Passing Interface,” Internet http://

www.mpi-forum.org/docs/mpi-20-html/mpi2-report.html, 1997.

[61] Graham, R., Shipman, G., Barrett, B., Castain, R., Bosilca, G., Lumsdaine, A., Los Alamos,

N., Bloomington, I., and Knoxville, T., “Open MPI: A High-performance, Heterogeneous

MPI,” Proceeding of the Conference HeteroPar, Vol. 6, Citeseer, 2006.

[62] Gropp, W., Lusk, E., Doss, N., and Skjellum, A., “A High-performance, Portable Implemen-

tation of the MPI Message Passing Interface Standard,” Parallel Computing, Vol. 22, No. 6,

1996.

[63] Cappello, F., “MPI versus MPI+ OpenMP on IBM SP for the NAS benchmarks,” Proceedings

of the 2000 ACM/IEEE Conference on Supercomputing (CDROM), IEEE Computer Society

Washington, DC, 2000.

[64] “Parallel Virtual File System Home Page,” http://www.pvfs.org/, accessed August

2009.

[65] Ligon, W. and Ross, R., “Implementation and Performance of a Parallel File System for High

Performance Distributed Applications,” Proceedings of the Fifth IEEE International Sympo-

sium on High Performance Distributed Computing, Vol. 480, 1996.

[66] Yu, W., Vetter, J., Canon, R. S., and Jiang, S., “Exploiting Lustre File Joining for Effective

Collective IO,” CCGRID ’07: Proceedings of the Seventh IEEE International Symposium on

Cluster Computing and the Grid, IEEE Computer Society, Washington, DC, USA, 2007.

[67] Cope, J., Oberg, M., Tufo, H., and Woitaszek, M., “Shared Parallel File Systems in Heteroge-

neous Linux Multi-cluster Environments,” Proceedings of the 6th LCI International Confer-

ence on Linux Clusters: The HPC Revolution, Citeseer, 2005.

[68] “IBM General Parallel File System,” GPFS Home Page http://www-03.ibm.com/

systems/clusters/software/gpfs/index.html, accessed August 2009. 57

[69] Welch, B., Unangst, M., Abbasi, Z., Gibson, G., Mueller, B., Small, J., Zelenka, J., and

Zhou, B., “Scalable Performance of the Panasas Parallel File System,” Proceedings of the

6th USENIX Conference on File and Storage Technologies table of contents, USENIX Asso-

ciation, Berkeley, 2008.

[70] ISO, “The ANSI C Standard (C99),” Tech. Rep. WG14 N1124, ISO/IEC, 1999. 58

Appendix 59

Chapter A

pCGNS Source Code & Documentation

A.1 Data Structure Index

A.1.1 Data Structures Here are the data structures with brief descriptions:

base_s (Struct to describe a base node in the file ) ...... 60

coords_s (Struct to describe a coords node in the file ) ...... 63

file_s (Struct to describe a CGNS file ) ...... 64

iter_s (Structure to describe nodes during iteration ) ...... 69

section_s (Struct to describe a section node in the file ) ...... 70

slice_s ...... 72

sol_s (Struct to describe a solution node in the file ) ...... 74

zone_s (Struct to describe a zone node in the file ) ...... 76

A.2 File Index

A.2.1 File List Here is a list of all files with brief descriptions:

benchmark.c ...... 80

open_close.c ...... 94

pcgns_util.c ...... 97

pcgns_util.h ...... 140

pcgnslib.c ...... 168

pcgnslib.h ...... 228 60

test_base.c ...... 252

test_queue.c ...... 256

test_unstructured.c ...... 260

test_zone.c ...... 265

thesis_benchmark.c ...... 268

A.3 Data Structure Documentation

A.3.1 base_s Struct Reference Struct to describe a base node in the file.

#include Collaboration diagram for base_s:

coords_s sol_s section_s

coords sols sections

zone_s

zones

base_s

Data Fields

• int idx

Index of the node.

• char basename [100+1]

Name of the node. 61

• zone_t zones ∗ Array of zone_t objects which describe the zones in the base.

• int nzones

Length of the array {zones}.

• hid_t group_id

HDF5 handle to the node.

• int cell_dim

Cell dimensions.

• int phys_dim

Physical dimensions.

A.3.1.1 Detailed Description Struct to describe a base node in the file.

Definition at line 98 of file pcgns_util.h.

A.3.1.2 Field Documentation

A.3.1.2.1 char basename[100+1]

Name of the node.

Definition at line 102 of file pcgns_util.h. 62

A.3.1.2.2 int cell_dim

Cell dimensions.

Definition at line 110 of file pcgns_util.h.

A.3.1.2.3 hid_t group_id

HDF5 handle to the node.

Definition at line 108 of file pcgns_util.h.

A.3.1.2.4 int idx

Index of the node.

Definition at line 100 of file pcgns_util.h.

A.3.1.2.5 int nzones

Length of the array {zones}.

Definition at line 106 of file pcgns_util.h.

A.3.1.2.6 int phys_dim

Physical dimensions.

Definition at line 112 of file pcgns_util.h.

A.3.1.2.7 zone_t zones ∗

Array of zone_t objects which describe the zones in the base. 63

Definition at line 104 of file pcgns_util.h.

The documentation for this struct was generated from the following file:

• pcgns_util.h

A.3.2 coords_s Struct Reference Struct to describe a coords node in the file. #include

Data Fields

• int idx

Index of the node.

• char coordname [100+1]

Name of the node.

• hid_t group_id

HDF5 handle to the node.

A.3.2.1 Detailed Description Struct to describe a coords node in the file.

Definition at line 30 of file pcgns_util.h.

A.3.2.2 Field Documentation

A.3.2.2.1 char coordname[100+1]

Name of the node. 64

Definition at line 34 of file pcgns_util.h.

A.3.2.2.2 hid_t group_id

HDF5 handle to the node.

Definition at line 36 of file pcgns_util.h.

A.3.2.2.3 int idx

Index of the node.

Definition at line 32 of file pcgns_util.h.

The documentation for this struct was generated from the following file:

• pcgns_util.h

A.3.3 file_s Struct Reference Struct to describe a CGNS file. 65

#include Collaboration diagram for file_s:

coords_s sol_s section_s

coords sols sections

zone_s

zones

base_s

bases

file_s

Data Fields

• int idx

The index of this file in {files}.

• char filename [100+1]

The name of this file.

• int isOpen

Flag to tell if this file is open.

• base_t bases ∗ Array of base_t objects which describe the bases in the file. 66

• int nbases

Length of the array {bases}.

• hid_t file_id

HDF5 handle to the file.

• hid_t plist_id

HDF5 property list of the file.

• MPI_Comm comm

MPI comm on which the file was opened.

• MPI_Info info

MPI info.

• int rank

MPI rank of this process in this comm.

• int size

MPI size of this comm.

A.3.3.1 Detailed Description Struct to describe a CGNS file.

Definition at line 116 of file pcgns_util.h.

A.3.3.2 Field Documentation 67

A.3.3.2.1 base_t bases ∗

Array of base_t objects which describe the bases in the file.

Definition at line 124 of file pcgns_util.h.

A.3.3.2.2 MPI_Comm comm

MPI comm on which the file was opened.

Definition at line 132 of file pcgns_util.h.

A.3.3.2.3 hid_t file_id

HDF5 handle to the file.

Definition at line 128 of file pcgns_util.h.

A.3.3.2.4 char filename[100+1]

The name of this file.

Definition at line 120 of file pcgns_util.h.

A.3.3.2.5 int idx

The index of this file in {files}.

Definition at line 118 of file pcgns_util.h.

A.3.3.2.6 MPI_Info info

MPI info. 68

Definition at line 134 of file pcgns_util.h.

A.3.3.2.7 int isOpen

Flag to tell if this file is open.

Definition at line 122 of file pcgns_util.h.

A.3.3.2.8 int nbases

Length of the array {bases}.

Definition at line 126 of file pcgns_util.h.

A.3.3.2.9 hid_t plist_id

HDF5 property list of the file.

Definition at line 130 of file pcgns_util.h.

A.3.3.2.10 int rank

MPI rank of this process in this comm.

Definition at line 136 of file pcgns_util.h.

A.3.3.2.11 int size

MPI size of this comm.

Definition at line 138 of file pcgns_util.h.

The documentation for this struct was generated from the following file:

• pcgns_util.h 69

A.3.4 iter_s Struct Reference Structure to describe nodes during iteration.

Data Fields

• int pos

• int counter

• char name ∗ • char label ∗

A.3.4.1 Detailed Description Structure to describe nodes during iteration.

Definition at line 522 of file pcgns_util.c.

A.3.4.2 Field Documentation

A.3.4.2.1 int counter

Definition at line 522 of file pcgns_util.c.

A.3.4.2.2 char label ∗

Definition at line 522 of file pcgns_util.c.

A.3.4.2.3 char name ∗

Definition at line 522 of file pcgns_util.c. 70

A.3.4.2.4 int pos

Definition at line 522 of file pcgns_util.c.

The documentation for this struct was generated from the following file:

• pcgns_util.c

A.3.5 section_s Struct Reference Struct to describe a section node in the file. #include

Data Fields

• int idx

Index of the node.

• char sectionname [100+1]

Name of the node.

• hid_t group_id

HDF5 handle to the node.

• ElementType_t type

Type of elements in the section.

• int range [2]

Range of element ids in the section.

• hid_t connectivity_id 71

HDF5 handle to ElementConnectivity.

• hid_t range_id

HDF5 handle to ElementRange.

A.3.5.1 Detailed Description Struct to describe a section node in the file.

Definition at line 40 of file pcgns_util.h.

A.3.5.2 Field Documentation

A.3.5.2.1 hid_t connectivity_id

HDF5 handle to ElementConnectivity.

Definition at line 52 of file pcgns_util.h.

A.3.5.2.2 hid_t group_id

HDF5 handle to the node.

Definition at line 46 of file pcgns_util.h.

A.3.5.2.3 int idx

Index of the node.

Definition at line 42 of file pcgns_util.h.

A.3.5.2.4 int range[2] 72

Range of element ids in the section.

Definition at line 50 of file pcgns_util.h.

A.3.5.2.5 hid_t range_id

HDF5 handle to ElementRange.

Definition at line 54 of file pcgns_util.h.

A.3.5.2.6 char sectionname[100+1]

Name of the node.

Definition at line 44 of file pcgns_util.h.

A.3.5.2.7 ElementType_t type

Type of elements in the section.

Definition at line 48 of file pcgns_util.h.

The documentation for this struct was generated from the following file:

• pcgns_util.h

A.3.6 slice_s Struct Reference #include

Data Fields

• SliceType_t type

• int rank

• int min ∗ • int max ∗ 73

• void data ∗ • int F

• int B

• int Z

• int Selector

• char name [100+1]

A.3.6.1 Detailed Description Definition at line 141 of file pcgns_util.h.

A.3.6.2 Field Documentation

A.3.6.2.1 int B

Definition at line 148 of file pcgns_util.h.

A.3.6.2.2 void data ∗

Definition at line 146 of file pcgns_util.h.

A.3.6.2.3 int F

Definition at line 147 of file pcgns_util.h.

A.3.6.2.4 int max ∗

Definition at line 145 of file pcgns_util.h. 74

A.3.6.2.5 int min ∗

Definition at line 144 of file pcgns_util.h.

A.3.6.2.6 char name[100+1]

Definition at line 151 of file pcgns_util.h.

A.3.6.2.7 int rank

Definition at line 143 of file pcgns_util.h.

A.3.6.2.8 int Selector

Definition at line 150 of file pcgns_util.h.

A.3.6.2.9 SliceType_t type

Definition at line 142 of file pcgns_util.h.

A.3.6.2.10 int Z

Definition at line 149 of file pcgns_util.h.

The documentation for this struct was generated from the following file:

• pcgns_util.h

A.3.7 sol_s Struct Reference Struct to describe a solution node in the file. 75

#include

Data Fields

• int idx

Index of the node.

• char solname [100+1]

Name of the node.

• hid_t group_id

HDF5 handle to the node.

A.3.7.1 Detailed Description Struct to describe a solution node in the file.

Definition at line 58 of file pcgns_util.h.

A.3.7.2 Field Documentation

A.3.7.2.1 hid_t group_id

HDF5 handle to the node.

Definition at line 64 of file pcgns_util.h.

A.3.7.2.2 int idx

Index of the node.

Definition at line 60 of file pcgns_util.h. 76

A.3.7.2.3 char solname[100+1]

Name of the node.

Definition at line 62 of file pcgns_util.h.

The documentation for this struct was generated from the following file:

• pcgns_util.h

A.3.8 zone_s Struct Reference Struct to describe a zone node in the file.

#include Collaboration diagram for zone_s:

coords_s sol_s section_s

coords sols sections

zone_s

Data Fields

• int idx

Index of the node.

• char zonename [100+1]

Name of the node.

• coords_t coords ∗ Array of coord_t objects which describe the coords in the zone. 77

• int ncoords

Length of the array {coords}.

• section_t sections ∗ Array of section_s objects which describe the element sections in the zone.

• int nsections

Length of the array {sections}.

• sol_t sols ∗ Array of sol_t objects which describe the solutions in the zone.

• int nsols

Length of the array {sols}.

• hid_t group_id

HDF5 handle to the node.

• int nijk ∗ Zone dimensions.

• ZoneType_t type

Zone type.

• hid_t grid_id

HDF5 handle to the grid node.

• hid_t flow_id 78

HDF5 handle to the flow node.

A.3.8.1 Detailed Description Struct to describe a zone node in the file.

Definition at line 68 of file pcgns_util.h.

A.3.8.2 Field Documentation

A.3.8.2.1 coords_t coords ∗

Array of coord_t objects which describe the coords in the zone.

Definition at line 74 of file pcgns_util.h.

A.3.8.2.2 hid_t flow_id

HDF5 handle to the flow node.

Definition at line 94 of file pcgns_util.h.

A.3.8.2.3 hid_t grid_id

HDF5 handle to the grid node.

Definition at line 92 of file pcgns_util.h.

A.3.8.2.4 hid_t group_id

HDF5 handle to the node.

Definition at line 86 of file pcgns_util.h. 79

A.3.8.2.5 int idx

Index of the node.

Definition at line 70 of file pcgns_util.h.

A.3.8.2.6 int ncoords

Length of the array {coords}.

Definition at line 76 of file pcgns_util.h.

A.3.8.2.7 int nijk ∗

Zone dimensions.

Definition at line 88 of file pcgns_util.h.

A.3.8.2.8 int nsections

Length of the array {sections}.

Definition at line 80 of file pcgns_util.h.

A.3.8.2.9 int nsols

Length of the array {sols}.

Definition at line 84 of file pcgns_util.h.

A.3.8.2.10 section_t sections ∗

Array of section_s objects which describe the element sections in the zone. 80

Definition at line 78 of file pcgns_util.h.

A.3.8.2.11 sol_t sols ∗

Array of sol_t objects which describe the solutions in the zone.

Definition at line 82 of file pcgns_util.h.

A.3.8.2.12 ZoneType_t type

Zone type.

Definition at line 90 of file pcgns_util.h.

A.3.8.2.13 char zonename[100+1]

Name of the node.

Definition at line 72 of file pcgns_util.h.

The documentation for this struct was generated from the following file:

• pcgns_util.h

A.4 File Documentation

A.4.1 benchmark.c File Reference #include "pcgnslib.h"

#include "stdio.h"

#include "stdlib.h"

#include "mpi.h"

#include "string.h"

#include "math.h" 81

Include dependency graph for benchmark.c:

benchmark.c

pcgnslib.h stdio.h stdlib.h string.h math.h

mpi.h hdf5.h

Defines

• #define MEGA_BYTES 256

• #define BUF_LENGTH (MEGA_BYTES 1024 1024/sizeof(double)) ∗ ∗ • #define N ((int) sqrt((double) BUF_LENGTH))

Functions

• int initialize (int argc, char argv[ ]) ∗ ∗∗ • int finalize ()

• int doTimer (const char msg, double time) ∗ • int doBandwidth (const char msg, double time) ∗ • int doBandwidthAgg (const char msg, double time) ∗ • int main (int argc, char argv[ ]) ∗

Variables

• int comm_size

• int comm_rank

• MPI_Info info

• int nijk [3][3]

• double x ∗ • double y ∗ 82

• double z ∗ • int min [3]

• int max [3]

• int fn

• int B

• int Z

• int C

• double t0

• double t1

• double ta

A.4.1.1 Detailed Description Author:

Kyle Horne Version:

0.2

A.4.1.2 LICENSE BSD style license

A.4.1.3 DESCRIPTION Test program for pcgns library

Definition in file benchmark.c.

A.4.1.4 Define Documentation

A.4.1.4.1 #define BUF_LENGTH (MEGA_BYTES 1024 1024/sizeof(double)) ∗ ∗ 83

Definition at line 20 of file benchmark.c.

A.4.1.4.2 #define MEGA_BYTES 256

Definition at line 19 of file benchmark.c.

A.4.1.4.3 #define N ((int) sqrt((double) BUF_LENGTH))

Definition at line 23 of file benchmark.c.

A.4.1.5 Function Documentation

A.4.1.5.1 int doBandwidth (const char msg, double time) ∗

Definition at line 108 of file benchmark.c.

Here is the caller graph for this function:

doBandwidth main

A.4.1.5.2 int doBandwidthAgg (const char msg, double time) ∗

Definition at line 125 of file benchmark.c.

Here is the caller graph for this function:

doBandwidthAgg main

A.4.1.5.3 int doTimer (const char msg, double time) ∗ 84

Definition at line 96 of file benchmark.c.

Here is the caller graph for this function:

doTimer main

A.4.1.5.4 int finalize (void)

Definition at line 86 of file benchmark.c.

Here is the caller graph for this function:

finalize main

A.4.1.5.5 int initialize (int argc, char argv[ ]) ∗ ∗∗

Definition at line 46 of file benchmark.c.

Here is the caller graph for this function:

initialize main

A.4.1.5.6 int main (int argc, char argv[ ]) ∗

Definition at line 142 of file benchmark.c. 85

Here is the call graph for this function:

free_file

free_base

cgp_close del_node free_coord

cgp_base_write node_exists free_zone free_section

cgp_coord_write new_node new_int_attb free_sol

cgp_zone_write

new_str

hdf5_version_str

cgp_open new_float new_str_attb main cgp_coord_write_data

new_int

doBandwidth

doBandwidthAgg next_file cleanup_files

doTimer node_idx2name node_name_finder

finalize get_str_attb num_nodes node_counter

initialize hdf5_format_str

A.4.1.6 Variable Documentation

A.4.1.6.1 int B

Definition at line 38 of file benchmark.c.

A.4.1.6.2 int C

Definition at line 40 of file benchmark.c. 86

A.4.1.6.3 int comm_rank

Definition at line 26 of file benchmark.c.

A.4.1.6.4 int comm_size

Definition at line 25 of file benchmark.c.

A.4.1.6.5 int fn

Definition at line 37 of file benchmark.c.

A.4.1.6.6 MPI_Info info

Definition at line 27 of file benchmark.c.

A.4.1.6.7 int max[3]

Definition at line 35 of file benchmark.c.

A.4.1.6.8 int min[3]

Definition at line 34 of file benchmark.c.

A.4.1.6.9 int nijk[3][3]

Definition at line 28 of file benchmark.c. 87

A.4.1.6.10 double t0

Definition at line 42 of file benchmark.c.

A.4.1.6.11 double t1

Definition at line 43 of file benchmark.c.

A.4.1.6.12 double ta

Definition at line 44 of file benchmark.c.

A.4.1.6.13 double x ∗

Definition at line 30 of file benchmark.c.

A.4.1.6.14 double y ∗

Definition at line 31 of file benchmark.c.

A.4.1.6.15 int Z

Definition at line 39 of file benchmark.c.

A.4.1.6.16 double z ∗

Definition at line 32 of file benchmark.c. 88

A.4.2 benchmark.c

00001 00002 00003 00004 00005 00006 00007 00008 00009 00010 00011 #include "pcgnslib.h" 00012 00013 #include "stdio.h" 00014 #include "stdlib.h" 00015 #include "mpi.h" 00016 #include "string.h" 00017 #include "math.h" 00018 00019 #define MEGA_BYTES 256

00020 #define BUF_LENGTH (MEGA_BYTES*1024*1024/sizeof(double)) 00021 //#define BUF_LENGTH (9) 00022 00023 #define N ((int) sqrt((double) BUF_LENGTH)) 00024 00025 int comm_size; 00026 int comm_rank; 00027 MPI_Info info; 00028 int nijk[3][3]; 00029

00030 double* x; 00031 double* y; 00032 double* z; 00033 00034 int min[3]; 89

00035 int max[3]; 00036 00037 int fn; 00038 int B; 00039 int Z; 00040 int C; 00041 00042 double t0; 00043 double t1; 00044 double ta; 00045

00046 int initialize(int* argc, char** argv[]) { 00047 MPI_Init(argc,argv); 00048 MPI_Comm_size(MPI_COMM_WORLD, &comm_size); 00049 MPI_Comm_rank(MPI_COMM_WORLD, &comm_rank); 00050 MPI_Info_create(&(info)); 00051 00052 nijk[0][0] = N; 00053 nijk[0][1] = N; 00054 nijk[0][2] = comm_size; 00055 nijk[1][0] = nijk[0][0]-1; 00056 nijk[1][1] = nijk[0][1]-1; 00057 nijk[1][2] = nijk[0][2]-1; 00058 nijk[2][0] = 0; 00059 nijk[2][1] = 0; 00060 nijk[2][2] = 0; 00061

00062 x = (double*) malloc(BUF_LENGTH*sizeof(double)); 00063 y = (double*) malloc(BUF_LENGTH*sizeof(double)); 00064 z = (double*) malloc(BUF_LENGTH*sizeof(double)); 00065 00066 int i,j; 00067 for(i=0;i

00069 x[i*N+j] = (double) (i); 90

00070 y[i*N+j] = (double) (j); 00071 z[i*N+j] = (double) (comm_rank); 00072 } 00073 } 00074 00075 min[0] = comm_rank; 00076 min[1] = 0; 00077 min[2] = 0; 00078 max[0] = comm_rank; 00079 max[1] = N-1; 00080 max[2] = N-1; 00081 00082 00083 return 0; 00084 } 00085 00086 int finalize() { 00087 free(x); 00088 free(y); 00089 free(z); 00090 00091 MPI_Finalize(); 00092 00093 return 0; 00094 } 00095

00096 int doTimer(const char* msg, double time) { 00097 double min; 00098 double max; 00099 double avg; 00100 MPI_Reduce(&time, &min, 1, MPI_DOUBLE, MPI_MIN, 0, MPI_COMM_WORLD); 00101 MPI_Reduce(&time, &max, 1, MPI_DOUBLE, MPI_MAX, 0, MPI_COMM_WORLD); 00102 MPI_Reduce(&time, &avg, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD); 00103 avg = avg/((double) comm_size); 00104 if(comm_rank==0) printf("%20s Time = { min: %-20f max: %-20f avg: %-20f} s n", \ 91

msg,min,max,avg); 00105 return 0; 00106 } 00107

00108 int doBandwidth(const char* msg, double time) { 00109 double min; 00110 double max; 00111 double avg; 00112

00113 double MB = ((double) BUF_LENGTH*sizeof(double))/(1024.0*1024.0); 00114 MPI_Reduce(&time, &max, 1, MPI_DOUBLE, MPI_MIN, 0, MPI_COMM_WORLD); 00115 max = MB/max; 00116 MPI_Reduce(&time, &min, 1, MPI_DOUBLE, MPI_MAX, 0, MPI_COMM_WORLD); 00117 min = MB/min; 00118 MPI_Reduce(&time, &avg, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD); 00119 avg = avg/((double) comm_size); 00120 avg = MB/avg; 00121 if(comm_rank==0) printf("%20s Band = { min: %-20f max: %-20f avg: %-20f} MB/s (local) n",msg,min,max,avg); \ 00122 return 0; 00123 } 00124

00125 int doBandwidthAgg(const char* msg, double time) { 00126 double min; 00127 double max; 00128 double avg; 00129

00130 double MB = ((double) BUF_LENGTH*sizeof(double))/(1024.0*1024.0)*((double) comm_size); 00131 MPI_Reduce(&time, &max, 1, MPI_DOUBLE, MPI_MIN, 0, MPI_COMM_WORLD); 00132 max = MB/max; 00133 MPI_Reduce(&time, &min, 1, MPI_DOUBLE, MPI_MAX, 0, MPI_COMM_WORLD); 00134 min = MB/min; 00135 MPI_Reduce(&time, &avg, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD); 00136 avg = avg/((double) comm_size); 92

00137 avg = MB/avg; 00138 if(comm_rank==0) printf("%20s Band = { min: %-20f max: %-20f avg: %-20f} MB/s (aggregate) n",msg,min,max,avg); \ 00139 return 0; 00140 } 00141

00142 int main(int argc, char* argv[]) { 00143 // Initialize varaibles 00144 initialize(&argc,&argv); 00145 00146 // Time the creation of a file 00147 t0 = MPI_Wtime(); 00148 cgp_open("benchmark.cgns", 0, MPI_COMM_WORLD, &info, &fn); 00149 t1 = MPI_Wtime(); 00150 doTimer("File Open", t1-t0); 00151 00152 // Time the creation of a base 00153 t0 = MPI_Wtime(); 00154 cgp_base_write(fn, "Base 1", 3, 3, &B); 00155 t1 = MPI_Wtime(); 00156 doTimer("Base Write", t1-t0); 00157 00158 // Time the creation of a zone 00159 t0 = MPI_Wtime(); 00160 cgp_zone_write(fn, B, "Zone 1", &(nijk[0][0]), 0, &Z); 00161 t1 = MPI_Wtime(); 00162 doTimer("Zone Write", t1-t0); 00163 00164 // Time the creation of coordinates X 00165 t0 = MPI_Wtime(); 00166 cgp_coord_write(fn,B,Z,0,"CoordinateX",&C); 00167 t1 = MPI_Wtime(); 00168 doTimer("Coord X Write", t1-t0); 00169 00170 // Time the write speed of coordinates X 93

00171 MPI_Barrier(MPI_COMM_WORLD); 00172 t0 = MPI_Wtime(); 00173 cgp_coord_write_data(fn,B,Z,C,min,max,x); 00174 t1 = MPI_Wtime(); 00175 MPI_Barrier(MPI_COMM_WORLD); 00176 ta = MPI_Wtime(); 00177 00178 doTimer("Coord X Write Data", t1-t0); 00179 doBandwidth("Coord X Write Data", t1-t0); 00180 doBandwidthAgg("Coord X Write Data", ta-t0); 00181 00182 // Time the creation of coordinates Y 00183 t0 = MPI_Wtime(); 00184 cgp_coord_write(fn,B,Z,0,"CoordinateY",&C); 00185 t1 = MPI_Wtime(); 00186 doTimer("Coord Y Write", t1-t0); 00187 00188 // Time the write speed of coordinates Y 00189 MPI_Barrier(MPI_COMM_WORLD); 00190 t0 = MPI_Wtime(); 00191 cgp_coord_write_data(fn,B,Z,C,min,max,y); 00192 t1 = MPI_Wtime(); 00193 MPI_Barrier(MPI_COMM_WORLD); 00194 ta = MPI_Wtime(); 00195 00196 doTimer("Coord Y Write Data", t1-t0); 00197 doBandwidth("Coord Y Write Data", t1-t0); 00198 doBandwidthAgg("Coord Y Write Data", ta-t0); 00199 00200 // Time the creation of coordinates Z 00201 t0 = MPI_Wtime(); 00202 cgp_coord_write(fn,B,Z,0,"CoordinateZ",&C); 00203 t1 = MPI_Wtime(); 00204 doTimer("Coord Z Write", t1-t0); 00205 94

00206 // Time the write speed of coordinates Z 00207 MPI_Barrier(MPI_COMM_WORLD); 00208 t0 = MPI_Wtime(); 00209 cgp_coord_write_data(fn,B,Z,C,min,max,z); 00210 t1 = MPI_Wtime(); 00211 MPI_Barrier(MPI_COMM_WORLD); 00212 ta = MPI_Wtime(); 00213 00214 doTimer("Coord Z Write Data", t1-t0); 00215 doBandwidth("Coord Z Write Data", t1-t0); 00216 doBandwidthAgg("Coord Z Write Data", ta-t0); 00217 00218 // Time closing of the file 00219 t0 = MPI_Wtime(); 00220 cgp_close(fn); 00221 t1 = MPI_Wtime(); 00222 doTimer("File Close", t1-t0); 00223 00224 finalize(); 00225 00226 return 0; 00227 }

A.4.3 open_close.c File Reference #include "pcgnslib.h"

#include "stdio.h"

#include "stdlib.h"

#include "mpi.h" 95

Include dependency graph for open_close.c:

open_close.c

pcgnslib.h stdio.h stdlib.h

mpi.h hdf5.h

Functions

• int main (int argc, char argv[ ]) ∗

A.4.3.1 Detailed Description

Author:

Kyle Horne

Version:

0.2

A.4.3.2 LICENSE BSD style license

A.4.3.3 DESCRIPTION Test program for pcgns library

Definition in file open_close.c.

A.4.3.4 Function Documentation

A.4.3.4.1 int main (int argc, char argv[ ]) ∗ 96

Definition at line 17 of file open_close.c.

Here is the call graph for this function:

del_node free_coord

cgp_close free_file free_base free_zone free_section

hdf5_format_str free_sol

hdf5_version_str main new_float

new_int

cgp_open new_node new_int_attb

new_str_attb

new_str

next_file cleanup_files

node_idx2name node_name_finder get_str_attb num_nodes node_counter

A.4.4 open_close.c

00001 00002 00003 00004 00005 00006 00007 00008 00009 00010 00011 #include "pcgnslib.h" 00012 00013 #include "stdio.h" 00014 #include "stdlib.h" 00015 #include "mpi.h" 97

00016

00017 int main(int argc, char* argv[]) { 00018 int err; 00019 int comm_size; 00020 int comm_rank; 00021 MPI_Info info; 00022 int fn; 00023 00024 err = MPI_Init(&argc,&argv); 00025 if(err!=MPI_SUCCESS) cgp_doError; 00026 err = MPI_Comm_size(MPI_COMM_WORLD, &comm_size); 00027 if(err!=MPI_SUCCESS) cgp_doError; 00028 err = MPI_Comm_rank(MPI_COMM_WORLD, &comm_rank); 00029 if(err!=MPI_SUCCESS) cgp_doError; 00030 err = MPI_Info_create(&(info)); 00031 if(err!=MPI_SUCCESS) cgp_doError; 00032 00033 err = cgp_open("open_close.cgns", 0, MPI_COMM_WORLD, &info, &fn); 00034 if(err!=0) cgp_doError; 00035 err = cgp_close(fn); 00036 if(err!=0) cgp_doError; 00037 00038 err = MPI_Finalize(); 00039 if(err!=MPI_SUCCESS) cgp_doError; 00040 return err; 00041 }

A.4.5 pcgns_util.c File Reference #include "pcgns_util.h"

#include "stdlib.h"

#include "string.h" 98

Include dependency graph for pcgns_util.c:

pcgns_util.c

pcgns_util.h stdlib.h string.h

pcgnslib.h

mpi.h hdf5.h

Data Structures

• struct iter_s

Structure to describe nodes during iteration.

Functions

• void cleanup_files (void)

Function to free the array {files} at exit.

• int next_file (int fn) ∗ • int free_file (file_t file) ∗ • int free_base (base_t base) ∗ • int free_zone (zone_t zone) ∗ • int free_coord (coords_t coords) ∗ • int free_section (section_t section) ∗ • int free_sol (sol_t sol) ∗ • int new_str (hid_t pid, const char name, const char value, int len) ∗ ∗ • int get_str (hid_t pid, const char name, int len, char value) ∗ ∗ • int new_str_attb (hid_t pid, const char name, const char value, int len) ∗ ∗ 99

• int get_str_attb (hid_t pid, const char name, int len, char value) ∗ ∗ • int new_int (hid_t pid, const char name, const int value) ∗ ∗ • int get_int (hid_t pid, const char name, int value) ∗ ∗ • int new_int_attb (hid_t pid, const char name, const int value) ∗ ∗ • int get_int_attb (hid_t pid, const char name, int value) ∗ ∗ • int new_float (hid_t pid, const char name, const float value) ∗ ∗ • int get_float (hid_t pid, const char name, float value) ∗ ∗ • int new_float_attb (hid_t pid, const char name, const float value) ∗ ∗ • int get_float_attb (hid_t pid, const char name, float value) ∗ ∗ • int new_node (hid_t pid, const char name, const char label, const char type) ∗ ∗ ∗ • int del_node (hid_t pid, const char name) ∗ • int node_exists (hid_t pid, const char name) ∗ • herr_t node_counter (hid_t gid, const char name, const H5L_info_t linfo, void vdata) ∗ ∗ ∗ Call-back function used to count the nodes with a particular label.

• int num_nodes (hid_t pid, const char label, int num) ∗ ∗ • herr_t node_idx_finder (hid_t gid, const char name, const H5L_info_t linfo, void vdata) ∗ ∗ ∗ Call-back function used to find the index of a nodes with a particular label.

• int node_name2idx (hid_t pid, const char label, const char name, int idx) ∗ ∗ ∗ • herr_t node_name_finder (hid_t gid, const char name, const H5L_info_t linfo, void ∗ ∗ vdata) ∗ Call-back function used to find the name of a node at index {idx}.

• int node_idx2name (hid_t pid, const char label, int idx, char name) ∗ ∗ • int hdf5_version_str (int len, char buf) ∗ • int hdf5_format_str (int len, char buf) ∗ 100

Variables

• file_t files = NULL ∗ Array of file_t’s which decribe the open files.

• int files_count = 0

Internal count of used slots in {files}.

• int files_size = 0

Internal count of the slots in {files}.

• slice_t write_queue = NULL ∗ Queue of IO write operations.

• int write_queue_len = 0

Length of write queue.

A.4.5.1 Detailed Description

Author:

Kyle Horne

Version:

0.2

A.4.5.2 LICENSE BSD style license

A.4.5.3 DESCRIPTION 101

Implementation of utility functions

Definition in file pcgns_util.c.

A.4.5.4 Function Documentation

A.4.5.4.1 void cleanup_files (void)

Function to free the array {files} at exit.

Definition at line 33 of file pcgns_util.c.

Here is the caller graph for this function:

cleanup_files next_file cgp_open main

A.4.5.4.2 int del_node (hid_t pid, const char name) ∗

Delete a node with parent {pid}

Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of node

Returns:

Error code

Definition at line 506 of file pcgns_util.c. 102

Here is the caller graph for this function:

cgp_array_write main

cgp_sol_write

cgp_base_write

del_node cgp_close

main cgp_coord_write

cgp_zone_write

cgp_section_write main

A.4.5.4.3 int free_base (base_t base) ∗

Free the memory of a base_t object

Parameters:

base [in]: Pointer the base

Returns:

Error code

Definition at line 103 of file pcgns_util.c.

Here is the call graph for this function:

free_coord

free_base free_zone free_section

free_sol

Here is the caller graph for this function:

cgp_base_write

free_base main free_file cgp_close 103

A.4.5.4.4 int free_coord (coords_t coords) ∗

Definition at line 153 of file pcgns_util.c.

Here is the caller graph for this function:

cgp_zone_write

free_coord free_zone free_base cgp_base_write main

free_file cgp_close

A.4.5.4.5 int free_file (file_t file) ∗

Free the memory of a file_t object Parameters:

file [in]: Pointer the file Returns:

Error code

Definition at line 69 of file pcgns_util.c.

Here is the call graph for this function:

free_coord

free_file free_base free_zone free_section

free_sol

Here is the caller graph for this function:

free_file cgp_close main

A.4.5.4.6 int free_section (section_t section) ∗

Free the memory of a section_t object 104 Parameters:

section [in]: Pointer the section

Returns:

Error code

Definition at line 163 of file pcgns_util.c.

Here is the caller graph for this function:

cgp_zone_write free_section free_zone free_base cgp_base_write main

free_file cgp_close

A.4.5.4.7 int free_sol (sol_t sol) ∗

Free the memory of a sol_t object Parameters:

sol [in]: Pointer the sol

Returns:

Error code

Definition at line 179 of file pcgns_util.c.

Here is the caller graph for this function:

cgp_zone_write free_sol free_zone free_base cgp_base_write main

free_file cgp_close

A.4.5.4.8 int free_zone (zone_t zone) ∗

Free the memory of a zone_t object 105 Parameters:

zone [in]: Pointer the zone

Returns:

Error code

Definition at line 121 of file pcgns_util.c.

Here is the call graph for this function:

free_coord

free_zone free_section

free_sol

Here is the caller graph for this function:

cgp_zone_write free_zone free_base cgp_base_write main

free_file cgp_close

A.4.5.4.9 int get_float (hid_t pid, const char name, float value) ∗ ∗

Read an float with parent {pid}, name {name}, and value {value} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of float

value [out]: Float to read

Returns:

Error code

Definition at line 412 of file pcgns_util.c. 106

A.4.5.4.10 int get_float_attb (hid_t pid, const char name, float value) ∗ ∗

Read an float attribute at location {pid} with {name} with {value} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of float

value [out]: Float to read

Returns:

Error code

Definition at line 456 of file pcgns_util.c.

A.4.5.4.11 int get_int (hid_t pid, const char name, int value) ∗ ∗

Read an integer with parent {pid}, name {name}, and value {value} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of integer

value [out]: Integer to read

Returns:

Error code

Definition at line 322 of file pcgns_util.c.

A.4.5.4.12 int get_int_attb (hid_t pid, const char name, int value) ∗ ∗

Read an integer attribute at location {pid} with {name} with {value} Parameters:

pid [in]: HDF5 locator for parent 107

name [in]: Name of integer

value [out]: Integer to read

Returns:

Error code

Definition at line 366 of file pcgns_util.c.

A.4.5.4.13 int get_str (hid_t pid, const char name, int len, char value) ∗ ∗

Read a string with parent {pid}, name {name}, and value {value} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of string

len [in]: Length of string in file

value [out]: Contents of string

Returns:

Error code

Definition at line 214 of file pcgns_util.c.

Here is the caller graph for this function:

get_str cgp_zone_read main

A.4.5.4.14 int get_str_attb (hid_t pid, const char name, int len, char value) ∗ ∗

Read a string attribute at location {pid} with {name} with {value} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of string 108

len [in]: Length of string in file

value [out]: Contents of string

Returns:

Error code

Definition at line 269 of file pcgns_util.c.

Here is the caller graph for this function:

cgp_base_read main

node_name_finder node_idx2name cgp_open main

get_str_attb node_counter num_nodes cgp_zone_read main

node_idx_finder node_name2idx

A.4.5.4.15 int hdf5_format_str (int len, char buf) ∗

Fill a buffer with the HDF5 format string Parameters:

len [in]: Length of buf

buf [out]: String with the HDF5 format

Returns:

Error code

Definition at line 706 of file pcgns_util.c.

Here is the caller graph for this function:

hdf5_format_str cgp_open main

A.4.5.4.16 int hdf5_version_str (int len, char buf) ∗

Fill a buffer with the HDF5 version number 109 Parameters:

len [in]: Length of buf

buf [out]: String with HDF5 version number

Returns:

Error code

Definition at line 692 of file pcgns_util.c.

Here is the caller graph for this function:

hdf5_version_str cgp_open main

A.4.5.4.17 int new_float (hid_t pid, const char name, const float value) ∗ ∗

Create a new float with parent {pid}, name {name}, and value {value} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of float

value [in]: Float to write

Returns:

Error code

Definition at line 390 of file pcgns_util.c.

Here is the caller graph for this function:

new_float cgp_open main

A.4.5.4.18 int new_float_attb (hid_t pid, const char name, const float value) ∗ ∗

Create a new float attribute at location {pid} with {name} with {value} 110 Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of float

value [in]: Float to write Returns:

Error code

Definition at line 434 of file pcgns_util.c.

A.4.5.4.19 int new_int (hid_t pid, const char name, const int value) ∗ ∗

Create a new integer with parent {pid}, name {name}, and value {value} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of integer

value [in]: Integer to write

Returns:

Error code

Definition at line 300 of file pcgns_util.c.

Here is the caller graph for this function:

new_int cgp_open main

A.4.5.4.20 int new_int_attb (hid_t pid, const char name, const int value) ∗ ∗

Create a new integer attribute at location {pid} with {name} with {value} Parameters:

pid [in]: HDF5 locator for parent 111

name [in]: Name of integer

value [in]: Integer to write

Returns:

Error code

Definition at line 344 of file pcgns_util.c.

Here is the caller graph for this function:

cgp_array_write main

cgp_sol_write

cgp_base_write

cgp_coord_write

new_int_attb new_node main

cgp_open

cgp_zone_write

cgp_section_write main

cgp_zone_read main

A.4.5.4.21 int new_node (hid_t pid, const char name, const char label, const char ∗ ∗ ∗ type)

Create a new node at with parent {pid} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of node

label [in]: Label of node

type [in]: Type of node

Returns:

Error code 112

Definition at line 480 of file pcgns_util.c.

Here is the call graph for this function:

new_int_attb

new_node

new_str_attb

Here is the caller graph for this function:

cgp_array_write main

cgp_sol_write

cgp_base_write

cgp_coord_write new_node main cgp_open

cgp_zone_write

cgp_section_write main

cgp_zone_read main

A.4.5.4.22 int new_str (hid_t pid, const char name, const char value, int len) ∗ ∗

Create a new string with parent {pid}, name {name}, and value {value} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of string

value [in]: Contents of string

len [in]: Length of string in file

Returns:

Error code

Definition at line 192 of file pcgns_util.c. 113

Here is the caller graph for this function:

cgp_open

new_str main cgp_zone_write

A.4.5.4.23 int new_str_attb (hid_t pid, const char name, const char value, int len) ∗ ∗

Create a new string attribute at location {pid} with {name} with {value}

Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of string

value [in]: Contents of string

len [in]: Length of string in file

Returns:

Error code

Definition at line 239 of file pcgns_util.c.

Here is the caller graph for this function:

cgp_open

cgp_base_write main

new_str_attb cgp_coord_write

new_node cgp_zone_write

cgp_array_write main cgp_sol_write

cgp_section_write main

cgp_zone_read main 114

A.4.5.4.24 int next_file (int fn) ∗

Return the number of the next availible file_t in {files} Parameters:

fn [out]: Index of next file Returns:

Error code

Definition at line 37 of file pcgns_util.c.

Here is the call graph for this function:

next_file cleanup_files

Here is the caller graph for this function:

next_file cgp_open main

A.4.5.4.25 herr_t node_counter (hid_t gid, const char name, const H5L_info_t linfo, ∗ ∗ void vdata) ∗

Call-back function used to count the nodes with a particular label.

Definition at line 525 of file pcgns_util.c.

Here is the call graph for this function:

node_counter get_str_attb

Here is the caller graph for this function:

cgp_base_read main

node_counter num_nodes cgp_open main

cgp_zone_read main 115

A.4.5.4.26 int node_exists (hid_t pid, const char name) ∗

Check if a node exists with name {name} and parent {pid}

Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of node

Returns:

{1=Exists, 0=Does not exist}

Definition at line 514 of file pcgns_util.c.

Here is the caller graph for this function:

cgp_array_write

main

cgp_sol_write

cgp_base_write

node_exists cgp_coord_write main

cgp_zone_write

cgp_section_write main

cgp_zone_read main

A.4.5.4.27 int node_idx2name (hid_t pid, const char label, int idx, char name) ∗ ∗

Convert a node id to a node name Parameters:

pid [in]: HDF5 locator for parent

label [in]: Label of node

idx [in]: Index of node

name [out]: Name of node 116

Definition at line 672 of file pcgns_util.c.

Here is the call graph for this function:

node_idx2name node_name_finder get_str_attb

Here is the caller graph for this function:

cgp_base_read main

node_idx2name cgp_open main

cgp_zone_read main

A.4.5.4.28 herr_t node_idx_finder (hid_t gid, const char name, const H5L_info_t linfo, ∗ ∗ void vdata) ∗

Call-back function used to find the index of a nodes with a particular label.

Definition at line 573 of file pcgns_util.c.

Here is the call graph for this function:

node_idx_finder get_str_attb

Here is the caller graph for this function:

node_idx_finder node_name2idx

A.4.5.4.29 int node_name2idx (hid_t pid, const char label, const char name, int idx) ∗ ∗ ∗

Convert a node name to a node id Parameters:

pid [in]: HDF5 locator for parent

label [in]: Label of node 117

name [in]: Name of node

idx [out]: Index of node Returns:

Error code

Definition at line 612 of file pcgns_util.c.

Here is the call graph for this function:

node_name2idx node_idx_finder get_str_attb

A.4.5.4.30 herr_t node_name_finder (hid_t gid, const char name, const H5L_info_t ∗ ∗ linfo, void vdata) ∗

Call-back function used to find the name of a node at index {idx}.

Definition at line 633 of file pcgns_util.c.

Here is the call graph for this function:

node_name_finder get_str_attb

Here is the caller graph for this function:

cgp_base_read main

node_name_finder node_idx2name cgp_open main

cgp_zone_read main

A.4.5.4.31 int num_nodes (hid_t pid, const char label, int num) ∗ ∗

Count the number of nodes of type {label} with parent {pid} Parameters:

pid [in]: HDF5 locator for parent 118

label [in]: Label of nodes

num [out]: Number of nodes

Returns:

Error code

Definition at line 554 of file pcgns_util.c.

Here is the call graph for this function:

num_nodes node_counter get_str_attb

Here is the caller graph for this function:

cgp_base_read main

num_nodes cgp_open main

cgp_zone_read main

A.4.5.5 Variable Documentation

A.4.5.5.1 file_t files = NULL ∗

Array of file_t’s which decribe the open files.

Definition at line 21 of file pcgns_util.c.

A.4.5.5.2 int files_count = 0

Internal count of used slots in {files}.

Definition at line 22 of file pcgns_util.c. 119

A.4.5.5.3 int files_size = 0

Internal count of the slots in {files}.

Definition at line 23 of file pcgns_util.c.

A.4.5.5.4 slice_t write_queue = NULL ∗

Queue of IO write operations.

Definition at line 25 of file pcgns_util.c.

A.4.5.5.5 int write_queue_len = 0

Length of write queue.

Definition at line 26 of file pcgns_util.c.

A.4.6 pcgns_util.c

00001 00002 00003 00004 00005 00006 00007 00008 00009 00010 00011 #include "pcgns_util.h" 00012 00013 #include "stdlib.h" 00014 #include "string.h" 00015 120

00016 //======// 00017 //== Begin Global Data ==// 00018 //======// 00019 00020

00021 file_t* files = NULL; 00022 int files_count = 0; 00023 int files_size = 0; 00024

00025 slice_t* write_queue = NULL; 00026 int write_queue_len = 0; 00027 00028 //======// 00029 //== Begin Function Definitions ==// 00030 //======// 00031 00033 void cleanup_files(void){ 00034 if(files!=NULL) free(files); 00035 } 00036

00037 int next_file(int* fn) { 00038 int err = 0; 00039 // Loop index 00040 int k; 00041 // Return the index of the next slot

00042 *fn = files_count; 00043 // Increment the number of used slots 00044 files_count++; 00045 // If this is the first execution, register to cleanup files at exit 00046 if(files_size==0) atexit(cleanup_files); 00047 // If the returned slot does not exist, extend the list so that it does 00048 if(files_size<=files_count) { 00049 // Keep track of how many slots were used 00050 int old_files_size = files_size; 00051 // Set the new size of the array 121

00052 files_size = 2*files_count; 00053 // Allocate a new array to replace the old one

00054 file_t* new_files = malloc(2*files_count*sizeof(file_t)); 00055 if(new_files==NULL) cgp_doError; 00056 // Copy the old array’s data to the new array 00057 for(k=0;k

00069 int free_file(file_t* file) { 00070 printTime; 00071 int err = 0; 00072 herr_t herr; 00073 // Loop index 00074 int k; 00075 // Free all the bases in this file 00076 printTime; 00077 for(k=0;knbases;k++) err = free_base(&(file->bases[k])); 00078 printTime; 00079 if(err!=0) cgp_doError; 00080 // If bases was allocated, free it 00081 if(file->bases!=NULL) free(file->bases); 00082 // Free the MPI info object 00083 err = MPI_Info_free(&(file->info)); 00084 if(err!=MPI_SUCCESS) cgp_doError; 00085 // Free the property list 00086 printTime; 122

00087 herr = H5Pclose(file->plist_id); 00088 printTime; 00089 if(herr<0) cgp_doError; 00090 // Free the HDF5 file 00091 printTime; 00092 herr = H5Fclose(file->file_id); 00093 printTime; 00094 if(herr<0) cgp_doError; 00095 // Set the file’s status to closed 00096 file->isOpen = FALSE; 00097 // Zero out HDF5 pointers 00098 file->plist_id = 0; 00099 file->file_id = 0; 00100 return 0; 00101 } 00102

00103 int free_base(base_t* base) { 00104 int err = 0; 00105 herr_t herr; 00106 // Loop index 00107 int k; 00108 // Free the zones in this base 00109 for(k=0;knzones;k++) err = free_zone(&(base->zones[k])); 00110 if(err!=0) cgp_doError; 00111 // If zones was allocated, free it 00112 if(base->zones!=NULL) free(base->zones); 00113 // Free the HDF5 group 00114 herr = H5Gclose(base->group_id); 00115 if(herr<0) cgp_doError; 00116 // Zero out the HDF5 pointer 00117 base->group_id = 0; 00118 return 0; 00119 } 00120

00121 int free_zone(zone_t* zone) { 123

00122 int err = 0; 00123 herr_t herr; 00124 // Loop index 00125 int k; 00126 // Free the zone dimensions 00127 if(zone->nijk!=NULL) free(zone->nijk); 00128 // Free the coords in this zone 00129 for(k=0;kncoords;k++) err = free_coord(&(zone->coords[k])); 00130 if(err!=0) cgp_doError; 00131 // If coords was allocated, free it 00132 if(zone->coords!=NULL) free(zone->coords); 00133 // Free the sections in this zone 00134 for(k=0;knsections;k++) err = free_section(&(zone->sections[k])); 00135 if(zone->sections!=NULL) free(zone->sections); 00136 // Free the sols in this zone 00137 for(k=0;knsols;k++) err = free_sol(&(zone->sols[k])); 00138 if(err!=0) cgp_doError; 00139 // If sols was allocated, free it 00140 if(zone->sols!=NULL) free(zone->sols); 00141 // Free the HDF5 groups 00142 herr = H5Gclose(zone->group_id); 00143 if(herr<0) cgp_doError; 00144 herr = H5Gclose(zone->grid_id); 00145 if(herr<0) cgp_doError; 00146 herr = H5Gclose(zone->flow_id); 00147 if(herr<0) cgp_doError; 00148 // Zero out the HDF5 pointer 00149 zone->group_id = 0; 00150 return 0; 00151 } 00152

00153 int free_coord(coords_t* coords) { 00154 herr_t herr; 00155 // Free the HDF5 group 00156 herr = H5Gclose(coords->group_id); 124

00157 if(herr<0) cgp_doError; 00158 // Zero out the HDF5 pointer 00159 coords->group_id = 0; 00160 return 0; 00161 } 00162

00163 int free_section(section_t* section) { 00164 herr_t herr; 00165 // Free the HDF5 groups 00166 herr = H5Gclose(section->group_id); 00167 if(herr<0) cgp_doError; 00168 herr = H5Gclose(section->connectivity_id); 00169 if(herr<0) cgp_doError; 00170 herr = H5Gclose(section->range_id); 00171 if(herr<0) cgp_doError; 00172 // Zero out the HDf5 pointers 00173 section->group_id = 0; 00174 section->connectivity_id = 0; 00175 section->range_id = 0; 00176 return 0; 00177 } 00178

00179 int free_sol(sol_t* sol) { 00180 herr_t herr; 00181 // Free the HDF5 group 00182 herr = H5Gclose(sol->group_id); 00183 if(herr<0) cgp_doError; 00184 // Zero out the HDF5 pointer 00185 sol->group_id = 0; 00186 return 0; 00187 } 00188 00189 00190 //======00191 125

00192 int new_str(hid_t pid, const char* name, const char* value, int len) { 00193 herr_t herr; 00194 // Set the dimension of the data to the length of the string, including the te rminator 00195 hsize_t dim = len+1; 00196 // Create a shape object for the data 00197 hid_t shape_id = H5Screate_simple(1,&dim, NULL); 00198 if(shape_id<0) cgp_doError; 00199 // Create the data in the file 00200 hid_t data_id = H5Dcreate2(pid, name, H5T_NATIVE_CHAR, shape_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); 00201 if(data_id<0) cgp_doError; 00202 // Write the data to the file 00203 herr = H5Dwrite(data_id, H5T_NATIVE_CHAR, H5S_ALL, H5S_ALL, H5P_DEFAULT, value ); 00204 if(herr<0) cgp_doError; 00205 // Close the data 00206 herr = H5Dclose(data_id); 00207 if(herr<0) cgp_doError; 00208 // Close the shape 00209 herr = H5Sclose(shape_id); 00210 if(herr<0) cgp_doError; 00211 return 0; 00212 } 00213

00214 int get_str(hid_t pid, const char* name, int len, char* value) { 00215 herr_t herr; 00216 // Open the data in the file 00217 hid_t data_id = H5Dopen2(pid, name, H5P_DEFAULT); 00218 if(data_id<0) cgp_doError; 00219 // Get the shape description 00220 hid_t shape_id = H5Dget_space(data_id); 00221 // Read the dimensions 00222 hsize_t dim; 00223 H5Sget_simple_extent_dims(shape_id, &dim, NULL); 126

00224 if(dim>len) cgp_doError; 00225 // Zero out the string 00226 memset(value,’ 0’,len+1); \ 00227 // Read the data from the file 00228 herr = H5Dread(data_id, H5T_NATIVE_CHAR, shape_id, shape_id, H5P_DEFAULT, valu e); 00229 if(herr<0) cgp_doError; 00230 // Close the data 00231 herr = H5Dclose(data_id); 00232 if(herr<0) cgp_doError; 00233 // Close the shape 00234 herr = H5Sclose(shape_id); 00235 if(herr<0) cgp_doError; 00236 return 0; 00237 } 00238

00239 int new_str_attb(hid_t pid, const char* name, const char* value, int len) { 00240 herr_t herr; 00241 // Create a shape object for the data 00242 hid_t shape_id = H5Screate(H5S_SCALAR); 00243 if(shape_id<0) cgp_doError; 00244 // Create a type for the data 00245 hid_t type_id = H5Tcopy(H5T_C_S1); 00246 if(type_id<0) cgp_doError; 00247 // Set the dimension of the data to the length of the string, including the te rminator 00248 herr = H5Tset_size(type_id, len+1); 00249 if(herr<0) cgp_doError; 00250 // Create the data in the file 00251 hid_t attb_id = H5Acreate(pid, name, type_id, shape_id, H5P_DEFAULT, H5P_DEFAU LT); 00252 if(attb_id<0) cgp_doError; 00253 // Write the data to the file 00254 herr = H5Awrite(attb_id, type_id, value); 00255 if(herr<0) cgp_doError; 127

00256 // Close the data 00257 herr = H5Aclose(attb_id); 00258 if(herr<0) cgp_doError; 00259 // Close the type 00260 herr = H5Tclose(type_id); 00261 if(herr<0) cgp_doError; 00262 // Close the shape 00263 herr = H5Sclose(shape_id); 00264 if(herr<0) cgp_doError; 00265 return 0; 00266 } 00267 00268 // This probably needs to be fixed to be more like get_str()

00269 int get_str_attb(hid_t pid, const char* name, int len, char* value) { 00270 herr_t herr; 00271 // Create a shape object for the data 00272 hid_t shape_id = H5Screate(H5S_SCALAR); 00273 if(shape_id<0) cgp_doError; 00274 // Create a type for the data 00275 hid_t type_id = H5Tcopy(H5T_C_S1); 00276 if(type_id<0) cgp_doError; 00277 // Set the dimension of the data to the length of the string, including the te rminator 00278 herr = H5Tset_size(type_id, len+1); 00279 if(herr<0) cgp_doError; 00280 // Open the data in the file 00281 hid_t attb_id = H5Aopen(pid, name, H5P_DEFAULT); 00282 if(attb_id<0) cgp_doError; 00283 // Read the data from the file 00284 herr = H5Aread(attb_id, type_id, value); 00285 if(herr<0) cgp_doError; 00286 // Close the data 00287 herr = H5Aclose(attb_id); 00288 if(herr<0) cgp_doError; 00289 // Close the type 128

00290 herr = H5Tclose(type_id); 00291 if(herr<0) cgp_doError; 00292 // Close the shape 00293 herr = H5Sclose(shape_id); 00294 if(herr<0) cgp_doError; 00295 return 0; 00296 } 00297 00298 //======00299

00300 int new_int(hid_t pid, const char* name, const int* value) { 00301 herr_t herr; 00302 // Set the dimension to be one element 00303 hsize_t dim = 1; 00304 // Create a shape object for the data 00305 hid_t shape_id = H5Screate_simple(1,&dim, NULL); 00306 if(shape_id<0) cgp_doError; 00307 // Create the data in the file 00308 hid_t data_id = H5Dcreate2(pid, name, H5T_NATIVE_INT, shape_id, H5P_DEFAULT, H 5P_DEFAULT, H5P_DEFAULT); 00309 if(data_id<0) cgp_doError; 00310 // Write the data to the file 00311 herr = H5Dwrite(data_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, value) ; 00312 if(herr<0) cgp_doError; 00313 // Close the data 00314 herr = H5Dclose(data_id); 00315 if(herr<0) cgp_doError; 00316 // Close the shape 00317 herr = H5Sclose(shape_id); 00318 if(herr<0) cgp_doError; 00319 return 0; 00320 } 00321

00322 int get_int(hid_t pid, const char* name, int* value) { 129

00323 herr_t herr; 00324 // Set the dimension to be one element 00325 hsize_t dim = 1; 00326 // Create a shape object for the data 00327 hid_t shape_id = H5Screate_simple(1,&dim, NULL); 00328 if(shape_id<0) cgp_doError; 00329 // Open the data in the file 00330 hid_t data_id = H5Dopen2(pid, name, H5P_DEFAULT); 00331 if(data_id<0) cgp_doError; 00332 // Read the data from the file 00333 herr = H5Dread(data_id, H5T_NATIVE_INT, shape_id, shape_id, H5P_DEFAULT, value ); 00334 if(herr<0) cgp_doError; 00335 // Close the data 00336 herr = H5Dclose(data_id); 00337 if(herr<0) cgp_doError; 00338 // Close the shape 00339 herr = H5Sclose(shape_id); 00340 if(herr<0) cgp_doError; 00341 return 0; 00342 } 00343

00344 int new_int_attb(hid_t pid, const char* name, const int* value) { 00345 herr_t herr; 00346 // Set the dimension to be one element 00347 hsize_t dim = 1; 00348 // Create a shape object for the data 00349 hid_t shape_id = H5Screate_simple(1,&dim, NULL); 00350 if(shape_id<0) cgp_doError; 00351 // Create the data in the file 00352 hid_t attb_id = H5Acreate(pid, name, H5T_NATIVE_INT, shape_id, H5P_DEFAULT, H5 P_DEFAULT); 00353 if(attb_id<0) cgp_doError; 00354 // Write the data to the file 00355 herr = H5Awrite(attb_id, H5T_NATIVE_INT, value); 130

00356 if(herr<0) cgp_doError; 00357 // Close the data 00358 herr = H5Aclose(attb_id); 00359 if(herr<0) cgp_doError; 00360 // Close the shape 00361 herr = H5Sclose(shape_id); 00362 if(herr<0) cgp_doError; 00363 return 0; 00364 } 00365

00366 int get_int_attb(hid_t pid, const char* name, int* value) { 00367 herr_t herr; 00368 // Set the dimension to be one element 00369 hsize_t dim = 1; 00370 // Create a shape object for the data 00371 hid_t shape_id = H5Screate_simple(1,&dim, NULL); 00372 if(shape_id<0) cgp_doError; 00373 // Open the data in the file 00374 hid_t attb_id = H5Aopen(pid, name, H5P_DEFAULT); 00375 if(attb_id<0) cgp_doError; 00376 // Read the data from the file 00377 herr = H5Aread(attb_id, H5T_NATIVE_INT, value); 00378 if(herr<0) cgp_doError; 00379 // Close the data 00380 herr = H5Aclose(attb_id); 00381 if(herr<0) cgp_doError; 00382 // Close the shape 00383 herr = H5Sclose(shape_id); 00384 if(herr<0) cgp_doError; 00385 return 0; 00386 } 00387 00388 //======00389

00390 int new_float(hid_t pid, const char* name, const float* value) { 131

00391 herr_t herr; 00392 // Set the dimension to be one element 00393 hsize_t dim = 1; 00394 // Create a shape object for the data 00395 hid_t shape_id = H5Screate_simple(1,&dim, NULL); 00396 if(shape_id<0) cgp_doError; 00397 // Create the data in the file 00398 hid_t data_id = H5Dcreate2(pid, name, H5T_NATIVE_FLOAT, shape_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); 00399 if(data_id<0) cgp_doError; 00400 // Write the data to the file 00401 herr = H5Dwrite(data_id, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL, H5P_DEFAULT, valu e); 00402 if(herr<0) cgp_doError; 00403 // Close the data 00404 herr = H5Dclose(data_id); 00405 if(herr<0) cgp_doError; 00406 // Close the shape 00407 herr = H5Sclose(shape_id); 00408 if(herr<0) cgp_doError; 00409 return 0; 00410 } 00411

00412 int get_float(hid_t pid, const char* name, float* value) { 00413 herr_t herr; 00414 // Set the dimension to be one element 00415 hsize_t dim = 1; 00416 // Create a shape object for the data 00417 hid_t shape_id = H5Screate_simple(1,&dim, NULL); 00418 if(shape_id<0) cgp_doError; 00419 // Open the data in the file 00420 hid_t data_id = H5Dopen2(pid, name, H5P_DEFAULT); 00421 if(data_id<0) cgp_doError; 00422 // Read the data from the file 00423 herr = H5Dread(data_id, H5T_NATIVE_FLOAT, shape_id, shape_id, H5P_DEFAULT, val 132

ue); 00424 if(herr<0) cgp_doError; 00425 // Close the data 00426 herr = H5Dclose(data_id); 00427 if(herr<0) cgp_doError; 00428 // Close the shape 00429 herr = H5Sclose(shape_id); 00430 if(herr<0) cgp_doError; 00431 return 0; 00432 } 00433

00434 int new_float_attb(hid_t pid, const char* name, const float* value) { 00435 herr_t herr; 00436 // Set the dimension to be one element 00437 hsize_t dim = 1; 00438 // Create a shape object for the data 00439 hid_t shape_id = H5Screate_simple(1,&dim, NULL); 00440 if(shape_id<0) cgp_doError; 00441 // Create the data in the file 00442 hid_t attb_id = H5Acreate(pid, name, H5T_NATIVE_FLOAT, shape_id, H5P_DEFAULT, H5P_DEFAULT); 00443 if(attb_id<0) cgp_doError; 00444 // Write the data to the file 00445 herr = H5Awrite(attb_id, H5T_NATIVE_FLOAT, value); 00446 if(herr<0) cgp_doError; 00447 // Close the data 00448 herr = H5Aclose(attb_id); 00449 if(herr<0) cgp_doError; 00450 // Close the shape 00451 herr = H5Sclose(shape_id); 00452 if(herr<0) cgp_doError; 00453 return 0; 00454 } 00455

00456 int get_float_attb(hid_t pid, const char* name, float* value) { 133

00457 herr_t herr; 00458 // Set the dimension to be one element 00459 hsize_t dim = 1; 00460 // Create a shape object for the data 00461 hid_t shape_id = H5Screate_simple(1,&dim, NULL); 00462 if(shape_id<0) cgp_doError; 00463 // Open the data in the file 00464 hid_t attb_id = H5Aopen(pid, name, H5P_DEFAULT); 00465 if(attb_id<0) cgp_doError; 00466 // Read the data from the file 00467 herr = H5Aread(attb_id, H5T_NATIVE_FLOAT, value); 00468 if(herr<0) cgp_doError; 00469 // Close the data 00470 herr = H5Aclose(attb_id); 00471 if(herr<0) cgp_doError; 00472 // Close the shape 00473 herr = H5Sclose(shape_id); 00474 if(herr<0) cgp_doError; 00475 return 0; 00476 } 00477 00478 //======00479

00480 int new_node(hid_t pid, const char* name, const char* label, const char* type) { 00481 int err = 0; 00482 herr_t herr; 00483 // Create the node in the file 00484 hid_t group_id = H5Gcreate2(pid, name, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);

00485 if(group_id<0) cgp_doError; 00486 // Set the label attribute; length is 32 for compatability 00487 err = new_str_attb(group_id, "label", label, 32); 00488 if(err!=0) cgp_doError; 00489 // Set the name attribute; length is 32 for compatability 00490 // The name should always correspond to the HDF5 group name 134

00491 err = new_str_attb(group_id, "name", name, 32); 00492 if(err!=0) cgp_doError; 00493 // Set the type attribute; length is 2 for compatability 00494 err = new_str_attb(group_id, "type", type, 2); 00495 if(err!=0) cgp_doError; 00496 // Set the flags attribure 00497 int val = 1; 00498 err = new_int_attb(group_id, "flags", &val); 00499 if(err!=0) cgp_doError; 00500 // Close the group 00501 herr = H5Gclose(group_id); 00502 if(herr<0) cgp_doError; 00503 return 0; 00504 } 00505

00506 int del_node(hid_t pid, const char* name) { 00507 herr_t herr; 00508 // Delete the node 00509 herr = H5Ldelete(pid, name, H5P_DEFAULT); 00510 if(herr<0) cgp_doError; 00511 return 0; 00512 } 00513

00514 int node_exists(hid_t pid, const char* name) { 00515 // Check for the existance of a node 00516 return H5Lexists(pid, name, H5P_DEFAULT)?TRUE:FALSE; 00517 } 00518 00519 //======00520

00522 struct iter_s {int pos; int counter; char* name; char* label;}; 00523

00525 herr_t node_counter(hid_t gid, const char* name, const H5L_info_t* linfo, void* v data) { 00526 int err = 0; 135

00527 herr_t herr; 00528 // Cast vdata as an iter_s

00529 struct iter_s* data = (struct iter_s*) vdata; 00530 // info object to hold data 00531 H5O_info_t info; 00532 // Get the info for the current object 00533 herr = H5Oget_info_by_name(gid, name, &info, H5P_DEFAULT); 00534 if(herr<0) cgp_doError; 00535 // If the object is a group 00536 if(info.type==H5O_TYPE_GROUP) { 00537 // Buffer to store the label 00538 char label[100+1]; 00539 // Open the group 00540 hid_t group_id = H5Gopen2(gid, name, H5P_DEFAULT); 00541 if(group_id<0) cgp_doError; 00542 // Read the label 00543 err = get_str_attb(group_id, "label", 100, label); 00544 if(err!=0) cgp_doError; 00545 // Close the group 00546 herr = H5Gclose(group_id); 00547 if(herr<0) cgp_doError; 00548 // If the label is the same as the given label, increment the counter 00549 if(strcmp(label,data->label)==0) data->counter++; 00550 } 00551 return 0; 00552 } 00553

00554 int num_nodes(hid_t pid, const char* label, int* num) { 00555 herr_t herr; 00556 // An iter_s to hold the iteration results 00557 struct iter_s data; 00558 // Initialize position to be -1 00559 data.pos = -1; 00560 // Initailize the counter to zero 00561 data.counter = 0; 136

00562 // Point the label to the given label

00563 data.label = (char*) label; 00564 // Iteration though the children of pid 00565 herr = H5Literate(pid, H5_INDEX_NAME, H5_ITER_INC, NULL, node_counter, &data);

00566 if(herr<0) cgp_doError; 00567 // Return the number of children whos label’s match label

00568 *num = data.counter; 00569 return 0; 00570 } 00571

00573 herr_t node_idx_finder(hid_t gid, const char* name, const H5L_info_t* linfo, void * vdata) { 00574 int err = 0; 00575 herr_t herr; 00576 // Cast vdata as an iter_s

00577 struct iter_s* data = (struct iter_s*) vdata; 00578 // Info for each object 00579 H5O_info_t info; 00580 // Get the info for the current object 00581 herr = H5Oget_info_by_name(gid, name, &info, H5P_DEFAULT); 00582 if(herr<0) cgp_doError; 00583 // If the object is a group 00584 if(info.type==H5O_TYPE_GROUP) { 00585 // Buffer for label 00586 char label[100+1]; 00587 // Buffer for name 00588 char nname[100+1]; 00589 // Open the group 00590 hid_t group_id = H5Gopen2(gid, name, H5P_DEFAULT); 00591 if(group_id<0) cgp_doError; 00592 // Read the label 00593 err = get_str_attb(group_id, "label", 100, label); 00594 if(err!=0) cgp_doError; 00595 // Read the name 137

00596 err = get_str_attb(group_id, "name", 100, nname); 00597 if(err!=0) cgp_doError; 00598 // Close the group 00599 herr = H5Gclose(group_id); 00600 if(herr<0) cgp_doError; 00601 // If the labels are the same 00602 if(strcmp(label,data->label)==0) { 00603 // If the names are the same, set the position to the counter 00604 if(strcmp(nname,data->name)==0) data->pos=data->counter; 00605 // Increment the counter 00606 data->counter++; 00607 } 00608 } 00609 return 0; 00610 } 00611

00612 int node_name2idx(hid_t pid, const char* label, const char* name, int* idx) { 00613 herr_t herr; 00614 // Data for iteration 00615 struct iter_s data; 00616 // Initialize position 00617 data.pos = -1; 00618 // Initialize counter 00619 data.counter = 0; 00620 // Point the label to the given label

00621 data.label = (char*) label; 00622 // Point the name to the given name

00623 data.name = (char*) name; 00624 // Iterate though all objects in this group 00625 herr = H5Literate(pid, H5_INDEX_NAME, H5_ITER_INC, NULL, node_idx_finder, &dat a); 00626 if(herr<0) cgp_doError; 00627 // Set the index to pos

00628 *idx = data.pos; 00629 return 0; 138

00630 } 00631

00633 herr_t node_name_finder(hid_t gid, const char* name, const H5L_info_t* linfo, voi d* vdata) { 00634 int err = 0; 00635 herr_t herr; 00636 // Cast vdata to an iter_s object

00637 struct iter_s* data = (struct iter_s*) vdata; 00638 // Info about the object 00639 H5O_info_t info; 00640 // Read the info 00641 herr = H5Oget_info_by_name(gid, name, &info, H5P_DEFAULT); 00642 if(herr<0) cgp_doError; 00643 // If the object is a group 00644 if(info.type==H5O_TYPE_GROUP) { 00645 // Buffer for the label 00646 char label[100+1]; 00647 // Buffer for the name 00648 char nname[100+1]; 00649 // Open the group 00650 hid_t group_id = H5Gopen2(gid, name, H5P_DEFAULT); 00651 if(group_id<0) cgp_doError; 00652 // Read the label 00653 err = get_str_attb(group_id, "label", 100, label); 00654 if(err!=0) cgp_doError; 00655 // Read the name 00656 err = get_str_attb(group_id, "name", 100, nname); 00657 if(err!=0) cgp_doError; 00658 // Close the group 00659 herr = H5Gclose(group_id); 00660 if(herr<0) cgp_doError; 00661 // If the label is the same as the given label 00662 if(strcmp(label,data->label)==0) { 00663 // If the counter is the same as the given coutner, copy the name 00664 if(data->counter==data->pos) strcpy(data->name, nname); 139

00665 // Increment the counter 00666 data->counter++; 00667 } 00668 } 00669 return 0; 00670 } 00671

00672 int node_idx2name(hid_t pid, const char* label, int idx, char* name) { 00673 herr_t herr; 00674 // Data for iteration 00675 struct iter_s data; 00676 // Initalize the position 00677 data.pos = idx; 00678 // Initialize the counter 00679 data.counter = 0; 00680 // Point the label

00681 data.label = (char*) label; 00682 // Point the name 00683 data.name = name; 00684 // Iterate through the objects 00685 herr = H5Literate(pid, H5_INDEX_NAME, H5_ITER_INC, NULL, node_name_finder, &da ta); 00686 if(herr<0) cgp_doError; 00687 return 0; 00688 } 00689 00690 //======00691

00692 int hdf5_version_str(int len, char* buf) { 00693 herr_t herr; 00694 // Three components of the HDF5 version 00695 unsigned int maj, min, rel; 00696 // Read the version 00697 herr = H5get_libversion(&maj, &min, &rel); 00698 if(herr<0) cgp_doError; 140

00699 // Preset the buffer to zeros 00700 memset(buf,’ 0’,len+1); \ 00701 // Write the verions string to the buffer 00702 sprintf(buf, "HDF5 Version %d.%d.%d", maj, min, rel); 00703 return 0; 00704 } 00705

00706 int hdf5_format_str(int len, char* buf) { 00707 herr_t herr; 00708 // Get a copy of the native float type 00709 hid_t type_id = H5Tcopy(H5T_NATIVE_FLOAT); 00710 if(type_id<0) cgp_doError; 00711 // Preset the buffer to zeros 00712 memset(buf,’ 0’,len+1); \ 00713 // Write the appropriate formate string to the buffer 00714 if (H5Tequal(type_id, H5T_IEEE_F32BE)) strcpy(buf, "IEEE_BIG_32"); 00715 else if (H5Tequal(type_id, H5T_IEEE_F32LE)) strcpy(buf, "IEEE_LITTLE_32"); 00716 else if (H5Tequal(type_id, H5T_IEEE_F64BE)) strcpy(buf, "IEEE_BIG_64"); 00717 else if (H5Tequal(type_id, H5T_IEEE_F64LE)) strcpy(buf, "IEEE_LITTLE_64"); 00718 else sprintf(buf, "NATIVE_%d", H5Tget_precision(type_id)); 00719 // Close the type 00720 herr = H5Tclose(type_id); 00721 if(herr<0) cgp_doError; 00722 return 0; 00723 }

A.4.7 pcgns_util.h File Reference #include "pcgnslib.h" 141

Include dependency graph for pcgns_util.h:

pcgns_util.h

pcgnslib.h

mpi.h hdf5.h

This graph shows which files directly or indirectly include this file:

pcgns_util.h

pcgns_util.c pcgnslib.c

Data Structures

• struct coords_s

Struct to describe a coords node in the file.

• struct section_s

Struct to describe a section node in the file.

• struct sol_s

Struct to describe a solution node in the file.

• struct zone_s

Struct to describe a zone node in the file.

• struct base_s

Struct to describe a base node in the file. 142

• struct file_s

Struct to describe a CGNS file.

• struct slice_s

Defines

• #define TRUE 1

• #define FALSE 0

Typedefs

• typedef struct coords_s coords_t

Struct to describe a coords node in the file.

• typedef struct section_s section_t

Struct to describe a section node in the file.

• typedef struct sol_s sol_t

Struct to describe a solution node in the file.

• typedef struct zone_s zone_t

Struct to describe a zone node in the file.

• typedef struct base_s base_t

Struct to describe a base node in the file.

• typedef struct file_s file_t 143

Struct to describe a CGNS file.

• typedef struct slice_s slice_t

Functions

• int next_file (int fn) ∗ • int free_file (file_t file) ∗ • int free_base (base_t base) ∗ • int free_zone (zone_t zone) ∗ • int free_coords (coords_t coords) ∗ • int free_section (section_t section) ∗ • int free_sol (sol_t sol) ∗ • int new_str (hid_t pid, const char name, const char value, int len) ∗ ∗ • int get_str (hid_t pid, const char name, int len, char value) ∗ ∗ • int new_str_attb (hid_t pid, const char name, const char value, int len) ∗ ∗ • int get_str_attb (hid_t pid, const char name, int len, char value) ∗ ∗ • int new_int (hid_t pid, const char name, const int value) ∗ ∗ • int get_int (hid_t pid, const char name, int value) ∗ ∗ • int new_int_attb (hid_t pid, const char name, const int value) ∗ ∗ • int get_int_attb (hid_t pid, const char name, int value) ∗ ∗ • int new_float (hid_t pid, const char name, const float value) ∗ ∗ • int get_float (hid_t pid, const char name, float value) ∗ ∗ • int new_float_attb (hid_t pid, const char name, const float value) ∗ ∗ • int get_float_attb (hid_t pid, const char name, float value) ∗ ∗ • int new_node (hid_t pid, const char name, const char label, const char type) ∗ ∗ ∗ • int del_node (hid_t pid, const char name) ∗ • int node_exists (hid_t pid, const char name) ∗ 144

• int num_nodes (hid_t pid, const char label, int num) ∗ ∗ • int node_name2idx (hid_t pid, const char label, const char name, int idx) ∗ ∗ ∗ • int node_idx2name (hid_t pid, const char label, int idx, char name) ∗ ∗ • int hdf5_version_str (int len, char buf) ∗ • int hdf5_format_str (int len, char buf) ∗

Variables

• file_t files ∗ Array of file_t’s which decribe the open files.

• int files_count

Internal count of used slots in {files}.

• int files_size

Internal count of the slots in {files}.

• slice_t write_queue ∗ Queue of IO write operations.

• int write_queue_len

Length of write queue.

A.4.7.1 Detailed Description Author:

Kyle Horne {[email protected]} Version:

0.2 145

A.4.7.2 LICENSE BSD style license

A.4.7.3 DESCRIPTION Header file for utility functions

Definition in file pcgns_util.h.

A.4.7.4 Define Documentation

A.4.7.4.1 #define FALSE 0

Definition at line 21 of file pcgns_util.h.

A.4.7.4.2 #define TRUE 1

Definition at line 17 of file pcgns_util.h.

A.4.7.5 Typedef Documentation

A.4.7.5.1 typedef struct base_s base_t

Struct to describe a base node in the file.

A.4.7.5.2 typedef struct coords_s coords_t

Struct to describe a coords node in the file.

A.4.7.5.3 typedef struct file_s file_t 146

Struct to describe a CGNS file.

A.4.7.5.4 typedef struct section_s section_t

Struct to describe a section node in the file.

A.4.7.5.5 typedef struct slice_s slice_t

A.4.7.5.6 typedef struct sol_s sol_t

Struct to describe a solution node in the file.

A.4.7.5.7 typedef struct zone_s zone_t

Struct to describe a zone node in the file.

A.4.7.6 Function Documentation

A.4.7.6.1 int del_node (hid_t pid, const char name) ∗

Delete a node with parent {pid} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of node

Returns:

Error code

Definition at line 506 of file pcgns_util.c. 147

Here is the caller graph for this function:

cgp_array_write main

cgp_sol_write

cgp_base_write

del_node cgp_close

main cgp_coord_write

cgp_zone_write

cgp_section_write main

A.4.7.6.2 int free_base (base_t base) ∗

Free the memory of a base_t object

Parameters:

base [in]: Pointer the base

Returns:

Error code

Definition at line 103 of file pcgns_util.c.

Here is the call graph for this function:

free_coord

free_base free_zone free_section

free_sol

Here is the caller graph for this function:

cgp_base_write

free_base main free_file cgp_close 148

A.4.7.6.3 int free_coords (coords_t coords) ∗

Free the memory of a coords_t object Parameters:

coords [in]: Pointer the coords

Returns:

Error code

A.4.7.6.4 int free_file (file_t file) ∗

Free the memory of a file_t object Parameters:

file [in]: Pointer the file

Returns:

Error code

Definition at line 69 of file pcgns_util.c.

Here is the call graph for this function:

free_coord

free_file free_base free_zone free_section

free_sol

Here is the caller graph for this function:

free_file cgp_close main

A.4.7.6.5 int free_section (section_t section) ∗

Free the memory of a section_t object 149 Parameters:

section [in]: Pointer the section

Returns:

Error code

Definition at line 163 of file pcgns_util.c.

Here is the caller graph for this function:

cgp_zone_write free_section free_zone free_base cgp_base_write main

free_file cgp_close

A.4.7.6.6 int free_sol (sol_t sol) ∗

Free the memory of a sol_t object Parameters:

sol [in]: Pointer the sol

Returns:

Error code

Definition at line 179 of file pcgns_util.c.

Here is the caller graph for this function:

cgp_zone_write free_sol free_zone free_base cgp_base_write main

free_file cgp_close

A.4.7.6.7 int free_zone (zone_t zone) ∗

Free the memory of a zone_t object 150 Parameters:

zone [in]: Pointer the zone

Returns:

Error code

Definition at line 121 of file pcgns_util.c.

Here is the call graph for this function:

free_coord

free_zone free_section

free_sol

Here is the caller graph for this function:

cgp_zone_write free_zone free_base cgp_base_write main

free_file cgp_close

A.4.7.6.8 int get_float (hid_t pid, const char name, float value) ∗ ∗

Read an float with parent {pid}, name {name}, and value {value} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of float

value [out]: Float to read

Returns:

Error code

Definition at line 412 of file pcgns_util.c. 151

A.4.7.6.9 int get_float_attb (hid_t pid, const char name, float value) ∗ ∗

Read an float attribute at location {pid} with {name} with {value} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of float

value [out]: Float to read

Returns:

Error code

Definition at line 456 of file pcgns_util.c.

A.4.7.6.10 int get_int (hid_t pid, const char name, int value) ∗ ∗

Read an integer with parent {pid}, name {name}, and value {value} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of integer

value [out]: Integer to read

Returns:

Error code

Definition at line 322 of file pcgns_util.c.

A.4.7.6.11 int get_int_attb (hid_t pid, const char name, int value) ∗ ∗

Read an integer attribute at location {pid} with {name} with {value} Parameters:

pid [in]: HDF5 locator for parent 152

name [in]: Name of integer

value [out]: Integer to read

Returns:

Error code

Definition at line 366 of file pcgns_util.c.

A.4.7.6.12 int get_str (hid_t pid, const char name, int len, char value) ∗ ∗

Read a string with parent {pid}, name {name}, and value {value} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of string

len [in]: Length of string in file

value [out]: Contents of string

Returns:

Error code

Definition at line 214 of file pcgns_util.c.

Here is the caller graph for this function:

get_str cgp_zone_read main

A.4.7.6.13 int get_str_attb (hid_t pid, const char name, int len, char value) ∗ ∗

Read a string attribute at location {pid} with {name} with {value} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of string 153

len [in]: Length of string in file

value [out]: Contents of string

Returns:

Error code

Definition at line 269 of file pcgns_util.c.

Here is the caller graph for this function:

cgp_base_read main

node_name_finder node_idx2name cgp_open main

get_str_attb node_counter num_nodes cgp_zone_read main

node_idx_finder node_name2idx

A.4.7.6.14 int hdf5_format_str (int len, char buf) ∗

Fill a buffer with the HDF5 format string Parameters:

len [in]: Length of buf

buf [out]: String with the HDF5 format

Returns:

Error code

Definition at line 706 of file pcgns_util.c.

Here is the caller graph for this function:

hdf5_format_str cgp_open main

A.4.7.6.15 int hdf5_version_str (int len, char buf) ∗

Fill a buffer with the HDF5 version number 154 Parameters:

len [in]: Length of buf

buf [out]: String with HDF5 version number

Returns:

Error code

Definition at line 692 of file pcgns_util.c.

Here is the caller graph for this function:

hdf5_version_str cgp_open main

A.4.7.6.16 int new_float (hid_t pid, const char name, const float value) ∗ ∗

Create a new float with parent {pid}, name {name}, and value {value} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of float

value [in]: Float to write

Returns:

Error code

Definition at line 390 of file pcgns_util.c.

Here is the caller graph for this function:

new_float cgp_open main

A.4.7.6.17 int new_float_attb (hid_t pid, const char name, const float value) ∗ ∗

Create a new float attribute at location {pid} with {name} with {value} 155 Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of float

value [in]: Float to write Returns:

Error code

Definition at line 434 of file pcgns_util.c.

A.4.7.6.18 int new_int (hid_t pid, const char name, const int value) ∗ ∗

Create a new integer with parent {pid}, name {name}, and value {value} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of integer

value [in]: Integer to write

Returns:

Error code

Definition at line 300 of file pcgns_util.c.

Here is the caller graph for this function:

new_int cgp_open main

A.4.7.6.19 int new_int_attb (hid_t pid, const char name, const int value) ∗ ∗

Create a new integer attribute at location {pid} with {name} with {value} Parameters:

pid [in]: HDF5 locator for parent 156

name [in]: Name of integer

value [in]: Integer to write

Returns:

Error code

Definition at line 344 of file pcgns_util.c.

Here is the caller graph for this function:

cgp_array_write main

cgp_sol_write

cgp_base_write

cgp_coord_write

new_int_attb new_node main

cgp_open

cgp_zone_write

cgp_section_write main

cgp_zone_read main

A.4.7.6.20 int new_node (hid_t pid, const char name, const char label, const char ∗ ∗ ∗ type)

Create a new node at with parent {pid} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of node

label [in]: Label of node

type [in]: Type of node

Returns:

Error code 157

Definition at line 480 of file pcgns_util.c.

Here is the call graph for this function:

new_int_attb

new_node

new_str_attb

Here is the caller graph for this function:

cgp_array_write main

cgp_sol_write

cgp_base_write

cgp_coord_write new_node main cgp_open

cgp_zone_write

cgp_section_write main

cgp_zone_read main

A.4.7.6.21 int new_str (hid_t pid, const char name, const char value, int len) ∗ ∗

Create a new string with parent {pid}, name {name}, and value {value} Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of string

value [in]: Contents of string

len [in]: Length of string in file

Returns:

Error code

Definition at line 192 of file pcgns_util.c. 158

Here is the caller graph for this function:

cgp_open

new_str main cgp_zone_write

A.4.7.6.22 int new_str_attb (hid_t pid, const char name, const char value, int len) ∗ ∗

Create a new string attribute at location {pid} with {name} with {value}

Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of string

value [in]: Contents of string

len [in]: Length of string in file

Returns:

Error code

Definition at line 239 of file pcgns_util.c.

Here is the caller graph for this function:

cgp_open

cgp_base_write main

new_str_attb cgp_coord_write

new_node cgp_zone_write

cgp_array_write main cgp_sol_write

cgp_section_write main

cgp_zone_read main 159

A.4.7.6.23 int next_file (int fn) ∗

Return the number of the next availible file_t in {files}

Parameters:

fn [out]: Index of next file

Returns:

Error code

Definition at line 37 of file pcgns_util.c.

Here is the call graph for this function:

next_file cleanup_files

Here is the caller graph for this function:

next_file cgp_open main

A.4.7.6.24 int node_exists (hid_t pid, const char name) ∗

Check if a node exists with name {name} and parent {pid}

Parameters:

pid [in]: HDF5 locator for parent

name [in]: Name of node

Returns:

{1=Exists, 0=Does not exist}

Definition at line 514 of file pcgns_util.c. 160

Here is the caller graph for this function:

cgp_array_write

main cgp_sol_write

cgp_base_write

node_exists cgp_coord_write main

cgp_zone_write

cgp_section_write main

cgp_zone_read main

A.4.7.6.25 int node_idx2name (hid_t pid, const char label, int idx, char name) ∗ ∗

Convert a node id to a node name Parameters:

pid [in]: HDF5 locator for parent

label [in]: Label of node

idx [in]: Index of node

name [out]: Name of node

Definition at line 672 of file pcgns_util.c.

Here is the call graph for this function:

node_idx2name node_name_finder get_str_attb

Here is the caller graph for this function:

cgp_base_read main

node_idx2name cgp_open main

cgp_zone_read main 161

A.4.7.6.26 int node_name2idx (hid_t pid, const char label, const char name, int idx) ∗ ∗ ∗

Convert a node name to a node id Parameters:

pid [in]: HDF5 locator for parent

label [in]: Label of node

name [in]: Name of node

idx [out]: Index of node Returns:

Error code

Definition at line 612 of file pcgns_util.c.

Here is the call graph for this function:

node_name2idx node_idx_finder get_str_attb

A.4.7.6.27 int num_nodes (hid_t pid, const char label, int num) ∗ ∗

Count the number of nodes of type {label} with parent {pid} Parameters:

pid [in]: HDF5 locator for parent

label [in]: Label of nodes

num [out]: Number of nodes Returns:

Error code

Definition at line 554 of file pcgns_util.c.

Here is the call graph for this function:

num_nodes node_counter get_str_attb 162

Here is the caller graph for this function:

cgp_base_read main

num_nodes cgp_open main

cgp_zone_read main

A.4.7.7 Variable Documentation

A.4.7.7.1 file_t files ∗

Array of file_t’s which decribe the open files.

Definition at line 21 of file pcgns_util.c.

A.4.7.7.2 int files_count

Internal count of used slots in {files}.

Definition at line 22 of file pcgns_util.c.

A.4.7.7.3 int files_size

Internal count of the slots in {files}.

Definition at line 23 of file pcgns_util.c.

A.4.7.7.4 slice_t write_queue ∗

Queue of IO write operations.

Definition at line 25 of file pcgns_util.c. 163

A.4.7.7.5 int write_queue_len

Length of write queue.

Definition at line 26 of file pcgns_util.c.

A.4.8 pcgns_util.h

00001 00002 00003 00004 00005 00006 00007 00008 00009 00010 00011 #ifndef PCGNS_UTIL_H_ 00012 #define PCGNS_UTIL_H_ 00013 00014 #include "pcgnslib.h" 00015 00016 #ifndef TRUE 00017 #define TRUE 1 00018 #endif 00019 00020 #ifndef FALSE 00021 #define FALSE 0 00022 #endif 00023 00024 00025 //======// 00026 //== Begin Datatypes ==// 00027 //======// 164

00028 00030 typedef struct coords_s { 00032 int idx; 00034 char coordname[100+1]; 00036 hid_t group_id; 00037 } coords_t; 00038 00040 typedef struct section_s { 00042 int idx; 00044 char sectionname[100+1]; 00046 hid_t group_id; 00048 ElementType_t type; 00050 int range[2]; 00052 hid_t connectivity_id; 00054 hid_t range_id; 00055 } section_t; 00056 00058 typedef struct sol_s { 00060 int idx; 00062 char solname[100+1]; 00064 hid_t group_id; 00065 } sol_t; 00066 00068 typedef struct zone_s { 00070 int idx; 00072 char zonename[100+1];

00074 coords_t* coords; 00076 int ncoords;

00078 section_t* sections; 00080 int nsections;

00082 sol_t* sols; 00084 int nsols; 00086 hid_t group_id;

00088 int* nijk; 00090 ZoneType_t type; 165

00092 hid_t grid_id; 00094 hid_t flow_id; 00095 } zone_t; 00096 00098 typedef struct base_s { 00100 int idx; 00102 char basename[100+1];

00104 zone_t *zones; 00106 int nzones; 00108 hid_t group_id; 00110 int cell_dim; 00112 int phys_dim; 00113 } base_t; 00114 00116 typedef struct file_s { 00118 int idx; 00120 char filename[100+1]; 00122 int isOpen;

00124 base_t* bases; 00126 int nbases; 00128 hid_t file_id; 00130 hid_t plist_id; 00132 MPI_Comm comm; 00134 MPI_Info info; 00136 int rank; 00138 int size; 00139 } file_t; 00140 00141 typedef struct slice_s { 00142 SliceType_t type; 00143 int rank;

00144 int* min; 00145 int* max; 00146 void* data; 00147 int F; 166

00148 int B; 00149 int Z; 00150 int Selector; 00151 char name[100+1]; 00152 } slice_t; 00153 00154 //======// 00155 //== Begin Global Data Prototypes ==// 00156 //======// 00157

00159 extern file_t* files; 00160 00162 extern int files_count; 00163 00165 extern int files_size; 00166

00168 extern slice_t* write_queue; 00169 00171 extern int write_queue_len; 00172 00173 //======// 00174 //== Begin Function Prototypes ==// 00175 //======// 00176

00180 int next_file(int* fn); 00181

00185 int free_file(file_t* file); 00186

00190 int free_base(base_t* base); 00191

00195 int free_zone(zone_t* zone); 00196

00200 int free_coords(coords_t* coords); 00201

00205 int free_section(section_t* section); 167

00206

00210 int free_sol(sol_t* sol); 00211 00212 //======00213

00220 int new_str(hid_t pid, const char* name, const char* value, int len); 00221

00228 int get_str(hid_t pid, const char* name, int len, char* value); 00229

00236 int new_str_attb(hid_t pid, const char* name, const char* value, int len); 00237

00244 int get_str_attb(hid_t pid, const char* name, int len, char* value); 00245 00246 //======00247

00253 int new_int(hid_t pid, const char* name, const int* value); 00254

00260 int get_int(hid_t pid, const char* name, int* value); 00261

00267 int new_int_attb(hid_t pid, const char* name, const int* value); 00268

00274 int get_int_attb(hid_t pid, const char* name, int* value); 00275 00276 //======00277

00283 int new_float(hid_t pid, const char* name, const float* value); 00284

00290 int get_float(hid_t pid, const char* name, float* value); 00291

00297 int new_float_attb(hid_t pid, const char* name, const float* value); 00298

00304 int get_float_attb(hid_t pid, const char* name, float* value); 00305 00306 //======00307 168

00314 int new_node(hid_t pid, const char* name, const char* label, const char* type); 00315

00320 int del_node(hid_t pid, const char* name); 00321

00326 int node_exists(hid_t pid, const char* name); 00327 00328 //======00329

00335 int num_nodes(hid_t pid, const char* label, int* num); 00336

00343 int node_name2idx(hid_t pid, const char* label, const char* name, int* idx); 00344

00350 int node_idx2name(hid_t pid, const char* label, int idx, char* name); 00351 00352 //======00353

00358 int hdf5_version_str(int len, char* buf); 00359

00364 int hdf5_format_str(int len, char* buf); 00365 00366 #endif

A.4.9 pcgnslib.c File Reference #include "pcgnslib.h"

#include "pcgns_util.h"

#include "stdio.h"

#include "stdlib.h"

#include "string.h" 169

Include dependency graph for pcgnslib.c:

pcgnslib.c

pcgns_util.h stdio.h stdlib.h string.h

pcgnslib.h

mpi.h hdf5.h

Functions

• int cgp_open (const char filename, int mode, MPI_Comm comm, MPI_Info info, int fn) ∗ ∗ ∗ • int cgp_close (int fn)

• int cgp_base_read (int fn, int B, char basename, int cell_dim, int phys_dim) ∗ ∗ ∗ • int cgp_base_write (int fn, char const basename, int cell_dim, int phys_dim, int B) ∗ ∗ • int cgp_nbases (int fn, int nbases) ∗ • int cgp_zone_read (int fn, int B, int Z, char zonename, int nijk) ∗ ∗ • int cgp_zone_type (int fn, int B, int Z, ZoneType_t zonetype) ∗ • int cgp_zone_write (int fn, int B, const char zonename, const int nijk, ZoneType_t type, int ∗ ∗ Z) ∗ • int cgp_nzones (int fn, int B, int nzones) ∗ • int cgp_coord_write (int fn, int B, int Z, DataType_t type, const char coordname, int C) ∗ ∗ • int cgp_coord_write_data (int fn, int B, int Z, int C, int min, int max, void coord_array) ∗ ∗ ∗ • int cgp_sol_write_data (int fn, int B, int Z, int S, int min, int max, void data) ∗ ∗ ∗ • int cgp_sol_write (int fn, int B, int Z, char solname, GridLocation_t location, int S) ∗ ∗ • int cgp_nsols (int fn, int B, int Z, int nsols) ∗ • int cgp_section_write (int fn, int B, int Z, char sectionname, ElementType_t type, int start, ∗ int end, int nbndry, int S) ∗ • int cgp_section_write_data (int fn, int B, int Z, int S, int min, int max, int elements) ∗ 170

• int cgp_array_write (int fn, int B, int Z, char arrayname, GridLocation_t location) ∗ • int cgp_array_write_data (int fn, int B, int Z, char arrayname, int min, int max, void data) ∗ ∗ ∗ ∗ • int queue_slice_write (SliceType_t type, int F, int B, int Z, void SN, int rank, int min, int ∗ ∗ max, void data) ∗ ∗ • int queue_flush (void)

Variables

• int node_counts [24]

• int preallocate = 0

A.4.9.1 Detailed Description Author:

Kyle Horne Version:

0.2

A.4.9.2 LICENSE BSD style license

A.4.9.3 DESCRIPTION Implimentation of functions provided by the pcgns library

Definition in file pcgnslib.c.

A.4.9.4 Function Documentation

A.4.9.4.1 int cgp_array_write (int fn, int B, int Z, char arrayname, GridLocation_t ∗ location)

Write an array to a zone 171 Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

arrayname [in]: Name of array

location [in]: Location of solution within each cell

Returns:

Error code

Definition at line 1191 of file pcgnslib.c.

Here is the call graph for this function:

del_node new_int_attb cgp_array_write new_node new_str_attb node_exists

Here is the caller graph for this function:

cgp_array_write main

A.4.9.4.2 int cgp_array_write_data (int fn, int B, int Z, char arrayname, int min, int ∗ ∗ ∗ max, void data) ∗

Write an array’s data in parallel Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone 172

arrayname [in]: Name of array

min [in]: Lower bound array for data

max [in]: Upper bound array for data

data [in]: Data to be written Returns:

Error code

Definition at line 1251 of file pcgnslib.c.

Here is the caller graph for this function:

main

cgp_array_write_data queue_flush main

A.4.9.4.3 int cgp_base_read (int fn, int B, char basename, int cell_dim, int phys_dim) ∗ ∗ ∗

Read info about a base Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

basename [out]: Name of the base

cell_dim [out]: Cell dimensions of the base

phys_dim [out]: Physical dimensions of the base Returns:

Error code

Definition at line 169 of file pcgnslib.c.

Here is the call graph for this function:

node_idx2name node_name_finder cgp_base_read get_str_attb

num_nodes node_counter 173

Here is the caller graph for this function:

cgp_base_read main

A.4.9.4.4 int cgp_base_write (int fn, char const basename, int cell_dim, int phys_dim, int ∗ B) ∗

Write a base to a file Parameters:

fn int[in]: Handle f the file

basename [in]: Name of the base to write

cell_dim [in]: Cell dimensions of the base

phys_dim [in]: Physical dimensions of the base

B [out]: Index of the base

Returns:

Error code

Definition at line 234 of file pcgnslib.c.

Here is the call graph for this function:

del_node free_coord

free_base free_zone free_section

cgp_base_write

new_node new_int_attb free_sol

node_exists new_str_attb

Here is the caller graph for this function:

cgp_base_write main 174

A.4.9.4.5 int cgp_close (int fn)

Close a previously opened file Parameters:

fn [in]: Handle of the file to close

Returns:

Error code

Definition at line 159 of file pcgnslib.c.

Here is the call graph for this function:

del_node free_coord

cgp_close free_file free_base free_zone free_section

free_sol

Here is the caller graph for this function:

cgp_close main

A.4.9.4.6 int cgp_coord_write (int fn, int B, int Z, DataType_t type, const char ∗ coordname, int C) ∗

Write coords group, but not data, to a grid Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

datatype [in]: Type of floats stored

coordname [in]: Name of the coords 175

C [out]: Index of the coords

Definition at line 667 of file pcgnslib.c.

Here is the call graph for this function:

del_node new_int_attb cgp_coord_write new_node new_str_attb node_exists

Here is the caller graph for this function:

cgp_coord_write main

A.4.9.4.7 int cgp_coord_write_data (int fn, int B, int Z, int C, int min, int max, void ∗ ∗ ∗ coord_array)

Write coords to a grid in parallel Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

C [in]: Index of the coords

range_min [in]: Array of lower bound index

range_max [in]: Array of upper bound index

coord_array [in]: Pointer to the data

Returns:

Error code

Definition at line 750 of file pcgnslib.c. 176

Here is the caller graph for this function:

main cgp_coord_write_data queue_flush main

A.4.9.4.8 int cgp_nbases (int fn, int nbases) ∗

Read the number of bases in a file Parameters:

fn [in]: Handle of the file

nbases [out]: Number of bases in the specified file

Returns:

Error code

Definition at line 318 of file pcgnslib.c.

A.4.9.4.9 int cgp_nsols (int fn, int B, int Z, int nsols) ∗

Read the number of solutions in a zone Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

nsols [out]: Number of solutions in the specified zone

Returns:

Error code

Definition at line 960 of file pcgnslib.c. 177

A.4.9.4.10 int cgp_nzones (int fn, int B, int nzones) ∗

Read the number of zones in a base

Parameters:

fn [in]: Handle of file

B [in]: Index of base

nzones [out]: Number of zones in the specified base

Returns:

Error code

Definition at line 660 of file pcgnslib.c.

A.4.9.4.11 int cgp_open (const char filename, int mode, MPI_Comm comm, MPI_Info ∗ ∗ info, int fn) ∗

Open a file for reading and writing

Parameters:

filename [in]: Name of the file to open

mode [in]: IO mode (read/write)

comm [in]: MPI communicator on which to open the file

info [in]: MPI info object to allow hints passed to MPI-IO

fn [out]: Handle of the opened file

Returns:

Error code

Definition at line 42 of file pcgnslib.c. 178

Here is the call graph for this function:

hdf5_format_str

hdf5_version_str

new_float

new_int

new_node new_int_attb cgp_open new_str_attb

new_str

next_file cleanup_files

node_idx2name node_name_finder get_str_attb

num_nodes node_counter

Here is the caller graph for this function:

cgp_open main

A.4.9.4.12 int cgp_section_write (int fn, int B, int Z, char sectionname, ElementType_t ∗ type, int start, int end, int nbndry, int S) ∗

Write the element connectivity groups for a section Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

C [in]: Index of the coords

sectionname [in]: Name of element section

type [in]: Type of element data

start [in]: Element lower bound index 179

end [in]: Element upper bound index

nbndry [in]: Number of boundary elements (unused)

S [out]: Section index

Returns:

Error code

Definition at line 976 of file pcgnslib.c.

Here is the call graph for this function:

del_node

new_int_attb

cgp_section_write new_node

new_str_attb node_exists

Here is the caller graph for this function:

cgp_section_write main

A.4.9.4.13 int cgp_section_write_data (int fn, int B, int Z, int S, int min, int max, int ∗ elements)

Write the element connectivity data for a section Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

C [in]: Index of the coords

S [in]: Section index

min [in]: Output array lower bound index 180

max [in]: Output array upper bound index

elements [in]: Pointer to the data

Returns:

Error code

Definition at line 1126 of file pcgnslib.c.

Here is the caller graph for this function:

main cgp_section_write_data queue_flush main

A.4.9.4.14 int cgp_sol_write (int fn, int B, int Z, char solname, GridLocation_t location, ∗ int S) ∗

Write a solution to a zone

Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

solname [in]: Name of solution

location [in]: Location of solution within each cell

S [out]: Index of solution

Returns:

Error code

Definition at line 876 of file pcgnslib.c. 181

Here is the call graph for this function:

del_node new_int_attb cgp_sol_write new_node new_str_attb node_exists

Here is the caller graph for this function:

cgp_sol_write main

A.4.9.4.15 int cgp_sol_write_data (int fn, int B, int Z, int S, int min, int max, void ∗ ∗ ∗ data)

Write a solution’s data in parallel Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

S [in]: Index of soltution

min [in]: Lower bound array for data

max [in]: Upper bound array for data

data [in]: Data to be written Returns:

Error code

Definition at line 813 of file pcgnslib.c.

Here is the caller graph for this function:

main cgp_sol_write_data queue_flush main 182

A.4.9.4.16 int cgp_zone_read (int fn, int B, int Z, char zonename, int nijk) ∗ ∗

Read info about a zone Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone to read

zonename [out]: Name of the zone

nijk [out]: Dimensions of the zone Returns:

Error code

Definition at line 325 of file pcgnslib.c.

Here is the call graph for this function:

get_str

new_node new_int_attb

cgp_zone_read node_exists new_str_attb

node_idx2name node_name_finder get_str_attb num_nodes node_counter

Here is the caller graph for this function:

cgp_zone_read main

A.4.9.4.17 int cgp_zone_type (int fn, int B, int Z, ZoneType_t zonetype) ∗

Read the type of a zone Parameters:

fn [in]: Handle of the file 183

B [in]: Index of the base

Z [in]: Index of the zone

zonetype [out]: Type of zone

Returns:

Error code

Definition at line 535 of file pcgnslib.c.

A.4.9.4.18 int cgp_zone_write (int fn, int B, const char zonename, const int nijk, ∗ ∗ ZoneType_t type, int Z) ∗

Write a zone to a base

Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

zonename [in]: Name of the zone to write

nijk [in]: Dimensions of the zone

type [in]: Type of zone

Z [out]: Index of the zone

Returns:

Error code

Definition at line 544 of file pcgnslib.c. 184

Here is the call graph for this function:

free_coord

del_node free_section

free_zone free_sol

cgp_zone_write new_node new_int_attb

new_str new_str_attb

node_exists

Here is the caller graph for this function:

cgp_zone_write main

A.4.9.4.19 int queue_flush (void)

Flush all the IO operations waiting in the queue

Returns:

Error code

Definition at line 1344 of file pcgnslib.c.

Here is the call graph for this function:

cgp_array_write_data

cgp_coord_write_data queue_flush cgp_section_write_data

cgp_sol_write_data

Here is the caller graph for this function:

queue_flush main 185

A.4.9.4.20 int queue_slice_write (SliceType_t type, int F, int B, int Z, void SN, int rank, ∗ int min, int max, void data) ∗ ∗ ∗

Queue an IO write operation for flushing later Parameters:

type [in]: Type of operation to queue

F [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

SN [in]: Pointer to array locator, which is an int for coordinates, solutions and sections, but a string for arrays

rank [in]: Rank of data to be written

min [in]: Pointer to the minumum location array

max [in]: Pointer to the maximum location array

data [in]: Pointer to the data to be written

Returns:

Error code

Definition at line 1315 of file pcgnslib.c.

Here is the caller graph for this function:

queue_slice_write main

A.4.9.5 Variable Documentation

A.4.9.5.1 int node_counts[24]

Initial value: 186

{ -1, -1, 1, 2,3, 3,6, 4,8,9, 4,10, 5,13,14, 6,15,18, 8,20,27, -1, -1,-1 }

Definition at line 21 of file pcgnslib.c.

A.4.9.5.2 int preallocate = 0

Definition at line 35 of file pcgnslib.c.

A.4.10 pcgnslib.c

00001 00002 00003 00004 00005 00006 00007 00008 00009 00010 00011 #include "pcgnslib.h" 00012 #include "pcgns_util.h" 00013 187

00014 #include "stdio.h" 00015 #include "stdlib.h" 00016 #include "string.h" 00017 00018 // Number of nodes per element 00019 // Index corresponds to the ElementType_t enum 00020 // Negative numbers indicate non-supported types 00021 int node_counts[24] = { 00022 -1, -1, 00023 1, 00024 2,3, 00025 3,6, 00026 4,8,9, 00027 4,10, 00028 5,13,14, 00029 6,15,18, 00030 8,20,27, 00031 -1, 00032 -1,-1 00033 }; 00034 00035 int preallocate = 0; 00036 00037 //======// 00038 //== Begin Function Definitions ==// 00039 //======// 00040 00041 //= File IO Prototypes =//

00042 int cgp_open(const char* filename, int mode, MPI_Comm comm, MPI_Info* info, int* fn) { 00043 int err; 00044 herr_t herr; 00045 // Test file to check for file’s existance

00046 FILE* test_file = fopen(filename, "rb"); 00047 // Flag for file’s existance 188

00048 int file_exists = (test_file==NULL)?FALSE:TRUE; 00049 // Flag for file’s HDF5 status 00050 int file_isHDF5 = FALSE; 00051 // If the file exists 00052 if(file_exists) { 00053 // Close the file 00054 fclose(test_file); 00055 // Test to see if it is an HDF5 file 00056 file_isHDF5 = H5Fis_hdf5(filename)?TRUE:FALSE; 00057 } 00058 // Create a new file reference in the global files array 00059 next_file(fn); 00060 // Create a pointer to this file in that array

00061 file_t* file = &(files[*fn]); 00062 // Set the file’s index

00063 file->idx = *fn; 00064 // Set the file’s state to open 00065 file->isOpen = TRUE; 00066 // Set the file’s name 00067 strcpy(file->filename,filename); 00068 // Set the file’s communicator 00069 file->comm = comm; 00070 // Set the MPI_Info for the file

00071 err = MPI_Info_dup(*info,&(file->info)); 00072 if(err!=MPI_SUCCESS) cgp_doError; 00073 // Set the rank on this communicator 00074 err = MPI_Comm_rank(comm,&(file->rank)); 00075 if(err!=MPI_SUCCESS) cgp_doError; 00076 // Set the size of this communicator 00077 err = MPI_Comm_size(comm,&(file->size)); 00078 if(err!=MPI_SUCCESS) cgp_doError; 00079 // Set the access property list 00080 file->plist_id = H5Pcreate(H5P_FILE_ACCESS); 00081 if(file->plist_id<0) cgp_doError; 00082 // Set the access property list to use MPI 189

00083 herr = H5Pset_fapl_mpio(file->plist_id, file->comm, file->info); 00084 if(herr<0) cgp_doError; 00085 // If the file exists and it is an HDF5 file, try to read is as CGNS 00086 if(file_exists&&file_isHDF5) { 00087 // Open the file with HDF5 and set the file_id 00088 file->file_id = H5Fopen(filename, H5F_ACC_RDWR, file->plist_id); 00089 if(file->file_id<0) cgp_doError; 00090 // Read the number of bases in the file 00091 num_nodes(file->file_id, "CGNSBase_t",&(file->nbases)); 00092 // Allocate space to store the bases

00093 file->bases = malloc(file->nbases*sizeof(base_t)); 00094 if(file->bases==NULL) cgp_doError; 00095 // Loop counter 00096 int k; 00097 // For each base in bases... 00098 for(k=0;knbases;k++) { 00099 // Buffer to store the name of the base 00100 char basename[100+1]; 00101 // Find the name of the k’th base in bases 00102 node_idx2name(file->file_id, "CGNSBase_t", k, basename); 00103 // Open that base 00104 file->bases[k].group_id = H5Gopen2(file->file_id, basename, H5P_DEFAULT) ; 00105 if(file->bases[k].group_id<0) cgp_doError; 00106 // Set the base’s index 00107 file->bases[k].idx = k; 00108 // Copy the base’s name 00109 strcpy(file->bases[k].basename,basename); 00110 // Iniialize it to have no zones 00111 file->bases[k].zones = NULL; 00112 file->bases[k].nzones = 0; 00113 } 00114 } 00115 // If the file does not exist, create an empty CGNS file 00116 else if((!file_exists)||(!file_isHDF5)) { 190

00117 // Create a new HDF5 file 00118 file->file_id = H5Fcreate(filename, H5F_ACC_TRUNC, H5P_DEFAULT, file-> plist_id); 00119 if(file->file_id<0) cgp_doError; 00120 // Create the needed attributes to describe the root node 00121 new_str_attb(file->file_id, "label", "Root Node of ADF File", 32); 00122 new_str_attb(file->file_id, "name", "HDF5 MotherNode", 32); 00123 new_str_attb(file->file_id, "type", "MT", 2); 00124 // Buffer for version information 00125 char version[100+1]; 00126 // Buffer for format information 00127 char format[100+1]; 00128 // Get the HDF5 format string 00129 hdf5_format_str(100,format); 00130 // Get the HDF5 version string 00131 hdf5_version_str(100,version); 00132 // Write the HDF5 format string to the file 00133 new_str(file->file_id, " format", format, strlen(format)); 00134 // Write the HDF5 version string to the file 00135 new_str(file->file_id, " hdf5version", version, 32); 00136 // Create a CGNS library version number 00137 new_node(file->file_id, "CGNSLibraryVersion", "CGNSLibraryVersion_t", "R4") ; 00138 // Open that version number group 00139 hid_t group_id = H5Gopen2(file->file_id, "CGNSLibraryVersion", H5P_DEFAULT) ; 00140 if(group_id<0) cgp_doError; 00141 // Set the files to be compatible with CGNS version 3.0 00142 float libversion = 3.0; 00143 // Write the version number to the file 00144 new_float(group_id, " data", &libversion); 00145 // Close the version number group 00146 herr = H5Gclose(group_id); 00147 if(herr<0) cgp_doError; 00148 191

00149 // set the file to have no bases 00150 file->bases = NULL; 00151 file->nbases = 0; 00152 } 00153 int dummy = 0; 00154 err = new_int(file->file_id, " dummy",&dummy); 00155 if(err!=0) cgp_doError; 00156 return 0; 00157 } 00158 00159 int cgp_close(int fn) { 00160 // Free the file referenced by fn 00161 printTime; 00162 del_node(files[fn].file_id, " dummy"); 00163 free_file(&(files[fn])); 00164 printTime; 00165 return 0; 00166 } 00167 00168 //= Base IO Prototypes =//

00169 int cgp_base_read(int fn, int B, char* basename, int* cell_dim, int* phys_dim) { 00170 if(fn>=files_count||fn<0) cgp_doError; 00171 if(!files[fn].isOpen) cgp_doError; 00172 if(B>files[fn].nbases||B<=0) cgp_doError; 00173 herr_t herr; 00174 // Pointer to the base being read

00175 base_t* base = &(files[fn].bases[B-1]); 00176 // Get the name of the base 00177 strcpy(basename, base->basename); 00178 00179 // Size of array to read 00180 hsize_t dim = 2; 00181 // Create shape object for shape of array 00182 hid_t shape_id = H5Screate_simple(1,&dim,NULL); 00183 if(shape_id<0) cgp_doError; 192

00184 // Open the data array in the file 00185 hid_t data_id = H5Dopen2(base->group_id, " data", H5P_DEFAULT); 00186 if(data_id<0) cgp_doError; 00187 // Buffer to store the data 00188 int data[2]; 00189 // Read the data array condaining this base’s dimensions 00190 herr = H5Dread(data_id, H5T_NATIVE_INT, shape_id, shape_id, H5P_DEFAULT, data) ; 00191 if(herr<0) cgp_doError; 00192 // Copy the data out of the array [The order here needs verification]

00193 *cell_dim = data[0]; 00194 base->cell_dim = data[0];

00195 *phys_dim = data[1]; 00196 base->phys_dim = data[1]; 00197 // Close the data array in the file 00198 herr = H5Dclose(data_id); 00199 if(herr<0) cgp_doError; 00200 // Destroy the shape object 00201 herr = H5Sclose(shape_id); 00202 if(herr<0) cgp_doError; 00203 00204 // Read Zone info 00205 // Read the number of zones in the base 00206 num_nodes(base->group_id, "Zone_t", &(base->nzones)); 00207 // Allocate space for zone descriptors

00208 base->zones = (zone_t*) malloc(base->nzones*sizeof(zone_t)); 00209 if(base->zones==NULL) cgp_doError; 00210 // Loop index 00211 int k; 00212 // For each zone in zones 00213 for(k=0;knzones;k++) { 00214 // Buffer to store the zone’s anme 00215 char zonename[100+1]; 00216 // Get the name of the k’th zone 00217 node_idx2name(base->group_id, "Zone_t", k, zonename); 193

00218 // Open the group for this zone 00219 base->zones[k].group_id = H5Gopen2(base->group_id,zonename,H5P_DEFAULT); 00220 if(base->zones[k].group_id<0) cgp_doError; 00221 // Set the zone’s index 00222 base->zones[k].idx = k; 00223 // Copy the zone’s name 00224 strcpy(base->zones[k].zonename,zonename); 00225 // Default to no coords or solutions 00226 base->zones[k].coords = NULL; 00227 base->zones[k].ncoords = 0; 00228 base->zones[k].sols = NULL; 00229 base->zones[k].nsols = 0; 00230 } 00231 return 0; 00232 } 00233

00234 int cgp_base_write(int fn, char const* basename, int cell_dim, int phys_dim, int* B) { 00235 if(fn>=files_count||fn<0) cgp_doError; 00236 if(!files[fn].isOpen) cgp_doError; 00237 herr_t herr; 00238 // Pointer to this file

00239 file_t* file = &(files[fn]); 00240 // Index 00241 int idx; 00242 // If this base already exists, replace the old one 00243 if(node_exists(file->file_id, basename)) { 00244 // Loop index 00245 int k; 00246 // Find this base in the list of bases for this file 00247 for(k=0;knbases;k++) if(strcmp(basename,file->bases[k].basename)==0) idx = k; 00248 // Free the old base in memory 00249 free_base(&(file->bases[idx])); 00250 // Delete the old base in the file 194

00251 del_node(file->file_id, basename); 00252 } 00253 // If this base does not yet exist, create one 00254 else { 00255 // Loop index 00256 int k; 00257 // Allocate space for a longer list of bases

00258 base_t* bases = malloc((file->nbases+1)*sizeof(base_t)); 00259 if(bases==NULL) cgp_doError; 00260 // Copy all the old bases to the new list 00261 for(k=0;knbases;k++) bases[k] = file->bases[k]; 00262 // Free the old list 00263 if(file->bases!=NULL) free(file->bases); 00264 // Point the file to use the new list 00265 file->bases = bases; 00266 // Increment the number of bases in the file 00267 file->nbases++; 00268 // Set the index to the last base in the list 00269 idx = file->nbases-1; 00270 } 00271 // Pointer to the current base

00272 base_t* base = &(file->bases[idx]); 00273 // Copy the name of the base 00274 strcpy(file->bases[idx].basename, basename); 00275 // Set the index of the base 00276 base->idx = idx; 00277 // Create no zones in the base 00278 base->zones = NULL; 00279 base->nzones = 0; 00280 // Set the base’s dimensions 00281 base->cell_dim = cell_dim; 00282 base->phys_dim = phys_dim; 00283 // Return the index of the base

00284 *B = idx+1; 00285 // Create the base in the file 195

00286 new_node(file->file_id, basename, "CGNSBase_t", "I4"); 00287 // Open the base 00288 hid_t group_id = H5Gopen2(file->file_id, basename, H5P_DEFAULT); 00289 if(group_id<0) cgp_doError; 00290 // Copy the group_id for later use 00291 file->bases[idx].group_id = group_id; 00292 // Set the size of the array to write 00293 hsize_t dim = 2; 00294 // Create a shape object for the array to write 00295 hid_t shape_id = H5Screate_simple(1,&dim,NULL); 00296 if(shape_id<0) cgp_doError; 00297 // Create the data in the file 00298 hid_t data_id = H5Dcreate2(group_id, " data", H5T_NATIVE_INT, shape_id, H5P_DE FAULT, H5P_DEFAULT, H5P_DEFAULT); 00299 if(data_id<0) cgp_doError; 00300 // Buffer to hold that data array 00301 int data[2]; 00302 // Fill the buffer [Order needs verification] 00303 data[0] = cell_dim; 00304 data[1] = phys_dim; 00305 // Write the data array to the file 00306 herr = H5Dwrite(data_id, H5T_NATIVE_INT, shape_id, shape_id, H5P_DEFAULT, data ); 00307 if(herr<0) cgp_doError; 00308 // Close the data array 00309 herr = H5Dclose(data_id); 00310 if(herr<0) cgp_doError; 00311 // Destroy the shape object 00312 herr = H5Sclose(shape_id); 00313 if(herr<0) cgp_doError; 00314 00315 return 0; 00316 } 00317

00318 int cgp_nbases(int fn, int *nbases) { 196

00319 // Read the number of bases in this file

00320 *nbases = files[fn].nbases; 00321 return 0; 00322 } 00323 00324 //= Zone IO Prototypes =//

00325 int cgp_zone_read(int fn, int B, int Z, char* zonename, int* nijk) { 00326 if(fn>=files_count||fn<0) cgp_doError; 00327 if(!files[fn].isOpen) cgp_doError; 00328 if(B>files[fn].nbases||B<=0) cgp_doError; 00329 if(Z>files[fn].bases[B-1].nzones||Z<=0) cgp_doError; 00330 herr_t herr; 00331 int err; 00332 // Pointer to the current base

00333 base_t* base = &(files[fn].bases[B-1]); 00334 // Pointer to the zone to be read

00335 zone_t* zone = &(files[fn].bases[B-1].zones[Z-1]); 00336 // Copy the zone’s name 00337 strcpy(zonename, zone->zonename); 00338 // Loop index 00339 int k; 00340 // If the zone already has a coordinates sub-group, read it 00341 if(node_exists(zone->group_id, "GridCoordinates")) { 00342 // Test if the existing HDF5 handle for the grid_id is valid 00343 // If it is, use it, otherwise open a new handle 00344 if(!H5Iis_valid(zone->grid_id)) zone->grid_id = H5Gopen2(zone->group_id, "G ridCoordinates", H5P_DEFAULT); 00345 if(zone->grid_id<0) cgp_doError; 00346 // Count the number of data arrays in the group 00347 num_nodes(zone->grid_id, "DataArray_t", &(zone->ncoords)); 00348 // Allocate memory to describe the data arrays

00349 zone->coords = (coords_t*) malloc(zone->ncoords*sizeof(coords_t)); 00350 if(zone->coords==NULL) cgp_doError; 00351 // For each ncoord in ncoords 00352 for(k=0;kncoords;k++) { 197

00353 // Buffer for the name of the coords 00354 char coordname[100+1]; 00355 // Read the name of the coords 00356 node_idx2name(zone->grid_id, "DataArray_t", k, coordname); 00357 // Open the data array 00358 zone->coords[k].group_id = H5Gopen2(zone->grid_id, coordname, H5P_DEFAUL T); 00359 if(zone->coords[k].group_id<0) cgp_doError; 00360 // Set the index of the data 00361 zone->coords[k].idx = k; 00362 // Copy the name of the coords 00363 strcpy(zone->coords[k].coordname,coordname); 00364 } 00365 } 00366 // If the zone does not have a coordinates sub-group, create one for it 00367 else { 00368 // Create a new group to hold the coords 00369 new_node(zone->group_id, "GridCoordinates", "GridCoordinates_t", "MT"); 00370 // Open the new group 00371 zone->grid_id = H5Gopen2(zone->group_id, "GridCoordinates", H5P_DEFAULT); 00372 if(zone->grid_id<0) cgp_doError; 00373 // Default to hold no coords data 00374 zone->ncoords = 0; 00375 zone->coords = NULL; 00376 } 00377 00378 // If the zone already has a flow sub-group, read it 00379 if(node_exists(zone->group_id, "GridCoordinates")) { 00380 // Test if the existing HDF5 handle for the flow_id is valid 00381 // If it is, use it, otherwise open a new handle 00382 if(!H5Iis_valid(zone->flow_id)) zone->flow_id = H5Gopen2(zone->flow_id, "Fl owSolution", H5P_DEFAULT); 00383 if(zone->flow_id<0) cgp_doError; 00384 // Count the number of data arrays in the group 00385 num_nodes(zone->flow_id, "DataArray_t", &(zone->ncoords)); 198

00386 // Allocate memory to describe the data arrays

00387 zone->sols = (sol_t*) malloc(zone->nsols*sizeof(sol_t)); 00388 if(zone->sols==NULL) cgp_doError; 00389 // For each sol in nsols 00390 for(k=0;knsols;k++) { 00391 // Buffer for the name of the coords 00392 char solname[100+1]; 00393 // Read the name of the coords 00394 node_idx2name(zone->flow_id, "DataArray_t", k, solname); 00395 // Open the data array 00396 zone->sols[k].group_id = H5Gopen2(zone->flow_id, solname, H5P_DEFAULT); 00397 if(zone->sols[k].group_id<0) cgp_doError; 00398 // Set the index of the data 00399 zone->sols[k].idx = k; 00400 // Copy the name of the coords 00401 strcpy(zone->sols[k].solname,solname); 00402 } 00403 } 00404 // If the zone does not have a flow sub-group, create one for it 00405 else { 00406 // Create a new group to hold the coords 00407 new_node(zone->group_id, "FlowSolution", "FlowSolution_t", "MT"); 00408 // Open the new group 00409 zone->flow_id = H5Gopen2(zone->group_id, "FlowSolution", H5P_DEFAULT); 00410 if(zone->flow_id<0) cgp_doError; 00411 // Default to hold no coords data 00412 zone->nsols = 0; 00413 zone->sols = NULL; 00414 } 00415 00416 // Count the number of element sections 00417 num_nodes(zone->group_id, "Elements_t", &(zone->nsections)); 00418 // Allocate memory to describe the element sections

00419 zone->sections = (section_t*) malloc(zone->nsections*sizeof(section_t)); 00420 // For each section in nsections 199

00421 for(k=0;knsections;k++) { 00422 // Buffer for the section’s name 00423 char sectionname[100+1]; 00424 // Read the name of the k’th section 00425 node_idx2name(zone->group_id, "Elements_t", k, sectionname); 00426 // Open the section 00427 zone->sections[k].group_id = H5Gopen2(zone->group_id, sectionname, H5P_DEFA ULT); 00428 if(zone->sections[k].group_id<0) cgp_doError; 00429 // Open the section’s connectivity 00430 zone->sections[k].connectivity_id = H5Gopen2(zone->sections[k].group_id, "E lementConnectivity", H5P_DEFAULT); 00431 if(zone->sections[k].connectivity_id<0) cgp_doError; 00432 // Open the section’s ranges 00433 zone->sections[k].range_id = H5Gopen2(zone->sections[k].group_id, "ElementR ange", H5P_DEFAULT); 00434 if(zone->sections[k].range_id<0) cgp_doError; 00435 // Set the index of the section 00436 zone->sections[k].idx = k; 00437 // Copy the name of the section 00438 strcpy(zone->sections[k].sectionname,sectionname); 00439 // Read the element type 00440 { 00441 // Dimensions of data 00442 hsize_t dims[1] = {2}; 00443 // Shape of data in memory and the file 00444 hid_t shape_id = H5Screate_simple(1,dims,NULL); 00445 // Open the array 00446 hid_t data_id = H5Dopen2(zone->sections[k].group_id, " data", H5P_DEFAUL T); 00447 // Buffer to hold results 00448 int data[2]; 00449 // Read array 00450 H5Dread(data_id,H5T_NATIVE_INT, shape_id,shape_id,H5P_DEFAULT,data); 00451 // Set the type 200

00452 zone->sections[k].type = data[0]; 00453 // Clsoe the array 00454 H5Dclose(data_id); 00455 // Close the shape 00456 H5Sclose(shape_id); 00457 } 00458 // Read the element range 00459 { 00460 // Dimensions of data 00461 hsize_t dims[1] = {2}; 00462 // Shape of data in memory and the file 00463 hid_t shape_id = H5Screate_simple(1,dims,NULL); 00464 // Open the array 00465 hid_t data_id = H5Dopen2(zone->sections[k].range_id, " data", H5P_DEFAUL T); 00466 // Buffer to hold results 00467 int data[2]; 00468 // Read array 00469 H5Dread(data_id,H5T_NATIVE_INT, shape_id,shape_id,H5P_DEFAULT,data); 00470 // Set the range 00471 zone->sections[k].range[0] = data[0]; 00472 zone->sections[k].range[1] = data[1]; 00473 // Clsoe the array 00474 H5Dclose(data_id); 00475 // Close the shape 00476 H5Sclose(shape_id); 00477 } 00478 } 00479 00480 // Count the number of flow solutions 00481 num_nodes(zone->group_id, "FlowSolution_t", &(zone->nsols)); 00482 // Allocate memory to describe the flow solutions

00483 zone->sols = (sol_t*) malloc(zone->nsols*sizeof(sol_t)); 00484 if(zone->sols==NULL) cgp_doError; 00485 // For each solution in nsols 201

00486 for(k=0;knsols;k++) { 00487 // Buffer for the solution’s name 00488 char solname[100+1]; 00489 // Read the name of the k’th solution 00490 node_idx2name(zone->group_id, "FlowSolution_t", k, solname); 00491 // Open the solution 00492 zone->sols[k].group_id = H5Gopen2(zone->group_id, solname, H5P_DEFAULT); 00493 if(zone->sols[k].group_id<0) cgp_doError; 00494 // Set the index of the solution 00495 zone->sols[k].idx = k; 00496 // Copy the name of the solution 00497 strcpy(zone->sols[k].solname,solname); 00498 } 00499 00500 // Read the zone type 00501 char ztype[100]; 00502 err = get_str(zone->group_id, "ZoneType/ data", 99, ztype); 00503 if(strcmp(ztype,"Structured")) zone->type = Structured; 00504 else if(strcmp(ztype,"Untructured")) zone->type = Unstructured; 00505 else cgp_doError; 00506 00507 // Read nijk 00508 // Set the size of the array to read 00509 int cols = (zone->type==Structured)?base->cell_dim:1; 00510 hsize_t dim[2] = {3, cols}; 00511 // Create the shape object for the array 00512 hid_t shape_id = H5Screate_simple(2,dim, NULL); 00513 if(shape_id<0) cgp_doError; 00514 // Open the array in the file 00515 hid_t data_id = H5Dopen2(zone->group_id, " data", H5P_DEFAULT); 00516 if(data_id<0) cgp_doError; 00517 // Read the array from the file to the local nijk 00518 herr = H5Dread(data_id, H5T_NATIVE_INT, shape_id, shape_id, H5P_DEFAULT, nijk) ; 00519 if(herr<0) cgp_doError; 202

00520 // Allocate space to store the data

00521 zone->nijk = malloc(3*cols*sizeof(int)); 00522 if(nijk==NULL) cgp_doError; 00523 // Read the array from the file to the global nijk 00524 herr = H5Dread(data_id, H5T_NATIVE_INT, shape_id, shape_id, H5P_DEFAULT, zone- >nijk); 00525 if(herr<0) cgp_doError; 00526 // Close the array 00527 herr = H5Dclose(data_id); 00528 if(herr<0) cgp_doError; 00529 // Destroy the shape 00530 herr = H5Sclose(shape_id); 00531 if(herr<0) cgp_doError; 00532 return 0; 00533 } 00534

00535 int cgp_zone_type(int fn, int B, int Z, ZoneType_t *zonetype) { 00536 // Read Zonetype from file and return in enum

00537 file_t* file = &(files[fn]); 00538 base_t* base = &(file->bases[B-1]); 00539 zone_t* zone = &(base->zones[Z-1]); 00540 *zonetype = zone->type; 00541 return 0; 00542 } 00543

00544 int cgp_zone_write(int fn, int B, const char* zonename, const int* nijk, ZoneType_t type, int* Z) { 00545 if(fn>=files_count||fn<0) cgp_doError; 00546 if(!files[fn].isOpen) cgp_doError; 00547 if(B>files[fn].nbases||B<=0) cgp_doError; 00548 herr_t herr; 00549 // Pointer to the base

00550 base_t* base = &(files[fn].bases[B-1]); 00551 // Index 00552 int idx; 203

00553 // Loop index 00554 int k; 00555 // If the zone exists, replace it 00556 if(node_exists(base->group_id, zonename)) { 00557 // Find the zone in this base 00558 for(k=0;knzones;k++) if(strcmp(zonename,base->zones[k].zonename)==0) idx = k; 00559 // Free the memory for the zone 00560 free_zone(&(base->zones[idx])); 00561 // Delete the zone from the file 00562 del_node(base->group_id,zonename); 00563 } 00564 // If the zone does not exist, initialize memory for it 00565 else { 00566 // Allocate a bigger list of zones

00567 zone_t* zones = (zone_t*) malloc((base->nzones+1)*sizeof(zone_t)); 00568 if(zones==NULL) cgp_doError; 00569 // Copy the old zones to the new list 00570 for(k=0;knzones;k++) zones[k] = base->zones[k]; 00571 // Deallocate the old list 00572 if(base->zones!=NULL) free(base->zones); 00573 // Point the base’s list to the new list 00574 base->zones = zones; 00575 // Increment the number of zones 00576 base->nzones++; 00577 // Set the index of the zone 00578 idx = base->nzones-1; 00579 } 00580 // Pointer to the current zone

00581 zone_t* zone= &(base->zones[idx]); 00582 // Copy the name of the zone 00583 strcpy(zone->zonename, zonename); 00584 // Set the index of the zone 00585 zone->idx = idx; 00586 // Default to no coords or solutions 204

00587 zone->coords = NULL; 00588 zone->ncoords = 0; 00589 zone->sections = NULL; 00590 zone->nsections = 0; 00591 zone->sols = NULL; 00592 zone->nsols = 0; 00593 // Return the index of the zone

00594 *Z = idx+1; 00595 00596 // Create the zone in the file 00597 new_node(base->group_id, zonename, "Zone_t", "I4"); 00598 // Open the new zone 00599 zone->group_id = H5Gopen2(base->group_id, zonename, H5P_DEFAULT); 00600 if(zone->group_id<0) cgp_doError; 00601 00602 // Write ZoneType 00603 zone->type = type; 00604 // Create a new node for the zone type 00605 new_node(zone->group_id, "ZoneType", "ZoneType_t", "C1"); 00606 // Open the new node 00607 hid_t group_id = H5Gopen2(zone->group_id, "ZoneType", H5P_DEFAULT); 00608 if(group_id<0) cgp_doError; 00609 // Write a string as the data for the new node 00610 if(type==Structured) new_str(group_id, " data", "Structured", strlen("Structur ed")-1); 00611 else if(type==Unstructured) new_str(group_id, " data", "Unstructured", strlen( "Unstructured")-1); 00612 else cgp_doError; 00613 // Close the node 00614 herr = H5Gclose(group_id); 00615 if(herr<0) cgp_doError; 00616 00617 // Write nijk 00618 // Size of array to write 00619 int cols = (type==Structured)?base->cell_dim:1; 205

00620 hsize_t dim[2] = {3, cols}; 00621 // Create shape object for array to write 00622 hid_t shape_id = H5Screate_simple(2,dim, NULL); 00623 if(shape_id<0) cgp_doError; 00624 // Create the data array in the file 00625 hid_t data_id = H5Dcreate2(zone->group_id, " data", H5T_NATIVE_INT, shape_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); 00626 if(data_id<0) cgp_doError; 00627 // Write the data to the array 00628 herr = H5Dwrite(data_id, H5T_NATIVE_INT, shape_id, shape_id, H5P_DEFAULT, nijk); 00629 if(herr<0) cgp_doError; 00630 // Allocate space to store the data

00631 zone->nijk = malloc(3*cols*sizeof(int)); 00632 if(zone->nijk==NULL) cgp_doError; 00633 // Read the data to the zone in memory 00634 herr = H5Dread(data_id, H5T_NATIVE_INT, shape_id, shape_id, H5P_DEFAULT, zone- >nijk); 00635 if(herr<0) cgp_doError; 00636 // Close the data array 00637 herr = H5Dclose(data_id); 00638 if(herr<0) cgp_doError; 00639 // Destroy the shape object 00640 herr = H5Sclose(shape_id); 00641 if(herr<0) cgp_doError; 00642 00643 // Write GridCoordinates 00644 // Create a node for the grid coordinates 00645 new_node(zone->group_id, "GridCoordinates", "GridCoordinates_t", "MT"); 00646 // Open the node 00647 zone->grid_id = H5Gopen2(zone->group_id, "GridCoordinates", H5P_DEFAULT); 00648 if(zone->grid_id<0) cgp_doError; 00649 00650 // Write FlowSolution 00651 // Create a node for the grid coordinates 206

00652 new_node(zone->group_id, "FlowSolution", "FlowSolution_t", "MT"); 00653 // Open the node 00654 zone->flow_id = H5Gopen2(zone->group_id, "FlowSolution", H5P_DEFAULT); 00655 if(zone->grid_id<0) cgp_doError; 00656 00657 return 0; 00658 } 00659

00660 int cgp_nzones(int fn, int B, int *nzones) { 00661 // Read the number of zones in this base

00662 *nzones = files[fn].bases[B-1].nzones; 00663 return 0; 00664 } 00665 00666 //= Grid IO Prototypes =//

00667 int cgp_coord_write(int fn, int B, int Z, DataType_t type, const char* coordname, int* C) { 00668 if(fn>=files_count||fn<0) cgp_doError; 00669 if(!files[fn].isOpen) cgp_doError; 00670 if(B>files[fn].nbases||B<=0) cgp_doError; 00671 if(Z>files[fn].bases[B-1].nzones||Z<=0) cgp_doError; 00672 herr_t herr; 00673 // Pointer to the current file

00674 file_t* file = &(files[fn]); 00675 // Pointer to the current base

00676 base_t* base = &(file->bases[B-1]); 00677 // Pointer to the current zone

00678 zone_t* zone = &(base->zones[Z-1]); 00679 // Index 00680 int idx; 00681 // Loop index 00682 int k; 00683 // If the coordinates already exist, replace them 00684 if(node_exists(zone->grid_id, coordname)) { 00685 // Find the coordinates to be replaced 207

00686 for(k=0;kncoords;k++) if(strcmp(coordname,zone->coords[k].coordname) ==0) idx = k; 00687 // Delete them 00688 del_node(zone->grid_id,coordname); 00689 } 00690 // If the coordinates do not exist, prepare the memory structures 00691 else { 00692 // New list of coords

00693 coords_t* coords = (coords_t*) malloc((zone->ncoords+1)*sizeof(coords_t)); 00694 if(coords==NULL) cgp_doError; 00695 // Copy old list to new list 00696 for(k=0;kncoords;k++) coords[k] = zone->coords[k]; 00697 // Free the old list 00698 if(zone->coords!=NULL) free(zone->coords); 00699 // Point the zone to the new list 00700 zone->coords = coords; 00701 // Incement the number of coords 00702 zone->ncoords++; 00703 // Set the coords index 00704 idx = zone->ncoords-1; 00705 } 00706 // Pointer to coords

00707 coords_t* coords = &(zone->coords[idx]); 00708 // Copy the name 00709 strcpy(coords->coordname, coordname); 00710 // Set the index 00711 coords->idx = idx; 00712 // Return the index

00713 *C = idx+1; 00714 00715 // Create a new node in the file 00716 new_node(zone->grid_id, coordname, "DataArray_t", "R8"); 00717 // Open the node 00718 coords->group_id = H5Gopen2(zone->grid_id, coordname, H5P_DEFAULT); 00719 if(coords->group_id<0) cgp_doError; 208

00720 00721 // Set the rank of the dimensions 00722 hsize_t rank = (zone->type==Structured)?base->cell_dim:1; 00723 // Dimensions of the array 00724 hsize_t dims[rank]; 00725 // Set these to correspond with the size of the zone 00726 for(k=0;knijk[(rank-1)-k]; 00727 00728 // Create a shape for the array 00729 hid_t shape_id = H5Screate_simple(rank,dims,NULL); 00730 if(shape_id<0) cgp_doError; 00731 // Property list for dataset 00732 hid_t plist_id = H5Pcreate(H5P_DATASET_CREATE); 00733 if(preallocate) herr = H5Pset_alloc_time(plist_id,H5D_ALLOC_TIME_EARLY); 00734 if(preallocate) herr = H5Pset_fill_time(plist_id, H5D_FILL_TIME_ALLOC); 00735 // Create the array in the file 00736 hid_t data_id = H5Dcreate2(coords->group_id, " data", H5T_NATIVE_DOUBLE, shape _id, H5P_DEFAULT, plist_id, H5P_DEFAULT); 00737 if(data_id<0) cgp_doError; 00738 // Close the array 00739 herr = H5Dclose(data_id); 00740 if(herr<0) cgp_doError; 00741 // Close the property list 00742 herr = H5Pclose(plist_id); 00743 if(herr<0) cgp_doError; 00744 // Close the shape 00745 herr = H5Sclose(shape_id); 00746 if(herr<0) cgp_doError; 00747 return 0; 00748 } 00749

00750 int cgp_coord_write_data(int fn, int B, int Z, int C, int* min, int* max, void* c oord_array) { 00751 int k; 00752 if(fn>=files_count||fn<0) cgp_doError; 209

00753 if(!files[fn].isOpen) cgp_doError; 00754 if(B>files[fn].nbases||B<=0) cgp_doError; 00755 if(Z>files[fn].bases[B-1].nzones||Z<=0) cgp_doError; 00756 if(C>files[fn].bases[B-1].zones[Z-1].ncoords||C<=0) cgp_doError; 00757 herr_t herr; 00758 // Pointer to the current file

00759 file_t* file = &(files[fn]); 00760 // Pointer to the current base

00761 base_t* base = &(file->bases[B-1]); 00762 // Pointer to the current zone

00763 zone_t* zone = &(base->zones[Z-1]); 00764 // Pointer to the current coords

00765 coords_t* coords = &(zone->coords[C-1]); 00766 00767 // Open the data 00768 hid_t data_id = H5Dopen2(coords->group_id, " data", H5P_DEFAULT); 00769 if(data_id<0) cgp_doError; 00770 00771 // Set the rank of the data 00772 hsize_t rank = (zone->type==Structured)?base->cell_dim:1; 00773 // Set the start position for the data write 00774 hsize_t start[rank]; 00775 for(k=0;k

00787 if(herr<0) cgp_doError; 00788 // Set the access property list for data transfer 00789 hid_t plist_id = H5Pcreate(H5P_DATASET_XFER); 00790 if(plist_id<0) cgp_doError; 00791 // Set MPI-IO collective communication 00792 herr = H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_COLLECTIVE); 00793 if(herr<0) cgp_doError; 00794 // Write the data in collective parallel I/O 00795 herr = H5Dwrite(data_id, H5T_NATIVE_DOUBLE, mem_shape_id, data_shape_id, plist _id, coord_array); 00796 if(herr<0) cgp_doError; 00797 // Close the property list 00798 herr = H5Pclose(plist_id); 00799 if(herr<0) cgp_doError; 00800 // Close the shape of the data in the file 00801 herr = H5Sclose(data_shape_id); 00802 if(herr<0) cgp_doError; 00803 // Close the shape of the data in memory 00804 herr = H5Sclose(mem_shape_id); 00805 if(herr<0) cgp_doError; 00806 // Close the data array 00807 herr = H5Dclose(data_id); 00808 if(herr<0) cgp_doError; 00809 00810 return 0; 00811 } 00812

00813 int cgp_sol_write_data(int fn, int B, int Z, int S, int* min, int* max, void* dat a) { 00814 int k; 00815 if(fn>=files_count||fn<0) cgp_doError; 00816 if(!files[fn].isOpen) cgp_doError; 00817 if(B>files[fn].nbases||B<=0) cgp_doError; 00818 if(Z>files[fn].bases[B-1].nzones||Z<=0) cgp_doError; 00819 if(S>files[fn].bases[B-1].zones[Z-1].nsols||S<=0) cgp_doError; 211

00820 herr_t herr; 00821 // Pointer to the current file

00822 file_t* file = &(files[fn]); 00823 // Pointer to the current base

00824 base_t* base = &(file->bases[B-1]); 00825 // Pointer to the current zone

00826 zone_t* zone = &(base->zones[Z-1]); 00827 // Pointer to the current coords

00828 sol_t* sol = &(zone->sols[S-1]); 00829 00830 // Open the data 00831 hid_t data_id = H5Dopen2(sol->group_id, " data", H5P_DEFAULT); 00832 if(data_id<0) cgp_doError; 00833 00834 // Set the rank of the data 00835 hsize_t rank = (zone->type==Structured)?base->cell_dim:1; 00836 // Set the start position for the data write 00837 hsize_t start[rank]; 00838 for(k=0;k

00854 // Set MPI-IO collective communication 00855 herr = H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_COLLECTIVE); 00856 if(herr<0) cgp_doError; 00857 // Write the data in collective parallel I/O 00858 herr = H5Dwrite(data_id, H5T_NATIVE_DOUBLE, mem_shape_id, data_shape_id, plist _id, data); 00859 if(herr<0) cgp_doError; 00860 // Close the property list 00861 herr = H5Pclose(plist_id); 00862 if(herr<0) cgp_doError; 00863 // Close the shape of the data in the file 00864 herr = H5Sclose(data_shape_id); 00865 if(herr<0) cgp_doError; 00866 // Close the shape of the data in memory 00867 herr = H5Sclose(mem_shape_id); 00868 if(herr<0) cgp_doError; 00869 // Close the data array 00870 herr = H5Dclose(data_id); 00871 if(herr<0) cgp_doError; 00872 00873 return 0; 00874 } 00875

00876 int cgp_sol_write(int fn, int B, int Z, char *solname, GridLocation_t location, i nt *S) { 00877 if(fn>=files_count||fn<0) cgp_doError; 00878 if(!files[fn].isOpen) cgp_doError; 00879 if(B>files[fn].nbases||B<=0) cgp_doError; 00880 if(Z>files[fn].bases[B-1].nzones||Z<=0) cgp_doError; 00881 herr_t herr; 00882 // Pointer to the current file

00883 file_t* file = &(files[fn]); 00884 // Pointer to the current base

00885 base_t* base = &(file->bases[B-1]); 00886 // Pointer to the current zone 213

00887 zone_t* zone = &(base->zones[Z-1]); 00888 // Index 00889 int idx; 00890 // Loop index 00891 int k; 00892 // If the solution already exist, replace it 00893 if(node_exists(zone->flow_id, solname)) { 00894 // Find the coordinates to be replaced 00895 for(k=0;knsols;k++) if(strcmp(solname,zone->sols[k].solname)==0) idx = k; 00896 // Delete them 00897 del_node(zone->flow_id,solname); 00898 } 00899 // If the coordinates do not exist, prepare the memory structures 00900 else { 00901 // New list of coords

00902 sol_t* sols = (sol_t*) malloc((zone->nsols+1)*sizeof(sol_t)); 00903 if(sols==NULL) cgp_doError; 00904 // Copy old list to new list 00905 for(k=0;knsols;k++) sols[k] = zone->sols[k]; 00906 // Free the old list 00907 if(zone->sols!=NULL) free(zone->sols); 00908 // Point the zone to the new list 00909 zone->sols = sols; 00910 // Incement the number of coords 00911 zone->nsols++; 00912 // Set the coords index 00913 idx = zone->nsols-1; 00914 } 00915 // Pointer to coords

00916 sol_t* sol = &(zone->sols[idx]); 00917 // Copy the name 00918 strcpy(sol->solname, solname); 00919 // Set the index 00920 sol->idx = idx; 214

00921 // Return the index

00922 *S = idx+1; 00923 00924 // Create a new node in the file 00925 new_node(zone->flow_id, solname, "DataArray_t", "R8"); 00926 // Open the node 00927 sol->group_id = H5Gopen2(zone->flow_id, solname, H5P_DEFAULT); 00928 if(sol->group_id<0) cgp_doError; 00929 00930 // Set the rank of the dimensions 00931 hsize_t rank = (zone->type==Structured)?base->cell_dim:1; 00932 // Dimensions of the array 00933 hsize_t dims[rank]; 00934 // Set these to correspond with the size of the zone 00935 if(location==Vertex) for(k=0;knijk[(rank-1)-k]; 00936 else if(location==CellCenter) for(k=0;knijk[rank+(r ank-1)-k]; 00937 00938 // Create a shape for the array 00939 hid_t shape_id = H5Screate_simple(rank,dims,NULL); 00940 if(shape_id<0) cgp_doError; 00941 // Property list for dataset 00942 hid_t plist_id = H5Pcreate(H5P_DATASET_CREATE); 00943 if(preallocate) herr = H5Pset_alloc_time(plist_id,H5D_ALLOC_TIME_EARLY); 00944 if(preallocate) herr = H5Pset_fill_time(plist_id, H5D_FILL_TIME_ALLOC); 00945 // Create the array in the file 00946 hid_t data_id = H5Dcreate2(sol->group_id, " data", H5T_NATIVE_DOUBLE, shape_id , H5P_DEFAULT, plist_id, H5P_DEFAULT); 00947 if(data_id<0) cgp_doError; 00948 // Close the array 00949 herr = H5Dclose(data_id); 00950 if(herr<0) cgp_doError; 00951 // Close the property list 00952 herr = H5Pclose(plist_id); 00953 if(herr<0) cgp_doError; 215

00954 // Close the shape 00955 herr = H5Sclose(shape_id); 00956 if(herr<0) cgp_doError; 00957 return 0; 00958 } 00959

00960 int cgp_nsols(int fn, int B, int Z, int* nsols) { 00961 if(fn>=files_count||fn<0) cgp_doError; 00962 if(!files[fn].isOpen) cgp_doError; 00963 if(B>files[fn].nbases||B<=0) cgp_doError; 00964 if(Z>files[fn].bases[B-1].nzones||Z<=0) cgp_doError; 00965 herr_t herr; 00966 // Pointer to the current file

00967 file_t* file = &(files[fn]); 00968 // Pointer to the current base

00969 base_t* base = &(file->bases[B-1]); 00970 // Pointer to the current zone

00971 zone_t* zone = &(base->zones[Z-1]); 00972 *nsols = zone->nsols; 00973 return 0; 00974 } 00975

00976 int cgp_section_write(int fn, int B, int Z, char* sectionname, ElementType_t type ,

00977 int start, int end, int nbndry, int* S) { 00978 00979 if(fn>=files_count||fn<0) cgp_doError; 00980 if(!files[fn].isOpen) cgp_doError; 00981 if(B>files[fn].nbases||B<=0) cgp_doError; 00982 if(Z>files[fn].bases[B-1].nzones||Z<=0) cgp_doError; 00983 herr_t herr; 00984 // Pointer to the current file

00985 file_t* file = &(files[fn]); 00986 // Pointer to the current base

00987 base_t* base = &(file->bases[B-1]); 216

00988 // Pointer to the current zone

00989 zone_t* zone = &(base->zones[Z-1]); 00990 00991 // Index 00992 int idx; 00993 // Loop index 00994 int k; 00995 00996 // If the section already exist, replace it 00997 if(node_exists(zone->grid_id, sectionname)) { 00998 // Find the section to be replaced 00999 for(k=0;knsections;k++) if(strcmp(sectionname,zone->sections[k]. sectionname)==0) idx = k; 01000 // Delete it 01001 del_node(zone->grid_id,sectionname); 01002 } 01003 // If the section does not exist, prepare the memory structures 01004 else { 01005 // New list of sections

01006 section_t* sections = (section_t*) malloc((zone->nsections+1)*sizeof( section_t)); 01007 if(sections==NULL) cgp_doError; 01008 // Copy old list to new list 01009 for(k=0;knsections;k++) sections[k] = zone->sections[k]; 01010 // Free the old list 01011 if(zone->sections!=NULL) free(zone->sections); 01012 // Point the zone to the new list 01013 zone->sections = sections; 01014 // Incement the number of coords 01015 zone->nsections++; 01016 // Set the coords index 01017 idx = zone->nsections-1; 01018 } 01019 // Pointer to coords

01020 section_t* section = &(zone->sections[idx]); 217

01021 // Copy the name 01022 strcpy(section->sectionname, sectionname); 01023 // Set the index 01024 section->idx = idx; 01025 // Set the type 01026 section->type = type; 01027 // Set the range 01028 section->range[0] = start; 01029 section->range[1] = end; 01030 // Return the index

01031 *S = idx+1; 01032 01033 // Create the section in the file 01034 new_node(zone->group_id, sectionname, "Elements_t", "I4"); 01035 // Open the new zone 01036 section->group_id = H5Gopen2(zone->group_id, sectionname, H5P_DEFAULT); 01037 if(section->group_id<0) cgp_doError; 01038 01039 // Write data 01040 { 01041 // Size of array to write 01042 hsize_t dim[1] = {2}; 01043 // Create shape object for array to write 01044 hid_t shape_id = H5Screate_simple(1,dim, NULL); 01045 if(shape_id<0) cgp_doError; 01046 // Create the data array in the file 01047 hid_t data_id = H5Dcreate2(section->group_id, " data", H5T_NATIVE_INT, shap e_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); 01048 if(data_id<0) cgp_doError; 01049 // Write the data to the array 01050 int data[2] = {type,0}; 01051 herr = H5Dwrite(data_id, H5T_NATIVE_INT, shape_id, shape_id, H5P_DEFAULT, d ata); 01052 if(herr<0) cgp_doError; 01053 // Close the data array 218

01054 herr = H5Dclose(data_id); 01055 if(herr<0) cgp_doError; 01056 // Destroy the shape object 01057 herr = H5Sclose(shape_id); 01058 if(herr<0) cgp_doError; 01059 } 01060 01061 // Create the ElementConnectivity in the section 01062 new_node(section->group_id, "ElementConnectivity", "DataArray_t", "I4"); 01063 // Open the new zone 01064 section->connectivity_id = H5Gopen2(section->group_id, "ElementConnectivity", H5P_DEFAULT); 01065 if(section->connectivity_id<0) cgp_doError; 01066 01067 // Create array for data 01068 // We choose not to support the MIXED type

01069 // The array size is (number of elements)*(nodes per element) 01070 // Write data 01071 { 01072 // Size of array to write

01073 hsize_t dim[1] = {(end-start+1)*node_counts[type]}; 01074 // Create shape object for array to write 01075 hid_t shape_id = H5Screate_simple(1,dim, NULL); 01076 if(shape_id<0) cgp_doError; 01077 // Property list for dataset 01078 hid_t plist_id = H5Pcreate(H5P_DATASET_CREATE); 01079 if(preallocate) herr = H5Pset_alloc_time(plist_id,H5D_ALLOC_TIME_EARLY); 01080 if(preallocate) herr = H5Pset_fill_time(plist_id, H5D_FILL_TIME_ALLOC); 01081 // Create the data array in the file 01082 hid_t data_id = H5Dcreate2(section->connectivity_id, " data", H5T_NATIVE_IN T, shape_id, H5P_DEFAULT, plist_id, H5P_DEFAULT); 01083 if(data_id<0) cgp_doError; 01084 // Close the data array 01085 herr = H5Dclose(data_id); 01086 if(herr<0) cgp_doError; 219

01087 // Close the property list 01088 herr = H5Pclose(plist_id); 01089 if(herr<0) cgp_doError; 01090 // Destroy the shape object 01091 herr = H5Sclose(shape_id); 01092 if(herr<0) cgp_doError; 01093 } 01094 01095 // Create the ElementRange in the section 01096 new_node(section->group_id, "ElementRange", "IndexRange_t", "I4"); 01097 // Open the new zone 01098 section->range_id = H5Gopen2(section->group_id, "ElementRange", H5P_DEFAULT); 01099 if(section->range_id<0) cgp_doError; 01100 01101 // Write data 01102 { 01103 // Size of array to write 01104 hsize_t dim[1] = {2}; 01105 // Create shape object for array to write 01106 hid_t shape_id = H5Screate_simple(1,dim, NULL); 01107 if(shape_id<0) cgp_doError; 01108 // Create the data array in the file 01109 hid_t data_id = H5Dcreate2(section->range_id, " data", H5T_NATIVE_INT, shap e_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); 01110 if(data_id<0) cgp_doError; 01111 // Write the data to the array 01112 int data[2] = {start,end}; 01113 herr = H5Dwrite(data_id, H5T_NATIVE_INT, shape_id, shape_id, H5P_DEFAULT, d ata); 01114 if(herr<0) cgp_doError; 01115 // Close the data array 01116 herr = H5Dclose(data_id); 01117 if(herr<0) cgp_doError; 01118 // Destroy the shape object 01119 herr = H5Sclose(shape_id); 220

01120 if(herr<0) cgp_doError; 01121 } 01122 01123 return 0; 01124 } 01125

01126 int cgp_section_write_data(int fn, int B, int Z, int S, int min, int max, int *el ements) { 01127 int k; 01128 if(fn>=files_count||fn<0) cgp_doError; 01129 if(!files[fn].isOpen) cgp_doError; 01130 if(B>files[fn].nbases||B<=0) cgp_doError; 01131 if(Z>files[fn].bases[B-1].nzones||Z<=0) cgp_doError; 01132 if(S>files[fn].bases[B-1].zones[Z-1].nsections||S<=0) cgp_doError; 01133 herr_t herr; 01134 // Pointer to the current file

01135 file_t* file = &(files[fn]); 01136 // Pointer to the current base

01137 base_t* base = &(file->bases[B-1]); 01138 // Pointer to the current zone

01139 zone_t* zone = &(base->zones[Z-1]); 01140 // Pointer to the current coords

01141 section_t* section = &(zone->sections[S-1]); 01142 01143 // Open the data 01144 hid_t data_id = H5Dopen2(section->connectivity_id, " data", H5P_DEFAULT); 01145 if(data_id<0) cgp_doError; 01146 01147 // Set the rank of the data 01148 hsize_t rank = 1; 01149 // Set the start position for the data write 01150 hsize_t start[rank];

01151 start[0] = (min-section->range[0])*node_counts[section->type]; 01152 // Compute the counts in each dimension 01153 hsize_t dims[rank]; 221

01154 dims[0] = (max-min+1)*node_counts[section->type]; 01155 // Create a shape for the data in memory 01156 hid_t mem_shape_id = H5Screate_simple(rank,dims,NULL); 01157 if(mem_shape_id<0) cgp_doError; 01158 // Create a shape for the data in the file 01159 hid_t data_shape_id = H5Dget_space(data_id); 01160 if(data_shape_id<0) cgp_doError; 01161 // Select a section of the array in the file 01162 herr = H5Sselect_hyperslab(data_shape_id, H5S_SELECT_SET, start, NULL, dims, N ULL); 01163 if(herr<0) cgp_doError; 01164 // Set the access property list for data transfer 01165 hid_t plist_id = H5Pcreate(H5P_DATASET_XFER); 01166 if(plist_id<0) cgp_doError; 01167 // Set MPI-IO collective communication 01168 herr = H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_COLLECTIVE); 01169 if(herr<0) cgp_doError; 01170 // Write the data in collective parallel I/O 01171 herr = H5Dwrite(data_id, H5T_NATIVE_INT, mem_shape_id, data_shape_id, plist_id , elements); 01172 if(herr<0) cgp_doError; 01173 // Close the property list 01174 herr = H5Pclose(plist_id); 01175 if(herr<0) cgp_doError; 01176 // Close the shape of the data in the file 01177 herr = H5Sclose(data_shape_id); 01178 if(herr<0) cgp_doError; 01179 // Close the shape of the data in memory 01180 herr = H5Sclose(mem_shape_id); 01181 if(herr<0) cgp_doError; 01182 // Close the data array 01183 herr = H5Dclose(data_id); 01184 if(herr<0) cgp_doError; 01185 01186 return 0; 222

01187 } 01188 01189 //= Array IO Prototypes =// 01190

01191 int cgp_array_write(int fn, int B, int Z, char *arrayname, GridLocation_t locatio n) { 01192 if(fn>=files_count||fn<0) cgp_doError; 01193 if(!files[fn].isOpen) cgp_doError; 01194 if(B>files[fn].nbases||B<=0) cgp_doError; 01195 if(Z>files[fn].bases[B-1].nzones||Z<=0) cgp_doError; 01196 herr_t herr; 01197 // Pointer to the current file

01198 file_t* file = &(files[fn]); 01199 // Pointer to the current base

01200 base_t* base = &(file->bases[B-1]); 01201 // Pointer to the current zone

01202 zone_t* zone = &(base->zones[Z-1]); 01203 // Index 01204 int idx; 01205 // Loop index 01206 int k; 01207 // If the solution already exist, replace it 01208 if(node_exists(zone->group_id, arrayname)) { 01209 // Delete them 01210 del_node(zone->group_id,arrayname); 01211 } 01212 01213 // Create a new node in the file 01214 new_node(zone->group_id, arrayname, "DataArray_t", "R8"); 01215 // Open the node 01216 hid_t group_id = H5Gopen2(zone->group_id, arrayname, H5P_DEFAULT); 01217 if(group_id<0) cgp_doError; 01218 01219 // Set the rank of the dimensions 01220 hsize_t rank = (zone->type==Structured)?base->cell_dim:1; 223

01221 // Dimensions of the array 01222 hsize_t dims[rank]; 01223 // Set these to correspond with the size of the zone 01224 if(location==Vertex) for(k=0;knijk[(rank-1)-k]; 01225 else if(location==CellCenter) for(k=0;knijk[rank+(r ank-1)-k]; 01226 01227 // Create a shape for the array 01228 hid_t shape_id = H5Screate_simple(rank,dims,NULL); 01229 if(shape_id<0) cgp_doError; 01230 // Property list for dataset 01231 hid_t plist_id = H5Pcreate(H5P_DATASET_CREATE); 01232 if(preallocate) herr = H5Pset_alloc_time(plist_id,H5D_ALLOC_TIME_EARLY); 01233 if(preallocate) herr = H5Pset_fill_time(plist_id, H5D_FILL_TIME_ALLOC); 01234 // Create the array in the file 01235 hid_t data_id = H5Dcreate2(group_id, " data", H5T_NATIVE_DOUBLE, shape_id, H5P _DEFAULT, plist_id, H5P_DEFAULT); 01236 if(data_id<0) cgp_doError; 01237 // Close the array 01238 herr = H5Dclose(data_id); 01239 // Close the property list 01240 herr = H5Pclose(plist_id); 01241 if(herr<0) cgp_doError; 01242 // Close the shape 01243 herr = H5Sclose(shape_id); 01244 if(herr<0) cgp_doError; 01245 // Close the group 01246 herr = H5Gclose(group_id); 01247 if(herr<0) cgp_doError; 01248 return 0; 01249 } 01250

01251 int cgp_array_write_data(int fn, int B, int Z, char* arrayname, int* min, int* max, void* data) { 01252 int k; 224

01253 if(fn>=files_count||fn<0) cgp_doError; 01254 if(!files[fn].isOpen) cgp_doError; 01255 if(B>files[fn].nbases||B<=0) cgp_doError; 01256 if(Z>files[fn].bases[B-1].nzones||Z<=0) cgp_doError; 01257 herr_t herr; 01258 // Pointer to the current file

01259 file_t* file = &(files[fn]); 01260 // Pointer to the current base

01261 base_t* base = &(file->bases[B-1]); 01262 // Pointer to the current zone

01263 zone_t* zone = &(base->zones[Z-1]); 01264 01265 // Open the group 01266 hid_t group_id = H5Gopen2(zone->group_id, arrayname, H5P_DEFAULT); 01267 // Open the data 01268 hid_t data_id = H5Dopen2(group_id, " data", H5P_DEFAULT); 01269 if(data_id<0) cgp_doError; 01270 01271 // Set the rank of the data 01272 hsize_t rank = (zone->type==Structured)?base->cell_dim:1; 01273 // Set the start position for the data write 01274 hsize_t start[rank]; 01275 for(k=0;k

01287 if(herr<0) cgp_doError; 01288 // Set the access property list for data transfer 01289 hid_t plist_id = H5Pcreate(H5P_DATASET_XFER); 01290 if(plist_id<0) cgp_doError; 01291 // Set MPI-IO collective communication 01292 herr = H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_COLLECTIVE); 01293 if(herr<0) cgp_doError; 01294 // Write the data in collective parallel I/O 01295 herr = H5Dwrite(data_id, H5T_NATIVE_DOUBLE, mem_shape_id, data_shape_id, plist _id, data); 01296 if(herr<0) cgp_doError; 01297 // Close the property list 01298 herr = H5Pclose(plist_id); 01299 if(herr<0) cgp_doError; 01300 // Close the shape of the data in the file 01301 herr = H5Sclose(data_shape_id); 01302 if(herr<0) cgp_doError; 01303 // Close the shape of the data in memory 01304 herr = H5Sclose(mem_shape_id); 01305 if(herr<0) cgp_doError; 01306 // Close the data array 01307 herr = H5Dclose(data_id); 01308 if(herr<0) cgp_doError; 01309 herr = H5Gclose(group_id); 01310 if(herr<0) cgp_doError; 01311 return 0; 01312 } 01313 01314 // The mallocs and memory copies in here need to be verified

01315 int queue_slice_write(SliceType_t type, int F, int B, int Z, void* SN, int rank, 01316 int* min, int* max, void* data) { 01317 int k;

01318 slice_t* slices = malloc((write_queue_len+1)*sizeof(slice_t)); 01319 for(k=0;k

01320 slice_t* slice = &(slices[write_queue_len]); 226

01321 01322 slice->type = type; 01323 slice->F = F; 01324 slice->B = B; 01325 slice->Z = Z; 01326 slice->rank = rank; 01327

01328 slice->min = (int*) malloc(rank*sizeof(int)); 01329 for(k=0;kmin[k] = min[k];

01330 slice->max = (int*) malloc(rank*sizeof(int)); 01331 for(k=0;kmax[k] = max[k]; 01332 slice->data = data;

01333 if(type!=Array&&type!=Empty) slice->Selector = *((int*) SN); 01334 else if(type==Array) strcpy(slice->name,(char*) SN); 01335 else {} 01336 01337 if(write_queue!=NULL) free(write_queue); 01338 write_queue = slices; 01339 write_queue_len++; 01340 return 0; 01341 } 01342 01343 // The mallocs and memory copies in here need to be verified 01344 int queue_flush(void){ 01345 int err; 01346 herr_t herr; 01347 int i,j,k; 01348 int max_queue_len = 0; 01349 int world_rank = 0; 01350 int world_size = 0; 01351 01352 err = MPI_Comm_rank(MPI_COMM_WORLD, &world_rank); 01353 err = MPI_Comm_size(MPI_COMM_WORLD, &world_size); 01354 err = MPI_Allreduce(&write_queue_len, &max_queue_len, 1, MPI_INT, MPI_MAX, MPI _COMM_WORLD); 227

01355 if(max_queue_len==0) max_queue_len = 1; 01356 01357 slice_t queue[max_queue_len]; 01358 for(k=0;k

01388 break; 01389 case(Elements): 01390 err = cgp_section_write_data(queue[k].F,queue[k].B,queue[k].Z,queue[k ].Selector,queue[k].min[0],queue[k].max[0],queue[k].data); 01391 break; 01392 case(Solution): 01393 err = cgp_sol_write_data(queue[k].F,queue[k].B,queue[k].Z,queue[k].Se lector,queue[k].min,queue[k].max,queue[k].data); 01394 break; 01395 case(Array): 01396 err = cgp_array_write_data(queue[k].F,queue[k].B,queue[k].Z,queue[k]. name,queue[k].min,queue[k].max,queue[k].data); 01397 break; 01398 } 01399 } 01400 for(k=0;k

A.4.11 pcgnslib.h File Reference #include "mpi.h"

#include "hdf5.h"

Include dependency graph for pcgnslib.h:

pcgnslib.h

mpi.h hdf5.h 229

This graph shows which files directly or indirectly include this file:

pcgnslib.h

pcgns_util.h benchmark.c open_close.c test_base.c test_queue.c test_unstructured.c test_zone.c thesis_benchmark.c

pcgns_util.c pcgnslib.c

Defines

• #define cgp_doError ;

• #define printTime

Typedefs

• typedef int DataType_t

Enumerations

• enum ZoneType_t { Structured, Unstructured }

• enum ElementType_t {

ElementTypeNull, ElementTypeUserDefined, NODE, BAR_2,

BAR_3, TRI_3, TRI_6, QUAD_4,

QUAD_8, QUAD_9, TETRA_4, TETRA_10,

PYRA_5, PYRA_13, PYRA_14, PENTA_6,

PENTA_15, PENTA_18, HEXA_8, HEXA_20,

HEXA_27, MIXED, NGON_n, NFACE_n }

• enum GridLocation_t { Vertex, CellCenter }

• enum SliceType_t {

Empty, Coords, Elements, Solution,

Array } 230

Functions

• int cgp_open (const char filename, int mode, MPI_Comm comm, MPI_Info info, int fn) ∗ ∗ ∗ • int cgp_close (int fn)

• int cgp_base_read (int fn, int B, char basename, int cell_dim, int phys_dim) ∗ ∗ ∗ • int cgp_base_write (int fn, char const basename, int cell_dim, int phys_dim, int B) ∗ ∗ • int cgp_nbases (int fn, int nbases) ∗ • int cgp_zone_read (int fn, int B, int Z, char zonename, int nijk) ∗ ∗ • int cgp_zone_type (int fn, int B, int Z, ZoneType_t zonetype) ∗ • int cgp_zone_write (int fn, int B, const char zonename, const int nijk, ZoneType_t type, int ∗ ∗ Z) ∗ • int cgp_nzones (int fn, int B, int nzones) ∗ • int cgp_coord_write (int fn, int B, int Z, DataType_t type, const char coordname, int C) ∗ ∗ • int cgp_coord_write_data (int fn, int B, int Z, int C, int min, int max, void coord_array) ∗ ∗ ∗ • int cgp_sol_write (int fn, int B, int Z, char solname, GridLocation_t location, int S) ∗ ∗ • int cgp_sol_write_data (int fn, int B, int Z, int S, int min, int max, void data) ∗ ∗ ∗ • int cgp_nsols (int fn, int B, int Z, int nsols) ∗ • int cgp_section_write (int fn, int B, int Z, char sectionname, ElementType_t type, int start, ∗ int end, int nbndry, int S) ∗ • int cgp_section_write_data (int fn, int B, int Z, int S, int min, int max, int elements) ∗ • int cgp_array_write (int fn, int B, int Z, char arrayname, GridLocation_t location) ∗ • int cgp_array_write_data (int fn, int B, int Z, char arrayname, int min, int max, void data) ∗ ∗ ∗ ∗ • int queue_slice_write (SliceType_t type, int F, int B, int Z, void SN, int rank, int min, int ∗ ∗ max, void data) ∗ ∗ • int queue_flush (void)

Variables

• int preallocate 231

A.4.11.1 Detailed Description Author:

Kyle Horne

Version:

0.2

A.4.11.2 LICENSE BSD style license

A.4.11.3 DESCRIPTION Header file for all public functions of the pcgns library

Definition in file pcgnslib.h.

A.4.11.4 Define Documentation

A.4.11.4.1 #define cgp_doError ;

Definition at line 17 of file pcgnslib.h.

A.4.11.4.2 #define printTime

Definition at line 21 of file pcgnslib.h.

A.4.11.5 Typedef Documentation

A.4.11.5.1 typedef int DataType_t

Definition at line 44 of file pcgnslib.h. 232

A.4.11.6 Enumeration Type Documentation

A.4.11.6.1 enum ElementType_t

Enumerator:

ElementTypeNull

ElementTypeUserDefined

NODE

BAR_2

BAR_3

TRI_3

TRI_6

QUAD_4

QUAD_8

QUAD_9

TETRA_4

TETRA_10

PYRA_5

PYRA_13

PYRA_14

PENTA_6

PENTA_15

PENTA_18

HEXA_8

HEXA_20 233

HEXA_27

MIXED

NGON_n

NFACE_n

Definition at line 31 of file pcgnslib.h.

A.4.11.6.2 enum GridLocation_t

Enumerator:

Vertex

CellCenter

Definition at line 45 of file pcgnslib.h.

A.4.11.6.3 enum SliceType_t

Enumerator:

Empty

Coords

Elements

Solution

Array

Definition at line 46 of file pcgnslib.h.

A.4.11.6.4 enum ZoneType_t 234 Enumerator:

Structured

Unstructured

Definition at line 30 of file pcgnslib.h.

A.4.11.7 Function Documentation

A.4.11.7.1 int cgp_array_write (int fn, int B, int Z, char arrayname, GridLocation_t ∗ location)

Write an array to a zone

Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

arrayname [in]: Name of array

location [in]: Location of solution within each cell

Returns:

Error code

Definition at line 1191 of file pcgnslib.c.

Here is the call graph for this function:

del_node new_int_attb cgp_array_write new_node

new_str_attb node_exists 235

Here is the caller graph for this function:

cgp_array_write main

A.4.11.7.2 int cgp_array_write_data (int fn, int B, int Z, char arrayname, int min, int ∗ ∗ max, void data) ∗ ∗

Write an array’s data in parallel Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

arrayname [in]: Name of array

min [in]: Lower bound array for data

max [in]: Upper bound array for data

data [in]: Data to be written

Returns:

Error code

Definition at line 1251 of file pcgnslib.c.

Here is the caller graph for this function:

main cgp_array_write_data queue_flush main

A.4.11.7.3 int cgp_base_read (int fn, int B, char basename, int cell_dim, int ∗ ∗ ∗ phys_dim)

Read info about a base 236 Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

basename [out]: Name of the base

cell_dim [out]: Cell dimensions of the base

phys_dim [out]: Physical dimensions of the base

Returns:

Error code

Definition at line 169 of file pcgnslib.c.

Here is the call graph for this function:

node_idx2name node_name_finder cgp_base_read get_str_attb num_nodes node_counter

Here is the caller graph for this function:

cgp_base_read main

A.4.11.7.4 int cgp_base_write (int fn, char const basename, int cell_dim, int phys_dim, ∗ int B) ∗

Write a base to a file Parameters:

fn int[in]: Handle f the file

basename [in]: Name of the base to write

cell_dim [in]: Cell dimensions of the base

phys_dim [in]: Physical dimensions of the base 237

B [out]: Index of the base Returns:

Error code

Definition at line 234 of file pcgnslib.c.

Here is the call graph for this function:

del_node free_coord

free_base free_zone free_section cgp_base_write

new_node new_int_attb free_sol

node_exists new_str_attb

Here is the caller graph for this function:

cgp_base_write main

A.4.11.7.5 int cgp_close (int fn)

Close a previously opened file Parameters:

fn [in]: Handle of the file to close

Returns:

Error code

Definition at line 159 of file pcgnslib.c.

Here is the call graph for this function:

del_node free_coord cgp_close free_file free_base free_zone free_section

free_sol 238

Here is the caller graph for this function:

cgp_close main

A.4.11.7.6 int cgp_coord_write (int fn, int B, int Z, DataType_t type, const char ∗ coordname, int C) ∗

Write coords group, but not data, to a grid Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

datatype [in]: Type of floats stored

coordname [in]: Name of the coords

C [out]: Index of the coords

Definition at line 667 of file pcgnslib.c.

Here is the call graph for this function:

del_node

new_int_attb cgp_coord_write new_node new_str_attb node_exists

Here is the caller graph for this function:

cgp_coord_write main

A.4.11.7.7 int cgp_coord_write_data (int fn, int B, int Z, int C, int min, int max, void ∗ ∗ coord_array) ∗

Write coords to a grid in parallel 239 Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

C [in]: Index of the coords

range_min [in]: Array of lower bound index

range_max [in]: Array of upper bound index

coord_array [in]: Pointer to the data

Returns:

Error code

Definition at line 750 of file pcgnslib.c.

Here is the caller graph for this function:

main cgp_coord_write_data queue_flush main

A.4.11.7.8 int cgp_nbases (int fn, int nbases) ∗

Read the number of bases in a file Parameters:

fn [in]: Handle of the file

nbases [out]: Number of bases in the specified file

Returns:

Error code

Definition at line 318 of file pcgnslib.c. 240

A.4.11.7.9 int cgp_nsols (int fn, int B, int Z, int nsols) ∗

Read the number of solutions in a zone Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

nsols [out]: Number of solutions in the specified zone

Returns:

Error code

Definition at line 960 of file pcgnslib.c.

A.4.11.7.10 int cgp_nzones (int fn, int B, int nzones) ∗

Read the number of zones in a base Parameters:

fn [in]: Handle of file

B [in]: Index of base

nzones [out]: Number of zones in the specified base

Returns:

Error code

Definition at line 660 of file pcgnslib.c.

A.4.11.7.11 int cgp_open (const char filename, int mode, MPI_Comm comm, MPI_Info ∗ ∗ info, int fn) ∗

Open a file for reading and writing 241 Parameters:

filename [in]: Name of the file to open

mode [in]: IO mode (read/write)

comm [in]: MPI communicator on which to open the file

info [in]: MPI info object to allow hints passed to MPI-IO

fn [out]: Handle of the opened file

Returns:

Error code

Definition at line 42 of file pcgnslib.c.

Here is the call graph for this function:

hdf5_format_str

hdf5_version_str

new_float

new_int

new_node new_int_attb cgp_open new_str_attb

new_str

next_file cleanup_files

node_idx2name node_name_finder

get_str_attb num_nodes node_counter

Here is the caller graph for this function:

cgp_open main 242

A.4.11.7.12 int cgp_section_write (int fn, int B, int Z, char sectionname, ElementType_t ∗ type, int start, int end, int nbndry, int S) ∗

Write the element connectivity groups for a section Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

C [in]: Index of the coords

sectionname [in]: Name of element section

type [in]: Type of element data

start [in]: Element lower bound index

end [in]: Element upper bound index

nbndry [in]: Number of boundary elements (unused)

S [out]: Section index

Returns:

Error code

Definition at line 976 of file pcgnslib.c.

Here is the call graph for this function:

del_node new_int_attb cgp_section_write new_node new_str_attb node_exists

Here is the caller graph for this function:

cgp_section_write main 243

A.4.11.7.13 int cgp_section_write_data (int fn, int B, int Z, int S, int min, int max, int ∗ elements)

Write the element connectivity data for a section Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

C [in]: Index of the coords

S [in]: Section index

min [in]: Output array lower bound index

max [in]: Output array upper bound index

elements [in]: Pointer to the data

Returns:

Error code

Definition at line 1126 of file pcgnslib.c.

Here is the caller graph for this function:

main cgp_section_write_data queue_flush main

A.4.11.7.14 int cgp_sol_write (int fn, int B, int Z, char solname, GridLocation_t ∗ location, int S) ∗

Write a solution to a zone Parameters:

fn [in]: Handle of the file 244

B [in]: Index of the base

Z [in]: Index of the zone

solname [in]: Name of solution

location [in]: Location of solution within each cell

S [out]: Index of solution

Returns:

Error code

Definition at line 876 of file pcgnslib.c.

Here is the call graph for this function:

del_node

new_int_attb

cgp_sol_write new_node

new_str_attb

node_exists

Here is the caller graph for this function:

cgp_sol_write main

A.4.11.7.15 int cgp_sol_write_data (int fn, int B, int Z, int S, int min, int max, void ∗ ∗ ∗ data)

Write a solution’s data in parallel Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

S [in]: Index of soltution 245

min [in]: Lower bound array for data

max [in]: Upper bound array for data

data [in]: Data to be written

Returns:

Error code

Definition at line 813 of file pcgnslib.c.

Here is the caller graph for this function:

main

cgp_sol_write_data

queue_flush main

A.4.11.7.16 int cgp_zone_read (int fn, int B, int Z, char zonename, int nijk) ∗ ∗

Read info about a zone

Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone to read

zonename [out]: Name of the zone

nijk [out]: Dimensions of the zone

Returns:

Error code

Definition at line 325 of file pcgnslib.c. 246

Here is the call graph for this function:

get_str

new_node new_int_attb

cgp_zone_read node_exists new_str_attb

node_idx2name node_name_finder get_str_attb num_nodes node_counter

Here is the caller graph for this function:

cgp_zone_read main

A.4.11.7.17 int cgp_zone_type (int fn, int B, int Z, ZoneType_t zonetype) ∗

Read the type of a zone Parameters:

fn [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

zonetype [out]: Type of zone Returns:

Error code

Definition at line 535 of file pcgnslib.c.

A.4.11.7.18 int cgp_zone_write (int fn, int B, const char zonename, const int nijk, ∗ ∗ ZoneType_t type, int Z) ∗

Write a zone to a base Parameters:

fn [in]: Handle of the file 247

B [in]: Index of the base

zonename [in]: Name of the zone to write

nijk [in]: Dimensions of the zone

type [in]: Type of zone

Z [out]: Index of the zone

Returns:

Error code

Definition at line 544 of file pcgnslib.c.

Here is the call graph for this function:

free_coord

del_node free_section

free_zone free_sol

cgp_zone_write new_node new_int_attb

new_str new_str_attb

node_exists

Here is the caller graph for this function:

cgp_zone_write main

A.4.11.7.19 int queue_flush (void)

Flush all the IO operations waiting in the queue Returns:

Error code

Definition at line 1344 of file pcgnslib.c. 248

Here is the call graph for this function:

cgp_array_write_data

cgp_coord_write_data queue_flush cgp_section_write_data

cgp_sol_write_data

Here is the caller graph for this function:

queue_flush main

A.4.11.7.20 int queue_slice_write (SliceType_t type, int F, int B, int Z, void SN, int ∗ rank, int min, int max, void data) ∗ ∗ ∗

Queue an IO write operation for flushing later Parameters:

type [in]: Type of operation to queue

F [in]: Handle of the file

B [in]: Index of the base

Z [in]: Index of the zone

SN [in]: Pointer to array locator, which is an int for coordinates, solutions and sections, but a string for arrays

rank [in]: Rank of data to be written

min [in]: Pointer to the minumum location array

max [in]: Pointer to the maximum location array

data [in]: Pointer to the data to be written Returns:

Error code 249

Definition at line 1315 of file pcgnslib.c.

Here is the caller graph for this function:

queue_slice_write main

A.4.11.8 Variable Documentation

A.4.11.8.1 int preallocate

Definition at line 35 of file pcgnslib.c.

A.4.12 pcgnslib.h

00001 00002 00003 00004 00005 00006 00007 00008 00009 00010 00011 #ifndef PCGNSLIB_H_ 00012 #define PCGNSLIB_H_ 00013 00014 #ifdef _DEBUG 00015 #define cgp_doError {printf("Error at %s:%u n",__FILE__, __LINE__); return 1;} \ 00016 #else 00017 #define cgp_doError ; 00018 #endif 00019 00020 //#define printTime printf("Time at %s:%u = %f n",__FILE__,__LINE__,MPI_Wtime()) \ 250

00021 #define printTime 00022 00023 #include "mpi.h" 00024 #include "hdf5.h" 00025 00026 //======// 00027 //== Begin Datatypes ==// 00028 //======// 00029 00030 typedef enum {Structured, Unstructured} ZoneType_t; 00031 typedef enum { 00032 ElementTypeNull, ElementTypeUserDefined, // 0 1 00033 NODE, // 2 00034 BAR_2, BAR_3, // 3 4 00035 TRI_3, TRI_6, // 5 6 00036 QUAD_4, QUAD_8, QUAD_9, // 7 8 9 00037 TETRA_4, TETRA_10, // 10 11 00038 PYRA_5, PYRA_13, PYRA_14, // 12 13 14 00039 PENTA_6, PENTA_15, PENTA_18, // 15 16 17 00040 HEXA_8, HEXA_20, HEXA_27, // 18 19 20 00041 MIXED, // 21 00042 NGON_n, NFACE_n // 22 23 00043 } ElementType_t; 00044 typedef int DataType_t; 00045 typedef enum {Vertex, CellCenter} GridLocation_t; 00046 typedef enum {Empty, Coords, Elements, Solution, Array} SliceType_t; 00047 00048 extern int preallocate; 00049 00050 //======// 00051 //== Begin Function Prototypes ==// 00052 //======// 00053 00054 //= File IO Prototypes =//

00062 int cgp_open(const char* filename, int mode, MPI_Comm comm, MPI_Info* info, int* 251

fn); 00063 00067 int cgp_close(int fn); 00068 00069 //= Base IO Prototypes =//

00077 int cgp_base_read(int fn, int B, char* basename, int* cell_dim, int* phys_dim); 00078

00086 int cgp_base_write(int fn, char const* basename, int cell_dim, int phys_dim, int* B); 00087

00092 int cgp_nbases(int fn, int *nbases); 00093 00094 //= Zone IO Prototypes =//

00102 int cgp_zone_read(int fn, int B, int Z, char* zonename, int* nijk); 00103

00110 int cgp_zone_type(int fn, int B, int Z, ZoneType_t *zonetype); 00111

00120 int cgp_zone_write(int fn, int B, const char* zonename, const int* nijk, ZoneType_t type, int* Z); 00121

00127 int cgp_nzones(int fn, int B, int *nzones); 00128 00129 //= Grid IO Prototypes =// 00130

00138 int cgp_coord_write(int fn, int B, int Z, DataType_t type, const char* coordname, int* C); 00139

00149 int cgp_coord_write_data(int fn, int B, int Z, int C, int* min, int* max, void* c oord_array); 00150 00151 //= Solution IO Prototypes =//

00160 int cgp_sol_write(int fn, int B, int Z, char *solname, GridLocation_t location, i nt *S); 00161

00171 int cgp_sol_write_data(int fn, int B, int Z, int S, int* min, int* max, void* dat 252

a); 00172

00179 int cgp_nsols(int fn, int B, int Z, int* nsols); 00180 00181 //= Unstructured Grid Prototypes =//

00194 int cgp_section_write(int fn, int B, int Z, char* sectionname, ElementType_t type ,

00195 int start, int end, int nbndry, int* S); 00196

00207 int cgp_section_write_data(int fn, int B, int Z, int S, int min, int max, int *el ements); 00208 00209 //= Array IO Prototypes =//

00217 int cgp_array_write(int fn, int B, int Z, char *arrayname, GridLocation_t locatio n); 00218

00228 int cgp_array_write_data(int fn, int B, int Z, char* arrayname, int* min, int* max, void* data); 00229 00230 //= Queue IO Prototypes =//

00242 int queue_slice_write(SliceType_t type, int F, int B, int Z, void* SN, int rank, 00243 int* min, int* max, void* data); 00244 00247 int queue_flush(void); 00248 #endif

A.4.13 test_base.c File Reference #include "pcgnslib.h"

#include "stdio.h"

#include "stdlib.h"

#include "mpi.h" 253

Include dependency graph for test_base.c:

test_base.c

pcgnslib.h stdio.h stdlib.h

mpi.h hdf5.h

Functions

• int main (int argc, char argv[ ]) ∗

A.4.13.1 Detailed Description

Author:

Kyle Horne

Version:

0.2

A.4.13.2 LICENSE BSD style license

A.4.13.3 DESCRIPTION Test program for pcgns library

Definition in file test_base.c.

A.4.13.4 Function Documentation

A.4.13.4.1 int main (int argc, char argv[ ]) ∗ 254

Definition at line 17 of file test_base.c.

Here is the call graph for this function:

node_exists

free_coord

cgp_base_write del_node free_base free_zone free_section

cgp_close free_file free_sol

main new_node new_int_attb

cgp_base_read num_nodes node_counter get_str_attb node_idx2name node_name_finder

new_str_attb

cgp_open new_str

hdf5_format_str

hdf5_version_str

new_float

new_int

next_file cleanup_files

A.4.14 test_base.c

00001 00002 00003 00004 00005 00006 00007 00008 00009 00010 00011 #include "pcgnslib.h" 00012 255

00013 #include "stdio.h" 00014 #include "stdlib.h" 00015 #include "mpi.h" 00016

00017 int main(int argc, char* argv[]) { 00018 int err; 00019 int comm_size; 00020 int comm_rank; 00021 MPI_Info info; 00022 int fn; 00023 int B; 00024 char basename[100]; 00025 int cell_dim = 3; 00026 int phys_dim = 3; 00027 00028 err = MPI_Init(&argc,&argv); 00029 if(err!=MPI_SUCCESS) cgp_doError; 00030 err = MPI_Comm_size(MPI_COMM_WORLD, &comm_size); 00031 if(err!=MPI_SUCCESS) cgp_doError; 00032 err = MPI_Comm_rank(MPI_COMM_WORLD, &comm_rank); 00033 if(err!=MPI_SUCCESS) cgp_doError; 00034 err = MPI_Info_create(&(info)); 00035 if(err!=MPI_SUCCESS) cgp_doError; 00036 00037 err = cgp_open("test_base.cgns", 0, MPI_COMM_WORLD, &info, &fn); 00038 if(err!=0) cgp_doError; 00039 err = cgp_base_write(fn, "Base 1", cell_dim, phys_dim, &B); 00040 if(err!=0) cgp_doError; 00041 err = cgp_base_read(fn, B, basename, &cell_dim, &phys_dim); 00042 if(err!=0) cgp_doError; 00043 err = cgp_close(fn); 00044 if(err!=0) cgp_doError; 00045 00046 err = MPI_Finalize(); 00047 if(err!=MPI_SUCCESS) cgp_doError; 256

00048 return err; 00049 }

A.4.15 test_queue.c File Reference #include "pcgnslib.h"

#include "stdio.h"

#include "stdlib.h"

#include "mpi.h"

Include dependency graph for test_queue.c:

test_queue.c

pcgnslib.h stdio.h stdlib.h

mpi.h hdf5.h

Functions

• int main (int argc, char argv[ ]) ∗

A.4.15.1 Detailed Description Author:

Kyle Horne Version:

0.2

A.4.15.2 LICENSE BSD style license

A.4.15.3 DESCRIPTION 257

Test program for pcgns library

Definition in file test_queue.c.

A.4.15.4 Function Documentation

A.4.15.4.1 int main (int argc, char argv[ ]) ∗

Definition at line 17 of file test_queue.c.

Here is the call graph for this function:

free_file

cgp_close del_node free_coord free_base

free_zone free_section

free_sol cgp_base_write node_exists

cgp_zone_write new_node new_int_attb

new_str

new_str_attb

hdf5_format_str

hdf5_version_str

main cgp_open new_float

new_int

cgp_zone_read next_file cleanup_files

num_nodes node_counter get_str_attb node_idx2name node_name_finder

get_str

cgp_array_write_data queue_flush cgp_coord_write_data queue_slice_write cgp_section_write_data

cgp_sol_write_data 258

A.4.16 test_queue.c

00001 00002 00003 00004 00005 00006 00007 00008 00009 00010 00011 #include "pcgnslib.h" 00012 00013 #include "stdio.h" 00014 #include "stdlib.h" 00015 #include "mpi.h" 00016

00017 int main(int argc, char* argv[]) { 00018 int err; 00019 int comm_size; 00020 int comm_rank; 00021 MPI_Info info; 00022 int fn; 00023 int B; 00024 int Z; 00025 char basename[100]; 00026 char zonename[100]; 00027 int cell_dim = 3; 00028 int phys_dim = 3; 00029 int nijk[3][3]; 00030 00031 nijk[0][0] = 10; 00032 nijk[0][1] = 10; 00033 nijk[0][2] = 1; 00034 nijk[1][0] = nijk[0][0]-1; 259

00035 nijk[1][1] = nijk[0][1]-1; 00036 nijk[1][2] = nijk[0][2]-1; 00037 nijk[2][0] = 0; 00038 nijk[2][1] = 0; 00039 nijk[2][2] = 0; 00040 00041 err = MPI_Init(&argc,&argv); 00042 if(err!=MPI_SUCCESS) cgp_doError; 00043 err = MPI_Comm_size(MPI_COMM_WORLD, &comm_size); 00044 if(err!=MPI_SUCCESS) cgp_doError; 00045 err = MPI_Comm_rank(MPI_COMM_WORLD, &comm_rank); 00046 if(err!=MPI_SUCCESS) cgp_doError; 00047 err = MPI_Info_create(&(info)); 00048 if(err!=MPI_SUCCESS) cgp_doError; 00049 00050 err = cgp_open("test_queue.cgns", 0, MPI_COMM_WORLD, &info, &fn); 00051 if(err!=0) cgp_doError; 00052 err = cgp_base_write(fn, "Base 1", cell_dim, phys_dim, &B); 00053 if(err!=0) cgp_doError; 00054 err = cgp_zone_write(fn, B, "Zone 1", &(nijk[0][0]), Structured, &Z); 00055 if(err!=0) cgp_doError; 00056 err = cgp_zone_read(fn, B, Z, zonename, &(nijk[0][0])); 00057 if(err!=0) cgp_doError; 00058 00059 int min = 0; 00060 int max = 0; 00061 double data = 0.0; 00062 err = queue_slice_write(Empty, fn, B, Z, NULL,1, &min, &max, &data); 00063 err = queue_flush(); 00064 00065 err = cgp_close(fn); 00066 if(err!=0) cgp_doError; 00067 00068 err = MPI_Finalize(); 00069 if(err!=MPI_SUCCESS) cgp_doError; 260

00070 return 0; 00071 }

A.4.17 test_unstructured.c File Reference #include "pcgnslib.h"

#include "stdio.h"

#include "stdlib.h"

#include "mpi.h"

Include dependency graph for test_unstructured.c:

test_unstructured.c

pcgnslib.h stdio.h stdlib.h

mpi.h hdf5.h

Functions

• int main (int argc, char argv[ ]) ∗

A.4.17.1 Detailed Description Author:

Kyle Horne Version:

0.2

A.4.17.2 LICENSE BSD style license

A.4.17.3 DESCRIPTION 261

Test program for pcgns library

Definition in file test_unstructured.c.

A.4.17.4 Function Documentation

A.4.17.4.1 int main (int argc, char argv[ ]) ∗

Definition at line 17 of file test_unstructured.c.

Here is the call graph for this function:

free_file

cgp_close free_base

cgp_base_write del_node free_coord

cgp_coord_write node_exists free_zone free_section

cgp_section_write new_node new_int_attb free_sol

main cgp_zone_write

new_str

hdf5_version_str

new_float new_str_attb

cgp_open new_int

next_file cleanup_files

node_idx2name node_name_finder get_str_attb num_nodes node_counter

hdf5_format_str

cgp_section_write_data queue_flush cgp_sol_write_data queue_slice_write cgp_array_write_data

cgp_coord_write_data 262

A.4.18 test_unstructured.c

00001 00002 00003 00004 00005 00006 00007 00008 00009 00010 00011 #include "pcgnslib.h" 00012 00013 #include "stdio.h" 00014 #include "stdlib.h" 00015 #include "mpi.h" 00016

00017 int main(int argc, char* argv[]) { 00018 int err; 00019 int comm_size; 00020 int comm_rank; 00021 MPI_Info info; 00022 int fn; 00023 int B; 00024 int Z; 00025 int S; 00026 char basename[100]; 00027 char zonename[100]; 00028 int cell_dim = 3; 00029 int phys_dim = 3; 00030 int nijk[3][1]; 00031 int k; 00032 00033 nijk[0][0] = 10; 00034 nijk[1][0] = nijk[0][0]-1; 263

00035 nijk[2][0] = 0; 00036 00037 err = MPI_Init(&argc,&argv); 00038 if(err!=MPI_SUCCESS) cgp_doError; 00039 err = MPI_Comm_size(MPI_COMM_WORLD, &comm_size); 00040 if(err!=MPI_SUCCESS) cgp_doError; 00041 err = MPI_Comm_rank(MPI_COMM_WORLD, &comm_rank); 00042 if(err!=MPI_SUCCESS) cgp_doError; 00043 err = MPI_Info_create(&(info)); 00044 if(err!=MPI_SUCCESS) cgp_doError; 00045 00046 err = cgp_open("test_unstructured.cgns", 0, MPI_COMM_WORLD, &info, &fn); 00047 if(err!=0) cgp_doError; 00048 err = cgp_base_write(fn, "Base 1", cell_dim, phys_dim, &B); 00049 if(err!=0) cgp_doError; 00050 err = cgp_zone_write(fn, B, "Zone 1", &(nijk[0][0]), Unstructured, &Z); 00051 if(err!=0) cgp_doError; 00052 00053 double x[10/comm_size]; 00054 double y[10/comm_size]; 00055 double z[10/comm_size]; 00056

00057 int min[1] = {10/comm_size*comm_rank}; 00058 int max[1] = {10/comm_size*(comm_rank+1)-1}; 00059 00060 for(k=0;k<10/comm_size;k++) { 00061 x[k] = (double) (min[0]+k); 00062 y[k] = 0.0; 00063 z[k] = 0.0; 00064 } 00065 00066 int Cx,Cy,Cz; 00067 00068 err = cgp_coord_write(fn,B,Z,Unstructured,"CoordinateX",&Cx); 00069 //~ err = cgp_coord_write_data(fn,B,Z,Cz,min,max,x); 264

00070 00071 err = cgp_coord_write(fn,B,Z,Unstructured,"CoordinateY",&Cy); 00072 //~ err = cgp_coord_write_data(fn,B,Z,Cy,min,max,y); 00073 00074 err = cgp_coord_write(fn,B,Z,Unstructured,"CoordinateZ",&Cz); 00075 //~ err = cgp_coord_write_data(fn,B,Z,Cz,min,max,z); 00076 00077 err = queue_slice_write(Coords, fn, B, Z, &Cx, 1, min, max, x); 00078 err = queue_slice_write(Coords, fn, B, Z, &Cy, 1, min, max, y); 00079 err = queue_slice_write(Coords, fn, B, Z, &Cz, 1, min, max, z); 00080 err = queue_flush(); 00081 00082 int start = 1; 00083 int end = 9; 00084 err = cgp_section_write(fn,B,Z,"Elements",BAR_2,start,end,0,&S); 00085

00086 int nelems = (comm_rank!=comm_size-1)?9/comm_size:9-(9/comm_size)*(comm_size-1 ); 00087 printf("%d:%d n",comm_rank,nelems); \ 00088 int emin = (9/comm_size)*comm_rank+1; 00089 int emax = (comm_rank!=comm_size-1)?(9/comm_size)*(comm_rank+1):9; 00090 int elements[nelems*2]; 00091 for(k=0;k

00092 elements[2*k] = k+emin; 00093 elements[2*k+1] = k+emin+1; 00094 } 00095 printf("%d:%d %d %d n",comm_rank,nelems,emin,emax); \ 00096 //~ err = cgp_section_write_data(fn,B,Z,S,emin,emax,&(elements[0])); 00097 00098 err = queue_slice_write(Elements, fn, B, Z, &S, 1, &emin, &emax, elements); 00099 err = queue_flush(); 00100 00101 err = cgp_close(fn); 00102 if(err!=0) cgp_doError; 00103 265

00104 err = MPI_Finalize(); 00105 if(err!=MPI_SUCCESS) cgp_doError; 00106 return 0; 00107 }

A.4.19 test_zone.c File Reference #include "pcgnslib.h"

#include "stdio.h"

#include "stdlib.h"

#include "mpi.h"

Include dependency graph for test_zone.c:

test_zone.c

pcgnslib.h stdio.h stdlib.h

mpi.h hdf5.h

Functions

• int main (int argc, char argv[ ]) ∗

A.4.19.1 Detailed Description

Author:

Kyle Horne

Version:

0.2

A.4.19.2 LICENSE BSD style license 266

A.4.19.3 DESCRIPTION Test program for pcgns library

Definition in file test_zone.c.

A.4.19.4 Function Documentation

A.4.19.4.1 int main (int argc, char argv[ ]) ∗

Definition at line 17 of file test_zone.c.

Here is the call graph for this function:

free_file

cgp_close del_node free_coord free_base

free_zone free_section

cgp_base_write free_sol node_exists main cgp_zone_write new_node new_int_attb

new_str

new_str_attb

hdf5_format_str

cgp_open hdf5_version_str

new_float

new_int cgp_zone_read next_file cleanup_files

num_nodes node_counter

get_str_attb node_idx2name node_name_finder

get_str

A.4.20 test_zone.c 267

00001 00002 00003 00004 00005 00006 00007 00008 00009 00010 00011 #include "pcgnslib.h" 00012 00013 #include "stdio.h" 00014 #include "stdlib.h" 00015 #include "mpi.h" 00016

00017 int main(int argc, char* argv[]) { 00018 int err; 00019 int comm_size; 00020 int comm_rank; 00021 MPI_Info info; 00022 int fn; 00023 int B; 00024 int Z; 00025 char basename[100]; 00026 char zonename[100]; 00027 int cell_dim = 3; 00028 int phys_dim = 3; 00029 int nijk[3][3]; 00030 00031 nijk[0][0] = 10; 00032 nijk[0][1] = 10; 00033 nijk[0][2] = 1; 00034 nijk[1][0] = nijk[0][0]-1; 00035 nijk[1][1] = nijk[0][1]-1; 268

00036 nijk[1][2] = nijk[0][2]-1; 00037 nijk[2][0] = 0; 00038 nijk[2][1] = 0; 00039 nijk[2][2] = 0; 00040 00041 err = MPI_Init(&argc,&argv); 00042 if(err!=MPI_SUCCESS) cgp_doError; 00043 err = MPI_Comm_size(MPI_COMM_WORLD, &comm_size); 00044 if(err!=MPI_SUCCESS) cgp_doError; 00045 err = MPI_Comm_rank(MPI_COMM_WORLD, &comm_rank); 00046 if(err!=MPI_SUCCESS) cgp_doError; 00047 err = MPI_Info_create(&(info)); 00048 if(err!=MPI_SUCCESS) cgp_doError; 00049 00050 err = cgp_open("test_zone.cgns", 0, MPI_COMM_WORLD, &info, &fn); 00051 if(err!=0) cgp_doError; 00052 err = cgp_base_write(fn, "Base 1", cell_dim, phys_dim, &B); 00053 if(err!=0) cgp_doError; 00054 err = cgp_zone_write(fn, B, "Zone 1", &(nijk[0][0]), Structured, &Z); 00055 if(err!=0) cgp_doError; 00056 err = cgp_zone_read(fn, B, Z, zonename, &(nijk[0][0])); 00057 if(err!=0) cgp_doError; 00058 err = cgp_close(fn); 00059 if(err!=0) cgp_doError; 00060 00061 err = MPI_Finalize(); 00062 if(err!=MPI_SUCCESS) cgp_doError; 00063 return 0; 00064 }

A.4.21 thesis_benchmark.c File Reference #include "pcgnslib.h"

#include "stdio.h"

#include "stdlib.h" 269

#include "math.h"

#include "mpi.h"

Include dependency graph for thesis_benchmark.c:

thesis_benchmark.c

pcgnslib.h stdio.h stdlib.h math.h

mpi.h hdf5.h

Functions

• int read_inputs (int argc, char argv) ∗ ∗∗∗ • int initialize (int argc, char argv) ∗ ∗∗∗ • int finalize (void)

• int main (int argc, char argv[ ]) ∗

Variables

• int comm_size

• int comm_rank

• MPI_Info info

• double data_size

• int N

• int Nl

• int pc

• int zpp

• int ppz

• int zc

• int zones ∗ 270

• int subzones ∗ • double x ∗ • double y ∗ • double z ∗ • int e ∗ • double u ∗ • double v ∗ • double w ∗ • double h ∗

A.4.21.1 Function Documentation

A.4.21.1.1 int finalize (void)

Definition at line 119 of file thesis_benchmark.c.

A.4.21.1.2 int initialize (int argc, char argv) ∗ ∗∗∗

Definition at line 62 of file thesis_benchmark.c.

Here is the call graph for this function:

initialize read_inputs

A.4.21.1.3 int main (int argc, char argv[ ]) ∗

Definition at line 135 of file thesis_benchmark.c. 271

Here is the call graph for this function:

cgp_array_write

cgp_coord_write node_exists

cgp_section_write del_node

cgp_sol_write new_node new_int_attb

cgp_base_write new_str_attb free_coord

cgp_close free_file free_base free_zone free_section

cgp_zone_write main free_sol cgp_open new_str

cgp_array_write_data hdf5_version_str

cgp_coord_write_data new_float

cgp_section_write_data new_int

cgp_sol_write_data

next_file cleanup_files finalize

node_idx2name node_name_finder initialize get_str_attb num_nodes node_counter

hdf5_format_str

A.4.21.1.4 int read_inputs (int argc, char argv) ∗ ∗∗∗

Definition at line 34 of file thesis_benchmark.c.

Here is the caller graph for this function:

read_inputs initialize

A.4.21.2 Variable Documentation 272

A.4.21.2.1 int comm_rank

Definition at line 8 of file thesis_benchmark.c.

A.4.21.2.2 int comm_size

Definition at line 7 of file thesis_benchmark.c.

A.4.21.2.3 double data_size

Definition at line 11 of file thesis_benchmark.c.

A.4.21.2.4 int e ∗

Definition at line 26 of file thesis_benchmark.c.

A.4.21.2.5 double h ∗

Definition at line 32 of file thesis_benchmark.c.

A.4.21.2.6 MPI_Info info

Definition at line 9 of file thesis_benchmark.c.

A.4.21.2.7 int N

Definition at line 12 of file thesis_benchmark.c. 273

A.4.21.2.8 int Nl

Definition at line 13 of file thesis_benchmark.c.

A.4.21.2.9 int pc

Definition at line 14 of file thesis_benchmark.c.

A.4.21.2.10 int ppz

Definition at line 16 of file thesis_benchmark.c.

A.4.21.2.11 int subzones ∗

Definition at line 20 of file thesis_benchmark.c.

A.4.21.2.12 double u ∗

Definition at line 28 of file thesis_benchmark.c.

A.4.21.2.13 double v ∗

Definition at line 29 of file thesis_benchmark.c.

A.4.21.2.14 double w ∗

Definition at line 30 of file thesis_benchmark.c. 274

A.4.21.2.15 double x ∗

Definition at line 22 of file thesis_benchmark.c.

A.4.21.2.16 double y ∗

Definition at line 23 of file thesis_benchmark.c.

A.4.21.2.17 double z ∗

Definition at line 24 of file thesis_benchmark.c.

A.4.21.2.18 int zc

Definition at line 17 of file thesis_benchmark.c.

A.4.21.2.19 int zones ∗

Definition at line 19 of file thesis_benchmark.c.

A.4.21.2.20 int zpp

Definition at line 15 of file thesis_benchmark.c.

A.4.22 thesis_benchmark.c

00001 #include "pcgnslib.h" 00002 #include "stdio.h" 00003 #include "stdlib.h" 275

00004 #include "math.h" 00005 #include "mpi.h" 00006 00007 int comm_size; 00008 int comm_rank; 00009 MPI_Info info; 00010 00011 double data_size; 00012 int N; 00013 int Nl; 00014 int pc; 00015 int zpp; 00016 int ppz; 00017 int zc; 00018

00019 int* zones; 00020 int* subzones; 00021

00022 double* x; 00023 double* y; 00024 double* z; 00025

00026 int* e; 00027

00028 double* u; 00029 double* v; 00030 double* w; 00031

00032 double* h; 00033

00034 int read_inputs(int* argc, char*** argv) { 00035 int k; 00036 if(comm_rank==0) {

00037 if(*argc<7) exit(1); 00038 for(k=1;k<*argc;k++) { 276

00039 if(strcmp((*argv)[k],"-ds")==0) { 00040 k++;

00041 sscanf((*argv)[k],"%lf",&data_size); 00042 printf("data_size=%lf n",data_size); \ 00043 }

00044 if(strcmp((*argv)[k],"-zpp")==0) { 00045 k++;

00046 sscanf((*argv)[k],"%d",&zpp); 00047 printf("zpp=%d n",zpp); \ 00048 }

00049 if(strcmp((*argv)[k],"-ppz")==0) { 00050 k++;

00051 sscanf((*argv)[k],"%d",&ppz); 00052 printf("ppz=%d n",ppz); \ 00053 } 00054 } 00055 } 00056 MPI_Bcast(&data_size,1,MPI_DOUBLE,0,MPI_COMM_WORLD); 00057 MPI_Bcast(&zpp,1,MPI_INT,0,MPI_COMM_WORLD); 00058 MPI_Bcast(&ppz,1,MPI_INT,0,MPI_COMM_WORLD); 00059 return 0; 00060 } 00061

00062 int initialize(int* argc, char*** argv) { 00063 int j,k; 00064 00065 MPI_Init(argc,argv); 00066 MPI_Comm_size(MPI_COMM_WORLD, &comm_size); 00067 MPI_Comm_rank(MPI_COMM_WORLD, &comm_rank); 00068 MPI_Info_create(&info); 00069 00070 read_inputs(argc,argv); 00071

00072 N = (data_size*1024*1024)/((double) sizeof(double)); 00073 pc = comm_size; 277

00074 zc = (pc*zpp)/ppz; 00075 Nl = N/comm_size/zpp; 00076

00077 zones = malloc(Nl*sizeof(int)); 00078 subzones = malloc(zpp*sizeof(int)); 00079 for(k=0;k

00080 zones[k] = comm_rank/ppz*zpp+k; 00081 subzones[k] = comm_rank%ppz; 00082 } 00083 00084 // Initialize Arrays

00085 x = malloc(Nl*sizeof(double)); 00086 y = malloc(Nl*sizeof(double)); 00087 z = malloc(Nl*sizeof(double)); 00088

00089 e = malloc(Nl*sizeof(int)); 00090

00091 u = malloc(Nl*sizeof(double)); 00092 v = malloc(Nl*sizeof(double)); 00093 w = malloc(Nl*sizeof(double)); 00094

00095 h = malloc(Nl*sizeof(double)); 00096 00097 double theta; 00098 double r; 00099 for(k=0;k

00100 j = Nl*subzones[0]+k; 00101 theta = ((double) j)/((double) Nl*zpp); 00102 r = theta;

00103 x[k] = r*cos(theta); 00104 y[k] = r*sin(theta); 00105 z[k] = r; 00106 e[k] = j+1; 00107 u[k] = x[k]; 00108 v[k] = y[k]; 278

00109 w[k] = z[k]; 00110 h[k] = r; 00111 } 00112 00113 //~ printf("%d: Nl %d n",comm_rank,Nl); \ 00114 //~ for(k=0;k

00135 int main(int argc, char* argv[]) { 00136 int k; 00137 int F; 00138 int B; 00139 00140 int nijk[3][1]; 00141 00142 initialize(&argc,&argv); 00143 279

00144 cgp_open("thesis_benchmark.cgns",0,MPI_COMM_WORLD, &info, &F); 00145 cgp_base_write(F,"Base",3,3,&B); 00146

00147 nijk[0][0] = Nl*ppz; 00148 nijk[1][0] = Nl*ppz; 00149 nijk[2][0] = 0; 00150 00151 int Z[zc]; 00152 int Cx[zc]; 00153 int Cy[zc]; 00154 int Cz[zc]; 00155 int E[zc]; 00156 int Su[zc]; 00157 int Sv[zc]; 00158 int Sw[zc]; 00159 00160 for(k=0;k

00167 cgp_section_write(F,B,Z[k],"Elements",NODE,1,Nl*ppz,0,&(E[k])); 00168 cgp_sol_write(F,B,Z[k],"MomentumX",Vertex,&(Su[k])); 00169 cgp_sol_write(F,B,Z[k],"MomentumY",Vertex,&(Sv[k])); 00170 cgp_sol_write(F,B,Z[k],"MomentumZ",Vertex,&(Sw[k])); 00171 cgp_array_write(F,B,Z[k],"phi",Vertex); 00172 } 00173 00174 double T0,T1; 00175 double t0,t1; 00176 00177 MPI_Barrier(MPI_COMM_WORLD); 00178 T0 = MPI_Wtime(); 280

00179 00180 MPI_Barrier(MPI_COMM_WORLD); 00181 t0 = MPI_Wtime(); 00182 for(k=0;k

00183 int min = subzones[k]*Nl; 00184 int max = (subzones[k]+1)*Nl-1; 00185 00186 cgp_coord_write_data(F,B,Z[zones[k]],Cx[zones[k]],&min,&max,x); 00187 cgp_coord_write_data(F,B,Z[zones[k]],Cy[zones[k]],&min,&max,y); 00188 cgp_coord_write_data(F,B,Z[zones[k]],Cz[zones[k]],&min,&max,z); 00189 } 00190 MPI_Barrier(MPI_COMM_WORLD); 00191 t1 = MPI_Wtime(); 00192 if(comm_rank==0) { 00193 printf("Coords n"); \ 00194 printf(" tTime=%lf n",comm_rank,t1-t0); \ \ 00195 printf(" tBandwidth=%lf n",comm_rank,3.0*data_size/(t1-t0)); \ \ 00196 } 00197 00198 MPI_Barrier(MPI_COMM_WORLD); 00199 t0 = MPI_Wtime(); 00200 for(k=0;k

00201 int min = subzones[k]*Nl; 00202 int max = (subzones[k]+1)*Nl-1; 00203 00204 cgp_sol_write_data(F,B,Z[zones[k]],Su[zones[k]],&min,&max,u); 00205 cgp_sol_write_data(F,B,Z[zones[k]],Sv[zones[k]],&min,&max,v); 00206 cgp_sol_write_data(F,B,Z[zones[k]],Sw[zones[k]],&min,&max,w); 00207 } 00208 MPI_Barrier(MPI_COMM_WORLD); 00209 t1 = MPI_Wtime(); 00210 if(comm_rank==0) { 00211 printf("Solutions n"); \ 00212 printf(" tTime=%lf n",t1-t0); \ \ 00213 printf(" tBandwidth=%lf n",3.0*data_size/(t1-t0)); \ \ 281

00214 } 00215 00216 MPI_Barrier(MPI_COMM_WORLD); 00217 t0 = MPI_Wtime(); 00218 for(k=0;k

00219 int min = subzones[k]*Nl; 00220 int max = (subzones[k]+1)*Nl-1; 00221 00222 cgp_array_write_data(F,B,Z[zones[k]],"phi",&min,&max,h); 00223 } 00224 MPI_Barrier(MPI_COMM_WORLD); 00225 t1 = MPI_Wtime(); 00226 if(comm_rank==0) { 00227 printf("Arrays n"); \ 00228 printf(" tTime=%lf n",t1-t0); \ \ 00229 printf(" tBandwidth=%lf n",data_size/(t1-t0)); \ \ 00230 } 00231 00232 MPI_Barrier(MPI_COMM_WORLD); 00233 t0 = MPI_Wtime(); 00234 for(k=0;k

00235 int min = subzones[k]*Nl; 00236 int max = (subzones[k]+1)*Nl-1; 00237 00238 min++; 00239 max++; 00240 cgp_section_write_data(F,B,Z[zones[k]],E[zones[k]],min,max,e); 00241 } 00242 MPI_Barrier(MPI_COMM_WORLD); 00243 t1 = MPI_Wtime(); 00244 if(comm_rank==0) { 00245 printf("Elements n"); \ 00246 printf(" tTime=%lf n",t1-t0); \ \ 00247 printf(" tBandwidth=%lf n",((double) sizeof(int))/((double) sizeof(double)) \ \ *data_size/(t1-t0)); 282

00248 } 00249 00250 MPI_Barrier(MPI_COMM_WORLD); 00251 t0 = MPI_Wtime(); 00252 cgp_close(F); 00253 MPI_Barrier(MPI_COMM_WORLD); 00254 t1 = MPI_Wtime(); 00255 if(comm_rank==0) printf("Close_Time=%lf n",t1-t0); \ 00256 00257 MPI_Barrier(MPI_COMM_WORLD); 00258 T1 = MPI_Wtime(); 00259 00260 if(comm_rank==0) { 00261 #printf("Total n"); \ 00262 printf("Total Time=%lf n",T1-T0); \ 00263 printf("Total Bandwidth=%lf n",(6.0+((double) sizeof(int))/((double) sizeof \ (double)))*data_size/(T1-T0)); 00264 } 00265 00266 finalize(); 00267 00268 return 0; 00269 }