Visualization of Climate Data with ParaView DATAR: Data science Aspects and Tools in Atmospheric Research November 14, 2018 | Dr. Herwig Zilken November 14, 2018 Slide 1 Contents • Overview about ParaView • Loading Data • Important Visualization Techniques • Animating Data • ParaView Python Scripting November 14, 2018 Slide 2 What is ParaView? • General purpose open-source data analysis and visualization application (two- and three-dimensional data sets) built on top of VTK • Provides a comprehensive suite of visualization algorithms • Supports many different file formats for both loading and exporting data sets. • Supported platforms: Linux, Windows, Mac • Processing Modes: • Stand-alone mode • Client server configuration (in parallel) • Batch (Python scripting) • Locally installed at JSC: • Linux Workstation Group • JURECA Visualization Partition November 14, 2018 Slide 3 The History of ParaView • 2000: collaboration between Kitware Inc. and Los Alamos National Laboratory, funding provided by the US Department of Energy ASCI program • 2002: first release of ParaView • September 2005 - May 2007: development of Paraview 3 by Kitware, Sandia National Lab. and other partners user interface more user friendly quantitative analysis framework • June 2013: ParaView 4.0 more cohesive GUI controls better multiblock interaction In situ integration into simulation and other applications (Catalyst) • Recent Releases: (currently ParaView 5.6) new VR backend (multi screen 3D projection, VR devices) November 14, 2018 Slide 4 ParaView‘s Architecture ParaView pvpython ParaWeb Catalyst Custom App Client (GUI) Interface Layer (Qt Widgets, Python Wrappings) ParaView Server VTK OpenGL MPI IceT … more … November 14, 2018 Slide 5 Getting Started • ParaView can be downloaded at www.paraview.org • precompiled versions available for Mac OS, Windows (32-bit and 64-bit) and Linux (64-bit) • sources for individual installations (e.g. using the mpi- version tailored to your system) • ParaView client launches like most applications: • Windows: launcher located at the Start-Menu • Linux: execute paraview from a command prompt • Mac OS: open the application bundle that you installed November 14, 2018 Slide 6 Visualization Pipeline • concept of a visualization pipeline as implemented in VTK Source Filter 1 Filter2 initial data Data 1 modify the Data 2 modify the Data 3 input or data in some data in some generated way way Mapper Actor Mapper Actor Mapper Actor Generates Adjust the Generates Adjust the Generates Adjust the Function calls visible Function calls visible Function calls visible to graphic properties to graphic properties to graphic properties system system system Renderer Create images of data November 14, 2018 Slide 7 User Interface Menu Bar Toolbars Pipeline Browser 3D View Properties & Information Panel Advanced toggle: Shows/hides advanced controls Menu Bar: access the majority of features Toolbars: quick access to the most commonly used Pipeline Browser: Overview about the data processing pipeline Properties & Information Panel: Parameters of the selected object in the pipeline November 14, 2018 Slide 8 Scientific Data • a data describes WHAT values are located WHERE. • WHERE: the structure of the data 1. Geometry: defines the (3D) location 2. Topology: describes the connectivity of points (cells) Example: three points at (x1,y1,z1), (x2,y2,z2), (x3,y3,z3) forming a triangle • WHAT: the attributes (values) of the data, e.g. temperature, pressure, … • typically the data is discrete (not continuous), given at a set of points in 3D space. November 14, 2018 Slide 9 Data Types: Structured Data Uniform Rectilinear Grid (Image Data) (vtkImageData) Non-uniform Rectilinear Grid (Rectilinear Grid) (vtkRectilinearData) Structured (Curvilinear) Grid (vtkStructuredData) November 14, 2018 Slide 10 Data Types: Unstructured Data Polygonal Mesh (Poly Data) (vtkPolyData) Unstructured Grid (vtkUnstructuredGrid) Multi-Block (several data sets grouped together) Hierarchical Adaptive Mesh Refinement (AMR) November 14, 2018 Slide 11 Data Attributes • Data attributes at grid points (grid data, node data) or on cells (cell data, element data) • General data attributes: Scalar Vector Tensor (n x n matrix) • Attributes with special meaning for the visualization process: Surface normals (3D vector) 2D or 3D texture coordinates • Any number of variables can be defined at points and cells November 14, 2018 Slide 12 Loading Data November 14, 2018 Slide 13 Loading Data: NetCDF • Typically climate data is stored in NetCDF containers • There are some NetCDF readers in ParaView: – A generic netCDF reader (respecting Climate and Forecast convention, see http://cfconventions.org/) – A netCDF MPAS reader for MPAS data – A netCDF POP reader for (parallel) ocean program data – A netCDF CAM reader for reading CAM data – A CDI netCDF reader for unstructured ICON data sets (plugin) November 14, 2018 Slide 14 Loading Data: HDF5 + XDMF • NetCDF readers sometimes are unflexibel and maybe not right for your file format • Solution: convert your data to HDF5 and generate corresponding XDMF file (NEXT SLIDE) • Data then can be loaded by ParaView‘s xdfm reader • The xdmf file gives additional flexibility on loading: - hyperslab for volume of interest or data subsampling - join scalars to build vectors - a few simple calculations possible (e.g. add two values) How To Convert: • Sometimes NetCDF-file is already a HDF5 container, check by: ncdump –k foo.nc output „netCDF-4“ or „netCDF-4 classic model“ means HDF5 • Otherwise convert NetCDF to HDF5 via nccopy –k 4 foo3.nc foo4.h5 another example with compression nccopy –k 4 – d 1 foo3.nc foo4.h5 November 14, 2018 Slide 15 XDMF • Description of the structure of HDF5 datasets, e.g. what data defines coordinates, what data defines data values (attributes), … • Small XML file (light data) in addition to (large) data file (heavy data, typically HDF5) • light data: XML file containing XDFM language statements and references to datasets in heavy data • heavy data: HDF5 files (or binary) • http://www.xdmf.org November 14, 2018 Slide 16 Loading Data: HDF5 + XDMF • Alternatively NetCDF can be converted by self made, simple Python script • Advantage: – Only the variables relevant for visualization can be selectively converted – Any desired coordinate transformation can be pre calculated (e.g. from spherical to cartesian coordinates, or from pressure to absolute height) – Additional variables can be derived from existing (e.g. magnitude of vectors, relative/fractional values) – A subset of the domain (region of interest) can be selected – Recalculate values to be cell centered (e.g. staggered grid) November 14, 2018 Slide 17 Sample Copy Script: open files, copy metadata, copy selected variables, cut off 10 gridpoints from border import h5py import netCDF4 as nc import numpy as np variables = ["QNRAIN", "QNICE", "CLDFRA"] cutoff = 10 fin = nc.Dataset("foo.nc", "r") fout = h5py.File("foo.h5", 'w') # copy global meta data (attributes) for attr in fin.ncattrs(): fout.attrs[attr] = getattr(fin, attr) for var in variables: v=fin.variables[var] # generate shape of output variable taking cutoff into account outshape = [] for i in range(len(v.shape)): outshape.append(v.shape[i]) # cut off border outshape[-1] = outshape[-1] - 2*cutoff outshape[-2] = outshape[-2] - 2*cutoff dset = fout.create_dataset(var, tuple(outshape), dtype=np.float32) # copy variable meta data (attributes) for attr in v.ncattrs(): dset.attrs[attr] = getattr(v, attr) # copy content of variable fout[var][:, :, :, :] = v[:, :, cutoff:-cutoff, cutoff:-cutoff] fin.close() fout.close() November 14, 2018 Slide 18 Sample Copy Script: calculate scalar from vector (result = sqrt(U10^2+V10^2)) import h5py import netCDF4 as nc import numpy as np fin = nc.Dataset("foo.nc", "r") fout = h5py.File("foo.h5", 'w') U10 = fin.variables["U10"] V10 = fin.variables["V10"] resultName = "Result" dset = fout.create_dataset(resultName, U10.shape, dtype=np.float32, compression="gzip") fout[resultName][:, :, :] = np.sqrt(U10[:, :, :]*U10[:, :, :] + V10[:, :, :]*V10[:, :, :]) fin.close() fout.close() November 14, 2018 Slide 19 Sample script to generate cartesian coordinates from lon, lat, level import h5py import netCDF4 as nc import numpy as np fin = nc.Dataset("foo.nc", "r") fout = h5py.File("foo.h5", 'w') latIn = fin['/lat'] # 1D variable lonIn = fin['/lon'] # 1D variable levIn = fin['/lev_2'] # 1D variable xOut = fOut.create_dataset("/x", (levIn.shape[0], latIn.shape[0], lonIn.shape[0]), dtype=np.float32) yOut = fOut.create_dataset("/y", (levIn.shape[0], latIn.shape[0], lonIn.shape[0]), dtype=np.float32) zOut = fOut.create_dataset("/z", (levIn.shape[0], latIn.shape[0], lonIn.shape[0]), dtype=np.float32) lat = np.zeros((latIn.shape[0], lonIn.shape[0]), dtype = np.float32) lon = np.zeros((latIn.shape[0], lonIn.shape[0]), dtype = np.float32) for i in range(lonIn.shape[0]): lat[:, i] = latIn for i in range(latIn.shape[0]): lon[i, :] = lonIn (continued on next slide) November 14, 2018 Slide 20 Sample Script to generate cartesian coordinates from lon, lat, level (continued) (continued) x = np.zeros((latIn.shape[0], lonIn.shape[0]), dtype = np.float32) y = np.zeros((latIn.shape[0], lonIn.shape[0]), dtype = np.float32) z = np.zeros((latIn.shape[0], lonIn.shape[0]), dtype = np.float32) for i in range(levIn.shape[0]): print "calculating coordinates for level %d" % (i) height = (137.0 - levIn[i])*.5+150 x = height * np.cos(lat*np.pi/180)*np.cos(lon*np.pi/180) y = height * np.cos(lat*np.pi/180)*np.sin(lon*np.pi/180)