Tools for Data Analysis

DATA PROCESSING AND ANALYSIS TOOLS FOR DATA ANALYSIS

13 NOV 2018 I SABINE SCHRÖDER, IEK-8 OUTLINE 1. Motivation 2. Tools and data standards 3. Commands, Interpreter, Programming: When to use what? 4. Tools in command line operators/viewers 5. Tools in interpreted languages 6. Tools in compiled languages 7. Summary

13 November 2018 Sabine Schröder Tools for data analysis MOTIVATION

13 November 2018 Sabine Schröder Tools for data analysis TOOLS AND DATA STANDARDS

tool1 A chosen data format influences the variety tool2 of available tools.

standard tool3 Sticking to data standards enhances the data availability of reusable tools. format tool4 Keep up to date: Not only data formats … develop, but also tools!

tooln

13 November 2018 Sabine Schröder Tools for data analysis COMMANDS, INTERPRETERS, COMPILED PROGRAMS

• Ease of learning command line tool: interpreter: programming: > print_helloWorld >>> print("Hello, PROGRAM Hello world!") WRITE(*,*) "Hello, world!" END PROGRAM Hello

13 November 2018 Sabine Schröder Tools for data analysis COMMANDS, INTERPRETERS, COMPILED PROGRAMS

• Ease of learning

• Getting up and running command line tool: interpreter: programming: • download • download and • license compiler installation of • install compiler interpreter • learn about compiler • learn syntax of • learn syntax of interpreter programming language • after programming, compile, then run

13 November 2018 Sabine Schröder Tools for data analysis COMMANDS, INTERPRETERS, COMPILED PROGRAMS

• Ease of learning

• Getting up and running

• Speed: a productivity vs. performance tradeoff command line tool: interpreter: programming: > ncwa --no_tmp_fl -y >>> from netCDF4 import PROGRAM Maxi max -v tpot test.nc Dataset USE netcdf max_tpot.nc >>> rootgrp = st=nf90_open("test.nc", Dataset("test.nc","r") NF90_NOWRITE, ncid) >>> print st=nf90_inq_varid(ncid, rootgrp.variables["tpot"][: "tpot", tpotId) ,:,:,:].max() st=nf90_get_var(ncid, tpotId, >>> rootgr.close() tpot) write(*,*) maxval(tpot) END PROGRAM Maxi 13 November 2018 Sabine Schröder Tools for data analysis COMMANDS, INTERPRETERS, COMPILED PROGRAMS

• Ease of learning

• Getting up and running

• Speed: a productivity vs. performance tradeoff command line tool: interpreter: programming: > ncwa --no_tmp_fl -y >>> from netCDF4 import PROGRAM Maxi max -v tpot test.nc Dataset USE netcdf max_tpot.nc >>> rootgrp = st=nf90_open("test.nc", Dataset("test.nc","r") NF90_NOWRITE, ncid) >>> print st=nf90_inq_varid(ncid,  0.16 s rootgrp.variables 0.05 [s" tpot"][: "tpot", tpotId) ,:,:,:].max() st=nf90_get_var( 0.03ncid s , tpotId, >>> rootgr.close() tpot) write(*,*) maxval(tpot) END PROGRAM Maxi 13 November 2018 Sabine Schröder Tools for data analysis COMMANDS, INTERPRETERS, COMPILED PROGRAMS

• Ease of learning

• Getting up and running

• Speed: a productivity vs. performance tradeoff

• Scope of use

• Requirements

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS/VIEWERS NCDUMP  NCGEN From CDL to Header • netCDF-3 • netCDF-4 • C/F77/JAVA program

Variable

Record

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE VIEWERS NCVIEW

http://meteora.ucsd.edu/~pierce/ncview_home_page.html

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE VIEWERS PANOPLY

https://www.giss.nasa.gov/tools/panoply/

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS NCO (NETCDF OPERATORS) http://nco.sourceforge.net/

• standalone, command-line programs to . derive new fields . compute statistics . hyperslab . manipulate metadata . regrid • input: . netCDF, HDF, DAP . flat files . GODAD (Group-Oriented Data Analysis and Distribution)

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS NCO (NETCDF OPERATORS) ncap2 (arithmetic processor (version 2)) ncap2 ncatted (attribute editor) arithmetically processes netCDF files ncbo (binary operator) instructions from command line ncclimo (climatology generator) or from file ncecat (ensemble concatenator) treats missing values nces (ensemble statistics) definition of dimensions possible ncflint (file interpolator) can link to the GNU Scientific Library (GSL) ncks (kitchen sink) can create derived fields ncpdq (permute dimensions quickly) ncra (record average) example: ncrcat (record concatenator) ncap2 -v -s 'a=3;b=4;c=sqrt(a^2+b^2)' in.nc ncremap (remapper) out.nc ncrename (renamer) ncwa (weighted average)

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS NCO (NETCDF OPERATORS) ncap2 (arithmetic processor (version 2)) ncatted ncatted (attribute editor) append, create, delete, modify, and overwrite ncbo (binary operator) attributes ncclimo (climatology generator) ncecat (ensemble concatenator) example: nces (ensemble statistics) ncatted -a history,global,a,c,'Data version ncflint (file interpolator) 2.0\n' in.nc ncks (kitchen sink) ncpdq (permute dimensions quickly) ncra (record average) ncrcat (record concatenator) ncremap (remapper) ncrename (renamer) ncwa (weighted average)

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS NCO (NETCDF OPERATORS) ncap2 (arithmetic processor (version 2)) ncbo ncatted (attribute editor) four operations: ncbo (binary operator) Addition, Subtraction, Multiplication, Division ncclimo (climatology generator) ncecat (ensemble concatenator) example: nces (ensemble statistics) ncbo --op_typ=sub 86_0112.nc ncflint (file interpolator) 85_0112.nc 86m85_0112.nc ncks (kitchen sink) ncpdq (permute dimensions quickly) ncra (record average) ncrcat (record concatenator) ncremap (remapper) ncrename (renamer) ncwa (weighted average)

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS NCO (NETCDF OPERATORS) ncap2 (arithmetic processor (version 2)) ncclimo ncatted (attribute editor) • Climatology modes: ncbo (binary operator) annual, monthly, daily ncclimo (climatology generator) • seasons: ncecat (ensemble concatenator) jfm,amj,jas,ond,on,fm,djf,mam,jja,son,ann nces (ensemble statistics) • number of timesteps-per-day in output ncflint (file interpolator) • automatic filename creation and splitting ncks (kitchen sink) of output files ncpdq (permute dimensions quickly) ncra (record average) example: ncrcat (record concatenator) ncclimo -C ann -m cism -h h -c caseid ncremap (remapper) -s 1851-e 1900 -i drc_in -o drc_out ncrename (renamer) ncwa (weighted average)

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS NCO (NETCDF OPERATORS) ncap2 (arithmetic processor (version 2)) ncecat ncatted (attribute editor) create one output file: ncbo (binary operator) • Record Aggregation – NetCDF3 ncclimo (climatology generator) • Group Aggregation – NetCDF4 ncecat (ensemble concatenator) nces (ensemble statistics) example: ncflint (file interpolator) ncecat -u realization 85_0[1-5].nc 85.nc ncks (kitchen sink) ncpdq (permute dimensions quickly) ncra (record average) ncrcat (record concatenator) ncremap (remapper) ncrename (renamer) ncwa (weighted average)

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS NCO (NETCDF OPERATORS) ncap2 (arithmetic processor (version 2)) nces ncatted (attribute editor) gridpoint statistics on variables across ncbo (binary operator) • an ensemble of input-files ncclimo (climatology generator) • input groups within each file ncecat (ensemble concatenator) statistics: nces (ensemble statistics) avg Mean value sqravg Square of the mean ncflint (file interpolator) avgsqr Mean of sum of squares max Maximum value min Minimum value mabs Maximum absolute value ncks (kitchen sink) mebs Mean absolute value mibs Minimum absolute value ncpdq (permute dimensions quickly) rms Root-mean-square (normalized by N) ncra (record average) rmssdn Root-mean square (normalized by N-1) sqrt Square root of the mean tabs Sum of absolute values ncrcat (record concatenator) ttl Sum of values ncremap (remapper) example: ncrename (renamer) nces 85_0[1-5].nc 85.nc ncwa (weighted average)

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS NCO (NETCDF OPERATORS) ncap2 (arithmetic processor (version 2)) ncflint ncatted (attribute editor) linear combination of input files: ncbo (binary operator) • weighted average ncclimo (climatology generator) • normalized weighted average ncecat (ensemble concatenator) • interpolation nces (ensemble statistics) ncflint (file interpolator) example: ncks (kitchen sink) ncflint -i time,86 85.nc 87.nc 86.nc ncpdq (permute dimensions quickly) ncra (record average) ncrcat (record concatenator) ncremap (remapper) ncrename (renamer) ncwa (weighted average)

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS NCO (NETCDF OPERATORS) ncap2 (arithmetic processor (version 2)) ncks ncatted (attribute editor) file creation and conversion with special ncbo (binary operator) features: ncclimo (climatology generator) • extract ncecat (ensemble concatenator) • hyperslab nces (ensemble statistics) • multi-slab ncflint (file interpolator) • sub-set ncks (kitchen sink) • translate ncpdq (permute dimensions quickly) ncra (record average) example: ncrcat (record concatenator) ncks -d time,5 -d lat,,0.0 -d lon,260.0,45.0 ncremap (remapper) -d lev,1000.0 in.nc out.nc ncrename (renamer) ncwa (weighted average)

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS NCO (NETCDF OPERATORS) ncap2 (arithmetic processor (version 2)) ncpdq ncatted (attribute editor) two distinct functions: ncbo (binary operator) • packing (pdq: Pack Data Quietly) ncclimo (climatology generator) • dimension permutation (pdq:Permute ncecat (ensemble concatenator) Dimensions Quickly) nces (ensemble statistics) ncflint (file interpolator) example: ncks (kitchen sink) packing: ncpdq (permute dimensions quickly) ncpdq in.nc out.nc ncra (record average) dimension permuation: ncrcat (record concatenator) ncpdq -a lon,-lat in.nc out.nc ncremap (remapper) ncrename (renamer) ncwa (weighted average)

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS NCO (NETCDF OPERATORS) ncap2 (arithmetic processor (version 2)) ncra ncatted (attribute editor) computes statistics of record variables across ncbo (binary operator) an arbitrary number of input-files ncclimo (climatology generator) statistics: ncecat (ensemble concatenator) avg Mean value sqravg Square of the mean nces (ensemble statistics) avgsqr Mean of sum of squares max Maximum value min Minimum value mabs Maximum absolute value ncflint (file interpolator) mebs Mean absolute value mibs Minimum absolute value ncks (kitchen sink) rms Root-mean-square (normalized by N) rmssdn Root-mean square (normalized by N-1) ncpdq (permute dimensions quickly) sqrt Square root of the mean tabs Sum of absolute values ncra (record average) ttl Sum of values ncrcat (record concatenator) example: ncremap (remapper) ncra -d time,11,13 85.nc 86.nc 87.nc ncrename (renamer) 8512_8602.nc ncwa (weighted average)

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS NCO (NETCDF OPERATORS) ncap2 (arithmetic processor (version 2)) ncrcat ncatted (attribute editor) concatenates record variables across an ncbo (binary operator) arbitrary number of input-files ncclimo (climatology generator) ncecat (ensemble concatenator) example: nces (ensemble statistics) ncrcat 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc ncflint (file interpolator) ncks (kitchen sink) ncpdq (permute dimensions quickly) ncra (record average) ncrcat (record concatenator) ncremap (remapper) ncrename (renamer) ncwa (weighted average)

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS NCO (NETCDF OPERATORS) ncap2 (arithmetic processor (version 2)) ncremap ncatted (attribute editor) remap to grid specified by a map file (weight- ncbo (binary operator) file), grid destination file, or a template file ncclimo (climatology generator) (data file on destination grid) ncecat (ensemble concatenator) nces (ensemble statistics) example: ncflint (file interpolator) ncremap -d dst.nc in.nc out.nc ncks (kitchen sink) ncpdq (permute dimensions quickly) ncra (record average) ncrcat (record concatenator) ncremap (remapper) ncrename (renamer) ncwa (weighted average)

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS NCO (NETCDF OPERATORS) ncap2 (arithmetic processor (version 2)) ncrename ncatted (attribute editor) renames dimensions, variables, attributes, ncbo (binary operator) and groups ncclimo (climatology generator) ncecat (ensemble concatenator) example: nces (ensemble statistics) ncrename -d lon,longitude -v lon,longitude ncflint (file interpolator) in.nc ncks (kitchen sink) ncpdq (permute dimensions quickly) ncra (record average) ncrcat (record concatenator) ncremap (remapper) ncrename (renamer) ncwa (weighted average)

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS NCO (NETCDF OPERATORS) ncap2 (arithmetic processor (version 2)) ncwa ncatted (attribute editor) computes statistics on variables in a single ncbo (binary operator) file over arbitrary dimensions, with options to ncclimo (climatology generator) specify weights, masks, and normalization ncecat (ensemble concatenator) statistics: nces (ensemble statistics) avg Mean value sqravg Square of the mean ncflint (file interpolator) avgsqr Mean of sum of squares max Maximum value min Minimum value mabs Maximum absolute value ncks (kitchen sink) mebs Mean absolute value mibs Minimum absolute value ncpdq (permute dimensions quickly) rms Root-mean-square (normalized by N) ncra (record average) rmssdn Root-mean square (normalized by N-1) sqrt Square root of the mean tabs Sum of absolute values ncrcat (record concatenator) ttl Sum of values ncremap (remapper) example: ncrename (renamer) ncwa -y max -v tpot test.nc max_tpot.nc ncwa (weighted average)

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS CDO (CLIMATE DATA OPERATORS) https://code.mpimet.mpg.de/projects/cdo

• collection of command-line operators to . manipulate . analyse climate and NWP model data (more than 600 operators available)

• input: . GRIB 1/2 . netCDF 3/4 . SERVICE, EXTRA and IEG

13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS CDO (CLIMATE DATA OPERATORS) Collections: • Dataset information • Information • Comparison of two datasets • File operations • Number of • Selection/Conditional selection parameters/levels/years/mont • Comparison hs/dates/timesteps/… • Modification • Show • Arithmetic standard_names/attributes/le • Statistical values/Correlation/Regression/EOF vels/date information/… • Interpolation • Grid description • Transformation • Import/Export • Miscellaneous/NCL

CDO has a Fortran interface: CDI 13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS CDO (CLIMATE DATA OPERATORS) Collections: • CopyDataset datasets information • Information • ConcatenateComparison ofdatasets two datasets • File operations • ReplaceNumber ofvariables • Selection/Conditional selection • Mergeparameters datasets/levels /years/mont • Comparison • Spliths/dates by /timesteps/… • Modification • codenumberShow /levels/grids/hour • Arithmetic sstandard_names/days/…/time selection/attributes /le • Statistical values/Correlation/Regression/EOF • Distributevels/date /informationcollect horizontal/… • Interpolation • gridGrid description • Transformation • Import/Export • Miscellaneous/NCL

CDO has a Fortran interface: CDI 13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS CDO (CLIMATE DATA OPERATORS) Collections: •• DatasetSelect/delete information fields • Information •• ComparisonSelect of two datasets • File operations • Numberparameters of /levels/grids • Selection/Conditional selection • parametersSelect /levels/years/mont • Comparison hstimesteps/dates/timesteps/hours/days/…/… • Modification •• ShowSelect lat-lon-box/index-box • Arithmetic • standard_namesSelect/delete grid/attributes cells /le • Statistical values/Correlation/Regression/EOF • velsResample/date information grid /… • Interpolation •• GridUse maskdescription file with conditions • Transformation (ifthen/ifnothen/ifthenelse/…) • Import/Export • Miscellaneous/NCL

CDO has a Fortran interface: CDI 13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS CDO (CLIMATE DATA OPERATORS) Collections: •eqDataset Equal information • Information •neComparison Not equal of two datasets • File operations •le NumberLess equal of • Selection/Conditional selection lt Lessparameters than /levels/years/mont • Comparison gehs Greater/dates /equaltimesteps /… • Modification •gt ShowGreater than • Arithmetic eqcstandard_names Equal constant/ attributes/le • Statistical values/Correlation/Regression/EOF necvels Not/date equal information constant/… • Grid description • Interpolation lec Less equal constant • Transformation ltc Less than constant • Import/Export gec Greater equal constant • Miscellaneous/NCL gtc Greater than constant

CDO has a Fortran interface: CDI 13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS CDO (CLIMATE DATA OPERATORS) Collections: •• DatasetModify information • Information • Comparison. metadata of two datasets • File operations • Number. fields of or part of a field • Selection/Conditional selection • parametersin a dataset/ levels/years/mont • Comparison • hsSet/dates attributes/date/time/timesteps/… • Modification • Showbounds/grids/levels/missing • Arithmetic standard_namesvalue/valid range/ attributes/le • Statistical values/Correlation/Regression/EOF • velsInvert/date latitudes/levels information/… • Grid description • Interpolation • Shift x/y • Transformation • Mask regions/boxes • Import/Export • Miscellaneous/NCL

CDO has a Fortran interface: CDI 13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS CDO (CLIMATE DATA OPERATORS) Collections: •• Datasetarithmetically information process datasets • Information • Comparisonvia expression of (twoscript datasets)s • File operations •• Numberoperators of • Selection/Conditional selection parameters(abs,sqrt,acos,log10,…)/levels/years/ mont • Comparison • hsOperate/dates /ontimesteps two fields/… • Modification • Show(add/sub /min/…) • Arithmetic • standard_namesMonthly/multiyear/attributes /le • Statistical values/Correlation/Regression/EOF vels[hourly/date/daily information/monthly//…seasonal ] • Grid description • Interpolation arithmetics (add, sub, mul, div) • Transformation • Days per month (add, sub, …) • Import/Export • Miscellaneous/NCL

CDO has a Fortran interface: CDI 13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS CDO (CLIMATE DATA OPERATORS) Collections: •• DatasetCumulative information values • Information •• ComparisonEnsemble/field of /zonal/meridional/two datasets • File operations • Numbergridbox/ verticalof /time • Selection/Conditional selection parametersselection/running/levels/time//yearshourly/mont/ • Comparison hsmonthly/dates//yearlytimesteps/seasonal/… / • Modification • Showmultiyear statistics • Arithmetic • standard_namesCorrelation in grid/attributes/time /le • Statistical values/Correlation/Regression/EOF• velsCovariance/date information in grid/time/… • Grid description • Interpolation • Regression • Transformation • Trends • Import/Export • EOF calculations • Miscellaneous/NCL

CDO has a Fortran interface: CDI 13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS CDO (CLIMATE DATA OPERATORS) Collections: •• Datasethorizontal information fields to a new grid • Information •• Comparisoninterpolation ofof two3D variablesdatasets • File operations • Numberfrom hybrid of model levels to • Selection/Conditional selection parametersheight or pressure/levels/ yearslevels/mont • Comparison • hsinterpolation/dates/timesteps in time/… between • Modification • Showtime steps and years • Arithmetic • standard_nameslinear/bilinear/cubic/attributes interpolation/le • Statistical values/Correlation/Regression/EOF • velsnearest/date neighbor/distance information/… - • Grid description • Interpolation weighted average remapping • Transformation • Import/Export • Miscellaneous/NCL

CDO has a Fortran interface: CDI 13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS CDO (CLIMATE DATA OPERATORS) Collections: • spectral to gridpoint • Information and vice versa • File operations • divergence and vorticity to U • Selection/Conditional selection and V wind • Comparison and vice versa • Modification • D and V to velocity potential and • Arithmetic stream function • Statistical values/Correlation/Regression/EOF • Interpolation • Transformation • Import/Export • Miscellaneous/NCL

CDO has a Fortran interface: CDI 13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS CDO (CLIMATE DATA OPERATORS) Collections: import and export data files which • Information cannot be read or written directly • File operations with CDO • Selection/Conditional selection • Comparison • Modification • Arithmetic • Statistical values/Correlation/Regression/EOF • Interpolation • Transformation • Import/Export • Miscellaneous/NCL

CDO has a Fortran interface: CDI 13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN COMMAND LINE OPERATORS CDO (CLIMATE DATA OPERATORS) Collections: • Filtering (band/low/highpass) • Information • Create pressure/temperature • File operations values for hydrostatic • Selection/Conditional selection atmosphere • Comparison • Potential temperature to in-situ • Modification temperature (and vice versa) • Arithmetic • Histogram • Statistical values/Correlation/Regression/EOF • Frost/strong wind/strong • Interpolation breeze/strong gale/hurrican • Transformation days • Import/Export • GrADS data descriptor file • Miscellaneous/NCL • ECHAM post processor

CDO has a Fortran interface: CDI 13 November 2018 Sabine Schröder Tools for data analysis TOOLS IN INTERPRETED LANGUAGES NCL (NCAR COMMAND LANGUAGE)

ncl 0> a=addfile("test.nc","r") (also OPeNDAP possible)