NCO User Guide A suite of netCDF operators Edition 5.0.2-alpha07, for NCO Version 5.0.2-alpha07 September 2021 by Charlie Zender Departments of Earth System Science and Computer Science University of California, Irvine Copyright c 1995{2021 Charlie Zender. This is the first edition of the NCO User Guide, and is consistent with version 2 of texinfo.tex. Published by Charlie Zender Department of Earth System Science 3200 Croul Hall University of California, Irvine Irvine, CA 92697-3100 USA Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. The license is available online at http://www.gnu.org/copyleft/fdl.html We gratefully acknowledge support for NCO development and maintenance provided by these institutions and programs: DOE ACME DE-SC0012998, LLNL-B625903, LLNL-B632442, NASA ACCESS NNX12AF48A and NNX14AH55A, and NSF SEI IIS-0431203, AGS-1541031, and OAC-2004993. This research was supported as part of the Energy Exascale Earth System Model (E3SM) project, formerly known as Accelerated Climate Modeling for Energy (ACME), funded by the U.S. Department of Energy, Office of Science, Office of Biological and Environmental Research. This material is based upon work supported by the National Science Foundation. The original author of this software, Charlie Zender, wants to improve it with the help of your suggestions, improvements, bug-reports, and patches. Charlie Zender <surname at uci dot edu> (yes, my surname is zender) Department of Earth System Science 3200 Croul Hall University of California, Irvine Irvine, CA 92697-3100 i Table of Contents Foreword ::::::::::::::::::::::::::::::::::::::::::::: 1 Summary :::::::::::::::::::::::::::::::::::::::::::: 5 1 Introduction ::::::::::::::::::::::::::::::::::::: 7 1.1 Availability::::::::::::::::::::::::::::::::::::::::::::::::::::: 7 1.2 How to Use This Guide ::::::::::::::::::::::::::::::::::::::::: 7 1.3 Operating systems compatible with NCO :::::::::::::::::::::::: 8 1.3.1 Compiling NCO for Microsoft Windows OS :::::::::::::::: 10 1.4 Symbolic Links :::::::::::::::::::::::::::::::::::::::::::::::: 10 1.5 Libraries :::::::::::::::::::::::::::::::::::::::::::::::::::::: 11 1.6 netCDF2/3/4 and HDF4/5 Support ::::::::::::::::::::::::::: 12 1.7 Help Requests and Bug Reports ::::::::::::::::::::::::::::::: 15 2 Operator Strategies :::::::::::::::::::::::::::: 17 2.1 Philosophy :::::::::::::::::::::::::::::::::::::::::::::::::::: 17 2.2 Climate Model Paradigm :::::::::::::::::::::::::::::::::::::: 17 2.3 Temporary Output Files ::::::::::::::::::::::::::::::::::::::: 17 2.4 Appending Variables :::::::::::::::::::::::::::::::::::::::::: 19 2.5 Simple Arithmetic and Interpolation ::::::::::::::::::::::::::: 19 2.6 Statistics vs. Concatenation ::::::::::::::::::::::::::::::::::: 20 2.6.1 Concatenators ncrcat and ncecat :::::::::::::::::::::::: 20 2.6.2 Averagers nces, ncra, and ncwa :::::::::::::::::::::::::: 21 2.6.3 Interpolator ncflint ::::::::::::::::::::::::::::::::::::: 21 2.7 Large Numbers of Files :::::::::::::::::::::::::::::::::::::::: 21 2.8 Large Datasets :::::::::::::::::::::::::::::::::::::::::::::::: 23 2.9 Memory Requirements::::::::::::::::::::::::::::::::::::::::: 24 2.9.1 Single and Multi-file Operators ::::::::::::::::::::::::::: 24 2.9.2 Memory for ncap2 :::::::::::::::::::::::::::::::::::::::: 26 2.10 Performance:::::::::::::::::::::::::::::::::::::::::::::::::: 26 3 Shared Features :::::::::::::::::::::::::::::::: 29 3.1 Internationalization:::::::::::::::::::::::::::::::::::::::::::: 29 3.2 Metadata Optimization :::::::::::::::::::::::::::::::::::::::: 29 3.3 OpenMP Threading ::::::::::::::::::::::::::::::::::::::::::: 30 3.4 Command Line Options ::::::::::::::::::::::::::::::::::::::: 31 3.4.1 Truncating Long Options ::::::::::::::::::::::::::::::::: 32 3.4.2 Multi-arguments :::::::::::::::::::::::::::::::::::::::::: 32 3.5 Sanitization of Input :::::::::::::::::::::::::::::::::::::::::: 33 3.6 Specifying Input Files ::::::::::::::::::::::::::::::::::::::::: 34 3.7 Specifying Output Files:::::::::::::::::::::::::::::::::::::::: 37 ii 3.8 Accessing Remote Files :::::::::::::::::::::::::::::::::::::::: 37 3.8.1 OPeNDAP :::::::::::::::::::::::::::::::::::::::::::::::: 39 3.9 Retaining Retrieved Files :::::::::::::::::::::::::::::::::::::: 42 3.10 File Formats and Conversion ::::::::::::::::::::::::::::::::: 42 3.10.1 File Formats :::::::::::::::::::::::::::::::::::::::::::: 43 3.10.2 Determining File Format :::::::::::::::::::::::::::::::: 44 3.10.3 File Conversion:::::::::::::::::::::::::::::::::::::::::: 45 3.10.4 Autoconversion :::::::::::::::::::::::::::::::::::::::::: 46 3.11 Large File Support ::::::::::::::::::::::::::::::::::::::::::: 47 3.12 Subsetting Files :::::::::::::::::::::::::::::::::::::::::::::: 48 3.13 Subsetting Coordinate Variables :::::::::::::::::::::::::::::: 53 3.14 Group Path Editing :::::::::::::::::::::::::::::::::::::::::: 53 3.14.1 Deletion, Truncation, and Flattening of Groups :::::::::: 54 3.14.2 Moving Groups :::::::::::::::::::::::::::::::::::::::::: 56 3.14.3 Dismembering Files:::::::::::::::::::::::::::::::::::::: 57 3.14.4 Checking CF-compliance::::::::::::::::::::::::::::::::: 60 3.15 C and Fortran Index conventions ::::::::::::::::::::::::::::: 63 3.16 Hyperslabs ::::::::::::::::::::::::::::::::::::::::::::::::::: 63 3.17 Stride :::::::::::::::::::::::::::::::::::::::::::::::::::::::: 66 3.18 Record Appending ::::::::::::::::::::::::::::::::::::::::::: 67 3.19 Subcycle ::::::::::::::::::::::::::::::::::::::::::::::::::::: 68 3.20 Interleave :::::::::::::::::::::::::::::::::::::::::::::::::::: 69 3.21 Multislabs:::::::::::::::::::::::::::::::::::::::::::::::::::: 71 3.22 Wrapped Coordinates :::::::::::::::::::::::::::::::::::::::: 74 3.23 Auxiliary Coordinates :::::::::::::::::::::::::::::::::::::::: 74 3.24 Grid Generation:::::::::::::::::::::::::::::::::::::::::::::: 77 3.25 Regridding ::::::::::::::::::::::::::::::::::::::::::::::::::: 86 Renormalization ::::::::::::::::::::::::::::::::::::::::::::::::: 87 Regridder Options Table::::::::::::::::::::::::::::::::::::::::: 90 3.26 Climatology and Bounds Support ::::::::::::::::::::::::::::: 97 3.27 UDUnits Support :::::::::::::::::::::::::::::::::::::::::::: 98 3.28 Rebasing Time Coordinate :::::::::::::::::::::::::::::::::: 101 3.29 Multiple Record Dimensions ::::::::::::::::::::::::::::::::: 102 3.30 Missing values::::::::::::::::::::::::::::::::::::::::::::::: 103 3.31 Chunking ::::::::::::::::::::::::::::::::::::::::::::::::::: 104 3.32 Compression :::::::::::::::::::::::::::::::::::::::::::::::: 111 3.32.1 Linear Packing ::::::::::::::::::::::::::::::::::::::::: 112 3.32.2 Precision-Preserving Compression::::::::::::::::::::::: 112 3.33 Deflation :::::::::::::::::::::::::::::::::::::::::::::::::::: 121 3.34 MD5 digests :::::::::::::::::::::::::::::::::::::::::::::::: 122 3.35 Buffer sizes:::::::::::::::::::::::::::::::::::::::::::::::::: 123 3.36 RAM disks :::::::::::::::::::::::::::::::::::::::::::::::::: 124 3.37 Unbuffered I/O ::::::::::::::::::::::::::::::::::::::::::::: 125 3.38 Packed data::::::::::::::::::::::::::::::::::::::::::::::::: 126 Standard Packing Algorithm:::::::::::::::::::::::::::::::::::: 126 Standard (Default) Unpacking Algorithm ::::::::::::::::::::::: 127 Non-Standard Packing and Unpacking Algorithms :::::::::::::: 127 Handling of Packed Data by Other Operators ::::::::::::::::::: 128 iii 3.39 Operation Types :::::::::::::::::::::::::::::::::::::::::::: 129 3.40 Type Conversion :::::::::::::::::::::::::::::::::::::::::::: 134 3.40.1 Automatic type conversion ::::::::::::::::::::::::::::: 134 3.40.2 Promoting Single-precision to Double ::::::::::::::::::: 136 3.40.3 Manual type conversion :::::::::::::::::::::::::::::::: 142 3.41 Batch Mode::::::::::::::::::::::::::::::::::::::::::::::::: 143 3.42 Global Attribute Addition ::::::::::::::::::::::::::::::::::: 143 3.43 History Attribute ::::::::::::::::::::::::::::::::::::::::::: 144 3.44 File List Attributes:::::::::::::::::::::::::::::::::::::::::: 145 3.45 CF Conventions ::::::::::::::::::::::::::::::::::::::::::::: 146 3.46 ARM Conventions ::::::::::::::::::::::::::::::::::::::::::: 150 3.47 Operator Version :::::::::::::::::::::::::::::::::::::::::::: 151 4 Reference Manual::::::::::::::::::::::::::::: 153 4.1 ncap2 netCDF Arithmetic Processor:::::::::::::::::::::::::: 154 4.1.1 Syntax of ncap2 statements:::::::::::::::::::::::::::::: 155 4.1.2 Expressions ::::::::::::::::::::::::::::::::::::::::::::: 156 4.1.3 Dimensions :::::::::::::::::::::::::::::::::::::::::::::: 159 4.1.4 Left hand casting :::::::::::::::::::::::::::::::::::::::: 160 4.1.5 Arrays and hyperslabs ::::::::::::::::::::::::::::::::::: 162 4.1.6 Attributes ::::::::::::::::::::::::::::::::::::::::::::::: 165 4.1.7 Value List ::::::::::::::::::::::::::::::::::::::::::::::: 167 4.1.8 Number literals:::::::::::::::::::::::::::::::::::::::::: 168 4.1.9 if statement ::::::::::::::::::::::::::::::::::::::::::::: 169 4.1.10 Print & String methods :::::::::::::::::::::::::::::::: 170 4.1.11 Missing values ncap2 ::::::::::::::::::::::::::::::::::: 173 4.1.12 Methods and functions ::::::::::::::::::::::::::::::::: 175 4.1.13 RAM variables:::::::::::::::::::::::::::::::::::::::::: 178 4.1.14 Where statement ::::::::::::::::::::::::::::::::::::::: 179 4.1.15 Loops :::::::::::::::::::::::::::::::::::::::::::::::::: 181 4.1.16