Understanding the folder structure for ordered products

Land Monitoring Tutorial Copernicus Land Monitoring Service – Global component

Prepared by: T. Jacobs (VITO) Aim of this tutorial Global Land Operations

When products are ordered using the legacy product access portal, they are prepared and delivered on FTP, using a specific folder structure.

This tutorial shows how to understand and work with that folder structure. Pre-requisites Global Land Operations

A utility for the file downloads

Examples: • FileZilla • Open source WGET utility or its Windows-equivalent WinWget • Curl • LFTP

A utility for unzipping the received ZIPs: • On Linux, the standard unzip will suffice • On Windows, modern systems can un-compress zip-files directly in the File Explorer. Alternatively, the use of 7-zip is recommended. Terminology Global Land Operations A product collection is a group of related products, typically consisting of a Near-Real Time produced product of a specific version and its consistent archive (time series), where available. Example: LAI 300m version 1 collection, consists of all the 300m resolution LAI products produced with processing algorithm version 1.

Each product typically combines • a data file (often netCDF4 format) at full resolution, with metadata attributes inside • a sub-sampled quicklook or browse image (for cataloguing and quick inspections) • a metadata record (INSPIRE-compliant XML)

An order is a request for delivery of one or multiple order items. Each order item is a copy of a specific product from a collection, in native or custom format (e.g. area subset in GeoTIFF format) and delivered on the Global Land FTP server or the portal itself.

Products and collections have a unique identifier that is provided in the metadata. Orders have a unique name. Order item names are unique within the order. Organization of the ordered products Global Land Operations Ordered products are made available in the following folder structure:

ordername └─ product identifier └─ product file(s)

Whereby • the ordername starts with C00 or M00 for catalogue orders and S00 for subscriptions. • the product identifier resembles the product filename, minus the c_gls_ prefix and filename extension (.nc, .tiff) • the product file(s) are • For custom orders: a ZIP file, itself containing one YYYYMMDD folder for the product's nominal date. • For orders in native format: the data file (.nc), quicklook (.tiff) and metadata (.xml) files Organization of the ordered products Global Land Operations

Example:

C0063555/ └─ NDVI300_202002010000_GLOBE_PROBAV_V1.0.1 └─ c_gls_NDVI300_202002010000_GLOBE_PROBAV_V1.0.1.zip

with the zip containing

c_gls_NDVI300_202002010000_GLOBE_PROBAV_V1.0.1.zip └─ 20200201 └─ c_gls_NDVI300-NDVI_202002010000_FRANCE_PROBAV_V1.0.1.tiff Polling subscription products Global Land Operations For subscription orders, the order name folder persists, while product identifier folders and below will change with every additional product.

Rather than parsing the emails for the hyperlinks they contain (prone to error, SPAM trouble), it is better to poll the FTP server recursively from the top-level folder (ordername) onwards.

S0063555/ └─ NDVI300_202002010000_GLOBE_PROBAV_V1.0.1 FileZilla: point to top folder and └─ … download it

With 10 days later adding Wget: options –r and –l S0063555/ └─ NDVI300_202002110000_GLOBE_PROBAV_V1.0.1 LFTP: use mirror command └─ … Curl: specify wildcard or multiple URLs on command line Bypassing the folder structure in the download Global Land Operations Most of the file transfer utilities have options to limit or enable the creation of local directories according to the folders in the remote Global Land FTP server.

FileZilla: point to the lowest level folder (or files) and download

Wget: combine –r (recursive) with options such as –nd (no local directories at all), -nH (no local directory with hostname) or --cut-dirs (do not replicate locally a number of directory levels)

Lftp mget command: does not create local directories, use -d to enable that mirror command: try using --no-empty-dirs

Curl: does not create local directories, use --create-dirs and -o to enable Bypassing the folder structure in the de-compression of the ZIP files Global Land Operations An order item can contain multiple files (e.g. one GeoTIFF per band). For orders of large time series, this can quickly run up the total number of files and slow down the directory navigation when all files are kept in the same folder.

The YYYYMMDD folder in the ZIP file can be handy for organizing the time series and avoiding such trouble.

Should you still desire to omit the YYYYMMDD folder when you un-compress the ZIP file, try the following options:

On Linux : unzip’s –j option (junk directories)

On Windows: 7-zip’s e command extracts without creating folders, x command extracts with the full paths.