The Unified Model

NCAS Unified Model Introduction

Part I: Overview of the UM system

University of Reading, 13-15 April 2016 The Unified Model

• The UM is a numerical modelling system, developed by the UK Met Office, and used for operational weather forecasting and climate prediction.

• It is licensed to the UK academic community for research. There are also collaborations between the Met Office and the academic community for research and development (e.g. JWCRP, MOAP).

• It is used by forecast centres and climate agencies around the world.

2 Seamless modelling

• The same core model is used across spatial and temporal scales: – Weather to climate timescales – Global and regional models – High to low resolution (horizontal and vertical) – Various model heights

• It can be used in atmosphere only mode, or coupled to: – UM ocean vn6.6.3 or earlier only (e.g. HadGEM2) – NEMO ocean and CICE sea-ice via OASIS coupler – UKCA chemistry and aerosols – JULES land-surface

• It can also be used to perform idealised studies, e.g: – Single Column Model (SCM) – Aquaplanet

3 © Crown copyright Met Office 4 Global & regional NWP system

Parallel suite 36 Global 17km 70 levels (N768L70)

EURO 4km 70 levels

UKV 1.5km 70 levels

[Retired NAE 12km 70 levels]

The UM is just one part of a larger forecast system. An example workflow:

Academic users can access this data

• Fixed resolutions: (at mid-latitudes) N48 N96 N144 N216 N320 N512 N768

96 x 73 192 x 145 288 x 217 432 x 325 640 x 481 1024 x 769 1536 x 1152 ~270 km ~135 km ~90 km ~60 km ~40 km ~25 km ~17 km 2.5° x 3.75° 1.88° x 1.25° 1.25° x 0.83° 0.83° x 0.56° 0.56° x 0.38° 0.35° x 0.23° 0.23° x 0.16°

• Common scientific configurations used across scales (weather, seasonal and climate):

o GA = Global Atmosphere (UM)

o GL = Global Land (JULES)

o GO = Global Ocean (NEMO)

o GSI = Global Sea Ice (CICE)

o GC = Global Coupled (all of the above coupled together) • These are ongoing developments with fixed releases. 7 Vertical resolutions

Standard resolutions Defined by model levels and height

Example: L85 with top at 85 km

NWP Climate

Run length 5 day operational forecast, Months (seasonal) 15 day ensemble forecast Years, decades, centuries+

Global resolution Testing: Low resolution: N320 (40 km) with 15 min ts N96 (135 km) with 20 min ts Operational: High resolution: N768 (17 km) with 7.5 min ts N512 (25 km) with 15 min ts

Calendar Gregorian Seasonal: Gregorian Other: 360 day

Dynamics Non-bit reproducible Bit-reproducible

Aerosols Climatological concentrations Interactive schemes with specified. emissions specified (not seasonal).

Based on information from Dan Copsey 9 HadGEM2(-ES)

Ocean HadGEM2-ES Physical climate biogeochemistry • Full Earth system model • Development completed in 2009 • Used for IPCC AR5 (CMIP5)

• N96 – 1° ocean Aerosols and chemistry (UKCA)

Interactive vegetation (TRIFFID)

Other versions available: Human emissions • -A (atmosphere only) • -AO (coupled atmosphere-ocean) • -CC (carbon-cycle) Earth system minus UKCA

10 HadGEM3(-AO)

UM atmosphere (GA) JULES land surface (GL) Under continuous development. There are versions with: • different code releases • different scientific configurations

OASIS3 Multiple resolutions: coupler • N96 – ORCA0.25 • N216 – ORCA0.25 NEMO ocean model (GO) CICE sea ice (GSI) • N512 – ORCA0.25 / ORCA12 (experimental)

Can run atmosphere-only

(Modified from a Met Office diagram.) 11 Other modelling systems

• MOGREPS: ensemble forecast – Based on Parallel Suites

• Nesting suite: sequence of spatial resolutions

• GloSea: seasonal prediction – Based on HadGEM3

• DePreSys: decadal prediction – Based on HadGEM3 (formerly HadCM3)

• UKESM(-Hi/Lo): Earth system model – Based on HadGEM3 + submodels – Under development

12 Terminology

Model version: defines the code base used. Model configuration: defines the science options, eg GA6.0, GO5.0. Modelling system: application with specific configuration (eg HadGEM3-GC2, GloSea5). Parallel suite: version of operational forecast system Job: particular instance defined and run by user through (UMUI) Experiment: grouping of jobs in UMUI Suite: Rose ‘job’

13 Major UM developments

Old New FCM NEMO & ENDGame Rose Shared dynamics dynamics CICE repositories

UM 4.5 UM 5.2 UM 6.1 UM 6.6 UM 7.1 UM 8.5 UM 9.0 UM 10.0

1998 2001 2005 2007 2008 2013 2014 2015

Climate models HadCM3 HadGEM1 HadGEM2 HadGEM3

NWP models PS12 PS19 PS20 … PS34 PS35 PS36

Dynamical core Infrastructure Ocean model 14 UM software (pre vn9.0)

PUMA UMUI Database of user jobs FCM Graphical job editor Code manager

Compilation and build HPC

Atmosphere model OASIS Reconfiguration Dynamical core Coupler Prepares initial Physics model state Diagnostics (STASH) [JULES and/or UKCA] Ocean model

Input file tools Output file tools Local / Jasmin Prepare ancillary data Data processing Analysis and visualisation 15 UM software (vn10.0 onwards)

PUMA MOSRS UMUI Rose Database of user jobs FCM Graphical job editor Cylc / Rose Code manager Job submission Workflow manager Compilation and build HPC

Atmosphere model OASIS Reconfiguration Dynamical core Coupler Prepares initial Physics model state Diagnostics (STASH) [JULES and/or UKCA] Ocean model

Input file tools Output file tools Local / Jasmin Prepare ancillary data Data processing Analysis and visualisation 16 NCAS supported machines

MOSRS PUMA

Code repositories Repositories: Web server (Trac) • Code (UM 10.x) User /home space • Rose suites NOC UMUI, Rose/Cylc and FCM UM submission Lander Rose VM MONSooN Polaris File systems: File systems: /home, /home and /work /projects HPC Wales RDF MASS

Jasmin 17 Upcoming UM developments

Short-term: • UKESM1 release – Based on GC3 – To include land ice sheets, ocean biogeochemistry, plus carbon and nitrogen cycles

Medium term: • N1024 -> N2048 global forecasts

Long term: • LFRiC replacement for the UM • GungHo highly scalable dynamical core

18 NCAS Unified Model Introduction

Part 2: Running the UM

University of Reading, 13-15 April 2016 UM User Interface

• The UMUI is a graphical interface to the UM comprising: – Job database (allowing the user to search for and browse jobs). – Window-based job editor (allowing the user to select the code version, scientific configuration and run-time option • The UMUI then creates the scripts and namelists which control the UM run. • It is used for UM versions 4.5 to 8.6. – Later versions use the UMUI replacement Rose. • It is available via the PUMA service, which provides: – All the latest UMUI changes – Centralised access to local files (hand-edits etc) – A common job database for all academic and MONSooN users

20 UMUI vocabulary

Experiments: e.g. xxab • A grouping of UM jobs • Identified by a folder icon and description in the UMUI • By default only your own experiments are displayed by the UMUI, although it is possible to search the full database. • Experiments can be created, copied and deleted Jobs: e.g. xxabc • Up to 26 jobs can be grouped in an experiment • Jobs can be created, copied and deleted • 2 jobs can be differenced (if using the same UM version) Navigation window -> input windows • Input panels within a job. • Note that these can very between UM versions.

21 22 Creating a UM job

• Choose a UM job from UKMO, NCAS or a colleague that is close to what you wish to run. – Either copy the job into your own job if they are in the same local database, – or upload (import) a basis file (the full description of a UM job which has come from another UMUI database) into your own job. • NCAS standard jobs should be available under the umui owner. • Basis files can be downloaded (exported) from the UMUI and emailed. – Note that additional UMUI files (hand-edits, STASHmasters etc) and data files may also need to be copied to the target system.

23 How UM jobs are defined

UM jobs are defined by: • A UM code version (e.g. vn8.2) • Specific UMUI settings (and hand-edits which alter these) • A particular horizontal and vertical resolution • A list of changes to the main code base, in the form of FCM branches. • A defined set of input files: start files, ancillary files and lateral boundary conditions. • STASH items specifying the diagnostic fields to be output (and STASHmaster files which define new diagnostics or alter standard ones). Modelling systems such as HadGEM3 or UKV span different UM versions and have specific “standard jobs” as they develop over time.

24 Submitting a job

UM runs are submitted directly from the UMUI.

PUMA HPC UMUI scripts (MONSooN/ARCHER) scripts namelists namelists code UM compile, process submit reconfigure & run

When a job is processed the UMUI: • Creates the namelists that the UM reads in at runtime. • Generates the scripts used to compile and run the model. When a job is submitted the UMUI: • Starts running the scripts • Copies the namelists and scripts to the HPC • Extracts the code and copies to the HPC

25 Running the UM

Three distinct steps (separate batch jobs): • Compilation -> Reconfiguration -> UM atmosphere model run • May also have a Post-processing & archiving step after the run.

The compilation is handled by the Flexible Configuration Management (FCM) system which: • manages code components; • creates Makefiles for compilation and loading; • compiles and links the code according to options selected; • creates an executable.

The reconfiguration is a stand-alone program which modifies UM atmosphere start files (and ocean files for UM 6.6.3 and earlier). • Compiled with FCM • Runs as a parallel application • Produces a new start file

26 UMUI settings

Select whether the compilation, reconfiguration and run required: • Compilation and Run Options -> Compile and run options for Atmosphere and Reconfiguration: Model start time and run length: • Input/Output Control and Resources -> Start Date and Run Length Options: Number of processors for the atmosphere run: • User Information and Submit Method -> Job submission method: Job time limit for the atmosphere run: • User Information and Submit Method -> Job submission method -> Qsub Number of processors & job time limit for the reconfiguration: • Reconfiguration -> General Reconfiguration -> Options For a given run length, the number of processors and length of time to request depends on the queue structure of the machine and the performance of the job. Testing is often required to work out the best options.

27 UM output listing

Log messages (output listing) from the UM are written to a .leave file: • e.g. xagmc000.xagmc.o98342.t14136.leave • Stored in: $HOME/output Output listing options controlled by UMUI Input/Output Control and Resources -> Output Choices The output listing can be quite large and confusing. Check output listing for • Timings reported at the end of the listing file • Key words like ERROR, ABORT, "file not found" Check presence of key files such as: • Executable after compilation • Start file after reconfiguration • Output files after model run

28 NCAS Unified Model Introduction

Part 3: UM Data Files

University of Reading, 13-15 April 2016 Contents

• UM file types

• Model input

• Model output

• Limited Area Models

• NEMO and CICE

30 UM and PP formats

There can be confusion about these two types of UM-related files.

UM format • The format read and written by the model. • Direct access files that consist of a primary header containing pointers to a series of secondary headers that then point to the data: – Implies mixed integer/real arrays conventionally handled with equivalence – Assumes that levels for a given variable are contiguous • Described in document F3: http://cms.ncas.ac.uk/wiki/Docs/MetOfficeDocs

PP format • Produced from UM files for data analysis (not used by the model). • Sequential files (header, data, header, data). • Files from the BADC and UKMO archive may be in this format.

31 UM files

Files in Files out

Initial files Restart files (also called start files or dumps) (or dumps)

Lateral boundary conditions (LBCs) Lateral boundary conditions (LBCs)

Ancillary files Post processing diagnostic files - forcing (also called fields files) - control Climate means

All model files are in UM format, but there are slight variations between restart files, ancillaries and fields files (see UM doc F3).

32 (CF-)NetCDF

Network Common Data Form: • Portable and self-describing data format. • Various supported and external libraries and tools. • Widely used and a standard format for many scientific communities. • http://www.unidata.ucar.edu/software/netcdf/

Climate and Forecast conventions: • Set of standards for describing data (metadata). • Definitive explanations of data variables (standard names). • Definitions for temporal and spatial properties of the data. • http://cfconventions.org/

33 NetCDF and the UM

• When you might come across NetCDF: – Input data sets may be in NetCDF (e.g. CMIP5 scenarios). – It can be more convenient to use NetCDF for analysing or sharing data. – NEMO and CICE read and write NetCDF files.

• There are tools for converting between (CF-)NetCDF and UM/PP formats.

• NCAS are working on a project to write CF-NetCDF directly from the UM.

34 • UM file types

• Model input

• Model output

• Limited Area Models

• NEMO and CICE

35 UM start dumps

• Start dumps are available on the National Computing Service (ARCHER) for standard resolutions under: /work/n02/n02/hum/ ($UMDIR)

• Restart dumps from other UM jobs can be used.

• For NWP, particular dates are available from the UK Met Office. Contact the NCAS modelling helpdesk: http://cms.ncas.ac.uk/wiki/CmsHelpdesk

36 Start dates

• For climate runs the start date is largely irrelevant • But for NWP experiments or case studies, the start date matters

• The UKMO archives only the last ~18 months of analyses which are used as start dumps. To start the UM from other dates, you need to use ECMWF data and convert this to a UM start dump.

• Starting from ECMWF data is available on the national HPC services. – For advice on starting from ECMWF data contact the helpdesk. – Example reconfiguration jobs are available under the “umui” owner.

37 The UM reconfiguration (i)

The reconfiguration is a standalone program which modifies (“reconfigures”) UM atmosphere or ocean start files to produce a new start file.

• This can be on a new grid (not for ocean files) – with different horizontal area – and/or different horizontal resolution – and/or different vertical resolution – if no new data is included, data from the original start file is interpolated to the new model grid/domain;

• It can also include different data fields from ancillary files – e.g. new orography data – e.g. new ozone data

38 The UM reconfiguration (ii)

• The reconfiguration is part of the UM code base but it has its own compilation stage to produce its own executable. • It runs on multiple processes (set separately from the model processors). • It can be run independently or immediately before a UM model run.

• Use the reconfiguration to: – Upgrade a start file from an earlier code version – Add fields including user-defined fields – Overwrite existing fields – Change resolution (not with ocean files)

39 UM ancillary files

• Ancillary fields describe externally prescribed conditions to be imposed on the model fields. – These may be time varying data applied regularly throughout the run. – Climate runs may use repeating year of data. – Also can be single fields to be applied to the initialisation. • The user needs to consider: – the horizontal and vertical resolution – the time variation – the 360 day/Gregorian calendar switch in the file header • In the UMUI ancillary file names and directories may be specified using environment variable. These are defined in an “ancillary versions file”: Atmosphere -> Ancillary and input data files -> In file related options -> Ancillary versions file

40 Standard ancillary files

Ancillary fields are grouped into files. Fixed: • Atmosphere -> Ancillary and input data files -> Other ancillary files and Lateral Boundary files • E.g., orography, land-sea mask, land fraction Climatologies: • Atmosphere -> Ancillary and input data files -> Climatologies & potential climatologies • Standard files are available on the national HPC services:

Ozone Shine and Li, Cariolle, SPARC SST / sea ice GISST, HadISST Soil fields AMIP

41 Example ancillary field

42 Control files

• The SW and LW radiation code use “spectral” files • Spectral data for the radiation scheme are generated by a separate pre- processing package and are stored in a spectral file. – This is not a simple matter and users will not normally generate their own spectral files. • Standard spectral files are available in the “ctldata/” directory of the UM installation. – Different spectral files are available on the national HPC services. • Spectral files are not in UM format but are Fortran namelists, e.g:

&R2SWSP L_PRESENT(0)=.TRUE., N_BAND=6, N_ABSORB=4, N_AEROSOL=13, TYPE_ABSORB=1,2,3,7, TYPE_AEROSOL=1,2,3,4,6,10,11,12,13,15,16,23,24,

43 • UM file types

• Model input

• Model output

• Limited Area Models

• NEMO and CICE

44 UM restart dumps

• Instantaneous dumps of the main model variables, written at regular intervals. – To restart the model for when things go wrong so that the run can be continued rather than starting again. – Start files for running long runs in ‘chunks’ – The basis for climate meaning.

• UMUI window: – Atmosphere -> control -> post-processing, dumping and meaning

• All restart dumps will be kept unless you set: – Post processing -> main switch & general questions • automatic post processing : yes • delete superseded restart dumps : yes – This works with and without archiving

45 Automatic resubmission

0 1 2 3 4 5 6 7 8 years

NRUN CRUN – automatic resubmission

• Always keep the same restart frequency throughout an experiment and for experiment inter-comparisons.

• To run a long climate experiment - set the restart dump frequency to a ‘reasonable’ rate - run the first chunk, check results are ok - select “continuation run (CRUN)” in the UMUI - resubmit the job

• Automatic data archiving, based on the UKMO system, has been implemented on the national HPC services.

46 Automatic archiving

Output from the UM Output from the UM

ARCHER temporary space MONSooN temporary space /work /projects

Medium-term RDF storage /nerc /nerc MASS archive JASMIN Your home institution

47 STASH

Spatial and Temporal Averaging and Storage Handling

For each diagnostic select: • Time profile: – When diagnostic will be output: the start & end time, and frequency – Whether time processing required, e.g. accumulation, mean, time series. • Domain profile: – Vertical: specify levels – Horizontal: limited area, meaning [zonal, vertical, meridional, horizontal], weighting. • Usage profile: – Select output unit for the diagnostic

48 49 STASH in the UMUI

• The UMUI windows for STASH are different from model windows.

• To add diagnostic: – Diagnostics -> load new diagnostics -> (double click on the section) – Available diagnostics are organised in sections. Some diagnostics have HELP information, a lot don’t! – Just because a diagnostic is available doesn’t mean it works!

• To view or edit the profiles: – Highlight the profile. – Profiles -> Edit profile -> Edit time/domain/usage

• Check setup (verify diagnostics) with Ctrl+v.

50 Adding new diagnostics

Users can add new diagnostics • Advice via the STASH user guide on the CMS website. • Users must provide a STASHmaster file to the UMUI: Atmosphere -> STASH -> User-STASHmaster files. • The standard STASHmaster file on PUMA is: /home/umui/umui/umui2.0/vn8.2/variables/STASHmaster_A

Users can copy the diagnostics settings from one UM job to another using: copy_stash • This script takes as input two UM basis files. • The two jobs must be at the same UM version.

51 Post processing files

• Output in fields file format (soon to be CF-netCDF) • Diagnostics set up via STASH and written to a “post processing unit”: – Atmosphere -> STASH • Post processing unit file sizes depend on options selected in the UMUI: – Post processing -> initialisation and processing – Options include • Profile 0: unpacked • Profile 5: new standard climate packing – Packed files save space when archiving. • Reduce file sizes at the end of the run (not if the run is to be continued) using “umpack”. This gets rid of padding and any extra headers.

52 Configuring post processing files

Unit number for fields files 60 – 69 Files labelled xxaabo.pp4 in $DATAW if not initialised Files labelled xxaabo.pdxxxx in $DATAM if re-initialised File names calculated from the re-initialisation process 53 Climate Meaning

• Climate runs using automatic resubmission, meaning the mechanism is safe over restarts.

• Diagnostics can be tagged to be picked up by the climate meaning system via STASH.

• Climate means are related to the dump frequency: – Atmosphere -> Control -> Post-processing, Dumping & Meaning -> Dumping and meaning -> NEXT – For example with a dumping frequency of 10 days, meaning periods of 3, 3, 4 and 10 will give: • monthly (3 x 10 day) means • seasonal (3 x 30 day) means • annual (4 x 90 day) means • decadal (10 x 360 day) means • File names will be of the form $RUNIDa.mm…, $RUNIDa.ms… • Climate meaning uses partial sum files to preserve precision across restarts (of the form $RUNIDa_s2[ab]).

54 • UM file types

• Model input

• Model output

• Limited Area Models

• NEMO and CICE

55 Current resolutions

• Operational global model: 17 km (previously 25 km) • Europe LAM: 4 km (operational) • On-demand LAM: 1.5 km (forecaster requested) • Storm scale LAM: 1 km (research model) • Research LAMs: as low as 50 m

12 km 4 km 1 km

Height of model orography (m)

57 Lateral boundary conditions

• LBC files are required by the Limited Area Model (LAM). • For standard areas (UK LAM and UK mesoscale): – Run the global UM to generate LBCs • 1024 x 769 L70 – global forecast resolution Atmosphere -> control -> output data files -> LBCs out – Run LAM Atmosphere -> ancillary and input data -> other ancillary -> LBC • For higher resolution standard areas: – On puma under owner “umui” there are example jobs for • 12 km, 4 km and 1 km UK mesoscale – These need start and ancillary files, some available on national HPC services. 58 Outputting LBCs

Specification of target grid

59 LAMPOS

For non-standard areas: • choose new domain using LAMPOS (available on puma)

• Create ancillary files: • Use the UKMO ancillary file creation system: CAP Available via the CMS web page • Run the global UM or larger LAM to generate LBCs • Run LAM

• Seek help and advice from NCAS CMS 60 Horizontal domain specification

Lat, lon in rotated grid

Used to put equator of rotated grid In the centre of the LAM domain LAMPOS or BADC Grid utilities

Obtained from CAP or reconfiguration

61 • UM file types

• Model input

• Model output

• Limited Area Models

• NEMO and CICE

62 Ocean and sea-ice

NEMO files: • All in NetCDF • Inputs cover full domain • By default one output file per process covering only that subdomain. – Construct global file after run using rebuild_nemo, e.g.: rebuild_nemo xdodto_CU150_19780901_19780930_grid_T 16 • NEMO -> Scientific Parameters and Sections -> Links to NEMO model

CICE files: • Restart files in binary format – There are tools to edit the start date and convert to NetCDF – (contact NCAS-CMS for guidance) • All other inputs and outputs in NetCDF (global files) • CICE -> Scientific Parameters and Sections -> Links to CICE model

63 ORCA grids

• Used by NEMO and CICE • Tripolar grids • Standard resolutions: – ORCA2 (2°) – ORCA1 (1°)

1 – ORCA025 ( /4°)

1 – ORCA12 ( /12°)

64 NCAS Unified Model Introduction

Part 4: FCM and the UM

University of Reading, 13-15 April 2016 What is FCM?

• Flexible Configuration Management system – Written by the Met Office. – A set of tools for managing and building source code. – Uses subversion for code management • Defines a common process and naming convention • Adds a layer on top of subversion

66 Overview of FCM

• FCM system consists of 3 components: – Integrated Configuration Management, wiki and issue tracking system – Extract system – Build system

67 Trac

• Integrated wiki and issue tracker – can be used for project documentation – keep track of bugs, development, etc • Browser for subversion repository • Timeline view of issues, wiki pages and subversion repository • https://puma.nerc.ac.uk/trac/UM

68 Extract System

Purpose: – Extracts source code from the repository ready for the build system

Features: – Combines code from a number of branches • Only if the modified files do not overlap – Mirrors code to a remote system (e.g. ARCHER) – Generates a configuration file for the build system

69 Build System

Purpose: – Builds the code assembled by the extract system and creates the model executable.

Features: – Parallel build – Incremental build – Build dependency analysis

70 Code Management Commands FCM System Components Local Working Subversion Copy Repository

Mirrored Pre-Built Extract Code Code Trac

Pre-Extracted Build PUMA Code

PUMA Build System 71 Features (1)

• Full history of source file – Accessible on the web via Trac

• Edit source code directly on PUMA

• Version control of directory tree – Copy, rename, add, delete files and directories

• Atomic commits – All or nothing is committed to the repository – Sequential changesets

72 Features (2)

• Access to all functions is via the fcm command – FCM command syntax is simple and easy to use – Simple GUI wrapper to the subversion CM commands – CM commands used only on PUMA – Build system is installed on ARCHER, MONSooN, etc

• Comprehensive User Guide – http://cms.ncas.ac.uk/chrome/site/FCM/user_guide/index.html

• Used by many other projects – NEMO, CICE, JULES, MONC, …

73 Repository Design

UM trunk src atmosphere ocean AC_assimilation control aerosols utility atmosphere_service configs include boundary_layer slab scripts branches tags vegetation

Admin

74 Subversion – some terminology

• The UM code is held in a subversion repository • The trunk is the consolidated code from the Met Office • A release is a specified revision of the trunk (identified by a revision keyword) • Branches are the method of making and tracking changes – held in the repository • A working copy is a local copy of a branch (possibly with changes to it) • An fcm patch is the mechanism by which code changes are exchanged with the Met Office.

75 Typical Workflow

Create Branch Commit to Repository Branch (PUMA)

Working Copy (PUMA) Checkout Modify Code

ARCHER Build

76 FCM and Revision Keywords

• FCM URLs are cumbersome • Keywords provide a shortcut • Specified with the fcm: prefix

Keyword Value um svn://puma/UM_svn/UM um_tr (or um-tr) svn://puma/UM_svn/UM/trunk um_br (or um-br) svn://puma/UM_svn/UM/branches

In a similar way revision keywords are used, so that you don’t have to remember the specific revision number that relates to a UM version. E.g. The revision keyword hg6.6.3 denotes the revision of the repository that is UM version 6.6.3, in this instance r2580.

77 UM Repository Diagram

EXETER Stable release vn8.0 Central UKMO Trunk vn7.8 vn7.9

RDG vn7.8 vn8.0 Trunk – UKMO Mirror

P/C: VN8.0_ncas

dev/user: VN8.0_user_branch

P/C: VN7.8_ncas Users always include this configuration branch in their jobs

dev/user: VN7.8_user_branch_X X } Same branch point from Trunk for everybody dev/user: VN7.8_user_branch_Y

Key:- - changeset P/S – shared package branch - stable release P/C – configuration branch dev/user – user development branch

78 UM Job Specifics

For HadGEM2 (hg6.6.3): • To create a branch use the script: – /usr/local/bin/create_HG2_branch • Make sure the standard branch is included in the job – fcm:um_br/pkg/Config/HG6.6.3_ncas

For all other UM Versions (vn7.x,vn8.x): • Use the ‘fcm gui’ or ‘fcm’ command line to create a branch • Include the standard branch in the job – fcm:um_br/pkg/Config/vnx.y_ncas

79 UMUI – specifying modifications

window: sub-model independent -> FCM configuration -> FCM configuration optional modifications

80 FCM Dos and Don’ts

DO: • Regularly commit your working copy changes to your branch at appropriate intervals. This means you can recover to a previous state easily.

DON’T: • Copy another user’s working copy, either in whole or part. A working copy contains hidden subversion files which, if edited or moved, will cause problems. • Copy sub-directories around within working copies, this can also lead to problems, for the same reason as above.

81 Getting Started

• Setup SVN_EDITOR to enable commit messages.

• Ensure that you have set up ssh-agent to allow login from PUMA to the remote machine (E.g. ARCHER) without the need to enter a password. (Instructions are available on the UM Tutorial Trac Wiki – see link below)

• UM FCM Tutorial: https://puma.nerc.ac.uk/trac/UM_TUTORIAL

82 Basic CM Sub-Commands

• The essential sub-commands are: – Checkout – Commit – Diff – Status – Branch – Merge • Help is available for all sub-commands: fcm help

83 FCM checkout

• Check out a working copy from the repository • URLs and revisions can be specified using keywords

ros@puma$ fcm checkout fcm:um_tr trunk_wc => svn checkout svn://puma/UM_svn/UM/trunk trunk_wc A trunk_wc/COPYRIGHT.txt A trunk_wc/CodeOwners.txt A trunk_wc/src A trunk_wc/src/control A trunk_wc/src/control/coupling … A trunk_wc/src/land/modules/params/c_vkman.F90 A trunk_wc/src/land/modules/params/nvegparm.F90 A trunk_wc/src/land/modules/params/c_surf.F90 Checked out revision 6719.

84 FCM status

• Inspect the status of working copy files & directories • See subversion handbook for status codes

fcm status -> svn status ? src/control/misc/um_paint_the_walls.F90 ! src/script/archiving/archfail A + src/ocean/mixed_layer/mixset_new.F90 D src/ocean/mixed_layer/mixset2.F90 M src/atmosphere/dynamics_solver/gcr_k-gcrk2a.F90 D src/configs/machines/cray-t3e/machine.cfg A src/configs/machines/cray-t3e/overrides128.cfg

85 FCM diff

• Used to show changes you’ve made to any file in your working copy, or relative to the base of the branch from which it came

• Specify the option --graphical or -g to use the tool xxdiff to display the differences

86 FCM commit

• Sends changes from working copy to the repository • All changes in your working copy are committed. – All or nothing – you can’t specify which files to commit • You will be prompted to enter a log message – Add your message above the delimiter line: “--Add your commit message ABOVE - do not alter this line or those below--” – Do not alter this delimiter line. Your changes will be ignored if you do.

87 Further Information

• Hands-on UM Tutorial https://puma.nerc.ac.uk/trac/UM_TUTORIAL Important: This is based on UM8.2. (All methods described are the same for HG6.6.3 EXCEPT branch creation. Remember to use the create_HG2_branch script when working with HG6.6.3) • FCM User Guide – Available from NCAS CMS website • FAQ – Available from the UM Trac Wiki along with links to other documents

88 NCAS Unified Model Introduction

Part 5: Rose and the UM

University of Reading, 13-15 April 2016 Rose and cylc

• Rose is a replacement for the UMUI from version 9.0 onwards. – “Jobs” are replaced by “suites” – Suites are under version control (using FCM) – There are tools for viewing, editing and controlling suites

• Cylc controls job submission and scheduling. – This simplifies complex workflows and cyclical suites.

90 Rosie suite database

Rose suites are held in a repository either locally (e.g. puma) or on the MOSRS (‘u’). • Suite-ids are prefixed with the repository name: e.g. puma-aa003, u-aa003 • There is a suite repository viewer (like the UMUI): rosie go • You can copy, create and checkout suites via the GUI or command-line tools: rosie create/copy/checkout

91 Rosie go

Click to copy a suite

92 Double-click to check-out and edit or run a suite Editing suites

You can open and edit suites through the Rose editor: • Scientific settings are similar to the UMUI. • Metadata provides information and consistency checking.

You can also edit the files directly: • Checked-out suites live in a directory: $HOME/roses// • Use FCM commands to save your changes

93 Rose edit

Click to run the suite

94 Running a suite

You can run a suite from the GUI or command-line: • rose suite-run

This launches a progress monitor GUI (gcylc): • From here you can view log files, stop and restart runs. • You can close this without affecting the run. To re-launch: – rose suite-gcontrol

95 Suite control GUI

96 Output files

• UM output data lives on the HPC and has (mostly) the same format as previously: $DATADIR/cylc-run//work//atmos/

• Log files (equivalent to leave files) are copied to puma, and can be accessed through Rose bush: http://puma.nerc.ac.uk/rose-bush – For MONSooN launch a browser from exvmsrose: http://localhost/rose-bush – Log files can also be viewed directly from: cylc-run//log/

97 Rose bush

98 MO Science Repository Service

• From vn10.0 the UM code is located on the Met Office Science Repository Service: https://code.metoffice.gov.uk

• This service hosts other Met Office codes (e.g. JULES, GCOM) and collaborative projects (e.g. GA, GL, GO and GSI).

• Access must be requested via NCAS-CMS.

99 Interaction with PUMA

Met Office Shared Repositories fcm:um.x (UM, JULES, etc) Updated every 10mins

PUMA Local Mirror of Shared fcm:um.xm Create branches Repositories & commit code changes Extract code into Rose suite

100 Archer and MONSooN

UM versions 10.0 to 10.3 are available on Archer and MONSooN. • Archer suites are submitted via PUMA. MONSooN • MONSooN suites are submitted via the Rose VM. Rose VM Checkout & edit suites PUMA Submit & monitor suites Checkout & edit suites

Submit & monitor suites XCM Build & run model Archer Build & run model Shared filesystem 101 Getting started with UM10.x

Follow the instructions on the CMS webpage: http://cms.ncas.ac.uk/wiki/RoseCylc

1. Register for a MOSRS account by emailing: [email protected]

2. Carefully follow the instructions for caching your password.

3. Follow the Rose set up instructions for MONSooN or Archer.

4. Work through the UM Rose tutorial.

102 Further information

For details of Rose commands and the syntax of configuration files, see the documentation: http://metomi.github.io/rose/doc/rose.html http://cylc.github.io/cylc/html/single/cug-html.html

For more information on the command line tools (for fcm, rose and cylc) use ‘help’, e.g.: $ rose help suite-run

If you are going to be developing suites, work through the Rose tutorials in the documentation.

103 NCAS Unified Model Introduction

Part 6: UM tools and facilities

University of Reading, 13-15 April 2016 File formats

UM Tools should provide a full range of input and output format possibilities

GRIB GRIB

ASCI ASCI

UM UM UM

netcdf netcdf PP PP

105 Conversion tools

GRIB GRIB ECMWF data Metview DECODE Fortran

reconfiguration convpp 64-bit PP Model 64-bit UM fftopp 32-bit PP Met Office graphics wconv.pl pptoanc xconv um32to64 convpp 32-bit netcdf IDL/ 64-bit PP Fortran ieee netcdf Python, idl 32-bit PP 32-bit UM (from UKMO) Ferret, Grads cdat

106 UM tools (i)

Data conversion Tool Input Output pptoanc PP UM (ancillary) xancil netCDF UM (ancillary) convpp UM PP ff2pp UM (fields format) 32-bit PP ieee UM 32-bit UM bigend UM byte-swapped UM xconv UM , PP and netCDF netCDF cf-python UM and PP CF-netCDF All tools and output 64-bit unless otherwise stated 107 UM tools (ii)

Data information Tool Purpose cumf Compares UM format files pumf Prints out header information for PP files uminfo Header information UM ppinfo Header information PP Data manipulation Tool Purpose umpack Removes spare headers and extra buffers from UM files fieldop Adds, subtracts, multiplies, divides UM (fields files) format

108 Xconv

• Xconv can be used to see what fields are contained within a data file and to look at the data values, either directly at the numerical values or at a gridbox fill plot of the data. • It can also convert model output into netCDF. Supported input formats: – Data output from the UM – PP format – GRIB format – netCDF format • Data manipulations available: – Spectral to gridpoint – Interpolation (bilinear or area-weighted) – Conversion to and from rotated grids

109 110 Central Ancillary Program

• For creating standard ancillary files can use CAP – Orography – Land-sea mask – Soil moistue, snow – Vegetation – Aerosol – Sea surface temperatue, sea-ice – Ozone • Typically used for running the LAM

• Contact the CMS helpdesk for more information: http://cms.ncas.ac.uk/wiki/CmsHelpdesk

111 Xancil

• Xancil provides a GUI for generating ancillary files from your own data. – Input data should be in NetCDF format. – Supports the creation of standard atmosphere and (UM) ocean ancillary files. – Allows the creation of custom ancillary files.

• Xancil can take command-line options. – Complex workflows can be automated using tcl scripts.

112 xancil

113 Visualisation and analysis

UM run UM fields files Python, idl MATLAB etc

ff2pp cf-python xconv

NDdiag Iris Ferret grads UKMO library Jplot (midl) cfplot

114 Jasmin Analysis Platform (JAP)

• Collection of tools commonly used for atmospheric and Earth observation science, including: – NetCDF and nco tools – Cf-python, cfplot and Iris – Xconv, CDAT and lots more

• Installed on all the Jasmin systems (at BADC, Bristol, Reading and Leeds) and on the RDF at Archer. • Can download as: – Virtual Machine, which will work on any system running VirtualBox – RPMs for RHEL/CentOS Linux https://github.com/cedadev/jasmin_scivm/wiki

115 Availability of tools

• The UM tools are available on Archer under: – $UMDIR/bin – $UMDIR/vnX.Y/cce/utils

• CMS-developed tools (xconv, xancil, cf-python, cfplot) can be downloaded to your own platform.

• Further information can be found through the CMS site: http://cms.ncas.ac.uk/wiki/ToolsAndUtilities

116 NCAS Unified Model Introduction

Part 7: Finale

University of Reading, 13-15 April 2016 UK UM support

• Through NCAS-CMS (NERC Centre for Atmospheric Science – Computational Modelling Services): http://cms.ncas.ac.uk • Modelling helpdesk: http://cms.ncas.ac.uk/wiki/CmsHelpdesk • Email contact: [email protected] • Use the Helpdesk or cms-support email where possible (rather than emailing the team directly).

118 Before contacting the helpdesk

1. Always search the helpdesk before raising a query. You may find that your query has already been answered. 2. On ARCHER please make your /home and /work directories readable by the CMS team so that we can help with any queries. chmod -R g+rX /home/n02/n02/ chmod -R g+rX /work/n02/n02/ 3. When contacting the CMS Helpdesk please make sure you supply the following information to help us answer your query more quickly. – Cut and paste any error messages – The job id of the UM job (e.g. xbcde) – The name of the UM output file ending .comp.leave or .leave

119 Other sources of support

Archer helpdesk: https://www.archer.ac.uk/safe/

MONSooN helpdesk: [email protected]

Met Office collaborative newsgroups (for general UM advice or scientific queries): • http://collab.metoffice.gov.uk/twiki/bin/view/Newsgroups/WebHome

Issue Archer MONSooN New account CMS MONsooN Over quota / budget run out CMS MONSooN UM specific issues CMS CMS Machine problems Archer MONSooN

120 The CMS team

• UM modelling support team: – Jeff Cole, Rosalyn Hatcher, Grenville Lister, Willie McGinty, Annette Osprey, Simon Wilson (50%) • UKCA modelling support: – Luke Abraham • PUMA administration: – Andy Heaps • Strategic projects : – Simon Read and Marc Stringer (UKESM) – David Hassel and Charles Roberts (CF-Python)

121 What CMS provides (i)

Machines and facilities: • Computing time on Archer. • Support for the UM on national HPC facility (Archer) and Met Office/NERC collaboration HPC service (MONSooN). • A common platform (puma) for the UMUI database, code repositories, and UM submission. • Support for the UM on local computing facilities (e.g. Polaris, Mobilis, HPC Wales).

Core activities: • Training days and user meetings • Documentation and information services • Set of UM “standard jobs”

122 What CMS provides (ii)

Strategic projects: • Development and support for visualisation and analysis tools (xconv, xancil, cf-python, cfplot). • Support for weather, climate and composition with networks of excellence using the UM. • Developments in partnership with UKMO through collaborative projects: – For example: UKHiGEM, CASCADE, CLIMIP, High resolution modelling, UKESM, SWAMMA.

123 Things to do now (i)

1. Register for an Archer account (if you haven’t done so already).

2. Set up ssh-agent to allow connection between your puma and Archer accounts: http://cms.ncas.ac.uk/wiki/ArcherSshAgent

3. Finish working through the tutorial exercises that are relevant to you.

4. Make sure to complete the FCM tutorial: https://puma.nerc.ac.uk/trac/UM_TUTORIAL

124 Things to do now (ii)

5. Copy a UM output file to your local system or Jasmin. Produce a plot of your favourite field using an available graphics package.

6. Browse the UMUI database to find a job that is similar to the one you want to run. Look at its options and settings.

7. Read the UM user guide: http://cms.ncas.ac.uk/wiki/Docs/MetOfficeDocs

8. Look at the Fortran code in the UM source code browser: http://cms.ncas.ac.uk/wiki/UM/CodeBrowsers

125