LSDTopoTools for Geomorphology, Hydrology, Ecology and Environmental Sciences
Simon Marius Mudd, David Milodowski, Stuart Grieve, Fiona Clubb, Martin Hurst, Declan Valters, Marie-Alice Harel Table of Contents
Preface by Simon M. Mudd ...... 2 Overview of the book ...... 3 1. Introduction ...... 5 1.1. What is this software? ...... 5 1.2. Why don’t we just use ArcMap/QGIS? It has topographic analysis tools...... 5 1.3. Summary ...... 6 1.4. Quickstart for those who don’t want to read the first 4 chapters ...... 6 2. Required and useful software ...... 7 2.1. Essential software ...... 7 2.1.1. Git ...... 7 2.1.2. A compiler and other tools associated with the source code ...... 8 2.1.3. GDAL ...... 9 2.1.4. Python ...... 9 2.2. Nonessential software ...... 10 2.2.1. An open source GIS: QGIS ...... 10 2.2.2. Documentation using asciidoctor ...... 10 2.3. Summary ...... 11 3. Preliminary steps ...... 12 3.1. The terminal and powershells ...... 12 3.2. Topographic data ...... 13 3.3. Data sources ...... 13 3.3.1. What data does LSDTopoToolbox take? ...... 13 3.3.2. Downloading data ...... 14 3.4. Projections and transformations ...... 16 3.5. GDAL ...... 17 3.5.1. Finding out what sort of data you’ve got ...... 17 3.5.2. Translating your raster into something that can be used by LSDTopoToolbox ...... 17 3.5.3. Clipping rasters with gdal ...... 21 3.6. Looking at your data (before you do anything with it)...... 21 3.6.1. Our lightweight python mapping tools ...... 21 3.7. Summary ...... 22 4. Getting LSDTopoTools ...... 23 4.1. How the code is structured ...... 23 4.1.1. Compiling the code ...... 23 4.1.2. Driver functions, objects and libraries ...... 23 4.1.3. The typical directory layout ...... 25 4.2. Summary ...... 25 4.3. Getting the code using Git ...... 25 4.3.1. Getting started with Git ...... 25 4.3.2. Pulling a repository from Github ...... 26 4.3.3. Making your own repository ...... 28 4.4. Where to get the code ...... 29 4.4.1. Latest release versions on GitHub ...... 30 4.4.2. CSDMS ...... 30 4.5. Summary ...... 30 5. First Analysis ...... 31 5.1. Preparing your data and folders ...... 31 5.2. Get and compile your first LSDTopoTools program ...... 33 5.2.1. Clone the code from Git ...... 33 5.2.2. If you would rather not use git ...... 34 5.2.3. Compile the code ...... 34 5.3. Running your first analysis ...... 34 5.3.1. First analysis: example data ...... 34 5.3.2. Placing the paramfile ...... 35 5.3.3. Modifying the parameter file ...... 36 5.3.4. Running the analyses (in this case, writing fill and hillshade rasters) ...... 40 5.3.5. Look at the output ...... 40 5.4. Summary ...... 41 6. Simple surface metrics (slope, curvature, aspect, etc) ...... 42 6.1. Modifying the parameter file ...... 42 6.2. Running the analyses (in this case, writing fill and hillshade rasters) ...... 46 6.3. Summary ...... 46 7. Chi analysis ...... 48 7.1. Background to chi analysis ...... 48 7.1.1. Topographic expression of climate, tectonics, and lithology ...... 48 7.2. Get the chi analysis tools ...... 48 7.2.1. Clone the code from Git ...... 49 7.2.2. Compile the code ...... 49 7.2.3. Get some example data ...... 49 7.3. Chi analysis, part 1: getting the channel profiles ...... 50 7.3.1. Overview ...... 51 7.3.2. Running the channel network extraction ...... 51 7.4. Chi profile analysis, part 2: constraining m/n and transforming profiles ...... 60 7.4.1. Steps involved to perform channel analysis ...... 61 7.4.2. Performing a statistical analysis to constrain the best fit m/n ratio ...... 61 7.4.3. The .tree file ...... 69 7.4.4. Visualizing the analysis ...... 70 7.4.5. A Sample Chi Analysis Workflow ...... 75 7.5. Summary ...... 75 8. Basinwide cosmogenic analysis ...... 76 8.1. Get the code and data basinwide cosmogenic analysis ...... 76 8.1.1. Get the source code for basinwide cosmogenics ...... 76 8.1.2. Getting example data: The San Bernardino Mountains ...... 77 8.1.3. Setting up your data directories and parameter files ...... 78 8.2. Calculating Topographic Shielding ...... 84 8.2.1. Steps for preparing the rasters for shielding calculations ...... 84 8.2.2. The shielding computation ...... 87 8.2.3. Embarrassingly parallel shielding ...... 87 8.3. Calculating erosion rates ...... 88 8.3.1. Compiling the code ...... 89 8.3.2. The output files ...... 89 8.4. Summary ...... 92 Appendix A: Software ...... 93 A.1. Essentials ...... 93 A.1.1. Git ...... 93 A.1.2. C++ tools ...... 93 A.1.3. Python ...... 93 A.1.4. GDAL ...... 94 A.2. Useful extras ...... 94 A.2.1. A virtual machine ...... 94 A.2.2. Geographic Information Software ...... 94 A.2.3. Ruby ...... 95 Appendix B: Setting up on Windows ...... 96 B.1. Working with the powershell ...... 96 B.1.1. Starting a powershell session ...... 96 B.2. Windows installation programs ...... 96 B.2.1. Package management ...... 97 B.2.2. Git ...... 97 B.3. Tools for C++ ...... 97 B.3.1. Cygwin ...... 98 B.3.2. C++ libraries ...... 99 B.4. Python ...... 99 B.5. GDAL windows installation ...... 99 B.6. Ruby ...... 99 B.6.1. Install Ruby using choco ...... 100 B.6.2. Fix rubygems on Windows ...... 100 B.6.3. Fix RubyDevKit on Windows ...... 100 B.6.4. Install some gems ...... 101 B.6.5. If you use Ruby with Java (you will probably not need this) ...... 101 B.7. Windows installation summary ...... 102 B.8. Turning your windows machine into a Linux machine ...... 102 B.9. Summary ...... 104 Appendix C: Setting up on Linux ...... 105 C.1. Git ...... 105 C.2. C++ tools ...... 105 C.2.1. C++ libraries ...... 105 C.3. Python ...... 106 C.3.1. Installing python ...... 106 C.3.2. Installing python packages ...... 107 C.4. Ruby ...... 107 C.4.1. Installing the asciidoctor documentation software ...... 108 C.5. Cloning or forking the documentation ...... 108 C.6. Summary ...... 109 Appendix D: Code Structure ...... 110 D.1. Source Files: Drivers, Headers, and Implementaions...... 110 D.2. A closer look at the source files ...... 111 D.2.1. LSDRaster ...... 111 D.2.2. LSDIndexRaster ...... 111 D.2.3. LSDFlowInfo ...... 111 D.2.4. LSDIndexChannel ...... 112 D.2.5. LSDChannel ...... 112 D.2.6. LSDJunctionNetwork ...... 112 D.2.7. LSDIndexChannelTree ...... 112 D.2.8. LSDChiNetwork ...... 112 D.2.9. LSDMostLikelyPartitionsFinder ...... 112 D.2.10. LSDStatsTools ...... 113 Figure 1. LSD logo Preface by Simon M. Mudd
Welcome to the documentation of the LSDTopoTools. This is, I am sure, obvious, but LSD stands for Land Surface Dynamics. Our group chose it because, as far as we know, it is a unique acronym and has no other meanings.
The project started around 2010 due to my increasing frustration with my inability to reproduce topographic analyses that I found in papers and saw at conferences. Some of the papers that had irreproducible analyses were my own! Like many scientists working with topographic data, I was using a geographic information system (GIS) to prepare figures and analyze topography, and after a long session of clicking on commercial software to get just that right figure, I did not have a record of the steps I took to get there. Mea culpa. However, I do not think I am the only person guilty of doing this! I wanted a way of doing topographic analysis that did not involve a sequence of mouse clicks.
A second motivation came when my PhD student, Martin Hurst, finished his PhD and left Edinburgh for warmer pastures in England. His PhD included several novel analyses that were clearly very useful, but also built using the python functionality in a certain commercial GIS and not very portable. I and my other PhD students wanted to run Martin’s analyses on other landscapes, but this proved to be a painful process that required a lot of emails and telephone calls between Martin and our group.
This motivated me to start writing my own software for dealing with topographic data. This seemed crazy at the time. Why were we trying to reinvent a GIS? The answer is that the resulting software, LSDTopoTools, IS NOT A GIS! It is a series of algorithms that are open-source and can be used to analyze topography, and the programs that run these analyses, which we call driver programs, are intended to be redistributed such that if you have the same topographic data as was used in the original analysis, you should be able to reproduce the analysis exactly. Also the code for the analysis is available so that there is no hiding behind cherry-picked figures.
We hope that others outside our group will find our tools useful, and this document will help users get our tools working on their systems. I do plead for patience: we have yet to involve anyone in the project that has any formal computer science of software engineering training! But we do hope to distribute beyond the walls of the School of GeoScience at the University of Edinburgh, so please contact us for help, questions or suggestions. Overview of the book
The purpose of this book is both to get you started using LSDTopoTools, and thus the early chapters contain both pre-requisite material and tutorials. The latter stages of the book are dedicated to using our driver functions (these are programs that are used to perform specific analyses). This latter part of the book focuses on research applications; we tend to write a series of driver functions for our publications which aim to each give some new geophysical, hydrological or ecological insight into the functioning of landscapes. Thus the latter half of the book is both long an not really structured like a textbook, and will expand as we conduct research. However, for those simply interested in learning how to get the code working and to perform some "routine" analyses the initial chapters are structured more like a book.
By routine I mean something that is accepted by most professionals such as basin NOTE extraction or gradient calculations, and is not likely to be controversial.
Chapter 1 goes into some more detail about the motivation behind the software, and involves a bit of commentary about open science. You are probably safe to skip that chapter if you do not like opinions.
Chapter 2 is a brief overview of the software you will need to get our software working on your computer, although the details about how to install things is mainly relegated to the appendices.
Chapter 3 describes the preliminary steps you need to take with your topographic data in order to get it into our software. If you have read about or taken a course on GIS, this will be vaguely familiar. It will introduce GDAL, which we find to be much better than commercial software for common tasks such as projections, coordinate transformations and merging of data.
Chapter 4 explains how to get our software from its various Github repositories, and has some basic details about the structure of the software.
Chapters 5-6 are the tutorial component of the book, and have been used in courses at the University of Edinburgh.
*The chapters thereafter consist of documentation of our driver functions that have been used for research, many of which feature in published papers.
Appendix A gives more detail about required software to get our package running.
Appendix B explains how to get LSDTopoTools running on Windows. It contains a quite a bit of text about why you don’t really want to install our software on Windows, since installation is much more reliable, functional, and easy on Linux. Don’t worry if you don’t have a Linux computer! We will explain how to create a "virtual" Linux computer on your Windows computer. This description of creating a virtual Linux machine should also work for users of OS X.
Appendix C explains how to get LSDTopoTools running on Linux.
Appendix D has some more details on how the code is structured. If you are obsessive you could go one step further and look at the documentation of the source code.
Appendix E explains the different options in the analysis driver functions, which allow simple analyses driven by a single program. Chapter 1. Introduction
1.1. What is this software?
Welcome to the LSDTopoTools. This tool was developed at the University of Edinburgh
• As a framework for implementing the latest developments in topographic analysis.
• As a framework for developing new topographic analysis techniques.
• As a framework for numerical modelling of landscapes.
• To improve the speed and performance of topographic analysis versus other tools (e.g., commercial GIS software).
• To enable reproducible topographic analysis in the research context.
The toolbox is organized around objects, which are used to store and manipulate specific kinds of data, and driver functions, which users write to interface with the objects.
The Source code documentation site tells you all about the objects: these pages get into the nitty gritty of the computational algorithms and I only recommend looking at them if you plan on developing the code.
For most readers of this documentation, you can exist in blissful ignorance of the implementation and simply stay on these pages to learn how to use the software for your topographic analysis needs.
1.2. Why don’t we just use ArcMap/QGIS? It has topographic analysis tools.
One of the things we do as geomorphologists is try to understand the physics and evolution of the Earth’s surface by analyzing topography. Many geomorphologists will take some topographic data and perform a large number of steps to produce and original analysis. Our code is designed to automate such steps as well as make these steps reproducible. If you send another geomorphologist your code and data they should be able to exactly reproduce your analysis. This is not true of work done in ArcMap.
• Our code automates things that would be slow in ArcMap.
• Our code is designed to be reproducible: it does not depend on one individuals mouse clicks.
• Our code uses the latest fast algorithms so it is much faster than ArcMap.
• Our code has topographic analysis algorithms designed and coded by us or designed by someone else but coded by us soon after publication that are not available in ArcMap. • Our code contains some elements of landscape evolution models which cannot be done in ArcMap.
1.3. Summary
You should have some idea now of why we built this software: we want open, reproducible programs for analyzing topography for applications in Geomorphology, Hydrology, Ecology and allied Geosciences.
1.4. Quickstart for those who don’t want to read the first 4 chapters
Some readers will already be familiar with the steps required to get LSDTopoTools working. Here is succinct overview of what you need to do to prepare for your first analysis:
Quick Instructions for preliminary data processing
1. Make sure you can open a terminal or powershell window.
2. Make sure you have a C++ compiler (we use g++) and the make tool installed.
3. Make sure you have git installed.
4. Make sure you have the GDAL utilities installed and working.
5. Get some topographic data and convert it to projected coordinates (we prefer WGS1984 UTM projections).
6. Make sure you have python with scipy including numpy and matplotlib working on your computer.
If all of the above steps make sense, you can probably just implement them and move on to the [First analysis] chapter. Otherwise, you should continue reading from here. Chapter 2. Required and useful software
LSDTopoTools is a collection of programs written in C++ that crunch topographic data. To run LSDTopoTools all that is really required is a functioning C++ compiler (which is a program that translates C++ code into 1s and 0s that your computer can understand), the make utility, and for specific components a few extra libraries. Most analyses will not need libraries.
As a standalone bit of software, LSDTopoTools does not require vast effort in installing the required software. However, if you want to look at the data produced by the software, or use some of our automation scripts, you will need to install additional packages on your system.
2.1. Essential software
This isn’t our actual software! It is all the extra bits of software that you need to IMPORTANT get working before you can use LSDTopoTools!
This list goes slightly beyond getting the tools alone to run; it includes the software you need to get recent versions of the software and to visualize the output.
Instructions for installing all of these software packages are in the appendices. There you will find instructions for installing the software on a windows operating system and for installing the software on a Linux operating system.
Table 1. A list of the essential software
Software Notes
Git Version control software that you can use to grab working versions of our code from Github.
A C++ compiler For compiling our software. We use GNU compiler g++. Note that we don’t call this directly, but rather call it via the make utility.
make The make utility: used to compile the code from makefiles
Various C++ For basic analysis, no libraries are needed. libraries For more specialized analysis, the libraries FFTW, Boost, MTL and PCL are required. See below for more information.
Python We use python for both automation and vizualisation (via matplotlib).
GDAL We use the GDAL utilities to prepare our datasets, e.g. to transform them into appropriate coordinate systems and into the correct formats.
2.1.1. Git
Git is version control software. Version control software helps you keep track of changes to your scripts, notes, papers, etc. It also facilitates communication and collaboration through the online communities github and bitbucket.
We post updated versions of our software to the Github site https://github.com/LSDtopotools. We also post version of the software used in publications on the CSDMS github site: https://github.com/csdms.
It is possible to simply download the software from these sites but if you want to keep track of our updates or modify the software it will be better if you have git installed on your computer.
2.1.2. A compiler and other tools associated with the source code
You will need a compiler to build the software, as it is written in c++. In addition you will need a few tools to go along with the compiler. The things you really need are:
1. A C++ compiler. We use the GNU compiler g++.
2. The make utility. Most of the code is compiled by calling g++ from this utility.
In addition the TNT library is required, but this doesn’t require installation and we package it with our software releases. If you are wondering what it is when you download our software, it is used to do linear algebra and handle matrices.
In addition, there are a few isolated bits of the code that need these other components. Most users will not need them, but for complete functionality they are required. First, some of our makefiles include flags for profiling and debugging. We try to remove these before we release the code on Github, but every now and then one sneaks through and the code won’t compile if you don’t have a debugger or profiler. It might save you some confusion down the line if you install:
1. The gdb utility. This is the gnu debugger.
2. The gprof utility. This allows you to see what parts of the code are taking up the most computational time.
Next, there are a few specialized tools that are only required by some of our more advanced components.
Requirements for LSDRasterSpectral
Some of our tools include spectral analysis, and to do spectral analysis you need the Fast Fourier Transform Library.
In the source code, you will find #include statements for these libraries, and corresponding library flags in the makefile: -lfftw3. In the RasterSpectral source files, we assume that you will have a fast fourier transform folder in your top level LSDTopoTools directory. If that paragraph doesn’t make any sense to you, don’t worry. We will go into more detail about the spectral tools within the specific chapters dedicated to those tools. You can download FFTWv3 here: http://www.fftw.org/download.html. Requirements for LSDRasterModel
Embedded within LSDTopoTools is a landscape evolution model. The model requires the Fast Fourier Transform Library (see above).
In addition it requires some numerical libraries:
1. Boost, a popular c++ library.
2. MTL is a library for working with sparse matrices, which are required for solving some of the equations in the landscape evolution model. You will need MTL 4. You don’t have to install anything for this, but Boost needs to be installed and this library goes in the boost/numeric/mtl subdirectory.
Requirements for swaths and point clouds
Okay, now things get a little more complicated because you want to use the Swath Profile tools or the LSDCloudBase object (which handles point clouds). These objects are dependent on a set of libraries used for analyzing point cloud data, namely:
1. The cmake utility. This is like make but is required for our tools that examine point clouds, since it is required by something called the point cloud library.
2. pcl: The Point Cloud Library.
3. libLAS: a library for working with LAS format data.
Unfortunately these are a bit time consuming to install, because they depend on all sorts of other bits of software that must be installed first. You should see the appendeces for details on how to install this software.
2.1.3. GDAL
The Geospatial Data Abstraction Library has fantastic tools for preparing your data. It performs operations like clipping data, patching data together, resampling data, reprojecting data and doing coordinate transformations. If you don’t know what those things are, don’t worry, we explain these things in the preliminary steps chapter.
You can install all of GDAL if you want, but really you will only need their utilities.
2.1.4. Python
Python is a programming language used by many scientists to visualize data and crunch numbers. We use it for visualization, and also for automating a number of tasks associated with topographic analysis.
You will need:
1. The python programming language a. Scipy, for scientific python. It includes lots of useful packages like
i. Numpy for fast numerics.
ii. Matplotlib for plotting.
iii.Pandas for data analysis.
In addition, some of our python tools require GDAL, so you will have to install GDAL for these to work.
2.2. Nonessential software
There are a number of software packages that are not required to run LSDTopoTools, but that you might find useful.
First, many people use geographic information software (GIS) to visualize data. If you work at a university or a private company, you might have a license to ArcGIS, a popular commercial GIS. However, if you are not part of a large institution or your institutional license does not allow home use, it can be convenient to have an open source alternative. In addition, if you want to edit our documentation or make your own fork for notes, you might consider using the same tools we do, which require the Ruby programming language.
2.2.1. An open source GIS: QGIS
The industry standard GIS is ArcGIS, and if you are at a university you might have a site license for this software. It is not so easy to get on a personal computer, however, so there are a number of open source options that can be used as an alternative.
One alternative, and the one that will be used in these tutorials, is QGIS.
If you are familiar with ArcMap, you should be able to become proficient at QGIS in a few days. In my experience, it also has the advantage of being more stable (i.e., it crashes less) than ArcMap.
One thing that is quite nice about QGIS is the number of plugins that are available.
You should download and install QGIS from their website, and click on the `Plugins tab to get some plugins. the OpenLayers plugin, which allows you to quickly load satellite and map information from all sorts of vendors.
2.2.2. Documentation using asciidoctor
This book, and various other notes and websites associated with the LSDTopoTools project, have been built using something called asciidoctor. Asciidoctor is used to produce cross-linked documents and documentation, and has been designed to simplify the tool chain that takes one from writing technical documentation to producing a book rather simple. You can read about its rationale here: http://asciidoctor.org/docs/what-is-asciidoc/. The software has worked well for us. If you want to get asciidoctor working, you will need to get some packages working in Ruby. The instructions can be found in the appendices.
2.3. Summary
This chapter has given an overview of what software is necessary to use LSDTopoTools. The appendices contain information about installing this software on both Windows and Linux operating systems. Chapter 3. Preliminary steps
In this section we go over some of the steps required before you use the LSDTopoTools software package. The most basic step is to get some topographic data! Topographic data comes in a number of formats, so it is often necessary to manipulate the data a bit to get it into a form LSDTopoTools will understand. The main ways in which you will need to manipulate the data are changing the projection of the data and changing its format. We explain raster formats and projections first, and then move on to the tool that is best suited for projecting and transforming rasters: GDAL. Finally we describe some tools that you can use to lave a look at your raster data before you send it to LSDTopoTools.
Quick Instructions for preliminary data processing
1. Get some data.
2. Use GDAL to see if the data is in a projected coordinate system.
3. If the data is not in a projected coordinate system, use GDAL to make it so.
4. Convert the data to a format that LSDTopoTools can read: the preferred format is the ENVI .bil format.
3.1. The terminal and powershells
Our software works primarily through a terminal (in Linux) or poweshell (in Windows) window. We don’t have installation notes for OSX but if you manage to get it working there you will also use a terminal window. A terminal or powershell window is an interface through which you can issue text- based commands to your computer.
In Windows, you can get powershell by searching for programs. If you are on Windows 8 (why are you on Windows 8??), use the internet to figure out how to get a powershell open.
Different flavors of Linux have different methods in which to open a terminal, but if you are using Ubuntu you can type Ctrl+Alt+T or you can find it in the application menu.
On other flavors of Linux (for example, those using a Gnome or KDE desktop) you can often get the terminal window by right-clicking anywhere on the desktop and selecting terminal option. In KDE a terminal is also called a "Konsole".
Once you have opened a terminal window you will see a command prompt. In linux the command prompt will look a bit like this:
user@server $ or just:
$ whereas the powershell will look a bit like this:
PS C:\Home >
Once you start working with our tools you will quickly be able to open a terminal window (or powershell) in your sleep.
3.2. Topographic data
Topographic data comes in a number of formats, but at a basic level most topographic data is in the form of a raster. A raster is just a grid of data, where each cell in the grid has some value (or values). The cells are sometimes also called pixels. With image data, each pixel in the raster might have several values, such as the value of red, green and blue hues. Image data thus has bands: each band is the information pertaining to the different colors.
Topographic data, on the other hand, is almost always single band: each pixel or cell only has one data value: the elevation. Derivative topographic data, such a slope or aspect, also tends to be in single band rasters.
It is possible to get topographic data that is not in raster format (that is, the data is not based on a grid). Occasionally you find topographic data built on unstructured grids, or point clouds, where each elevation data point has a location in space associated with it. This data format takes up more space than raster data, since on a aster you only need to supply the elevation data: the horizontal positions are determined by where the data sits in the grid. Frequently LiDAR data (LiDAR stands for Light Detection and Ranging, and is a method for obtaining very high resolution topographic data) is delivered as a point cloud and you need software to convert the point cloud to a raster.
For most of this book, we will assume that your data is in raster format.
3.3. Data sources
Before you can start analyzing topography and working with topographic data, you will need to get data and then get it into the correct format. This page explains how to do so.
3.3.1. What data does LSDTopoToolbox take?
The LSDTopoToolbox works predominantly with raster data; if you don’t know what that is you can read about it here: http://en.wikipedia.org/wiki/Raster_data. In most cases, the raster data you will start with is a digital elevation model (DEM). Digital elevation models (and rasters in general) come in all sorts of formats. LSDTopoToolbox works with three formats:
Table 2. File input and output options
Data type file extension Description
Ascii .asc This format is in plain text and can be read by a text editor. The advantage of this format is that you can easily look at the data, but the disadvantage is that the file size is extremely large (compared to the other formats, .flt).
Float .flt with a This is a binary file format meaning that you can’t use a text editor header file to look at the data. with extension The file size is greatly reduced compared to .asc data, however. .hdr. This format does not retain georeferencing information.
ENVI bil .bil with a This is the recommended format, because it works best with format header file GDAL (see the section GDAL), with extension and because it retains georeferencing information. .hdr.
Why don’t we use GeoTiff?
GeoTIFF is a widely used raster format that has the advantage of containing georeferencing and the raster data in a single file. The disadvantage is that for C++ code you need to have two libraries (libtiff and libgeotiff) installed before you can read GeoTIFF files. Because there are many open source, easily installed tools for converting GeoTIFF files (for example, the GDAL utilities and the python GDAL bindings) we have opted for portability and not included the GeoTIFF libraries in our software. If you have GeoTIFF files, you will need to convert them to a supported format before using LSDTopoTools.
Below you will find instructions on how to get data into the correct format: data is delivered in a wide array of formats (e.g., ESRI bil, DEM, GeoTiff) and you must convert this data before it can be used by LSDTopoTools.
3.3.2. Downloading data
If you want to analyze topography, you should get some topographic data! The last decade has seen incredible gains in the availability and resolution of topographic data. Today, you can get topographic data from a number of sources. The best way to find this data is through search engines, but below are some common sources:
Table 3. Sources of topographic data
Source Data type Description and link
opentopograph LiDAR Lidar raster and point cloud data, funded by the National Science y foundation. http://www.opentopography.org/ Source Data type Description and link
U.S. LiDAR and Lidar raster and point cloud data, and IFSAR (5 m resolution or Interagency IfSAR better), collated by NOAA. http://www.csc.noaa.gov/inventory/# Elevation Inventory
USGS national Various United States elevation data hosted by the United States Geological map viewer (including Survey. Mostly data from the United States. IfSAR and http://viewer.nationalmap.gov/basic/ LiDAR, and satellite imagery)
EarthExplorer Various Another USGS data page. THis has more global coverage and is a (including good place to download SRTM 30 mdata. LiDAR, IfSAR, http://earthexplorer.usgs.gov/ ASTER and SRTM data)
Spanish LiDAR LiDAR This site has lidar data from Spain: http://centrodedescargas.cnig.es/CentroDescargas/buscadorCatalog o.do?codFamilia=LIDAR
Finland LiDAR LiDAR Finland’s national LiDAR dataset: https://tiedostopalvelu.maanmittauslaitos.fi/tp/kartta?lang=en
Denmark LiDAR Denmark’s national LiDAR dataset: LiDAR http://download.kortforsyningen.dk/
Environment LiDAR LiDAR holdings of the Environment Agency (UK): Agency (UK) http://www.geostore.com/environment- LiDAR agency/WebStore?xml=environment-agency/xml/application.xml
Trentino (Italy) LiDAR Lidar from the Trentio, a province in the Italian Alps: LiDAR http://www.lidar.provincia.tn.it:8081/WebGisIT/pages/webgis.faces Global datasets
There are several global topographic datasets. The oldest of these is gtopo30, which was completed in 1996 and contains ~1 km resolution global data. This was followed by the Shuttle Radar Topography Mission (SRTM) that produced a 90 meter resolution DEM in 2003; followed by the Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) 30 meter resolution global DEM in 2009. In 2014 SRTM released a global 30 meter dataset. 2015 has seen the release of the WorldDEM, a 12 meter resolution topographic dataset.
GTOPO30, SRTM and ASTER data are all freely available for download. The WorldDEM is a commercial product. If you are looking for a global dataset at reasonable resolution that is not commercial, you will most likely choose between ASTER and SRTM. You can download ASTER and SRTM data at the same site and make your own comparisons, but ASTER has widely publicised data quality issues so we recommend the SRTM 30 meter data.
3.4. Projections and transformations
Many of our readers will be aware that our planet is well approximated as a sphere. Most maps and computer screens, however, are flat. This causes some problems.
To locate oneself on the surface of the Earth, many navigational tools use a coordinate system based on a sphere, first introduced by the "father or geography" Eratosthenes of Cyrene. Readers will be familiar with this system through latitude and longitude.
A coordinate system based on a sphere is called a geographic coordinate system. For most of our topographic analysis routines, a geographic coordinate system is a bit of a problem because the distance between points is measured in angular units and these vary as a function of position on the surface of the planet. For example, a degree of longitude is equal to 111.320 kilometers at the equator, but only 28.902 kilometers at a latitude of 75 degrees! For our topographic analyses tools we prefer to measure distances in length rather than in angular units.
To convert locations on a the surface of a sphere to locations on a plane (e.g., a paper map or your computer screen), a map projection is required. All of the LSDTopoTools analysis routines work on a projected coordinate system.
There are many projected coordinate systems out there, but we recommend the Universal Transverse Mercator (UTM) system, since it is a widely used projection system with units of meters.
So, before you do anything with topographic data you will need to:
1. Check to see if the data is in a projected coordinate system
2. Convert any data in a geographic coordinate systems to a projected coordinate system.
Both of these tasks can be done quickly an easily with GDAL software tools. 3.5. GDAL
Now that you know something about data formats, projections and transformations (since you read very carefully the preceding sections), you are probably hoping that there is a simple tool with which you can manipulate your data. Good news: there is! If you are reading this book you have almost certainly heard of GIS software, which is inviting since many GIS software packages have a nice, friendly and shiny user interface that you can use to reassuringly click on buttons. However, we do not recommend that you use GIS software to transform or project your data. Instead we recommend you use GDAL.
GDAL (the Geospatial Data Abstraction Library) is a popular software package for manipulating geospatial data. GDAL allows for manipulation of geospatial data in the Linux operating system, and for most operations is much faster than GUI-based GIS systems (e.g., ArcMap).
Here we give some notes on common operations in GDAL that one might use when working with LSDTopoTools. Much of these operations are carried out using GDAL’s utility programs, which can be downloaded from http://www.gdal.org/gdal_utilities.html. The appendices have instructions on how to get the GDAL utilities working. You will also have to be able to open a terminal or powershell. Instructions on how to do this are in the appendices.
3.5.1. Finding out what sort of data you’ve got
One of the most frequent operations in GDAL is just to see what sort of data you have. The tool for doing this is gdalinfo which is run with the command line:
$ gdalinfo filename.ext where filename.ext is the name of your raster.
This is used mostly to:
a. See what projection your raster is in.
b. Check the extent of the raster.
This utility can read Arc formatted rasters but you need to navigate into the folder of the raster and use the .adf file as the filename. There are sometimes more than one .adf files so you’ll just need to use ls -l to find the biggest one.
3.5.2. Translating your raster into something that can be used by LSDTopoToolbox
Say you have a raster but it is in the wrong format (LSDTopoToolbox at the moment only takes .bil, .flt and .asc files) and in the wrong projection. Note: LDSTopoToolbox performs many of its analyses on the basis of projected coordinates.
You will need to be able to both change the projection of your rasters and change the format of your raster. The two utilities for this are:
• gdalwarp
• gdaltranslate
Changing raster projections with gdalwarp
The preferred coordinate system is WGS84 UTM coordinates. For convert to this coordinate system you use gdalwarp. The coordinate system of the source raster can be detected by gdal, so you use the flag -t_srs to assign the target coordinate system. Details about the target coordinate system are in quotes, you want:
+proj=utm +zone=XX +datum=WGS84' where XX is the UTM zone. You can find a map of UTM zones here: http://www.dmap.co.uk/utmworld.htm. For example, if you want zone 44 (where the headwaters of the Ganges are), you would use:
'+proj=utm +zone=XX +datum=WGS84'
Put this together with a source and target filename:
$ gdalwarp -t_srs '+proj=utm +zone=XX +datum=WGS84' source.ext target.ext so one example would be:
$ gdalwarp -t_srs '+proj=utm +zone=44 +datum=WGS84' diff0715_0612_clip.tif diff0715_0612_clip_UTM44.tif note that if you are using UTM and you are in the southern hemisphere, you should use the +south flag:
$ gdalwarp -t_srs '+proj=utm +zone=19 +south +datum=WGS84' 20131228_tsx_20131228_tdx.height.gc.tif Chile_small.tif
There are several other flags that could be quite handy (for a complete list see the GDAL website).
• -of format: This sets the format to the selected format. This means you can skip the step of changing formats with gdal_translate. We will repeat this later but the formats for LSDTopoTools are ◦ AAIGrid for ASCII files. These files are huge so try not to use them.
◦ EHdr for ESRI float files. This used to be the only binary option but GDAL seems to struggle with it and it doesn’t retain georeferencing.
◦ ENVI for ENVI rasters. This is the preferred format. GDAL deals with these files well and they retain georeferencing. We use the extension bil with these files.
So, for example, you could output the file as:
$ gdalwarp -t_srs '+proj=utm +zone=44 +datum=WGS84' -of ENVI diff0715_0612_clip.tif diff0715_0612_clip_UTM44.bil
Or for the southern hemisphere:
$ gdalwarp -t_srs '+proj=utm +zone=19 +south +datum=WGS84' -of ENVI 20131228_tsx_20131228_tdx.height.gc.tif Chile_small.bil
• -tr xres yres: This sets the x and y resolution of the output DEM. It uses nearest neighbour resampling by default. So say you wanted to resample to 4 metres:
$ gdalwarp -t_srs '+proj=utm +zone=44 +datum=WGS84' -tr 4 4 diff0715_0612_clip.tif diff0715_0612_clip_UTM44_rs4.tif
LSDRasters assume square cells so you need both x any y distances to be the IMPORTANT same
• -r resampling_method: This allows you to select the resampling method. The options are:
◦ near: nearest neighbour resampling (default, fastest algorithm, worst interpolation quality).
◦ bilinear: bilinear resampling.
◦ cubic: cubic resampling.
◦ cubicspline: cubic spline resampling.
◦ lanczos: Lanczos windowed sinc resampling.
◦ average: average resampling, computes the average of all non-NODATA contributing pixels. (GDAL >= 1.10.0)
◦ mode: mode resampling, selects the value which appears most often of all the sampled points. (GDAL >= 1.10.0) So for example you could do a cubic resampling with:
$ gdalwarp -t_srs '+proj=utm +zone=44 +datum=WGS84' -tr 4 4 -r cubic diff0715_0612_clip.tif diff0715_0612_clip_UTM44_rs4.tif
• -te
• UTM South: If you are looking at maps in the southern hemisphere, you need to use the +south flag:
$ gdalwarp -t_srs '+proj=utm +zone=44 +south +datum=WGS84' -of ENVI diff0715_0612_clip.tif diff0715_0612_clip_UTM44.bil
Changing raster format with gdal_translate
Suppose you have a raster in UTM coordinates (zones can be found here: http://www.dmap.co.uk/utmworld.htm) but it is not in .flt format. You can change the format using gdal_translate (note the underscore). gdal_translate recognizes many file formats, but for LSDTopoTools you want either:
• The ESRI .hdr labelled format, which is denoted with EHdr.
• The ENVI .hdr labelled format, which is denoted with ENVI. ENVI files are preferred since they work better with GDAL and retain georeferencing.
To set the file format you use the -of flag, an example would be:
$ gdal_translate -of ENVI diff0715_0612_clip_UTM44.tif diff0715_0612_clip_UTM44.bil
Where the first filename.ext is the source file and the second is the output file.
Potential filename errors
It appears that GDAL considers filenames to be case-insensitive, which can cause data management problems in some cases. The following files are both considered the same:
Feather_DEM.bil feather_dem.bil
This can result in an ESRI .hdr file overwriting an ENVI .hdr file and causing the code to fail to load the data. To avoid this ensure that input and output filenames from GDAL processes are unique. 3.5.3. Clipping rasters with gdal
You might also want to clip your raster to a smaller area. This can sometimes take ages on GUI-based GISs. An alternative is to use gdalwarp for clipping:
$ gdalwarp -te
$ gdalwarp -te
Since this is a gdalwarp operation, you can add all the bells and whistles to this, such as:
• changing the coordinate system,
• resampling the DEM,
• changing the file format.
The main thing to note about the -te operation is that the clip will be in the coordinates of the source raster (input.tif). You can look at the extent of the raster using gdalinfo.
3.6. Looking at your data (before you do anything with it).
You might want to have a look at your data before you do some number crunching. To look at the data, there are a number of options. The most common way of looking at topographic data is by using a Geographic Information System (or GIS). The most popular commercial GIS is ArcGIS. Viable open source alternatives are [QGIS] is you want something similar to ArcGIS, and Whitebox if you want something quite lightweight.
3.6.1. Our lightweight python mapping tools
If you would like something really lightweight, you can use our python mapping tools, available here: https://github.com/LSDtopotools/LSDMappingTools. These have been designed for internal use for our group, so at this point they aren’t well documented. However if you know a bit of python you should be able to get them running. You will need python with numpy and matplotlib.
To look at a DEM, you will need to download LSDMappingTools.py and TestMappingTools.py from the Gitub repository. The latter program just gives some examples of usage. At this point all the plotting functions do are plot the DEM and plot a hillshade, but if you have python working properly you can plot something in a minute or two rather than having to set up a GIS. 3.7. Summary
You should now have some idea as to how to get your hands on some topographic data, and how to use GDAL to transform it into something that LSDTopoTools can use. Chapter 4. Getting LSDTopoTools
There are several ways to get our tools, but before you start downloading code, you should be aware of how the code is structured. Much of this chapter covers the details about what you will find when you download the code and how to download it with git, but if you just want to get started you can always skip to the final section.
Quick Instructions for preliminary data processing
1. Make sure you have a C++ compiler on your computer.
2. Use git to download one of the LSDTopoTools packages. You can find some of them here: https://github.com/LSDtopotools.
4.1. How the code is structured
Okay, if you are getting the LSDTopoTools for the first time, it will be useful to understand how the code is structured. Knowing the structure of the code will help you compile it (that is, turn the source code into a program).
4.1.1. Compiling the code
The software is delivered as C++ source code. Before you can run any analyses, you need to compile it, using something called a compiler. You can think of a compiler as a translator, that translates the source code (which looks very vaguely like English) into something your computer can understand, that is in the form of 1s and 0s.
The C++ source code has the extensions .cpp and .hpp. In addition, there are files with the extension .make, which give instructions to the compiler through the utility make.
Don’t worry if this all sounds a bit complex. In practice you just need to run make (we will explain how to do that) and the code will compile, leaving you with a program that you can run on your computer.
4.1.2. Driver functions, objects and libraries
LSDTopoTools consists of three distinct components:
• Driver functions: These are programs that are used to run the analyses. They take in topographic data and spit out derivative data sets.
• Objects: The actual number crunching goes on inside objects. Unless you are interested in creating new analyses or are part of the development team, you won’t need to worry about objects.
• Libraries: Some of the software need separate libraries to work. The main one is the TNT library that handles some of the computational tasks. Unless otherwise stated, these are downloaded with the software and you should not need to do anything special to get them to work.
When you download the code, the objects will sit in a root directory. Within this directory there will be driver_function_* directories as well as a TNT directory.
Driver functions
If you are using LSDTopoTools simply to produce derivative datasets from your topographic data, the programs you will use are driver functions. When compiled these form self enclosed analysis tools. Usually they are run by calling parameter files that point to the dataset you want to analyze, and the parameters you want to use in the analysis.
The .make files, which have the instructions for how the code should compile, are located in the driver_functions_* folders.
For example, you might have a driver function folder called /home/LSDTopoTools/driver_functions_chi, and it contains the following files:
$ pwd /home/LSDTopoTools/driver_functions_chi $ ls chi_get_profiles_driver.cpp chi_step1_write_junctions_driver.cpp chi_get_profiles.make chi_step1_write_junctions.make chi_m_over_n_analysis_driver.cpp chi_step2_write_channel_file_driver.cpp chi_m_over_n_analysis.make chi_step2_write_channel_file.make
In this case the .make files are used to compile the code, and the .cpp files are the actual instructions for the analyses.
Objects
LSDTopoTools contains a number of methods to process topographic data, and these methods live within objects. The objects are entities that store and manipulate data. Most users will only be exposed to the driver functions, but if you want to create your own analyses you might have a look at the objects.
The objects sit in the directory below the driver functions. They all have names starting with LSD, so, for examples, there are objects called LSDRaster, LSDFlowInfo, LSDChannel and so on. Each object has both a .cpp and a .hpp file.
If you want the details of what is in the objects in excruciating detail, you can go to our automatically generated documentation pages, located here: http://www.geos.ed.ac.uk/~s0675405/LSD_Docs/index.html. Libraries
THe objects in LSDTopoTools required something called the Template Numerical Toolkit, which handles the rasters and does some computation. It comes with the LSDTopoTools package. You will see it in a subfolder within the folder containing the objects. This library compiled along with the code using instructions from the makefile. That is, you don’t need to do anything special to get it to compile or install.
There are some other libraries that are a bit more complex which are used by certain LSDTopoTools packages, but we will explain those in later chapters when we cover the tools that use them.
4.1.3. The typical directory layout
Figure 2. The typical directory structure of LSDTopoTools.
4.2. Summary
You should now have some idea as to where to retrieve the code, and what you will find in your directories once it is downloaded. We are now ready to actually move on to using the code!
4.3. Getting the code using Git
The development versions of LSDTopoTools live at the University of Edinburgh’s code development pages, sourceEd, and if you want to be voyeuristic you can always go to the timeline there and see exactly what we are up to.
If you actually want to download working versions of the code, however, your best bet is to go to one of our open-source working versions hosted on Github. To get code on Github you will need to know about the version control system git.
What follows is an extremely abbreviated introduction to git. If you want to know more about it, there are thousands of pages of documentation waiting for you online. Here we only supply the basics.
4.3.1. Getting started with Git
We start with the assumption that you have installed git on your computer. If it isn’t installed, you should consult the appendices for instructions on how to install it.
You can call git with:
$ git
Note: Much of what I will describe below is also described in the Git book, available online.
If it is your first time using git, you should configure it with a username and email:
$ git config --global user.name "John Doe" $ git config --global user.email [email protected]
Now, if you are the kind of person who cares what the internet thinks of you, you might want to set your email and username to be the same as on your Github account (this is easily done online) so that your contributions to open source projects wil be documented online.
You can config some other stuff as well, if you feel like it, such as your editor and merge tool. If you don’t know what those are, don’t bother with these config options:
$ git config --global merge.tool vimdiff $ git config --global core.editor emacs
Note: If you want a local configuration, you need to be in a repository (see below) and use the --local instead of --global flag.
You can check all your options with
$ git config --list core.repositoryformatversion=0 core.filemode=true core.bare=false core.logallrefupdates=true core.editor=emacs user.name=simon.m.mudd [email protected] merge.tool=vimdiff
4.3.2. Pulling a repository from Github
Okay, once you have set up git, you are ready to get some code! To get the code, you will need to clone it from a repository. Most of our code is hosted on Github, later in this chapter we <
Go to Github and navigate to a repository you want to grab (in git parlance, you will clone the repository). Here is one that you might try: https://github.com/LSDtopotools/LSDTopoTools_ChiMudd2014.
If you look at the right side of this website there will be a little box that says HTTPS clone URL. Copy the contents of this box. In your powershell or terminal window type
$ git clone https://github.com/LSDtopotools/LSDTopoTools_ChiMudd2014.git
The repository will be cloned into the subdirectory LSDTopoTools_ChiMudd2014. Congratulations, you just got the code!
Keeping the code up to date
Once you have the code, you might want to keep up with updates. To do this, you just go to the directory that contains the repository whenever you start working and run
$ git pull -u origin master
The origin is the place you cloned the repository from (in this case a specific Github repository) and master is the branch of the code you are working on. Most of the time you will be using the master branch, but you should read the git documentation to find out how to branch your repository.
Keeping track of changes
Once you have an updated version of the code you can simply run it to do your own analyses. But if you are making modification to the code, you probably will want to track these changes. To track changes use the git commit command.
If you change multiple files, you can commit everything in a folder (including all subdirectories) like this:
$ git commit -m "This is a message that should state what you've just done." .
Or you can commit individual, or multiple files:
$ git commit -m "This is a message that should state what you've just done." a.file $ git commit -m "This is a message that should state what you've just done." more.than one.file 4.3.3. Making your own repository
If you start to modify our code, or want to start keeping track of your own scripts, you might create your own repositories using git and host them on Github, Bitbucket or some other hosting website.
First, you go to a directory where you have some files you want to track. You will need to initiate a git repository here. This assumes you have git installed. Type:
git init to initiate a repository. If you are downloading an LSDTopoTools repository from github, you won’t need to init a repository.
So now you gave run git init in some folder to initiate a repository. You will now want to add files with the add command::
$ ls a.file a_directory $ git add a.file a_directory
Git adds all files in a folder, including all the files in a named subdirectoy.
If you want to add a specific file(s), you can do something like this:
$ git add *.hpp $ git add A_specific.file
Committing to a repository
Once you have some files in a repository,
$ git commit -m "Initial project version" .
Where the . indicates you want everything in the current directory including subfolders.
Pushing your repository to Github
Github is a resource that hosts git repositories. It is a popular place to put open source code. To host a repository on Github, you will need to set up the repository before syncing your local repository with the github repository. Once you have initiated a repository on Github, it will helpfully tell you the URL of the repository. This URL will look something like this: https://github.com/username/A_repository.git.
To place the repository sitting on your computer on Github, you need to use the push command. For example:
$ git remote add origin https://github.com/simon-m-mudd/OneD_hillslope.git $ git push -u origin master Counting objects: 36, done. Delta compression using up to 64 threads. Compressing objects: 100% (33/33), done. Writing objects: 100% (36/36), 46.31 KiB, done. Total 36 (delta 8), reused 0 (delta 0) To https://github.com/simon-m-mudd/OneD_hillslope.git * [new branch] master -> master Branch master set up to track remote branch master from origin.
Once you have uploaded an initial copy, you will need to keep it in sync with local copies. You can push things to github with::
$ git push -u origin master
One thing that can go wrong is that your repository will be out of sync, and you will get messages like this:
! [rejected] master -> master (non-fast-forward) error: failed to push some refs to 'https://github.com/simon-m-mudd/OneD_hillslope.git' hint: Updates were rejected because the tip of your current branch is behind hint: its remote counterpart. Merge the remote changes (e.g. 'git pull') hint: before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details.
You can try to fix this by making a pull request:
$ git pull origin and if you are lucky you will not have to engage in conflict resolution. If you do get a conflict (for example if someone else has pushed a change and you started from an outdated file), you will need to merge the files. Doing that is beyond the scope of this documentation, but there are many resources on the web for using git so help is only a few electrons away.
4.4. Where to get the code
Okay, now to actually get the code! There are several versions floating around. They are not all the same!. The best two ways to get the code both involve downloading the code from github. One way to do this is to go into a repository and download the repository as a .zip file.
If, however, you are planning on keeping up with updates, it is probably better to use the software git to get the code.
4.4.1. Latest release versions on GitHub
We post the latest release versions of the software on GitHub.
The github site is: https://github.com/LSDtopotools. This site contains a number of offerings: documentation, notes, automation scripts, plotting scripts and other goodies. At the moment (September, 2015) the "latest release" offerings here are rather limited, but we hope to change this in the near future.
4.4.2. CSDMS
When we publish papers that use new algorithms, we tend to post the code on the Community Surface Dynamics Modeling System website, found here: http://csdms.colorado.edu/wiki/Main_Page.
These versions are the ones used in the publications, so they are representative of the code used in the papers but not the latest versions. Currently our CSDMS offerings are:
• A tool for examining river profiles: http://csdms.colorado.edu/wiki/Model:Chi_analysis_tools
• A tool for finding channel heads: http://csdms.colorado.edu/wiki/Model:DrEICH_algorithm
• A tool for measuring hillslope length: http://csdms.colorado.edu/wiki/Model:Hilltop_flow_routing
• A tool for finding bedrock outcrops: http://csdms.colorado.edu/wiki/Model:SurfaceRoughness
4.5. Summary
You should now have some idea as to where to retrieve the code, and what you will find in your directories once it is downloaded. We are now ready to actually move on to using the code! Chapter 5. First Analysis
If you have jumped directly to here
You might be so eager to start that you have jumped directly here without reading any of the background material. This is fine as long as you know what you are doing! To start this chapter, you should:
• Make sure you can open a terminal or powershell window.
• Make sure you have a c++ compiler (we use g++) and the make tool installed.
• Make sure you have git installed.
• Make sure you have the GDAL utilities installed and working.
• Get some topographic data and convert it to projected coordinates (we prefer WGS1984 UTM projections).
• Make sure you have python with scipy including numpy and matplotlib working on your computer.
If you understand all of the preliminary steps, you are ready to move on to your first analysis. If not, the previous chapters will get you up to speed.
5.1. Preparing your data and folders
Don’t be messy! Your life will be much more pleasant if you set up a sensible directory structure before your start performing analyses. The programs in LSDTopoTools allow you to both read and write data from directories of your choice, so there is no need to pile everything in one directory. In fact, we would recommend keeping your data quite separate from the code. Terminal window cheat sheet
Here are some commands and shortcuts that you will find useful while using the terminal window. They are things that work in Linux (and should therefore work on OSX, since it is built on Linux). Not all will work in a Windows powershell, which is why if you are running Windows we recommend, creating a virtual Linux machine on your Windows computer. Useful commands are:
• Navigate to a directory: $ cd /directory/name/.
• Go down a directory level: $ cd ..
• See what directory you are in: $ pwd
• See what files are in this directory: $ ls
• Copy a file: cp /my/directory/a.file /another/directory/new.filename
• Move a file: mv /my/directory/a.file /another/directory/new.filename
There are also a number of keyboard shortcuts that are useful:
• Use tab to autocomplete (start typing a filename and hit tab and Linux will complete the filename for you!)
• Use ctrl-a to go to the beginning of a line in the terminal.
• Use ctrl-e to go to the end of a line in the terminal.
• Use the up and dawn arrows to go through your history of commands.
• If you highlight text with you mouse you can hit the center mouse button to copy text.
How you organize your directories is, of course, up to you, but we can gently suggest a directory structure. Because LSDTopoTools is distributed from several different GitHub repositories, It probably makes sense to make one directory to house all the different repositories, and another to house your data.
The tutorials will be based on a structure where the repositories are located in a folder home/LSDTT_repositories/ and the data is located in a folder home/topographic_data/. If you have a different directory structure just substitute in your directories when running the examples. If you do this for a living (like we do), you might want to set up a sensible structure for you topographic data, for example by having folders for the type of data, e.g.: $ pwd home/topographic_data/ $ ls Aster30m SRTM90m lidar IfSAR SRTM30m
Then within each data type I’ve arranged geographically, e.g.:
$ pwd home/topographic_data/ $ cd lidar $ ls California Colorado Italy Switzerland
Again, the way you organize this data is totally up to you, but you will save yourself from substantial amounts of stress later if you set up a sensible directory structure from the start.
5.2. Get and compile your first LSDTopoTools program
Okay, the first step is to navigate to the folder where you will keep your repositories. In this example, that folded is called /home/LSDTT_repositories. In a terminal window, go there with the cd command:
$ cd /home/LSDTT_repositories/
You can use the pwd command to make sure you are in the correct directory. If you don’t have the directory, use mkdir to make it.
5.2.1. Clone the code from Git
Now, clone the repository from GitHub. The repository in the first example is here: https://github.com/LSDtopotools/LSDTopoTools_AnalysisDriver. The command to clone is:
$ pwd /home/LSDTT_repositories/ $ git clone https://github.com/LSDtopotools/LSDTopoTools_AnalysisDriver.git 5.2.2. If you would rather not use git
Perhaps you feel like being difficult and have decided not to use git, because you find its name offensive or because Linus Torvalds once threw an apple core at your cat.
In that case you can download a zipped version of the repository, and unzip it
$ pwd /home/LSDTT_repositories/ $ wget https://github.com/LSDtopotools/LSDTopoTools_AnalysisDriver/archive/master.zip $ gunzip master.zip
5.2.3. Compile the code
Okay, now you should have the code. You will still be sitting in the directory /home/LSDTT_repositories/, so navigate up to the directory LSDTopoTools_AnalysisDriver/Analysis_driver/.
$ pwd /home/LSDTT_repositories/ $ cd LSDTopoTools_AnalysisDriver $ cd Analysis_Driver
You can now compile the code with
$ make -f Drive_analysis_from_paramfile.make
I am afraid there will be a lot of warnings. We apologize for being naughty programmers. However, after all of those warnings you should be able to type ls and see a program called LSDTT_analysis_from_paramfile.out. Congratualtions! You have compiled your first LSDTopoTools program. You are now ready to do some analysis.
5.3. Running your first analysis
We are going to run the first example on some example data. For the purposes of this example, we are going to put the data into a folder called home/topographic_data/IfSAR. We called the folder IfSAR since the data is derived from IfSAR. Again, you can call the data whatever you like, but you need to adjust the path names for your directory structure.
5.3.1. First analysis: example data
Navigate into your data folder and download the data using the wget tool We have placed several example datasets on a github repository. Today we will be working with a topographic map from Scotland, you can get it with:
$ wget https://github.com/LSDtopotools/ExampleTopoDatasets/raw/master/WhiteadderDEM.tif
This data is in .tif format! Quite a lot of the data you might download from the web is in this format. LSDTopoTools doesn’t read tif files (that is a job for the future), so you need to convert to a valid file format. We will convert the data using GDAL: see the section Translating your raster into something that can be used by LSDTopoToolbox.
To convert the data to ENVI bil format (which is our preferred format) type:
$ gdal_translate -of ENVI WhiteadderDEM.tif WhiteadderDEM.bil
Now, see if the file is there
$ ls WhiteadderDEM.bil WhiteadderDEM.hdr WhiteadderDEM.bil.aux.xml WhiteadderDEM.tif
If you looked at the file in a GIS you might have additional files with the extension .aux.xml. The important thing is that you now have files with the extensions bil and hdr.
Important: There are two formats that use the file extension bil: the ENVI format (which is the one we use) and an ESRI format. Make sure your bil files are in ENVI format. You can always check using gdalinfo.
5.3.2. Placing the paramfile
The code is flexible enough that the parameter file can be in a different location from the data, but I think it is good practice to keep the parameter files with the data. The parameter file not only runs the software, but more importantly it is a reproducible record of your analyses! So if you are doing research you should save these files. The software is designed so that if you send someone else the parameter file and the DEM they can reproduce your analysis exactly. This differentiates our software from GUI driven software like ArcMap and QGIS.
Okay, lets actually move the file. An example file came with the source code. You can use the cp` command to make a copy of this file in your data folder.
TIP:Keep two terminal windows open, one in the directory with your data, and one in the directory with the compiled code.
Note:Your directories will have different names than those shown here so modify the paths shown below appropriately: If you follow our advice, go into the folder with the data and copy across the parameter file:
$ pwd /home/topographic_data/IfSAR $ cp /home/LSDTT_repositories/LSDTopoTools_AnalysisDriver/Analysis_driver/Example.LSDTT_driver Example.LSDTT_driver
Alternatively you can copy from the Analysis_driver folder:
$ pwd /home/LSDTT_repositories/LSDTopoTools_AnalysisDriver/Analysis_driver /home/topographic_data/IfSAR $ cp Example.LSDTT_driver /home/topographic_data/IfSAR/Example.LSDTT_driver
Now make sure you are in the topography folder and check to see if the file is there:
$ pwd /home/topographic_data/IfSAR $ ls Example.LSDTT_driver WhiteadderDEM.bil.aux.xml WhiteadderDEM.tif WhiteadderDEM.bil WhiteadderDEM.hdr
We will use Example.LSDTT_driver as a template, but will want to change some of the values in that file. You might want to keep the original, so we suggest making a copy of the parameter file:
cp Example.LSDTT_driver Whiteadder.LSDTT_driver
5.3.3. Modifying the parameter file
Before you run the program you need to modify the parameter file. The parameter file is just plain text, so you will need a text editor to modify it. You can modify it in your favorite text editor, but don’t use a program that inserts a bunch of stupid formatting like Wordpad or Word.
In fact most text editors in Windows systems have the unfortunate habit of inserting diabolical hidden characters, called control characters that you will never see or notice if you just look at text but will completely screw up the program. We have endeavored to remove these characters within our code, but I highly recommend editing the parameter file either in linux, or using a text editor on Windows that won’t insert these characters.
For windows, we use Pspad or Atom For linux, we use either emacs or vim
These linux text editors take a bit of getting used to, so unless you are going to start writing code, you should probably stick with Pspad or Atom.
In many text editors, you can select the text formatting. It turns out there are different formattings for different operating systems. You should use the magic of the internet to determine how to change the text formatting. Many editors have the options MAC, DOS, and UNIX formatting. You want UNIX formatting.
1. Okay, lets get started modifying this file. Open it in your text editor. It will look a little bit like this::
# This is a driver file for LSDTopoTools # Any lines with the # symbol in the first row will be ignored
# File information dem read extension: bil dem write extension: bil read path: /home/topographic_data/IfSAR read fname: WhiteadderDEM
# Parameters for various topographic analysis min_slope_for_fill: 0.0001 # pixel_threshold_for_channel_net: 200
# The different analyses to be run write fill: true write hillshade: true # write channel_net: true
2. These files have a specific format. Any line that starts with the # symbol is ignored: you can put comments here.
3. Lines with parameters are separated with a colon (:). The text before the colon is the parameter, and the text after the colon is the parameter value. For example, in the above file, dem read extension is the parameter and bil is the parameter value.
• The parameter names are NOT case sensitive: dem read extension is the same as DEM rEaD extenSION as far as the program is concerned.
• The parameter values ARE case sensitive: bil is NOT the same as BIL.
• The program will only understand the parameter name if you get it exactly correct. So if you misspell or put an underscore where a space should be, the program will not be able to understand. So be careful when editing these files!!
4. Okay, first, we want to make sure the file path and the file names are correct. These two lines:: dem read extension: bil dem write extension: bil
tell the program that you want to read and write ENVI files. That is our intention, so we will leave these lines alone. The default is bil so you could actually delete these two lines and the program would still work.
5. Next are lines for the read path and the read fname. If you didn’t have lines for these it would default to the path of the parameter file and the name of the parameter file, excluding everything after the .. However I would recommend assigning these. To figure out what the path to your data is, first make sure the data is there using ls and then type pwd to get the path:
$ pwd /home/topographic_data/IfSAR
6. The read fname is the name of the DEM WITHOUT the extension. So if the DEM is called WhiteadderDEM.bil then the read fname would be WhiteadderDEM. These names are CASE SENSITIVE.
You should modify your parameter file with the correct directory:
# This is a driver file for LSDTopoTools # Any lines with the # symbol in the first row will be ignored
# File information dem read extension: bil dem write extension: bil read path: /home/topographic_data/IfSAR read fname: WhiteadderDEM
# Parameters for various topographic analysis min_slope_for_fill: 0.0001 # pixel_threshold_for_channel_net: 200
# The different analyses to be run write fill: true write hillshade: true # write channel_net: true
IMPORTANT Do not just copy the above file: your directory will be different!
7. You can also change the path and name of the files you write. The keywords are write path and write fname. For example: write path: /home/smudd/a_different/directory write fname: DifferentDEMname
If you leave these blank then the output will just write to the read directory. For now don’t add write path information.
8. Further down there are some parameters:
# Parameters for various topographic analysis min_slope_for_fill: 0.0001 # pixel_threshold_for_channel_net: 200
The first one min_slope_for_fill sets a minimum topographic slope after the fill function. The fill function makes sure there are no internally drained basins in the DEM, and is a standard task in topographic analysis.
The parameter name has underscores: don’t replace these with spaces or the IMPORTANT program won’t understand!
The parameter is actually a bit redundant since the default for this parameter is 0.0001, so deleting this line wouldn’t change the output. However, the line is left in if you want to change it.
The next line has a # symbol in front so is ignored by the program.
9. The next bit tells the program what you want to do with the DEM.:
# The different analyses to be run write fill: true write hillshade: true # write channel_net: true
In this case these instructions are telling the program to write the fill DEM and the hillshade DEM. The program will not write a channel network (write channel_net) since this line has a # as its first character.
You might be asking: doesn’t ArcMap and QGIS have fill and hillshade functions? They do indeed, but for large rasters our code is much faster, and using our parameter files you can create reproducible analyses that can easily be sent to collaborators, students, clients, etc.
These functions will only run if the parameter value is true.
10.Okay, save your changes to the parameter file; we will now move on to performing the analyses. 5.3.4. Running the analyses (in this case, writing fill and hillshade rasters)
1. You need to run the program (LSDTT_analysis_from_paramfile.out) from the folder containing the program. We would suggest keeping two terminal windows open, one in which you are in the directory of the data, and another where you are in the directory of the program. You can always find out what directory you are in by using the command pwd.
2. LSDTT_analysis_from_paramfile.out runs with two arguments:
3. The path to the parameter file.
4. The name of the parameter file.
5. You should have already found the path to your data (go to your data folder and type pwd). The name of the parameter file includes extension. So to run the program you type this:
./LSDTT_analysis_from_paramfile.out /home/topographic_data/IfSAR Whiteadder.LSDTT_driver
The ./ is a Linux thing. When you run programs you tell Linux that the program is in this directory with ./.
6. Once you’ve entered the above command, there will be some output to screen telling you what the code is doing, and then it will finish.
7. LSDTT_analysis_from_paramfile.out has put the output in the data folder, so use ls in this folder to see if the data is there:
$ ls Example.LSDTT_driver WhiteadderDEM.tif Whiteadder_hs.hdr WhiteadderDEM.bil Whiteadder_fill.bil Whiteadder.LSDTT_driver WhiteadderDEM.bil.aux.xml Whiteadder_fill.hdr WhiteadderDEM.hdr Whiteadder_hs.bil
8. Hey, look at that! There are a bunch of new files. There are two new rasters, each with a bil file and a hdr file. These are the fill raster: Whiteadder_fill.bil and the hillshade raster Whiteadder_hs.bil.
5.3.5. Look at the output
1. Now that you’ve done some analyses, you can look at the data in either your favorite GIS or using python. QGIS.
Viewing data in QGIS
Start QGIS. Then open the resulting raster using the Add raster layers dialog: a. image:: ./_images/QGIS/AddRaster.jpg
#. Choose the raster. You want the ``bil`` files:
b. image:: ./_images/QGIS/OpenRaster.jpg
#. If you open the hillshade, you should get a nice pretty picture like this:
c. image:: ./_images/QGIS/WAHS.jpg
#. Great! Now you have got a handle on some of the basics of running ``LSDTT_analysis_from_paramfile.out`` and looking at the results.
5.4. Summary
By now you should be able to clone one of our programs from Github, compile it, and run it on your computer. Now we can move on to more complex topographic analyses. Chapter 6. Simple surface metrics (slope, curvature, aspect, etc)
By now you should have compiled the program LSDTT_analysis_from_paramfile.out. If you haven’t done this, go to the previous chapter: First Analysis.
If you have jumped directly to here
1. You need to have a compiler, the make utility, and the GDAL utilities available. If you don’t know what these are go to the previous chapters.
2. You need to get and compile the source code; see here: [get-and-compile-source].
3. You should make a folder for your data; see here: [make-folder].
4. If you don’t have data, get the example data; see here: [get-example-data].
5. Move the parameter file to the data folder; see here: [param-to-data-folder].
6. Rename your parameter file (so you don’t confuse this with other analyses). You can do this using the cp command:
$ cp Example.LSDTT_driver A_sensible_name.LSDTT_driver
the extension to the parameter file (in this case .LSDTT_driver) doesn’t matter. NOTE You could call the file Wacky.Monkey_Gumballs if you wanted. It is the format of this file that is important, not the filename.
7. Open two terminal windows (Instructions here: :ref:`getting-onto-servers`).
6.1. Modifying the parameter file
1. You should be starting with a parameter file in your data folder that has been copied from somewhere else. You will now need to modify this file.
2. Open the file in your favorite text editor. If this is from a previous analysis, you might see something a bit like this: # This is a driver file for LSDTopoTools # Any lines with the # symbol in the first row will be ignored
# File information dem read extension: bil dem write extension: bil read path: /home/smudd/topographic_tools/topographic_data/NextMap/Scotland read fname: WhiteadderDEM
# Parameters for various topographic analysis min_slope_for_fill: 0.0001 # pixel_threshold_for_channel_net: 200
# The different analyses to be run write fill: true write hillshade: true # write channel_net: true
3. If a line has a # symbol, that line is a comment and the program LSDTT_analysis_from_paramfile.out will ignore it.
4. First, we need to deal with the file format and location. The file information in this case is:
# File information dem read extension: bil dem write extension: bil read path: /home/smudd/topographic_tools/topographic_data/NextMap/Scotland read fname: WhiteadderDEM
This will work for me, but you will have to change this line:
read path: /home/smudd/topographic_tools/topographic_data/NextMap/Scotland
to your path:
read path: /home/s01429521/my/sensible/path/Scotland/
To figure out what the path is you can type pwd in the terminal window when you are in your data folder.
If you are not using the example data, you will need to change the read fname to the name of your data. So for example, if you are starting with a DEM called Sierra.bil, the read fname will be Sierra. Some notes abot parameter files
• The parameter names are NOT case sensitive: dem read extension is the same as DEM rEaD extenSION as far as the program is concerned.
• The parameter values ARE case sensitive: bil is NOT the same as BIL.
• The program will only understand the parameter name if you get it exactly correct. So if you misspell or put an underscore where a space should be, the program will not be able to understand. So be careful when editing these files!!
5. Now lets move on to parameter values. At the moment these are:
# Parameters for various topographic analysis min_slope_for_fill: 0.0001 # pixel_threshold_for_channel_net: 200
The first and third lines above are comments, and are ignored by the program. We don’t actually need the min_slope_for_fill parameter for this run, but if you leave that it it won’t affect the program.
6. To get our simple surface metrics, we are going to use a polyfit function. This fits a polynomial to the topographic surface over a fixed window, and then calculates topographic metrics of this polynomial rather than calculating metrics on the data itself. This technique is employed to smooth high frequency noise, such as that from pits and mounds caused by falling trees.
For LiDAR data, we have found that you want a polyfit window that is around 7 metres in radius. This is based on work by Roering et al., 2010 and Hurst et al., 2012. For coarser data, you probably want to smooth over at least 1 pixel radius, so if you havbe a 10m DEM your window radius should be >10m.
In this example we are using old NextMap data (it was processed around 2007). Sadly this data isn’t so great: it is full of lumps. The data resolution is 5 metres, but we are going to use a polyfit radius of 15.1 metres to make sure we get three pixels on each side of the centre pixel.
The keyword is polyfit_window_radius, so in your parameter file you should have these lines:
# Parameters for various topographic analysis min_slope_for_fill: 0.0001 # pixel_threshold_for_channel_net: 200 polyfit_window_radius: 15.1
7. We also want to add some lines to the parameter file to designate a method for calculating slope. The default method is called d8. It takes the slope between a pixel and its steepest downslope neighbor. For this example, we want the polyfit method, wherein the data is fit with a polynomial and the slope is determined by differentiating this polynomial. To switch the slope method to polyfit, you use the flag for slope_method:
# Methods used in analyses slope_method: polyfit
The first line above (# Methods used in analyses) is a comment so ignored by the program, but it is useful to add these comments to the parameter files so that other people can tell what you are doing.
8. Now you should tell the program what rasters to write. In this case we want curvature, aspect, and slope, so this section of the parameter file should look like:
# The different analyses to be run write slope: true write curvature: true write aspect: true
9. Okay, save your changes to the parameter file; we will now move on to performing the analyses. It should look like this:
# This is a driver file for LSDTopoTools # Any lines with the # symbol in the first row will be ignored
# File information dem read extension: bil dem write extension: bil read path: /home/smudd/topographic_tools/topographic_data/NextMap/Scotland read fname: WhiteadderDEM
# Parameters for various topographic analysis min_slope_for_fill: 0.0001 # pixel_threshold_for_channel_net: 200 slope_method: polyfit polyfit_window_radius: 15.1
# The different analyses to be run write slope: true write aspect: true write curvature: true 6.2. Running the analyses (in this case, writing fill and hillshade rasters)
1. You will now need to run the program LSDTT_analysis_from_paramfile.out. Some details about running this program are in the first tutorial ([First-analysis]) in case you have forgotten.
2. I renamed my parameter file `Whiteadder_Surf.LSDTT_driver, so to run the code you need to type the following into the command line::
$ ./LSDTT_analysis_from_paramfile.out /home/smudd/topographic_tools/topographic_data/NextMap/Scotland Whiteadder_Surf.LSDTT_driver
3. The program will spit out text to screen as it works. Once it is finished, you can look at the data in your favorite GIS. You can check to see if all the files are there by going into your data folder and typing ls. You should see something like:
$ ls Example.LSDTT_driver Whiteadder_fill.bil Whiteadder.LSDTT_driver Whiteadder_Surf_curvature.hdr WhiteadderDEM.bil Whiteadder_fill.hdr Whiteadder_Surf_aspect.bil Whiteadder_Surf.LSDTT_driver WhiteadderDEM.hdr Whiteadder_hs.bil Whiteadder_Surf_aspect.hdr Whiteadder_Surf_slope.bil WhiteadderDEM.tif Whiteadder_hs.hdr Whiteadder_Surf_curvature.bil Whiteadder_Surf_slope.hdr
4. Something is actually a little funny here. Why are the output files all named Whiteadder_Surf…? The default output file names are the name of the parameter file with the extension removed. If you want to change the output names you can either change the name of the parameter file, or, alternatively, you can include the write file name in the parameter file:
write fname: MyFileName
5. Another thing to note: If you use ArcMap to calculate curvature, it will get the sign wrong! Ridgetops have negative curvature and valleys have positive curvature. This is reversed in ArcMap. Our software gives the correct curvature.
6.3. Summary
You should now be able to extract some simple topographic metrics from a DEM using our Driver_analysis program. Chapter 7. Chi analysis
In the late 1800s, G.K. Gilbert proposed that bedrock channel incision should be proportional to topographic gradients and the amount of water flowing in a channel. Since then, many geomorpholgists have attempted to extract information about erosion rates from channel profiles. Chi analysis is a method of extracting information from channel profiles that attempts to compare channels with different discharges first proposed by Leigh Royden and colleagues at MIT. LSDTopoTools has a number of tools for performing chi analysis.
This document gives instructions on how to use the segment fitting tool for channel profile analysis developed by the Land Surface Dynamics group at the University of Edinburgh. The tool is used to examine the geometry of channels using the integral method of channel profile analysis. For background to the method, and a description of the algorithms, we refer the reader to Mudd et al. (2014). For background into the strengths of the integral method of channel profile analysis, the user should read Perron and Royden (2013, ESPL).
This document guides the user through the installation process, and explains how to use the model. You will need a c compiler for this tutorial. If you have no idea what a c compiler is, see the appendix. Visualisation of the model results is performed using Python scripts. We recommend installing Python(x,y) or Anaconda and running the scripts within Spyder (which is installed with Python(x,y) and Anaconda).
Both the recommended compiler and Python(x,y) are open source: you do not need to buy any 3rd party software (e.g., Matlab) to run our topographic analysis!
7.1. Background to chi analysis
Chi analysis, or \chi analysis, is a means of normalizing channel gradient for either drainage area or discharge.
Chi analysis is a method for examining channel profiles. If you are familiar with chi analysis, you can skip ahead to the section [Get and compile the chi analysis tools]. This section covers the background to the method, and why it is useful.
7.1.1. Topographic expression of climate, tectonics, and lithology
Sorry the sicetion is under construction! But if you want to read more about chi analysis, have a look at these papers: Perron and Royden, 2013; Royden and Perron, 2013; Mudd et al., 2014.
7.2. Get the chi analysis tools
First navigate to the folder where you will keep your repositories. In this example, that folder is called /home/LSDTT_repositories. In a terminal window, go there with the cd command: $ cd /home/LSDTT_repositories/
You can use the pwd command to make sure you are in the correct directory. If you don’t have the directory, use mkdir to make it.
7.2.1. Clone the code from Git
Now, clone the repository from GitHub:
$ pwd /home/LSDTT_repositories/ $ git clone https://github.com/LSDtopotools/LSDTopoTools_ChiMudd2014.git
Alternatively, get the zipped code
If you don’t want to use git, you can download a zipped version of the code:
$ pwd /home/LSDTT_repositories/ $ wget https://github.com/LSDtopotools/LSDTopoTools_ChiMudd2014/archive/master.zip $ gunzip master.zip
GitHub zips all repositories into a file called master.zip, so if you previously NOTE downloaded a zipper repository this will overwrite it.
7.2.2. Compile the code
Okay, now you should have the code. You will still be sitting in the directory /home/LSDTT_repositories/, so navigate up to the directory LSDTopoTools_ChiMudd2014/driver_functions_MuddChi2014/.
$ pwd /home/LSDTT_repositories/ $ cd LSDTopoTools_ChiMudd2014 $ cd driver_functions_MuddChi2014
There are a number of makefiles (thse with extension .make in this folder). These do a number of different things that will be explained later in this chapter. We will compile them as we go through the various different types of chi analysis.
7.2.3. Get some example data
We are going to use example data from the Mandakini River in Northern India. This river was the focus of a study by Rahul Devrani and others, you can find it here: http://onlinelibrary.wiley.com/doi/10.1002/2015GL063784/full (the paper is open access).
Again, we want to make sure our data is arranged somewhere sensible. Make a directory for datasets and perhaps a folder specific to India. Again, you don’t need to follow the same naming conventions as in these examples, but you will have to remember the directory names!
I would open a second terminal window (one should already be open in the driver_functions_MuddChi2014 folder) and navigate to the data folder:
$ cd /home/LSDTT_Data/India/
The SRTM data from the catchment is stored on the data repository at GitHub: https://github.com/LSDtopotools/ExampleTopoDatasets.
You probably don’t want to clone this repository since it contains a lot of DEMs, so why don’t you just download the relevant files directly:
$ wget https://github.com/LSDtopotools/ExampleTopoDatasets/raw/master/Mandakini.bil $ wget https://github.com/LSDtopotools/ExampleTopoDatasets/raw/master/Mandakini.hdr
We are also going to use a parameter file, which can be grabbed from Github:
$ wget https://github.com/LSDtopotools/ExampleTopoDatasets/raw/master/example_parameter_files/Ma ndakini.driver
Okay, you are now ready to move on to the examples.
7.3. Chi analysis, part 1: getting the channel profiles
Our chi analysis method involves two steps. The first extracts a channel profile from a DEM. This processes is separated from the rest of the chi analysis for memory management reasons: these steps involve a DEM but once they are completed chi analysis can proceed with much smaller .chan files Quick guide
If you already know more or less what you are doing, but need a quick reminder, here are the steps involved:
1. Download your DEM.
2. Project it into a projected coordinate system (we usually use UTM).
3. Export the DEM in .flt or .bil format. See the section on using GDAL .
4. If the programs aren’t complied, make them with: chi_step1_write_junctions.make and chi_step2_write_channel_file.make
5. Run the program chi1_write_junctions.exe on your DEM.
6. Import the junction raster (*.JI.flt) into a GIS and pick a junction (this is easiest if you also import the stream order (*_SO.flt) and hillshade (*_HS.flt).
7. Run chi2_write_channel_file.exe to get the .chan file. Once you do this you are ready to move on to section two: running the chi analysis!
7.3.1. Overview
In this section we will extract a channel network, and from this channel network we will choose a junction (or junctions) from which we will run chi analyses. Later in this chapter we will go over tools for running chi analysis across all the channels in a basin.
7.3.2. Running the channel network extraction
The channel extraction code requires two steps. In the first step, the toolkit takes the raw DEM and prints several derived datasets from it. The main dataset used for the next step is the junction index dataset. The second step involves selecting a junction from which the chi analysis proceeds.
Compiling the source code for junctions and channels
1. This example will use the files that we downloaded in the previous section. If you have your own files, you will need to substitites the correct file and directory names.
2. Make sure the necessary files are in your directory: $ pwd /home/LSDTT_Data/India/
$ ls Mandakini.driver Mandakini.bil Mandakini.hdr
3. The driver file must contain three lines. The first line is the name of the DEM without the extension. In this example the name is Mandakini. The next line is a minimum slope for the fill function. The default is 0.0001. The third line is the threshold number of pixels that contribute to another pixel before that pixel is considered a channel. You can play with these numbers a bit, in this example, I have set the threshold to 300 (it is a 90m DEM so in the example the threshold drainage area is 2.5x106 m2). Here are the first 3 lines of the file:
mandakini 0.0001 300
You can check this in Linux with the less command. Just type less Mandakini.drver to see the file and q to quit.
4. Okay, if you have been following along, you should have two terminal windows open. One should be open in the folder containing your data, and the other should be open in the folder with the source code driver functions.
Table 4. The two terminal windows
Data terminal window Source code terminal window
[source,paramfile] [source,paramfile] ------$ pwd $ pwd /home/LSDTT_data/India/ /home/LSDTT_repositories/LSDTopoTools_ChiMu $ ls dd2014/driver_functions_MuddChi2014/ Mandakini.driver Mandakini.bil Mandakini.hdr $ ls ---- chi_get_profiles_driver.cpp chi_get_profiles.make chi_m_over_n_analysis_driver.cpp chi_m_over_n_analysis.make chi_step1_write_junctions_driver.cpp chi_step1_write_junctions.make chi_step2_write_channel_file_discharge.cpp chi_step2_write_channel_file_discharge.make chi_step2_write_channel_file_driver.cpp chi_step2_write_channel_file.make ---- 5. In the source code terminal window, you need to compile two programs (step1 and step2):
$ make -f chi_step1_write_junctions.make <
6. This will make two programs, chi1_write_junctions.exe and chi2_write_channel_file.exe. Once you have done this, you need to run the driver program.
Writing junctions
1. For writing junctions, the driver program is called chi1_write_junctions.exe. It takes 2 arguments. The first is the path name into the folder where your data is stored, and the second is the name of the driver file. To run the program, just type the program name and then the path name and driver file name.
WARNING The path has to end with a ‘/’ symbol.
If you are working in Linux, then the program name should be proceeded with a ./ symbol. Here is a typical example:
$ ./chi1_write_junctions.exe /home/LSDTT_data/India/ Mandakini.driver
To run the code you need to be in the source code folder containing the .exe IMPORTANT file, NOT the folder with the data.
All the output from the software, however, will be printed to the data folder. That is, the software and data are kept separately.
2. In later sections you will see that the driver file has the same format for all steps, but for this step only the first three lines are read. The driver file has a bunch of parameters that are described later but there is a file in the distribution called Driver_cheat_sheet.txt that has the details of the parameter values.
3. This is going to churn away for a little while. If you have used incorrect filenames the code should tell you. The end result will be a large number of new files: The code prints
• A filled DEM (with _fill in the filename),
• A hillshade raster (with _HS in the filename),
• Information about the stream orders (file with _SO in the filename),
• A file with information about the junctions (_JI in the filename).
So your directory will be full of files like this: $ ls Mandakini.hdr Mandakini.driver Mandakini.bil Mandakini_HS.bil Mandakini_CP.bil Mandakini_HS.hdr Mandakini_CP.hdr Mandakini_JI.bil Mandakini_JI.hdr Mandakini_fill.bil Mandakini_SO.bil Mandakini_fill.hdr Mandakini_SO.hdr
4. You will then need to load these files into ArcMap and look at them. alternative to ArcMap is Whitebox which has the advantage of being open source. QGIS is another good open source alternative to ArcMap.
5. You want to look at the channel network and junctions. So at a minimum you should import
• the hillshade raster
• the stream order raster (_SO in filename) and
• the junction index raster (_JI in filename)
into your preferred GIS.
The stream order raster will display the channel network, with each channel having a stream order. The junction index file is the key file, you will need information from this file for the next step. In the image below, the channel network is in cool colours and the junctions are in warm colours. Each junction has a unique integer value, called the junction index. Figure 3. Stream network, with junctions in red and orange pixels
6. Now, find the part of the map where you want to do the chi analysis. You need to choose the junction at the downstream end of the channels where you will do your analysis. Use the identify tool (it looks like an i in a blue circle on ArcMap) to get the number of the junction that you want to use as the lowest junction in the channel network. In the below image the arrow points to junction number 51. Figure 4. Finding the junction number
7. Each junction has one and only one receiver junction, whereas it can have multiple donor junctions. When you choose a junction, the extracted channel traces down to the node one before the receiver junction. It then traces up the channel network, following the path that leads to the node the furthest flow distance from the outlet junction. That is, when junctions are reached as the algorithm moves upstream the upstream channel is determined by flow distance not drainage area. Below we show an image of this.
Extracting the .chan file
1. Now that you have the junction number, you need to run the second program. Before you run this program, you need to write a file that contains the parameters for the chi analysis.
2. The first 3 lines of this file MUST be the same as the driver file in step 1. The code does not check this so you need to make sure on your own this is the case.
3. The next two rows of the driver file are the junction number from which you want to extract the network. and something that controls how the channel network is "pruned". This is the ratio in area between the main stem and a tributary that must be exceeded for a tributary to be included in the analysis. If this number is 1 you only get the main stem. The smaller the number the more tributaries you get. A reasonable number seems to be ~0.02. Here is an example file: Mandakini 0.0001 300 51 0.01
4. There can be more information in the driver file (for example, parameters for a chi analysis), but the channel network extraction program will ignore these; it only looks at the first 5 lines of the driver function.
5. From here you run the program chi2_write_channel_file.exe. You need to include the path name and the name of the chi parameter file. In Linux the program should be proceeded with ./. Here is an example:
./chi2_write_channel_file.exe /home/smudd/topographic_tools/test_suites/Mandakini/ chi_parameters.driver
6. This will generate several files
• A DEM with basin in the filename. Immediately before the .flt extension the junction number will also be listed. This file is a raster containing the outline of the contributing pixels to the basin drained by the extracted channel network.
• A .chan file (with ChanNet and the junction number in the filename) will be printed.
• A .csv file. This file can be imported into ArcMap or other GIS software.
ArcMap should be able to see the '.csv' file.
Figure 5. Adding a csv file in ArcMap
If you load this layer and right click on it, you should be able to load the xy data Figure 6. Showing x and y data in ArcMap
Loading the csv file will give you a shapefile with the channel nodes, and loading the basin file will give you the basin. Here is the basin and the channel for junction 51 of the Mandakini dataset Figure 7. The extraced basin with its channels
Note how the channel extends downstream from the selected junction. It stops one node before the next junction. This way you can get an entire tributary basin that stops one node short of its confluence with the main stem channel.
Format of the .chan file
The segment fitting algorithm (see part 2) works on a "channel" file (we use the extension .chan to denote a channel file). The channel file starts with six lines of header information that is used to reference the channel to a DEM. f the channel is not generated from a DEM these six rows can contain placeholder values. The six rows are:
Table 5. File input and output options
Keyword Description
Nrows number of rows
Ncols number of columns
Xllcorner location in the x coordinate of the lower left corner
Yllcorner location in the y coordinate of the lower left corner
Node_spacing the spacing of nodes in the DEM
NoDataVal the value used to indicate no data
This header information is not used in the segment analysis; it is only preserved for channel data to have some spatial reference so that scripts can be written to merge data from the channel files with DEM data.
The rest of the channel file consists of rows with 9 columns.
• The first column is the channel number (we use c++ style zero indexing so the main stem has channel number 0).
• The second column is the channel number of the receiver channel (the channel into which this channel flows). The mainstem channel flows into itself, and currently the code can only handle simple geometries where tributaries flow into the main stem channel only, so this column is always 0.
• The third column is the node number on the receiver channel (which, recall, must be the main stem) into which the tributary flows. The main stem is defined to flow into itself. Suppose the main stem has 75 nodes. The third column would then be 74 for the main stem (because of zero indexing: the first node in the main stem channel is node 0. Nodes are organized from upstream down, so the most upstream node in the main stem channel is node zero. Suppose tributary 1 entered the main stem on the 65th node of the main stem. The third column for tributary 1 would be 64 (again, due to 0 indexing). • The 4th column is the node index that refers back to the LSDFlowInfo object.
• The 5th column is the row in a DEM the node occupies.
• The 6th column is the column in a DEM the node occupies.
• The 7th column is the flow distance from the outlet of the node. It should be in metres.
• The 8th column is the elevation of the node. It should be in meters.
• The 9th column is the drainage area of the node. It should be in meters squared.
Many of these columns are not used in the analysis but are there to allow the user to refer the channel file back to a DEM. Columns are separated by spaces so rows will have the format:
Chan_number receiver_chan receiver_node node_index row col flow_dist elev drainage_area
Here are the first few lines of the example file (Mandakini_ChanNet_51.chan):
648 587 290249.625 3352521.5 88.81413269 -9999 0 0 271 4864 70 322 71253.0625 4609.008789 3762552.25 0 0 271 4999 71 322 71164.25 4609 3770440.25 0 0 271 5140 72 322 71075.4375 4602 3778328.25 0 0 271 5288 73 322 70986.625 4596 3786216 0 0 271 5444 74 323 70861.02344 4591 3794104 0 0 271 5615 75 323 70772.21094 4574 3928199.25 0 0 271 5799 76 323 70683.39844 4571 3936087.25 0 0 271 5992 77 324 70557.79688 4565 3975527 0 0 271 6190 78 325 70432.19531 4564 3983414.75
Now that you have the .chan file you are ready to move to the next section of the chi analysis: part 2: constraining m/n and transforming profiles. This may have seen like quite a few steps, but once you get familiar with the workflow the entire process should take no more than a few minutes.
7.4. Chi profile analysis, part 2: constraining m/n and transforming profiles
This is part 2 of the chi profile analysis documentation. Part 1 can be found here in the section part 1: getting the channel profiles. This section of the documentation assumes you have created a .chan file. The format of the channel file is described in the section Format of the .chan file. These "channel" files contain information about flow distance, elevation and other properties of a channel network that are used to create a profile transformed into so-called "chi" (\chi) space. The transformation involves integrating drainage area along the length of the channel, so that comparisons of channel steepness can be made for channels of different drainage area.
The analysis is similar to slope-area analysis but has some advantages; the primary advantage of the method is that it does not take the gradient of noisy topographic data, and so is less noisy than slope- area analysis.
The main disadvantage is that the method assumes channel profiles are well described by predictions of the stream power erosion law. The various advantages and disadvantages of the method are described by Perron and Royden, 2013 ESPL.
7.4.1. Steps involved to perform channel analysis
After preparing the data (:ref:`chi-analysis-get-profiles`), performing the channel analysis involves 3 steps, including visualization of the data.
1. Performing a statistical analysis to constrain the best fit m/n ratio
2. Extracting the transformed chi-elevation profiles
3. Visualizing the analysis
7.4.2. Performing a statistical analysis to constrain the best fit m/n ratio
Once the profile data has been converted into a .chan file, the data can then be processed to determine the most likely m/n ratio for individual channels and also via the collinearity test (see Mudd et al (2014)).
Compiling the code
The code for running the statistical analysis to find the most likely m/n ratio can be compiled by caling the makefile chi_m_over_n_analysis.make.
If you are using a windows machine and have installed Cygwin you need to ensure that you have installed the make utility. The below instructions use the linux command prompt symbol ($), but as long as you have make and a compiler installed on widows these instructions should also work in a powershell terminal. See the section: The terminal and powershells for more details.
To make the file navigate the folder that contains it and run:
$ make -f chi_m_over_n_analysis.make
This will create the program chi_m_over_n_analysis.exe. Running the code
The program 'chi_m_over_n_analysis.exe` is run with 2 arguments to the command line.
• The first argument is the path name of the path where the .chan file is located, along with a driver file that contains the parameters of the analysis. All data will be printed to files in this path.
• The second argument is the name of the driver file. We typically use a .driver extension for the driver file but this is not a requirement.
For example, we call the program with:
./chi_m_over_n_analysis.exe /home/smudd/topographic_tools/test_suites/Mandakini/ chi_parameters.driver
The ./ leading chi_m_over_n_analysis.exe is only necessary on a linux system. The driver file contains a number of parameters for running the analysis. This file is used on several different programs so not all parameters are used by chi_m_over_n_analysis.exe. The parameters must be listed in the correct order and there cannot be any extra information between parameters (e.g., a string describing them). The parameters are:
Table 6. Cheat sheet for driver file
Row number Description Notes
1 The prefix of The original name of the DEM. If your .chan file is called the channel mandakini_ChanNet_21729.chan, then the first row of the driver file file should be mandakini.
2 Minimum Not used by chi_m_over_n_analysis.exe. slope for fill function
3 Number of Not used by chi_m_over_n_analysis.exe. contributing pixels for a channel
4 Junction This is the junction number that was specified when the .chan file number of was created. So if your .chan file is called channel. mandakini_ChanNet_21729.chan, then the fourth row should be 21729.
5 Area fraction Not used by chi_m_over_n_analysis.exe. for pruning
6 A_0 A reference drainage area for integrating to chi space (m^2) Row number Description Notes
7 Minimum The minimum number of pixels in a segment. See Mudd et al segment length (2014) for guidance. Values between 10-20 are recommended. The computational time required is a highly nonlinear inverse function of this parameter. 20 might lead to a run lasting a few minutes, whereas 5 might take many hours (or even days). We recommend starting with a value of 14.
8 \sigma The standard deviation of error on the DEM, and also error from geomorphic noise (e.g., boulders in the channel). In meters. See Mudd et al (2014) paper) For SRTM this should be something like 10-30 m. The larger this number, the fewer segments you will get (see Mudd et al (2014)).
9 starting m/n The starting m/n value to test the likelihood of.
10 \Delta m/n The change in m/n value that you want to loop over. For example, suppose the starting m/n is 0.2 and the change in m/nis 0.05, then the next m/n value after 0.2 is 0.25.
11 n m/n values The number of m/n values you want to loop through.
12 Target nodes The maximum number of nodes you want to run through the partitioning algorithm at a time. Recommended values are 80-140. The computational time is nonlinearly related to this parameter, 80 might take several minutes whereas 140 will take many hours, and 200 will take months.
13 Monte Carlo The number of iterations on the Monte Carlo routine that finds the iterations statistics for every node in the channel network rather than a subset of nodes.
14 This Not used chi_m_over_n_analysis.exe. One day we will remove this parameters is parameter but today is not that day. legacy.
15 Vertical drop The vertical drop over which slope-area analysis will be for S-A, in performed. meters
16 Horizontal The horizontal interval over which slope area analysis will be drop for S-A, in performed meters Row number Description Notes
17 Max change in The maximum change in drainage area of an interval for slope- A (in m^2) area analysis as a fraction of the area at the midpoint of an interval.
18 Target skip The "target skip", which is the average number of nodes the routine skips when it is trying to compute the best segments. If the DEM is 90m resolution, for example, the resolution the algorithm will work at is ~300 meters. Here is an example file:
And here is the cheat sheet (also included in driver_cheat_sheet.txt):
Table 7. Cheat sheet for driver file
Example value Description
e_bathdem filename prefix
0.0001 minimum slope, don’t change
300 N contributing pixels for a channel. Could reduce to, say 100 or even 50.
1332 junction number, this will change
0.05 area frac for channel pruning. 1= mainstem only, low numbers= more tribs
1000 A_0 for chi analysis: probably don’t need to change.
20 minimum segment length. Should be between 5-20.
20 sigma: some estimate of uncertainty in elevation data. Smaller = more segments
0.15 starting m/n for best for m/n testing
0.025 increment of m/n for best for m/n testing
20 number of m/n values tested for m/n testing
90 target length of nodes to be analy\ed for segments. Should be between 80-150
250 number of iterations for Monte Carlo analysis. 250 seems okay
0.95 Not used anymore!
20 Vertical interval for sampling for S-A analysis. Should be scaled to DEM resolution
500 Horizontal interval for sampling for S-A analysis. Should be scaled to DEM resolution Example value Description
0.2 An area thinning fraction for S-A analysis. 0.2 is probably about right.
2 The mean number of data nodes you skip for each node of segment analysis. For LiDAR this can be 10 or more. Nextmap can be 2-10. SRTM 0-2.
Once this program has run, it will print out a file with the filename prefix and an extension of .movern.
This program is computationally expensive! Increasing the target length of nodes to be analyzed and reducing the minimum segment length increases the WARNING computational time required in a highly nonlinear fashion. Increasing the skip value can reduce computational time required.
You can expect the computation to take several minutes (e.g. minimum segment length ~20, target nodes ~100, skip set so mainstem has 300-500 nodes analysed) to many hours (e.g. minimum segment length of 5, target nodes of 120-140, skip set such that thousands of nodes are analysed).
The 'movern' file
The .movern file is produced by the statistical analysis of the channel network in order to find the most likely m/n ratio. The filename contains information about parameter values; these are parsed by the visualisation algorithms. The format of the filename is the filename prefix, followed by BFmovern, followed by the sigma values, the skip value, the minimum segment length value the target nodes value and the junction number, all separated by the underscore symbol (_). The file then has the extension .movern. For example, the filename:
mandakini_BFmovern_20_2_20_120_51.movern
Indicates that
• \sigma = 20
• skip = 2
• minimum segment length = 20
• target nodes = 120
• and the junction number being analyzed is 51.
The format of the file is:
Table 8. The format of the .movern file Row Value
1 In the first column of the first row there is a placeholder value, -99, followed by the m/n ratios tested each followed by a space.
2 In the first column is a placeholder value, -99, followed by the mean AICc (from n_iterations iterations) for each tested m/n ratio for the collinearity test. These are separated by spaces.
3 In the first column is a place holder value of -99, followed by the standard deviation of the AICc for the collinearity test. When fits are extremely poor, the likelihood approaches zero. Calculating the AICc involves taking the logarithm of the likelihood, to avoid this, the code assigns a very small number to 0 likelihoods. This results in a high, but not infinite, value of AICc. These poor fits will have a standard deviation of zero.
Even rows The first column is the channel number. thereafter The following columns are the mean AICc values for that channel.
Odd rows The first column is the channel number. thereafter The following columns are the standard deviations of the AICc values for that channel.
Here is an example file:
-99 0.15 0.175 -99 4008 4008 -99 0 0 0 2004 2004 0 0 0 1 2004 2004 1 0 0 2 2004 2004 2 0 0 3 1766.89 1788.39 3 608.033 583.88 4 1905.04 1973.54 4 422.523 238.852 5 2004 2004 5 0 0 6 1975.36 1882.18 6 224.595 450.995
Performing a sensitivity analysis on the best fit m/n ratio
For structurally or tectonically complex landscapes, it can be difficult to constrain the m/n ratio. In such cases, it is wise to perform a sensitivity analysis of the best fit m/n ratio. To facilitate this, we provide a python script, movern_sensitivity_driver_generation.py, that generates a number of driver files with the parameters minimum_segment_length, sigma, mean_skip and target_nodes that vary systematically.
To run this script you will need to change the data directory and the filename of the original driver file within the script.
You will need to modify the script before you run it. On lines 41 and 43 you need to modify the data directory and driver name of your files:
# set the directory and filename DataDirectory = "/home/smudd/topographic_tools/test_suites/Mandakini/"
DriverFileName = "chi_parameters.driver"
If you are running this file in Spyder from a windows machine, the path name will have slightly different formatting:
DataDirectory = "m:\\topographic_tools\\test_suites\\Mandakini\\"
If you run from the command line you will need to navigate to the folder that contains NOTE the script.
The driver files will be numbered (e.g., my_driver.1.driver, my_driver.2.driver, etc.):
$ ls *.driver chi_parameters.1.driver chi_parameters.2.driver chi_parameters.driver
You can run these with:
$ ./chi_m_over_n_analysis.exe /home/smudd/topographic_tools/test_suites/Mandakini/ my_driver.1.driver
Or if you want to run them with no hangup and nice:
$ nohup nice ./chi_m_over_n_analysis.exe /home/smudd/topographic_tools/test_suites/Mandakini/ my_driver.1.driver &
And then just keep running them in succession until you use up all of your CPUs (luckily at Edinburgh we have quite a few)!
We have also written an additional python script called Run_drivers_for_mn.py which simply looks for all the drivers in folder and sends them to your server. Once again, you’ll need to modify this python script before you run it in order to give the script the correct data directory. In this case the relevant line is line 18:
DataDirectory = "/home/smudd/topographic_tools/test_suites/Mandakini/"
You can run it from command line with:
$ python Run_drivers_for_mn.py
Again, you’ll need to be in the directory holding this file to run it (but it doesn’t have to be in the same directory as the data).
This will send all drivers to your servers to be run, so if you generated 1000 WARNING drivers, then 1000 jobs will be sent to the server. Use with caution!
Extracting the transformed chi-elevation profiles
The next stage of the analysis is to extract the chi (\chi) profiles. To compile the program to extract the chi profiles, you need to use the makefile chi_get_profiles.make. The program is compiled with:
$ make -f chi_get_profiles.make
The makefile compiles a program called chi_get_profiles.exe. This is run, like chi_m_over_n_analysis.exe , with two arguments: the path name and the driver name:
$ ./chi_get_profiles.exe /home/smudd/topographic_tools/test_suites/Mandakini/ chi_parameters.driver
The driver file has exactly the same format as the driver file for chi_m_over_n_analysis.exe. A chi profile will be produced for each m/n value outlined by these elements in the driver file:
• 7th row: The starting (lowest) m/n value to test if it is the most likely.
• 8th row: The change in m/n value that you want to loop over (suppose the starting m/n is 0.2 and the change in m/n is 0.05, then the next m/n value after 0.2 is 0.25).
• 9th row: The number of m/n values you want to loop through.
Users may wish to modify these values in the driver file from the original values to explore only those values which have "plausible" values of the m/n ratio (see Mudd et al (2014) for guidance).
For each m/n ratio tested, the code produces a file with the extension .tree and the string within the filename fullProfileMC_forced. This filename also contains the m/n value so for example a filename might be called:
pa_basin_fullProfileMC_forced_0.3_5_2_20_100_3124.tree
The numbers in the filename are arranged in the following order: m/n ratio, \sigma value, mean skip, minimum segment length and target nodes. The final number before the extension (here, 3124) is copied from the 4th row of the driver file: it is the junction number. Users can assign different numbers to different basins to facilitate automation of data analysis.
7.4.3. The .tree file
The .tree file has as many rows as there are nodes in the channel network. There will be more nodes in the .tree file than in the .chan file because the code extends all tributaries to the outlet. Each row has 23 columns. The columns are
• 1st column: The channel number (like in .chan file)
• 2nd column: The receiver channel (like in .chan file)
• 3rd column: The node on receiver channel (like in .chan file)
• 4th column: The node index (like in .chan file)
• 5th column: The row of the node (like in .chan file)
• 6th column: The column of the node (like in the .chan file)
• 7th column: The flow distance of the node
• 8th column: The chi coordinate of the node
• 9th column: The elevation of the node
• 10th column: The drainage area of the node
• 11th column: The number of data points used to calculate node statistics. Because of the skipping algorithm (see Mudd et al (2013 draft manuscript)) not all nodes are analysed each iteration.
• 12th column: The mean M_{\chi} value for the node.
• 13th column: The standard deviation of the M_{\chi} value for the node.
• 14th column: The standard error of the M_{\chi} value for the node. • 15th column: The mean B_{\chi} value for the node.
• 16th column: The standard deviation of the B_{\chi} value for the node.
• 17th column: The standard error of the B_{\chi} value for the node.
• 18th column: The mean DW value for the node.
• 19th column: The standard deviation of the DW value for the node.
• 20th column: The standard error of the DW value for the node.
• 21st column: The mean fitted elevation for the node.
• 22nd column: The standard deviation of the fitted elevation for the node.
• 23rd column: The standard error of the fitted elevation for the node.
7.4.4. Visualizing the analysis
We have also provided python scripts for visualizing the data (that is "LSDVisualisation": our code uses British spelling).
AICc_plotting.py
This script makes a plot of the AICc as a function of the m/n ratio for each channel as well as for the collinearity test. The mean and standard deviation of the AICc is plotted. In addition the m/n ratio with the minimum AICc value is highlighted, and there is a horizontal dashed line depicting the minimum AICc value plus the standard deviation of the minimum AICc value. This dashed line can help the user determine which m/n ratios are plausible (see Mudd et al (2014)). Here is an example:
Figure 8. The AICc as a function of m/n. The m/n with the minimum AICc is the most likely m/n
To run the AICc_plotting.py script you must modify the path name and filename after line 35 of the script. The file it takes is a .movern file. Plotting m/n sensitivity: AICc_plotting_multiple.py
This script looks through a directory (you need to change the DataDirectory variable in the script) for any files with BFmovern_.movern in them and plots the AICc results. The code extracts the parameter values from the filename so each plotted figure has the parameter values in the title. Note that this script plots to file instead of to screen. You can change the kind of output file by changing the parameter OutputFigureFormat. See MatPlotLib documentation for options, but possibilities include jpg and pdf.
Plotting chi profiles and M_{\chi} values chi_visualisation.py
This script makes three figures. First, you must define the path name and the filename after line 39 of the script. This script takes a .tree file. The first figure is a plot of the channels in chi (\chi) space. The transformed data is in a semi-transparent solid line and the best fit segments are in dashed lines. Each tributary is plotted with a different colour. Here is an example:
Figure 9. Profiles of elevation as a function of \chi.
These examples are derived from numerical model landscapes and are "perfect". NOTE Natural channels will be considerably noisier.**
The second figure generated by this script is a figure showing the gradient in \chi-elevation space as a function of \chi. The gradient in \chi-elevation is indicative of a combination of erosion rate and erodibility, so these plots allow perhaps a clearer idea of the different segments identified by the segment fitting algorithms. The colour scheme from the first figure is carried forward, so that it is easy to identify the characteristics of each tributary. Here is an example: Figure 10. A plot of M_{\chi} as a function of \chi.
The third figure displays the longitudinal channel profiles (elevation as a function of flow distance), but these channel profiles are coloured by the gradient in \chi-elevation space. Here is an example:
Figure 11. A plot of M_{\chi} in profiles.
Plotting the sensitivity of best fit m/n values to parameters: bf_movern_sensitivity.py
This script looks in the directory DataDirectory for all the files with BFmovern_.movern in the filename and then compiles the best fit m/n ratio from these files. It then produces box and whisker plots of the best fit m/n ratio. The red line is the median m/n ratio, the box shows the 25th and 75th percentile m/n ratios, the whiskers show the data range, with outliers (as determined by the Matplotlib function boxplot) as "+" symbols. The boxes are notched; these give 85% confidence intervals to the median value after bootstrapping 10,000 times. Here is an example plot: Figure 12. Box and whisker plots of m/n ratios: this shows how m/n varies as a function of segment fitting parameters.
Plotting the spatial distribution of M_{\chi} values: raster_plotter_2d_only.py
This script contains two functions. One is for plotting the tributaries superimposed on a hillshade (or any other raster) and another is for plotting the M_{\chi} values superimposed on a raster (we usually do this on the hillshade).
For this to work, the .chan file must be referenced to a coordinate system. That means that the row and column information in the .chan file corresponds to the xllcorner, yllcorner and node spacing data in the first few lines of the .chan file. If you have the LSDTopoToolbox this will happen automatically, but if you are writing a script to generate your own .chan files you`ll need to ensure that your channel nodes have the correct row and column data.
The DEM used for the hillshade does not need to be the same as the DEM used for the .chan file (that is, it can have different n_rows and n_columns etc, so you can, in principle, do a chi analysis on a clipped version of a DEM but then plot the results on the full DEM extent). The two functions are:
1. coloured_chans_like_graphs: This takes two strings: the filename of the raster and the filename of the .tree file. You have to include the full path name for both of these files.
The colouring scheme used for the channels is the same as in the elevation and M_{\chi} plots made by chi_visualisation.py. Here is an example: Figure 13. A map showing the channels (colored by their number) within a basin
2. m_values_over_hillshade: Again, this takes two strings: the filename of the raster and the filename of the .tree file. You have to include the full path name for both of these files.
The colouring scheme on the M_\chi values is the same as in the chi_visualisation.py plot where M_\chi is plotted on the channel profile. Here is an example:
Figure 14. The M_{\chi} of channels within a basin
To run one or the other of these function you need to scroll to the bottom of the script and comment or uncomment one of the function lines. 7.4.5. A Sample Chi Analysis Workflow
Link to the full-size version of the flowchart
Figure 15. The workflow for finding the m over n ratio
7.5. Summary
You should now be able to extract some simple topographic metrics from a DEM using our Driver_analysis program. Chapter 8. Basinwide cosmogenic analysis
The LSDCosmoBasin toolkit is designed to automate calculation of basinwide erosion rates determined estimated from the concentration of in situ cosmogenic nuclides. Currently 10Be and 26Al are supported.
The toolkit requires:
• Data on cosmogenic samples.
• A file containing filenames of the topographic data, and optional filenames for shielding rasters.
• A parameter file.
The toolkit then produces * A csv file that contains results of the analysis. * A text file that can be copied into the CRONUS online calculator for data comparison.
8.1. Get the code and data basinwide cosmogenic analysis
This section goes walks you through getting the code and example data, and also describes the different files you will need for the analysis.
8.1.1. Get the source code for basinwide cosmogenics
First navigate to the folder where you will keep your repositories. In this example, that folder is called /home/LSDTT_repositories. In a terminal window, go there with the cd command:
$ cd /home/LSDTT_repositories/
You can use the pwd command to make sure you are in the correct directory. If you don’t have the directory, use mkdir to make it.
Clone the code from Git
Now, clone the repository from GitHub:
$ pwd /home/LSDTT_repositories/ $ git clone https://github.com/LSDtopotools/LSDTopoTools_CRNBasinwide.git
Alternatively, get the zipped code
If you don’t want to use git, you can download a zipped version of the code: $ pwd /home/LSDTT_repositories/ $ wget https://github.com/LSDtopotools/LSDTopoTools_CRNBasinwide/archive/master.zip $ gunzip master.zip
GitHub zips all repositories into a file called master.zip, so if you previously NOTE downloaded a zipper repository this will overwrite it.
Compile the code
Okay, now you should have the code. You will still be sitting in the directory /home/LSDTT_repositories/, so navigate up to the directory LSDTopoTools_BasinwideCRN/driver_functions_BasinwideCRN/.
$ pwd /home/LSDTT_repositories/ $ cd LSDTopoTools_CRNBasinwide $ cd driver_functions_CRNBasinwide
There are a number of makefiles (thse with extension .make in this folder). These do a number of different things that will be explained later in this chapter.
8.1.2. Getting example data: The San Bernardino Mountains
We have provided some example data. This is on our Github example data website.
The example data has a number of digital elevation models in various formats, but for these examples we will be only using one dataset, from the San Bernardino Mountains in California.
You should make a folder for your data using mkdir somewhere sensible. For the purposes of this tutorial I’ll put it in the following folder:
$ pwd /home/ExampleDatasets/SanBernardino/
Again, we will only take the data we need, so use wget to download the data: $ wget https://github.com/LSDtopotools/ExampleTopoDatasets/raw/master/SanBern.bil $ wget https://github.com/LSDtopotools/ExampleTopoDatasets/raw/master/SanBern.hdr $ wget https://github.com/LSDtopotools/ExampleTopoDatasets/raw/master/example_parameter_files/Sa nBern_CRNData.csv $ wget https://github.com/LSDtopotools/ExampleTopoDatasets/raw/master/example_parameter_files/Sa nBern_CRNRasters.csv $ wget https://github.com/LSDtopotools/ExampleTopoDatasets/raw/master/example_parameter_files/Sa nBern.CRNParam
You should now have the following files in your data folder:
$ pwd /home/ExampleDatasets/SanBernardino/ $ ls SanBern.bil SanBern_CRNData.csv SanBern.CRNParam SanBern.hdr SanBern_CRNRasters.csv
The file SanBern_CRNRasters.csv will need to be modified with the appropriate IMPORTANT paths to your files!
8.1.3. Setting up your data directories and parameter files
Before you can run the code, you need to set up some data structures.
If you downloaded the example data, these files will already exist. These instructions NOTE are for when you need to run CRN analysis on your own datasets.
1. You can keep your topographic data separate from your cosmogenic data, if you so desire. You’ll need to know the directory paths to these data.
2. In a single folder (again, it can be separate from topographic data), you must put a i) parameter file, a cosmogenic data file, and a raster filenames file .
3. These three files must have the same prefix, and each have their own extensions.
• The parameter file has the extension: .CRNParam.
• The cosmogenic data file has the extension _CRNData.csv.
• The raster filenames file has the extension _CRNRasters.csv.
4. For example, if the prefix of your files is SanBern, then your three data files will be SanBern.CRNParam, SanBern_CRNData.csv, and SanBern_CRNRasters.csv.
5. If the files do not have these naming conventions, the code WILL NOT WORK! Make sure you have named your files properly.
The parameter file
The parameter file contains some values that are used in the calculation of both shielding and erosion rates.
IMPORTANT This file must have the extension .CRNParam. The extension is case sensitive.
The parameter file could be empty, in which case parameters will just take default values. However, you may set various parameters. The format of the file is:
parameter_name: parameter_value
So for example a parameter file might look like:
An example CRNparam file
min_slope: 0.0001 source_threshold: 12 search_radius_nodes: 1 threshold_stream_order: 1 theta_step: 30 phi_step: 30 Muon_scaling: Braucher write_toposhield_raster: true write_basin_index_raster: true
There cannot be a space between the parameter name and the ":" character, so IMPORTANT min_slope : 0.0002 will fail and you will get the default value.
In fact all of the available parameters are listed above, and those listed above are default values. The parameter names are not case sensitive. The parameter values are case sensitive. These parameters are as follows:
Table 9. File input and output options
Keyword Input type default Description
min_slope float 0.0001 The minimum slope between pixels used in the filling function (dimensionless) Keyword Input type default Description source_thresho int 12 The number of pixels that must drain into a pixel ld to form a channel. This parameter makes little difference, as the channel network only plays a role in setting channel pixels to which cosmo samples will snap. This merely needs to be set to a low enough value that ensures there are channels associated with each cosmogenic sample. search_radius_ int 1 The number of pixels around the location of the nodes cosmo location to search for a channel. The appropriate setting will depend on the difference between the accuracy of the GPS used to collect sample locations and the resolution of the DEM. If you are using a 30 or 90m DEM, 1 pixel should be sufficient. More should be used for LiDAR data. threshold_stre int 1 The minimum stream or which the sample am_order snapping routine considers a 'true' channel. The input is a Strahler stream order. theta_step 30 int Using in toposhielding calculations. This is the step of azimuth (in degrees) over which shielding and shadowing calculations are performed. Codilean (2005) recommends 5, but it seems to work without big changes differences with 15. An integer that must be divisible by 360 (although if not the code will force it to the closest appropriate integer). phi_step int 30 Using in toposhielding calculations. This is the step of inclination (in degrees) over which shielding and shadowing calculations are performed. Codilean (2005) recommends 5, but it seems to work without big changes differences with 10. An integer that must be divisible by 360 (although if not the code will force it to the closest appropriate integer). Keyword Input type default Description
Muon_scaling string Braucher The scaling scheme for muons. Options are "Braucher", "Schaller" and "Granger". If you give the parameter file something other than this it will default to Braucher scaling. These scalings take values reported in COSMOCALC as described by Vermeesch 2007.
write_toposhiel bool true If true this writes a toposhielding raster if one d_raster does not exist. Saves a bit of timebut will take up some space on your hard disk!
write_basin_in bool true For each DEM this writes an LSDIndexRaster to dex_raster file with the extension _BASINS that has each of the basins that have been found for CRN analysis listed by basinID.
The cosmogenic data file
This file contains the actual cosmogenic data: it has the locations of samples, their concentrations of cosmogenics (10Be and 26Al) and the uncertainty of these concentrations.
The cosmogenic data file must have the extension _CRNData.csv. The extension is IMPORTANT case sensitive.
This is a .csv file: that is a comma separated value file. It is in that format to be both excel and pandas friendly.
The first row is a header that names the columns, after that there should be 7 columns (separated by commas) and unlimited rows. The seven columns are:: sample_name, sample_latitude, sample_longitude, nuclide, concentration, AMS_uncertainty, standardisation Important notes about _CRNData.csv files
• The sample name should not start with a number or have spaces.
• The latitude and longitude should be in decimal degrees. Negative latitude indicates southern hemisphere.
• Nuclide can be either "Be10" or "Al26". Any other option will be rejected.
• Concentration is in atoms/gram
• AMS uncertainty is also in atoms/gram
• Standardisation is the name of the standards used in the AMS measurements. This is not always so easy to find in published papers!! The defaults are "07KNSTD" for 10Be and "KNSTD" for 26Al. These seem to be used by many people after 2007 when Kuni Nishiizumi made them available (or at least that is when he published the paper). If the samples are from before 2007 and you don’t know the standard use, you should use "KNSTD" for 10Be and 26Al. There are many more standards floating around, but the Nishiizumi one seem the most widely used. The options are (take a deep breath), for 10Be:
Options for 10Be standardisation
"07KNSTD", "KNSTD", "NIST_Certified", "LLNL31000", "LLNL10000", "LLNL3000", "LLNL1000" "LLNL300", "NIST_30000", "NIST_30200", "NIST_30300", "NIST_30600", "NIST_27900" "S555","S2007", "BEST433", "BEST433N", "S555N", "S2007N"
And for 26Al:
Options for 26Al standardisation
"KNSTD", "ZAL94", "SMAL11", "0", "ZAL94N", "ASTER", "Z92-0222"
An example file would look like this (this is not real data): An example _CRNData.csv file
Sample_name,Latitude,Longitude,Nuclide,Concentration,Uncertainty,Standardisation LC07_01,-32.986389,-71.4225,Be10,100000,2500,07KNSTD LC07_04,-32.983528,-71.415556,Be10,150000,2300,07KNSTD LC07_06,-32.983028,-71.415833,Al26,4000,2100,KNSTD LC07_08,-32.941333,-71.426583,Be10,30000,1500,07KNSTD LC07_10,-33.010139,-71.435389,Be10,140000,25000,07KNSTD LC07_11,-31.122417,-71.576194,Be10,120502,2500,07KNSTD
If you followed the instructions earlier in the section Getting example data: The San Bernardino Mountains then you will have a CRNdata.csv file called Binnie_CRNData.csv in your data folder.
The raster names file
This file contains names of rasters that you want to analyze.
The raster names file must have the extension _CRNRasters.csv. The extension is IMPORTANT case sensitive.
This file is a csv file that has as many rows as you have rasters that cover your CRN data. Each row can contain between 1 and 4 columns.
• The first column is the FULL path name to the Elevation raster and its prefix (that is, without the .bil, e.g.:
/home/smudd/basin_data/Chile/CRN_basins/Site01/Site_lat26p0_UTM19_DEM
• The next column is either a full path name to a snow shielding raster or a snow shielding effective depth. Both the raster and the single value should have units of g/cm^2 snow depth. If there is no number here the default is 0.
• The next column is either a full path name to a self shielding raster or a self shielding effective depth. Both the raster and the single value should have units of g/cm^2 shielding depth. If there is no number here the default is 0.
• The next column is the FULL path to a toposhielding raster. If this is blank the code will run topographic shielding for you. Note: topographic shielding is the most computationally demanding step in the cosmo analysis.
A typical file might will look like this: An example CRNRasters.csv file
/home/smudd/basin_data/Site01/Site01_DEM,0,0,/home/smudd/basin_data/Site01/Site01_DEM_T opoShield /home/smudd/basin_data/Site02/Site02_DEM,5,10 /home/smudd/basin_data/Site03/Site03_DEM,5,/home/smudd/basin_data/Site03/Site03_DEM_Sel fShield /home/smudd/basin_data/Site04/Site04_DEM,/home/smudd/basin_data/Site04/Site04_DEM_SnowS hield,/home/smudd/basin_data/Site04/Site04_DEM_SelfShield /home/smudd/basin_data/Site05/Site05_DEM /home/smudd/basin_data/Site06/Site06_DEM,/home/smudd/basin_data/Site06/Site06_DEM_SnowS hield
8.2. Calculating Topographic Shielding
Cosmogenic nuclides are produced at or near the Earth’s surface by cosmic rays, and these rays can be blocked by topography (i.e., big mountains cast "shadows" for cosmic rays).
In most cases, you will not have topographic shielding rasters available, and will need to calculate these.
Shielding calculation are computationally intensive, much more so than the actual erosion rate computations. Because of the computational expense of shielding calculations, we have prepared a series of tools for speeding this computation.
The topographic shielding routines take the rasters from the _CRNRasters.csv file and the _CRNData.csv file and computes the location of all CRN basins. They then clips a DEM around the basins (with a pixel buffer set by the user). These clipped basins are then used to make the shielding calculations and the erosion rate calculations.
This process of clipping out each basin spans a large number of new DEM that require a new directory structure. A python script is provided to set up this directory structure in order to organize the new rasters.
This process uses a large amount of storage on the hard disk because a new WARNING DEM will be written for each CRN basin.
8.2.1. Steps for preparing the rasters for shielding calculations
Creation of subfolders for your topographic datasets
The first step is to create some subfolders to store topographic data. We do this using a python script
1. First, place the _CRNRasters.csv and _CRNData.csv file into the same folder, and make sure the _CRNRasters.csv file points to the directories that contain the topographic data. If you are working with the example data (see section Getting example data: The San Bernardino Mountains), you should navigate to the folder with the data (for this example, the folder is in /home/ExampleDatasets/SanBernardino/):
$ pwd /home/ExampleDatasets/SanBernardino/ $ ls Binnie_CRNData.csv SanBern_data_analysis.CRNfiles SanBern.hdr SanBern.bil SanBern_data_analysis.CRNparam
You will then need to modify SanBern_data_analysis.CRNfiles to reflect your directory:
/home/ExampleDatasets/SanBernardino/Binnie_CRNdata.csv /home/ExampleDatasets/SanBernardino/SanBern
The first line points to the CRNdata.csv file, and the second points to the DEM that will be analyzed. Note that if you are doing this on your own data you can have multiple lines for different DEMs.
In addition in this case we are not supplying and shielding rasters. For more details about the format of this file see the section: The raster names file.
2. Second, run the python script PrepareDirectoriesForBasinSpawn.py.
• You can clone this script from GitHub; find it here: https://github.com/LSDtopotools/LSDAutomation You will also need the file LSDOSystemTools.py from this repository. The LSDOSystemTools.py file contains some scripts for making sure directories are in the correct format, and for changing filenames if you happen to be switching between Linux and Windows. It is unlikely that you will need to concern yourself with its contents, as long as it is present in the same folder as the PrepareDirectoriesForBasinSpawn.py file.
The scripts can be downloaded directly using:
$ wget https://github.com/LSDtopotools/LSDAutomation/raw/master/PrepareDirectoriesForBasinS pawn.py $ wget https://github.com/LSDtopotools/LSDAutomation/raw/master/LSDOSystemTools.py
• You will need to scroll to the bottom of the script and change the path (which is simply the directory path of the _CRNRasters.csv file.
• You will need to scroll to the bottom of the script and change the prefix (which is simply prefix of the _CRNRasters.csv file; that is the filename before _CRNRasters.csv so if the filename is YoYoMa_CRNRasters.csv then prefix is YoYoMa. Note this is case sensitive.
In this example, scroll to the bottom of the file and change it to: if __name__ == "__main__": path = "/home/ExampleDatasets/SanBernardino" prefix = "SanBern" PrepareDirectoriesForBasinSpawn(path,prefix)
• This python script does several subtle things like checking directory paths and then makes a new folder for each DEM. The folders will contain all the CRN basins located on the source DEM.
If you are using the example data, the rather trivial result will be a directory called SanBern.
Spawning the basins
Now you will run a C++ program that spawns small rasters that will be used for shielding calculations. First you have to compile this program.
1. To compile, navigate to the folder /home/LSDTT_repositories/LSDTopoTools_CRNBasinwide/driver_functions_CRNBasinwide/. If you put the code somewhere else, navigate to that folder. Once you are in the folder with the driver functions, type:
$ make -f Spawn_DEMs_for_CRN.make
2. The program will then compile (you may get some warnings—ignore them.)
3. In the /driver_functions_CRNTools/ folder, you will now have a program Spawn_DEMs_for_CRN.exe. You need to give this program two arguments.
4. You need to give Spawn_DEMs_for_CRN.exe, the path to the data files (i.e., _CRNRasters.csv and _CRNData.csv), and the prefix, so if they are called YoMa_CRNRaster.csv the prefix is YoMa). Run this with:
PS> Spawn_DEMs_for_CRN.exe PATHNAME DATAPREFIX
in windows or:
$ ./Spawn_DEMs_for_CRN.exe PATHNAME DATAPREFIX
in Linux.
In our example, you should run: $ ./Spawn_DEMs_for_CRN.exe /home/ExampleDatasets/SanBernardino/ SanBern
The PATHNAME MUST have a frontslash at the end. WARNING /home/ExampleDatasets/SanBernardino/ will work whereas /home/ExampleDatasets/SanBernardino will lead to an error.
5. Once this program has run, you should have subfolders containing small DEMs that contain the basins to be analyzed. There will be one for every cosmogenic sample that lies within the DEM.
6. You will also have files that contain the same PATHNAME and PREFIX but have _Spawned added to the prefix. For example, if your original prefix was CRN_test, the new prefix will be CRN_test_Spawned.
7. In the file PREFIX_Spawned_CRNRasters.csv you will find the paths and prefixes of all the spawned basins.
8.2.2. The shielding computation
The shielding computation is the most computationally expensive step of the CRN data analysis. Once you have spawned the basins (see above section, Steps for preparing the rasters for shielding calculations), you will need to run the shielding calculations.
1. You will first need to compile the program that calculates shielding. This can be compiled with:
$ make -f Shielding_for_CRN.make
2. The compiled program (Shielding_for_CRN.exe) takes two arguments: the PATHNAME and the PREFIX.
3. You could simply run this on a single CPU after spawning the basins; for example if the original data had the prefix CRN_test before spawning, you could run the program with:
$ ./Shielding_for_CRN.exe PATHNAME CRN_test_Spawned
where PATHNAME is the path to your _CRNRasters.csv, _CRNData.csv, and .CRNParam (these files need to be in the same path).
If you only wanted to do a subset of the basins, you can just delete rows from the NOTE *_Spawned_CRNRasters.csv file as needed.
8.2.3. Embarrassingly parallel shielding
We provide a python script for running multiple basins using an embarrassingly parallel approach. It is written for our cluster: if your cluster uses qsub or equivalent, you will need to write your own script. However, this will work on systems where you can send jobs directly. 1. To set the system up for embarrassingly parallel runs, you need to run the python script ManageShieldingComputation.py, which can be found here: https://github.com/LSDtopotools/LSDAutomation. You can download it with:
$ wget https://github.com/LSDtopotools/LSDAutomation/raw/master/ManageShieldingComputation.py
2. In ManageShieldingComputation.py, navigate to the bottom of the script, and enter the path, prefix, and NJobs. NJobs is the number of jobs into which you want to break up the shielding computation.
3. Once you run this computation, you will get files with the extension _bkNN where NN is a job number.
4. In addition a text file is generated, with the extension _ShieldCommandPrompt.txt, and from this you can copy and paste job commands into a Linux terminal.
These commands are designed for the GeoSciences cluster at the WARNING University of Edinburgh: if you use qsub you will need to write your own script.
5. Note that the parameters for the shielding calculation are in the .CRNParam files. We recommend:
theta_step:8 phi_step: 5
These are based on extensive sensitivity analyses and balance computational speed with accuracy. Errors will be << 1% even in landscapes with extremely high relief. Our forthcoming paper has details on this.
6. Again, these computations take a long time. Don’t start them a few days before your conference presentation!!
7. Once the computations are finished, there will be a shielding raster for every spawned basin raster. In addition, the _CRNRasters.csv file will be updated to reflect the new shielding rasters so that the updated parameter files can be fed directly into the erosion rate calculators.
8.3. Calculating erosion rates
Okay, now you are ready to get the erosion rates. You’ll need to run the function from the directory where the compiled code is located (in our example, /home/LSDTT_repositories/LSDTopoTools_CRNBasinwide/), but it can work on data in some arbitrary location. 8.3.1. Compiling the code
To compile the code, go to the driver function folder (in the example, /home/LSDTT_repositories/LSDTopoTools_CRNBasinwide/driver_functions_CRNBasinwide) and type:
$ ./Basinwide_CRN.exe pathname_of_data file_prefix
• The pathname_of_data is just the path to where your data is stored, in this example that is /home/ExampleDatasets/SanBernardino/.
IMPORTANT You MUST remember to put a / at the end of your pathname.
• The filename is the PREFIX of the files you need for the analysis (that is, without the extension). In the example this prefix is SanBern (or SanBern_Spawned if you spawned separate shielding basins)
8.3.2. The output files
There are two output files. Both of these files will end up in the pathname that you designated when calling the program.
The first is called file_prefix_CRNResults.csv and the second is called file_prefix_CRONUSInput.txt where file_prefix is the prefix you gave when you called the program.
So, for example, if you called the program with:
$ ./basinwide_CRN.exe /home/ExampleDatasets/SanBernardino/ SanBern
The outfiles will be called:
SanBern_CRNResults.csv SanBern_CRONUSInput.txt
The _CRONUSInput.txt is formatted to be cut and pasted directly into the CRONUS calculator. The file has some notes (which are pasted into the top of the file): Header of the *_CRONUSInput.txt file
->IMPORTANT nuclide concentrations are not original! They are scaled to the 07KNSTD!! ->Scaling is averaged over the basin for snow, self and topographic shielding. ->Snow and self shielding are considered by neutron spallation only. ->Pressure is an effective pressure that reproduces Stone scaled production that is calculated on a pixel by pixel basis. ->Self shielding is embedded in the shielding calculation and so sample thickness is set to 0.
The _CRNResults.csv is rather long. It contains the following data in comma separated columns:
Table 10. Columns in the _CRNResults.csv file
Colum Name Units Description n
1 basinID Integer A unique identifier for each CRN sample.
2 sample_ string The name of the sample name
3 Nuclide string The name of the nuclide. Must be either 10Be or 26Al
4 latitude decimal The latitude. degrees
5 longitu decimal The longitude. de degrees
6 concent atoms/g The concentration of the nuclide. This is adjusted for the recent standard ration (e.g., 07KNSTD), so it may not be the same as in the original dataset.
7 concent atoms/g The concentration uncertainty of the nuclide. ration_ Most authors report this as only the AMS uncertainty. uncert The concentration is adjusted for the recent standard (e.g., 07KNSTD), so it may not be the same as in the original dataset.
8 erosion g cm-2 The erosion rate in mass per unit area: this is from the full spatially rate yr-1 distributed erosion rate calculator.
9 erosion g cm-2 The erosion rate uncertainty in mass per unit area: this is from the full rate yr-1 spatially distributed AMS_un erosion rate calculator. The uncertainty is only that derived from AMS cert uncertainty.
10 muon_u g cm-2 The erosion rate uncertainty in mass per unit area derived from muon ncert yr-1 uncertainty. Colum Name Units Description n
11 product g cm-2 The erosion rate uncertainty in mass per unit area derived from ion_unc yr-1 uncertainty in the production rate. ert
12 total_un g cm-2 The erosion rate uncertainty in mass per unit area that combines all cert yr-1 uncertainties.
13 AvgPro float The average production scaling correction for the basin. dScalin (dimens g ionless)
14 Average float The average topographic shielding correction for the basin. TopoShi (dimens elding ionless)
15 Average float The average self shielding correction for the basin. SelfShie (dimens lding ionless)
16 Average float The average snow shielding correction for the basin. SnowSh (dimens ielding ionless)
17 Average float The average combined shielding correction for the basin. Combin (dimens edScali ionless) ng
18 outlet_l decimal The latitude of the basin outlet. atitude degrees
19 OutletP hPa The pressure of the basin outlet (calculated based on NCEP2 data after ressure CRONUS).
20 centroi decimal The latitude of the basin centroid. d_latitu degrees de
21 centroi hPa The pressure of the basin centroid (calculated based on NCEP2 data after dPressu CRONUS). re
22 Centroi hPa This is the pressure needed to get basin averaged production scaling: dEffPre it is a means of translating the spatially distributed production data into ssure a single value for the CRONUS calculator. Colum Name Units Description n
23 Erosion g cm-2 The erosion rate you would get if you took production weighted scaling Rate_CO yr-1 and used SMOCA cosmocalc. LC_in_g _percm 2_peryr
24 Erosion -1mm kyr The erosion rate you would get if you took production weighted scaling Rate_CO and used SMOCA cosmocalc. Assumes 2650 kg m-2. LC_mm perkyr
-2 25 Erosion -1mm kyr The erosion rate to check against cosmocalc. Assumes 2650 kg m . Rate_in _mmpe rkyr
-2 26 Erosion -1mm kyr Uncertainty in the erosion rate. Assumes 2650 kg m . Rate_tot aluncer tainty_i n_mmp erkyr
8.4. Summary
You should now be able take date from basinwide cosmogenics and convert these into erosion rates. Appendix A: Software
There are quite a few different components you need to get working on your system to perform the examples in this book (sorry!). Some of these are essential, and some are optional and included for completeness. In the following appendices, instructions for getting them installed on Windows or Linux are included. Since the author of this book does not have any computers running on friut-based operating system, I’m afraid instructions for getting the software working on such systems will require a bit of homework, but in theory installation on such systems should be similar to installation on Linux systems.
A.1. Essentials
The following tools are core to the contents of this book, and will need to be installed before you can work on the exercises in the book.
A.1.1. Git
Git is version control software. It helps you keep track of changes to your scripts, notes, papers, etc. It also facilitates communication and collaboration through the online communities github and bitbucket. The source code for LSDTopoTools is on github so by using Git you can make sure you always have the latest version of the software.
A.1.2. C++ tools
A number of scientific programs are written in these languages so information on how to get them working on your windows machine is incluided here for completeness.
To get these working you will need
1. The compiler. This is what translates the program into something the computer can understand.
2. The tool make, which automates building programs.
3. The tool gdb. which stands for gnu debugger, a tool for debugging code.
4. The tool gprof, which is a profiler: it allows you to see which parts of your code are using the most computational resources.
A.1.3. Python
Python is a programming language used by many scientists to visualize data and crunch numbers. You can also use it to automate data management.
You will need:
1. The python programming language a. Scipy, for scientific python. It includes lots of useful packages like
i. Numpy for fast numerics.
ii. Matplotlib for plotting.
iii.Pandas for data analysis.
b. pip for python package management.
A.1.4. GDAL
GDAL (the Geospatial Data Abstraction Library) is used to manipulate topographic data so that it can be fed into LSDTopoTools.
A.2. Useful extras
You could find these tools useful. In particular, my documentation is written using something called asciidoctor, which is implemented in a programming language called Ruby.
A.2.1. A virtual machine
This isn’t exactly optional software, but rather what you might use if it becomes too much trouble to install things on either a Windows or an OS X operating system. A virtual machine is software that allows you to emulate an operating system within another operating systems (for example, you can create a Linux operating system within a Windows computer).
The software tools described in this book are all rather easy to install on a Linux system, and for other systems some components are rather tricky to get working. It therefore can be advantageous to run the tools described in this book within a virtual machine.
Several options exist. My Windows computer has a virtual Linux system installed with the VMware player but you do have to find the section on personal use to get the free version. Another option is the VirtualBox system.
A.2.2. Geographic Information Software
If you want to look at the data produced by LSDTopoTools, you could use our lightweight python tools, but in many cases you will want to use a GIS.
The common options are:
• ArcGIS The GIS most frequently used by commercial enterprise and government. It is commercial software and rather expensive. If your organisation has a licence, fantastic. However not all users of our software will have a licence so all tutorials in this book will be based on open source software.
• QGIS A GIS that behaves much like ArcGIS, but is open source. • Whitebox A very lightweight, open source GIS written in java. This is handy since it is quite portable: there is nothing to install, just copy the .jar file on your computer and away you go!
A.2.3. Ruby
Ruby is a programming language used frequently by web developers and has many package for building documentation and automating collection of data over the internet. In this book we will really only use it for documentation, but there is a large ecosystem of open source tools available in Ruby. It hasn’t been adopted to a great extent by the scientific community but you may still find useful tools, particularly if you are gathering online information.
The main reason we use Ruby is to generate our documentation using Asciidoctor, so if you fancy contributing to the documentation of getting the latest version, you will need to get Ruby and some associated tools.
You will need:
1. The Ruby programming language
a. Rubygems for updating ruby.
b. bumdler for managing updates and making sure scripts are up to date.
c. RubyDev kit which is needed for some other Ruby packages.
d. asciidoctor for making notes, documentation and books.
e. Ruby DevKit which is used by some Ruby extensions.
2. In addition you will need Node.js for some of the Ruby tools to work. Appendix B: Setting up on Windows
For the kind of scientific tools we will be using, Windows can sometimes be a bit difficult, since many scientific developers work in Linux and so installation for their software is easiest in a Linux environment. However, you can get Windows to behave a bit like Linux which will make your life easier for the purposes of installing the required software for the examples here.
Alternatively, you can get a "virtual" Linux machine running on your Windows machine.
Let me type that in bold text: it will be far easier to do things in a Linux environment than in a Windows environment. I strongly encourage you to build a virtual machine.
B.1. Working with the powershell
Much of what we do will be through a powershell window. This is a text-based interface into your windows system that allows you to install software and run scripts and code. It functions like a linux terminal.
First of all, you need to get a powershell running. On my windows version, you just type powershell into the bit of Windows where you searh for files or programs (it varies based on what windows system you are using). You should be able to find a way to get a powershell on your version of windows through the power of the internet.
B.1.1. Starting a powershell session
First, you will need to open an administrator powershell. In your powershell window, type
PS> Start-Process powershell -Verb runAs
NOTE The PS> denotes the powershell propmpt.
You might not have administrator priviliges on your computer. In that case you will need to convince the administrator to install everything for you, or you can WARNING ask them to install a linux virtual machine, which is described in the section [Turning-your-windows-machine-into-a-linux-machine].
B.2. Windows installation programs
Take a deep breath. I am afraid this is going to take a while. It can be done, but you may need psychological treatment before the end. You might consider just installing a Linux Virtual machine instead of trying to get the software installed within a Windows environment. B.2.1. Package management
If you are a windows user, you are probably used to installing software on windows the traditional way using installation files. The tools we are using involve many linked programs, and installing them by downloading windows installers from their respective websites can be tiresome. An alternative is to use something called a machine package manager, which makes installing software a bit less of a hassle.
Chocolatey package manager
Here we will use a machine package manager built specifically for windows called chocolatey.
To install chocolatey, type in the following in your administrative powershell:
PS> iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))
You then just have to sit back and wait while it installs.
To test if chocolatey is working after the installation, type choco -v. If you get an error NOTE saying it is not a recognized name of a cmdlet or path, then simply close your powershell windows and open new ones.
Once you have chocolatey installed, you can get the additional packages that are required for the tools we use in this book by using the choco command.
B.2.2. Git
You can install git by downloading the installation package from the Github desktop website. Alternatively, you can use chocolatey:
PS> choco install git
You can check if it works in the command line by calling
PS> git --version
If you install git desktop, you will get a git powershell link on your desktop. This powershell is quite useful when using git since it does things like highlights files in different colours depending on status (e.g., if they have been modified or not or of they are being tracked).
B.3. Tools for C++
There are several options for installing C++ and fortran compilers on your Windows machine. Two popular options, Mingw and https://www.cygwin.com/Cygwin] install something that behaves a bit like a Linux operating system (but perhaps it is easier just to set up Linux within your Windows computer?).
Another option for C++ is to install the developer toolkit from Microsoft, Visual Studio express. You can install the 2013 version using chocolatey:
PS> choco install visualstudioexpress2013windowsdesktop
You can also install the software by downloading from the Visual Studio website.
B.3.1. Cygwin
To install Cygwin, you must first install the program setup-x86.exe for a 32 bit system or setup-x86_64 for a 64 bit system.
When you run setup-*.exe, you will get a window that looks like this:
Figure 16. Cygwin setup.
Scroll down the the devel menu and select the following packages:
binutuls gcc core g++ gfortran gdb make
You can also install all sorts of other things from cygwin like Ruby, Git and Python, but you don’t need to do that if you’ve already installed them. In fact, you might want to make sure git is not selected if you have installed the Git powershell from the github desktop application.
Once you have selected the things you need, select next. You might need to install a bunch of additional packages because your selected packages depend on them. If this is your first time installing cygwin go and get a drink or have lunch since installation will take some time.
B.3.2. C++ libraries
Some of our more specialized components require libraries. These are very difficult to install on Windows, and you will possibly self harm if you attempt to do so. Why don’t you make a virtual Linux machine instead?
B.4. Python
Python in Windows can be a little bit more difficult to manage than python in Linux. You can get python through chocolatey, but it makes installing extra packages a bit of a pain.
Instead, you probably want to install python as part of a package that includes all the handy python libraries and associated programs.
The popular ones are:
1. Python(x,y) contains all manner of useful packages for scientists. I have found, however, that updating this is not so straightforward; if you have a look at the Python(x,y) downlaods page you will see that the updates have dependencies that dead end, so if you are on, say, Python(x,y) 2.6.X you will need to reinstall Python(x,y) 2.7.X if you want continued updates. Python(x,y) is also huge: it includes many python packages, most of which you will never use.
2. Winpython is another python package that has a nice updating interface.
3. Anaconda is another scientific package that includes all sorts of smaller packages. It seems to be able to handle updates better than Python(x,y) through its conda updating interface.
4. miniconda uses the same conda updating interface as anaconda, the difference is that with miniconda things arene’t installed automatically, so you will have to figure out what you want and then use conda to install new packages (e.g., pandas, scipy, gdal, etc.)
B.5. GDAL windows installation
You can download GDAL for windows from this website: https://trac.osgeo.org/gdal/wiki/DownloadingGdalBinaries. If you are on Windows, however, you might want to just use the GDAL bindings in python. Or, you can skip all of this and build a virtual linux machine on your windows computer.
B.6. Ruby
You can check to see if Ruby is installed on your system by typing PS> ruby -v and you can check the Rubygems version with
PS> gem -v
B.6.1. Install Ruby using choco
If these are not installed, install them using chocolatey (in an administrator powershell). While you are at it, install the RubyDev and Node.js package:
PS> choco install ruby PS> choco install rubygems PS> choco install ruby2.devkit PS> choco install node.js
B.6.2. Fix rubygems on Windows
At the time of this writing, rubygems is "broken" on Windows: if you try to install a gem file you will get an error that says you cannot make an SSL connection. You will need to fix this problem by copying a trust certificate into you rubygems folder.
1. First download the file AddTrustExternalCARoot-2048.pem.
It seems some browsers add formatting to this file when downloaded (Chrome WARNING does this, for example). I only managed to get this fix to work after downloading the file using Internet Explorer.
2. Next, find the location of your gems installation:
PS> gem which rubygems C:/Ruby21/lib/ruby/2.1.0/rubygems.rb
1. Go to the folder and then go into the SLL_certs subfolder. In this case it is at: C:/Ruby21/lib/ruby/2.1.0/rubygems/SSL_certs. Copy the .pem file into this folder. Rubygems should now work.
B.6.3. Fix RubyDevKit on Windows
The devkit sometimes does not work the first time around. If you get error messages about the devkit, you should go to the devkit folder (you need to find it on your system) and run: PS> ruby dk.rb init
This will generate a file called config.yml. Sometimes, the devkit installer will not find your Ruby installation, so you will need to add the path to your Ruby installation to this file. For example, if your Ruby instalation is sitting in C:/Ruby21/, then you should modify the config.yml file to look like:
The config.yml file:
# This configuration file contains the absolute path locations of all # installed Rubies to be enhanced to work with the DevKit. This config # file is generated by the 'ruby dk.rb init' step and may be modified # before running the 'ruby dk.rb install' step. To include any installed # Rubies that were not automagically discovered, simply add a line below # the triple hyphens with the absolute path to the Ruby root directory. # # Example: # # --- # - C:/ruby19trunk # - C:/ruby192dev # --- - C:/Ruby21/
B.6.4. Install some gems
From here we can install some useful ruby gems. Asciidoctor is really great for writing documentation. Bundler is useful for keeping Ruby packages up to date.
PS> gem install asciidoctor PS> gem install bundler
B.6.5. If you use Ruby with Java (you will probably not need this)
This is for users who are trying to get a Ruby extension that uses Java installed. If you use an extension that need a java link, you will need the gem
PS> gem install rjb -v '1.4.9'
But on my system this failed because you need to define the java runtime home. To do this, you need to figure out where your java installation is, and then define an environemnt variable $JAVA_HOME to point to this directory. To do so, you should do this (in an administrator Powershell):
PS> [Environment]::SetEnvironmentVariable("JAVA_HOME", "C:\Progra~2\Java\jre_7_55", "Machine")
Note that the powershell abbreviates Program files (x86) as Progra~2 and Program Files as Progra~1.
You can check to see if the appropriate path has been set with:
PS> Get-ChildItem Env:
Unfortuately this only works in an administrator window.
B.7. Windows installation summary
If you actually managed to install things on Windows without permanent emotional scarring, I offer my sincerest congratulations. However, if you are just skipping ahead, why don’t you make your life easier and make a virtual Linux machine inside your Windows computer?
B.8. Turning your windows machine into a Linux machine
The header of this section is a bit misleading, what you are really going to do is use software to create a virtual version of Linux within your Windows computer.
There are a number of options, popular ones include:
• Parallels This software is proprietary.
• VMWare There are several flavours of this. The free version is VMware Player.
• VirtualBox This is open source.
Here I’ll walk you through setting up Linux using VMware. It just happened to be the one I tried first and it works, please do not take this as an endorsement. One disadvantage is it doesn’t seem to have an Apple version. If you use Apple you’ll need to try to go through a similar process using VirtualBox, which does have a version for Mac operating systems.
But, here is how you set up the VMware player. You will need a reasonable amount of storage (say at least 30GB: you will not be able to get this back!) and RAM (say at least 4 GB, but 8GB is better…note that this is only used when the virtual machine is on) A very old computer probably won’t work. If you’ve got a computer purchased in the last few years things will probably be fine. Note that the virtual machine permanently takes up some portion of your hard disk (you can release this protion back to your windows machine if you uninstall the virtual machine).
1. First, download VMware player. The download is currently here: https://my.vmware.com/web/vmware/free#desktop_end_user_computing/vmware_player/7_0.
2. Run the installation package. Brew a cup of tea while you wait for it to install. Maybe surf the internet a bit.
3. BEFORE you set up a virtual machine, you will need to download a linux operating system!
a. We are going to use Ubuntu, just because it is stable, popular and has good documentation. WARNING: I first attempted an installation with 64-bit Ubuntu, but it turns out my computer doesn’t allow guest 64 bit operating systems. To fix this I just downloaded the 32 bit version of Ubuntu, which worked. However, many of my students have sucessfully installed 65 bit Ubuntu.
b. Find the downloads link and download the latest version. It will be an iso disk image. This will take a while. Put that time to good use.
4. Once that finishes downloading, you can set up your virtual box. First, open VMware Player.
These menus seem to change with new releases, so just try to pick the most WARNING sensible menu options if they don’t match the instructions.
a. Now click on the "Create a New Virtual Machine" option.
5. It will ask you how you want to install the operating system. Tell it you want to use the Ubuntu disk image you just downloaded:
6. You will need to add some username information, and then you will have to pick a location for the Virtual Machine. I made a folder called c:\Ubuntu for it:
7. Now allocate disk space to the virtual machine. This disk space cannot be used by your windows operating system!!. I decided to use a single file to store the disk since it should be faster.
8. The next page will say it is ready to create the virtual machine, but it has a default Memory (in my case 1 GB) allocated. I wanted more memory so I clicked on the customize hardware button: This allowed me to increase the memory to 2GB.
Memory will be used when the virtual machine is on, but when not in use the NOTE memory will revert to your original operating system.
You can change the amount of memory allocated to your virtual machine by NOTE changing the virtual machine settings from the VMware start page.
The size of the DEM you can analyse will be limited by your memory. Give the NOTE virtual machine as much memory as you can spare if you are running analysis on big DEMs. 9. You might be asked to install some VMware Linux tools. You should do this, as some things won’t work if it isn’t installed.
10.Installing the operating system within the virtual machine will take ages. You might schedule this task for your lunch hour, which is what I did. My chicken shawarma wrap was delicious, thank you for asking.
11.When Ubuntu has installed, it will look for software updates. You should install these. This will also take ages. Maybe you have a book to read?
12.Finally, you should be aware that the default keyboard layout is US. Getting UBUNTU to recognize a different keyboard is a bit of a pain.
a. First go to system settings.
b. Then click on language support.
c. It will need to install some stuff.
d. Go to text entry.
e. In the lower left corner click on the + button.
f. Add your country’s input source.
B.9. Summary
By now you should be able to pull up a powershell and call the essential programs we will be working with in this course. Appendix C: Setting up on Linux
Setting up your system on Linux is considerably easier than setting up on Windows. Before doing anything, open a terminal window. The $ symbol below indicates commands typed into the terminal window.
In Ubuntu, the terminal window is opened with ctrl+alt+T. You can also find it in the applications menu under accessories.
These commands are for Ubuntu and Debian flavors of Linux. Other flavors of Linux use different package managers, such as yum. If you don’t use Debian of WARNING Ubuntu, you will need to look up the installation guidelines for the programs below.
C.1. Git
To check if git is working, type
$ git --version
If it isn’t installed, install it with
$ sudo apt-get install git
C.2. C++ tools
You can check if these are working by typing (this assumes you are using the GNU compilers)
$ g++
You can install these with
$ sudo apt-get install g++
These seem to install gdb and make, which is convenient.
C.2.1. C++ libraries
For more specialized versions of the code, you will need some libraries. Installing these can sometimes be tedious, sou you might want to wait until you actually need them before you install. Spectral analysis
Any analyses that use the RasterSpectral objects, which includes the LSDRasterModel, require the fast fourier transform libraries. In the source code, you will find #include statements for these libraries, and corresponding library flags in the makefile: -lfftw3. In the RasterSpectral source files, we assume that you will have a fast fourier transform folder in your toplevel LSDTopoTools directory.
You can download FFTWv3 here: http://www.fftw.org/download.html
Installation should be fairly easy. Go to the FFTW download folder and run
$ ./configure $ make $ make install
The landscape evolution model
Our landscape evolution model (LSDRasterModel) requires FFTW, Boost and [MTL].
BoostBoost contains a large number of header only libraries. You will need to know where you have unzipped them! But the good news is that you don’t need to install anything.
More information is here: http://www.boost.org/doc/libs/1_59_0/more/getting_started/unix- variants.html
C.3. Python
To check if it is working, just type
$ python
If it is working, it will tell you the version and you will get a command prompt that looks like this:
>>>
You should have version 2.7 or above.
C.3.1. Installing python
If you don’t have python, you should install both python and pip, which manages python packages. To do that type: $ sudo apt-get install python2.7 $ sudo apt-get install python-pip
C.3.2. Installing python packages
To check if python packages are there, just try to import them. First start a python session:
$ python
Then, try to import a package. For example:
>>> import matplotlib
If the package does not exist, you will get an error message like this:
Traceback (most recent call last): File "
You can install all the packages at once with:
$ sudo apt-get install python-numpy python-scipy python-matplotlib ipython ipython- notebook python-pandas python-sympy python-nose
You can upgrade packages with the pip command:
$ pip install PackageNameHere --upgrade
C.4. Ruby
As mentioned in the main section of the book, the ruby programming language is used to build the documentation of our software. You can skip this part if you are viewing the documentation from the pdf or webpages and aren’t planning on contributing to the documentation. (Though you if you are contributing new features to the software you are encouraged to!)
You can see if Ruby is on your system by typing:
$ ruby -v If it isn’t there, install with (this should seem routine by now):
$ sudo apt-get install ruby-full
If you already have ruby installed, you might need to check you have the development packages too, along with rubygems, which is a package installer for ruby libraries and add-ons (similar to pip for Python):
$ sudo apt-get install ruby-dev $ sudo apt-get install rubygems
C.4.1. Installing the asciidoctor documentation software
After you have installed ruby, ruby-devel, and rubygems you can now proceed to getting the actual ruby packages that make the documentation:
$ sudo gem install asciidoctor $ sudo gem install bundler
If bundler fails to install, you may be missing the ttfunk package.:
$ sudo gem install ttfunk
Thankfully, the gem installer is quite helpful at telling you which packages are missing and how to quickly install them. That’s it now, you can proceed to cloning the documentation from github.
C.5. Cloning or forking the documentation
The documentation is under version control on github, just like the actual software source code. If you’re a developer, you can clone it direct from the original repository, otherwise you will need to fork it first into your own repo, and then clone from there to your local machine. You do this by visiting http://github.com/LSDtopotools/LSDTT_book and clicking 'fork' (assuming you have a github account of course). Then, on your local machine, you can do:
$ git clone https://github.com/LSDTopoTools/LSDTT_book.git and the documentation will be cloned into a directory called LSDTT_book from where you ran the command. cd into the LSDTT_book directory, and run bundler install: $ bundler install
Then, to build the documentation in full from the source files, run:
$ bundler exec rake book:build or to just build html:
$ bundler exec rake book:build_html
In a few seconds bundler will have completed and you will have a smart looking copy of the documentation in both pdf and html form. Any time you make changes to the documentation source files, run the bundler exec command again to update your local copy. Remember to commit and push your changes regularly to the remote repository on your githb account.
C.6. Summary
Well, you have probably spent some time installing all of these software packages so relax for a bit and enjoy the beverage of your choice! Appendix D: Code Structure
You can happily use LSDTopoTools oblivious to the inner-workings of the code and how it all connects together to produce a suite of topographic analyses at your disposal. Ignorance is, as they say, bliss. No knowledge of Classes, Drivers, Objects, and so on, is needed to run the basic analyses supplied in the distribution of this software. However, you might wish to write your own driver functions, and may even be considering contributing to the core parts of the code. This way you can make your topographic techniques and algorithmic awesomeness available to like-minded geoscientists. Earlier, we mentioned breifly how the code is structured in the main body of this documentation. For those intrepid interrogators of empirical inquisition who have made it four levels down into the Appendix section, first of all, we salute you, and we reward you with a deeper look into how the code is structured, should you wish to contribute your own additions to the repository.
D.1. Source Files: Drivers, Headers, and Implementaions.
There are two main parts of the LSDTopoTools source code: the driver functions and the core component source files. The core files contain the implementations of topographic analyses, and many other functions for creating rasters, sorting data, performin file operations, writing output, and so on. All the core source files are name like LSDSomeFile. pp. They also come in pairs: a LSDSomeClass.cpp source file comes with a LSDSomeClass.hpp header file. The header file describes the interface to a parituclar source file. In other words, LSDSomeClass.hpp tells us what all the methods and data structures of a particular source file (LSD.cpp) are. For functions (or class methods to be precise…) the header file tells us what type of parameters the functions take as arguments, and what data types (if any) these functions return. They also describe how data structures are stored. For example, we could look in LSDRaster.hpp to see how the data members of a Raster are defined. We would see that an LSDRaster object has some meta-data telling us about the extent and coordinates of the the raster data, and an array that stores each pixel value in the raster. It would also tell us which functions (methods) would return information about the LSDRaster object. If we wanted to know the number of rows in a raster object we would see in the header file there is a getNRows() function, and so on. The .cpp files tell us how these functions are implemented, e.g. what exactly each function does when parameters are passed to it, and how it maniplulates the data stored with the object.
In general, although it is not required, we have kept one Class to one pair of header and implementation files. (Although there are a couple of exceptions to this). So in the LSDRaster.hpp and LSDRaster.cpp core files, you will find the declaration and implementation of the LSDRaster class, respectively. In short, hpp tells us what is there and how to use it, cpp tells us how it works. For a full description of all the LSDTopoTools objects and classes you can visit the automatically generated doxygen documentation.
The driver files are separated off into their own folder(s) and as their name suggests, they are responsible for driving a particular analysis by calling on different objects and functions defined in the core source files. The core LSDFooBar-type files we talked about previously don’t actually do much on their own — they are just a bunch of class data structures and methods. This is where the driver files come in: driver files are written to perform a certain kind of topographic analysis or model simulation using some form of input data created by you, the user. Driver files call upon different parts of the object files to perform topographic analyses. An example driver file may call upon LSDRaster to create a Raster object to store data from an input DEM, then it might pass this LSDRaster object to LSDFlowInfo to calculate the flow routing paths in the raster, and so on, until we have completed a useful analysis and written some output data to analyse.
Since there is such a variety of functions defined in the source code, there are potentially hundreds if not thousands of possible analyses that could be performed with the code. The driver files provided with the TopoTools distribution are designed to accomplish some of the more common topographic analysis tasks, e.g. extracting basins from a raster, calculating the location of channel heads in a landscape, running a chi-analysis on river channels, or extracting swath profiles, and so on. However, you are free to write your own driver files of course! What’s more, they provide an exact record of the analysis you have just done, and so are inherrently reproducible if you want to share your findings with others.
Another way to think of drivers, if you are familiar with C or C++ programming, is that TIP they contain the int main() { } bit of a typical program. I.e. the main workflow of a program.
D.2. A closer look at the source files
D.2.1. LSDRaster
LSDRaster is the class that contains methods for mainipulating arrays of raster data loaded from a DEM file. There are also data members that store metadata about the raster, such as the typical header data you would find in a DEM file. It can perform typical analyses such as creating a hillshade raster, calculating curvature of ridgetops, calculating the location of channel heads in a landscape, filling sinks and pits in a DEM and so on — mainly analyses that are performed on the entire raster dataset. It also does the basic reading and writing to DEM files.
D.2.2. LSDIndexRaster
A simple and fairly short class that stores rasters data as integers, used for indexing of raster data, calculating bounding boxes for rasters with lots of NoData values around the edge (or surounding basins). Very similar to LSDRaster class except is only used for when an index or mask of a given raster is needed.
D.2.3. LSDFlowInfo
This class performs operations such as calculating flow direction, calculating upstream contributing pixels in a raster, calculating sources based on threshold pixel methods, and other flow related information (based on topographic analysis — not based on any particular hydrological fluid flow calculations). Note how this object has include statements for LSDRaster and LSDIndexRaster — it returns these type of objects from many of its methods. It uses the FastScape algorithm of Braun and Willet (2014).
D.2.4. LSDIndexChannel
This object contains the node indexes as well as the row and col indices for individual channel segments.
D.2.5. LSDChannel
LSDChannel is our first class that inherits the public methods and data members of another, namely LSDIndexChannel. This means we have direct access to the public members of that class. Note, in the source code:
class LSDChannel: public LSDIndexChannel { public: // Some method declarations
Indicates this class inherits from LSDIndexChannel. LSDChannel stores information about the actual channel characteristics, e.g. elevation etc.
D.2.6. LSDJunctionNetwork
JunctionNetwork contains the main parts of the implemented FastScape algorithm for creating channel junction networks that can be searched for network connectivity. The main inputs are a FlowInfo object and list of source nodes. It also contains functions for returning IndexChannel objects after calculating the layout of channel networks.
D.2.7. LSDIndexChannelTree
This object spawns vectors of LSDIndexChannels. They can be indexed by the LSDChannel network, but can also be independent of the channel network, storing longest channels from sources, for example. The class is designed to be flexible, it can be used either with the LSDFlowInfo or LSDJunctionNetwork classes.
D.2.8. LSDChiNetwork
This is used to perform chi-related analyses on channels.
D.2.9. LSDMostLikelyPartitionsFinder
This object is principally used to identify segments of differing channel steepness in chi-zeta space. It contains the implementation of the segment fitting algorithm, including the statistical model that determines the most likely partitionining combination of channels. D.2.10. LSDStatsTools
The StatsTools files contain a number of classes and standalone functions that perform statistical analyses. A number of other utility functions are implemented here, such as file name parsing and formatting.