DemoDoc Documentation Release
Sourav
Apr 24, 2017
Contents
1 SunPy User Guide 3 1.1 Installation...... 3 1.2 A brief tour of SunPy...... 10 1.3 Acquiring Data with SunPy...... 16 1.4 Data Types in SunPy...... 42 1.5 Plotting in SunPy...... 52 1.6 Time in SunPy...... 54 1.7 Region of Interest...... 57 1.8 Customizing SunPy...... 57 1.9 Troubleshooting...... 58
2 API Reference 61 2.1 SunPy...... 61 2.2 SunPy cm...... 61 2.3 SunPy coordinates...... 61 2.4 SunPy database...... 64 2.5 SunPy image...... 64 2.6 SunPy instr...... 64 2.7 SunPy io...... 65 2.8 SunPy lightcurve...... 65 2.9 SunPy map...... 65 2.10 SunPy net...... 67 2.11 SunPy physics...... 67 2.12 SunPy roi...... 67 2.13 SunPy spectra...... 67 2.14 SunPy sun...... 68 2.15 SunPy time...... 68 2.16 SunPy util...... 68 2.17 SunPy visualization...... 68 2.18 SunPy wcs...... 68
3 Developer’s Guide 69 3.1 Developer’s Guide Overview...... 69 3.2 Version Control...... 69 3.3 Coding Standards...... 74 3.4 Global Settings...... 75 3.5 Documentation...... 75
i 3.6 Testing...... 80
4 Reporting Bugs 85
5 SSWIDL/SunPy Cheat Sheet 87
Python Module Index 89
ii DemoDoc Documentation, Release
Welcome to the SunPy documentation. SunPy is a community-developed, free and open-source solar data analysis environment for Python. You can find the official SunPy website at sunpy.org.
Contents 1 DemoDoc Documentation, Release
2 Contents CHAPTER 1
SunPy User Guide
Welcome to the SunPy User Guide. SunPy is a community-developed, free and open-source solar data analysis environment. It is meant to provide the core functionality and tools to analyze solar data with Python. This guide provides a walkthrough of the major features in SunPy. For more details checkout the API Reference.
Installation
SunPy is a Python package for solar physics. It relies on and enables the use of the wider ecosystem of scientific Python packages for solar physics. Therefore a working SunPy installation is more about installing the scientific Python ecosystem than SunPy itself. SunPy is Python 2.7.x, 3.4.x and 3.5.x compatible. If you are new to Python and scientific Python then continue to follow this guide to get setup with the whole envi- ronment. If you already have a working Python / Scientific Python environment then you can skip to the Advanced Installation Instructions section.
Installing Scientific Python and SunPy
If you do not currently have a working scientific Python distribution this guide will set you up with the Anaconda sci- entific Python distribution. Anaconda makes it easy to install and manage your scientific Python packages. Alternative scientific Python options exist and can be found later in the Advanced Installation Instructions section. Anaconda contains a free distribution of Python and a large number of common scientific packages. Anaconda is very powerful and easy to use. Installing Anaconda provides (almost) all the packages you need to use SunPy. To install the Anaconda Python distribution follow the instructions here. You will need to select the correct download for your platform and follow the install procedure. Note that although Anaconda makes it simple to switch between Python versions, we recommend that new users install the latest Python 3.x version of Anaconda. This is because Python 2.7 is scheduled to be deprecated in 2020.
3 DemoDoc Documentation, Release
Installing SunPy on top of Anaconda
To install SunPy launch a system command prompt or the ‘Anaconda Command Prompt’ (under Windows). First configure conda for sunpy downloads: conda config--add channels conda-forge to install SunPy: conda install sunpy
You now have a working SunPy installation. You can check your SunPy install by following the instructions in Testing SunPy.
Note: Currently Glymur / JPEG2000 support is not tested under Anaconda on any platforms. However, an external installation of openJPEG should work with the glymur installation in conda.
Updating SunPy to a New Version
When a new version of SunPy is released you can update to the latest version by running: conda update sunpy
Advanced SunPy Installation
If you do not wish to use Anaconda to install Scientific Python or you already have a scientific Python installation there are other options for installing SunPy.
Advanced Installation Instructions
This document provides details on things you need to know to install and manage your own scientific Python + SunPy installation. If you have never installed or used scientific Python we recommend that you follow the Anaconda installation instructions.
Alternative Scientific Python Installation Instructions
There are many alternatives to Anaconda as a way of installing a scientific Python environment, there are various other platform specific ways to install scientific Python:
Linux
Overview
Warning: We highly recommend that users use the Anaconda python distribution. Instructions are available here anaconda_install.
4 Chapter 1. SunPy User Guide DemoDoc Documentation, Release
Almost all versions of Linux ship with a recent enough version of Python, so it is unlikely that you will need to install Python yourself. If you do not have python you can find the source at the Python official site. Install using sudo apt-get install python2.7-dev or a similar command, depending on your linux distro. Anything like python-dev provides you the required Python development headers.
Debian based (eg. Ubuntu)
On Ubuntu, most of the pre-reqs are available in the Ubuntu software repos and can be installed using apt-get: sudo apt-get install python-dev sudo apt-get install python-qt4 sudo apt-get install git-core sudo apt-get install python-numpy sudo apt-get install python-scipy sudo apt-get install python-matplotlib sudo apt-get update
Now we shall install pip.
Pip
Most Python distributions ship with a tool called easy_install which assists with installing Python packages. Although easy_install is capable of installing most of the dependencies needed for SunPy itself, a more powerful tool called pip provides a more flexible installation (including support for uninstalling, upgrading, and installing from remote sources such as GitHub) and should be used instead. Use easy_install to install pip: sudo easy_install pip
You are now ready to install SunPy and its dependencies.
Mac OS X
Overview
Warning: We highly recommend that users use the Anaconda python distribution. Instructions are available here anaconda_install.
Because of the wide variety of methods for setting up a Python environment on Mac OS X, users have a number of options with how to install SunPy and its dependencies. We have a number of solutions listed below. Choose the solution which best suits you. For users who wish to have more control over the installation of Python, several alternative installation methods are provided below, including instructions for Macports and Homebrew. The following instructions are not recommended for beginners. OS X comes pre-loaded with Python but each versions of OS X (Mountain Lion, Snow Leopard, etc.) ships with a different version of Python. In the instructions below, we therefore recommend that you install your own version of Python. You will then have two versions of Python living on your system at the same time. It can be confusing to make sure that when you install packages they are installed for the correct Python so beware.
1.1. Installation 5 DemoDoc Documentation, Release
Python 2.7.3+
Download and install the latest version of 32 bit Python 2.7.3
XCode tools / Compiler
If you are using MacOS X, you will need to the XCode command line tools. One way to get them is to install XCode. If you are using OS X 10.7 (Lion) or later, you must also explicitly install the command line tools. You can do this by opening the XCode application, going to Preferences, then Downloads, and then under Components, click on the Install button to the right of Command Line Tools. Alternatively, on 10.7 (Lion) or later, you do not need to install XCode, you can download just the command line tools from https://developer.apple.com/downloads/index. action (requires an Apple developer account).
Homebrew
Homebrew is a tool for helping to automate the installation of a number of useful tools and libraries on Mac OS X. It is similar to Macports, but attempts to improve on some of the pitfalls of Macports. Note that if you have already installed either fink or Macports on your system, it is recommended that you uninstall them before using Homebrew. Next, install and update homebrew:
/usr/bin/ruby-e"$(/usr/bin/curl -fsSL https://raw.github.com/mxcl/homebrew/master/
˓→Library/Contributions/install_homebrew.rb)" brew doctor
Using homebrew, install Qt and some of the other dependencies needed for compilation later on by pip:
brew-v install gfortran pkgconfig git openjpeg readline pyqt
Now on to the next steps.
Git
Head over and download
Pip
Most Python distributions ship with a tool called easy_install which assists with installing Python packages. Although easy_install is capable of installing most of the dependencies needed for SunPy itself, a more powerful tool called pip which provides a more flexible installation (including support for uninstalling, upgrading, and installing from remote sources such as GitHub) and should be used instead. Use easy_install to install pip:
sudo easy_install pip
You are now ready to install scipy, numpy, and matplotlib.
6 Chapter 1. SunPy User Guide DemoDoc Documentation, Release
Scientific Libraries
If pip installed properly, then you can install NumPy simply with:
pip install numpy
Now under Lion, install the stable version of SciPy (0.10) by running:
pip install scipy
Mountain Lion users will need to install the development version of SciPy (0.11) by executing the following line: pip install -e git+https://github.com/scipy/scipy#egg=scipy-dev Now on to matplotlib On Lion, install matplotlib like any other package:
pip install matplotlib
Mountain Lion users will have to use the development version as of this writing:
pip install git+https://github.com/matplotlib/matplotlib.git#egg=matplotlib-dev
Done! You are now ready to install SunPy itself .
Installing SunPy on top of an existing Scientific Python Environment
These instructions assume you have a scientific Python distribution with access to the pip command installed.
Prerequisites
You will need a compiler suite and the development headers for Python and Numpy in order to build SunPy. On Linux, using the package manager for your distribution will usually be the easiest route, while on MacOS X you will need the XCode command line tools. The instructions for building Numpy from source are also a good resource for setting up your environment to build Python packages.
Note: If you are using MacOS X, you will need to the XCode command line tools. One way to get them is to install XCode. If you are using OS X 10.7 (Lion) or later, you must also explicitly install the command line tools. You can do this by opening the XCode application, going to Preferences, then Downloads, and then under Components, click on the Install button to the right of Command Line Tools. Alternatively, on 10.7 (Lion) or later, you do not need to install XCode, you can download just the command line tools from the Apple developer site (requires an Apple developer account).
SunPy’s Requirements
SunPy consists of many submodules that each have their own requirements. You do not need to fulfil all the require- ments if you only intend on using parts of SunPy. It is however strongly recommended to have all the dependencies installed (with the potential exception of glymur). SunPy has the following strict requirements:
1.1. Installation 7 DemoDoc Documentation, Release
• Python 2.7 • NumPy 1.6.0 or later • SciPy 0.10.0 or later • AstroPy 1.0.0 or later SunPy also depends on other packages for optional features. However, note that these only need to be installed if those particular features are needed. SunPy will import even if these dependencies are not installed. • Matplotlib[ Highly Recommended] 1.3.0 or later: For ~sunpy.lightcurve, ~sunpy.map, ~sunpy.spectra, ~sunpy.instr and ~sunpy.visualization. • pandas 0.10 or later: For ~sunpy.lightcurve. • sqlalchemy: For the ~sunpy.database package. • suds-jurko: For ~sunpy.net. • beautifulsoup4: For ~sunpy.spectra.Callisto Spectrograms and ~sunpy.net.helio • requests: For the ~sunpy.net.jsoc submodule. • wcsaxes: For sunpy.map plotting improvements. • glymur 0.5.9 or later: To enable reading of JPEG2000 files. Glymur requires the installation of the OpenJPEG C library. • pytest: To run tests. The packages that will be installed as dependencies by default and are the ones required to import the core datatypes ~sunpy.map, ~sunpy.lightcurve and ~sunpy.spectra. These are the strict requirements and the following optional pack- ages:
Using pip
There are multiple options depending on how many optional dependencies you want to install: To install SunPy with pip including optional dependencies (recommended), simply run: pip install sunpy[all]
To install SunPy with no optional dependencies: pip install sunpy
To install SunPy with net-based dependencies (suds and beautifulsoup): pip install sunpy[net]
To install SunPy with database dependencies (sqlalchemy): pip install sunpy[database]
Warning: Users of the Anaconda python distribution should follow the instructions for anaconda_install.
Note: You will need a C compiler (e.g. gcc or clang) to be installed.
8 Chapter 1. SunPy User Guide DemoDoc Documentation, Release
Note: If you get a PermissionError this means that you do not have the required administrative access to install new packages to your Python installation. In this case you may consider using the --user option to install the package into your home directory. You can read more about how to do this in the pip documentation. Alternatively, if you intend to do development on other software that uses SunPy, such as an affiliated package, consider installing SunPy into a virtualenv. Do not install SunPy or other third-party packages using sudo unless you are fully aware of the risks.
Testing SunPy
The easiest way to test your installed version of SunPy is running correctly is to use the sunpy.self_test() function: import sunpy sunpy.self_test(online=False) which will run many of the SunPy tests. The tests should run and print out any failures, which you can report at the SunPy issue tracker.
Installing the Development Version of SunPy
The latest (bleeding-edge) development version of SunPy can be cloned from github using this command: git clone git://github.com/sunpy/sunpy.git
Note: If you wish to participate in the development of SunPy, see developer-docs. This document covers only the basics necessary to install SunPy.
Once inside the source directory that has been clone from GitHub you can install SunPy using: python setup.py install
Note: This command will need access to system folders append –user to install SunPy into your home directory.
Troubleshooting
If you get an error mentioning that you do not have the correct permissions to install SunPy into the default site-packages directory, you should try installing with: pip install sunpy--user which will install into a default directory in your home directory.
1.1. Installation 9 DemoDoc Documentation, Release
Building documentation
Note: Building the documentation is in general not necessary unless you are writing new documentation or do not have internet access, because the latest (and archive) versions of SunPy’s documentation are available at docs.sunpy.org .
Building the documentation requires the SunPy source code and some additional packages: • Sphinx (and its dependencies) 1.0 or later • Graphviz
Note: Sphinx also requires a reasonably modern LaTeX installation to render equations. Per the Sphinx documenta- tion, for the TexLive distribution the following packages are required to be installed: • latex-recommended • latex-extra • fonts-recommended For other LaTeX distributions your mileage may vary. To build the PDF documentation using LaTeX, the fonts-extra TexLive package or the inconsolata CTAN package are also required.
There are two ways to build the SunPy documentation. The most straightforward way is to execute the command (from the sunpy source directory):
python setup.py build_sphinx-lo
The documentation will be built in the doc/build/html directory, and can be read by pointing a web browser to doc/build/html/index.html. The LaTeX documentation can be generated by using the command:
python setup.py build_sphinx-b latex
The LaTeX file SunPy.tex will be created in the doc/build/latex directory, and can be compiled using pdflatex. The above method builds the API documentation from the source code.
A brief tour of SunPy
This brief tutorial will walk you through some of the functionality offered by SunPy. Start by reading this tutorial and trying out some of the examples demonstrated. Once you’ve completed the tutorial check out the rest of the User Guide for a more thorough look at the functionality available.
Sample Data
This tour makes use of a number of sample data files which you will need to download. If have not already done so please follow the instruction here Downloading Sample Data.
10 Chapter 1. SunPy User Guide DemoDoc Documentation, Release
Maps
Maps are the primary data type in SunPy they are spatially and / or temporally aware data arrays. There are maps for a 2D image, a time series of 2D images or temporally aligned 2D images. Creating a Map SunPy supports many different data products from various sources ‘out of the box’. We shall use SDO’s AIA instru- ment as an example in this tutorial. The general way to create a Map from one of the supported data products is with the ~sunpy.map.map() function from the ~sunpy.map submodule. ~sunpy.map.map() takes either a filename, a list of filenames or a data array and header. We can test map with: This returns a map named aia which can be manipulated with standard SunPy map commands. For more information about maps checkout the map guide and the SunPy map.
Lightcurve
SunPy handles time series data, fundamental to the study of any real world phenomenon, by creating a lightcurve object. A lightcurve consists of two parts; times and measurements taken at those times. The data can either be in your current Python session, alternatively within a local or remote file. Let’s create some fake data and pass it into a lightcurve object. Within LightCurve.create, we have a dictionary that contains a single entry with key “signal” containing a list of 1000 entries (0-999). The accompanying set of times is passed in via the index keyword argument. If no times are passed into index, a default set of time indices is generated. For more information about lightcurves, check out the lightcurve guide and the and the SunPy lightcurve.
Spectra
SunPy has spectral support for instruments which have such a capacity. CALLISTO, an international network of Solar Radio Spectrometers, is a specific example. For more information about spectra, check out the spectra guide and the SunPy spectra.
Plotting
SunPy uses a matplotlib like interface to it’s plotting so more complex plots can be built by combining SunPy with matplotlib. Let’s begin by creating a simple plot of an AIA image. To make things easy, SunPy includes several example files which are used throughout the docs. These files have names like sunpy.data.sample.AIA_171_IMAGE and sunpy.data.sample.RHESSI_IMAGE. Try typing the below example into your interactive Python shell. If everything has been configured properly you should see an AIA image with a red colormap, a colorbar on the right-hand side and a title and some labels. There is lot going on here, but we will walk you through the example. Briefly, the first line is just importing SunPy. On the second line we create a SunPy Map object which is basically just a spatially-aware image or data array. On the last line we then plot the map object, using the built in ‘quick plot’ function peek(). SunPy uses a matplotlib like interface to it’s plotting so more complex plots can be built by combining SunPy with matplotlib. For more information check out Plotting in SunPy.
1.2. A brief tour of SunPy 11 DemoDoc Documentation, Release
Solar Physical Constants
SunPy contains a convenient list of solar-related physical constants. Here is a short bit of code to get you started:
>>> from sunpy.sun import constants as con
# one astronomical unit (the average distance between the Sun and Earth) >>> print con.au Name = Astronomical Unit Value = 1.495978707e+11 Error = 0.0 Units = m Reference = IAU 2012 Resolution B2
# the solar radius >>> print con.radius Name = Solar radius Value = 695508000.0 Error = 26000.0 Units = m Reference = Allen's Astrophysical Quantities 4th Ed.
Not all constants have a shortcut assigned to them (as above). The rest of the constants are stored in a dictionary. The following code grabs the dictionary and gets all of the keys.:
>>> solar_constants= con.constants >>> solar_constants.keys() ['solar flux unit', 'surface area', 'average density', 'radius', 'surface gravity', 'ellipticity', 'visual magnitude', 'center density', 'average angular size', 'absolute magnitude', 'sunspot cycle', 'effective temperature', 'aphelion distance', 'mean energy production', 'mass conversion rate', 'average intensity', 'volume', 'metallicity', 'moment of inertia', 'escape velocity', 'perihelion distance', 'GM', 'oblateness', 'mean distance', 'age', 'mass', 'luminosity', 'center temperature']
You can also use the function sunpy.constants.print_all() to print out a table of all of the values available. These constants are provided as a convenience so that everyone is using the same (accepted) values. For more information check out SunPy sun.
Quantities and Units
Many capabilities in SunPy make use of physical quantities that are specified with units. SunPy uses ~astropy.units to implement this functionality. For example, the solar radius above is a physical quantity that can be expressed in length units. In the example above from sunpy.sun import constants as con con.radius ˓→"Allen's Astrophysical Quantities 4th Ed."> shows the solar radius in units of meters. It is simple to express the same physical quantity in different units: con.radius.to('km') To get the numerical value of the solar radius in kilometers - without the unit information - use 12 Chapter 1. SunPy User Guide DemoDoc Documentation, Release con.radius.to('km').value 695508.0 Quantities and units are simple and powerful tools for keeping track of the units you’re working in, and make it easy to convert the same physical quantity into different units. To learn more about the capabilities of quantities and units, please consult the astropy tutorial. SunPy’s approach to the adoption of quantities and units in the codebase is described here. Here’s a simple example of the power of units. Suppose you have the radius of a circle and would like to calculate its area. The following code implements this >>> import numpy as np >>> import astropy.units asu >>> @u.quantity_input(radius=u.m) ... def circle_area(radius): ... return np.pi * radius ** 2 The first line imports numpy, and the second line imports astropy’s units module. The beginning of the third line (the “@” symbol) indicates that what follows is a Python decorator. In this case, the decorator allows us to specify what kind of unit the function input variable “radius” in the following function “circle_area” should have. In this case, it is meters. The decorator checks that the input is convertible to the units specified in the decorator. Calculating the area of a circle with radius 4 meters using the function defined above is simple circle_area(4 * u.m) The units of the returned area are what we expect, namely the meters squared (m2). However, we can also use other units of measurement; for a circle with radius 4 kilometers circle_area(4 * u.km) Even although the input value of the radius was not in meters, the function does not crash; this is because the input unit is convertible to meters. This also works across different systems of measurement, for example circle_area(4 * u.imperial.foot) However, if the input unit is not convertible to meters, then an error is thrown >>> circle_area(4 * u.second) ... UnitsError: Argument 'radius' to function 'circle_area' must be in units convertable ˓→to 'm'. Also, if no unit is specified, an error is thrown >>> circle_area(4) ... TypeError: Argument 'radius' to function has 'circle_area' no 'unit' attribute. You ˓→may want to pass in an astropy Quantity instead. Using units allows the user to be explicit about what the function expects. Units also make conversions very easy to do. For example, if you want the area of a circle in square feet, but were given measurements in meters, then circle_area((4 * u.m).to(u.imperial.foot)) 1.2. A brief tour of SunPy 13 DemoDoc Documentation, Release or >>> circle_area(4 * u.m).to(u.imperial.foot ** 2) Astropy units and quantities are very powerful, and are used throughout SunPy. To find out more about units and quantities, please consult the the astropy tutorial and documentation Working with Times SunPy also contains a number of convenience functions for working with dates and times. Here is a short example: >>> import sunpy.time # parsing a standard time strings >>> sunpy.time.parse_time('2004/02/05 12:00') datetime.datetime(2004, 2, 5, 12, 0) # This returns a datetime object. All SunPy functions which require # time as an input sanitize the input using parse_time. >>> sunpy.time.day_of_year('2004-Jul-05 12:00:02') 187.50002314814816 # the julian day >>> sunpy.time.julian_day((2010,4,30)) 2455316.5 # TimeRange objects are useful for representing ranges of time >>> time_range= sunpy.time.TimeRange('2010/03/04 00:10','2010/03/04 00:20') >>> time_range.center datetime.datetime(2010, 3, 4, 0, 15) For more information about working with time in SunPy checkout the time guide. Getting at Data Querying the VSO There are a couple different ways to query and download data from the VSO using SunPy. The method you should use depends first on your preference with respect to query style: the main method of querying uses a syntax that is unique to SunPy and may require some getting used to, but is extremely flexible and powerful. A second “legacy” API also exists which works is very much the same way as VSO_GET in IDL. Further, for each of the two query APIs there are interactive and non-interactive versions available, depending on the type of work you are doing. The below example demonstrates a simple query for SOHO EIT data using the non-interactive version of the main API: >>> from sunpy.net import vso # create a new VSOClient instance >>> client= vso.VSOClient() # build our query >>> result= client.query( 14 Chapter 1. SunPy User Guide DemoDoc Documentation, Release ... vso.attrs.Time((2011,9, 20,1), (2011,9, 20,2)), ... vso.attrs.Instrument('eit')) # print the number of matches >>> print("Number of records found: {}".format(len(result))) Number of records found: 4 # download matches to /download/path >>> res= client.get(result, path="/download/path/ {file}").wait() Note that specifying a path is optional and if you do not specify one the files will simply be downloaded into a temporary directory (e.g. /tmp/xyz). For more information about vso client checkout the vso guide. Database Package The database package offers the possibility to save retrieved data (e.g. via the :mod:’sunpy.net.vso’ package) onto a local or remote database. The database may be a single file located on a local hard drive (if a SQLite database is used) or a local or remote database server. This makes it possible to fetch required data from the local database instead of downloading it again from a remote server. Querying a database is straightforward, as this example using VSO, shows. The example demonstrates the useful feature which prevents storing the same data twice: >>> from sunpy.database import Database >>> from sunpy.net.vso.attrs import Time, Instrument >>> db= Database('sqlite:///') >>> entries= db.fetch( ... Time('2012-08-05','2012-08-05 00:00:05'), ... Instrument('AIA')) >>> assert entries is None >>> len(db) 4 >>> entries= db.fetch( ... Time('2012-08-05','2012-08-05 00:00:05'), ... Instrument('AIA')) >>> entries is None False >>> len(entries) 4 >>> len(db) 4 Explanation: first, entries is None because the query has never been used for querying the database -> query the VSO, add new entries to database, remember query hash. In the second fetch, entries is not None because the query has already been used and returns a list of database entries. For more information check out the Using the database package. Querying Helioviewer.org SunPy can be used to make several basic requests using the The Helioviewer.org API including generating a PNG and downloading a JPEG 2000 image and loading it into a SunPy Map. A simple example of a helioviewer query and generating a plot of the result follows: 1.2. A brief tour of SunPy 15 DemoDoc Documentation, Release >>> from sunpy.net.helioviewer import HelioviewerClient >>> import matplotlib.pyplot as plt >>> from matplotlib.image import imread >>> hv= HelioviewerClient() >>> file= hv.download_png('2099/01/01', 4.8,"[SDO,AIA,AIA,304,1,100]", x0=0, y0=0, ˓→width=512, height=512) >>> im= imread(file) >>> plt.imshow(im) >>> plt.axis('off') >>> plt.show() This downloads a PNG image of the latest AIA 304 image available on Helioviewer.org. In the ~sunpy.net.helioviewer.HelioviewerClient.download_png command the value, 4.8, refers to the image resolution in arcseconds per pixel (larger values mean lower resolution), x0 and y0 are the center points about which to focus and the width and height are the pixel values for the image dimensions. For more information checkout the helioviewer guide. Acquiring Data with SunPy Downloading Sample Data SunPy provides a number of sample data files which are referenced in the documentation and examples. These files are available to download onto your local machine so that you can try out the code in the documentation. To download the sample data simply run the following command: import sunpy.data sunpy.data.download_sample_data() With a bad Internet connection it helps to set the overwrite=False parameter in dowload_sample_data. This will download the data to your sample-data directory which can be customized by editing the sunpyrc file (see Customizing SunPy). After running this you can then import the sample data files shortcuts which are used below by simply importing the module like so: import sunpy.data.sample If the sample files are not available for some reason that you will get an error on import. Downloading Data from the VSO The main interface which SunPy provides to search for and download data is provided by SunPy’s VSO module. This module provides an interface to the Virtual Solar Observatory (VSO) which is a service which presents a homogeneous interface to heterogeneous data-sets and services. Using the VSO, a user can query multiple data providers simulta- neously, and then download the relevant data. SunPy uses the VSO through the vso module, which was developed through support from the European Space Agency Summer of Code in Space (ESA-SOCIS) 2011. Setup SunPy’s VSO module is in sunpy.net. It can be imported as follows: >>> from sunpy.net import vso >>> client=vso.VSOClient() 16 Chapter 1. SunPy User Guide DemoDoc Documentation, Release This creates your client object. Obtaining data via the VSO is a two-stage process. You first ask the VSO to find the data you want. The VSO queries various data-providers looking for your data. If there is any data that matches your request, you choose the data you want to download. The VSO client handles the particulars of how the data from the data provider is downloaded to your computer. Searching the VSO To search the VSO, your query needs at minimum a start time, an end time, and an instrument. Two styles of con- structing the query are supported by SunPy’s VSO module. The first style is very flexible, as it allows users to issue complex queries in a single command. This query style is described below. The second query style - known as the legacy query is useful for making quick VSO queries, and is based on the function call to SSWIDL’s VSO query client. The section below first describe the more flexible query style. The next section then describes the legacy query. The final section describes how to download data from those query results. Constructing a Query Let’s start with a very simple query. We could ask for all SOHO/EIT data between January 1st and 2nd, 2001. >>> qr= client.query(vso.attrs.Time('2001/1/1','2001/1/2'), vso.attrs.Instrument( ˓→'eit')) The variable qr is a Python list of response objects, each one of which is a record found by the VSO. You can find how many records were found by typing >>> len(qr) 122 To get a little bit more information about the records found, try >>> print(qr) ... Now, let’s break down the arguments of client.query to understand better what we’ve done. The first argument: vso.attrs.Time('2001/1/1', '2001/1/2') sets the start and end times for the query (any date/time format understood by SunPy’s parse_time function can be used to specify dates and time). The second argument: vso.attrs.Instrument('eit') sets the instrument we are looking for. The third argument: vso.attrs.Wave(142*u.AA, 123*u.AA) sets the values for wavelength i.e, for wavemax(maximum value) and similarly wavemin(for minimum value) for the query. Also the u.AA part comes from astropy.units.Quantity where AA is Angstrom. It should be noted that specifying spectral units in arguments is necessary or an error will be raised. To know more check astropy.units. So what is going on here? The notion is that a VSO query has a set of attribute objects - described in vso.attrs - that are specified to construct the query. For the full list of vso attributes, use >>> help(vso.attrs) 1.3. Acquiring Data with SunPy 17 DemoDoc Documentation, Release Note that due to a current bug in the VSO, we do not recommend that the extent object vso.attrs.Extent be in your query. Instead, we recommend that any extent filtering you need to do be done on the queries made without setting a value to the vso.attrs.Extent object. As we will see, this query style can take more than two arguments, each argument separated from the other by a comma. Each of those arguments are chained together using a logical AND. This query style allows you to combine these VSO attribute objects in complex ways that are not possible with the legacy query style. So, let’s look for the EIT and MDI data on the same day: >>> qr=client.query(vso.attrs.Time('2001/1/1','2001/1/2'), vso.attrs.Instrument('eit ˓→')| vso.attrs.Instrument('mdi')) >>> len(qr) 3549 >>> print(qr) ... The two instrument types are joined together by the operator “|”. This is the OR operator. Think of the above query as setting a set of conditions which get passed to the VSO. Let’s say you want all the EIT data from two separate days: >>> qr=client.query(vso.attrs.Time('2001/1/1','2001/1/2')| vso.attrs.Time('2007/8/9 ˓→','2007/8/10'), vso.attrs.Instrument('eit')) >>> len(qr) 227 Each of the arguments in this query style can be thought of as setting conditions that the returned records must satisfy. You can set the wavelength; for example, to return the 171 Angstrom EIT results >>> import astropy.units asu >>> qr=client.query(vso.attrs.Time('2001/1/1','2001/1/2'), vso.attrs.Instrument('eit ˓→'), vso.attrs.Wave(171*u.AA,171*u.AA) ) >>> len(qr) 4 Using the Legacy Query Style If you just need to do a quick query or don’t want to do anything too complicated you can use the legacy query style. Here is the first example from the above section executed using a legacy query. As before, we want EIT data between 2001/01/01 and 2001/01/02 >>> qr=client.query_legacy(tstart='2001/01/01', tend='2001/01/02', instrument='EIT') which is almost identical to what you would type in SSWIDL. So, what’s happening with this command? The client is going out to the web to query the VSO to ask how many files EIT images are in the archive between the start of 2001/01/01 and the start of 2001/01/02. The same query can also be performed using a slightly different syntax. For example >>> qr=client.query_legacy('2001/1/1','2001/1/2', instrument='EIT') both gives the same result. The variable qr is a Python list of response objects, each one of which is a record found by the VSO. How many records have been found? You can find that out be typing >>> len(qr) 122 To get a little bit more information, try 18 Chapter 1. SunPy User Guide DemoDoc Documentation, Release >>> print(qr) ... The Solarsoft legacy query has more keywords available: to find out more about the legacy query, type: >>> help(client.query_legacy) As an example, let’s say you just want the EIT 171 Angstrom files for that data. These files can be found by >>> qr=client.query_legacy(tstart='2001/01/01', tend='2001/01/02', instrument='EIT', ˓→min_wave='171', max_wave='171', unit_wave='Angstrom') which yields four results, the same as the VSO IDL client. Downloading data All queries return a query response list. This list can then used to get the data. This list can also be edited as you see fit. For example you can further reduce the number of results and only get those. So having located the data you want, you can download it using the following command: >>> res=client.get(qr, path='/ThisIs/MyPath/to/Data/{file}.fits') This downloads the query results into the directory /ThisIs/MyPath/to/Data naming each downloaded file with the filename {file} obtained from the VSO , and appended with the suffix .fits. The {file} option uses the file name obtained by the VSO for each file. You can also use other properties of the query return to define the path where the data is saved. For example, to save the data to a subdirectory named after the instrument, use >>> res=client.get(qr, path='/ThisIs/MyPath/to/Data/{instrument}/{file}.fits') If you have set your default download directory in your sunpyrc configuration file then you do not need to identify a path at all. All you data will be downloaded there. Note that the download process is spawned in parallel to your existing Python session. This means that the remainder of your Python script will continue as the download proceeds. This may cause a problem if the remainder of your script relies on the presence of the downloaded data. If you want to resume your script after all the data has been downloaded then append .wait() to the get command above, i.e., >>> res=client.get(qr, path='/Users/ireland/Desktop/Data/{instrument}/{file}.fits'). ˓→wait() More information on the options available can be found through the standard Python help command. Using SunPy’s HEK module The Heliophysics Event Knowledgebase (HEK) is a repository of feature and event information about the Sun. Entries are generated both by automated algorithms and human observers. SunPy accesses this information through the hek module, which was developed through support from the European Space Agency Summer of Code in Space (ESA- SOCIS) 2011. 1. Setting up the client SunPy’s HEK module is in sunpy.net. It can be imported into your session as follows: 1.3. Acquiring Data with SunPy 19 DemoDoc Documentation, Release >>> from sunpy.net import hek >>> client= hek.HEKClient() This creates a client that we will use to interact with the HEK. 2. A simple query To search the HEK, you need a start time, an end time, and an event type. Times are specified as strings or Python date- time objects. Event types are specified as upper case, two letter strings, and are identical to the two letter abbreviations found at the HEK website, http://www.lmsal.com/hek/VOEvent_Spec.html. >>> tstart='2011/08/09 07:23:56' >>> tend='2011/08/09 12:40:29' >>> event_type='FL' >>> result= client.query(hek.attrs.Time(tstart,tend),hek.attrs.EventType(event_type)) The first line defines the search start and end times. The second line specifies the event type, in this ‘FL’ or flare. Line 4 goes out to the web, contacts the HEK, and queries it for the information you have requested. Event data for ALL flares available in the HEK within the time range 2011/08/09 07:23: 56 UT - 2011/08/09 12:40:20 UT will be returned, regardless of which feature recognition method used to detect the flare. Let’s break down the arguments of client.query. The first argument: hek.attrs.Time(tstart,tend) sets the start and end times for the query. The second argument: hek.attrs.EventType(event_type) sets the type of event to look for. Since we have defined event_type = ‘FL’, this sets the query to look for flares. We could have also set the flare event type using the syntax: hek.attrs.FL There is more on the attributes of hek.attrs in section 4 of this guide. 3. The result So, how many flare detections did the query turn up? >>> len(result) 19 The object returned by the above query is a list of Python dictionary objects. Each dictionary consists of key-value pairs that exactly correspond to the parameters listed at http://www.lmsal.com/hek/VOEvent_Spec.html. You can inspect all the dictionary keys very simply: >>> result[0].keys() [u'skel_startc1', u'concept', u'frm_versionnumber', u'hrc_coord', u'refs_orig',.... 20 Chapter 1. SunPy User Guide DemoDoc Documentation, Release and so on. Remember, the HEK query we made returns all the flares in the time-range stored in the HEK, regardless of the feature recognition method. The HEK parameter which stores the the feature recognition method is called “frm_name”. Using list comprehensions (which are very cool), it is easy to get a list of the feature recognition methods used to find each of the flares in the result object, for example: >>> [elem["frm_name"] for elem in result] [u'asainz', u'asainz', u'asainz', u'asainz', u'asainz', u'asainz', u'asainz', u'SSW Latest Events', u'SEC standard', u'Flare Detective - Trigger Module', u'Flare Detective - Trigger Module', u'SSW Latest Events', u'SEC standard', u'Flare Detective - Trigger Module', u'Flare Detective - Trigger Module', u'Flare Detective - Trigger Module', u'Flare Detective - Trigger Module', u'Flare Detective - Trigger Module'] It is likely each flare on the Sun was actually detected multiple times by many different methods. 4. More complex queries The HEK client allows you to make more complex queries. There are two key features you need to know in order to make use of the full power of the HEK client. Firstly, the attribute module - hek.attrs - describes ALL the parame- ters stored by the HEK as listed in http://www.lmsal.com/hek/VOEvent_Spec.html, and the HEK client makes these parameters searchable. To explain this, let’s have a closer look at hek.attrs. The help command is your friend here; scroll down to section DATA you will see: >>> help(hek.attrs) AR = 1.3. Acquiring Data with SunPy 21 DemoDoc Documentation, Release etc etc... The object hek.attrs knows the attributes of the HEK. You’ll see that one of the attributes is a flare object: FL= We can replace hek.attrs.EventType(‘FL’) with hek.attrs.FL - they do the same thing, setting the query to look for flare events. Both methods of setting the event type are provided as a convenience Let’s look further at the FRM attribute: >>> help(hek.attrs.FRM) Help on FRM in module sunpy.net.hek.attrs object: class FRM(__builtin__.object) | Data descriptors defined here: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) | | ------| Data and other attributes defined here: | | Contact = Let’s say I am only interested in those flares identified by the SSW Latest Events tool. I can retrieve those entries only from the HEK with the following command: >>> result= client.query( hek.attrs.Time(tstart,tend), hek.attrs.EventType(event_ ˓→type), hek.attrs.FRM.Name =='SSW Latest Events') >>> len(result) 2 We can also retrieve all the entries in the time range which were not made by SSW Latest Events with the following command: >>> result= client.query( hek.attrs.Time(tstart,tend), hek.attrs.EventType(event_ ˓→type),hek.attrs.FRM.Name !='SSW Latest Events') >>> len(result) 17 22 Chapter 1. SunPy User Guide DemoDoc Documentation, Release We are using Python’s comparison operators to filter the returns from the HEK client. Other comparisons are possible. For example, let’s say I want all the flares that have a peak flux of over 4000.0: >>> result= client.query(hek.attrs.Time(tstart,tend), hek.attrs.EventType(event_ ˓→type), hek.attrs.FL.PeakFlux> 4000.0) >>> len(result) 1 Multiple comparisons can be included. For example, let’s say I want all the flares with a peak flux above 1000 AND west of 800 arcseconds from disk center of the Sun >>> result= client.query(hek.attrs.Time(tstart,tend), hek.attrs.EventType(event_ ˓→type), hek.attrs.Event.Coord1> 800, hek.attrs.FL.PeakFlux> 1000.0) Multiple comparison operators can be used to filter the results back from the HEK. The second important feature about the HEK client is that the comparisons we’ve made above can be combined using Python’s logical operators. This makes complex queries easy to create. However, some caution is advisable. Let’s say I want all the flares west of 50 arcseconds OR have a peak flux over 1000.0: >>> result= client.query(hek.attrs.Time(tstart,tend), hek.attrs.EventType(event_ ˓→type), (hek.attrs.Event.Coord1> 50) or (hek.attrs.FL.PeakFlux> 1000.0)) and as a check >>> [elem["fl_peakflux"] for elem in result] [None, None, None, None, None, None, None, 2326.86, 1698.83, None, None, 2360.49, 3242.64, 1375.93, 6275.98, 923.984] >>> [elem["event_coord1"] for elem in result] [51, 51, 51, 924, 924, 924, 69, 883.2, 883.2, 69, 69, 883.2, 883.2, 883.2, 1.3. Acquiring Data with SunPy 23 DemoDoc Documentation, Release 883.2, 883.2] Note that some of the fluxes are returned as “None”. This is because some feature recognition methods for flares do not report the peak flux. However, because the location of event_coord1 is greater than 50, the entry from the HEK for that flare detection is returned. Let’s say we want all the flares west of 50 arcseconds AND have a peak flux over 1000.0: >>> result= client.query(hek.attrs.Time(tstart,tend), hek.attrs.EventType(event_ ˓→type), (hek.attrs.Event.Coord1> 50) and (hek.attrs.FL.PeakFlux> 1000.0)) >>> [elem["fl_peakflux"] for elem in result] [2326.86, 1698.83, 2360.49, 3242.64, 1375.93, 6275.98] >>> [elem["event_coord1"] for elem in result] [883.2, 883.2, 883.2, 883.2, 883.2, 883.2] In this case none of the peak fluxes are returned with the value None. Since we are using an and logical operator we need a result from the (hek.attrs.FL.PeakFlux > 1000.0) filter. Flares that have None for a peak flux cannot provide this, and so are excluded. The None type in this context effectively means “Don’t know”; in such cases the client returns only those results from the HEK that definitely satisfy the criteria passed to it. 5. Getting data for your event The ‘hek2vso’ module allows you to take an HEK event and acquire VSO records specific to that event. >>> from sunpy.net import hek2vso >>> h2v= hek2vso.H2VClient() There are several ways to use this capability. For example, you can pass in a list of HEK results and get out the corresponding VSO records. Here are the VSO records returned via the tenth result from the HEK query in Section 2 above: >>> result= client.query(hek.attrs.Time(tstart,tend),hek.attrs.EventType(event_type)) >>> vso_records= h2v.translate_and_query(result[10]) >>> len(vso_records[0]) 31 Result 10 is an HEK entry generated by the “Flare Detective” automated flare detection algorithm running on the AIA 193 angstrom waveband. The VSO records are for full disk AIA 193 images between the start and end times of this event. The ‘translate_and_query’ function uses exactly that information supplied by the HEK in order to find the relevant data for that event. Note that the VSO does not generate records for all solar data, so it is possible that an HEK entry corresponds to data that is not accessible via the VSO. You can also go one step further back, passing in a list of HEK attribute objects to define your search, the results of which are then used to generate their corresponding VSO records: >>> from sunpy.net import hek >>> q= h2v.full_query((hek.attrs.Time('2011/08/09 07:23:56','2011/08/09 12:40:29'), ˓→hek.attrs.EventType('FL'))) The full capabilities of the HEK query module can be used in this function (see above). Finally, for greater flexibility, it is possible to pass in a list of HEK results and create the corresponding VSO query attributes. 24 Chapter 1. SunPy User Guide DemoDoc Documentation, Release >>> vso_query= hek2vso.translate_results_to_query(result[10:11]) >>> vso_query[0] [ ˓→56), None)>, ˓→')>] This function allows users finer-grained control of VSO queries generated from HEK results. The ‘hek2vso’ module was developed with support from the 2013 Google Summer of Code. Querying Helioviewer.org with SunPy SunPy can be used to make several basic requests using the The Helioviewer.org API including generating a PNG and downloading a JPEG 2000 image and loading it into a SunPy Map. The SunPy Helioviewer client requires installing two other pieces of software. The first OpenJPEG is an open source library for reading and writing JPEG2000 files. To install OpenJPEG, please follow the instructions at the OpenJPEG homepage. The other package you will need is Glymur. Glymur is an interface between Python and the OpenJPEG libraries. Please follow the instructions here to install Glymur on your system. To interact with the Helioviewer API, users first create a “HelioviewerClient” instance. The client instance can then be used to make various queries against the API using the same parameters one would use when making a web request. Nearly all requests require the user to specify the data they are interested in and this can be done using one of two methods: 1. Call “get_data_sources()” to get a list of the data that is available, and use the source id numbers referenced in the result to refer to a particular dataset, or, 2. Specify the four components of a Helioviewer.org data source or layer: observatory, instrument, detector and measurement. Let’s begin by getting a list of data sources available on the server using the get_datasources method: from sunpy.net.helioviewer import HelioviewerClient hv= HelioviewerClient() datasources= hv.get_data_sources() # print a list of datasources and their associated ids for observatory, instruments in datasources.items(): for inst, detectors in instruments.items(): for det, measurements in detectors.items(): for meas, params in measurements.items(): print("%s %s: %d"% (observatory, params['nickname'], params['sourceId ˓→'])) At time of writing (2014/01/06) Helioviewer provides JP2 images from AIA, HMI, LASCO C2/C3, EIT, MDI, STEREO A/B COR1/2 & EUVI, SWAP and SXT. New sources of JP2 images are being added every few months; please use the code snippet above to get an up-to-date list of available data sources. Suppose we next want to download a PNG image of the latest AIA 304 image available on Helioviewer.org. We could use the explicit approach as shown in the following example.: >>> from sunpy.net.helioviewer import HelioviewerClient >>> import matplotlib.pyplot as plt >>> from matplotlib.image import imread 1.3. Acquiring Data with SunPy 25 DemoDoc Documentation, Release >>> hv= HelioviewerClient() >>> file= hv.download_png('2099/01/01', 4.8,"[SDO,AIA,AIA,304,1,100]", x0=0, y0=0, ˓→width=512, height=512) >>> im= imread(file) >>> plt.imshow(im) >>> plt.axis('off') >>> plt.show() Where 4.8 refers to the image resolution in arcseconds per pixel (larger values mean lower resolution), the “1” and “100” in the layer string refer to the visibility (visible/hidden) and opacity, x0 and y0 are the center points about which to focus and the width and height are the pixel values for the image dimensions. Note that the filename of the returned file has the date and time of the request, not of any of the times shown in the image itself. This is not a bug. Helioviewer.org finds images closest to the requested time. Since the user may ask for images from multiple sources, and each of them may have a different observation time, the problem becomes which time is the most appropriate to associate with the resultant image. Helioviewer.org doesn’t choose between the images times, but instead uses the request time to construct the image filename. This means that the image file names for request times in the future (like in this example) can look a little unusual compared to the times in the image. If we find that the source id for AIA 304 is is 13, we could make the same request using: hv.download_png('2099/01/01', 4.8,"[13,1,100]", x0=0, y0=0, width=512, height=512) Now suppose we wanted to create a composite PNG image using data from two different AIA wavelengths and LASCO C2 coronagraph data. The layer string is extended to include the additional data sources, and opacity is throttled down 26 Chapter 1. SunPy User Guide DemoDoc Documentation, Release for the second AIA layer so that it does not completely block out the lower layer.: >>> from sunpy.net.helioviewer import HelioviewerClient >>> import matplotlib.pyplot as plt >>> from matplotlib.image import imread >>> hv= HelioviewerClient() >>> file= hv.download_png('2099/01/01',6,"[SDO,AIA,AIA,304,1,100],[SDO,AIA,AIA,193, ˓→1,50],[SOHO,LASCO,C2,white-light,1,100]", x0=0, y0=0, width=768, height=768) >>> im= imread(file) >>> plt.imshow(im) >>> plt.axis('off') >>> plt.show() Next, let’s see how we can download a JPEG 2000 image and load it into a SunPy Map object. The overall syntax is similar to the download_png request, expect instead of specifying a single string to indicate which layers to use, here we can specify the values as separate keyword arguments.: >>> from sunpy.net.helioviewer import HelioviewerClient >>> import matplotlib.pyplot as plt >>> from astropy.units import Quantity >>> from sunpy.map import Map >>> hv= HelioviewerClient() >>> filepath= hv.download_jp2('2012/07/05 00:30:00', observatory='SDO', instrument= ˓→'HMI', detector='HMI', measurement='continuum') >>> hmi= Map(filepath) 1.3. Acquiring Data with SunPy 27 DemoDoc Documentation, Release >>> xrange= Quantity([200, 550],'arcsec') >>> yrange= Quantity([-400, 200],'arcsec') >>> hmi.submap(xrange, yrange).peek() Every JP2 file provided by the Helioviewer Project has been processed to generate an image that can be used for browse purposes. This typically involves following the standard image processing procedure used by each instrument team to convert their science data into an image for a webpage. The JP2 image is then scaled between 0 and 255 (byte-scaled). Please note that the JP2 image data is NOT the same as the original science data. In the example above, SunPy queries Helioviewer for the relevant JP2 file closest to the input time, downloads it, and selects a color table based on the JP2 image meta data for plotting. The color table is that used by the Helioviewer Project to display JP2 images in their browse clients. For more information about using querying Helioviewer.org, see the Helioviewer.org API documentation at: http: //helioviewer.org/api/. Using the database package The database package offers the possibility to save retrieved data (e.g. via the sunpy.net.vso package) onto a local or remote database. The database may be a single file located on a local hard drive (if a SQLite database is used) or a local or remote database server (see the SQLAlchemy documentation for a list of supported databases) This makes it possible to fetch required data from the local database instead of downloading it again from a remote server. The package sunpy.database was developed as part of Google Summer of Code (GSOC) 2013. 28 Chapter 1. SunPy User Guide DemoDoc Documentation, Release 1. Connecting and initializing the database To start a connection to an existing or a new database, instantiate a Database object. >>> from sunpy.database import Database >>> database= Database('sqlite:///sunpydata.sqlite') The database object in our example above connects to a new SQLite database with the file name “sunpydata.sqlite” in the current directory. The first parameter of Database receives one mandatory argument: a URL which describes how to connect to the database. This value is directly passed to sqlalchemy.create_engine(). The supported format of this URL is described by the documentation of sqlalchemy.create_engine() as follows: “The string form of the URL is dialect+driver://user:password@host/dbname[? key=value..], where dialect is a database name such as mysql, oracle, postgresql, etc., and driver the name of a DBAPI, such as psycopg2, pyodbc, cx_oracle, etc.” Note that a connection is only established when it’s really needed, i.e. if some query is sent to the database to read from it. Transactions can also be committed explicitly using the Database.commit() method. Warning: If you are using Database objects in an interactive Python session you must not forget to call the Database.commit() method on them explicitly before quitting the Python session! Otherwise, all changes on the altered databases are lost! Note: You can set the default database url in the sunpy config file under the ‘database’ section. See Customizing SunPy for information on the config file. A database section might look like this: [database] url= sqlite:////home/user/sunpy/my_database.sqlite 2. Adding new entries Before explaining how to add new entries, it is important to know what information is saved in an entry. Each database entry is an instance of the class tables.DatabaseEntry with the following attributes: 1.3. Acquiring Data with SunPy 29 DemoDoc Documentation, Release Attribute Description id A unique ID number. By default it is None, but automatically set to the maximum number plus one when an entry is added to the database. source The source is the name of an observatory or the name of a network of observatories. provider The name of the server which provides the retrieved data. physobs A physical observable identifier used by VSO. fileid The file ID is a string defined by the data provider that should point to a specific data product. The association of fileid to the specific data may change sometime, if the fileid always points to the latest calibrated data. observa- The date and time when the observation of the data started. tion_time_start observa- The date and time when the observation of the data ended. tion_time_end instrument The instrument which was used to observe the data. size The size of the data in kilobytes (-1 if unknown). wavemin The value of the measured wave length. wavemax This is the same value as wavemin. The value is stored twice, because each suds.sudsobject.QueryResponseBlock which is used by the vso package contains both these values. path A local file path where the according FITS file is saved. down- The date and time when the files connected to a query have been downloaded. Note: this is not load_time the date and time when this entry has been added to a database! starred Entries can be starred to mark them. By default, this value is False. fits_header_entriesA list of tables.FitsHeaderEntry instances. fits_key_commentsA list of tables.FitsKeyComment instances. tags A list of tables.Tag instances. • The id attribute is automatically set if an entry is added to a database. • The attributes source, provider, physobs, fileid, observation_time_start, observation_time_end, instrument, size, wavemin, and wavemax are set by methods which use the VSO interface. In particular, these are Database.add_from_vso_query_result(), Database.download() and possibly Database.fetch(). • The attributes path and download_time are set by the method Database.download() and also possibly by Database.fetch(). starred is set or changed via the method Database.star() or unstar(), respectively. Analogously, tags is set via the methods Database.tag() and Database. remove_tag(). • The attribute fits_header_entries is set by the methods Database.download(), Database. add_from_dir(), and Database.add_from_file(). 2.1 Adding entries from one FITS file The method Database.add_from_file() receives one positional argument (either a string or a file-like object) which is used to add at least one new entry from the given FITS file to the database. Why “at least one” and not “exactly one”? The reason is that each database entry does not represent one file but one FITS header of a file. That means, if you pass a file which has 5 FITS headers in it, 5 entries will be added to the database. The file in the following example (sunpy.data.sample.AIA_171_IMAGE) has only one FITS header, that is why just one entry is added to the database. However, if you are working with Hinode/SOT files you may notice that for each file you get two entries, one which refers to the observation and another that contains some (useless - as discussed in the fitsbits mailing list) telemetry data. The method saves the value of path by either simply passing on the value of the received argument (if it was a string) or by reading the value of file.name where file is the passed argument. If the path cannot be determined, it stays 30 Chapter 1. SunPy User Guide DemoDoc Documentation, Release to None (the default value). The values of wavemin and wavemax are only set if the wavelength unit of the passed FITS file can be found out or if the attribute default_waveunit of the database object is set. These values are then used to convert from the used unit to nanometers. The rationale behind this behaviour is that it makes querying for wavelengths more flexible. tl;dr: wavelengths are always stored in nanometers! The value of the attribute instrument is simply set by looking up the FITS header key INSTRUME. The value of observation_time_start is set by searching for the FITS header key DATE-OBS or DATE_OBS. Analogously, obser- vation_time_end is set by searching for DATE-END or DATE_END. Finally, the whole FITS header is stored in the attribute fits_header_entries as a list of tables.FitsHeaderEntry instances. All FITS comments are stored in the attribute fits_key_comments which is a list of tables.FitsKeyComment instances. Using the function len on a Database object returns the number of saved database entries. To get the first entry of the database, database[0] is used (the ID number of the entries does not matter, database[0] always re- turns the oldest saved entry of the database). If the database had been empty, this expression would have raised an IndexError. In section 3, more advanced formats of the slicing syntax are introduced. >>> import sunpy.data >>> sunpy.data.download_sample_data(overwrite=False) >>> import sunpy.data.sample >>> database.add_from_file(sunpy.data.sample.AIA_171_IMAGE) >>> len(database) 1 >>> entry= database[0] >>> entry.path == sunpy.data.sample.AIA_171_IMAGE True >>> entry.wavemin, entry.wavemax (17.1, 17.1) >>> entry.instrument 'AIA_3' >>> entry.observation_time_start, entry.observation_time_end (datetime.datetime(2011, 3, 19, 10, 54, 0, 340000), None) >>> len(entry.fits_header_entries) 170 >>> for fits_header_entry in entry.fits_header_entries[:10]: ... print' {entry.key}\n\t{entry.value}'.format(entry=fits_header_entry) SIMPLE True BITPIX 32 NAXIS 2 NAXIS1 1024 NAXIS2 1024 EXTEND True COMMENT FITS (Flexible Image Transport System) format is defined in 'Astronomy and ˓→Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H ORIGIN SDO/JSOC-SDP DATE 2011-03-19T11:08:25 TELESCOP SDO/AIA 1.3. Acquiring Data with SunPy 31 DemoDoc Documentation, Release >>> for fits_key_comment in entry.fits_key_comments: ... print' {comment.key}\n\t{comment.value}'.format(comment=fits_key_comment) NAXIS number of data axes NAXIS1 length of data axis 1 DATASUM data unit checksum updated 2011-03-19T11:08:18 EXTEND FITS dataset may contain extensions BITPIX number of bits per data pixel SIMPLE file does conform to FITS standard CHECKSUM HDU checksum updated 2011-03-19T11:08:18 NAXIS2 length of data axis 2 2.2 Adding entries from a directory of FITS files Adding all FITS files from a certain directory works by calling the method Database.add_from_dir() and pass- ing the desired directory to it. By setting the keyword argument ignore_already_added to True, no exception is raised if it is attempted to add an already existing entry (In this case, setting this parameter is required because the file sunpy. data.sample.AIA_171_IMAGE was already added which is located in the directory sampledata_dir). >>> from sunpy import config >>> sampledata_dir= config.get("downloads","sample_dir") >>> database.default_waveunit='angstrom' >>> database.add_from_dir(sampledata_dir, ignore_already_added=True) >>> len(database) 25 2.3 Adding entries using the VSO interface 2.3.1 Adding entries from a VSO query result A VSO query result can be used to add new entries to the database. The number of database entries that will be added is equal to the value of len(qr) in the following code snippet. Note that the method Database. add_from_vso_query_result() does not download any files, though. If you want to add new entries using the VSO and also want to download files at the same time, take a look at the following two sections. >>> from sunpy.net import vso >>> client= vso.VSOClient() >>> qr= client.query( ... vso.attrs.Time('2011-05-08','2011-05-08 00:00:05'), ... vso.attrs.Instrument('AIA')) >>> len(qr) 4 >>> database.add_from_vso_query_result(qr) >>> len(database) 29 32 Chapter 1. SunPy User Guide DemoDoc Documentation, Release 2.3.2 Downloading The method Database.download() queries the VSO by passing the given query on to sunpy.net.vso. VSOClient.query(). Note that not the number of records of the resulting query result determines the number of entries that will be added to the database! The number of entries that will be added depends on the total number of FITS headers. The download method also accepts an optional keyword argument path which is passed as-is to sunpy.net.vso.VSOClient.get() and determines the value of the path attribute of each entry. >>> database.download( ... vso.attrs.Time('2012-08-05','2012-08-05 00:00:05'), ... vso.attrs.Instrument('AIA')) >>> len(database) 33 2.3.3 “Clever” Fetching The method Database.fetch() queries the database if the given query has already been used once to add entries using the method Database.download(). Otherwise, the given query is used to download and add new data via Database.download(). This means: If you are not sure whether the required data already exists in the database, use Database.fetch() and you use a method to save bandwidth! As you can see in the following example, the first call of the fetch method results in a list of entries. These are the entries which have been returned by querying the database. This is the reason why the number of saved entries in the database is still 32 even after the fetch method has been called. The second fetch call has not been done already on a download call, therefore the given query is used to download new data and add these resulting entries to the database. Because the query result translates to 4 records, 4 new entries are added to the database after the fetch call. >>> entries= database.fetch( ... vso.attrs.Time('2012-08-05','2012-08-05 00:00:05'), ... vso.attrs.Instrument('AIA')) >>> entries is None False >>> len(entries) 4 >>> len(database) 33 >>> entries= database.fetch( ... vso.attrs.Time('2013-08-05','2013-08-05 00:00:05'), ... vso.attrs.Instrument('AIA')) >>> entries is None True >>> len(database) 37 2.4 Adding entries manually Although usually not required, it is also possible to add database entries by specifying the parameters manually. To do so, you simply pass the values as keyword arguments to tables.DatabaseEntry as follows: >>> from sunpy.database.tables import DatabaseEntry >>> entry= DatabaseEntry(instrument='EIT', wavemin=25.0) >>> database.add(entry) 1.3. Acquiring Data with SunPy 33 DemoDoc Documentation, Release >>> entry in database False >>> database.commit() >>> entry in database True >>> len(database) 38 Note that the in operator works only as expected after the Database.commit() method has been called! 3. Displaying entries in a table Meanwhile, 37 entries have been added, all of them saving a lot of data. How can selected data of each entry be displayed? Fortunately, there is a helper function to do this: tables.display_entries() takes two arguments: the first one is an iterator of tables.DatabaseEntry instances. Remember that an instance of Database yields instances of tables.DatabaseEntry instances, so you can simply pass a database object. The second argument is an iterable of the resulting columns in the table to be displayed. Each string in this iterable is used to access the en- try’s attribute values. In the following example, the values of entry.id, entry.observation_time_start, entry.observation_time_end, entry.instrument, entry.wavemin, and entry.wavemax are dis- played (where entry stands for the respective database entry). Note that N/A is displayed if the value cannot be found or is not set. >>> from sunpy.database.tables import display_entries >>> print display_entries(database, ... ['id','observation_time_start','observation_time_end', ... 'instrument','wavemin','wavemax']) id observation_time_start observation_time_end instrument wavemin wavemax ------1 2011-03-19 10:54:00 N/A AIA_3 17.1 17.1 2 N/A N/A N/A N/A N/A 3 2013-09-21 16:00:06 N/A AIA_2 19.3 19.3 4 2011-03-19 10:54:00 N/A AIA_3 17.1 17.1 5 2014-04-09 06:00:12 N/A AIA_3 17.1 17.1 6 2011-09-22 00:00:00 2011-09-22 00:00:00 BIR N/A N/A 7 N/A N/A N/A N/A N/A 8 2002-06-25 10:00:10 N/A EIT 19.5 19.5 9 2002-02-20 11:06:00 2002-02-20 11:06:43 RHESSI N/A N/A 10 N/A N/A N/A N/A N/A 11 N/A N/A N/A N/A N/A 12 N/A N/A N/A N/A N/A 13 N/A N/A N/A N/A N/A 14 N/A N/A N/A N/A N/A 15 N/A N/A N/A N/A N/A 16 N/A N/A N/A N/A N/A 17 N/A N/A N/A N/A N/A 18 N/A N/A N/A N/A N/A 19 N/A N/A N/A N/A N/A 20 2010-10-16 19:12:18 2010-10-16 19:12:22 RHESSI N/A N/A 21 N/A N/A N/A N/A N/A 22 N/A N/A N/A N/A N/A 23 N/A N/A N/A N/A N/A 24 2012-10-30 15:30:01 N/A AIA_4 9.4 9.4 25 2012-01-01 00:16:07 N/A SWAP 17.4 17.4 26 2011-05-08 00:00:00 2011-05-08 00:00:01 AIA 17.1 17.1 27 2011-05-08 00:00:00 2011-05-08 00:00:01 AIA 21.1 21.1 28 2011-05-08 00:00:02 2011-05-08 00:00:03 AIA 9.4 9.4 34 Chapter 1. SunPy User Guide DemoDoc Documentation, Release 29 2011-05-08 00:00:03 2011-05-08 00:00:04 AIA 33.5 33.5 30 2012-08-05 00:00:01 2012-08-05 00:00:02 AIA 9.4 9.4 31 2012-08-05 00:00:01 2012-08-05 00:00:02 AIA 9.4 9.4 32 2012-08-05 00:00:02 2012-08-05 00:00:03 AIA 33.5 33.5 33 2012-08-05 00:00:02 2012-08-05 00:00:03 AIA 33.5 33.5 34 2013-08-05 00:00:01 2013-08-05 00:00:02 AIA 9.4 9.4 35 2013-08-05 00:00:01 2013-08-05 00:00:02 AIA 9.4 9.4 36 2013-08-05 00:00:02 2013-08-05 00:00:03 AIA 33.5 33.5 37 2013-08-05 00:00:02 2013-08-05 00:00:03 AIA 33.5 33.5 38 N/A N/A EIT 25.0 N/A In Section 2.1, “Adding entries from one FITS file”, it has already been shows that the index operator can be used to access certain single database entries like database[5] or even use negative indices such as in database[-2] to get the second-latest entry. It is also possible to use the more advanced slicing syntax: database[:] returns a list of all database entries (and is hereby an alias for list(database)), database[::2] returns a list of every 2nd entry, starting with the oldest one. As you can imagine, database[9::10] starts with the 10th entry and returns a list of every 10th entry from there. >>> print display_entries(database[9::10], ... ['id','observation_time_start','observation_time_end', ... 'instrument','wavemin','wavemax']) id observation_time_start observation_time_end instrument wavemin wavemax ------10 N/A N/A N/A N/A N/A 20 2010-10-16 19:12:18 2010-10-16 19:12:22 RHESSI N/A N/A 30 2012-08-05 00:00:01 2012-08-05 00:00:02 AIA 9.4 9.4 4. Removing entries database.remove() can be used to remove database entries from the SunPy database. It takes a tables. DatabaseEntry object as argument. For example, let us imagine we want to only have database entries which have some observation time saved. To remove all entries where the value of both observation_time_start and observation_time_end is None, one can simply iterate over the database and uses the Database.remove() method to remove those where the just described predicate is true: >>> for database_entry in database: ... if database_entry.observation_time_start is None and database_entry. ˓→observation_time_end is None: ... database.remove(database_entry) ... >>> len(database) 22 >>> print display_entries(database, ... ['id','observation_time_start','observation_time_end', ... 'instrument','wavemin','wavemax']) id observation_time_start observation_time_end instrument wavemin wavemax ------1 2011-03-19 10:54:00 N/A AIA_3 17.1 17.1 3 2013-09-21 16:00:06 N/A AIA_2 19.3 19.3 4 2011-03-19 10:54:00 N/A AIA_3 17.1 17.1 5 2014-04-09 06:00:12 N/A AIA_3 17.1 17.1 6 2011-09-22 00:00:00 2011-09-22 00:00:00 BIR N/A N/A 8 2002-06-25 10:00:10 N/A EIT 19.5 19.5 9 2002-02-20 11:06:00 2002-02-20 11:06:43 RHESSI N/A N/A 1.3. Acquiring Data with SunPy 35 DemoDoc Documentation, Release 20 2010-10-16 19:12:18 2010-10-16 19:12:22 RHESSI N/A N/A 24 2012-10-30 15:30:01 N/A AIA_4 9.4 9.4 25 2012-01-01 00:16:07 N/A SWAP 17.4 17.4 26 2011-05-08 00:00:00 2011-05-08 00:00:01 AIA 17.1 17.1 27 2011-05-08 00:00:00 2011-05-08 00:00:01 AIA 21.1 21.1 28 2011-05-08 00:00:02 2011-05-08 00:00:03 AIA 9.4 9.4 29 2011-05-08 00:00:03 2011-05-08 00:00:04 AIA 33.5 33.5 30 2012-08-05 00:00:01 2012-08-05 00:00:02 AIA 9.4 9.4 31 2012-08-05 00:00:01 2012-08-05 00:00:02 AIA 9.4 9.4 32 2012-08-05 00:00:02 2012-08-05 00:00:03 AIA 33.5 33.5 33 2012-08-05 00:00:02 2012-08-05 00:00:03 AIA 33.5 33.5 34 2013-08-05 00:00:01 2013-08-05 00:00:02 AIA 9.4 9.4 35 2013-08-05 00:00:01 2013-08-05 00:00:02 AIA 9.4 9.4 36 2013-08-05 00:00:02 2013-08-05 00:00:03 AIA 33.5 33.5 37 2013-08-05 00:00:02 2013-08-05 00:00:03 AIA 33.5 33.5 There are more possible ways to remove entries: You can remove every 2nd entry by iterating over database[::2] and then calling passing every yielded entry to Database.remove(). You can even do advanced operations easily like for example removing every entry with a certain observation start time, instrument, and FITS header entry pair. This requires knowledge of the Database.query() method though, which will be covered in section 7, “Querying the database”. 5. Editing entries 5.1 Starring and unstarring entries The database package supports marking certain entries as “starred”. This concept may be familiar to you from graphi- cal applications such as E-Mail clients or photo managers. The method Database.star() marks the passed entry as starred. Let’s say we are for some reason interested in all values that have a wavelength of 20nm or higher, so we mark those as starred: >>> for database_entry in database: ... if database_entry.wavemin> 20: ... database.star(database_entry) >>> print display_entries( ... filter(lambda entry: entry.starred, database), ... ['id','observation_time_start','observation_time_end', ... 'instrument','wavemin','wavemax']) id observation_time_start observation_time_end instrument wavemin wavemax ------27 2011-05-08 00:00:00 2011-05-08 00:00:01 AIA 21.1 21.1 29 2011-05-08 00:00:03 2011-05-08 00:00:04 AIA 33.5 33.5 32 2012-08-05 00:00:02 2012-08-05 00:00:03 AIA 33.5 33.5 33 2012-08-05 00:00:02 2012-08-05 00:00:03 AIA 33.5 33.5 36 2013-08-05 00:00:02 2013-08-05 00:00:03 AIA 33.5 33.5 37 2013-08-05 00:00:02 2013-08-05 00:00:03 AIA 33.5 33.5 So remove the mark from these entries, the method Database.unstar() works the same way. 5.2 Setting and removing tags The starring concept is somewhat “binary” in a way that one can only decide whether to mark an entry as starred or not. To add some more information by assigning keywords to entries, the database package also supports tags. The 36 Chapter 1. SunPy User Guide DemoDoc Documentation, Release tags property of a database object holds all tags that are saved in this database. Let us assign the tag spring to all entries that have been observed in March, April, or May on any day at any year: >>> for database_entry in database: ... if database_entry.observation_time_start.month in [3,4,5]: ... database.tag(database_entry,'spring') >>> database.tags [ 5.3 Changing custom attributes What still annoys me a bit is that there are entries in the database where the end of the observation time is None. Let us change them to the same value to the start of the observation time (because we can and because it looks prettier, not because it is accurate). The Database.edit() method receives the database entry to be edited and any number of keyword arguments which describe which values to change and how. >>> for database_entry in database: ... if database_entry.observation_time_end is None: ... database.edit(database_entry, observation_time_end=database_entry. ˓→observation_time_start) ... >>> print display_entries( ... database, ... ['id','observation_time_start','observation_time_end', ... 'instrument','wavemin','wavemax']) id observation_time_start observation_time_end instrument wavemin wavemax ------1 2011-03-19 10:54:00 2011-03-19 10:54:00 AIA_3 17.1 17.1 3 2013-09-21 16:00:06 2013-09-21 16:00:06 AIA_2 19.3 19.3 4 2011-03-19 10:54:00 2011-03-19 10:54:00 AIA_3 17.1 17.1 5 2014-04-09 06:00:12 2014-04-09 06:00:12 AIA_3 17.1 17.1 6 2011-09-22 00:00:00 2011-09-22 00:00:00 BIR N/A N/A 8 2002-06-25 10:00:10 2002-06-25 10:00:10 EIT 19.5 19.5 9 2002-02-20 11:06:00 2002-02-20 11:06:43 RHESSI N/A N/A 20 2010-10-16 19:12:18 2010-10-16 19:12:22 RHESSI N/A N/A 24 2012-10-30 15:30:01 2012-10-30 15:30:01 AIA_4 9.4 9.4 25 2012-01-01 00:16:07 2012-01-01 00:16:07 SWAP 17.4 17.4 26 2011-05-08 00:00:00 2011-05-08 00:00:01 AIA 17.1 17.1 27 2011-05-08 00:00:00 2011-05-08 00:00:01 AIA 21.1 21.1 28 2011-05-08 00:00:02 2011-05-08 00:00:03 AIA 9.4 9.4 29 2011-05-08 00:00:03 2011-05-08 00:00:04 AIA 33.5 33.5 30 2012-08-05 00:00:01 2012-08-05 00:00:02 AIA 9.4 9.4 31 2012-08-05 00:00:01 2012-08-05 00:00:02 AIA 9.4 9.4 1.3. Acquiring Data with SunPy 37 DemoDoc Documentation, Release 32 2012-08-05 00:00:02 2012-08-05 00:00:03 AIA 33.5 33.5 33 2012-08-05 00:00:02 2012-08-05 00:00:03 AIA 33.5 33.5 34 2013-08-05 00:00:01 2013-08-05 00:00:02 AIA 9.4 9.4 35 2013-08-05 00:00:01 2013-08-05 00:00:02 AIA 9.4 9.4 36 2013-08-05 00:00:02 2013-08-05 00:00:03 AIA 33.5 33.5 37 2013-08-05 00:00:02 2013-08-05 00:00:03 AIA 33.5 33.5 You may ask yourself now “Why can’t I simply use database_entry.observation_time_end = database_entry.observation_time_start”? Well, the answer is: you can, but it has one major disadvan- tage: you cannot undo this operation if you don’t use the methods of the database objects such as Database.edit. See the section 6, “Undoing and redoing operations”, to see how undoing and redoing works. 6. Undoing and redoing operations A very handy feature of the database package is that every operation can be reverted that changes the database in some way. The Database class has the methods Database.undo() and Database.redo() to undo and redo the last n commands, respectively. Note: The undo and redo history are only saved in-memory! That means in particular, that if you work on a Database object in an interactive Python session and quit this session, the undo and redo history are lost. In the following snippet, the operations from the sections 5.3, 5.2, and 5.1 are undone. Note the following changes: there are no longer any tags anymore saved in the database, there is no entry which is starred and there are again entries with no end of observation time. >>> database.undo(4) # undo the edits from 5.3 (4 records have been affected) >>> database.undo(6) # undo tagging some entries with the tag 'spring' >>> database.undo(4) # undo starring entries >>> print display_entries( ... database, ... ['id','observation_time_start','observation_time_end', ... 'instrument','wavemin','wavemax','tags','starred']) id observation_time_start observation_time_end instrument wavemin wavemax tags starred ------1 2011-03-19 10:54:00 N/A AIA_3 17.1 17.1 N/A No 3 2013-09-21 16:00:06 N/A AIA_2 19.3 19.3 N/A No 4 2011-03-19 10:54:00 N/A AIA_3 17.1 17.1 N/A No 5 2014-04-09 06:00:12 N/A AIA_3 17.1 17.1 N/A No 6 2011-09-22 00:00:00 2011-09-22 00:00:00 BIR N/A N/A N/A No 8 2002-06-25 10:00:10 N/A EIT 19.5 19.5 N/A No 9 2002-02-20 11:06:00 2002-02-20 11:06:43 RHESSI N/A N/A N/A No 20 2010-10-16 19:12:18 2010-10-16 19:12:22 RHESSI N/A N/A N/A No 24 2012-10-30 15:30:01 N/A AIA_4 9.4 9.4 N/A No 25 2012-01-01 00:16:07 N/A SWAP 17.4 17.4 N/A No 26 2011-05-08 00:00:00 2011-05-08 00:00:01 AIA 17.1 17.1 N/A No 27 2011-05-08 00:00:00 2011-05-08 00:00:01 AIA 21.1 21.1 N/A Yes 28 2011-05-08 00:00:02 2011-05-08 00:00:03 AIA 9.4 9.4 N/A No 29 2011-05-08 00:00:03 2011-05-08 00:00:04 AIA 33.5 33.5 N/A Yes 30 2012-08-05 00:00:01 2012-08-05 00:00:02 AIA 9.4 9.4 N/A No 31 2012-08-05 00:00:01 2012-08-05 00:00:02 AIA 9.4 9.4 N/A No 32 2012-08-05 00:00:02 2012-08-05 00:00:03 AIA 33.5 33.5 N/A Yes 33 2012-08-05 00:00:02 2012-08-05 00:00:03 AIA 33.5 33.5 N/A Yes 34 2013-08-05 00:00:01 2013-08-05 00:00:02 AIA 9.4 9.4 N/A No 35 2013-08-05 00:00:01 2013-08-05 00:00:02 AIA 9.4 9.4 N/A No 38 Chapter 1. SunPy User Guide DemoDoc Documentation, Release 36 2013-08-05 00:00:02 2013-08-05 00:00:03 AIA 33.5 33.5 N/A Yes 37 2013-08-05 00:00:02 2013-08-05 00:00:03 AIA 33.5 33.5 N/A Yes The redo method reverts the last n operations that have been undone. If not that many operations can be redone (i.e. any number greater than 14 in this example), an exception is raised. You can see that the redo call reverts the original state: the tags appeared again and all entries with a wavelength >20nm are starred again. Also, there are no entries with no stored end of observation time anymore. >>> database.redo(14) # redo all undone operations >>> print display_entries( ... database, ... ['id','observation_time_start','observation_time_end', ... 'instrument','wavemin','wavemax','tags','starred']) id observation_time_start observation_time_end instrument wavemin wavemax tags ˓→starred ------ ˓→-- 1 2011-03-19 10:54:00 2011-03-19 10:54:00 AIA_3 17.1 17.1 spring No 3 2013-09-21 16:00:06 2013-09-21 16:00:06 AIA_2 19.3 19.3 N/A No 4 2011-03-19 10:54:00 2011-03-19 10:54:00 AIA_3 17.1 17.1 spring No 5 2014-04-09 06:00:12 2014-04-09 06:00:12 AIA_3 17.1 17.1 spring No 6 2011-09-22 00:00:00 2011-09-22 00:00:00 BIR N/A N/A N/A No 8 2002-06-25 10:00:10 2002-06-25 10:00:10 EIT 19.5 19.5 N/A No 9 2002-02-20 11:06:00 2002-02-20 11:06:43 RHESSI N/A N/A N/A No 20 2010-10-16 19:12:18 2010-10-16 19:12:22 RHESSI N/A N/A N/A No 24 2012-10-30 15:30:01 2012-10-30 15:30:01 AIA_4 9.4 9.4 N/A No 25 2012-01-01 00:16:07 2012-01-01 00:16:07 SWAP 17.4 17.4 N/A No 26 2011-05-08 00:00:00 2011-05-08 00:00:01 AIA 17.1 17.1 spring No 27 2011-05-08 00:00:00 2011-05-08 00:00:01 AIA 21.1 21.1 spring Yes 28 2011-05-08 00:00:02 2011-05-08 00:00:03 AIA 9.4 9.4 spring No 29 2011-05-08 00:00:03 2011-05-08 00:00:04 AIA 33.5 33.5 spring Yes 30 2012-08-05 00:00:01 2012-08-05 00:00:02 AIA 9.4 9.4 N/A No 31 2012-08-05 00:00:01 2012-08-05 00:00:02 AIA 9.4 9.4 N/A No 32 2012-08-05 00:00:02 2012-08-05 00:00:03 AIA 33.5 33.5 N/A Yes 33 2012-08-05 00:00:02 2012-08-05 00:00:03 AIA 33.5 33.5 N/A Yes 34 2013-08-05 00:00:01 2013-08-05 00:00:02 AIA 9.4 9.4 N/A No 35 2013-08-05 00:00:01 2013-08-05 00:00:02 AIA 9.4 9.4 N/A No 36 2013-08-05 00:00:02 2013-08-05 00:00:03 AIA 33.5 33.5 N/A Yes 37 2013-08-05 00:00:02 2013-08-05 00:00:03 AIA 33.5 33.5 N/A Yes 7. Querying the database The API for querying databases is similar to querying the VSO using the method sunpy.net.vso.VSOClient. query(). The Database.query() method accepts any number of ORed query attributes (using |) and combines them using AND. It returns a list of matched database entries. The special thing about querying databases is that all attributes support the unary operator ~ to negate specific attributes. Example: the query ~Instrument('EIT') returns all entries that have not been observed with the EIT. 7.1 Using VSO attributes Using the attributes from sunpy.net.vso.attrs is quite intuitive: the simple attributes and the Time attribute work exactly as you expect it. Note though that the near parameter of sunpy.net.vso.attrs.Time is ignored! The reason for this is that its behaviour is not documented and that it is different depending on the server which is requested. The following query returns the data that was added in section 2.3.2, “Downloading”: 1.3. Acquiring Data with SunPy 39 DemoDoc Documentation, Release >>> print display_entries( ... database.query(vso.attrs.Time('2012-08-05','2012-08-05 00:00:05'), vso.attrs. ˓→Instrument('AIA')), ... ['id','observation_time_start','observation_time_end', ... 'instrument','wavemin','wavemax'], sort= True) id observation_time_start observation_time_end instrument wavemin wavemax ------30 2012-08-05 00:00:01 2012-08-05 00:00:02 AIA 9.4 9.4 31 2012-08-05 00:00:01 2012-08-05 00:00:02 AIA 9.4 9.4 32 2012-08-05 00:00:02 2012-08-05 00:00:03 AIA 33.5 33.5 33 2012-08-05 00:00:02 2012-08-05 00:00:03 AIA 33.5 33.5 When using the sunpy.net.vso.attrs.Wave attribute, you have to specify a unit using astropy.units. Quantity. If not an error is raised. This also implies that there is no default unit that is used by the class. To know how you can specify a detail using astropy check astropy.units. >>> from astropy import units as u >>> print display_entries( ... database.query(vso.attrs.Wave(1.0*u.nm, 2.0*u.nm)), ... ['id','observation_time_start','observation_time_end', ... 'instrument','wavemin','wavemax'], sort= True) id observation_time_start observation_time_end instrument wavemin wavemax ------1 2011-03-19 10:54:00 2011-03-19 10:54:00 AIA_3 17.1 17.1 25 2012-01-01 00:16:07 2012-01-01 00:16:07 SWAP 17.4 17.4 26 2011-05-08 00:00:00 2011-05-08 00:00:01 AIA 17.1 17.1 3 2013-09-21 16:00:06 2013-09-21 16:00:06 AIA_2 19.3 19.3 4 2011-03-19 10:54:00 2011-03-19 10:54:00 AIA_3 17.1 17.1 5 2014-04-09 06:00:12 2014-04-09 06:00:12 AIA_3 17.1 17.1 8 2002-06-25 10:00:10 2002-06-25 10:00:10 EIT 19.5 19.5 7.2 Database-specific attributes There are 5 additional query attributes supported by the database package. They can be imported from the submodule sunpy.database.attrs and are in particular: • Starred • Tag • Path • DownloadTime • FitsHeaderEntry The following query searches for all entries that have the tag ‘spring’ or (inclusive or!) are starred and have not the FITS header key ‘WAVEUNIT’ with the value ‘Angstrom’: >>> import sunpy.database.attrs as dbattrs >>> print display_entries( ... database.query(dbattrs.Tag('spring')| dbattrs.Starred(),~dbattrs. ˓→FitsHeaderEntry('WAVEUNIT','Angstrom')), ... ['id','observation_time_start','observation_time_end', ... 'instrument','wavemin','wavemax','tags','starred'], sort= True) id observation_time_start observation_time_end instrument wavemin wavemax tags ˓→starred ------ ˓→-- 40 Chapter 1. SunPy User Guide DemoDoc Documentation, Release 1 2011-03-19 10:54:00 2011-03-19 10:54:00 AIA_3 17.1 17.1 spring No 26 2011-05-08 00:00:00 2011-05-08 00:00:01 AIA 17.1 17.1 spring No 27 2011-05-08 00:00:00 2011-05-08 00:00:01 AIA 21.1 21.1 spring Yes 28 2011-05-08 00:00:02 2011-05-08 00:00:03 AIA 9.4 9.4 spring No 29 2011-05-08 00:00:03 2011-05-08 00:00:04 AIA 33.5 33.5 spring Yes 32 2012-08-05 00:00:02 2012-08-05 00:00:03 AIA 33.5 33.5 N/A Yes 33 2012-08-05 00:00:02 2012-08-05 00:00:03 AIA 33.5 33.5 N/A Yes 36 2013-08-05 00:00:02 2013-08-05 00:00:03 AIA 33.5 33.5 N/A Yes 37 2013-08-05 00:00:02 2013-08-05 00:00:03 AIA 33.5 33.5 N/A Yes 4 2011-03-19 10:54:00 2011-03-19 10:54:00 AIA_3 17.1 17.1 spring No 5 2014-04-09 06:00:12 2014-04-09 06:00:12 AIA_3 17.1 17.1 spring No 8. Caching All entries that are saved in the database are also saved in a cache in-memory. The type of the cache is determined at the initialization of the database object and cannot be changed after that. The default type is caching.LRUCache (least-recently used) and the other one which is supported is caching.LFUCache (least-frequently used). Per default the cache size is float('inf'), i.e. infinite. To set the cache size after the database object has been initialized, use the method Database.set_cache_size(). If the new size is smaller than the current number of database entries, entries are removed according to the cache type until the number of entries is equal to the given cache size. The following call to Database.set_cache_size() sets the cache size to 10 and therefore removes the 5 entries that been used least recently. >>> database.set_cache_size(10) >>> print display_entries( ... database, ... ['id','observation_time_start','observation_time_end', ... 'instrument','wavemin','wavemax']) id observation_time_start observation_time_end instrument wavemin wavemax ------1 2011-03-19 10:54:00 2011-03-19 10:54:00 AIA_3 17.1 17.1 3 2013-09-21 16:00:06 2013-09-21 16:00:06 AIA_2 19.3 19.3 4 2011-03-19 10:54:00 2011-03-19 10:54:00 AIA_3 17.1 17.1 5 2014-04-09 06:00:12 2014-04-09 06:00:12 AIA_3 17.1 17.1 8 2002-06-25 10:00:10 2002-06-25 10:00:10 EIT 19.5 19.5 24 2012-10-30 15:30:01 2012-10-30 15:30:01 AIA_4 9.4 9.4 25 2012-01-01 00:16:07 2012-01-01 00:16:07 SWAP 17.4 17.4 33 2012-08-05 00:00:02 2012-08-05 00:00:03 AIA 33.5 33.5 36 2013-08-05 00:00:02 2013-08-05 00:00:03 AIA 33.5 33.5 37 2013-08-05 00:00:02 2013-08-05 00:00:03 AIA 33.5 33.5 Opening Files with SunPy SunPy has wrapped several libraries in order to make input and output as painless as possible. Below is a brief tour of the IO libraries wrapped for SunPy. 1. Wrapped Libraries Currently three IO libraries are wrapped for easy use with SunPy. 1. Astropy’s Fits 2. Glymur’s JPEG 2000 C library wrapper 1.3. Acquiring Data with SunPy 41 DemoDoc Documentation, Release 3. ANA C library Part of SunPys convenience is that these file types have general purpose high-level IO functions.: import sunpy.io as io io.read_file(filename) io.read_file_header(filename) io.write_file(filename) These are designed to take a filename and breakdown the name in order to automatically calculate the call required to either read or write this file. SunPy has a list of known file types which is compared against when using the high-level IO functions. When reading a data file, the function will return a list of (data, header) pairs depending on how HDUs exist in the file. It important to remember this. Further, you can force the filetype from this interface, like so: io.read_file(filename,filetype='filetype of your choice') Valid values for filetype are ‘fits’, ‘jp2’ and ‘ana’ This will work for the three high-level IO functions. Full documentation for compatible files is located here iofits, iojp2 and ioana. Data Types in SunPy Maps Maps in SunPy are are 2-dimensional data associated with a coordinate system. In this guide, we will cover some of the basic functionality of maps. Once you’ve read through this guide check out the SunPy map for a more thorough look at SunPy maps. There you can see what instruments are currently supported or you can access the code reference for each instrument-specific map subclass. Creating maps To make things easy, SunPy can download several example files which are used throughout the docs. These files have names like ~sunpy.data.sample.AIA_171_IMAGE and ~sunpy.data.sample.RHESSI_IMAGE. To create the sam- ple sunpy.map.sources.sdo.AIAMap type the following into your interactive Python shell: >>> import sunpy >>> import sunpy.map >>> import sunpy.data.sample >>> my_map= sunpy.map.Map(sunpy.data.sample.AIA_171_IMAGE) If you have not downloaded the data already you should get an error and some instruction on how to download the sample data. The variable my_map is a SunPy map object. To create one from a local FITS file try the following: >>> my_map= sunpy.map.Map('/mydirectory/mymap.fits') SunPy should automatically detects the type of file (e.g. FITS), what instrument it is associated with (e.g. AIA, EIT, LASCO) and will automatically look in the appropriate places for the FITS keywords it needs to interpret the coordinate system. If the type of FITS file is not recognized then SunPy will try some default FITS keywords and return a ~sunpy.map.GenericMap but results may vary. SunPy can also create maps from the jpg2000 files from helioviewer.org. 42 Chapter 1. SunPy User Guide DemoDoc Documentation, Release Creating Custom Maps It is also possible to create maps using custom data (e.g. from a simulation). To do this you need to provide ~sunpy.map.map_factory.MapFactory with both the data array as well as some basic meta information. If no header is given then some default values as assumed. Here is a simple example: >>> import numpy as np >>> data= np.arange(0,100).reshape(10,10) >>> header={'cdelt1': 10,'cdelt2': 10,'telescop':'sunpy'} >>> my_map= sunpy.map.Map(data, header) The keys in the header follows the FITS standard. Inspecting maps A map contains a number of data-associated attributes. To get a quick look at your map simply type: >>> my_map= sunpy.map.Map(sunpy.data.sample.AIA_171_IMAGE) >>> my_map SunPy AIAMap ------Observatory: SDO Instrument: AIA 3 Detector: AIA Measurement: 171.0 Angstrom Wavelength: 171.0 Angstrom Obs Date: 2011-03-19 10:54:00 dt: 1.999601 s Dimension: [ 1024. 1024.] pix scale: [ 2.4 2.4] arcsec / pix array([[ 0.3125, -0.0625, -0.125 , ..., 0.625 , -0.625 , 0. ], [ 1. , 0.1875, -0.8125, ..., 0.625 , -0.625 , 0. ], [-1.1875, 0.375 , -0.5 , ..., -0.125 , -0.625 , -1.1875], ..., [-0.625 , 0.0625, -0.3125, ..., 0.125 , 0.125 , 0.125 ], [ 0.5625, 0.0625, 0.5625, ..., -0.0625, -0.0625, 0. ], [ 0.5 , -0.125 , 0.4375, ..., 0.6875, 0.6875, 0.6875]]) This will show a representation of the data as well as some of its associated attributes. A number of other at- tributes are also available, for example the ~sunpy.map.GenericMap.date, ~sunpy.map.GenericMap.exposure_time, ~sunpy.map.GenericMap.center, ~sunpy.map.GenericMap.xrange, ~sunpy.map.GenericMap.yrange and others (see ~sunpy.map.GenericMap): >>> map_date= my_map.date >>> map_exptime= my_map.exposure_time >>> map_center= my_map.center >>> map_xrange= my_map.xrange >>> map_yrange= my_map.yrange To get a list of all of the attributes check the documentation by typing: >>> help(my_map) From SunPy version 0.6 many attributes return ~astropy.units.quantity.Quantity objects, please refer to the as- tropy.units. 1.4. Data Types in SunPy 43 DemoDoc Documentation, Release The meta data for the map is accessed by >>> header= my_map.meta This references the meta data dictionary with the header information as read from the source file. Getting at the data The data in a SunPy Map object is accessible through the ~sunpy.map.GenericMap.data attribute. The data is imple- mented as a NumPy ~numpy.ndarray, so for example, to get the 0th element in the array >>> my_map.data[0,0] 0.3125 >>> my_map.data[0][0] 0.3125 One important fact to remember is that the first index is for the y direction while the second index is for the x direction. For more information about indexing please refer to the Numpy documentation. Data attributes like ~numpy.ndarray.dtype and ~sunpy.map.GenericMap.dimensions are accessible through the Sun- PyGenericMap object >>> my_map.dimensions Pair(x= Here the dimensions attribute is similar to the ~numpy.ndarray.shape attribute, however returning an ~as- tropy.units.quantity.Quantity. If you’d like to use the data in a SunPy ~sunpy.map.GenericMap object elsewhere, you can use either of the following: >>> var= my_map.data >>> var= my_map.data.copy() Python makes use of pointers so if you want to alter the data and keep the original data in the map intact make sure to copy it. Some basic statistical functions on the data array are also passed through to Map objects: >>> my_map.min() -2.0 >>> my_map.max() 9429.125 >>> my_map.mean() 235.91531443595886 but you can also access all the other ~numpy.ndarray functions and attributes but accessing the data array directly. For example: >>> my_map.data.std() 292.43424704677756 Plotting As is true of all of the SunPy data objects, the SunPy ~sunpy.map.GenericMap object (and all of its instrument-specific sub-classes) has its own built-in plot methods so that it is easy to quickly view your map. To create a plot just type: 44 Chapter 1. SunPy User Guide DemoDoc Documentation, Release >>> my_map.peek() This will open a matplotlib plot on your screen. In addition, to enable users to modify the plot it is possible to grab the matplotlib axes object by using the ~sunpy.map.GenericMap.plot() command. This makes it possible to use the SunPy plot as the foundation for a more complicated figure. For a bit more information about this and some examples see Plotting in SunPy. Note: If the wcsaxes package is not installed the ~sunpy.map.GenericMap.plot() and ~sunpy.map.GenericMap.peek() methods assume that the data is not rotated, i.e. the solar y axis is oriented with the columns of the array. If this condition is not met, when the map is plotted a warning will be issued. You can create an oriented map by using ~sunpy.map.GenericMap.rotate() before you plot the Map. Plotting Keywords For Map ~matplotlib.pyplot.imshow does most of the heavy lifting in the background while SunPy makes a number of choices for you so that you don’t have to (e.g. colortable, plot title). Changing these defaults is made possible through two simple interfaces. You can pass any ~matplotlib.pyplot.imshow keyword into the plot command to override the defaults for that particular plot. The following plot changes the default AIA color table to use an inverse Grey color table. You can view or make changes to the default settings through the ~sunpy.map.GenericMap.plot_settings dictionary. In the following example we change the title of the plot by changing the ~sunpy.map.GenericMap.plot_settings property. Colormaps and Normalization Image data is generally shown in false color in order to better identify it or to better visualize structures in the image. Matplotlib handles this colormapping process through the ~matplotlib.colors module. This process involves two steps: the data array is first mapped onto the range 0-1 using an instance of ~matplotlib.colors.Normalize or a subclass; then this number is mapped to a color using an instance of a subclass of a ~matplotlib.colors.colormap. SunPy provides the colormaps for each mission as defined by the mission teams. The Map object chooses the ap- propriate colormap for you when it is created as long as it recognizes the instrument. To see what colormaps are available: >>> import sunpy.cm >>> sunpy.cm.cmlist.keys() ['sohoeit304', 'sdoaia211', 'sohoeit195', 'trace1600', 'sdoaia94', 'trace284', 'trace1216', 'sdoaia304', 'trace1700', 'yohkohsxtal', 'trace195', 'sdoaia335', 'sdoaia1600', 'traceWL', 'stereohi2', 'sdoaia193', 'stereohi1', 'rhessi', 'trace171', 'trace1550', 'sohoeit284', 'stereocor2', 'hmimag', 'stereocor1', 'sdoaia1700', 'yohkohsxtwh', 'sohoeit171', 'hinodexrt', 'sdoaia131', 'sdoaia171', 'hinodesotintensity', 'sdoaia4500', 'soholasco3', 'soholasco2'] The SunPy colormaps are registered with matplotlib so you can grab them like you would any other colormap: >>> import matplotlib.pyplot as plt >>> import sunpy.cm You need to import sunpy.cm or sunpy.map for this to work: 1.4. Data Types in SunPy 45 DemoDoc Documentation, Release >>> cmap= plt.get_cmap('sdoaia171') The following plot shows off all of the colormaps. These can be used with the standard commands to change the colormap. So for example if you wanted to plot an AIA image but use an EIT colormap, you would do so as follows. or you can just change the colormap for the map itself as follows: >>> smap.plot_settings['cmap']= plt.get_cmap('sohoeit171') The normalization is also set automatically and is chosen so that all the data from minimum to maximum is displayed as best as possible for most cases. This means that it is never necessary to touch the data such as applying a function such sqrt or log to the data to make your plot look good. There are many normalizations available from matplotlib such as ~matplotlib.colors.Lognorm. Other more exotic normalizations are also made available from Astropy. Just like the colormap the default normalization can be changed through the plot_settings dictionary or directly for the individual plot by passing a keyword argument. The following example shows the difference between a linear and logarithmic normalization on an AIA image. Note how the color in the colorbar does not change since these two maps share the same colormap while the data values associated with each color do because the normalization is different. Masking and Clipping Data It is often necessary for the purposes of display or otherwise to ignore certain data in an image. For example large data value could be due to cosmic ray hits and should be ignored. The most straightforward way to ignore this kind of data in plots without altering the data is to clip it. This can be achieved very easily when initializing the normalization variable. For example: >>> import matplotlib.colors as colors >>> norm= colors.Normalize(vmin=smap.min(), vmax=smap.mean()+3 *smap.std()) This clips out many of the brightest pixels. If you’d like to see what areas of your images got clipped set the following values: >>> cmap= cmap.plot_settings['cmap'] >>> cmap.set_over('red', 1.0) >>> cmap.set_under('green', 1.0) This will color the areas above and below in red and green respectively (similar to this example). You can use the following colorbar command to display these choices: >>> plt.colorbar(extend='both') Here is an example of this put to use on an AIA image. If you see how the image displays by default you’ll see that it does not look that pretty. This is because the image contains some negative values which are throwing off the normalization. In order to fix this we need to adjust our normalization to not display negative values. We can also brighten the image by clipping the high values though this will mean that the bright regions look ‘saturated’. This is achieved in the following plot. Another method to ignore bad data is to mask the data. A mask is a boolean array and so can give you much more fine-grained control over what is not being displayed. A ~numpy.ma.MaskedArray is a subclass of a numpy array so it has all of the same properties with the addition of an associated boolean array which holds the mask. 46 Chapter 1. SunPy User Guide DemoDoc Documentation, Release Composite Maps and Overlaying Maps The Map() method described above can also handle a list of maps. If a series of maps are supplied as inputs, Map() will return a list of maps as the output. However, if the ‘composite’ keyword is set to True, then a ~sunpy.map.CompositeMap object is returned. This is useful if the maps are of a different type (e.g. different in- struments). For example, to create a simple composite map: >>> my_maps= sunpy.map.Map(sunpy.data.sample.EIT_195_IMAGE, sunpy.data.sample.RHESSI_ ˓→IMAGE, composite=True) A ~sunpy.map.CompositeMap is different from a regular SunPy ~sunpy.map.GenericMap object and therefore differ- ent associated methods. To list which maps are part of your composite map use: >>> my_maps.list_maps() [ ˓→] The following code adds a new map (which must be instantiated first), sets its transparency to 25%, turns on contours from 50% to 90% for the second map, and then plots the result. This is not a particularly pretty plot but it shows what SunPy can do! Working with your map Part of the philosophy of the map object is to provide most of the basic functionality that a scientist would want therefore a map also contains a number of map-specific methods such as resizing a map or grabbing a subview. To get a list of the methods available for a map type: >>> help(my_map) and check out the methods section! Mapcubes A ~sunpy.map.MapCube is an ordered list of maps. By default, the maps are ordered by their observation date, from earlier maps to later maps. A ~sunpy.map.MapCube can be created by supplying multiple existing maps: >>> map1= sunpy.map.Map(sunpy.data.sample.AIA_171_IMAGE) >>> map2= sunpy.map.Map(sunpy.data.sample.EIT_195_IMAGE) >>> mc= sunpy.map.Map([map1, map2], cube= True) or by providing a directory full of image files: >>> mc= sunpy.map.Map('path/to/my/files/ *.fits', cube=True) The earliest map in the mapcube can be accessed by simply indexing the maps list: >>> mc.maps[0] Mapcubes can hold maps that have different shapes. To test if all the maps in a ~sunpy.map.MapCube have the same shape: >>> mc.all_maps_same_shape() True 1.4. Data Types in SunPy 47 DemoDoc Documentation, Release It is often useful to return the image data in a ~sunpy.map.MapCube as a single three dimensional Numpy ~numpy.ndarray: >>> mc.as_array() Note that an array is returned only if all the maps have the same shape. If this is not true, an error (ValueError) is returned. If all the maps have nx pixels in the x-direction, and ny pixels in the y-direction, and there are n maps in the mapcube, the ~numpy.ndarray array that is returned has shape (ny, nx, n). The data of the first map in the ~sunpy.map.MapCube appears in the ~numpy.ndarray in position [:, :, 0], the data of second map in po- sition [:, :, 1], and so on. The order of maps in the ~sunpy.map.MapCube is reproduced in the returned ~numpy.ndarray. The meta data from each map can be obtained using: >>> mc.all_meta() This returns a list of map meta objects that have the same order as the maps in the ~sunpy.map.MapCube. Coalignment of Mapcubes A typical data preparation step when dealing with time series of images is to coalign images taken at different times so that features in different images remain in the same place. A common approach to this problem is to take a representative template that contains the features you are interested in, and match that to your images. The location of the best match tells you where the template is in your image. The images are then shifted to the location of the best match. This aligns your images to the position of the features in your representative template. SunPy provides a function to coalign the maps inside the ~sunpy.map.MapCube. The implementation of this func- tionality requires the installation of the scikit-image library, a commonly used image processing library. To coalign a ~sunpy.map.MapCube, simply import the function and apply it to your ~sunpy.map.MapCube: >>> from sunpy.image.coalignment import mapcube_coalign_by_match_template >>> coaligned= mapcube_coalign_by_match_template(mc) This will return a new ~sunpy.map.MapCube, coaligned to a template extracted from the center of the first map in the ~sunpy.map.MapCube, with the map dimensions clipped as required. The coalignment algorithm provides many more options for handling the coalignment of ~sunpy.map.MapCube type: >>> help(mapcube_coalign_by_match_template) for a full list of options and functionality. If you just want to calculate the shifts required to compensate for solar rotation relative to the first map in the ~sunpy.map.MapCube without applying them, use: >>> from sunpy.image.coalignment import calculate_match_template_shift >>> shifts= calculate_match_template_shift(mc) This is the function used to calculate the shifts in ~sunpy.map.MapCube coalignment function above. Please see ~sunpy.image.coalignment.calculate_match_template_shift to learn more about its features. Shifts calculated using calculate_match_template_shift can be passed directly to the coalignment function. Compensating for solar rotation in Mapcubes Often a set of solar image data consists of fixing the pointing of a field of view for some time and observing. Features on the Sun will rotate according to the Sun’s rotation. 48 Chapter 1. SunPy User Guide DemoDoc Documentation, Release A typical data preparation step when dealing with time series of these types of images is to shift the images so that features do not appear to move across the field of view. This requires taking in to account the rotation of the Sun. The Sun rotates differentially, depending on latitude, with features at the equator moving faster than features at the poles. SunPy provides a function to shift images in ~sunpy.map.MapCube following solar rotation. This function shifts an image according to the solar differential rotation calculated at the latitude of the center of the field of view. The image is not differentially rotated. This function is useful for de-rotating images when the effects of differential rotation in the ~sunpy.map.MapCube can be ignored (for example, if the spatial extent of the image is small, or when the duration of the ~sunpy.map.MapCube is small; deciding on what ‘small’ means depends on your application). To apply this form of solar derotation to a ~sunpy.map.MapCube, simply import the function and apply it to your ~sunpy.map.MapCube: >>> from sunpy.physics.transforms.solar_rotation import mapcube_solar_derotate >>> derotated= mapcube_solar_derotate(mc) For more info see ~sunpy.physics.transforms.solar_rotation.mapcube_solar_derotate. If you just want to calculate the shifts required to compensate for solar rotation relative to the first map in the ~sunpy.map.MapCube without applying them, use: >>> from sunpy.physics.transforms.solar_rotation import calculate_solar_rotate_shift >>> shifts= calculate_solar_rotate_shift(mc) Please consult the docstring of the ~sunpy.image.coalignment.mapcube_coalign_by_match_template function in order to learn about the features of this function. Lightcurves Time series data are a fundamental part of many data analysis projects as much in heliophysics as other areas. SunPy therefore provides a lightcurve object to handle this type of data. Once you’ve read through this guide check out the SunPy lightcurve for a more thorough look at SunPy LightCurve and to see what data sources it currently supports. 1. Creating a Lightcurve from a data source To create a ~sunpy.lightcurve.LightCurve object from one of the supported data sources, import the object into your session. Unlike the ~sunpy.map.GenericMap object and its instrument-specific subclasses, the instrument sub-classes of ~sunpy.lightcurve.LightCurve provide a way to download their own data on creating. The following example creates a ~sunpy.lightcurve.GOESLightCurve for the specified time range: >>> from sunpy.lightcurve import GOESLightCurve >>> from sunpy.time import TimeRange >>> tr= TimeRange('2013/07/21','2013/07/22') >>> goes= GOESLightCurve.create(tr) The ~sunpy.lightcurve.GOESLightCurve will go off and download the data that is needed and therefore requires an internet connection. 2. Creating Custom Lightcurve It is also very easy to create lightcurves using custom data. >>> from sunpy.lightcurve import LightCurve >>> light_curve= LightCurve.create({"param1": range(24 *60)}) 1.4. Data Types in SunPy 49 DemoDoc Documentation, Release Within ~sunpy.lightcurve.LightCurve.create, we have a dictionary that contains a single entry with key param1 con- taining a list of 1440 entries (0-1439). As there are no times provided, so a default set of times are generated. A LightCurve object must be supplied with some data when it is created. The data can either be in your current Python session, in a local file, or in a remote file. Let’s create some fake data and pass it into a LightCurve object: >>> from sunpy.lightcurve import LightCurve >>> light_curve= LightCurve.create({"param1": range(24 * 60)}) The first line imports the lightcurve object. Let’s look at the argument in LightCurve.create. The argument is a dictionary that contains a single entry with key “param1” with a value of a list of 1440 entries (from 0 to 1439) - these are our ‘fake data’ measurements. Since no other times are provided, a default set of times are provided. You can provide your own times very simply using the ‘index’ keyword, as is shown below: >>> import datetime >>> base= datetime.datetime.today() >>> dates= [base- datetime.timedelta(minutes=x) for x in range(0, 24 * 60)] >>> light_curve= LightCurve.create({"param1": range(24 * 60)}, index=dates) This gives the measurements “param1” a set of times, in this case, 1440 minutes beginning at the current local time. Under the hood, this has created a pandas ~pandas.DataFrame object with a column name “param1”, with an index of times. 3. Inspecting maps & Getting at the data A lightcurve holds both data as well as meta data. The meta data for the lightcurve is accessed by >>> header= goes.meta This references the meta data dictionary with the header information as read from the source file. A word of caution, many data sources provide little to no meta data so this variable might be empty. The data in a SunPy ~sunpy.lightcurve.LightCurve object is accessible through the ~sunpy.map.LightCurve.data at- tribute. The data is implemented as a Pandas ~pandas.DataFrame, so to get a look at what data you have available >>> goes.data You can also get a quick overview of what data you have available like so: >>> goes.data.info() LightCurves are columnar data so to get at a particular datum you need to first index the column then the element you want. To get the names of the available columns: >>> goes.data.columns So you can access the 0th element in the column xrsa with: >>> goes.data['xrsa'][0] You can also grab all of the data at a particular time: >>> goes.data['xrsa']['2013-07-21 23:59'] This will return a list of entries with times that match the accuracy of the time you provide. Finally if you want to get at the x or y values: 50 Chapter 1. SunPy User Guide DemoDoc Documentation, Release >>> x= goes.data.index >>> y= goes.data.values You can read more about indexing at the pandas documentation website. 3. Plotting The SunPy LightCurve object has its own built-in plot methods so that it is easy to quickly view your lightcurve. To create a plot just type: This will open a matplotlib plot on your screen. The ~sunpy.lightcurve.LightCurve.peek() function provides a custom view on the data while ~sunpy.lightcurve.LightCurve.plot() provides a more generic plot. In addition, to enable users to modify the plot it is possible to grab the matplotlib axes object by using the ~sunpy.lightcurve.LightCurve.plot() command. This makes it possible to use the SunPy plot as the foundation for a more complicated figure. For a bit more information about this and some examples see Plotting in SunPy. Here is one a more complicated example which makes use of this methodology. Here is another more advanced example. Click the source link to see the that generated this plot. Spectra Warning: This module is under development! Use at your own risk. Spectrograms SunPy currently supports reading dynamic spectra from e-Callisto instruments. The main class that is used for this is CallistoSpectrogram. SunPy also comes with an example image that shows a radio burst observed at Rosse Observatory (aka. BIR; Birr Castle, Co. Offaly, Ireland) that can be found in sunpy.data.sample.CALLISTO_IMAGE. We now notice that there seems to be something interesting that has been cut off at the corner of the image, so we use the extend method to request more data from the server. It optionally takes the amount of minutes we want to request from the server (negative values mean we want to add data that was registered before our existing local data), if none are given it defaults to 15 minutes (the size of one e-Callisto file). more= image.extend() more.peek() We will, for the purposes of this demonstration, continue working with the original image, though. You can then perform automatic constant background subtraction by using the subtract_bg() method. The re- sulting image will be clipped at 0 using the min parameter of peek in order to avoid negative values. If you want to see the background determined by the automatic subtraction, you can use the auto_const_bg() method and visualize the resulting data using pyplot.plot().: plt.figure() bg= image.auto_const_bg() plt.plot(image.freq_axis, bg) plt.xlabel("Frequency [MHz]") plt.ylabel("Intensity") plt.show() 1.4. Data Types in SunPy 51 DemoDoc Documentation, Release Now let us say we want to isolate the interesting bit (which starts around 10:38) from the boring background; there is a method called in_interval() that allows us to take the part of an image that is within a specified interval. Leaving out the second argument it defaults to the end time of the file. To get rid of the noise, we could also clip low intensities by setting vmin. If we want more context, we can also join together different images into a large one in time (note that this does more than just concatenating the array and the axes – it also considers possible overlap or gaps).: c1= CallistoSpectrogram.read('BIR_20110922_101500_01.fit') c2= CallistoSpectrogram.read('BIR_20110922_103000_01.fit') d= CallistoSpectrogram.join_many([c1, c2]) We could also get the from_range method to get data between those two points directly from the archive and joined together (though that will fetch all frequencies of BIR): from sunpy.spectra.sources.callisto import CallistoSpectrogram d= CallistoSpectrogram.from_range('BIR','2011-09-22 10:15:00','2011-09-22 10:45:00 ˓→') Plotting in SunPy SunPy makes use of matplotlib for all of its plotting needs as such it tries to follow the matplotlib plotting philosophy. It is therefore useful to go over how matplotlib works as background. 1. Matplotlib Tutorial The tutorial provided here is a summary of one that can be found in the matplotlib usage documentation. Matplotlib provides two main pathways for plotting. One is meant for interactive use (e.g. command-line) and the other for non-interactive use (e.g. modules). It is important to recognize though that the interactive-use pathway (referred to as pyplot) just provides shortcuts for doing many of the more advanced non-interactive functions in the background. It is therefore possible to switch between the two as necessary and it is possible to use pyplot in a non- interactive way. In this manner pyplot is just a shortcut to making it quicker to set up plot axes and figures. In order to get access to the full interactive capabilities of pyplot it is necessary to turn this feature on. Pylab is another matplotlib usage scenario but it is essentially just pyplot with the interactive capabilities turned on and numpy and matplotlib imported into the main namespace. 2. Pyplot Here is a simple example of pyplot usage. The ~matplotlib.pyplot.show command opens a plot on the screen and blocks execution until the plot window is closed. The ~matplotlib.pyplot.show command only works once. If you were to call ~matplotlib.pyplot.show again after the above code is executed nothing happens. This confusing behavior is something that the matplotlib devs get complaints about often and so this may change. A discussion about this can be found here. Don’t be confused by another command called ~matplotlib.pyplot.draw. This is only used while in interactive mode. To turn on interactivity for pyplot use the command >>> plt.ion() In interactive mode, the plot will appear at the first ~matplotlib.pyplot.plot command and most commands will update the plot as you call them. Here is some example code: 52 Chapter 1. SunPy User Guide DemoDoc Documentation, Release >>> plt.plot(range(10), range(10)) >>> plt.title("Simple Plot") In this example, you’ll see that the title appears right on the plot when you call it. Note that in this case the ~mat- plotlib.pyplot.show command is useless as the plot shows up right when you create it. Also note that some commands will not automatically update the plot and you have to use the ~matplotlib.pyplot.draw command. The following command >>> plt.ioff() turns off interactivity. 3. Advanced Pyplot If you need more fine-grained control over plots the recommended path is to use pyplot and access the figures and axes objects. This is shown in the following example. In matplotlib, ~matplotlib.figure.Figure is the top-level container for all plot elements and ~matplotlib.axes.Axes is the top-level container for a particular plot. So the above example, creates a figure then creates an axes and populates the plot in ax. With this method you now have your hands on the ~matplotlib.axes.Axes object so you can do things like change the labels on the x and y axes or add a legend. In the previous section, pyplot took care of creating these objects for you so you didn’t have to worry about creating them yourself. 4. SunPy Plotting Standards To be consistent with matplotlib, SunPy has developed a standard plotting policy which supports both simple and advanced matplotlib usage. The following examples focus on the map object but they should be applicable across all of the data objects. 4.1 peek() For quick and easy access to a plot all SunPy base objects (e.g. maps, spectra, lightcurves) define their own ~sunpy.map.mapbase.GenericMap.peek command which will create a plot for you and show it without you having to deal with any matplotlib setup. This is so that it is easy to take a quick look at your data. For example you can make the following plot. This creates a plot window with all axes defined, a plot title, and the image of the map data defined by the contents of the map. In non-interactive mode the plot window blocks the command line terminal and must be closed before doing anything else. 4.2 plot() For more advanced plotting the base SunPy objects also provide a ~sunpy.map.mapbase.GenericMap.plot command. This command is similar to the pyplot ~matplotlib.pyplot.imshow command in that it will create a figure and axes object for you if you haven’t already. When you create a plot with ~sunpy.map.GenericMap.peek or ~sunpy.map.GenericMap.plot, if possible SunPy will use wcsaxes to represent coordinates on the image accurately, for more information see Plotting Maps with wcsaxes. Using ~sunpy.map.GenericMap.plot it is possible to customise the look of the plot by combining SunPy and matplotlib commands, for example you can over plot contours on the Map: 1.5. Plotting in SunPy 53 DemoDoc Documentation, Release In this example, the ~matplotlib.figure.Figure and ~wcsaxes.WCSAxes instances are created explicitly, and then used to modify the plot: It is possible to create the same plot, explicitly not using wcsaxes, however, this will not have the features of wcsaxes which include correct representation of rotation and plotting in different coordinate systems. Plotting Maps with wcsaxes By default SunPy map checks if the wcsaxes package has been installed. If it is installed, then wc- saxes is used to improve the representation of world coordinates, and calling ~sunpy.map.GenericMap.plot or ~sunpy.map.GenericMap.peek() will use wcsaxes for plotting. Unless a standard matplotlib.axes.Axes object is cre- ated. To explicitly create a wcsaxes.WCSAxes instance do the following >>> fig= plt.figure() >>> ax= plt.subplot(projection=smap) when plotting on a ~wcsaxes.WCSAxes axes, it will by default plot in pixel coordinates, you can override this behavior and plot in ‘world’ coordinates by getting the transformation from the axes with ax.get_transform('world'). Note: World coordinates are always in degrees so you will have to convert to degrees.: >>> smap.plot() >>> ax.plot((100*u.arcsec).to(u.deg), (500*u.arcsec).to(u.deg), ... transform=ax.get_transform('world')) Finally, here is a more complex example using SunPy maps, wcsaxes and Astropy units to plot a AIA image and a zoomed in view of an active region. Time in SunPy Working with times and time ranges is a standard task in solar data analysis as such SunPy strives to provide con- venient and easy methods to do the simple stuff. Python already provides an object for a time or date through date- time.datetime. SunPy builds upon its functionality. 1. Parsing Times Solar data is associated with a number of different time formats. SunPy provides a simple parsing function which can deal with most every format that a user may encounter. Called sunpy.time.parse_time(), this function takes a string as input and returns a datetime object. Here are few examples of formats which sunpy.time.parse_time() accepts: >>> from sunpy.time import parse_time >>> parse_time('2007-05-04T21:08:12') >>> parse_time('2007/05/04T21:08:12') >>> parse_time('20070504T210812') >>> parse_time('2007-May-04 21:08:12') >>> parse_time('20070504_210812') datetime.datetime(2007, 5, 4, 21, 8, 12) Each of the above returns the same datetime object datetime.datetime(2007, 5, 4, 21, 8, 12). One of the most standard time formats used in solar physics is the number of seconds since 1979 January 01. The parse_time function also accepts this as input, e.g.: 54 Chapter 1. SunPy User Guide DemoDoc Documentation, Release >>> parse_time(894316092.00000000) datetime.datetime(2007, 5, 4, 21, 8, 12) All SunPy functions which require time as an input sanitize the input using parse_time. 2. Time Ranges A very standard task in data analysis is to have to deal with pairs of times or time ranges. This occurs very often with plotting or when searching for data. To deal with time ranges SunPy provides the sunpy.time.TimeRange object. A TimeRange object can be created very easily by providing it with two time strings, a start time and an end time: >>> from sunpy.time import TimeRange >>> time_range= TimeRange('2010/03/04 00:10','2010/03/04 00:20') You can also pass the start and end times as a tuple: >>> time_range= TimeRange(('2010/03/04 00:10','2010/03/04 00:20')) This object makes use of parse_time() so it can accept a wide variety of time formats. A time range object can also be created by providing a start time and a duration. The duration must be provided as a datetime.timedelta object or time-equivalent astropy.units.Quantity example: >>> import astropy.units asu >>> time_range= TimeRange('2010/03/04 00:10', 400 * u.second) or: >>> from datetime import timedelta >>> time_range= TimeRange('2010/03/04 00:10', timedelta(0, 400)) The time range objects provides a number of useful functions. For example, you can easily get the time at the center of your interval or the length of your interval in minutes or days or seconds: >>> time_range.center datetime.datetime(2010, 3, 4, 0, 13, 20) >>> time_range.minutes It also makes it easy to create new time ranges. The functions next() and previous() do an inplace update to the object by either adding or subtracting the same time interval . This could be useful if you need to step through a number of time ranges. For example, if you needed time ranges that spanned 30 minutes over a period of 4 hours you could do: >>> for a in range(8): ... print(time_range.next()) Start: 2010-03-04 00:16:40 End: 2010-03-04 00:23:20 Center:2010-03-04 00:20:00 Duration:0.00462962962963 days or 0.111111111111 hours or 6.66666666667 minutes or 400.0 seconds 1.6. Time in SunPy 55 DemoDoc Documentation, Release Start: 2010-03-04 00:23:20 End: 2010-03-04 00:30:00 Center:2010-03-04 00:26:40 Duration:0.00462962962963 days or 0.111111111111 hours or 6.66666666667 minutes or 400.0 seconds Start: 2010-03-04 00:30:00 End: 2010-03-04 00:36:40 Center:2010-03-04 00:33:20 Duration:0.00462962962963 days or 0.111111111111 hours or 6.66666666667 minutes or 400.0 seconds Start: 2010-03-04 00:36:40 End: 2010-03-04 00:43:20 Center:2010-03-04 00:40:00 Duration:0.00462962962963 days or 0.111111111111 hours or 6.66666666667 minutes or 400.0 seconds Start: 2010-03-04 00:43:20 End: 2010-03-04 00:50:00 Center:2010-03-04 00:46:40 Duration:0.00462962962963 days or 0.111111111111 hours or 6.66666666667 minutes or 400.0 seconds Start: 2010-03-04 00:50:00 End: 2010-03-04 00:56:40 Center:2010-03-04 00:53:20 Duration:0.00462962962963 days or 0.111111111111 hours or 6.66666666667 minutes or 400.0 seconds Start: 2010-03-04 00:56:40 End: 2010-03-04 01:03:20 Center:2010-03-04 01:00:00 Duration:0.00462962962963 days or 0.111111111111 hours or 6.66666666667 minutes or 400.0 seconds Start: 2010-03-04 01:03:20 End: 2010-03-04 01:10:00 Center:2010-03-04 01:06:40 Duration:0.00462962962963 days or 0.111111111111 hours or 6.66666666667 minutes or 400.0 seconds A time range can also be easily split into sub-intervals of equal length, for example to split a TimeRange object into two new TimeRange objects: 56 Chapter 1. SunPy User Guide DemoDoc Documentation, Release time_range.split(2) Check out the code reference for the sunpy.time.TimeRange object for more information. Region of Interest Warning: This module is under development. An region of interest (ROI) is an object that contains some basic information about a particular time range in the form of string descriptors. For example, an ROI might denote an interval of troublesome instrument data, such as an encounter with the South Antarctic Anomaly (SAA). 1. Creating an ROI You can create an ROI object with the following: >>> from sunpy.roi import * >>> result= roi(times=['2011-02-15 04:34:09','2011-02-15 04:48:21'],description='UV ˓→occult.',source='LYRA LYTAF') This creates an roi called result for the specific time range. Querying the newly created ROI gives the following: >>> result SunPy Region-of-interest (ROI) object ------Source: LYRA LYTAF Start time: 2011-02-15T04:34:09 End time: 2011-02-15T04:48:21 Event description: UV occult. Check out the code reference for the time range object for more information. Customizing SunPy The sunpyrc file Sunpy uses a sunpyrc configuration file to customize certain properties. You can control a number of key features of SunPy such as where your data will download to. SunPy looks for sunpyrc in two locations, in the following order: 1. .sunpy/sunpyrc, for the user’s default customizations. 2. INSTALL/sunpy/data/sunpyrc. You can find where SunPy is installed by printing out sunpy.__file__ after importing SunPy. To display where the currently active sunpyrc file was loaded from, one can do the following: >>> import sunpy >>> sunpy.print_config() 1.7. Region of Interest 57 DemoDoc Documentation, Release To maintain your own customizations place a copy of the default sunpyrc file into .sunpy. Do not edit the default file directly as every time you install or update SunPy, this file will be overwritten. See below for the example config file. Dynamic settings You can also dynamically change the default settings in a python script or interactively from the python shell. All of the settings are stored in a Python ConfigParser instance called sunpy.config, which is global to the sunpy package. Settings can be modified directly, for example: import sunpy sunpy.config.set('downloads','download_dir','/home/user/Downloads') A sample sunpyrc file Troubleshooting Contents • Troubleshooting – Crotate Warning – Obtaining sunpy version – System Info – sunpy install location – .sunpy directory location – Report a problem Crotate Warning The SunPy map class has a custom rotate functionality, similar to IDL’s ROT function. This uses a Python C-API extension which should be compiled by installing sunpy. If for any reason this build process fails, you will not be able to use the C-API rotate code, but will be able to still use all the functionality of map. If this happens you will encounter the following warning upon using the rotate method >>> rot_map= mymap.rotate(10) sunpy/map/map.py:829: Warning: The C extension sunpy.image.Crotate is not installed, ˓→falling back to the interpolation='spline' of order=3 warnings.warn("The C extension sunpy.image.Crotate is not installed, falling back ˓→to the interpolation='spline' of order=3" ,Warning) What happens is, because the C-API extension is not found, the rotate() function defaults to the spline interpolation method of order 3 which is implemented in scipy. To fix the C-API you should try and reinstall SunPy, if this still fails please ask the mailing list for assistance. 58 Chapter 1. SunPy User Guide DemoDoc Documentation, Release Obtaining sunpy version To find out your sunpy version number, import it and print the __version__ attribute: >>> import sunpy >>> sunpy.__version__ System Info To quickly collect information on your system, you can use our convenience function system_info which you can run through: >>> import sunpy >>> sunpy.util.system_info() The output should look something like: ======SunPy Installation Information Sunday, 18. November 2012 11:06PM UT ====== ########### General ########### OS: Mac OS X 10.8.2 (i386) Python: 2.7.3(64bit) #################### Required libraries #################### SunPy: 0.1 NumPy: 1.6.2 SciPy: 0.10.1 Matplotlib: 1.2.x PyFITS: 3.0.8 pandas: 0.8.1 ####################### Recommended libraries ####################### beautifulsoup4: 4.1.1 PyQt: 4.9.4 SUDS: 0.4' This information is especially useful if you are running into a bug and need help. sunpy install location You can find what directory sunpy is installed in by importing it and printing the __file__ attribute: >>> import sunpy >>> sunpy.__file__ 1.9. Troubleshooting 59 DemoDoc Documentation, Release .sunpy directory location Each user should have a .sunpy/ directory which should contain a sunpyrc file. To locate your .sunpy/ directory, use sunpy.print_config(): >>> import sunpy as sun >>> sun.print_config() On unix-like systems, this directory is generally located in your HOME directory. On windows, it is in your documents and settings directory by default. If you would like to use a different configuration directory, you can do so by specifying the location in your SUNPY_CONFIGDIR environment variable. Report a problem If you are having a problem with sunpy, search the mailing lists first: it is possible that someone else has already run into your problem. If not, please provide the following information in your e-mail to the mailing list: • your operating system; (Linux/UNIX users: post the output of uname -a) • sunpy version: >>> import sunpy >>> sunpy.util.system_info() • how you obtained sunpy. • any customizations to your sunpyrc file (see Customizing SunPy). • Please try to provide a minimal, standalone Python script that demonstrates the problem. This is the critical step. If you can’t post a piece of code that we can run and reproduce your error, the chances of getting help are significantly diminished. Very often, the mere act of trying to minimize your code to the smallest bit that produces the error will help you find a bug in your code that is causing the problem. You will likely get a faster response writing to the mailing list than filing a bug in the bug tracker. If your problem has been determined to be a bug and can not be quickly solved, the issues may be filed a bug in the tracker so the issue doesn’t get lost. 60 Chapter 1. SunPy User Guide CHAPTER 2 API Reference SunPy SunPy cm SunPy coordinates The SunPy coordinates submodule is an implementation of the common solar physics coordinate frames using the Astropy coordinates framework. Warning: The accuracy of the transformations in this module have not been rigorously verified. They have been compared to sunpy.wcs and match to numerical precision. Independent verification will be added at a later date. Getting Started The easiest interface to the coordinates module is through the ~astropy.coordinates.SkyCoord class: >>> import astropy.units asu >>> from astropy.coordinates import SkyCoord >>> import sunpy.coordinates >>> c= SkyCoord(-100 *u.arcsec, 500*u.arcsec, frame='helioprojective') >>> c= SkyCoord(x=-72241.0 *u.km, y=361206.1*u.km, z=589951.4*u.km, frame= ˓→'heliocentric') >>> c= SkyCoord(70 *u.deg,-30 *u.deg, frame='heliographic_stonyhurst') >>> c SunPy implements support for the following solar physics coordinate systems: 61 DemoDoc Documentation, Release • Helioprojective (Cartesian) ~sunpy.coordinates.frames.HelioProjective • Heliocentric ~sunpy.coordinates.frames.HelioCentric • Heliographic Stonyhurst ~sunpy.coordinates.frames.HelioGraphicStonyhurst • Heliographic Carrington ~sunpy.coordinates.frames.HelioGraphicCarrington for a complete description of these frames see sunpy.coordinates.frames, for a more detailed description of the frames see Thompson (2006) ~astropy.coordinates.SkyCoord and all other ~astropy.coordinates objects also support array coordinates. These work the same as single-value coordinates, but they store multiple coordinates in a single object. When you’re going to apply the same operation to many different coordinates, this is a better choice than a list of ~astropy.coordinates.SkyCoord objects, because it will be much faster than applying the operation to each ~astropy.coordinates.SkyCoord in a for loop. >>> c= SkyCoord([-500, 400] *u.arcsec, [100, 200]*u.arcsec, frame='helioprojective') >>> c ˓→rsun=695508.0 km): (Tx, Ty) in arcsec [(-500.0, 100.0), (400.0, 200.0)]> >>> c[0] ˓→rsun=695508.0 km): (Tx, Ty) in arcsec (-500.0, 100.0)> Accessing Coordinates Individual coordinates can be accessed via attributes on the SkyCoord object, but the names of the components of the coordinates for each frame differ. For a full description of all the properties of the frames see sunpy.coordinates.frames. HelioProjective For the helioprojective frame the coordinates are access as Tx and Ty representing theta x and y. These are the same coordinates that are often referred to as ‘solar-x’ and ‘solar-y’. >>> c= SkyCoord(-500 *u.arcsec, 100*u.arcsec, frame='helioprojective') >>> c.Tx Heliocentric Heliocentric normally a Cartesian frame so the coordinates are accessed as x,y,z: >>> c= SkyCoord(-72241.0 *u.km, 361206.1*u.km, 589951.4*u.km, frame='heliocentric') >>> c.x 62 Chapter 2. API Reference DemoDoc Documentation, Release HeliographicStonyhurst and HeliographicCarrington Both the heliographic frames use latitude, longitude and radius which are accessed as follows: >>> c= SkyCoord(70 *u.deg,-30 *u.deg, frame='heliographic_stonyhurst') >>> c.lat Design of the Coordinates Module This module works by defining a collection of Frames (sunpy.coordinates.frames), which exists on a transformation graph, where the transformations between the coordinate frames are then defined and registered with the transforma- tion graph (sunpy.coordinates.transformations). Currently, the SunPy frames are not transformable to the frames in Astropy, as there is no transformation defined between the two sets of frames. Positions within these Frames are stored as a Representation of a coordinate, a representation being a descrip- tion of a point in a Cartesian, spherical or cylindrical system (sunpy.coordinates.representation). A frame that contains a representation of one or many points is said to have been ‘realized’. For a more in depth look at the design and concepts of the Astropy coordinates system see astropy-coordinates- overview Frames and SkyCoord The ~astropy.coordinates.SkyCoord class is a high level wrapper around the astropy.coordinates package. It provides an easier way to create and transform coordinates, by using string representations for frames rather than the classes themselves and some other usability improvements, for more information see the ~astropy.coordinates.SkyCoord doc- umentation. The main advantage provided by ~astropy.coordinates.SkyCoord is the support it provides for caching Frame at- tributes. Frame attributes are extra data specified with a frame, some examples in sunpy.coordinates are dateobs or L0 and B0 for observer location. Only the frames where this data is meaningful have these attributes, i.e. only the Helioprojective frames have L0 and B0. However, when you transform into another frame and then back to a projective frame using SkyCoord it will remember the attributes previously provided, and repopulate the final frame with them. If you were to do transformations using the Frames alone this would not happen. The most important implication for this in sunpy.coordinates is the rsun parameter in the projective frames. If you create a projective frame with a rsun attribute, if you convert back to a projective frame it will be set correctly. It should also be noted that, if you create a Heliographic frame and then transform to a projective frame with an rsun attribute, it will not match the radius coordinate in the Heliographic frame. This is because you may mean to be describing a point above the defined ‘surface’ of the Sun. Coordinates and WCS The sunpy.coordinates package provides a mapping between FITS-WCS CTYPE convention and the coordinate frames as defined in sunpy.coordinates. This is used via the astropy.wcs.utils.wcs_to_celestial_frame function, with which the SunPy frames are registered upon being imported. This list is used by packages such as wcsaxes to convert from astropy.wcs.WCS objects to coordinate frames. 2.3. SunPy coordinates 63 DemoDoc Documentation, Release The sunpy.map.GenricMap class creates astropy.wcs.WCS objects as amap.wcs, however, it adds some ex- tra attributes to the ~astropy.wcs.WCS object to be able to fully specify the coordinate frame. It adds heliographic_longitude, heliographic_latitude and dsun. If you want to obtain a un-realized coordinate frame corresponding to a ~sunpy.map.GenericMap object you can do the following: >>> from astropy.wcs.utils import wcs_to_celestial_frame >>> import sunpy.coordinates >>> import sunpy.map >>> from sunpy.data.sample import AIA_171_IMAGE >>> amap= sunpy.map.Map(AIA_171_IMAGE) >>> wcs_to_celestial_frame(amap.wcs) ˓→L0=0.0 deg, B0=-7.064078 deg, rsun=695508.0 km)> sunpy.coordinates Package Attribution Some of this documentation was adapted from Astropy under the terms of the BSD License. This package was developed by Pritish Chakraborty as part of GSOC 2014 and Stuart Mumford. SunPy database Submodules SunPy image SunPy instr sunpy.instr.lyra Module Functions sunpy.instr.rhessi Module 64 Chapter 2. API Reference DemoDoc Documentation, Release Functions SunPy io File Readers SunPy lightcurve Overview One of core classes in SunPy is a LightCurve or timeseries. A number of instruments are supported through subclasses of the base ~sunpy.lightcurve.LightCurve class. To see Instrument LightCurve Classes for a list of all of them. Creating a LightCurve LightCurves can either be creating manually or automatically by downloading their own data (the most common case). Too create a custom ~sunpy.lightcurve.LightCurve see the example in the class documentation below. Subclasses of ~sunpy.lightcurve.LightCurve for specific instrument provide their own methods for downloading their data. For more information see Instrument LightCurve Classes. Instrument LightCurve Classes The generic method to create an instrument-specific LightCurve find the instrument subclass of interest and follow the following example: >>> from sunpy.lightcurve import GOESLightCurve >>> from sunpy.time import TimeRange >>> tr= TimeRange('2013/07/21','2013/07/22') >>> goes= GOESLightCurve.create(tr) The ~sunpy.lightcurve.LightCurve.create method will go off and download the data needed to populate the instance. The following instrument classes are supported. SunPy map Overview One of core classes in SunPy is a Map. A SunPy Map object is simply a spatially-aware data array, often an image. In order to make it easy to work with image data in SunPy, the Map object provides a number of methods for commonly performed operations. 2D map objects are subclasses of ~sunpy.map.MapBase and all Map objects are created using the Map factory ~sunpy.map.Map. A number of instrument are supported by subclassing this base object. See Instrument Map Classes to see a list of all of them. More complex subclasses are also available. See Map Classes. 2.7. SunPy io 65 DemoDoc Documentation, Release Creating Map Objects SunPy Map objects are constructed using the special factory class ~sunpy.map.Map: x= sunpy.map.Map('file.fits') The result of a call to ~sunpy.map.Map will be either a ~sunpy.map.mapbase.GenericMap object, or a subclass of ~sunpy.map.mapbase.GenericMap which either deals with a specific type of data, e.g. ~sunpy.map.sources.sdo.AIAMap or ~sunpy.map.sources.soho.LASCOMap (see Map Classes to see a list of all of them), or if no instrument matches, a 2D map ~sunpy.map.mapbase.GenericMap. Using Map Objects Once a map object has been created using ~sunpy.map.Map it will be a instance or a subclass of the ~sunpy.map.mapbase.GenericMap class. Irrespective of the instrument the map is constructed for, all maps behave the same and are interchangeable with one another. It is possible to manipulate the map or access meta data about the map from the methods and properties of the map class. The following documentation of ~sunpy.map.mapbase.GenericMap lists the attributes and methods that are available on all Map objects. Map Classes Defined in sunpy.map.sources are a set of ~sunpy.map.GenericMap subclasses which convert the specific metadata and other differences in each instruments data to the standard ~sunpy.map.GenericMap interface. These ‘sources’ also define things like the colormap and default normalisation for each instrument. These subclasses also provide a method, which describes to the Map factory which data and metadata pairs match its instrument. Instrument Map Classes Writing a new Instrument Map Class Any subclass of ~sunpy.map.GenericMap which defines a method named ~sunpy.map.GenericMap.is_datasource_for will automatically be registered with the Map factory. The is_datasource_for method describes the form of the data and metadata for which the ~sunpy.map.GenericMap subclass is valid. For example it might check the value of the INSTRUMENT key in the metadata dictionary. This makes it straightforward to define your own ~sunpy.map.GenericMap subclass for a new instrument or a custom data source like simulated data. These classes only have to be imported for this to work, as demonstrated by the following example. import sunpy.map class FutureMap(sunpy.map.GenericMap): def __init__(self, data, header, **kwargs): super(FutureMap, self).__init__(data, header, **kwargs) # Any Future Instrument specific keyword manipulation # Specify a classmethod that determines if the data-header pair matches # the new instrument @classmethod def is_datasource_for(cls, data, header, **kwargs): """Determines if header corresponds to an AIA image""" return header.get('instrume','').startswith('FUTURESCOPE') 66 Chapter 2. API Reference DemoDoc Documentation, Release This class will now be available through the Map factory as long as this class has been defined, i.e. imported into the current session. If you do not want to create a method named is_datasource_for you can manually register your class and matching method using the following method import sunpy.map sunpy.map.Map.register(FutureMap, FutureMap.some_matching_method) SunPy net SunPy physics SunPy roi SunPy spectra Overview One of core classes in SunPy is a Spectrum and Spectrogram. These representing a spectrum as a specific point in time or one as a function of time, respectively. Individual instruments are supported through subclasses. To see what instrument are supported see Spectrum Classes and Spectrogram Classes. Warning: This module is under development! Use at your own risk. Spectrum Spectrum Classes None yet. Spectrogram Spectrogram Classes There are a series of subclasses which are specialised for each instrument. 2.10. SunPy net 67 DemoDoc Documentation, Release SunPy sun SunPy time SunPy util SunPy visualization SunPy wcs 68 Chapter 2. API Reference CHAPTER 3 Developer’s Guide Developer’s Guide Overview This article describes the guidelines to be followed by developers working on SunPy. You if you are thinking of contributing to SunPy please read the following carefully. Version Control Source-code for SunPy is managed using Git, a Distributed Version Control system. Code branches are hosted on GitHub.com, a free project hosting website for Open-Source software. Creating Your Own Repo Overview Each person contributing to SunPy should create their own code repository on GitHub by forking the master repository or repo. All development is then done on that fork, using topic branches to isolate work on different features. New contributors can then initiate pull requests to have their code incorporated into the SunPy master repository. Regular contributors can become members of the SunPy team on GitHub. Code will be reviewed by regular contributors and comments will usually be provided before code is accepted. Getting Started Creating your own repo on GitHub is easy to do. If you followed the SunPy installation instructions you should already have git installed. Go ahead and create a free account on create an account on GitHub. Github has some great resources to help. Here is a quick overview of the process. Adding an SSH key to GitHub Next, you need to tell GitHub who you are. In order to push any code to GitHub you need to create a public SSH key and associate it with your GitHub account. For instructions on how this is done, see the article on GitHub on Setting up git under “Set Up SSH Keys”. You only need to do this once, although if you plan to work from multiple computers 69 DemoDoc Documentation, Release you will need to go through the process for each computer you wish to work on. Once you have created your account and associated a public SSH key it, you are ready to go. Using HTTPS If you do not fancy using SSH you can access GitHub using HTTP/HTTPS. A few things to note. Using HTTP only allows cloning of public repositories, while HTTPS allows cloning of private repositories but also allows you to have push access. This way you can type in your username and password to access your repositories. Identifying yourself Begin by identifying yourself to git (so all of your commits have this information) and logging in to GitHub: git config-- global user.name"Firstname Lastname" git config-- global user.email"[email protected]" Forking SunPy Each contributor to SunPy has their own copy of the SunPy master repo. When working on the code, changes are made to this copied repo, and only when the changes are completed, and have been verified to work, are they pull requested back to the upstream repo. GitHub provides a simple mechanism to setup your own personal repo by providing an option to fork a repository. When you create a fork of a GitHub project, a copy of the repo will automatically be created for you, and a link will be provided which you can use to download the code to your machine and begin working on it. To begin, fork the main SunPy repo on GitHub by clicking on the Fork button on the SunPy project page Next, you need to download the forked repository. Clone the fork to your local machine, edit and run: git clone [email protected]:your_username/sunpy.git or: git clone http://github.com/sunpy/sunpy.git By default your fork of the repo on GitHub is identified by the name origin. In order to keep the fork up to date with the main repo, it is useful to add it as a remote in git: git remote add upstream https://github.com/sunpy/sunpy.git To stay up to date you can grab the latest changes to the SunPy master using the commands: git pull upstream master This will merge the upstream code automatically with your code so you don’t need to worry about it overwriting your changes. After running either of these commands, your local copy of your personal repo is just a copy of the main repo. This is the same procedure that you will use in the future to keep yourself synchronised with the main repo. To make sure everything is setup correctly, let’s make some changes to our personal local repo and push those to our personal repo on GitHub. Go ahead and modify one of the files, or create a new file (and then run git add). Commit and push the changes to GitHub: git commit-a-m"My first commit" git push You local repo is now synced with GitHub and ahead of the main repo as it contains your personal contribution. Remember to commit after you’ve done a unit of work (i.e. often). This will make it easier for you (in the future) and everyone else to understand what you are doing. Also make sure to make your commit statements clear and understandable. 70 Chapter 3. Developer’s Guide DemoDoc Documentation, Release Installing SunPy In order to use the version of SunPy located in your personal repository. You need to install it using the setup.py script located in the top-level folder. The setup.py script has several flags: :: develop : Installs SunPy and builds all external libraries. build or build_ext: (Re)Builds the external libraries. clean –all: Cleans all build files Use the setup.py script like so: sudo python setup.py develop If you are interested in having different versions of sunpy in your machine and you want to switch from one to another you could use virtual environments. This is an easy task if you used conda as your python package manager. After a standard conda installation, assuming you have also installed the latest stable version of sunpy, you then proceed to create a new environment as: conda create-n sunpy-dev python=2.7 sunpy This will create a new environment called sunpy-dev with all of the dependencies needed by sunpy. We the proceed to change to the new environment: source activate sunpy-dev Then we need to remove the stable version from this environment conda remove sunpy to then install the version in your git repository cd to/sunpy/git/repository python setup.py develop At this stage you can use the development version in which you are working on. If you want to go back to the stable installation you can just change the environment by source deactivate Conclusion That’s it! You now have your own personal SunPy repo to develop on. You could hack away at it to your heart’s content, pushing changes to your fork on GitHub to share with others and to ensure that you have a backup online. But what about when you want to start contributing back to the main SunPy repo? That is the topic of the next section. Branches Developers should create topic branches within their repos for most of their main coding. Every repo starts with a single branch called master, which seldom needs to be used. Instead, work on any particular feature, bug, or portion of the code is done in its own separate branch. This way changes on any particular issue are isolated from other unrelated changes. Users can even work on several different branches simultaneously. To create a new branch run: git branch branchname To switch to the new branch: 3.2. Version Control 71 DemoDoc Documentation, Release git checkout branchname (or alternatively, git checkout -b branchname will accomplish the above). Developers should create new branches for the features they are working on. When they have finished making changes and the code has been tested and verified to be working well, the code can be merged back into the SunPy repo. This is usually done through something called a pull request. Example Workflow Before we get started Here is an example workflow for a SunPy developer on any given day. Before beginning this tutorial, follow the above instructions to grab a copy of the SunPy repo. Grabbing other people’s changes The first thing you want to do before you start coding anything new is to pull in the latest code that others have written since you last did any coding. To do this, run git pull: git pull upstream master This will ensure that you don’t edit a file that has changed since your last pull which will lead to merge conflicts later on. Code away Assuming there are no merge conflicts (which shouldn’t happen unless two people are working on the same part of the same file), then you are ready to begin coding. If there are conflicts check out our conflicts section. Push your changes to GitHub As you code away on your local repo, you will need to keep git aware of what you are doing and also your remote copy up to date. To add a file, create the file then run: git add If you delete a file run: git rm To move a file: git mv To check to see if git is happy run: git status which will give you a report of what has happened so far. Once you are at a good stopping point you should “commit” your changes. This will provide you an opportunity to describe what you have done so far. To do this type: git commit-a-m"description of your changes" After doing this you are ready to push your changes to your repo online with the command: 72 Chapter 3. Developer’s Guide DemoDoc Documentation, Release git push The local and remote copies of your repo are now synced. Contributing to the main repo Once you have made your desired changes, and committed and pushed your personal branch, you need to decide whether or not to merge those changes back into the main SunPy repo. If the changes you made are finished and have been tested and proven stable (see the testing section below), then they can be merged into SunPy. For now, lets assume that your changes are complete and they are ready to be added to the main SunPy repo. All contributed code to SunPy must be submitted as a “pull request”. To do this go to the github website and to your repo (remember to select the branch) then click on the “Pull Request” button (in the upper right hand corner next to the Fork button which you’ve used before). All initial pull requests must be made to the master branch unless they are a fix for specific version. This will submit your code to a review. You will likely receive some constructive comments on your code. To address these you can simply work on your code and push those changes to your local repo. Those changes will be reflected in your pull request. Once a member of the SunPy dev team approves your pull request then your code will be merged into the main SunPy repo and your code will be part of the main SunPy code. Congratulations! And that’s it! It may seem like a lot at first but once you go through the motions a few times it becomes very quick. Conflict resolution It may so happen that when you try to sync with the main repo there is a conflict error. This means that someone else has been working on the same section of code that you have. In such cases, the merge command will issue a conflict warning and will then expect you do the merge yourself. You can type: git mergetool to go through the conflicts. This command will likely open some merging tools which are already available on your computer. For example, on Mac OS X, it will open FileMerge (if you have XCode installed). You can check on your progress by typing: git status Once you are done, you should then commit your changes, in this case the resolution of the conflict with: git commit-m"Resolved conflict between my and online version of file.py" You can then proceed to push this change up to your branch. Backporting contribution Sometimes a contribution needs to be backported to the latest stable branch, this may be due to a bug being fixed or something similar. There are different ways to do so, if the contribution contains just a couple of commits, then the easiest is to cherry-pick them. Assuming you are in the branch of your new feature (eg. new_feature), this is what you need to do: First you need to find out which commits you want to copy to the other branch: git log Download/update the upstream branches to your local machine: git fetch upstream Create a new branch from the version you want to backport, X.y: git checkout-b new_feature_X.y upstream/X.y 3.2. Version Control 73 DemoDoc Documentation, Release Copy the commits using cherry-pick, xxxxxxxx (yyyyyyyy) refers to the oldest (newest) commit you want to backport. ^ at the end of the oldest is to include it, otherwise will take the ones after that point: git cherry-pick xxxxxxxx^..yyyyyyyy Push that new branch to your repository on github: git push origin new_feature_X.y Once done, then you can create a new pull request to the X.y branch. Remember to keep the same title that the original but adding [X.y] at the beginning. Also add a reference to the original pull request in the comments with the appropriate format: #pr-number. Coding Standards All code that is part of the SunPy project should follow The Style Guide for Python (PEP 8) and the coding style and conventions proposed by Astropy. Additionally, all code that goes in the trunk should be checked using PyLint. PyLint is an open source tool which analyzes Python code and checks for compliance with PEP8, as well as common coding errors and other potentially confusing or erroneous code statements. Checking the SunPy trunk code this helps to ensure some baseline level of quality and consistency for the code, and also helps to prevent potential problems from slipping through the cracks into the production code. If you followed the installation instructions for devs, pylint should already be installed on your system. To run PyLint on a file, simply call pylint from the command-line, passing in the name of the file you wish to check: pylint file.py By default PyLint will print lines with potential problems along with a summary report. To disable the summary report you can add either -rn or –reports=no to the command: pylint-rn file.py Further, a paver task has been created so that all of the SunPy code can be checked at once: paver pylint The output from PyLint will look something like: C: 87: Line too long (635/80) C:135: Line too long (98/80) R: 22:plot_fits: Too many local variables (22/15) R: 80:aia_color_table: Too many statements (59/50) W: 14: Unused import cm W: 16: Unused import Circle Each line includes a line number, the category of the warning message, and a short description of the issue encountered. The categories include: • [R]efactor for a “good practice” metric violation • [C]onvention for coding standard violation • [W]arning for stylistic problems, or minor programming issues • [E]rror for important programming issues (i.e. most probably bug) • [F]atal for errors which prevented further processing 74 Chapter 3. Developer’s Guide DemoDoc Documentation, Release PyLint checks a wide range of different things so the first time you run PyLint on a file you will likely get a large number of warnings. In some cases the warnings will help you to spot coding mistakes or areas that could be improved with refactoring. In other cases, however, the warning message may not apply and what you have there is exactly as it should be. In these cases it is possible to silence PyLint for that line. PyLint warning messages can be disabled at three different levels: globally (using a .pylintrc file), file-wide, and for a single line. (To be finished...) Global Settings SunPy makes use of a settings file (sunpyrc). This file contains a number of global settings such as where files should be downloaded by default or the default format for displaying times. When developing new functionality check this file and make use of the default values if appropriate or, if needed, define a new value. More information can be found in Customizing SunPy. Documentation All code must be documented. Undocumented code will not be accepted into SunPy. Documentation should follow the guidelines in PEP 8 and PEP 257 (Docstring conventions). Documentation for modules, classes, and functions should follow the NumPy/SciPy documentation style guide. We provide an example of good documentation below or you can just browse some of SunPy code itself for examples. All of the SunPy documentation (like this page!) is built by Sphinx and must therefore adhere to Sphinx guidelines. Sphinx Overview Sphinx is a tool for generating high-quality documentation in various formats (HTML, pdf, etc) and is especially well-suited for documenting Python projects. Sphinx works by parsing files written using a a Mediawiki-like syntax called reStructuredText. In addition to parsing static files of reStructuredText, Sphinx can also be told to parse code comments. In fact, in addition to what you are reading right now, the Python documentation was also created using Sphinx. Usage All of the SunPy documentation is contained in the doc/source folder and code comments. To generate the doc- umentation you must have Sphinx (as well as Numpydoc and astropy-helpers) installed on your computer. Enter the doc/source folder and run: make html This will generate HTML documentation for SunPy. To clean up and delete the generated documentation run: make clean For more information on how to use Sphinx, consult the Sphinx documentation. The rest of this section will describe how to document the SunPy code in order to guarantee that well-formatted documentation will be created. doctest The example codes in the Guide section of the docs are configured with the Sphinx doctest extension. This will test the example code to make sure it runs correctly, it can be executed using: 3.4. Global Settings 75 DemoDoc Documentation, Release sphinx-build-t doctest-b doctest./../build from inside the doc/source folder. Use of quantities and units Much code perform calculations using physical quantities. SunPy uses astropy’s quantities and units implementation to store, express and convert physical quantities. New classes and functions should adhere to SunPy’s quantity and unit usage guidelines. This document sets out SunPy’s reasons and requirements for the usage of quantities and units. Briefly, SunPy’s policy is that all user-facing function/object arguments which accept physical quantities as input **MUST* accept astropy quantities, and ONLY astropy quantities*. Developers should consult the Astropy Quantities and Units page for the latest updates on using quantities and units. The astropy tutorial on quantities and units also provides useful examples on their capabilities. Astropy provides the decorator ~astropy.units.quantity_input that checks the units of the input arguments to a function against the expected units of the argument. We recommend using this decorator to perform function argument unit checks. The decorator ensures that the units of the input to the function are convertible to that specified by the decorator, for example import astropy.units asu @u.quantity_input(myangle=u.arcsec) def myfunction(myangle): return myangle**2 This function only accepts arguments that are convertible to arcseconds. Therefore, >>> myangle(20 * u.degree) returns the expected answer but >>> myangle(20 * u.km) raises an error. The following is an example of a use-facing function that returns the area of a square, in units that are the square of the input length unit: @u.quantity_input(side_length=u.m) def get_area_of_square(side_length): """ Compute the area of a square. Parameters ------side_length : `~astropy.units.quantity.Quantity` Side length of the square Returns ------area : `~astropy.units.quantity.Quantity` Area of the square. """ return (side_length ** 2) 76 Chapter 3. Developer’s Guide DemoDoc Documentation, Release This more advanced example shows how a private function that does not accept quantities can be wrapped by a function that does: @u.quantity_input(side_length=u.m) def some_function(length): """ Does something useful. Parameters ------length : `~astropy.units.quantity.Quantity` A length. Returns ------length : `~astropy.units.quantity.Quantity` Another length """ # the following function either # a] does not accept Quantities # b] is slow if using Quantities result= _private_wrapper_function(length.convert('meters').value) # now convert back to a quantity result= Quantity(result_meters, units_of_the_private_wrapper_function) return result In this example, the non-user facing function _private_wrapper_function requires a numerical input in units of meters, and returns a numerical output. The developer knows that the result of _private_wrapper_function is in the units units_of_the_private_wrapper_function, and sets the result of some_function to return the answer in those units. Examples Modules Each module or package should begin with a docstring describing its overall purpose and functioning. Below that meta-tags containing author, license, email and credits information may also be listed. Example: """This is an example module comment. An explanation of the purpose of the module would go here and will appear in the generated documentation """ # # TODO # Developer notes and todo items can be listed here and will not be # included in the documentation. # __authors__=["Keith Hughitt","Steven Christe","Jack Ireland","Alex Young"] __email__="[email protected]" __license__="xxx" 3.5. Documentation 77 DemoDoc Documentation, Release For details about what sections can be included, see the section on documenting modules in the NumPy/SciPy style guide. Classes Class docstrings should include a clear and concise docstring explaining the overall purpose of the class, required and optional input parameters, and the return value. Additionally, notes, references and examples are encouraged. Example (sunpy.map.Map) """ Map(data, header) A spatially-aware data array based on the SolarSoft Map object Parameters ------data : numpy.ndarray, list A 2d list or ndarray containing the map data header : dict A dictionary of the original image header tags Attributes ------header : dict A dictionary representation of the image header date : datetime Image observation time det : str Detector name inst : str Instrument name meas : str, int Measurement name. For AIA this is the wavelength of image obs : str Observatory name r_sun : float Radius of the sun name : str Nickname for the image type (e.g. "AIA 171") center : dict X and Y coordinate for the center of the sun in arcseconds scale: dict Image scale along the x and y axes in arcseconds/pixel Examples ------>>> aia = sunpy.map.Map(sunpy.data.sample.AIA_171_IMAGE) >>> aia.T Map([[ 0.3125, 1. , -1.1875, ..., -0.625 , 0.5625, 0.5 ], [-0.0625, 0.1875, 0.375 , ..., 0.0625, 0.0625, -0.125 ], [-0.125 , -0.8125, -0.5 , ..., -0.3125, 0.5625, 0.4375], ..., [ 0.625 , 0.625 , -0.125 , ..., 0.125 , -0.0625, 0.6875], [-0.625 , -0.625 , -0.625 , ..., 0.125 , -0.0625, 0.6875], [ 0. , 0. , -1.1875, ..., 0.125 , 0. , 0.6875]]) >>> aia.header['cunit1'] 'arcsec' 78 Chapter 3. Developer’s Guide DemoDoc Documentation, Release >>> aia.show() >>> import matplotlib.cm as cm >>> import matplotlib.colors as colors >>> aia.peek(cmap=cm.hot, norm=colors.Normalize(1, 2048)) See Also ------numpy.ndarray Parent class for the Map object References ------| http://docs.scipy.org/doc/numpy/reference/arrays.classes.html | http://docs.scipy.org/doc/numpy/user/basics.subclassing.html | http://www.scipy.org/Subclasses """ Functions Functions should include a clear and concise docstring explaining the overall purpose of the function, required and optional input parameters, and the return value. Additionally, notes, references and examples are encouraged. Example (numpy.matlib.ones): def ones(shape, dtype=None, order='C'): """ Matrix of ones. Return a matrix of given shape and type, filled with ones. Parameters ------shape : {sequence of ints, int} Shape of the matrix dtype : data-type, optional The desired data-type for the matrix, default is np.float64. order : {'C', 'F'}, optional Whether to store matrix in C- or Fortran-contiguous order, default is 'C'. Returns ------out : matrix Matrix of ones of given shape, dtype, and order. See Also ------ones : Array of ones. matlib.zeros : Zero matrix. Notes ----- If `shape` has length one i.e. ``(N,)``, or is a scalar ``N``, `out` becomes a single row matrix of shape ``(1,N)``. Examples ------ 3.5. Documentation 79 DemoDoc Documentation, Release >>> np.matlib.ones((2,3)) matrix([[ 1., 1., 1.], [ 1., 1., 1.]]) >>> np.matlib.ones(2) matrix([[ 1., 1.]]) """ a= ndarray.__new__(matrix, shape, dtype, order=order) a.fill(1) return a For details about what sections can be included, see the section on documenting functions in the NumPy/SciPy style guide. Trouble-shooting Sphinx can be very particular about formatting, and the warnings and errors outputted aren’t always obvious. Below are some commonly-encountered warning/error messages along with a human-readable translation: WARNING: Duplicate explicit target name: “xxx”. If you reference the same URL, etc more than once in the same document sphinx will complain. To avoid, use double- underscores instead of single ones after the URL. ERROR: Malformed table. Column span alignment problem at line offset n Make sure there is a space before and after each colon in your class and function docs (e.g. attribute : type, instead of attribute: type). Also, for some sections (e.g. Attributes) numpydoc seems to complain when a description spans more than one line, particularly if it is the first attribute listed. WARNING: Block quote ends without a blank line; unexpected unindent. Lists should be indented one level from their parents. ERROR: Unkown target name: “xxx” In addition to legitimate errors of this type, this error will also occur when variables have a trailing underscore, e.g., xxx_. WARNING: Explicit markup ends without a blank line; unexpected unindent. This usually occurs when the text following a directive is wrapped to the next line without properly indenting a multi- line text block. WARNING: toctree references unknown document ‘...’ / WARNING: toctree contains reference to nonexisting document This pair of errors is due to the way numpydoc scrapes class members. Testing This is a brief tutorial on how to write and run SunPy unit tests. SunPy makes use of the great package pytest 80 Chapter 3. Developer’s Guide DemoDoc Documentation, Release Writing a unit test Consider a simple module stuff.py that contains the simple function shown below.: def double(x): return 2 * x We can write a test case for this function by defining a new function containing the test (or tests) we want to perform. Suppose we want to check that the correct behaviour occurs when we pass a value of 5 to double(). We would write the test function like this: def test_answer(): assert double(5) == 10 There are two things to note here. Firstly, names of test cases should always begin with test_. This is because pytest searches for test cases named this way. Secondly, we use assert to assert our expectation of what the result of the test should be. In this example, the test returns true and so the test passes. The example given above is one in which the function and test reside in the same module. In SunPy, functions and tests are separated and the latter can be found in the tests directory within the directory containing the module. The convention is to have one test module per module, with the names for the test modules being the same as those for the modules prefixed with test_. For example, the modules xml.py and multimethod.py in sunpy/util have corresponding test modules test_xml.py and test_multimethod.py in sunpy/util/tests. There are some tests for functions and methods in SunPy that require a working connection to the internet. pytest is configured in a way that it iterates over all tests that have been marked as online and checks if there is an established connection to the internet. If there is none, the test is skipped, otherwise it is run. Marking tests is pretty straightforward in pytest: use the decorator @pytest.mark.online to mark a test function as needing an internet connection. Writing a unit test for a figure You can write SunPy unit tests that test the generation of matplotlib figures by adding the decorator sunpy.tests.helpers.figure_test. Here is a simple example: import matplotlib.pyplot as plt from sunpy.tests.helpers import figure_test @figure_test def test_simple_plot(): plt.plot([0,1]) The current figure at the end of the unit test, or an explicitly returned figure, has its hash compared against an es- tablished hash library (more on this below). If the hashes do not match, the figure has changed, and thus the test is considered to have failed. All such tests are automatically marked with the pytest mark pytest.mark.figure. See the next section for how to use marks. You will need to update the library of figure hashes after you create a new figure test or after a figure has intentionally changed due to code improvement. After you have confirmed that any conflicting hashes are associated with desired changes in figures, copy the hash-library file listed at the end of the test report to sunpy/tests/. Be forewarned that the hash library will likely need to be updated for multiple versions of Python. Running unit tests To find and run all the SunPy unit tests, simply run 3.6. Testing 81 DemoDoc Documentation, Release py.test from the root of the SunPy tree (i.e. the directory containing INSTALL.TXT, sunpy, doc, etc.). This will produce a lot of output and you’ll probably want to run only selected test modules at a time. This is done by specifying the module on the command line, e.g.: py.test sunpy/util/tests/test_xml.py for the tests for sunpy.util.xml. To run only tests that been marked with a specific pytest mark using the decorator @pytest.mark (see the section Writing a unit test), use the following command (where MARK is the name of the mark): py.test-k MARK To exclude (i.e. skip all tests with a certain mark, use the following code (where MARK is the name of the mark): py.test-k-MARK Note that pytest is configured to skip all tests with the mark online if there is no connection to the internet. This cannot be circumvented, i.e. it cannot be forced to run a test with the mark online if there is no working internet connection (rename the mark to something else to call the test function anyway). To get more information about skipped and xfailed tests (xfail means a test has passed although it has been marked as @pytest.mark.xfail), you need to use the option -rs for skipped tests and -rx for xfailed tests, respectively. Or use -rxs for detailed information on both skipped and xfailed tests. When to write unit tests A rule of thumb for unit testing is to have at least one unit test per public function. Testing Your Code Before Committing When you commit your changes and make a Pull Request to the main SunPy repo on GitHub, your code will be tested by Travis CI to make sure that all the tests pass and the documentation builds without any warnings. Before you commit your code you should check that this is the case. There is a helper script in sunpy/tools/pre-commit.sh that is designed to run these tests automatically every time you run git commit to install it copy the file from sunpy/tools/pre- commit.sh to sunpy/.git/hooks/pre-commit, you should also check the script to make sure that it is configured properly for your system. Continuous Integration SunPy makes use of the Travis CI service. This service builds a version of SunPy and runs all the tests. It also integrates with GitHub and will report the test results on any Pull Request when they are submitted and when they are updated. The Travis CI server not only builds SunPy from source, but currently it builds all of SunPy’s dependencies from source as well using pip, all of this behaviour is specified in the .travis.yml file in the root of the SunPy repo. New Functionality For SunPy, we would encourage all developers to thoroughly cover their code by writing unit tests for each new function created. 82 Chapter 3. Developer’s Guide DemoDoc Documentation, Release Developers who want to take an aggressive approach to reducing bugs may even wish to consider adopting a practice such as Test Drive Development (TDD) whereby unit tests are written before any actual code is written. The tests begin by failing, and then as they code is developed the user re-runs the tests until all of them are passing. Bugs discovered In addition to writing unit tests new functionality, it is also a good practice to write a unit test each time a bug is found, and submit the unit test along with the fix for the problem. This way we can ensure that the bug does not re-emerge at a later time. 3.6. Testing 83 DemoDoc Documentation, Release 84 Chapter 3. Developer’s Guide CHAPTER 4 Reporting Bugs All bugs are kept track of on the GitHub issue tracker. If you run into any unexpected behavior or have a question please send an email or add an issue directly to the issue tracker. 85 DemoDoc Documentation, Release 86 Chapter 4. Reporting Bugs CHAPTER 5 SSWIDL/SunPy Cheat Sheet SolarSoft (SSWIDL) is a popular IDL software library for solar data analysis, and in fact, many parts of SunPy are inspired by data structures and functions in SSWIDL. Though IDL and Python are very different it sometimes helps to consider how to translate simple tasks between the two languages. The primary packages which provide much of the functionality for scientific data analysis in Python are NumPy and SciPy. In the following we assume that those packages are available to you and that you are imported Numpy is imported as np with the following import statement: import numpy as np import scipy as sp In the following examples, a and b could be arrays. For python the arrays must be numpy arrays which can be created simply through: np.array(a) where a is a python list of numbers. Relational Operators IDL Python a EQ b a == b a LT b a < b a GT b a > b a GE b a >= b a LE b a <= b a NE b a != b Logical Operators IDL Python a and b a and b a or b a or b Math Functions 87 DemoDoc Documentation, Release IDL Python cos(a) np.cos(a) alog(a) np.log(a) alog10(a) np.alog(a) exp(a) np.exp(a) Math Constants IDL Python !pi np.pi exp(1) np.e Arrays Sequences IDL Python indgen(10) np.arange(0,10) findgen(10) np.arange(0,10,dtype=np.float) Array Creation IDL Python dblarr(3,5) np.zeros((3,5)) intarr(3,5) np.zeros((3,5),dtype=np.int) dblarr(3,5)+1 np.ones((3,5)) intarr(3,5)+9 np.zeros((3,5),dtype=np.int) + 9 boolarr(10) np.zeros(10,dtype=bool) identity(3) np.identity(3) Many more examples can be found on this page 88 Chapter 5. SSWIDL/SunPy Cheat Sheet Python Module Index s sunpy.map, 65 89 DemoDoc Documentation, Release 90 Python Module Index Index E environment variable HOME, 60 SUNPY_CONFIGDIR, 60 H HOME, 60 S sunpy.map (module), 65 SUNPY_CONFIGDIR, 60 91