SMILI Documentation Release 0.0.1

SMILI Developper Team

Jun 21, 2019

Contents

1 Table of Contents 3

i ii SMILI Documentation, Release 0.0.1

Sparse Modeling Imaging Library for Interferometry This website is the documentation for SMILI. SMILI is a python-interfaced library for interferometric imaging using sparse sampling techniques. SMILI is mainly designed for very long baseline interferometry, and has been under the active development primarily for the Event Horizon Telescope. This documentation describes its basic usage with some example data sets. However, SMILI has yet been actively and dynamically developed for many new topics and challenges of the EHT. The documentation is not perfect and sometimes outdated due to dynamical changes in the data structure. Please contact to Kazu Akiyama at NRAO/MIT Haystack Observatory if you have any questions about this library. You may contact with following other core developers, too. • Kazu Akiyama (The Main Developer) at NRAO/MIT Haystack Observatory • Fumie Tazaki (Developer) (Japanese Only) at NAOJ • Shiro Ikeda (Developer) at the Institute of Statistical Mathematics • Kotaro Moriyama (Developer) at NAOJ/MIT Haystack Observatory

Contents 1 SMILI Documentation, Release 0.0.1

2 Contents CHAPTER 1

Table of Contents

1.1 Installation

1.1.1 Requirements

SMILI consists of python modules and Fortran/C internal libraries called from python modules. Here, we summarize required python packages and external packages for SMILI. You will also need ds9 for some functions such as setting imaging regions (CLEAN box) interatively.

1.1.2 Python Packages and Modules

SMILI has been tested and developed in Python 2.7 environments provided by the Anaconda package. We recommend using Anaconda for SMILI. All of mandatory packages will be automatically installed during installation. There are some optional packages that can be used for SMILI. • ehtim: https://github.com/achael/eht-imaging • ehtplot: https://github.com/chanchikwan/ehtplot

1.1.3 External Libraries (For MacPort users, Ubuntu/Debian users)

Fortran/C internal libraries of SMILI use following external libraries. This path has been tested for • Mac OS X 10.012/10.13 with MacPort’s gcc 8 • Ubuntu 2016LTS & 2018LTS. 1) OpenBLAS

3 SMILI Documentation, Release 0.0.1

We use OpenBLAS, which is the fastest library among publicly-available BLAS implementations. Our recommendation is to build up OpenBLAS by yourself with a compile option USE_OPENMP=1 and use it for our library. The option USE_OPENMP=1 enables OpenBLAS to perform paralleled multi-threads calculations, which will accelerate our library. In most of cases, you can compile this library by

# Clone the current repository git clone https://github.com/xianyi/OpenBLAS

# Compile and install cd OpenBLAS make USE_OPENMP=1 make PREFIX="Your install directory; such as /usr/local or $HOME/local"

˓→install

Note that for MacOSX, USE_OPENMP=1 option does not work and should be omitted. You may need superuser to install OpenBLAS (i.e. to run the last command). SMILI uses pkg-config to find appropriate compiler flags for OpenBLAS. Once the library is installed, you can check if the package configuration file can be accessible. You can type

pkg-config --debug openblas

If you can see the correct package configuration file in the output (should be $PRE- FIX/lib/pkgconfig/openblas.pc), you are all set with OpenBLAS. Otherwise, you need to set PKG_CONFIG_PATH to your pkgconfig directory by, for instance

export PKG_CONFIG_PATH="Your prefix for OpenBLAS such as /usr/local"/lib/

˓→pkgconfig:$PKG_CONFIG_PATH

Then you can check by ‘‘pkg-conﬁg –debug openblas” if the path is correct. Some Other Tips: If you are using Ubuntu, RedHat and its variant, the default OpenBLAS package, which is installable with apt-get/aptitude or yum, seems compiled without this option (USE_OPENMP=1), so we recommend compiling OpenBLAS by yourself. If you are using macOS, unfortunately, this option is not available so far. You may use a package available in a popular package system (e.g. MacPort, Fink, Homebrew). 2) FFTW3 We use FFTW3, which is one of the fastest library among publicly-available FFT library. For non-Ubuntu users, our recommendation is to build up FFTW3 by yourself.

In most of cases, you can compile this library by

# Download the library (in case of version 3.3.7) wget http://www.fftw.org/fftw-3.3.7.tar.gz # you should check the latest

˓→version tar xzvf fftw-3.3.7.tar.gz cd fftw-3.3.7

# Compile and install ./configure --prefix="install directory; such as /usr/local, $HOME/local"--

˓→enable-openmp --enable-threads --enable-shared make make install

4 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

You may need superuser to install FFTW3 (i.e. to run the last command). SMILI uses pkg-config to find appropriate compiler flags for FFTW3. Once the library is installed, you can check if the package configuration file can be accessible. You can type

pkg-config --debug fftw3

If you can see the correct package configuration file in the output (should be $PRE- FIX/lib/pkgconfig/fftw3.pc), you are all set with OpenBLAS. Otherwise, you need to set PKG_CONFIG_PATH to your pkgconfig directory by, for instance

export PKG_CONFIG_PATH="Your prefix such as /usr/local"/lib/pkgconfig:$PKG_

˓→CONFIG_PATH

Then you can check by ‘‘pkg-conﬁg –debug fftw3” if the path is correct. Some Other Tips: If you are using Ubuntu, the default fftw3 package, which is installable with apt- get/aptitude seems compiled with the option for Openmp (–enable-openmp). So, you don’t need to install it by yourself.

1.1.4 External Libraries (for homebrew users in MacOS)

1) pyenv and Anaconda installation: Since Anaconda conﬂicts with Homebrew, it should be installed via pyenv.

brew install pyenv export PATH=$HOME/.pyenv/shims:$PATH

Then install Anaconda.

pyenv install -l | grep anaconda2 pyenv install anaconda2-X # select anaconda 2 version pyenv global anaconda2-X python --version # check versions

2) OPENBLAS installation: It is mostly same to the above one, but you will need to install gcc.

# Clone the current repository git clone https://github.com/xianyi/OpenBLAS

# Install gcc49 brew install gcc49 sudo ln -sf /usr/local/bin/gcc-4.9 /usr/bin/gcc sudo ln -sf /usr/local/bin/g++-4.9 /usr/bin/g++

# Install OPENBLAS make USE_OPENMP=1CC=gcc make PREFIX=/usr/local install

3) FFTW3 installation: No net change from the above one.

# Download the source code wget http://www.fftw.org/fftw-3.X.X.tar.gz tar xzvf fftw-3.X.X.tar.gz cd fftw-3.X.X

# Install ./configure prefix="/usr/local" --enable-openmp --enable-threads --enable-

˓→shared (continues on next page)

1.1. Installation 5 SMILI Documentation, Release 0.0.1

(continued from previous page) make make install

1.1.5 Downloading SMILI

You can download the code from github.

# Clone the repository git clone https://github.com/astrosmili/smili

1.1.6 Installing SMILI

For compiling the whole library, you need to work in your SMILI directory. cd(Your SMILI Directory)

Generate Makefiles with ./configure. If you have correct paths to package-config files for OpenBLAS and FFTW3, you would not need any options.

./configure

If you don’t have paths to these ﬁles, then you need to specify them manually prior to type ./conﬁgure

# Example for OpenBLAS export OPENBLAS_LIBS="-LYOURPREFIX/lib -lopenblas" export OPENBLAS_CFLAGS="-IYOURPREFIX/include"

# Example for FFTW3 export FFTW3_LIBS="-LYOURPREFIX/lib -lfftw3" export FFTW3_CFLAGS="-IYOURPREFIX/include"

Make and compile the library. The internal C/Fortran Library will be compiled into python modules, and then the whole python modules will be added to the package list of your Python environment. make install

If you can load following modules in your python interpretator, SMILI is probably installed successfully.

# import SMILI from smili import imdata, uvdata, imaging

(IMPORTANT NOTE; 2018/04/26) Previously, you needed to type autoconf before ./conﬁgure command. This is no longer necessary. (IMPORTANT NOTE; 2018/01/04) Previously, you needed to add a PYTHONPATH to your SMILI Directory. This is no longer required, because the make command will run setup.py and install SMILI into the package list of your Python environment.

1.1.7 Updating SMILI

We strongly recommend cleaning up the entire library before updating.

6 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

cd(Your SMILI Directory) make uninstall

Then, you can update the repository with git pull.

git pull

Now, the repository has updated. You can follow the above section Installing SMILI for recompiling your SMILI.

1.2 Tutorial and Example Scripts

1.2.1 Images

The most basic class for the image in SMILI is imdata.IMFITS object.Here, we show its basic usage. You can see list of functions at in this page.

[1]:% matplotlib inline from smili import imdata, util

# this is for plotting import matplotlib.pyplot as plt import matplotlib.cm as cm import matplotlib as mpl

Creating a blank image

Let’s start from a blank image. You can make a blank image.

[2]: # Create a blank image image= imdata.IMFITS( # pixel size in the specified angular unit (you can specify dy if you want a non-

˓→square image pixel) dx=2, # number of pixels (you can also specify ny if you want a non-square field of

˓→view) nx= 256, # angular unit (e.g., rad, deg, amin/arcmin, asec/arcsec, mas, uas) angunit="uas" )

# Plot image (We will explain this function later) image.imshow(scale="linear", colorbar="True") [2]: (, )

1.2. Tutorial and Example Scripts 7 SMILI Documentation, Release 0.0.1

SMILI also can edit the location of the origin by nxref and nyref in the unit of the pixel number. Note that nxref=1, nyref=1 means the center will be set to the center of the leftmost and bottom pixel.

[3]: # Create a blank image image= imdata.IMFITS( dx=2, nx= 256, nxref= 64, nyref= 64, angunit="uas" )

# Plot image image.imshow(scale="linear", colorbar="True") [3]: (, )

8 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

Loading images from FITS ﬁles

Here, we use an FITS image reconstructed with SMILI from a 43 GHz 3C 273 data set of the Boston University Blazar Group.

[4]: image= imdata.IMFITS( "./BU_3C273_sampleimage.fits", # filename angunit="mas" # default angular unit that you want to use )

# Plot image image.imshow(scale="linear", colorbar="True") [4]: (, )

The popular FITS format, for instance, images from DIFMAP/AIPS can be loaded without a problem. However, there are some options to load images with non-popular FITS formats.

[ ]: # Default image= imdata.IMFITS( "example.fits", imfitstype="standard", # this is the deafult option for SMILI FITS, AIPS/DIFMAP

˓→FITS files. angunit="mas" )

# FITS files generated by eht-im image= imdata.IMFITS( "example.fits", imfitstype="ehtim", # this is the option if you want to load fits files generated

˓→by eht-imaging library angunit="mas" )

# AIPS CC Table image= imdata.IMFITS( "example.fits", imfitstype="aipscc", # this is the option if you want to load AIPS CC tables

˓→instead of the image data (continues on next page)

1.2. Tutorial and Example Scripts 9 SMILI Documentation, Release 0.0.1

(continued from previous page) angunit="mas" )

Plot images

SMILI’s default plotting function is imdata.IMFITS.imshow. This function is inheriting pyplot.imshow so you can use all of arguments in pyplot.imshow to customize your plots.

[5]: # linear scale plt.figure() plt.title("linear, afmhot") image.imshow( scale="linear", colorbar="True")

# Gamma scale plt.figure() plt.title("gamma scale, jet") image.imshow( scale="gamma", gamma=0.5, cmap=cm.jet, colorbar="True")

# Log Scale plt.figure() plt.title("log scale, inferno_r, mjy") image.imshow(scale="log", dyrange=1000, fluxunit="mjy", cmap=cm.inferno_r, colorbar=

˓→"True") [5]: (, )

10 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

If you set the beam information, then you can also compute Jy/beam

[6]: # Set beam to image. image.set_beam(majsize=0.3,minsize=0.3,angunit="mas")

# Plot convolved image with beam. image.imshow(fluxunit="jy",saunit="beam", scale="log", dyrange=100, colorbar=True) [6]: (, )

1.2. Tutorial and Example Scripts 11 SMILI Documentation, Release 0.0.1

Extracting information of images

You can get low level infromation of IMFITS object here. • imdata.IMFITS.header: Header information • imdata.IMFITS.data: 4 dimensional array of images.

[7]: # Header information (dictionary) print("Header Information") print(image.header)

# Example: extract header parameters. nx= image.header["nx"] ny= image.header["ny"] nxref= image.header["nxref"] nyref= image.header["nyref"] dx= image.header["dx"] *util.angconv("deg","mas") dy= image.header["dy"] *util.angconv("deg","mas")

print("Example information read from the header") print(" Number of pixels: (nx, ny) = (%d, %d)"% (nx,ny)) print(" Reference pixel: (nxref, nyref) = (%d, %d)"% (nxref,nyref)) print(" Pixel size: (dx, dy) = (%1.2f, %1.2f) [mas]"% (dx,dy)) Header Information {’telescope’: ’VLBA’, ’observer’: ’BM413X’, ’nyref’: 216.0, ’bmaj’: 8.

˓→333333333333333e-08, ’instrument’: ’VLBA’, ’nsref’: 1, ’nf’: 1, ’nx’: 256, ’ny’:

˓→256, ’bmin’: 8.333333333333333e-08, ’nfref’: 1.0, ’ns’: 1, ’df’: 64000000.0,

˓→’object’: ’3C273’, ’dateobs’: None, ’nxref’: 40.0, ’dx’: -5.5555555555555e-09,

˓→’dy’: 5.55555555555555e-09, ’ds’: 1, ’bpa’: 0.0, ’f’: 43007500000.0, ’s’: 1, ’y’: 2.

˓→05238841111, ’x’: 187.277915537} Example information read from the header Number of pixels: (nx, ny) = (256, 256) Reference pixel: (nxref, nyref) = (40, 216) Pixel size: (dx, dy) = (-0.02, 0.02) [mas]

12 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

[8]: # low level image array information (in Jy/pixel) print("Shape of the image array:") print(image.data.shape)

print("") print("Array:") print(image.data) Shape of the image array: (1, 1, 256, 256)

Array: [[[[0.00000000e+00 0.00000000e+00 0.00000000e+00 ... 9.21214561e-06 1.46166836e-05 2.08356582e-04] [0.00000000e+00 0.00000000e+00 0.00000000e+00 ... 5.90111877e-06 7.22100152e-06 2.47679859e-05] [0.00000000e+00 0.00000000e+00 0.00000000e+00 ... 3.56583850e-06 4.00781271e-06 8.47375404e-06] ... [0.00000000e+00 0.00000000e+00 0.00000000e+00 ... 0.00000000e+00 0.00000000e+00 0.00000000e+00] [0.00000000e+00 0.00000000e+00 0.00000000e+00 ... 0.00000000e+00 0.00000000e+00 0.00000000e+00] [0.00000000e+00 0.00000000e+00 0.00000000e+00 ... 0.00000000e+00 0.00000000e+00 0.00000000e+00]]]]

There are many additonal functions to get some information about images. • Total Flux and Peak Intensity

[9]: totalflux= image.totalflux() peak= image.peak(fluxunit="Jy", saunit="pixel") print("total flux = %1.2f Jy"% (totalflux)) print("peak flux = %1.2f Jy/pixel"% (peak)) total flux = 10.23 Jy peak flux = 0.04 Jy/pixel

• Grid coordinates

[10]: # Image extent: compatible with extent argument of pyplot.imshow imextent= image.get_imextent(angunit="mas") print("Image Extent") print(imextent) print("")

# 1D GRID x, y= image.get_xygrid(twodim=False, angunit="mas") print("1D coodirnate grids") print(x) print("")

# 2D GRID x, y= image.get_xygrid(twodim=True, angunit="mas") print("2D coodirnate grids")

util.matplotlibrc(ncols=2, width=500, height=300) fig, axs= plt.subplots(ncols=2) plt.sca(axs[0]) (continues on next page)

1.2. Tutorial and Example Scripts 13 SMILI Documentation, Release 0.0.1

(continued from previous page) plt.title("x-axis") plt.imshow(x, extent=imextent, cmap=cm.jet, origin="lower") plt.xlabel("Relative RA (mas)") plt.ylabel("Relative Dec (mas)") plt.colorbar(label="x coordinate (mas)") plt.sca(axs[1]) plt.title("y-axis") plt.imshow(y, extent=imextent, cmap=cm.jet, origin="lower") plt.xlabel("Relative RA (mas)") plt.ylabel("Relative Dec (mas)") plt.colorbar(label="y coordinate (mas)") mpl.rcdefaults() Image Extent [ 0.79 -4.33 -4.31 0.81]

1D coodirnate grids [ 0.78 0.76 0.74 0.72 0.7 0.68 0.66 0.64 0.62 0.6 0.58 0.56 0.54 0.52 0.5 0.48 0.46 0.44 0.42 0.4 0.38 0.36 0.34 0.32 0.3 0.28 0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 -0. -0.02 -0.04 -0.06 -0.08 -0.1 -0.12 -0.14 -0.16 -0.18 -0.2 -0.22 -0.24 -0.26 -0.28 -0.3 -0.32 -0.34 -0.36 -0.38 -0.4 -0.42 -0.44 -0.46 -0.48 -0.5 -0.52 -0.54 -0.56 -0.58 -0.6 -0.62 -0.64 -0.66 -0.68 -0.7 -0.72 -0.74 -0.76 -0.78 -0.8 -0.82 -0.84 -0.86 -0.88 -0.9 -0.92 -0.94 -0.96 -0.98 -1. -1.02 -1.04 -1.06 -1.08 -1.1 -1.12 -1.14 -1.16 -1.18 -1.2 -1.22 -1.24 -1.26 -1.28 -1.3 -1.32 -1.34 -1.36 -1.38 -1.4 -1.42 -1.44 -1.46 -1.48 -1.5 -1.52 -1.54 -1.56 -1.58 -1.6 -1.62 -1.64 -1.66 -1.68 -1.7 -1.72 -1.74 -1.76 -1.78 -1.8 -1.82 -1.84 -1.86 -1.88 -1.9 -1.92 -1.94 -1.96 -1.98 -2. -2.02 -2.04 -2.06 -2.08 -2.1 -2.12 -2.14 -2.16 -2.18 -2.2 -2.22 -2.24 -2.26 -2.28 -2.3 -2.32 -2.34 -2.36 -2.38 -2.4 -2.42 -2.44 -2.46 -2.48 -2.5 -2.52 -2.54 -2.56 -2.58 -2.6 -2.62 -2.64 -2.66 -2.68 -2.7 -2.72 -2.74 -2.76 -2.78 -2.8 -2.82 -2.84 -2.86 -2.88 -2.9 -2.92 -2.94 -2.96 -2.98 -3. -3.02 -3.04 -3.06 -3.08 -3.1 -3.12 -3.14 -3.16 -3.18 -3.2 -3.22 -3.24 -3.26 -3.28 -3.3 -3.32 -3.34 -3.36 -3.38 -3.4 -3.42 -3.44 -3.46 -3.48 -3.5 -3.52 -3.54 -3.56 -3.58 -3.6 -3.62 -3.64 -3.66 -3.68 -3.7 -3.72 -3.74 -3.76 -3.78 -3.8 -3.82 -3.84 -3.86 -3.88 -3.9 -3.92 -3.94 -3.96 -3.98 -4. -4.02 -4.04 -4.06 -4.08 -4.1 -4.12 -4.14 -4.16 -4.18 -4.2 -4.22 -4.24 -4.26 -4.28 -4.3 -4.32]

2D coodirnate grids

14 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

• two dimensional image array

[11]: imarr= image.get_imarray() imarr [11]: array([[[[0.00000000e+00, 0.00000000e+00, 0.00000000e+00, ..., 9.21214561e-06, 1.46166836e-05, 2.08356582e-04], [0.00000000e+00, 0.00000000e+00, 0.00000000e+00, ..., 5.90111877e-06, 7.22100152e-06, 2.47679859e-05], [0.00000000e+00, 0.00000000e+00, 0.00000000e+00, ..., 3.56583850e-06, 4.00781271e-06, 8.47375404e-06], ..., [0.00000000e+00, 0.00000000e+00, 0.00000000e+00, ..., 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], [0.00000000e+00, 0.00000000e+00, 0.00000000e+00, ..., 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], [0.00000000e+00, 0.00000000e+00, 0.00000000e+00, ..., 0.00000000e+00, 0.00000000e+00, 0.00000000e+00]]]])

Editing Images

From the above functions, you can compute some quantities, do analysis and edit images. We also have several functions to edit images (e.g. shift, convolving, regridding) which will be frequently used. Here, we show some representatitive functions. • Shifting: When you want to shift the image, peakshift, comshift, and refshift are available.

[12]: # Check peak position print(image.peakpos())

# Shift the peak position to the origin image.peakshift().imshow(scale="linear", colorbar="True") {’y0’: 0.07999999999999992, ’x0’: 0.059999999999999394, ’angunit’: ’mas’} [12]: (, )

1.2. Tutorial and Example Scripts 15 SMILI Documentation, Release 0.0.1

[13]: # Check the position of center of mass (COM). print(image.compos())

# Shift the COM position to the origin image.comshift().imshow(scale="linear", colorbar="True") {’y0’: -0.5030372749420002, ’x0’: -0.5017984082235737, ’angunit’: ’mas’} [13]: (, )

16 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

[14]: # Shift the center to the arbitral location image.refshift(x0=2, y0=2).imshow(scale="linear", colorbar="True") [14]: (, )

1.2. Tutorial and Example Scripts 17 SMILI Documentation, Release 0.0.1

• Image copy and re-gridding

[15]: # New grid parameters newdx= image.header["dx"] *util.angconv("deg","mas")/2 newnx= image.header["nx"] *2

# Making an image based on new grids blank= imdata.IMFITS(dx=newdx,nx=newnx,angunit="mas") image_regrid= blank.cpimage(image) # regridding the image

plt.title("new re-gridded image") image_regrid.imshow(scale="linear", colorbar="True") [15]: (, )

18 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

• Convolution

[17]: image.convolve_gauss(majsize=1.0, angunit="mas").imshow(scale="linear", colorbar="True

˓→") [17]: (, )

1.2. Tutorial and Example Scripts 19 SMILI Documentation, Release 0.0.1

• Soft/Hard/minimum Thresholding

[18]: image.hard_threshold(threshold=0.2, relative=True, save_totalflux=False).imshow(scale=

˓→"linear", colorbar="True") [18]: (, )

20 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

[19]: image.soft_threshold(threshold=0.2, relative=True, save_totalflux=False).imshow(scale=

˓→"linear", colorbar="True") [19]: (, )

1.2. Tutorial and Example Scripts 21 SMILI Documentation, Release 0.0.1

Mininum threshold is slightly different from hard thresholding, since the former resets all of pixels where their brightness is smaller than a given threshold. On the other hand, hard thresholding resets all of pixels where the absolute of their brightness is smaller than the threshold.

[20]: image.min_threshold(threshold=0.2, relative=True, save_totalflux=False).imshow(scale=

˓→"linear", colorbar="True") [20]: (, )

22 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

Save images to ﬁles

There are a couple of formats that SMILI can output

[21]: # to Image FITS file (w/ AIPS CC table equivalent with the pixel brightness

˓→information) image.to_fits("save_test.fits")

# to DIFMAP model file image.to_fits("save_test.mod")

Acknowledgement

This notebook makes use of 43 GHz VLBA data from the VLBA-BU Blazar Monitoring Program (VLBA-BU- BLAZAR; http://www.bu.edu/blazars/VLBAproject.html), funded by NASA through the Fermi Guest Investigator Program. The VLBA is an instrument of the National Radio Astronomy Observatory. The National Radio Astronomy Observatory is a facility of the National Science Foundation operated by Associated Universities, Inc.

1.2.2 Image regions

In interferometric imaging or any image processing, often we need to set regions where the intensity will be solved or to mask images. SMILI provide a class for handling geometric description of such regions.

1.2. Tutorial and Example Scripts 23 SMILI Documentation, Release 0.0.1

This notebook provides an example usage how to read, set and save image regions.

[1]:% matplotlib inline from pylab import * from smili import imdata, util

An instance of ds9 was found to be running before we could start the ’xpans’ name server. You will need to perform a bit of manual intervention in order to connect this existing ds9 to Python.

For ds9 version 5.7 and beyond, simply register the existing ds9 with the xpans name server by selecting the ds9 File->XPA->Connect menu option. Your ds9 will now be fully accessible to pyds9 (e.g., it appear in the list returned by the ds9_targets() routine).

For ds9 versions prior to 5.7, you cannot (easily) register with xpans, but you can view ds9’s File->XPA Information menu option and pass the value associated with XPA_METHOD directly to the Python DS9() constructor, e.g.:

d = DS9(’a000101:12345’)

The good news is that new instances of ds9 will be registered with xpans, and will be known to ds9_targets() and the DS9() constructor.

Here, we use an FITS image reconstructed with SMILI from a 43 GHz 3C 273 data set of the Boston University Blazar Group.

[2]: # Load the FITS image image= imdata.IMFITS("./BU_3C273_sampleimage.fits", angunit="mas")

# Plot the image in linear and log scales util.matplotlibrc(ncols=2, width=500, height=300) fig, axs= plt.subplots(ncols=2)

plt.sca(axs[0]) plt.title("3C 273 Example Image: Linear Scale") image.imshow(colorbar=True, fluxunit="mjy", cmap=cm.jet)

plt.sca(axs[1]) plt.title("3C 273 Example Image: Log Scale") image.imshow(scale="gamma", fluxunit="mjy", colorbar=True, cmap=cm.jet)

mpl.rcdefaults()

24 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

Interactively set imaging regions using SAO DS9

SMILI can intractively set regions with SAO DS9 using pyds9. To load your image to DS9, you can use open_pyds9 method of your image object.

[ ]: # Open this image in DS9 image.open_pyds9()

Or alternatively, you can create a imdata.IMRegion object and use open_pyds9 method.

[ ]: # Initialize an IMRegion object imregion= imdata.IMRegion()

# Open the image in DS9. If this imregion has already some regions, it will be sent

˓→to DS9 as well. imregion.open_pyds9(image)

You can see your image in DS9 and set regions.

1.2. Tutorial and Example Scripts 25 SMILI Documentation, Release 0.0.1

26 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

You can set a region by clicking EDIT > Region. Currently, SMILI can handle circles, squares and ellipses in DS9. Here, ellipses are set to the area with signiﬁcant ﬂux densities.

1.2. Tutorial and Example Scripts 27 SMILI Documentation, Release 0.0.1

28 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

1.2. Tutorial and Example Scripts 29 SMILI Documentation, Release 0.0.1

You can pull regions from DS9 with load_pyds9 methods in imdata.IMFITS or imdata.IMRegion objects.

[ ]: # eighther of commands will work to pull regions from DS9 imregion= image.load_pyds9() imregion= imregion.load_pyds9(image)

The imdata.IMRegion class is inheriting pandas.DataFrame. You can use all functions available for pandas.DataFrame.

[4]: imregion [4]: shape xc yc width height radius maja mina angle \ 0 circle 61.83334 80.4166 NaN NaN 214.16716 NaN NaN NaN 1 circle -98.79166 -101.6250 NaN NaN 214.16716 NaN NaN NaN 2 circle -270.12500 -305.0834 NaN NaN 239.36694 NaN NaN NaN 3 circle -377.20834 -519.2500 NaN NaN 273.55490 NaN NaN NaN 4 circle -655.62500 -562.0834 NaN NaN 283.40834 NaN NaN NaN 5 circle -912.62500 -744.1250 NaN NaN 369.12282 NaN NaN NaN 6 circle -1137.50000 -1108.2084 NaN NaN 422.75350 NaN NaN NaN

angunit 0 uas 1 uas 2 uas 3 uas 4 uas 5 uas 6 uas

In addition, it has additional functions. For instance, you can plot your regions with plot method.

[5]: # plot the original image as a reference image.imshow(colorbar=True, fluxunit="mjy", cmap=cm.jet)

# plot regions. You can use any arguments of pyplot.plot for customizing your plots. imregion.plot(color="white", angunit="mas")

30 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

You can save this region ﬁle to a csv ﬁle with to_csv method, and load it again with imdata.read_imregion function

[6]: # save this region file to a csv file imregion.to_csv("imregion.csv")

# load this region file from the csv file imregion= imdata.read_imregion("imregion.csv")

If you have a pixel-coordinated DS9 region ﬁle, you can also read it with load_ds9reg method

[ ]: # Load DS9 region file (must be using pixel-based coordinates) imregion_ds9= imdata.IMRegion() imregion_ds9= imregion_ds9.load_ds9reg(regfile="xxx.reg",image=image)

Manually set/edit imaging regions

You can also add regions from SMILI. This would be useful to pipeline imaging scripts.

[7]: # Adding a box imregion_box= imdata.IMRegion() imregion_box= imregion_box.add_box(xc=-2,yc=-2,width=2,angle=20,angunit="mas")

# Check image.imshow() imregion_box.plot(color="white")

1.2. Tutorial and Example Scripts 31 SMILI Documentation, Release 0.0.1

[8]: # Adding a circle imregion_circ= imdata.IMRegion() imregion_circ= imregion_circ.add_circle(xc=0,yc=0,radius=0.5,angunit="mas")

# Check image.imshow() imregion_circ.plot(color="white")

32 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

[9]: # Adding a circle imregion_ellip= imdata.IMRegion() imregion_ellip= imregion_ellip.add_ellipse(xc=-1,yc=-1,maja=2,mina=1,angle=20,

˓→angunit="mas")

# Check image.imshow() imregion_ellip.plot(color="white")

1.2. Tutorial and Example Scripts 33 SMILI Documentation, Release 0.0.1

There are some other useful functions to edit regions

[10]: # Shifting imaging regions. imregion_shift= imregion.copy() imregion_shift.shift(dx=-1,dy=0,angunit="mas")

# Check image.imshow(colorbar=True) imregion.plot(color="white", label="original", angunit="mas") imregion_shift.plot(color="yellow", label="shifted", angunit="mas")

34 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

[11]: # Zoom regions imregion_zoom= imregion.copy() imregion_zoom.zoom(fx=3)

# Check image.imshow(colorbar=True) imregion.plot(color="white", label="original") imregion_zoom.plot(color="yellow")

1.2. Tutorial and Example Scripts 35 SMILI Documentation, Release 0.0.1

You can also add regions. Create region1 and region2.

[12]: # Create two imaging regions imregion1= imregion.copy() imregion2= imdata.IMRegion().add_box(xc=-2,yc=-2,width=2,angle=0,angunit="mas")

# Take a sum imregion3= imregion1+ imregion2

# Check image.imshow(colorbar=True) imregion3.plot(color="white", angunit="mas")

36 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

Masking image with imregion

You can mask images with winmod methods in imdata.IMFITS/imdata.IMRegion objects.

[13]: # These commands are equivalent editimage1= imregion.winmod(image,save_totalflux=False) editimage2= image.winmod(imregion, save_totalflux=False)

# Check: indeed you can see all brightness outside regions are cleared. util.matplotlibrc(ncols=3, width=500, height=300) fig, axs= plt.subplots(ncols=3)

plt.sca(axs[0]) plt.title("3C 273 Original Image") image.imshow(scale="gamma", fluxunit="mjy", colorbar=True,cmap=cm.jet) imregion.plot(color="white", angunit="mas")

plt.sca(axs[1]) plt.title("3C 273 Masked1") editimage1.imshow(scale="gamma", fluxunit="mjy", colorbar=True, cmap=cm.jet) imregion.plot(color="white", angunit="mas")

plt.sca(axs[2]) plt.title("3C 273 Masked2") editimage2.imshow(scale="gamma", fluxunit="mjy", colorbar=True, cmap=cm.jet) (continues on next page)

1.2. Tutorial and Example Scripts 37 SMILI Documentation, Release 0.0.1

(continued from previous page) imregion.plot(color="white", angunit="mas")

mpl.rcdefaults()

You can also creat a “mask image”, which can be loaded to imaging functions as well

[14]: # Create a mask image maskimage= imregion.maskimage(image)

# Check: indeed you can see all brightness outside regions are cleared. util.matplotlibrc(ncols=2, width=500, height=300) fig, axs= plt.subplots(ncols=2)

plt.sca(axs[0]) plt.title("3C 273 Original Image") image.imshow(scale="gamma", fluxunit="mjy", colorbar=True,cmap=cm.jet) imregion.plot(color="white", angunit="mas")

plt.sca(axs[1]) plt.title("3C 273 Masked1") maskimage.imshow(scale="gamma", colorbar=True, cmap=cm.jet) imregion.plot(color="white", angunit="mas")

mpl.rcdefaults()

38 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

Acknowledgement

1.2.3 Interferometric data sets

In SMILI, interfreometric data sets are handle by uvdata module. Here, we show a basic usage of uvdata module using a VLBA data set of 3C 273 at 43 GHz of the Boston University Blazar Group.

[1]:% matplotlib inline from smili import uvdata, imdata, imaging, util

# for plotting import matplotlib.pyplot as plt

Load/Save uvﬁts ﬁle and select a stokes parameter

SMILI can load a single-source uvfits files generated by eht-imaging library, AIPS task SPLIT, and DIFMAP. uvdata.UVFITS object handles data loaded from UVFITS file.

[14]: # Include class to load uvfits data uvfits= uvdata.UVFITS("3C273DEC16.UVP") Filename: 3C273DEC16.UVP No. Name Ver Type Cards Dimensions Format 0 PRIMARY 1 GroupsHDU 155 (3, 4, 1, 4, 1, 1) float32 1913 Groups

˓→7 Parameters 1 AIPS NX 1 BinTableHDU 31 8R x 7C [1E, 1E, 1J, 1J, 1J, 1J, 1J] 2 AIPS FQ 1 BinTableHDU 29 1R x 6C [1J, 4D, 4E, 4E, 4J, 32A] 3 AIPS AN 1 BinTableHDU 72 10R x 14C [8A, 3D, 0D, 1J, 1J, 1E, 1E, 4E,

˓→ 1A, 1E, 8E, 1A, 1E, 8E]

Loading HDUs in the input UVFITS files. Primary HDU was loaded. AIPS FQ Table was loaded. Subarray 1 was found in an AIPS AN table

Checking loaded HDUs. 1 Subarray settings are found. No AIPS SU tables were found. Assuming that this is a single source UVFITS file.

Reading FQ Tables Frequency Setup ID: 1 IF Freq setups (Hz): if_freq_offset ch_bandwidth if_bandwidth sideband 0 0.0 64000000.0 64000000.0 1 1 80000000.0 64000000.0 64000000.0 1 2 144000000.0 64000000.0 64000000.0 1 3 208000000.0 64000000.0 64000000.0 1 Note: Central Frequency of ch=i at IF=j (where i,j=1,2,3...) freq(i,j) = reffreq + (i-1) * ch_bandwidth(j) * sideband + if_freq_offset(j) (continues on next page)

1.2. Tutorial and Example Scripts 39 SMILI Documentation, Release 0.0.1

(continued from previous page)

Reading AN Tables Sub Array ID: 1 Frequency Setup ID: 1 Reference Frequency: 43007500000 Hz Reference Date: 2016-12- AN Table Contents: id name x y z mnttype 0 1 BR -2.112065e+06 -3.705357e+06 4.726814e+06 0 1 2 FD -1.324009e+06 -5.332182e+06 3.231962e+06 0 2 3 HN 1.446375e+06 -4.447940e+06 4.322306e+06 0 3 4 KP -1.995679e+06 -5.037318e+06 3.357328e+06 0 4 5 LA -1.449753e+06 -4.975299e+06 3.709124e+06 0 5 6 MK -5.464075e+06 -2.495248e+06 2.148297e+06 0 6 7 NL -1.308726e+05 -4.762317e+06 4.226851e+06 0 7 8 OV -2.409150e+06 -4.478573e+06 3.838617e+06 0 8 9 PT -1.640954e+06 -5.014816e+06 3.575412e+06 0 9 10 SC 2.607849e+06 -5.488069e+06 1.932740e+06 0

Reading Source Information from Primary HDU Frequency Setup ID: 1 Sources: id source radec equinox 0 1 3C273 12h29m06.6997s +02d03m08.5983s 2000.0

Reading Primary HDU data VisData.sort: 0 indexes have wrong station orders (ant1 > ant2). VisData.sort: Data have been sorted by utc, ant1, ant2, subarray

Here are representative methods to edit uvdata.UVFITS objects. • avspc() perform weighted-average complex visibilities in frequency directions. • weightcal() perform recalculate sigmas and weights of data from scatter in complex visibilities over speciﬁed frequency and time segments.

[15]: # select Stokes I: This will create a single stokes uvfits object. uvfits_I= uvfits.select_stokes("I")

# You can edit uvfits object by using following procedures #uvfits_avspc = uvfits.avspc() #uvfits_weightcal = uvfits.weightcal() Stokes I data will be calculated from input RR and LL data

You can save uvfits files by to_uvfits instance.

[16]: uvfits_I.to_uvfits("3C273DEC16.stokesI.uvfits")

Create a Visibility Table for Imaging

SMILI’s imaging function is designed to use visibility data sets stored in two dimensional tables. There are three major table formats for (1) visibilities and amplitudues (uvdata.VisTable), (2) bi-spectram (closure phases; uvdata.BSTable), and (3) closure amplitudes (uvdata.CATable). All of tables inherit a popular pandas.DataFrame class, so you can use data table instances like pandas.DataFrame. Here we start from the visbility data table class uvdata.VisTable.

40 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

[17]: # Read visibilities from uvdata.VisTable objects vtable= uvfits_I.make_vistable()

vtable.loc[0:10, :] [17]: utc gsthour freq stokesid ifid chid \ 0 2016-12-24 09:38:44.998856 15.868862 4.300750e+10 1 0 0 1 2016-12-24 09:38:44.998856 15.868862 4.300750e+10 1 0 0 2 2016-12-24 09:38:44.998856 15.868862 4.300750e+10 1 0 0 3 2016-12-24 09:38:44.998856 15.868862 4.300750e+10 1 0 0 4 2016-12-24 09:38:44.998856 15.868862 4.300750e+10 1 0 0 5 2016-12-24 09:38:44.998856 15.868862 4.300750e+10 1 0 0 6 2016-12-24 09:38:44.998856 15.868862 4.300750e+10 1 0 0 7 2016-12-24 09:38:44.998856 15.868862 4.300750e+10 1 0 0 8 2016-12-24 09:38:44.998856 15.868862 4.300750e+10 1 0 0 9 2016-12-24 09:38:44.998856 15.868862 4.300750e+10 1 0 0 10 2016-12-24 09:38:44.998856 15.868862 4.300750e+10 1 0 0

ch u v w uvdist subarray st1 \ 0 0 6.105191e+07 2.229138e+08 -2.445611e+08 2.311231e+08 1 1 1 0 -3.265773e+08 7.199128e+07 -4.043123e+08 3.344182e+08 1 1 2 0 1.085653e+08 2.017154e+08 -1.513385e+08 2.290754e+08 1 1 3 0 4.243206e+07 1.527676e+08 -1.959372e+08 1.585510e+08 1 1 4 0 -1.230807e+08 8.189730e+07 -2.950599e+08 1.478378e+08 1 1 5 0 1.034190e+08 1.293095e+08 -5.416693e+07 1.655791e+08 1 1 6 0 6.721655e+07 1.714831e+08 -1.822360e+08 1.841861e+08 1 1 7 0 -3.604489e+08 4.221343e+08 -6.135220e+08 5.550863e+08 1 1 8 0 -3.875396e+08 -1.509233e+08 -1.597306e+08 4.158903e+08 1 2 9 0 4.751136e+07 -2.119897e+07 9.322340e+07 5.202621e+07 1 2 10 0 -1.862026e+07 -7.014639e+07 4.862363e+07 7.257569e+07 1 2

st2 st1name st2name amp phase weight sigma 0 2 BR FD 1.885444 43.039765 3.850429 0.509619 1 3 BR HN 1.089305 45.338093 2.541920 0.627219 2 4 BR KP 3.986017 36.372964 2.925850 0.584620 3 5 BR LA 3.399448 -30.419323 1.374593 0.852929 4 7 BR NL 7.270494 28.013670 5.071963 0.444030 5 8 BR OV 3.207430 -34.795414 3.058038 0.571845 6 9 BR PT 1.974857 -52.271663 6.026638 0.407345 7 10 BR SC 4.721735 -13.935714 3.301363 0.550368 8 3 FD HN 2.406819 28.885509 169.609238 0.076785 9 4 FD KP 8.917510 -21.874527 3.286803 0.551586 10 5 FD LA 5.005936 63.771332 82.484375 0.110107

By default, we are using a simple csv format to save/load tables.

[18]: # Save visibilities to a csv file vtable.to_csv("visibility.csv")

# Load visibility information from the save csv file vtable= uvdata.read_vistable("visibility.csv")

uvdata.VisTable has simple plot methods uvplot and raplot for plotting uv-coverages and radial plots. Each function is mocking pyploy.plot or pyplot.errorbar functions, and you can use almost all of arguments for pyplot.plot or pyplot.errorbar.

[19]: # You can check fundamental properties of observations and visibilities util.matplotlibrc(ncols=3, width=500, height=300) (continues on next page)

1.2. Tutorial and Example Scripts 41 SMILI Documentation, Release 0.0.1

(continued from previous page) fig, axs= plt.subplots(ncols=3)

plt.sca(axs[0]) plt.title("(u,v) coverage") vtable.uvplot() plt.xlim(1.5,-1.5) plt.ylim(1.5,-1.5)

plt.sca(axs[1]) plt.title("Radial plot of visibility amplitudes") vtable.radplot(datatype="amp",color="red")

plt.sca(axs[2]) plt.title("Radial plot of visibility phases") vtable.radplot(datatype="phase",color="blue")

uvdata.VisTable has more ﬂexible plotter vplot for various combination of data types. Available types of data are [“utc”,”gst”,”amp”,”phase”,”sigma”,”real”,”imag”,”u”,”v”,”uvd”,”snr”]. In this example, we show the utc time dependence of visibilities. You can also pick up a certain baseline information, where the available baselines are conﬁrmed by vtable.baseline_list function.

[20]: util.matplotlibrc(ncols=3, width=500, height=300) fig, axs= plt.subplots(ncols=3)

plt.sca(axs[0]) plt.title("Time development of a visibility amplitudes") vtable.vplot(axis1="utc",axis2="amp",color="red")

plt.sca(axs[1]) plt.title("Time development of a visibility phases") vtable.vplot(axis1="utc",axis2="phase",color="blue")

plt.sca(axs[2]) plt.title("Visibility amplitudes of a selected baseline ('BR','FD')") vtable.vplot(axis1="utc",axis2="amp",baseline=('BR','FD'),color="green") plt.show()

# By the way, you can get the list of baselines by baseline_list= vtable.baseline_list()

42 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

Make tables for bispectra and log closure amplitudes

From uvdata.VisTable, you can make tables for closure quantities such as bispectra (uvdata.BSTable) and log closure amplitudes (uvdata.CATable). The classes of both table also inherit pandas.DataFrame class, so you can use this class like pandas.DataFrame.

[21]: # Calculate bispectra from vistable.VisTable objects btable= vtable.make_bstable() #btable = vtable.make_bstable(redundant=[["AA", "AP"], ["JC", "SM"]]) # removing

˓→trivial closure quatitites if the array has a redundant site

# Calculate log closure amplitudes from vistable.VisTable objects ctable= vtable.make_catable() #ctable = vtable.make_catable(redundant=[["AA", "AP"], ["JC", "SM"]]) # removing

˓→trivial closure quatitites if the array has a redundant site 2%| | 141/7647 [00:00<00:05, 1401.74it/s] (1/5) Sort data (2/5) Tagging data 100%|| 7647/7647 [00:05<00:00, 1326.31it/s] 21%| | 51/248 [00:00<00:00, 505.69it/s] Number of Tags: 248 (3/5) Checking Baseline Combinations 100%|| 248/248 [00:00<00:00, 565.98it/s] 0%| | 1/248 [00:00<00:31, 7.80it/s] Detect 25 combinations for Closure Phases (4/5) Forming Closure Phases 100%|| 248/248 [00:28<00:00, 8.66it/s] 0%| | 0/7647 [00:00

1.2. Tutorial and Example Scripts 43 SMILI Documentation, Release 0.0.1

Detect 25 combinations for Closure Amplitudes (4/5) Forming Closure Amplitudes 100%|| 248/248 [00:35<00:00, 18.17it/s] (5/5) Creating CATable object

Data saving/loading funcitons are simiar to uvdata.VisTable.

[22]: # Save btable.to_csv("cphase.csv") ctable.to_csv("logcamp.csv")

# Load btable= uvdata.read_bstable("cphase.csv") ctable= uvdata.read_catable("logcamp.csv")

You can plot fundamental properties of observations and closure values.

[23]: util.matplotlibrc(ncols=2, width=500, height=300) fig, axs= plt.subplots(ncols=2)

plt.sca(axs[0]) plt.title("Radial plot of closure phases") btable.radplot(color="red")

plt.sca(axs[1]) plt.title("Radial plot of log closure amplitudes") ctable.radplot(color="blue") plt.show()

uvdata.BSTable and uvdata.CATable have vplot methods similar to uvdata.VisTable.

[24]: util.matplotlibrc(nrows=2,ncols=2, width=500, height=300) fig, axs= plt.subplots(nrows=2,ncols=2, sharex=False, sharey=False) plt.subplots_adjust(hspace=0.4)

plt.sca(axs[0,0]) plt.title("Minimum UV distance vs closure phases") btable.vplot(axis1="uvdmin", axis2="phase", color="red")

plt.sca(axs[0,1]) plt.title("Maximum UV distance vs log closure amplitudes") (continues on next page)

44 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

(continued from previous page) ctable.vplot(axis1="uvdmax", axis2="logamp", color="blue") plt.sca(axs[1,0]) plt.title("Closure phases of a triangle ('BR','FD','HN')") btable.vplot(axis1="utc", axis2="phase", triangle=('BR','FD','HN'),color="green") plt.sca(axs[1,1]) plt.title("Log closure amplitudes of a quadrature ('BR','FD','HN','KP')") ctable.vplot(axis1="utc", axis2="logamp", quadrature=('BR','FD','HN','KP'),color=

˓→"orange") plt.show()

# By the way, you can get the combinations of baselines by triangle_list= btable.triangle_list() quadrature_list= ctable.quadrature_list()

Acknowledgement

1.2.4 Imaging

We show a quick example of imaging using imaging.lbfgs module. In this notebook, we do not go to details of its theoretical backgroud nor imaging procedures (tuning parameters, iterative imaging, self-calbiration, etc.); here we focus on going through the basic functions. We also note that the imaging function has been dynamical updated, so its

1.2. Tutorial and Example Scripts 45 SMILI Documentation, Release 0.0.1

usage may change on short time scales. We use a VLBA data set of 3C 273 at 43 GHz of the Boston University Blazar Group.

[1]:% matplotlib inline from smili import uvdata,imdata,imaging,util

# for plotting import matplotlib.pyplot as plt import matplotlib.cm as cm #from IPython.display import IFrame

Closure Imaging

First, you need to load the uvﬁts data into uvdata.UVFITS object.

[2]: # Load the UVFITS and FITS for a sample image uvfits= uvdata.UVFITS("3C273DEC16.UVP").select_stokes("I").avspc() # select Stokes I

˓→and average over multiple IFs Filename: 3C273DEC16.UVP No. Name Ver Type Cards Dimensions Format 0 PRIMARY 1 GroupsHDU 155 (3, 4, 1, 4, 1, 1) float32 1913 Groups

˓→ 1A, 1E, 8E, 1A, 1E, 8E]

Loading HDUs in the input UVFITS files. Primary HDU was loaded. AIPS FQ Table was loaded. Subarray 1 was found in an AIPS AN table

Checking loaded HDUs. 1 Subarray settings are found. No AIPS SU tables were found. Assuming that this is a single source UVFITS file.

46 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

(continued from previous page) 1 2 FD -1.324009e+06 -5.332182e+06 3.231962e+06 0 2 3 HN 1.446375e+06 -4.447940e+06 4.322306e+06 0 3 4 KP -1.995679e+06 -5.037318e+06 3.357328e+06 0 4 5 LA -1.449753e+06 -4.975299e+06 3.709124e+06 0 5 6 MK -5.464075e+06 -2.495248e+06 2.148297e+06 0 6 7 NL -1.308726e+05 -4.762317e+06 4.226851e+06 0 7 8 OV -2.409150e+06 -4.478573e+06 3.838617e+06 0 8 9 PT -1.640954e+06 -5.014816e+06 3.575412e+06 0 9 10 SC 2.607849e+06 -5.488069e+06 1.932740e+06 0

Reading Source Information from Primary HDU Frequency Setup ID: 1 Sources: id source radec equinox 0 1 3C273 12h29m06.6997s +02d03m08.5983s 2000.0

Reading Primary HDU data /Users/akiyama/.pyenv/versions/anaconda2-5.3.1/lib/python2.7/site-packages/astropy/io/

˓→fits/fitsrec.py:687: UserWarning: Field ’ORBPARM’ has a repeat count of 0 in its

˓→format code, indicating an empty field. ’indicating an empty field.’.format(key)) VisData.sort: 0 indexes have wrong station orders (ant1 > ant2). VisData.sort: Data have been sorted by utc, ant1, ant2, subarray Stokes I data will be calculated from input RR and LL data

As you can see the above log, this data set has full Stokes parameters. Here, we average data over IFs and select Stokes I. Then we create visibility tables.

[3]: # Table objects of visibilities vtable= uvfits.make_vistable() # Complex Visibilities btable= vtable.make_bstable() # Bispectra ctable= vtable.make_catable() # Log closure amplitudes 5%| | 93/1912 [00:00<00:01, 923.43it/s] (1/5) Sort data (2/5) Tagging data 100%|| 1912/1912 [00:02<00:00, 880.08it/s] 32%| | 20/62 [00:00<00:00, 197.23it/s] Number of Tags: 62 (3/5) Checking Baseline Combinations 100%|| 62/62 [00:00<00:00, 208.34it/s] 2%| | 1/62 [00:00<00:11, 5.19it/s] Detect 22 combinations for Closure Phases (4/5) Forming Closure Phases 100%|| 62/62 [00:10<00:00, 5.90it/s] 5%| | 98/1912 [00:00<00:01, 971.01it/s] (5/5) Creating BSTable object (1/5) Sort data (2/5) Tagging data 100%|| 1912/1912 [00:01<00:00, 1001.59it/s] 35%| | 22/62 [00:00<00:00, 194.14it/s]

1.2. Tutorial and Example Scripts 47 SMILI Documentation, Release 0.0.1

Number of Tags: 62 (3/5) Checking Baseline Combinations 100%|| 62/62 [00:00<00:00, 241.42it/s] 0%| | 0/62 [00:00

Here, we would try a simple imaging with full closure quantities and also error-increased amplitude data sets. Full closure imaging is non-convex optimization, so therefore the resultant image will be affected by a choise of the initial image. In particular, you need non-zero (not empty image) for closure imaging, since closure imaging do not have sensitivities in the absolute position and needs a reference for a relative position. Here, we can start from a circular Gaussian, of which beam size is corresponding to the minor beam size of the synthesized beam. You can estimate beam size parameters from vistable.

[4]: beamprm= vtable.fit_beam(angunit="mas") print(beamprm) {u’minsize’: 0.1207788291535404, u’pa’: -15.238042928900784, u’majsize’: 0.

˓→6421714803233262, u’angunit’: ’mas’}

For starting imaging, we need to deﬁne the image FOV, pixel size. SMILI will do it by setting the initial image.

[5]: # This command can create an empty image. The final option uvfitsfile is optional,

˓→and can read header informations from uvfits file. initimage= imdata.IMFITS(dx=0.02, nx=128, ny=128, nxref=20, nyref=128-20, angunit=

˓→"mas", uvfitsfile="3C273DEC16.UVP")

# set beam initimage.set_beam(**beamprm)

# add a Gaussian beamprm["majsize"]= beamprm["minsize"] initimage= initimage.add_gauss(totalflux=1, **beamprm) # this can add a circular or ˓→elliptical gaussian to the image

# show imagee initimage.imshow(scale="gamma", fluxunit="mJy", saunit="pixel", colorbar=True) [5]: (, )

48 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

Before starting imaging, let’s make a table for visibility amplitudes. Since they could have residual gain errors, let’s add additional errors for residual errrors. We can copy tables for full complex visibilities and add potential systematic errors.

[6]: atable= vtable.copy() # Table for amplitudes atable= atable.add_error(error=atable["amp"] *0.05) # add 5% errors in quadrature

Then, now is the time to make the ﬁrst image! The simplest imaging function is imaging.lbfgs.imaging.

[7]: imprm={}

# Data sets to be used #imprm["vistable"]=vtable # If we use full complex visibility imprm["amptable"]=atable # use amplitudes imprm["bstable"]=btable # use closure phases imprm["catable"]=ctable # use closure amplitudes

# Regularization functions: # we have 5 reglarization functions: # l1-norm: "l1_xxxxx" # Total Variation (TV): "tv_xxxxx" # Total Squared Variation (TSV): "tsv_xxxxx" # Relative Entropy (KL divergence): "kl_xxxxx" # GS Entropy: "gs_xxxxxx" # # The most important parameter is "lambda", which is the hypter parameter of

˓→imaging. # If xx_lambda > 0: then the corresponding regularization function will be used. # # l1-norm imprm["l1_lambda"]=1. #imprm["l1_prior"]=prior image # this option will use weighted l1 regularization

˓→using prior image # # TSV imprm["tsv_lambda"]=1. #imprm["tsv_prior"]= prior image # this option will use weighted TSV regularization

˓→using prior image (continues on next page)

1.2. Tutorial and Example Scripts 49 SMILI Documentation, Release 0.0.1

(continued from previous page) # # TV imprm["tv_lambda"]=1. #imprm["tv_prior"]= prior image # this option will use weighted-TV regularization

˓→using prior image # # KL-MEM imprm["kl_lambda"]=-1. #imprm["kl_prior"]= prior image # prior image. If it is not specified, then flat

˓→prior will be adopted. # # GS-MEM imprm["gs_lambda"]=-1 #imprm["gs_prior"]= prior image # prior image. If it is not specified, then flat

˓→prior will be adopted.

# Other regularization functions for scaling amplitudes or fixing the phase center. # # Total flux regularization imprm["tfd_lambda"]=1 imprm["totalflux"]=vtable.amp.max() # # Cent-of-mass regularization imprm["cen_lambda"]=0.1 imprm["cen_alpha"]=3.0 # the power of the centroid. 1 gives the regularization using

˓→the exact center of the mass. # The higher value gives regularization

˓→using the region close to the peak of the image.

# iteration number imprm["niter"]=1000

# imaging window: if you want to use the imaging region to limit your imaging pixels. #imprm["imregion"]=imregion

Closure imaging is non-convex optimization, and sometimes we fails on some local minima with many noises. We may kill some noise artifacts by blurring, thresholding and do imaging iteratively. We may also use reweighted sparse regularizations, which can enhance sparcity of images. Here, we use blurring and also reweighting l1-norm to kill noises in the image.

[ ]: for i in range(5): if i ==0: editimage= initimage else: editimage= outimage.soft_threshold(0.01).convolve_gauss( **beamprm) # ˓→thresholding & bluring the image outimage= imaging.lbfgs.imaging(editimage, **imprm)

[9]: outimage.imshow(scale="linear", cmap=cm.afmhot, fluxunit="mJy", saunit="pixel",

˓→colorbar=True) [9]: (, )

50 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

We can also do iterative imaging with self-calibrations

[ ]: # This is the loop for selfcalibration for i in range(4): # Imaging part if i<2: # for the first two times, we will use amplitudes imprm["amptable"]= atable imprm["vistable"]= None imprm["cen_lambda"]=1 # we need the centroid regularization during closure

˓→imaging, since the absolute potition information will be lost. else: imprm["amptable"]= None imprm["vistable"]= vtable imprm["cen_lambda"]=-1 # don't forget to switch off the centroid

˓→regularizer, since now we have absolute position information from full complex

˓→visibilities.

for j in range(5): if j ==0: editimage= initimage else: editimage= outimage.soft_threshold(0.01).convolve_gauss( **beamprm). ˓→comshift(alpha=3) # thresholding & bluring the image outimage= imaging.lbfgs.imaging(editimage, **imprm)

# here we do selfcal caltable= uvfits.selfcal(outimage) uvfits= uvfits.apply_cltable(caltable)

# recreate vistable and amptable vtable= uvfits.make_vistable() atable= vtable.copy() atable= atable.add_error(error=atable.amp * 0.05)

[11]: outimage.imshow(scale="linear", cmap=cm.afmhot, fluxunit="mJy", saunit="pixel",

˓→colorbar=True) [11]: (, )

1.2. Tutorial and Example Scripts 51 SMILI Documentation, Release 0.0.1

Image inspections

You can check the goodness of ﬁtting between the image and data sets by following functions: 1. “eval_image” returns a model visibilities which is Fourier coefﬁcient of the intensity of the image. 2. “residual_image” calculates the residual between real and model visibilities.

3. “chisq_image” estimates the a (mean square) standardized residual, 휒 (휒r). These functions can be adapted to complex visibilities, bispectra, and log closure amplitudes. Here, we show an example for modeling complex visibilities.

[12]: # model complex visibilities based on an input image vtable_model= vtable.eval_image(outimage)

# residual between real and model visibilities vtable_residual= vtable.residual_image(outimage)

# You can also check the visibilies as a function of projected baseline length util.matplotlibrc(ncols=3, width=500, height=300) fig, axs= plt.subplots(ncols=3)

plt.sca(axs[0]) plt.title("Model image") outimage.imshow(cmap=cm.afmhot)

plt.sca(axs[1]) plt.title("Radial plot: Real part of Visibilities") vtable.radplot(datatype="real",label="Data",color="red") vtable_model.radplot(datatype="real",label="Model",color="blue") vtable_residual.radplot(datatype="real",label="Residual",color="green") plt.legend(loc='upper right',markerscale=1.,handlelength=0.1)

plt.sca(axs[2]) plt.title("Radial plot: Imag part of Visibilities") vtable.radplot(datatype="imag",label="Imag",color="red") (continues on next page)

52 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

(continued from previous page) vtable_model.radplot(datatype="imag",label="Model",color="blue") vtable_residual.radplot(datatype="imag",label="Residual",color="green") plt.show()

# You can estimate chisquares and reduced chisquares. print("Complex visibilities : chisquare and reduced chisquare are %1.2f and %1.2f."

˓→%(vtable.chisq_image(outimage))) print("Visibility amplitudes : chisquare and reduced chisquare are %1.2f and %1.2f."

˓→%(vtable.chisq_image(outimage,amptable=True))) Imaging Window: Not Specified. We calcurate the image on all the pixels. Imaging Window: Not Specified. We calcurate the image on all the pixels.

Imaging Window: Not Specified. We calcurate the image on all the pixels. Complex visibilities : chisquare and reduced chisquare are 12059.28 and 3.15. Imaging Window: Not Specified. We calcurate the image on all the pixels. Visibility amplitudes : chisquare and reduced chisquare are 5567.42 and 2.91.

You can also adapt the same functions to bispectra and log closure amplitudes.

[13]: # model bispectra based on an input image btable_model= btable.eval_image(outimage)

# residual between real and model bispectra btable_residual= btable.residual_image(outimage)

# model log closure amplitudes based on an input image ctable_model= ctable.eval_image(outimage)

# residual between real and model bispectra ctable_residual= ctable.residual_image(outimage)

# You can also check the visibilities as a function of projected baseline length util.matplotlibrc(ncols=3, width=500, height=300) fig, axs= plt.subplots(ncols=3)

plt.sca(axs[0]) plt.title("Model image") outimage.imshow(cmap=cm.afmhot)

plt.sca(axs[1]) plt.title("Radial plot of closure phases") btable.radplot(label="Real",color="red") btable_model.radplot(label="Model",color="blue") btable_residual.radplot(label="Residual",color="green") plt.legend(loc='upper right',markerscale=1.,handlelength=0.1) (continues on next page)

1.2. Tutorial and Example Scripts 53 SMILI Documentation, Release 0.0.1

(continued from previous page)

plt.sca(axs[2]) plt.title("Radial plot of log closure amplitudes") ctable.radplot(label="Real",color="red") ctable_model.radplot(label="Model",color="blue") ctable_residual.radplot(label="Residual",color="green") plt.show()

# You can estimate chisquares and reduced chisquares. print("Closure phases : chisquare and reduced chisquare are %1.2f and %1.2f."

˓→%(btable.chisq_image(outimage))) print("Log closure amplitudes: chisquare and reduced chisquare are %1.2f and %1.2f."

˓→%(ctable.chisq_image(outimage))) Imaging Window: Not Specified. We calcurate the image on all the pixels. Imaging Window: Not Specified. We calcurate the image on all the pixels. Imaging Window: Not Specified. We calcurate the image on all the pixels. Imaging Window: Not Specified. We calcurate the image on all the pixels.

Imaging Window: Not Specified. We calcurate the image on all the pixels. Closure phases : chisquare and reduced chisquare are 2964.93 and 2.02. Imaging Window: Not Specified. We calcurate the image on all the pixels. Log closure amplitudes: chisquare and reduced chisquare are 1799.69 and 1.28.

You can make summary pdf ﬁgures and csv ﬁle, showing residuals, residual statistics and chisquares between the image and data sets. Here are example outputs. • plot_model_fcv.pdf • plot_model_cphase.pdf • plot_model_logcamp.pdf

[ ]: # You can make a summary plot for image inspections. # Complex visibilities vtable.plot_model_fcv(outimage,filename="plot_model_fcv.pdf")

# Closure phases btable.plot_model(outimage,filename="plot_model_cphase.pdf")

# Log closure amplitudes ctable.plot_model(outimage,filename="plot_model_logcamp.pdf")

Acknowledgement

54 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

Program. The VLBA is an instrument of the National Radio Astronomy Observatory. The National Radio Astronomy Observatory is a facility of the National Science Foundation operated by Associated Universities, Inc.

1.3 License

SMILI is licensed under GPLv3 as described in LICENSE.txt. SMILI also includes some internal packages from other libraries (LBFGSB, FINUFFT, MFISTA), for which corresponding LICENSE files are included. • L-BFGS-B: http://users.iems.northwestern.edu/~nocedal/lbfgsb.html • FINUFFT: https://github.com/flatironinstitute/finufft

1.4 References

We are now preparing a software paper (Akiyama et al. in prep.) describing the latest snapshot of some new functions after Kuramochi et al. 2018. Until this paper will come out, please cite following two papers for SMILI. • Akiyama et al. 2017a, ApJ, 838, 1 • Akiyama et al. 2017b, AJ, 153, 159 There are some other related publications using SMILI. • Honma et al. 2014, PASJ, 66, 95H • Ikeda et al. 2016, PASJ, 68, 45I • Obuchi et al. 2017, PLoSO, 1288012O • Kuramochi et al. 2018, ApJ, 858, 56K

1.5 Developer Team and Related Links

1.5.1 Main members of the Developer Team

• Kazu Akiyama (Primary Developer) at NRAO/MIT Haystack Observatory • Fumie Tazaki (Japanese Only) at NAOJ • Kotaro Moriyama at NAOJ/MIT Haystack Observatory • Shiro Ikeda at the Institue of Statistical Mathematics • Mahito Sasada at Hiroshima University • Ilje Cho at Korea Astronomy and Space Science Institute • Mareki Honma at NAOJ Many collaborators and visiting/summer students at MIT Haystack Observatory and NAOJ contributed to test and develop functions.

1.3. License 55 SMILI Documentation, Release 0.0.1

1.5.2 Institution or Research Group

• Event Horizon Telescope Ofﬁcial • Event Horizon Telescope Japan Ofﬁcial (Japanese Only) • MIT Haystack Observatory • National Radio Astronomy Observatory

1.6 Acknowledgement

Development of SMILI has been ﬁnancially supported by the following programs. • Japan Society for the Promotion of Science Postdoctoral Fellowships for Research Abroad • National Radio Astronomy Observatory Jansky Fellowship program • MEXT/JSPS KAKENHI (grant Numbers 24540242, 25120007, and 25120008) • National Science Foundation Astronomy and Astrophysics Grants (AST-1440254; AST-1614868) • National Science Foundation Research Experiences for Undergraduates program • NASA Massachusetts Space Grant Consortium • University of Science and Technology: the Overseas Research Exchange program • National Research Foundation of Korea: Global PhD Fellowship Grant (NRF-2015H1A2A1033752) We also thank Michael Johnson, Andrew Chael, Katie Bouman, Chi-Kwan Chan, and many other EHT Imaging friends for fruitful discussions.

1.7 smili package

1.7.1 Subpackages smili.imdata package

Submodules smili.imdata.imdata module smili.imdata.imregion module

Module contents smili.uvdata package

Subpackages smili.uvdata.cltable package

56 Chapter 1. Table of Contents SMILI Documentation, Release 0.0.1

Submodules smili.uvdata.cltable.cltable module

Module contents smili.uvdata.uvﬁts package

Submodules smili.uvdata.uvﬁts.uvﬁts module

Module contents smili.uvdata.uvtable package

Submodules smili.uvdata.uvtable.bstable module smili.uvdata.uvtable.catable module smili.uvdata.uvtable.gvistable module smili.uvdata.uvtable.tools module smili.uvdata.uvtable.uvtable module smili.uvdata.uvtable.vistable module

Module contents

Module contents smili.imaging package

Submodules smili.imaging.lbfgs module

1.7. smili package 57 SMILI Documentation, Release 0.0.1

Module contents

1.7.2 Submodules

1.7.3 smili.util module

1.7.4 Module contents

58 Chapter 1. Table of Contents