The SP PREP Data Preparation Package for the Hinode Spectro-Polarimeter

Solar Phys (2013) 283:601–629 DOI 10.1007/s11207-012-0205-4

The SP_PREP Data Preparation Package for the Hinode Spectro-Polarimeter

B.W. Lites · K. Ichimoto

Received: 25 September 2012 / Accepted: 26 November 2012 / Published online: 4 January 2013 © Springer Science+Business Media Dordrecht 2012

Abstract The Hinode/Spectro-Polarimeter (SP) is the ﬁrst space-borne precision spectro- polarimeter for the study of solar phenomena. It is primarily intended for measuring the solar photospheric vector magnetic ﬁeld at high spatial and spectral resolution. This objec- tive requires that the data are calibrated and conditioned to a high degree of precision. We describe how the calibration package SP_PREP for the SP operates.

Keywords Instrumentation and data management · Polarization, optical

1. Overview

The Focal Plane Package (FPP) of the Hinode/Solar Optical Telescope (SOT) includes a precision spectro-polarimeter (SP) that operates at the neutral iron lines at 6302 Å. Because of the nature of this dual-beam polarimeter, the data calibration is involved and requires many steps. This article outlines the polarimeter calibration procedures that are executed by the SolarSoft IDL package SP_PREP. Descriptions of the SP, FPP, SOT, and the Hinode mission are given by Lites et al. (2012), Tsuneta et al. (2008), and Kosugi et al. (2007). We refer to Lites et al. (2012) for on-orbit performance characteristics of the SP. In the same way that the SP instrument has inherited features from the Advanced Stokes Polarimeter (ASP: Elmore et al., 1992), some of the calibration procedures developed for the SP have substantial heritage in the data-reduction procedures of the ASP. Prior to detailed design of the Hinode/SP, a proof-of-concept spectro-polarimeter was implemented at the National Solar Observatory. That instrument led to the development of the Diffraction Limited Spectro-Polarimeter (DLSP: Sankarasubramanian et al., 2006). The data-reduction procedure developed for the DLSP was formulated as a prototype for the SP data-reduction

B.W. Lites () High Altitude Observatory, National Center for Atmospheric Research, P.O. Box 3000, Boulder, CO 80307, USA e-mail: [email protected]

K. Ichimoto Hida Observatory, Kyoto University, Takayama, Gifu 506-1314, Japan 602 B.W. Lites, K. Ichimoto package described here. Even though the DLSP and SP data-reduction packages share some common elements and routines, in reality the instruments and the data they produce differ sufficiently such that most of the structure of the DLSP package had to be completely refor- mulated. A similar history is occured with the polarization calibration: the SP polarization- calibration procedure (Ichimoto et al., 2008) has some heritage in that of the ASP (Sku- manich et al., 1997), yet in the end the two procedures have little in common. Most SP observations are carried out in “operations” of typically 30 minutes or longer. The thermal characteristics of the instrument discovered shortly after launch necessitated an elaborate two-stage calibration procedure that requires two passes through each typical SP operation. The first pass finds empirical drifts of the image in the CCD focal plane during an operation and, after temporally smoothing these empirically determined drifts, they are used in the second pass to re-position the image in the focal plane and correct for gain variations uniquely associated with the drifting image. The calibration pipeline routinely processes all Hinode raw level0 SP data. The level0 data are organized into FITS files, each containing data resulting from the onboard accumulation/demodulation of a sequence of 0.1-second exposures of the CCD. Typical level0 FITS files result from accumulation of a few to more than ten seconds. In addition to fully calibrated Stokes profiles (the SP level1 data), SP_PREP provides quick-look output that is suitable for scientific analyses that require neither quantitative measures of the vector magnetic field, nor other analyses that require full spectral resolution of the Stokes profiles. This level1 processing comprises the following sequence of corrections to the data: i) digital wrap-around of the Stokes-I profiles and restoration of the spectra for onboard bit-shifting, ii) dark- and flat-field corrections, iii) removal of instrumentally induced polarization, iv) rectification of spectra so that the dispersion is along a pixel row and alignment of the spectra vertically between the two beams of the dual-beam polarimeter, v) merging of the two polarization beams from the dual-beam polarimeter, vi) correction of the spectral-line curvature, the thermal drift of the spectrum in the wavelength direction, and orbital Doppler shift, vii) rotation of the polarization frame of reference to the standard frame (+Q along solar East–West), viii) reversal of the spectrum direction so that increasing spectral pixel corresponds to the direction toward longer wavelengths, ix) compensation for residual I → Q, U, and V crosstalk, x) shifting the spectrum along pixel columns to correct for thermal flexure of the instrument in that direction, xi) correction for the slowly varying intensity response of the instrument (vignetting of the image plane) in the slit-scan direction, and xii) conversion of the data back to the same format as the unprocessed data (integers, possibly bit-shifted). This article describes in some detail the procedures employed to acquire and process the data needed for calibration (Section 2). Section 3 describes how these calibrations are applied in practice to produce the calibrated level1 Stokes-profile images and the ancillary L level1 data products such as the longitudinal and transverse “apparent flux density” [Bapp T and Bapp] as derived from the level1 Stokes profiles.

2. Determining the Data Needed for Hinode/SP Calibration This section describes the methods used to acquire and construct data needed to carry out the routine calibration of maps from the Hinode/SP. 2.1. Adjustment for Unsigned Stokes I , Bit-Shifting

Data from the SOT reformatting program are presented as level0 FITS ﬁles containing signed integers. The SP_PREP program converts these integers to 32-bit ﬂoating-point numbers for further processing. SP_PREP for the Hinode Spectro-Polarimeter 603

The reformatting program presents all four Stokes spectral images as signed 16-bit integers in a single FITS file, but unlike Stokes Q, U,andV , the Stokes-I signal is processed and compressed on-orbit as unsigned integers. Owing to the high bias level of the Stokes-I signal (Section 2.3), the Stokes-I images frequently wrap beyond the signed integer 15-bit boundary and thus may appear to have negative values when standard FITS reading programs are used. The SP_PREP procedure detects this apparent wraparound and adjusts the signal appropriately by adding 215 to Stokes I , where needed, prior to any further processing. If digital overflow occurs onboard in the unsigned Stokes-I 16-bit integers during longer exposures, it will cause double wraparound in the level0 signed integer FITS data. This double wraparound is difficult to correct for in an automatic manner. To avoid digital overflow of the summed images for longer exposures, the onboard FPP processor allows downward shifting by one bit of any of the four demodulated Stokes images (sums and differences of CCD images over one modulation cycle: a half-rotation of the rotating retarder, or eight CCD exposures) prior to summing in the onboard FPP “smart memory”. This bit-shifting is carried out most frequently for Stokes I , but it may also be done for both I and V ,or for all Stokes parameters I,Q,U,andV as indicated by the FPP keyword SPBSHFT.The SP_PREP routine restores the data to their unshifted values prior to any processing of the data. The process of bit-shifting truncates the least significant bit, leading to an average negative, non-recoverable bias of one-half of the least significant bit for each measurement summed onboard. It is possible to acquire data near the quiet-Sun center with digital overflow in Stokes I for integrations as short as 12.4 seconds (144 CCD exposures), and with the standard dark bias offset (Section 2.3). This wraparound of Stokes I is difficult to correct for in post- processing, and will contain serious artifacts when the data are taken with the usual case of significant on-orbit JPEG compression. The level1 SP data are written to FITS files in a manner identical to that of level0 data; that is, after all processing, the data are bit-shifted in the same way as the original data, then converted to 16-bit signed integers. Therefore, it is necessary to restore level1 SP data to their original form by following the same procedures for reading the data as carried out in SP_PREP, for example by using the SolarSoft routine READL1_SBSP.

2.2. Acquisition of Dark Images

The dark level of the SP cameras has a first-order influence only on the Stokes-I profiles because the Hinode/SOT onboard polarization modulation/demodulation scheme performs differences of measured intensities to arrive at Stokes Q, U,andV . In order not to bias Stokes I , the dark offset needs to be known to an accuracy specified by the science goals of each observation. Most science requirements are met if the Stokes-I levels are known to about 1 %. However, in very dark umbrae where the mean continuum intensity is below 10 % of the quiet-Sun continuum, errors of about 1 % in the dark level might result in an erroroftheStokes-I profile that could adversely affect some analyses. The SP contains no shutter, consequently there is no way to precisely measure the dark images on a frequent basis from launch to the end of the mission. Images taken well beyond the east and west limbs do not provide a pure measure of the dark bias because they are slightly contaminated by scattered light from the solar disk (see, for example, Lites et al., 2010). Many dark images were obtained in the first days of the mission prior to opening the entrance door to the SOT. Many combinations were measured of selected CCD ampli- fiers, amplifier gain, summing of pixels in the direction along the slit (hereafter indicated by 604 B.W. Lites, K. Ichimoto the “slit direction”), summing of data in the slit scan direction (hereafter “scan direction”), spectral region-of-interest (ROI), and JPEG compression. In addition to the period prior to opening the entrance door, eclipses offer another possibility for measurement of the SP dark level. For example, in the first year of operation there were five orbits of either a partial or total solar eclipse. The annual spacecraft eclipse season from approximately mid-May to early August provides another opportunity to obtain dark images. The spacecraft sees only the night-time Earth for a few minutes every orbit during this eclipse season. These eclipses have provided data suggesting the possibility of a longer-term drift of the SP CCD camera dark level (see Section 2.4). Prior to 2009, dark images from the first days of the mission prior to opening the SOT entrance door are used by SP_PREP. From 2009 onward, dark images from the spacecraft eclipse season are employed. When allowed by other planning restrictions, dark- and flat- field data are obtained annually during the eclipse season. Examination of the on-orbit data obtained prior to opening the SOT entrance door revealed that the SP camera amplifier 1, set to gain index 2, gives the lowest read noise. Throughout, we processed dark data and gathered flat-field data for this combination only, and nearly all science data taken with the SP to the time of this writing have been carried out with these parameter settings. Furthermore, the spectral lines of interest ended up very close to the center of the CCD array, so most science data presented to date cor- respond to the standard 112-pixel spectral ROI [56 – 167] on each side of the SP CCD (CCDSIDE0 and CCDSIDE1).1 There are exceptional occasions when other spectral regions available to the SP are observed, for example when studying the O I lines at 6300.3 Å for issues surrounding the O I abundance (Allende Prieto, Lambert, and Asplund, 2001; Centeno and Socas-Navarro, 2008)ortheTiI lines at 6303.7 Å (Lites et al., 1998; Beck, 2011), whose strength increases greatly in sunspots. For this reason, the standard dark/flat observations were extended in 2010 to observe the full range of wavelengths available to the SP.

2.3. Instrument Dark Offset Settings

Owing to the strong dark bias of the standard observing settings, digital overflow can occur for integrations longer than about ten seconds. It is possible to adjust the zero-level offset of the SP CCD cameras downward, thereby allowing longer integrations without overflow. Such observations have been in demand, and observations with lower offset have been carried out on demand since August 2007, and are now generally the norm for SP observations. The initial setting for this offset is indicated by header keywords CAMDACA and CAMDACB = 7. By changing this programmable parameter to 5, one achieves a camera dark bias lower by a factor of two. Dark data for the two bias levels were obtained on-orbit prior to opening the SOT entrance door. These data have been used to determine the level of adjustment of the bias needed to transform the average dark images between the CAMDACA/B = 5or 7 offset settings. This adjustment is incorporated into the SP_PREP procedure. All recent eclipse season dark/flat measurements are taken with CAMDACA/B = 5.

1For the Hinode dual-beam polarimeter, the dual-beam system separates the two polarizations near the focal plane of the spectrograph and images them separately on two sides of the SP CCD. Throughout, the two images are referred to as CCDSIDE0 and CCDSIDE1. SP_PREP for the Hinode Spectro-Polarimeter 605

Figure 1 Shown are image-averaged intensities of dark-level data taken during the spacecraft eclipse season on 18 – 19 May 2010. These data are SP normal-mode observations with an exposure time of 4.8 seconds. Ten sequences were taken during eclipse periods of successive orbits, each sequence had about four minutes duration. Left panel: symbols indicate measurements for each of the two beam images CCDSIDE0 and CCD- SIDE1. Variations are shown in Data Numbers (DN) with mean removed. Variations of CCDSIDE1 are offset downward by 30 DN for clarity. Right panel: Expanded view of the ﬁfth sequence (starting at about 01:40 UT). Here the ordinate is also presented in DN, with an offset applied to CCDSIDE1 so that variations of both CCDSIDEs may be seen together. Data containing obvious defects such as radiation hits are omitted from these presentations.

2.4. Fluctuations in the Dark Bias Level

It is of value to know if the bias of the SP camera amplifier signal undergoes significant variation with time, or with the temperature of the CCD and its readout electronics. Ex- amination of dark-level variations with temperature and time are problematic because time series of dark data are not possible to obtain on-orbit outside of the spacecraft eclipse season, and even then the duration of those measurements is limited to the few minutes of the spacecraft eclipse. Comparison of measurements of the average dark level from the various occasions where it has been monitored reveal measurable differences. Over the short term, of minutes to hours, fluctuations of about 0.2 % of the bias level are visible: Figure 1. The left panel of that figure shows an apparent slow drift of the average dark level for both CCDSIDEs, but these drifts are not exactly the same. These data were taken during 18 – 19 May 2010, and each of the ten sequences consisted of images repeated at the nominal SP “normal mode” cadence of 4.8 seconds during about four minutes. The right panel of Figure 1 presents the variation of the data sequence obtained around 1:40 UT on 19 May 2010. This sequence is representative of the other sequences shown in the left panel of this figure. The nature of this drift in the right panel is suggestive of a thermally induced bias variation of about 0.1 %. The bias also undergoes longer-term fluctuations of an as yet unexplained nature that are also apparently uncorrelated with temperature. Solar (not terrestrial) eclipse observations during the first nine months of operation seemed to indicate a slow increase in the dark bias. It was found, however, that this trend did not persist, and in November 2007 off-limb observations revealed that the bias level had apparently reverted back to its pre-first-light levels. These fluctuations can be on the order of 1 – 2 % of the bias. The bias level is on the order of half the actual solar continuum photon-signal level for CAMDACA/B = 7, and about a quarter of the photon signal level for CAMDACA/B = 5. Because there is no way at present to anticipate these changes in bias, no temporal de- pendence of the bias is applied in the SP_PREP analysis. This issue of bias drift is worthy 606 B.W. Lites, K. Ichimoto of more intensive investigation, and users of SP data should be aware that there are potential errors in the dark level that can be as much as a few percent of the disk-center continuum level.

2.5. Average Spectral–Spatial Dark Images

For SP data from 2006 – 2009, the dark data obtained prior to opening the SOT entrance door on 23 October 2006 are used to construct dark-level correction images. See Table 1 for a summary of the dark- and flat-field data applicable to each year of operation of the Hin- ode/SP. The SP camera settings corresponding to the “standard” settings (used for most of the first 1.5 years of the mission) represent the largest grouping of these data, and are therefore used to construct mean dark-level images for both normal and fast observing modes. Inspection of each image individually allows one to reject the dark images that suffer from obvious radiation tracks or electromagnetic interference. The latter is present in many of the dark images, but usually at low levels identifiable only as a subtle large-scale pattern at or below the noise level of the dark sequence Stokes-Q, U,andV images. As explained above, for 2010 and later, dark levels determined during the eclipse season are used to form the dark-correction images. Examples of the averaged dark images for normal-mode observations are presented in Figure 2. The most obvious feature of the dark images is the shift-and-repeat pattern every 16 rows; this is a result of the manufacturing procedure for the CCD. This pattern shows prominently in the display in Figure 2 because of the scaling: ± 0.5 % of the mean dark level of each image. The level of these fluctuations is very low: 0.25 % rms of the bias signal, or typically 0.2 % rms of the continuum level of the quiet-Sun intensity at disk center. The most significant difference among these data is that the 2006 pre-door-opening measurements were obtained with lossless compression, whereas the 2009 and 2010 observations were subjected to the standard on-orbit JPEG compression. The effect of the compression is a smoothing of and some ringing of the sharp single-row signal enhancement occurring every 16 pixel rows. Relative to the 2009 observations, considerably more data were available in 2010, which led to much smoother dark images seen in the rightmost image of Figure 2.

2.6. Flat-Field Data Acquisition

All flat-field corrections are extracted from on-orbit SP maps of the quiet Sun obtained very close to the center of the disk. To obtain a representation of both the slit-width variations and the CCD pixel-by-pixel gain response, typically one averages many independent realizations of quiet-Sun spectral images. This averaging is required due to the large (almost 8 %) rms contrast of the granulation as observed at the angular resolution of the combined SOT and SP. For example, to reduce the statistical fluctuation in the flat-field correction from granulation to an rms error of 0.1 % (or about 0.6 % peak-to-peak), it would be necessary to have 802 = 6400 independent realizations of the granulation. The contrast along the slit may be even higher in the wings of the solar spectral lines because of intensity-correlated Doppler shifts. Therefore even more realizations to reach the desired flat-field accuracy at all wavelengths. Early in the mission, the flat-field correction was derived from quiet-Sun normal-mode maps (obtained for scientific purposes) that cover the full range of the SP slit scanner. The successive steps are separated by only 0.16, a distance significantly smaller than the typical scale of granulation, so that nearby map positions are not statistically independent realizations of the granulation variations. For that reason, a significant number of large normal- mode maps were needed to achieve a good flat field. After the failure of the Hinode onboard SP_PREP for the Hinode Spectro-Polarimeter 607

Table 1 Summary of SP dark and ﬂat-ﬁeld data application.

Analysis period Applied data Acquisition method

Dark Level 2006 October 2006 Pre-door opening 2007 October 2006 Pre-door opening 2008 October 2006 Pre-door opening 2009 October 2006 Pre-door opening 2010 May 2010 Spacecraft eclipse 2011 July 2011 Spacecraft eclipse 2012 June/July 2012 Spacecraft eclipse

Flat Field, Standard Spectral Range 2006 2006 – 2008 Quiet Sun maps 2007 2006 – 2008 Quiet Sun maps 2008 2006 – 2008 Quiet Sun maps 2009 July 2009 Spacecraft eclipse 2010 May 2010 Spacecraft eclipse 2011 July 2011 Spacecraft eclipse 2012 June/July 2012 Spacecraft eclipse

Flat Field, Full Spectral Range 2006 July 2009 Spacecraft eclipse 2007 July 2009 Spacecraft eclipse 2008 July 2009 Spacecraft eclipse 2009 July 2009 Spacecraft eclipse 2010 July 2009 Spacecraft eclipse 2011 June/July 2012 Spacecraft eclipse 2012 June/July 2012 Spacecraft eclipse high-speed X-band telemetry in December 2007 (Shimizu, 2009), the restricted telemetry rate discourages acquisition of these large quiet-Sun maps. From 2009 onward, flat-field data are acquired in a different mode (see Table 1 for a summary of the dark and flat-field data applicable to each year of operation of the Hinode/SP). Following from flat-field procedures for the Hinode filtergraph (FG) data and similar procedures used for flat-fielding at ground-based telescopes (in particular in practice at the Dunn Solar Telescope of the National Solar Observatory in Sunspot, NM, USA and at the German VTT on Tenerife), the correlation-tracker mirror (CTM) is programmed to dither its position in two axes at a high rate, thereby rapidly sweeping the granulation image over the slit. This motion is rapid enough to effectively blur the solar image during the typical integration time of normal mode (4.8 seconds) or fast mode (3.2 seconds) SP data. To additionally ensure statistical independence of the intensity pattern, the flat-field procedure acquires a map of the dithered image with 256 positions of the slit, each separated by eight slit scan steps (1.2). This procedure permits acquisition of a good flat-field image within the current limits of the Hinode telemetry. In practice, the Hinode/SP flat-field and dark data are acquired during the Hinode terrestrial eclipse season when dark data may be obtained very close in time to those of the flat-field data. Since 2009, these data are acquired once per year. Additionally, in 2009 and 2012, dark- and flat-field data were acquired for the full 224 wavelength pixel range of the 608 B.W. Lites, K. Ichimoto

Figure 2 Averaged SP normal-mode dark images [D(λ,y)] used for correction of the SP dark bias are shown for three epochs: the on-orbit dark measurements acquired before opening the SOT entrance door in October 2006, and the spacecraft eclipse season measurements in 2009 and 2010. All displays are fractional, saturating at ± 0.5% of the mean dark level of each image. Data for imaging of both polarization beams (CCDSIDEs) are presented. The 2006 data were obtained with lossless compression, whereas in 2009 and 2010 the data were subjected to the standard JPEG Q = 75 compression.

CCD, as compared to the standard pixel range [56,167] used in most observations. These data are needed for specialized programs designed to observe the O I line at 6300.3 Å or the Ti I line at 6303.7 Å. Specialized analyses of all flat-field data are necessary to extract data needed for the three aspects of the flat-field correction, as described in the following. A flow chart depicting the process described in Sections 2.7 to 2.11 is shown in Figure 3. These processes are carried out by routines ancillary to the main SP_PREP routines.

2.7. Determining the Vignetting Function for the SP

The combined optical system of Hinode/SOT/SP imparts a slow variation of intensity of the solar image over maps constructed from SP spectra. The largest variation occurs in the slit scan direction, as is apparent at the extreme ends of the range of the SP slit-scanning mechanism, but variations along the slit occur at a lower level. This slow variation was determined early in the mission from an average of coarsely sampled quiet disk-center SP SP_PREP for the Hinode Spectro-Polarimeter 609

Figure 3 Shown is a flowchart indicating the order of processes described in Sections 2.7 to 2.11,which lead to the parameters used in the calibration of Hinode science data. maps that cover the full range of SP slit scan and slit length. Each of these maps was first adjusted to remove the known mean solar limb-darkening function (Neckel and Labs, 1994), then a seventh-order polynomial in two dimensions was fit to the average of the maps. The result is shown in Figure 4 as a contour plot superimposed upon a gray scale image of the fit. The variation of this “vignetting function” is seen in that figure to have a minimum at the extreme right side of the image of 12 % less than its value at the center of the map.

2.8. Intensity Fluctuation Along the Slit

An accurate determination of the intrinsic intensity variation along the length of the SP slit is necessary. The slit of the Hinode/SP is a narrow (12 µm) gap in a deposited reflective coating. It has significant variations in width at small scales and, as a result, the intensity at the focal plane has a correspondingly large (5 % rms) fluctuation [W(y, k),wherey is the dimension along the slit and k =[1, 2] is the index of the dual-beam polarization image] in the direction perpendicular to the spectral dispersion (see Figure 5). The difference of the intensity pattern between the two CCDSIDEs shown in the lower part of that figure reveals a smooth variation that indicates the differing response of the two CCDSIDEs. All these variations are compensated for by the calibration procedures described here. One challenge of Hinode/SP data processing is the thermal flexure of the instrument. Thermal variations during the orbit cause the image on the CCD to move by typically ten pixels along the slit direction during an orbit. Although this flexure presents a periodic variation of orbital duration, it was found that, in general, the drift could not be predicted precisely based on measured temperatures within the instrument. It was deemed necessary to 610 B.W. Lites, K. Ichimoto

Figure 4 Normalized variation of the vignetting function [Vg(x,y)] in the focal plane of the SP slit is shown as a gray-scale image. Contours are plotted at intervals of 0.01 relative to the maximum intensity. This vignetting function varies strongest along the direction of the slit scanner (x-direction).

Figure 5 Normalized variation of the intensity pattern [W(y, k)] along the slit is shown for the two sides of the CCD on which the orthogonal polarizations are imaged (CCDSIDE0: solid curve; CCDSIDE1: dashed curve). This variation arises mostly from slit-width variations. The lower dotted curve shows the difference of the variations CCDSIDE0 – CCDSIDE1, offset to 0.85; the zero difference is indicated by the long-dashed line.

measure this drift, then shift the spectra accordingly during the data-reduction process. The corresponding drift in the dispersion direction from thermal flexure may also be determined empirically. The spectral drift is a combination of Doppler shifts arising from the spacecraft orbital motion and the thermal flexure. This drift is easily determined in SP_PREP from the bulk (slit-averaged) shift of the absorption lines in the spectral dimension. Similarly, the SP_PREP routine employs the instrumentally induced intensity fluctuation pattern along the slit [W(y, k)] to measure the drift of the image in that direction. To follow the drift of the spectral image along the slit direction, it is necessary to precisely identify the location of the fixed signature of the slit width variations [W(y, k)]for the larger 8 % rms intensity fluctuations that arise from solar granulation. The empirical determination of this thermal drift along the slit relies on an accurate knowledge of W(y, k). Determining W(y, k) from on-orbit data alone would have been problematic, but using SP_PREP for the Hinode Spectro-Polarimeter 611 pre-launch sunlight calibration data (with poor seeing that made the granular contrast very low!), preliminary estimations of W(y, k) were possible. These data were then used with on-orbit data to refine W(y, k). The first steps of the analysis of a flat-field data set are to dark-correct each observation, then cross-correlate an initial guess of W(y, k) with the continuum intensity fluctuation along the slit direction [δIc(y)]. To enhance the stability of the shift determination, this 2 2 ∂ W(y,k) ∂ δIc(y) cross-correlation is performed on the numerical second derivatives of ∂y2 and ∂y2 , not on the intensities themselves. This procedure allows one to extract the temporal evolution of the thermal drift along the slit [δyth(t)] during the flat-field measurement. The shifts are temporally smoothed before s [ ] use, the smoothed version being denoted as δyth(t). The measured variations δIc(y) for s each flat-field observation are then shifted individually by the corresponding δyth(t) and averaged to arrive at a refined estimate of W(y, k). The variations W(y, k) so determined s are then archived for use in the calibration procedure for the determination of δyth(t) for each [ ] [ s ] science observation. The measured δyth(t) and smoothed δyth(t) thermal drifts along the s slit and their counterparts for the spectral drifts [δλth(t) and δλth(t)] are plotted for every SP observation sequence at output of the routine processing.2

2.9. Intrinsic Pixel-by-Pixel Gain Variation

We describe in this section the procedure to extract the intrinsic gain variation of the CCD and nearby optics. These gain variations are distinguished from the large-scale vignetting, which varies primarily with the slit scan position, and the fixed intensity variations along the slit, which shift up and down in the focal plane depending primarily upon the phase of the spacecraft orbit. However, it is necessary to remove the effects of the vignetting and slit-width variation prior to processing the flat-field data to obtain the pixel-by-pixel gain variation. Furthermore, several other properties of the spectra must be determined prior to this processing, as described in the following.

2.9.1. The Spectral Skew δy(λ)

In the SP, the spectral dispersion does not precisely follow a CCD pixel row, which slightly skews the intensity pattern caused by the slit-width variations [W(y, k)] within the spectral image. This “spectral skew” is easily determined from any SP spectral image via correlating the strong intensity fluctuations due to granulation (and to a lesser degree W(y, k)) along the slit direction of the observed Stokes-I spectrum. The observations for the flat-field measurements serve well to determine the spectral skew. For any observed Stokes-I spectral image, the variation may be determined as a function of wavelength [δy(λ)]. These functions are presumed to be linear to high order, therefore they are fit with a linear function, and the linear variations are then averaged over all the flat-field observations. The results are shown in Figure 6 for the two CCDSIDEs. For the purpose of cross-correlation, the reference variation along the slit of the intensity pattern is chosen as the average variation over spectral pixels 6 – 12 of CCDSIDE0. Also determined is the net shift of the spectrum along the slit between the two CCDSIDEs, amounting to about −2.6 pixels. This shift is shown in Figure 6. It arises within the modified Savart plate that acts as the polarizing beam splitter

2These plots along with the calibrated data may be accessed and browsed at the HAO/NCAR Hinode/SP data download site http://www.csac.hao.ucar.edu/csac/dataHostSearch.jsp by clicking on the link “thermal drift” for drift along the slit, and “line center”. The ordinates of the plots are labeled in pixels. 612 B.W. Lites, K. Ichimoto

Figure 6 Spectral skew, or orientation of the dispersion direction relative to the CCD pixel rows and columns, is shown as a function of the standard range of wavelength pixels. A shift in the spectrum of CCDSIDE1 (on the right) relative to CCDSIDE0 (left) of about −2.6 pixels is evident.

located near the focal plane of the SP. The skew has negligible temporal variation. Its year- to-year ﬂuctuation amounts to about 0.01 pixels across the typical SP wavelength range (112 pixels). The offset between the CCDSIDEs shows differences of order 0.05 pixels from year to year.

s 2.9.2. The Spectral Curvature δλ0(y) and Wavelength Shift Between CCDSIDEs δL(y)

The spectrograph imparts a curvature of the spectrum lines at the focal plane of the SP with a maximum excursion of about seven spectral pixels (see raw data images in Figure 7). This curvature may be extracted from flat-field data averaged in time over an operation (see Section 2.9.3). From this average flat image, the spectral profile, averaged over a segment near the center of the slit, is cross-correlated with each spectral row of the average flat image, thus yielding a preliminary curvature function of the spectrum. The curvature function is then refined by forming a new average spectral profile, averaged over the full length of the slit after appropriately shifting the individual profiles, then cross-correlating this new average profile again with the individual profiles along the slit. This curvature as a function of pixels along the slit [δλ0(y), see Figure 8] is fit with a fourth-order polynomial to yield s a function δλ0(y) that is needed for routine data reduction by SP_PREP. The curvature s δλ0(y) does not vary appreciably in time. The rms of the difference in δλ0(y) between 2009 and 2010 eclipse data is 0.03 pixels (Figure 8): comparable to the noise of the measurement and devoid of systematic effects. To merge the Stokes spectra from the two CCDSIDEs, it is necessary to determine the relative displacement in wavelength between the two CCDSIDEs[δL(y), see Figure 9]asa function of pixels along the slit. This quantity is determined accurately via cross-correlation of paired spectral profiles from each CCDSIDE of the average flat image. Figure 9 shows that the shift between the CCDSIDEs is about two spectral pixels, but it also differs by only s about 0.024 pixels from 2009 to 2010. The fitted δλ0(y) and the measured δL(y) from each eclipse season are archived and then used in SP_PREP to correct the data. SP_PREP for the Hinode Spectro-Polarimeter 613

Figure 7 Various stages of processing of the flat-field data are shown. Images left to right: i) single image corrected for dark level only, ii) single image with corrections for slit-width variation and large-scale vignetting, iii) average of corrected images for 244 observations of the 2009 eclipse season data, iv) the resulting 2009 flat-field image after dividing by the mean spectral profile (saturating at ± 3%),and v) the same as the previous, but for 2010 eclipse season data where 2410 separate observations were available. Data are shown for CCDSIDE0 only; results are similar for CCDSIDE1.

2.9.3. Processing Flat-Field Data to Obtain the Pixel-by-Pixel Gain Function G(λ, y, k)

The dark-corrected eclipse-season flat-field data are processed to compensate first for limb darkening, then for the large-scale vignetting, and finally for the intensity variations along the slit direction before the data are averaged. Similar to the way the slit intensity variation was determined in Section 2.8, the smoothed empirical variation of the thermal drift along s the slit δyth(t) is determined via cross-correlation of the numerical second derivatives of W(y, k) and δIc(y) for each observation in a flat-field map. The known function W(y, k) is shifted via Fourier techniques using the smoothed s δyth(t) determined as described in Section 2.8 for each of the flat-field maps, then its inverse is applied to the spectral images to remove the structure in intensity associated with slit-width variations. This procedure is somewhat complicated because the pattern W(y, k) 614 B.W. Lites, K. Ichimoto

Figure 8 Curvature of the spectrum along the length of the slit is shown [δλ0(y), thin solid curve], along with the smooth fit s to the data [δλ0(y), thin dashed curve]. The two curves are nearly indistinguishable in the plot, therefore we also plot the difference between fit and data magnified by a factor of ten (thick curve). The difference between the curvature data obtained in the 2009 and 2010 eclipse seasons (multiplied by ten) is shown, shifted downward for clarity (thick dashed lines; the straight dashed line at −4pixels is the level of zero difference). There is no significant variation of the slit curvature with time.

Figure 9 Thick solid curve shows the inferred variation of the wavelength shift between CCDSIDEs[δL(y)] as a function of pixels along the slit direction. The dashed lines show the difference of δL(y) between the 2009 and 2010 eclipse measurements; the straight dashed line at −2.10 pixels is the level of zero difference. The average difference of δL(y) between 2009 and 2010 is negligible: 0.024 pixels.

is slightly skewed along wavelength, so W(y, k)−1 is first shifted along the slit according to the skew variation δy(λ) as described in Section 2.9.1. After correcting the individual images for the effects described above, they are averaged to obtain a mean spectral image for each operation: the “average flat image”. The resulting images, one for each of the flat-field map operations, are in principle free of the intensity variations along the slit and the slowly varying limb darkening and vignetting functions. The image labeled “Avg. Flat Image” in Figure 7 presents an example of this stage of processing of the flat field. Note that the spectrum lines in that image are broadened considerably relative to the two individual images to its left, owing to averaging data over extended periods where the spectrum undergoes thermal and orbital Doppler shift. The method adopted by SP_PREP is to divide each spectrum row of the average flat image by an average spectral profile [P(λ)] to remove its spectral signature. P(λ) is constructed by shifting the spectra in wavelength according to the pre-determined spectral- SP_PREP for the Hinode Spectro-Polarimeter 615

s curvature function [δλ0(y)], then averaging the shifted profiles along the entire slit. It was found that some residual variation of the spectral profile due to solar structure is still present in the average flat images after dividing by P(λ)when the average is taken along the entire slit. These residuals arise mostly from residual variations in Doppler shift of the spectrum profile along the slit. To minimize the influence of these variations, the spectrum is divided by a 100-pixel running mean of the wavelength-shifted profile [P(λ,y)]. This procedure is similar to that described by Beck et al. (2005). The results of the division are shown as the “line free” images in Figure 7 for the 2009 and 2010 data. The 2009 image is the result of processing one flat-field map, and the 2010 image is an average over all ten maps of that eclipse season. The very small residuals of line Doppler shifts visible in the 2009 image are not seen in the more extensive data of 2010.

2.10. Acquisition of the Inverse Instrument Response Matrix X−1

The calibration of the FPP polarimeters has been described by Ichimoto et al. (2008). That work outlines the procedure that leads to a description of the instrument response matrix [X] as determined from data obtained prior to the launch of Hinode.Thema- trix X specifies the modification of the polarization state by the entire SP system (telescope/spectrograph/polarization analyzer/electronics) to produce the measured demodulated output signals. The response matrix [X] has subtle variations as a function of the spatial and spectral positions within the CCD focal plane, and also as a function of the slit-scan position. From the pre-launch data for X, the values of its inverse X−1(λ, y), to be applied to the data in order to effect the polarization calibration, were calculated at each spectral and spatial point, then fit with a low-order (cubic) polynomial in the focal plane of the CCD. The SP calibration data are then stored as a set of 4 × 4 spatial/spectral fit coefficients describing the cubic behavior for each of the 4 × 4elementsofX−1(λ, y), at each of the nine measured positions spanning the range of the slit-scanner, and for each of the two CCDSIDEs. The polarization calibration was carried out using ground-support hardware and soft- ware. The calibration was accomplished using polymer sheet polarizers that covered the entrance to the telescope. No calibration optics are incorporated into the flight system so it is not possible to repeat the polarization calibration on-orbit. The SP images for flight data are reversed in the dimension along the slit during the reformatting process to have solar North at the top of the image (as is the case for the filtergraph images). For this reason, SP_PREP reverses the recovered X−1 data top-to-bottom. Finally, X−1 is re-binned a factor of two smaller along the slit if the data that are pro- cessedaresummedinthatdirection(i.e., a fast mode).

2.11. Determination of the Residual Crosstalk Correction

After all calibration corrections described above have been applied, it is evident that there is a small amount of residual polarization in the Stokes-Q, U,andV images. This polarization is visible in the continuum outside the spectrum lines on both small and large scales. The small-scale variations appear primarily as horizontal “streaks” of polarization that affect nearly all wavelengths of the Stokes Q, U,andV . This structure is minimal, especially in Stokes V , for the standard two CCDSIDE dual-beam calibrated data, but is considerably larger in the single-beam data obtained on a regular basis from 2008 onward. See Figure 10, where these streaks are even visible in the raw Hinode/SP polarization continua prior to 616 B.W. Lites, K. Ichimoto

Figure 10 Two sets of I,Q,U,andV images on the left represent raw normal-mode SP data from CCD- SIDE0 (beam 1) and CCDSIDE1 (beam 2) obtained on 1 May 2007 at 11:26 UT. The corresponding calibrated and merged data are shown in the four images at the right.

processing. Evidently, the small-scale residual polarization is due to residual image motion causing crosstalk from Stokes I → Q, U,andV . Compensation for this source of crosstalk is described in Section 3.2.9. Another source of residual polarization appears as a large-scale variation along the slit dimension (and to a lesser extent along the spectral dimension), as illustrated in Figure 11. Its larger scale suggests that it arises from uncertainties in the instrument response matrix X. Indeed, the crosstalk from I → Q, U,andV was difficult to measure from the ground-based pre-launch calibration because a source of unpolarized light was essentially unavailable. On- orbit, one has the spectral continuum near the center of the solar disk to provide light that may be considered as essentially unpolarized for usual applications of Hinode/SP data. The flat-field calibration data provide a useful data source to determine corrections for residual I → Q, U,andV crosstalk. The flat-field data cover the full range of the slit scanner, and Figure 12 shows that the residual crosstalk depends on the position of the slit scanner in addition to varying within the focal plane of the SP. The approach taken by SP_PREP is to apply a fourth-order polynomial fit to the residual continuum polarization in the direction along the slit for each of the images Q/I , U/I,andV/I, and at each measured position of the slit scanner. The coefficients of the fit are then archived for use in correction of crosstalk of the science data (see Section 3.2.9). SP_PREP for the Hinode Spectro-Polarimeter 617

Figure 11 Stokes-Q, U,andV polarization spectral images processed by SP_PREP are shown for two observations of the flat-field calibration data taken on 19 May 2010 (as indicated at the top of the figure), but all residual crosstalk corrections are bypassed. Horizontal is the spectral dimension, vertical the length along the slit. The images are scaled to saturate at ± 0.25 %. In some images there is a noticeable gradient along the slit at all continuum wavelengths, viz. Stokes U at scan position −898, and Stokes Q at scan position 670.

The images in Figure 12 derive from recent data, and the residual crosstalk is much more prominent in Stokes Q. This has not always been the case; at the beginning of the mission, the crosstalk in Q was comparable to that of U, as shown by the plots in Figure 13.The amplitude of the crosstalk in Figure 12 is about half that shown in Figure 13 because the latter has been corrected by the instrument response matrix, which nearly doubles the signal of the raw data. It is evident from Figure 13 that the residual crosstalk in Stokes Q increases substan- tially with time. Residual polarization was nearly negligible in 2006 and 2007, typically being on the order of a few ×10−4, but it increased significantly in later years. Aging of optics, particularly oblique reflective surfaces, may decrease their reflectivity and thus cause the polarization to change. At the moment, there is no way to be certain which optical elements might be responsible for the changing instrumental polarization, or why the residual polarization presents its distinct large-scale pattern in the instrument focal plane as revealed by the Stokes-Q image in Figure 12. 618 B.W. Lites, K. Ichimoto

Figure 12 Crosstalk is shown for the raw full-size fast map obtained on 18 May 2012. Images top to bottom are Q/I , U/I,andV/I scaled so that white is 0.1 % and black is −0.1 %. Stokes Q/I shows maximum excursions approaching −0.1 %, Stokes U/I shows a barely perceptible pattern of a different shape, and Stokes V/I shows nearly no crosstalk. The strength of the pattern seen in Stokes Q/I is ﬁrst noticed to change in 2007 and has increased steadily since.

3. Routine Implementation of Calibration in SP Data

Calibration of SP data by SP_PREP requires two passes through the raw level0 data. The first pass specifies empirical, smoothed fits to the thermal drifts of the image along the slit direction and the thermal plus Doppler drifts in the dispersion direction. SP observations are commanded in operations that take minutes to hours. A series of short SP maps repeated at a single location is considered a single operation. It is commanded as a single instance of a program that is uploaded to the spacecraft. Operations usually result in a continuous series of individual FITS files, each file containing the images of the spectrum in the four Stokes parameters. SP_PREP accepts as input a list of FITS files that are considered by the program to be one continuous operation. Therefore, it is important that the data be identified with single operations prior to passing them to SP_PREP. After the smoothed fits are determined, the second pass through the data performs the actual calibration of the level0 SP data, and writes the calibrated level1 data. The functioning of these two passes through the data are described in more detail in the following, as depicted in the flow chart of Figure 14.

3.1. Pass 1: Determining the Image Drift at the CCD Camera

At the outset of this process, the archived calibration data described in Section 2 are recovered. The level0 data are ﬁrst subjected to correction for dark level, then the pixel-by-pixel SP_PREP for the Hinode Spectro-Polarimeter 619

Figure 13 Level of residual continuum polarization relative to the continuum intensity [Ic]isshownasa function of the index of the slit scanner for the full-range of the slit scanner. Rows of panels indicate the residual crosstalk at the indicated positions y along the slit. Plots in each panel represent data from [2006, 2007, 2009, 2010, and 2011]. The polarization in Stokes Q/I increases with time (curves for 2006 and 2007 nearly overlap), U/I shows a weak linear behavior that is constant in time, and no significant crosstalk is ever present in V/I. gain corrections are applied. Defective data files, such as those containing telemetry dropouts, are flagged, then rejected from further processing. s The smoothed variation of the image displacement in the direction along the slit [δyth(t)] is determined for each operation via correlation with the known slit intensity variation [W(y, k)] as summarized in Section 2.8. In the spectral direction, the drift in wavelength is the average along the slit of the minimum intensity wavelength pixel of the 6302.5 Å line [δλth(t)]. Prior to smoothing, the time series of variations are subjected to a median filter that rejects points that individually lie well beyond the local median of the data series. Further- more, spectra where the slit lies entirely outside of the limb of the solar image are identified. 620 B.W. Lites, K. Ichimoto

Figure 14 This flow chart depicts the two passes through the level0 Hinode SP science data needed for calibration. The first pass, shown in the left column under Pass 1, determines the drifts of the spectral images in the focal plane of the SP for a given science sequence (a spatial map, for example). These drifts are then smoothed temporally prior to input to the second and final pass through the data: Pass 2. Flat-field data and other data needed for calibration are determined as depicted in the flow chart of Figure 3. The processes represented in this figure are described in detail in Sections 3.1 through 3.2.10.

These data beyond the limb are not included in the smoothing. The smoothed wavelength s s drift [δλth(t)] and the smoothed drift along the slit [δyth(t)] result from “boxcar” running means of the unsmoothed variations; the smoothing window is about four minutes wide.3 During this first pass, temporal gaps in the data are identified. If a temporal gap exists of more than eight times the mean time between files, this gap defines a temporal boundary between separate, usually unplanned, segments of an SP operation. Temporal gaps occur due to a variety of situations such as saturation of the allotted SOT data volume on the onboard data recorder and loss of data during downlink. If an operation contains more than one segment, each segment is smoothed separately. Special treatment is necessary at the end of segments, details of which are not described here.

3.2. Pass 2: Production of the level1 SP Data

Upon reading each of the raw level0 data ﬁles during the second pass through the data, the basic functions of applying the digital wrap-around and bit-shift adjustments are again applied. The following sections describe the adjustments to the data in successive order.

3Data during the ﬁrst two weeks of the mission experienced much more rapid thermal movement of the image. For that period, a smoothing window of about 50 seconds was adopted. SP_PREP for the Hinode Spectro-Polarimeter 621

3.2.1. Preparation of the Polarization Calibration Matrix X−1(λ, y)

Recovery of the values of the inverse of the instrument response matrix [X−1, see Sec- tion 2.10], appropriate for SP data obtained at any position of the slit scanner, is carried out by performing a linear interpolation of the ﬁt coefﬁcients in the slit-scan direction, then reconstructing the smoothed spatial/spectral variations of X−1(λ, y) in the focal plane of the SP for the appropriate spectral ROI.

3.2.2. Dark- and Gain-Correction

The images are first corrected for dark offset with the appropriate dark images for the observation epoch (Section 2.5), then the corresponding pixel-by-pixel gain correction [G(λ, y, k), Section 2.9.3] is applied. Both the dark and flat images are adjusted for the spectral and spatial ROI on the CCD selected for each particular observation. If the observations are binned along the slit, the full-field dark and gain correction arrays are re-binned in that direction before application. Tests for image correction demonstrate that re-binning the full-field gain images to mimic fast mode observations (binning two pixels in the dimension along the slit) using the IDL REBIN procedure results in gain- or dark-corrections that are shifted upward one re-binned pixel along the slit relative to the onboard binned science data. This shift has been verified with flat-field data taken in normal and fast modes during the 2010 eclipse season. Evidently, the onboard binning procedure results in this shift. For data binned along the slit, the appropriate shifts are applied to the re-binned flat- and dark data prior to application to the science data.

3.2.3. Correction for the Slit-Width Variations

Compensation for the intensity variations arising from thermal drifts of the image at the focal plane of the CCD is carried out in the same manner as was described in Section 2.9.3 s for the flat-field data. The drift [δyth(t)] of the fixed slit-width intensity pattern [W(y, k)] used in this correction was derived in the first pass through the data, as outlined above.

3.2.4. Polarization Calibration

The inverse of the instrument polarization response matrix [X−1] extracted from ﬁt parameters representing the variation of this matrix within the ﬁeld of view of the SP (as described in Section 3.2.1) is applied directly to the data.

3.2.5. Removal of Spectral Skew and Offset Between CCDSIDEs

Spectral skew is taken into account for the correction of the slit-width intensity variation as described above, but up to this point in the processing (i.e. after the ﬂat-ﬁeld corrections and polarization calibration have been accomplished) the spectra are not corrected spatially to remove either this skew or the shift along the slit between the two CCDSIDEs (see Figure 6). The skew is removed at this point to avoid sub-pixel interpolation of the gain and polarization calibration matrices. Skew is removed by vertically shifting the individual spectral images of a map (in the direction along the slit, or perpendicular to the wavelength variation) in a fashion similar to that described by Skumanich et al. (1997). In principle, the skew indicates a rotation of the image in the focal plane, but owing to its small sub-pixel magnitude, rotation of the images would impart unnecessary interpolation in the wavelength direction. To preserve the integrity of the spectral information, interpolation is carried out in the direction along the slit only. 622 B.W. Lites, K. Ichimoto

3.2.6. Merging the CCDSIDEs

This operation sums the Stokes-I images from the two CCDSIDEs, and differences the polarization images Q, U,andV . The careful merging of the two CCDSIDE polarization images minimizes crosstalk among the Stokes parameters that arises from residual image motion (Lites, 1987; Judge et al., 2004; Casini, de Wijn, and Judge, 2012). With the spectral-skew correction applied as described in Section 3.2.5, at this point in the processing the images from the two CCDSIDEs are aligned in the direction along the slit. However, the spectral images still need to be aligned in the spectral direction. This is critical to extract the Stokes polarization profiles. In this operation the Stokes-I images from the two CCDSIDEsare aligned with the variation δL(y) discussed in Section 2.9.2 and shown in Figure 9.Resid- ual spectral shifts are observed to remain after shifting, therefore the individual CCDSIDE Stokes-I images are cross-correlated in wavelength, then the derived spectral-shift parameters along the slit are fit with a third-order polynomial – avoiding pixels that might fall within sunspot umbrae – resulting in a measurement of the residual spectral shifts [δLr(y)]. All four Stokes spectra from CCDSIDE1 are then shifted with the combination δL(y) + δLr(y). The flat-field correction does not compensate for any difference of throughput between the two CCDSIDEs. For the SP, typically CCDSIDE1 is only about 67 % of CCDSIDE0. The gain of the individual images must be very similar for the merging process to remove the effects of seeing-induced crosstalk. At this merging stage the spectral–spatial average Stokes-I intensity of each CCDSIDE is used as a measure to balance the intensities. The ratio of these average intensities is used to adjust all Stokes images from CCDSIDE1.After this renormalization of CCDSIDE1, the Stokes-I images from each CCDSIDE are summed and the Stokes-Q, U,andV images are subtracted, resulting in single merged images of Stokes I,Q,U,andV .

3.2.7. Removal of Spectral Curvature and Wavelength Drift

The four Stokes images are shifted and interpolated in wavelength via Fourier techniques to s remove the spectral curvature using the curvature function δλ0(y) that is pre-determined as s described in Section 2.9.2. At the same time, the wavelength drift of the spectrum [δλth(t)], estimated as described in Section 3.1, is removed. After this step, each spectral column of each image represents a single wavelength, and the spectrum has been shifted in bulk so that the mean position of the 6301.5 Å line falls at spectral pixel 29. The latter choice is an arbi- trary one representing a mean position of this line over many orbits. No absolute wavelength calibration is determined for SP data as part of the SP_PREP calibration procedure.

3.2.8. Rotation of the Polarization Reference Frame and Reversal of the Wavelength Direction

The reference frame for polarization in Hinode data is such that +Q is oriented along the solar East–West direction. The azimuth of the transverse component of the solar magnetic field is defined to be zero toward solar West and increases in the counter-clockwise direction (toward solar North). The SOT orientation is such that the SP slit is very nearly aligned with the spacecraft y-axis which is maintained closely along the solar North–South direction. During pre-launch polarization calibration of the SOT, the reference frame chosen was such that +Q is aligned along the axis perpendicular to the spacecraft y-axis. Thus, in nominal operation, the SP polarization reference frame corresponds to the chosen frame for the transverse magnetic field vector. The FITS header of the SOT/SP data contains the keyword SP_PREP for the Hinode Spectro-Polarimeter 623

CROTA2, which indicates the sum of the instrument rotation angle and the satellite roll angle (in degrees), which in turn speciﬁes the rotation angle of the spectrograph slit, positive clockwise relative to solar North. The SP_PREP program rotates the linear polarization parameters Q and U to bring +Q along solar East–West. Because of this, SP_PREP applies the following transformation of coordinate frame to the Stokes-Q and U images into the rectiﬁed Q and U : CROTA2 CROTA2 Q = Q cos 2π + U sin 2π , (1) 180 180 CROTA2 CROTA2 U =−Q sin 2π + U cos 2π . (2) 180 180 If the spacecraft has a substantial roll away from its nominal orientation, the SP_PREP program will therefore correct the polarization axis to its nominal value. However, up to the time of writing the spacecraft roll has always been held at its nominal value. The Stokes spectral images are next mirrored along the vertical axis so that the wavelength increases with increasing horizontal pixel number.

3.2.9. Compensation for Residual Stokes-I → Q, U, and V Crosstalk

A small amount of spurious continuum polarization, not corrected for by the instrument response matrix X, must be removed from the data. Using the fit to this crosstalk determined from the flat-field data as described in Section 2.11, the data are first adjusted to compensate for the large-scale variations following the method described by Skumanich et al. (1997). Take δSk(x, y) as the measured, residual crosstalk values, where Sk is the relative polarization of the continuum in the three polarizations k = 0, 1, 2beingQ/Ic,U/Ic,andV/Ic, respectively, for example as shown in Figure 13.There,y is the pixel count along the slit and x is the slit-scan position. The correction then applied to each of the three Stokes polarization images [Sk ] to yield the corrected images [Sk ]isgivenby = − Sk(λ,x,y) Sk(λ,x,y) δSk(x, y)I (λ, x, y). (3)

In practice, the values for δSk(x, y) determined from the flat-field data are interpolated in x before applying this correction. In normal operation, the dual-beam Hinode/SP drastically reduces the amount of Stokes- I → Q, U,andV crosstalk arising from image motion or evolution. This source of instrumental polarization is never large because of the effective operation of the onboard correlation tracker [CT]. The stability of the image is excellent and little instrumentally induced crosstalk is present. However, the CT does not remove the small amount of high-frequency jitter arising from the spacecraft momentum wheels, nor does it eliminate crosstalk arising from the evolving solar features. Since the failure of the X-band telemetry in December 2007, the SP is usually operated with a single CCDSIDE instead of dual-beam mode in order to conserve telemetry. The dramatic reduction of crosstalk among the Stokes parameters as a result of merging the two polarimeter beams is, of course, not available in the single-beam mode. For the single- as well as for the dual-beam modes, a final correction to the measured polarization is effected by the SP_PREP program. This empirical correction is applied to the SP data to reduce the amount of I → Q, U,andV crosstalk. This correction depends upon a reliable measure of the polarization in the continuum. For this purpose, the continuum is defined as all wavelengths [λc] for which the Stokes-I profile averaged along the slit that has values greater than the median for Stokes I for the image. Then one can derive the 624 B.W. Lites, K. Ichimoto

Figure 15 Stokes-Q, U,andV spectra are shown before (left) and after (right) correcting for the image-motion-induced residual crosstalk (see Equation (5)). The data are from a half-height (512 pixels along the slit) normal-mode single-sided SP map obtained on 19 September 2008. Single-sided maps typically show measurable crosstalk from image motion. The Stokes images are scaled to saturate at ± 0.4 % of the median Stokes I . The horizontal streaks of spurious polarization are removed by the procedure.

wavelength-averaged polarization-to-continuum intensity ratio for every individual Stokes spectrum of the image Sk(λ, y) dλ P (y) = λc . (4) k I (λ, y) dλ λc c On a spectrum-by-spectrum basis, the residual crosstalk from the image motion is then removed as follows: = − Sk (λ, y) Sk(λ, y) Pk(y)I (λ, y). (5) This final adjustment of the polarization is applied uniformly to all Stokes-Q and U images. Stokes V can present a problem because, within the spectral range covered by SP in its normal operation, only about 2.4 Å of the spectrum are observed, and even less than this when compensation for Doppler shifts and spectral curvature have been applied. Within sunspots and strong plage, the Stokes-V signature from the two lines included in the nominal bandpass is visible to the ends of this range when viewed at high sensitivity, making it difficult to find a “continuum” wavelength for Stokes V . However, when such strong polarization is present, the residual crosstalk correction is small enough that usually it may be ignored. Thus, SP_PREP adopts a threshold for the fractional Stokes- V polarization integrated over the entire observed wavelength range [V = |V(λ)| dλ/ I(λ)dλ] such that when V>0.02 the above correction is not applied to Stokes V . This value for the threshold V was determined empirically by examination of Stokes spectra. Figure 15 shows the effect of this residual line-by-line crosstalk correction for single- sided normal-mode data.

3.2.10. Shift of the Spectra to Compensate for Thermal Drift Along the Slit

During the ﬁrst pass through the data, the drift of the image in the direction along the slit s [δyth(t)] is determined (see Section 3.1). At this point, the data are shifted along the slit by − s an amount δyth(t). Figure 10 shows the comparison of the raw Stokes dual-beam spectra with the ﬁnal calibrated Stokes spectra for a single observation. SP_PREP for the Hinode Spectro-Polarimeter 625

3.2.11. Conversion of the Data to Bit-Shifted Integers

Before writing the data to level1 FITS ﬁles, the data are converted back to their original integer form (see Section 2.1). Furthermore, the original bit-shifting applied onboard is re- applied, if applicable. Output as integers saves a factor of two in ﬁnal data storage, but it must be remembered that often the Stokes-I images will wrap beyond the 15-bit boundary of unsigned integers, thus will appear to have negative numbers when read with a standard FITS reader. Users of SP data must be cognizant of this and the bit-shift issue, when using both level0 and level1 SP data. Two simple SolarSoft utilities, READL0_SBSP and READL1_SBSP, correctly rectify the data, and should always be used to access them.

3.2.12. Output to level1 FITS Files

Some FITS header values are updated and added as a result of processing the SP data from level0 to level1. The keywords SPWLSHFT and SPSLSHFT are added to indicate the shift (in pixels) applied in the spectral direction and along the slit, respectively, as a result of correction of the thermal drift. Keywords SPWLSFT0 and SPSLSFT0 add the thermal drift predicted in wavelength, and the unsmoothed shift along the slit [δyth, see Section 2.8], respectively. Modifications to the header are also added as a result of co-alignment of images derived from SP maps with the reference images for SOT: the FG G-band images (Centeno et al., 2009). The keyword FGSPROT is added to indicate the overall rotation of the SP slit relative to the G-band pixel columns. The value of FGSPROT is in degrees clockwise from the north end of an FG pixel column to the north end of the SP slit. The existing keywords XSCALE and YSCALE are updated to reflect the average values of the SP images as determined by Centeno et al. (2009). Similarly, the values of the solar coordinates at the center of each SP observation, XCEN and YCEN, are updated. Because the level1 images are mirrored along a vertical axis relative to level0 so that the wavelength increases with increasing spectral pixel, the dispersion keyword CDELT1 is reversed in sign. Finally, the wavelength reference keyword CRVAL1 is updated to indicate the shift of all spectra such that the Fe I line at 6301.5091 Å falls at spectral pixel 29 for the standard spectral ROI (pixels 56 – 167 on the CCD). Because the level1 SP data result from merging of the two CCDSIDEs, the level1 SP FITS data are three-dimensional [λ,y,4], as compared to the original four-dimensional level0 data [λ,y,2, 4]. One may distinguish level0 data from level1 data by the name of the FITS file. The level0 data begin with SP4D (for four-dimensional), whereas the level1 data begin with SP3D and end with C.fits.

3.2.13. Generation of the level1 Quick-Look Data Products

Table 2 presents the parameters generated by the SP level1 processing procedure SP_PREP. The procedure outputs these data as images for a given SP operation (map) in a simple IDL save-ﬁle format, named date_time_stksimg.save andstoredinthelevel1 archives. Most of the output variables are therefore images dimensioned [nx, ny] corresponding to the [horizontal, vertical] dimensions in the [scan direction, direction along slit]. Exceptions are: • CTRLINE and VCS, which are three-dimensional arrays [x,y,2]: the third dimension indexing the two Fe I lines at 6301.5 and 6302.5 Å. • The quantities associated with the processing only at each slit position are thus dimensioned [nx]: AVCTR, DOPCV, FITAV, FITWW, FTIME, SLTDR, SPCDR, and WDELW. 626 B.W. Lites, K. Ichimoto

Table 2 Hinode Spectro-Polarimeter level1 Quick-Look Data Products

Parameter Description Units

AVCTR empirical location of 6301.5 Å line center pixels − BLAPP longitudinal apparent flux density (Lites et al., 2008)Mxcm2 − BTAPP transverse apparent flux density (Lites et al., 2008)Mxcm2 CONTI continuum intensity data units CTRLINE min. int. wavelength of 6301.5 and 6302.5 Å pixels DCLNTN solar N–S disk position, North of disk center arcsec DOPCV orbital Doppler shift DOP_CVR, from header pixels MUVAL cosine of heliocentric angle FITAV smoothed empirical spectral drift correction pixels FITSP smoothed predicted spectral drift correction pixels FITWW smoothed empirical thermal drift along slit pixels FTIME Universal time from start of operation minutes PII I/Ic integrated over lines 2 + 2 PLC Q U /Ic integrated over continuum wavelengths 2 2 PLL Q + U /Ic integrated over lines POLNET measure of fractional linear polarization (Lites et al., 2008) PQ Q/Ic integrated over lines 2 2 2 PTOT Q + U + V /Ic integrated over lines PU U/Ic integrated over lines PV signed mean V/Ic integrated over lines (Lites et al., 2008) RGTASN solar E–W disk position, West of disk center arcsec RUFAZ estimate for field azimuth CCL from solar West radians SCATLP scattered light profile data units SLAT solar Carrington latitude degrees SLONG solar Carrington longitude degrees SLTDR Ichimoto prediction, thermal drift along slit pixels SPCDR Ichimoto/Kubo prediction, thermal spectral drift pixels UTIMED Universal time from start of day hours VCS Stokes V zero-crossing wavelength, 6301.5 and 6302.5 Å pixels WDELW empirical thermal drift along slit pixels

• The scattered light spectral profile [SCATLP] dimensions the number of wavelength pixels [nλ]. This profile represents an average over the regions of each map with a net polarization [variable PTOT] Ptot ≤ 0.0035. The detailed definitions of many of these level1 quick-look quantities may be found in Lites et al. (2008). The most important of these are the quantities BLAPP, BTAPP, and RU- FAZ, which represent the longitudinal apparent flux density, the transverse apparent flux L T density, and the approximate field azimuth; see Bapp, Bapp,andφr in Lites et al. (2008). The user of the quick-look data should always keep in mind that these quantities result from simple “magnetographic” line-integrated measures of the polarization and as such are no substitute for a proper inversion of the spectral-line profiles as given by the SP level2 data L T processing. Specifically, the Bapp and Bapp measures assume specific atmospheric structure appropriate for mean quiet-Sun conditions, and also assume a magnetic fill fraction of unity. SP_PREP for the Hinode Spectro-Polarimeter 627

The wavelength-integrated line polarization quantities VTOT, POLNET,andPTOT are deﬁned by Equations (1), (2), and (3), respectively, of Lites et al. (2008).

4. Discussion

The SP_PREP package has been adapted over time to function in an autonomous manner for most Hinode/SP data. However, there are some conditions under which it fails to calibrate the data properly. The first concern is whether data sets contain telemetry dropouts. Nor- mally, prior to processing by SP_PREP, the data are automatically scanned for the presence of telemetry dropouts. Telemetry dropouts were very rare during the first year of operation when the onboard X-band transmitter was operative. Since December 2007, however, dropouts are more frequent, and a significant number of dropouts are missed by the automatic dropout-finding procedures. For this reason, all data sets processed to level1 are examined individually for the presence of dropouts and other problems. The data files that contain dropouts identified either automatically or manually are eliminated from further processing, so that the level1 data simply do not contain data files with dropouts. A database registering the level0 files with telemetry dropouts is maintained and updated continuously (the “blacklist”). Other conditions may cause failure of the routine SP_PREP processing. The most frequent of these occurs when SP maps cross the solar limb at mid-latitudes. At some point in these maps most of the slit records data off the solar limb, resulting in only a short segment of the length of the slit illuminated by the solar disk. Because the thermal drift is determined by correlating the continuum intensity pattern with the fixed pattern that results from irreg- ularities of the slit width, the illuminated portion along the slit is insufficient to extract a good correlation. Another example of failed SP_PREP processing occurs when commands are issued that lead to multiple digital wrap-around of the unsigned Stokes-I two-byte integers. This condition leads to occasional negative values of the intensity that cause multiple failures of SP_PREP. Most of these extraordinary errors may be overcome by executing the processing manually on an individual case basis, but this treatment requires a significant ef- fort and attention, so correction of problems by manual processing causes significant delays in posting such data sets to the finished level1 data base. The routine and manual processing service is provided to the community through support of the Hinode program by NASA. This service will continue as long as funding is available from NASA or other sources. A significant minority of SP data is affected by these problems, but they are easily identified by visual inspection of the data output from the automatic SP_PREP processing. Therefore, if users of SP data wish to run the SP_PREP package on level0 data themselves, they need to carefully check the output from the routine for the problems discussed in this section. Even when some problems require manual processing of the data, they often cause only a slight vertical misalignment of the data for a small portion of a map, so the data of interest to the user may not be affected by the processing problem. Most of the manual processing problems are visible in the standard output plot of the thermal s drift along the slit [δyth(t)] and its smoothed counterpart [δyth(t)]. The user of data processed by SP_PREP should be aware of another source of error in the dark and flat-field correction arising from the JPEG compression/restoration of the observed spectra during its downlink. The JPEG compression is a nonlinear, nonlocal operation. Structure in the raw quiet-Sun spectra used for extraction of the flat-field images is the product of the varying high-contrast granulation and the drifting intensity pattern from the slit-width fluctuations. The resulting variable intensity pattern perpendicular to the spectral 628 B.W. Lites, K. Ichimoto dispersion is modified by the JPEG compression errors. Depending on the degree of compression, these effects limit the precision to which we may correct the compressed Stokes spectra for flat-field effects. A flat-field correction of higher precision could be derived for a less aggressive compression. We have not yet explored flat-fielding such higher-quality data because to date most SP data have JPEG compression factors Q = 75 or lower. Corre- spondingly, the dark images now routinely gathered during the spacecraft eclipse season are subjected to JPEG compression. As a result, the intrinsic dark pattern of the CCD response is modified by the compression: compare the 2006 uncompressed data of Figure 2 to those of 2009 and later that are subject to compression. Note, however, that the striking differences in Figure 2 are so clear only because of the image scaling – the shift-and-repeat pattern of the CCD results in structure of only a few DN. In conclusion, we note that the functions of the SP_PREP package are quite involved and complex, as dictated by the nature of spectro-polarimetric science data. Much of the com- plication of this processing arose as a result of the unexpectedly large thermal drifts of the image on the detector; drifts noticed only when the spacecraft was on-orbit and delivering science data. Through experience with the Hinode/SP data, and data of its ground-based pre- decessors (ASP, DLSP, and others), we believe that the nature of these data is such that data reduction packages cannot be constructed before one has access to the actual science data. Unlike ground-based missions, however, the Hinode/SP data have nearly constant properties over the life of the mission so far, so once a working data reduction package is available, automatic processing of the full data stream may proceed with confidence.

Acknowledgements We thank the many people in the USA, Japan, and the UK whose involvement in developing the Hinode mission was indispensable to its success. We also thank A. de Wijn for his critique of the manuscript prior to submission, and an anonymous referee, who provided many very helpful comments and suggestions. Hinode is a Japanese mission developed and launched by ISAS/JAXA, collaborating with NAOJ as a domestic partner, NASA and STFC (UK) as international partners. Scientiﬁc operation of the Hinode mission is conducted by the Hinode science team organized at ISAS/JAXA. This team mainly consists of scientists from institutes in the partner countries. Support for the post-launch operation is provided by JAXA and NAOJ (Japan), STFC (UK), NASA, ESA, and NSC (Norway). The FPP project at LMSAL and HAO is supported by NASA contract NNM07AA01C. The National Center for Atmospheric Research is sponsored by the National Science Foundation.

References

Allende Prieto, C., Lambert, D.L., Asplund, M.: 2001, Astrophys. J. Lett. 556, L63. doi:10.1086/322874. Beck, C.: 2011, Astron. Astrophys. 525, A133. doi:10.1051/0004-6361/201015484. Beck, C., Schmidt, W., Kentischer, T., Elmore, D.: 2005, Astron. Astrophys. 437, 1159. doi:10.1051/ 0004-6361:20052662. Casini, R., de Wijn, A.G., Judge, P.G.: 2012, Astrophys. J. 757, 45. doi:10.1088/0004-637X/757/1/45. Centeno, R., Socas-Navarro, H.: 2008, Astrophys. J. Lett. 682, L61. doi:10.1086/590405. Centeno, R., Lites, B., de Wijn, A.G., Elmore, D.: 2009, In: Lites, B., Cheung, M., Magara, T., Mariska, J., Reeves, K. (eds.) The Second Hinode Science Meeting: Beyond Discovery – Toward Understanding CS-415, Astron. Soc. Pac., San Francisco, 323. Elmore, D.F., Lites, B.W., Tomczyk, S., Skumanich, A.P., Dunn, R.B., Schuenke, J.A., Streander, K.V.,Leach, T.W., Chambellan, C.W., Hull, H.K.: 1992, In: Goldstein, D.H., Chipman, R.A. (eds.), Polarization Analysis and Measurement, SPIE Conf. Ser. 1746, 22. Ichimoto, K., Lites, B., Elmore, D., Suematsu, Y., Tsuneta, S., Katsukawa, Y., Shimizu, T., Shine, R., Tarbell, T., Title, A., Kiyohara, J., Shinoda, K., Card, G., Lecinski, A., Streander, K., Nakagiri, M., Miyashita, M., Noguchi, M., Hoffmann, C., Cruz, T.: 2008, Solar Phys. 249, 233. ADS:2008SoPh..249..233I, doi:10.1007/s11207-008-9169-9. Judge, P.G., Elmore, D.F., Lites, B.W., Keller, C.U., Rimmele, T.: 2004, Appl. Opt. 43, 3817. doi:10.1364/AO.43.003817. SP_PREP for the Hinode Spectro-Polarimeter 629

Kosugi, T., Matsuzaki, K., Sakao, T., Shimizu, T., Sone, Y., Tachikawa, S., Hashimoto, T., Minesugi, K., Ohnishi, A., Yamada, T., Tsuneta, S., Hara, H., Ichimoto, K., Suematsu, Y., Shimojo, M., Watanabe, T., Shimada, S., Davis, J.M., Hill, L.D., Owens, J.K., Title, A.M., Culhane, J.L., Harra, L.K., Doschek, G.A., Golub, L.: 2007, Solar Phys. 243,3.ADS:2007SoPh..243....3K, doi:10.1007/s11207-007-9014-6. Lites, B.W.: 1987, Appl. Opt. 26, 3838. doi:10.1364/AO.26.003838. Lites, B.W., Thomas, J.H., Bogdan, T.J., Cally, P.S.: 1998, Astrophys. J. 497, 464. doi:10.1086/305451. Lites, B.W., Kubo, M., Socas-Navarro, H., Berger, T., Frank, Z., Shine, R., Tarbell, T., Title, A., Ichimoto, K., Katsukawa, Y., Tsuneta, S., Suematsu, Y., Shimizu, T., Nagata, S.: 2008, Astrophys. J. 672, 1237. doi:10.1086/522922. Lites, B.W., Casini, R., Manso Sainz, R., Jurcák,ˇ J., Ichimoto, K., Ishikawa, R., Okamoto, T.J., Tsuneta, S., Bellot Rubio, L.: 2010, Astrophys. J. 713, 450. doi:10.1088/0004-637X/713/1/450. Lites, B.W., Akin, D.L., Card, G., Cruz, T., Duncan, D.W., Edwards, C.G., Elmore, D.E., Hoffmann, C., Katsukawa, Y., Katz, N., Kubo, M., Ichimoto, K., Shimizu, T., Shine, R., Streander, K.V., Suematsu, A., Tarbell, T., Title, A., Tsuneta, S.: 2012, Solar Phys. doi:10.1007/s11207-012-0206-3. Neckel, H., Labs, D.: 1994, Solar Phys. 153, 91. ADS:1994SoPh..153...91N, doi:10.1007/BF00712494. Sankarasubramanian, K., Lites, B., Gullixson, C., Elmore, D., Hegwer, S., Streander, K., Rimmele, T., Fletcher, S., Gregory, S., Sigwarth, M.: 2006, In: Casini, R., Lites, B.W. (eds.) Solar Polarization 4 CS-358, Astron. Soc. Pac., San Francisco, 201. Shimizu, T.: 2009, In: Lites, B., Cheung, M., Magara, T., Mariska, J., Reeves, K. (eds.) The Second Hinode Science Meeting: Beyond Discovery – Toward Understanding CS-415, Astron. Soc. Pac., San Fran- cisco, 1. Skumanich, A., Lites, B.W., Martinez Pillet, V., Seagraves, P.: 1997, Astrophys. J. 110, 357. doi:10.1086/313004. Tsuneta, S., Ichimoto, K., Katsukawa, Y., Nagata, S., Otsubo, M., Shimizu, T., Suematsu, Y., Nakagiri, M., Noguchi, M., Tarbell, T., Title, A., Shine, R., Rosenberg, W., Hoffmann, C., Jurcevich, B., Kushner, G., Levay, M., Lites, B., Elmore, D., Matsushita, T., Kawaguchi, N., Saito, H., Mikami, I., Hill, L.D., Owens, J.K.: 2008, Solar Phys. 249, 167. ADS:2008SoPh..249..167T, doi:10.1007/s11207-008-9174-z.