STEREO-pipeline -- post-processing for STEREO/SECCHI
This is a collection of routines that comprise a STEREO spacecraft post-processing pipeline, to model (and remove) background sources from the STEREO heliospheric imaging data. It is kept as a collection of AutoLoader files in part because that style of development has become the norm in the solar physics community due to the widespread adoption of SolarSoft (http://ssw.lmsal.com), and in part because some of the tools are general-purpose enough that hiding them in a module could be counterproductive. To use the package, you must first install Perl and PDL, then unpack the package into your file system and add an entry +<location>/STEREO-pipeline to your @PDLLIB variable inside PDL (where <location> is the location into which you unpack the pipeline).
The routines accept Level 1 (photometrically calibrated) synoptic data from the STEREO/SECCHI instruments and extract from them a background model that is then subtracted and discarded. They produce Thomson-scattering images of the transient solar wind and transient features within it. Typical background rejection is 3-4 orders of magnitude (i.e. the background is reduced from a few x 10^-11 mean solar radiances to under 10^-14 mean solar radiances in the HI-1 pipeline). In the case of data from STEREO-A HI-2, this means that output noise levels are within a factor of ~3 of the photon noise floor of the data.
The routines generally manipulate batches of data, because spatiotemporal differences between the individual images are used to separate background from desired features. The batches are merged after production, to produce individual Level 2 output files.
In standard usage, the pipeline organizes HI-1 data into 7 day batches that are staggered at 1/2 day intervals. The batch epoch is midnight UT on 1-Jan-2007, some four months before the official start of science data, and Batch 0 begins at that time and lasts for 7 days. Batch 1 begins at noon UT on 1-Jan-2007; batch 730 begins at midnight on 1-Jan-2008. The batches are merged after processing, to produce near-full-frame data with similar filenames as the original L1 data that were accepted for use in the pipeline. The filenames are modified to reflect the Level 2 status of the files.
Most of the routines use a rudimentary options hash to allow adjustment of the algorithm's behavior. If you do not specify an options hash, or omit an option, then sensible defaults are applied. Where one routine calls another, the options hash is "passed through". The options are case sensitive and may not be abbreviated (which is why it is called "rudimentary"). See the individual modules' documentation for a description of options accepted by each.
Here is a list of the routines that are included in the package, with a brief description of each. The first few are the main entry points; after that, they are listed in alphabetical order.
This is the main entry point to process an "official" batch for the pipeline. It accepts a batch number and an options hash, attempts to locate all Level 1 data within the time for that batch, adn processes them with hi1_pipeline.
This is the main HI-1 processing pipeline. It accepts a list ref containing the names of all the files to process, and an options hash (optional). The output files are deposited by default into the subdir "processed" of the current working directory.
The metapipeline processes a tranche of unofficial batches of files. It was used for the initial work (in 2008-Dec) using the pipeline algorithms, and was necessary because the starfield distortion-map algorithm was quite slow at first and could not be run for each batch. As of 2013, the distortion-map algorithm is fast enough to be run independently for each batch of files, but the metapipeline is retained both for reference and in case it proves useful.
As of Winter 2013, the pipeline has been tested on HI-1A and HI-2A data. It is likely to work on HI-1B data (and testing will proceed in Spring 2013) but will not work without modification on HI-2B data due to the different focus characteristics of that instrument.
The final data product represents photometric excess brightness of moving objects only: as with an unpolarized coronagraph, it is not possible to separate the steady F coronal background from a hypothetical steady K coronal component -- and therefore the brightness in the output .FITS files represents only excess brightness above a quasi-steady background level that contains both F and some K. For certain bright features (e.g. coronal mass ejections or corotating interaction regions in the solar wind) the background is negligible; for others (e.g. small "puffs" or plasmoids in the solar wind) it is not. The amplitude of bright features is photometrically correct to within the limits of the instrument photometric calculation and may be used to calculate, e.g., excess electron surface densities or variations therein - but, for example, the brightnesses of two faint features separated by more than a few days in time may not be compared meaningfully and directly.
The background-subtracted images also contain many artifacts, and the user is warned to watch for them. In particular:
Stellar "telegraph" print-through
Pointlike sources undergo small fluctuations in apparent brightness as they cross the pixel grid. This effect is suppressed for faint stars, but bright stars' and planets' apparent brightness fluctuates enough (in an absolute sense) to remain once the primary bright point of the planet is removed.
Fourier motion-filter artifacts
Venus and Mercury can, at certain phases, produce long-lasting artifacts that extend through space and time. The artifacts can include converging and/or diverging motions or persistent bright spots. The artifacts are visually recognizable and should be avoided.
Dust events from the spacecraft can produce single-frame bright streaks or several-frame bright points. Data dropouts produce single-frame black rectangles. Both types of event are amplified by the Fourier motion filter, and can extend through several frames. These artifacts are recognizable and should be avoided.
In HI-2A images, the Earth occulter at the left side of the frame can contain strangely moving artifacts. These are not masked out, but should be ignored.
Starfield motion artifacts
The pipeline relies on steady orbital motion and pointing from the spacecraft, to identify the starfield. It attempts to recognize and avoid pointing shifts such as the occasional roll maneuvers, but can be fooled by the occasional pointing excursions that are seen throughout the mission. When small, these excursions are compensated by the pipeline - but when large, they cause persistent starfield-like artifacts that last for one batch time (7 days).
If you have questions, concerns, or bug reports, please contact the author at the e-mail address below, or submit an Issue to the github site.
Copyright (c) 2011-2013 Craig DeForest
This file may be modified and/or distributed under either the Gnu Public License (v2) or the Perl Artistic license.
The author thanks the STEREO mission and the SECCHI teams at the U.K.'s Rutherford Appleton Labs and the U.S.'s Naval Research Laboratory for making their data available; and the Perl Data Language development team for their tireless work on improving the language. Several individuals contributed greatly to the scientific effort embodied in this work. Of particular note are T. Howard, J. Tappin, and C. Eyles.
The work to develop this image processing pipeline was funded through multiple grants from NASA and the NSF. The underlying algorithm is documented in an Astrophysical Journal article.