Xexanoth

What's Changed

• Remove depreciated code and files from repo
• generalized apo2d code to remove apo name references and replace with sos
• add code to create quick plots of SOS pipeline extractions (for qa - not run automatically)
• add code to parse and monitor SOS runtimes
• move python code into internal boss_drp python module
• add code to support ToOs
• Add daily_log code base to produce daily logging HTML status pages
• Update file organization of output files (field groups and new organization structure)
• Added plot_QA monitoring code to idlspec2d
• Add code to use arcs to help trace (determine shift between flat and field by modeling shift in arcs between the 2 times and positions
• adjust maximum sky-line shift correction from 3.5 to 5.5 (to help correct bad wavelength calibrations)
• add readthedocs documentation
• add fits extension names to files files produced by the idl code
• Add code to calculate fiber off amounts (on sky and in the wok)
• Add code to convert ps qa plots to pdf
• Add code to produce flat field throughput QA plots
• add code to reject bad calibrations in planning reductions

Full Changelog: v6_1_3...v6_2_0

Kaalut

• Fixed general bugs
• Fix bug to set delta_ra/delta_dec in the spall_lite file
• Fixed offset_fiber flag not being set
• Reset to fits table null header flags
• fixed fibermask (and ZWARNING) bitmasks for plate data

Taranushi

• Fixed coadded metadata
• fixed Healpix path (healpixgrp) columns to use integer healpix group values
• Added fits columns descriptions to fits headers
• Fixed SDSS_id bug in SDSSV plates
• added targeting flags
• fixed SDSSV plate (field) epoch coadds bug
  • only coadd a single plugging
• streamlined running of full pipeline catchup runs
• added model column back to specfull and speclite files
• added observatory specific custom coadds
• added code to update targeting flags within the spAll files

Olkoth

• Added SimpleDust2023 preliminary dustmap
• Added dither flag to spplan files
• updates to spcalib to save additional information and help with calibration library tests
• Added check of hartmann door status
• Migrated parts of pipeline to python
  • spplan
  • readfibermap - as an initial step to the pipeline (updated readplugmap.pro for updated format)
  • fieldmerge - builds on previous spAll file (does not start from scratch daily)
  • fieldlist
  • spSpec_reformat - converts spSpec files to spec files (and produces pngs)
• Added basic fits datamodel
  • par files with columns
  • python/idl codes to handle models
• updated sos to add a second SN2 calculation
• updated for sdsscore migration
• added codes for FPS epoch and custom (target level) coadds
• fix bug with combining resolution arrays
• Add additional qa plots and files (with debug flag)
• Updated LCO configuration for updated Flat lamps
• inclusion of SDSS_ID

Daoloth

• updated SOS for LCO
• Updated SOS controller to the systemd (systemctl) standard 
  - optimize SOS calibration matching
  - clean old SOS warnings when rerunning an exposure
  - clarification of Arc Lamp Status messages
• removed photolog dependency
• Update idlutils dependency to v6.X.X (SDSSV version)
• updates to add LCO location
  - Add an option to override flat/arc rejection
• update main pipeline for LCO
  - add observatory to final products
  - fixed double counting of Exposure time
  - add an option to run pipeline without SDSS-V database access
  - add an option to use arc from another field
  - update Arc lamp line file, opFibers, and gains for LCO
  - migrate from bundled fiber setup to monolithic slit
  - trim early LCO blue data where there is no flat lamp coverage
• fix bad line labels in PNGS
• migrated uurundaily and uubatchpbs to python
• setup to run fieldmerge separately from daily run
• update sphdrfix for SDSS-V (in SDSSCORE)
  - add sdR_hdrfix.py 
• Fix bug in spall fils where models were incorrect
• remove offset fibers and "bhm_gua*" targets from SN2
• Correct v0.5 magnitudes with RevisedMagnitude Table
• fix bugs in healpix columns
• fix bug in spField headers
• Add BOSS_log function for night logs
• Add GAIA_DR2 ids

Abhoth

• Updates to SOS and idlspec2d for LCO operation
  - r1/b1 vs r2/b2
  - pass LCO lat/long/alt option to pass to heliocentric/barycentric velocity correction
  - determine LCO vs APO at plan level and pass that to the pipeline
• Updated readplugmap & prerun_readplugmap
  - Update to fibermap fits formats (added a summary extension with fibermap metadata)
  - Merged FPS and plate fibermap datamodels
  - modify to use parallax and propermotion from fps configuration files, otherwise get from database
  - fixed bugs to properly read fibermap from fits file if available rather then recreating it
  - clean plugmap
  - add code to get survey name for plates
• Automate selection of parameters/options for pipeline for plate vs fps data
• added wrapper for yanny_par (yanny_par_fc) to add fold_case for string matching
• added v_rad fits header keyword and modified helio_rv description to conform with the definition of heliocentric vs barycentric RVs.
  - New keyword (v_rad) was chosen to be consistent between APOGEE, ASTRA, and BOSS
• Exclude Nan mags (mag < -99) for SOS SN2 fits
• Changed sdss_spec_image error bar shading code to address shading errors
• Add columns to spall for
  - catalogIDs from various catalog crossmatch versions
  - delta_RA_LIST and delta_DEC_LIST
  - SPEC2* (LCO)
• Updated fieldlist
  - fixed bugs to fill out fieldlist dat file
  - add columns associated with b2/r2 (LCO)
• Corrected condition meta data for spall for single exposures
• Updated spplan2d and spplan1d
  - Automatically determine plate/legacy/fps data
  - merged plate and fps spplan2d/1d models
  - Add observatory header keyword
  - Updated code to filter out engineering data and reference baddata.par to flag other bad data
  - add options to exclude dithers
• updated daily run scripts to enable LCO master and give option to have a 2nd daily run (if necessary)
• update run_plugmap_supplements
  - corrected bug in get_RJCE_ext
  - get field cadence information from designID rather then fieldid
• updated run_PyXCSAO to use splog.py (python version of idl splog) to enable live logging

Ghroth

We implement a number of changes to fix bugs and small changes to both the main IDLspec2d pipeline and the SOS mountain pipeline

• Update spplan2d
  - to utilize twilight flats (if no field-mjd matched flats are available)
  - allow skip of commission fields
  - allow test exposures with field = -999 to run with fieldid=0
• Addition of a prototype spAll-lite file
• Addition of spectrophotometric QA plots on a field-mjd basis
• Fixed a bug in Extinction values from dust3d model
  - The addition of columns identifying utilized extinction source (EBV_TYPE)
  - Updated SFD_EBV column name in spall (and other steps) to EBV
• Corrected bug in loadSN2Value to allow updates of SN2 values
• Updated APO red CCD gain values to SDSS-V updated CCD
  - modified sdssproc to read these from a new opGain.par file to clean up and allow for easier future updates
• Added LCO arc lamp line file (lampHeNe.dat) and included an option to use in fitarcimage.pro
• modified SN2 and fieldquality color/values to be consistent
  - both in fieldlist and spall
  - scales SN2 of fields with < 3600s exposure to 3600s before comparing with reference values of SN2_b <10 and SN2_r<22
• Updated Fieldlist
  - added fieldcadence (after retrieving from targetdb via run_plugmap_supplements)
• Updated spAll
  - RACAT, DECCAT, MJD_FINAL are reported as double
  - additional decimal point included in RA/DEC_LIST
  - sfd_ebv → ebv
  - added ebv_type
  - added cadence
  - Implemented an option to skip the step to build specprimary and nspecobs columns (but still allow as post step)
  - optimized the code to produce specprimary, nspecobs, etc.
  - added FIBER_RA, FIBER_DEC, and specobjid (required for sas_mwm_healpix)
  - specobjid is actually reproduced in this step (to ensure consistency
  - copying them from the specreduce1d step outputs was producing inconsistent values due to data types)
• Utilizing astrometry values from target_db instead of crossmatching to Gaia for retrieval
• modified read_sos.py to get parent confSummary files (for dithered fields) if it is not included in the fibermap fits file
• Cleaned up fibermap/spall columns
  - replacing -999. with NANs
• fixed bug with utilizing fibermask for calibrations, when more than 1 calibration frame of type is included in the plan

Bast

We implement a modification to the daily target level coadd algorithm. This modification splits the coadd into a two-stage process, with the first stage combining the blue and red camera (including outlier rejection), and the second stage combining the (already red+blue combined) exposures (without outlier rejection). This strategy was implemented to improve the coadds of exposures with significant flux differences seen in the SDSS-V FPS era. EXP_DISP_MED was added to SpAll to characterize the flux dispersion between the exposures. All exposures are included as extensions in the full versions of the spec files for custom post-pipeline coadds. Additional columns of lists of exposure level metadata are included in the SpAll file for reference. The reading of the confSummary files was replaced with the confSummaryF files to incorporate the more accurate fiber positions (as measured by the FVC loop). Additionally, several other bug fixes were incorporated into the main IDLspec2d and SOS pipelines.

Hypnos

We implement several bug fixes to the flux calibration and flat fielding of the reduced spectra. We also implement improvements to the final spectral metadata included in the spec-field-mjd-catalogid.fits file headers and the spAll extensions and file. These improvements included propagating some information on a per-target rather than per-field basis since we coadd on a target level. We switch from coadding via an RA-DEC match to a coadd based on catalogid. We also switched off the interpolated flux calibration vectors for all fields with less than 60 standards (in practice only leaving this on for BHM-RM fields).

Amaterasu

The initial version of the idlspec2d and SOS for the FPS.

• Implement the new confSummary files for FPS in place of the plate plugmaps.
• Optimize the readplugmap for multiple calls
• Implement single MJD coadd of all observations of a field
• Correct specobjid data types to produce correct output
• Implement gaia dust extinction maps for galactic latitudes <15
• Replaced reference to plate with fields
  fieldID < 16000 are plates
  16000 < fieldID < 100000 are FPS commissioning Fields
  fieldID > 100000 are FPS Survey Fields
  fieldID strings are zero-padded 6-character strings
• Clarify output magnitudes are fiber2mags