notes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## data reduction pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Files\n",
    "\n",
    "axis_info.parquet - time step information \n",
    "\n",
    "train/test/planet_id/ARIS-CHO_signam.parquet \n",
    "\n",
    "train/test/planet_id/FGS1_signam.parquet \n",
    "\n",
    "train/test_adc_info.csv - instrumental calibration (gain, offset) information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### FSG1\n",
    "\n",
    "- File 135000 rows at 0.1s time intervals, flattened to 1024 collums\n",
    "- un-flatten numpy.reshape(135000, 32, 32)\n",
    "- multiply by matched gain [train/test_adc_info.csv]\n",
    "- add offset [train/test_adc_info.csv]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ARIS-CH0\n",
    "\n",
    "- File 11250 rows at time intervals specified in [axis_info.parquet], flattened to 1024 collums\n",
    "- un-flatten numpy.reshape(11250, 32, 356)\n",
    "- multiply by matched gain [train/test_adc_info.csv]\n",
    "- add offset [train/test_adc_info.csv]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Callibration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Callibration files\n",
    "\n",
    "$image = gain * \\frac{exposure - bias - (dark*Delta_t)}{flat - bias} + offset$\n",
    "\n",
    "- exposure: \n",
    "- dark: [train/test]/[planet_id]/[ARIS-CHO/FSG1]_calibration/dark.parquet\n",
    "- flat: [train/test]/[planet_id]/[ARIS-CHO/FSG1]_calibration/flat.parquet\n",
    "- bias: [train/test]/[planet_id]/[ARIS-CHO/FSG1]_calibration/read.parquet\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- ignore all dead pixels from [train/test]/[planet_id]/[ARIS-CHO/FSG1]_calibration/dead.parquet.\n",
    "\n",
    "        fit spectrum with masked dead pixels and then set dead pixel values to continuum?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- correct to linear response using polynomial given in [train/test]/[planet_id]/[ARIS-CHO/FSG1]_calibration/linear_corr.parquet."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GPT Idea"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Summary Flowchart\n",
    "\n",
    "\t1.\tBias Subtraction: \n",
    "\tBias_Corrected Data = Raw Data - Master Bias Frame\n",
    "\n",
    "\t2.\tDark Current Correction:\n",
    "\tDark_Corrected Data = Bias_Corrected Data - (Master Dark Frame * delta_t*)\n",
    "\n",
    "\t3.\tGain Correction:\n",
    "\tGain_Corrected Data = (Dark_Corrected Data * Gain) + Offset\n",
    "\tDivide if units of gain in ADU/e\n",
    "\tTimes if units of gain in e/ADU\n",
    "\n",
    "\t4.\tLinearity Correction:\n",
    "\tLinear_Corrected Data = f(Gain_corrected Data[i,j])\n",
    "\n",
    "\t5.\tFlat Field Correction:\n",
    "\tFlat_field_corrected data = Linear_Corrected Data/Normalised Master Flat Field\n",
    "\n",
    "\t6.\tDead Pixel Correction\n",
    "\tDon't mask - median pixels\n",
    "\n",
    "\t7.\t(Optional) Cosmic Ray Removal\n",
    "\tOutlier detection\n",
    "\n",
    "\t8.\t(Optional) Background Subtraction\n",
    "\tBackground_corrected Data = Flat_field_corrected Data - Background level\n",
    "\n",
    "\t9.\tWavelength Calibration\n",
    "\tAccurate wavelength values to pixels (Done?)\n",
    "\tWavlength_calibrated_data = Wavelenght function\n",
    "\n",
    "\t10.\t(Optional) Flux Calibration\n",
    "\tFlux_calibration data = Wavelength_calibrated/Instrument Response function\n",
    "\t\n",
    "\t11.\tQuality Checks\n",
    "\t12.\tData Saving and Documentation\n",
    "\t13.\tProceed to Data Analysis"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}