Merge pull request #159 from mperrin/docs_stage_steps

Add more documentation

docs/source/Installation-and-dependencies.rst

@@ -32,5 +32,4 @@ With the anaconda environment created, move to the cloned directory and install
 	python make_psfmasks.py
 
 
-
 Finally, and very importantly, you will need to download reference files to support the functioning of the :code:`webbpsf` and :code:`webbpsf_ext`. Instructions to do this can be found at the respective package websites (`WebbPSF <https://webbpsf.readthedocs.io/en/latest/installation.html#installing-the-required-data-files>`_, `webbpsf_ext <https://github.com/JarronL/webbpsf_ext>`_). Ensure that if you edit your .bashrc file that you reopen and close your terminal to fully apply the changes (:code:`source ~/.bashrc` or :code:`source ~/.zshrc` may also work)

docs/source/imagetools.rst

@@ -1,6 +1,89 @@
 .. _imagetools:
 
 Image Tools
------------
+============
 
 TBD: Detailed documentation of the ``spaceklip.imagetools`` module will go here.
+
+
+
+Steps for background removal and pixel cleaning
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+subtract_background
+--------------------
+See :py:meth:`~spaceKLIP.imagetools.ImageTools.subtract_background` docstring for details. 
+
+If you have dedicated background exposures (exp type=SCI_BG or REF_BG), subtract those from your SCI or REF data. If there are multiple ints in the BG exposure, they are median-combined before the subtraction. 
+
+subtract_median
+--------------------
+See :meth:`~spaceKLIP.imagetools.ImageTools.subtract_median` docstring for details. 
+
+Subtract the median background level from each frame. The median is performed after masking out bright sources, using sigma clipping or similar techniques (see function docstring for the choices of how to do this.)
+
+Recommended for NIRCam only. 
+
+
+fix_bad_pixels
+--------------
+See :meth:`~spaceKLIP.imagetools.ImageTools.fix_bad_pixels` docstring for details.
+
+Flexible, complex set of multiple different ways to detect and replace bad pixels. Very complex set of options; see tutorial notebooks for examples of usage.
+
+replace_nans
+--------------
+See :meth:`~spaceKLIP.imagetools.ImageTools.replace_nans` docstring for details.
+
+Simple step to replace any NaNs with zero.
+
+
+Steps for Image Registration and Alignment
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+update_nircam_centers
+---------------------
+
+See :meth:`~spaceKLIP.imagetools.ImageTools.update_nircam_centers` docstring for details.
+
+Update header metadata for locations of the coronagraphs. This uses a table of better center locations measured from NIRCam commissioning data by J. Leisenring and J. Girard.  *This is a temporary step which should eventually be unnecessary, after a planned update of the SIAF calibratoin data.*  Replaces the header values for the CRPIX locations for the mask locations.  Only applies to NIRCam data, naturally. 
+
+
+**Details:** For each file, read current CRPIXn values. Use APERNAME to look up crpix values from `crpix_jarron`. Use FILTER to look up filter shift from `filter_shifts_jarron`. Compute deltas based on crpix_jarron and filter_shifts relative to Siaf x/ysciref values. Apply those deltas to change the CRPIX values. 
+
+**TBD: Document where these come from. Make some simple plots of those values to include here**.
+
+ * This step modifies the WCS headers, but not the pixel data. 
+
+recenter_frames
+---------------
+See :meth:`~spaceKLIP.imagetools.ImageTools.recenter_frames` docstring for details.
+
+To better measure the location of the star with respect to the coronagraph, we create a simulation of the star behind the coronagraph (using webbpsf), and cross-correlate this with the observed PSF. The cross correlation peak is used to infer the offset of the star relative to the mask center. The measured offset is used to shift the first frame to be centered. Then subsequent frames are aligned to that first frame.
+
+The accuracy of this algorithm is around 7 milliarcsec according to testing.
+
+This step also shifts to account for the coronagraph not being precisely centered in the subarray. After this step, the star center will be at the center of the pixel array.
+
+This step only works for NIRCam data; for MIRI data it will apply zero shifts, i.e. do nothing.
+
+ * This step modifies the pixel data to apply shifts
+
+
+align_frames
+------------
+See :meth:`~spaceKLIP.imagetools.ImageTools.align_frames` docstring for details.
+
+This step applies shifts to the image pixel data to align frames. All subsequent frames are aligned to the first frame of the first science integration. (I.e. the second roll and all references are aligned to the first roll).
+
+This measures and applies relative shifts between subsequent frames and the first frame.
+
+pad_frames
+----------
+See :meth:`~spaceKLIP.imagetools.ImageTools.pad_frames` docstring for details.
+
+Pad empty space around frames to give space to rotate and align during pyklip. THis puts a region of NAN pixels around the outside.
+
+

docs/source/index.rst

@@ -53,6 +53,7 @@ Compatible Simulated Data: `Here <https://stsci.box.com/s/cktghuyrwrallb401rw5y5
 
    TLDR
    Installation-and-dependencies
+   recommendations
 
 .. toctree::
    :maxdepth: 1

docs/source/recommendations.rst

@@ -0,0 +1,22 @@
+Recommendations and Best Practices
+==================================
+
+Getting started
+---------------
+
+Use the Tutorial notebooks. Note, the tutorials are all Jupyter Notebooks - they can be read online, though the readthedocs rendering is a bit awkward in parts - but they're really best used by downloading the
+notebooks and running them locally. These will let you reproduce some of the results from the Direct Observations of Exoplanetary Systems ERS program. 
+
+
+Tuning and Improving Reductions for NIRCam data
+-----------------------------------------------
+
+ - TIPS TO BE ADDED HERE
+
+Tuning and Improving Reductions for MIRI data
+-----------------------------------------------
+
+ - Understand whether your science and PSF stars are at similar or different exposure levels. If the PSF stars are much brighter than the science target, there may be detector nonlinearities, or issues in scaling the background subtraction. 
+ - Look for multiple PSF references and assemble a PSF library
+ - TIPS TO BE ADDED HERE
+

docs/source/spaceKLIP.rst

@@ -49,6 +49,15 @@ spaceKLIP.database module
    :undoc-members:
    :show-inheritance:
 
+spaceKLIP.fnoise_clean module
+-----------------------------
+
+.. automodule:: spaceKLIP.fnoise_clean
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
 spaceKLIP.imagetools module
 ---------------------------
 

docs/source/stage1.rst

@@ -1,6 +1,102 @@
 .. _stage1:
-
 Stage 1
--------
+=======
+
+The following steps are part of the ``Coron1Pipeline`` custom version of the JWST pipeline's ``Detector1Pipeline``.
+
+Note, the ``coron1pipeline.run_obs()`` function provides a succinct interface to create and invoke Coron1Pipeline on all
+observations in a spaceKLIP Database; that's the recommended way to invoke Coron1Pipeline.
+
+
+group_scale
+-----------
+Identical to default Detector1Pipeline
+
+dq_init
+-----------
+Identical to default Detector1Pipeline
+
+saturation
+----------
+For *MIRI*, identical to default Detector1Pipeline
+
+For *NIRCam*, depending on parameters ``flag_rcsat`` and ``grow_diagonal`` and ``n_pix_grow_sat``, may do some custom flagging of which pixels
+should be flagged around saturated pixels.
+
+**Custom step, TBD write more docs for this**
+
+Parameters you may wish to adjust for your data:
+ * ``flag_rcsat``. Default = False.
+ * ``grow_diagonal``. Default = False.
+ * ``n_pix_grow_sat``
+
+ipc
+-----------
+Identical to default Detector1Pipeline
+
+firstframe, lastframe, reset
+----------------------------
+(Only for MIRI)
+Identical to default Detector1Pipeline
+
+superbias
+----------
+(Only for NIRCam)
+Identical to default Detector1Pipeline
+
+refpix
+--------
+(Applies to both NIRCam and MIRI, but in different orders with other pipeline steps -- before linearity for NIRCam, after for MIRI??)
+
+**Custom step, TBD  write docs here for this**
+
+For MIRI, runs identical to default Detector1Pipeline.
+
+For NIRCam, for subarray images, runs a custom step to do "psuedo" reference pixel subtraction using pixels around the edge of the subarray.
+
+Parameters you may wish to adjust for your data:
+ * ``nlower, nupper, nleft, nright``: number of pixels around each edge to treat as pseudo-refpix. (left and right may not actually do anything, TBC?)
+
+
+linearity
+----------
+Identical to default Detector1Pipeline
+
+rscd
+-----
+(Only for MIRI)
+Identical to default Detector1Pipeline
+
+dark_current, charge_migration, jump
+-------------------------------------
+Identical to default Detector1Pipeline
+
+subtract_ktc
+------------
+(Only for NIRCam)
+Custom step for removal of kTc noise.
+
+**Custom step, TBD  write docs here for this**
+
+subtract_1overf
+----------------
+(Only for NIRCam)
+Custom step for removal of one over f noise, optimized for coronagraphic data using. This can be run either at the groups stage or at the ramp stage (i.e. before
+or after ramp fitting)
+
+**Custom step, TBD  write docs here for this**
+
+ramp_fit
+--------
+Identical to default Detector1Pipeline
+
+apply_rateints_outliers
+-----------------------
+Custom step to Flag additional outliers by comparing rateints and refit ramp
+
+**Custom step, TBD  write docs here for this**
+
+gain_scale
+----------
+Identical to default Detector1Pipeline. Applied to both the rate and rateints outputs, as standard.
 
-TBD: Detailed documentation of spaceKLIP Stage 1 processing will go here.

docs/source/stage2.rst

@@ -1,6 +1,28 @@
 .. _stage2:
 
 Stage 2
--------
+=======
+
+The following steps are part of the ``Coron2Pipeline`` custom version of the JWST pipeline's ``Detector2Pipeline``.
+
+Note, the ``coron2pipeline.run_obs()`` function provides a succinct interface to create and invoke Coron2Pipeline on all
+observations in a spaceKLIP Database; that's the recommended way to invoke Coron2Pipeline.
+
+
+All regular pipeline stage2 steps, mostly
+------------------------------------------
+
+
+The stage 2 pipeline steps from regular ``Image2Pipeline`` are mostly invoked.  
+
+Background subtraction is skipped, to be performed later using functions in spaceKLIP image analyses. 
+
+
+OutlierDetection
+----------------
+
+An outlier detection step is added. This is a step from the regular pipeline, so what's different here? Not sure, may be that this is run as part
+of stage 2 instead of stage 3?  **TBC, write better docs of this**
+
+This step is skipped for images with exptype = NRC_TA or NRC_TACONFIRM.
 
-TBD: Detailed documentation of spaceKLIP Stage 2 processing will go here.

docs/source/tutorials/tutorial_MIRI_contrast_analyses.ipynb

@@ -318,7 +318,7 @@
     "## Extract measurements of the companion HD 65426 b\n",
     "\n",
     "\n",
-    "We need to know where the companion is. Using [whereistheplanet.org](https://whereistheplanet.org) for 2022 Aug 15:\n",
+    "We need to know where the companion is. Using [whereistheplanet.com](https://whereistheplanet.com) for 2022 Aug 15:\n",
     "```\n",
     "RA Offset = 419.183 +/- 7.853 mas \n",
     "Dec Offset = -697.621 +/- 6.277 mas \n",

docs/source/tutorials/tutorial_NIRCam_contrast_analyses.ipynb

@@ -251,7 +251,7 @@
     "This means that we can and should mask that companion out of the contrast calculations. \n",
     "\n",
     "\n",
-    "Using [whereistheplanet.org](https://whereistheplanet.org) for the date of observations, 2022 Aug 15:\n",
+    "Using [whereistheplanet.com](https://whereistheplanet.com) for the date of observations, 2022 Aug 15:\n",
     "```\n",
     "RA Offset = 416.618 +/- 0.045 mas \n",
     "Dec Offset = -703.443 +/- 0.051 mas \n",

spaceKLIP/imagetools.py

@@ -1531,6 +1531,13 @@ def update_nircam_centers(self):
         center from Jarron and update the current reference pixel position to
         reflect the true mask center. Account for filter-dependent distortion.
         Might not be required for simulated data.
+
+        This step uses lookup tables of information derived from NIRCam
+        commissioning activities CAR-30 and CAR-31, by J. Leisenring and J. Girard,
+        and subsequent reanalyses using additional data from PSF SGD observations.
+
+        This information will eventually be applied as updates into the SIAF,
+        after which point this step will become not necessary.
         
         Returns
         -------