From 0f79350163ad2e196e3259f84a55917580fb8c9f Mon Sep 17 00:00:00 2001 From: Marshall Perrin Date: Tue, 6 Feb 2024 07:58:27 -0500 Subject: [PATCH 1/3] add initial brief docs for stage 1 steps --- docs/source/spaceKLIP.rst | 9 ++++ docs/source/stage1.rst | 99 +++++++++++++++++++++++++++++++++++++-- 2 files changed, 105 insertions(+), 3 deletions(-) diff --git a/docs/source/spaceKLIP.rst b/docs/source/spaceKLIP.rst index 9c4a7bf2..5212062b 100644 --- a/docs/source/spaceKLIP.rst +++ b/docs/source/spaceKLIP.rst @@ -49,6 +49,15 @@ spaceKLIP.database module :undoc-members: :show-inheritance: +spaceKLIP.fnoise_clean module +----------------------------- + +.. automodule:: spaceKLIP.fnoise_clean + :members: + :undoc-members: + :show-inheritance: + + spaceKLIP.imagetools module --------------------------- diff --git a/docs/source/stage1.rst b/docs/source/stage1.rst index 774d8b4d..28ddb550 100644 --- a/docs/source/stage1.rst +++ b/docs/source/stage1.rst @@ -1,6 +1,99 @@ .. _stage1: - Stage 1 -------- +======= + +The following steps are part of the ``Coron1Pipeline`` custom version of the JWST pipeline's ``Detector1Pipeline``. + + +group_scale +----------- +Identical to default Detector1Pipeline + +dq_init +----------- +Identical to default Detector1Pipeline + +saturation +---------- +For *MIRI*, identical to default Detector1Pipeline + +For *NIRCam*, depending on parameters ``flag_rcsat`` and ``grow_diagonal`` and ``n_pix_grow_sat``, may do some custom flagging of which pixels +should be flagged around saturated pixels. + +**Custom step, TBD write more docs for this** + +Parameters you may wish to adjust for your data: + * ``flag_rcsat``. Default = False. + * ``grow_diagonal``. Default = False. + * ``n_pix_grow_sat`` + +ipc +----------- +Identical to default Detector1Pipeline + +firstframe, lastframe, reset +---------------------------- +(Only for MIRI) +Identical to default Detector1Pipeline + +superbias +---------- +(Only for NIRCam) +Identical to default Detector1Pipeline + +refpix +-------- +(Applies to both NIRCam and MIRI, but in different orders with other pipeline steps -- before linearity for NIRCam, after for MIRI??) + +**Custom step, TBD write docs here for this** + +For MIRI, runs identical to default Detector1Pipeline. + +For NIRCam, for subarray images, runs a custom step to do "psuedo" reference pixel subtraction using pixels around the edge of the subarray. + +Parameters you may wish to adjust for your data: + * ``nlower, nupper, nleft, nright``: number of pixels around each edge to treat as pseudo-refpix. (left and right may not actually do anything, TBC?) + + +linearity +---------- +Identical to default Detector1Pipeline + +rscd +----- +(Only for MIRI) +Identical to default Detector1Pipeline + +dark_current, charge_migration, jump +------------------------------------- +Identical to default Detector1Pipeline + +subtract_ktc +------------ +(Only for NIRCam) +Custom step for removal of kTc noise. + +**Custom step, TBD write docs here for this** + +subtract_1overf +---------------- +(Only for NIRCam) +Custom step for removal of one over f noise, optimized for coronagraphic data using. This can be run either at the groups stage or at the ramp stage (i.e. before +or after ramp fitting) + +**Custom step, TBD write docs here for this** + +ramp_fit +-------- +Identical to default Detector1Pipeline + +apply_rateints_outliers +----------------------- +Custom step to Flag additional outliers by comparing rateints and refit ramp + +**Custom step, TBD write docs here for this** + +gain_scale +---------- +Identical to default Detector1Pipeline. Applied to both the rate and rateints outputs, as standard. -TBD: Detailed documentation of spaceKLIP Stage 1 processing will go here. \ No newline at end of file From 0e0c138d494286f975530867db970ed7930eac78 Mon Sep 17 00:00:00 2001 From: Marshall Perrin Date: Wed, 20 Mar 2024 16:38:54 -0400 Subject: [PATCH 2/3] update and add to docs --- docs/source/imagetools.rst | 85 ++++++++++++++++++- docs/source/stage1.rst | 3 + docs/source/stage2.rst | 26 +++++- .../tutorial_MIRI_contrast_analyses.ipynb | 2 +- .../tutorial_NIRCam_contrast_analyses.ipynb | 2 +- spaceKLIP/imagetools.py | 7 ++ 6 files changed, 120 insertions(+), 5 deletions(-) diff --git a/docs/source/imagetools.rst b/docs/source/imagetools.rst index 18877978..e7bf1c50 100644 --- a/docs/source/imagetools.rst +++ b/docs/source/imagetools.rst @@ -1,6 +1,89 @@ .. _imagetools: Image Tools ------------ +============ TBD: Detailed documentation of the ``spaceklip.imagetools`` module will go here. + + + +Steps for background removal and pixel cleaning +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +subtract_background +-------------------- +See :py:meth:`~spaceKLIP.imagetools.ImageTools.subtract_background` docstring for details. + +If you have dedicated background exposures (exp type=SCI_BG or REF_BG), subtract those from your SCI or REF data. If there are multiple ints in the BG exposure, they are median-combined before the subtraction. + +subtract_median +-------------------- +See :meth:`~spaceKLIP.imagetools.ImageTools.subtract_median` docstring for details. + +Subtract the median background level from each frame. The median is performed after masking out bright sources, using sigma clipping or similar techniques (see function docstring for the choices of how to do this.) + +Recommended for NIRCam only. + + +fix_bad_pixels +-------------- +See :meth:`~spaceKLIP.imagetools.ImageTools.fix_bad_pixels` docstring for details. + +Flexible, complex set of multiple different ways to detect and replace bad pixels. Very complex set of options; see tutorial notebooks for examples of usage. + +replace_nans +-------------- +See :meth:`~spaceKLIP.imagetools.ImageTools.replace_nans` docstring for details. + +Simple step to replace any NaNs with zero. + + +Steps for Image Registration and Alignment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +update_nircam_centers +--------------------- + +See :meth:`~spaceKLIP.imagetools.ImageTools.update_nircam_centers` docstring for details. + +Update header metadata for locations of the coronagraphs. This uses a table of better center locations measured from NIRCam commissioning data by J. Leisenring and J. Girard. *This is a temporary step which should eventually be unnecessary, after a planned update of the SIAF calibratoin data.* Replaces the header values for the CRPIX locations for the mask locations. Only applies to NIRCam data, naturally. + + +**Details:** For each file, read current CRPIXn values. Use APERNAME to look up crpix values from `crpix_jarron`. Use FILTER to look up filter shift from `filter_shifts_jarron`. Compute deltas based on crpix_jarron and filter_shifts relative to Siaf x/ysciref values. Apply those deltas to change the CRPIX values. + +**TBD: Document where these come from. Make some simple plots of those values to include here**. + + * This step modifies the WCS headers, but not the pixel data. + +recenter_frames +--------------- +See :meth:`~spaceKLIP.imagetools.ImageTools.recenter_frames` docstring for details. + +To better measure the location of the star with respect to the coronagraph, we create a simulation of the star behind the coronagraph (using webbpsf), and cross-correlate this with the observed PSF. The cross correlation peak is used to infer the offset of the star relative to the mask center. The measured offset is used to shift the first frame to be centered. Then subsequent frames are aligned to that first frame. + +The accuracy of this algorithm is around 7 milliarcsec according to testing. + +This step also shifts to account for the coronagraph not being precisely centered in the subarray. After this step, the star center will be at the center of the pixel array. + +This step only works for NIRCam data; for MIRI data it will apply zero shifts, i.e. do nothing. + + * This step modifies the pixel data to apply shifts + + +align_frames +------------ +See :meth:`~spaceKLIP.imagetools.ImageTools.align_frames` docstring for details. + +This step applies shifts to the image pixel data to align frames. All subsequent frames are aligned to the first frame of the first science integration. (I.e. the second roll and all references are aligned to the first roll). + +This measures and applies relative shifts between subsequent frames and the first frame. + +pad_frames +---------- +See :meth:`~spaceKLIP.imagetools.ImageTools.pad_frames` docstring for details. + +Pad empty space around frames to give space to rotate and align during pyklip. THis puts a region of NAN pixels around the outside. + + diff --git a/docs/source/stage1.rst b/docs/source/stage1.rst index 28ddb550..a625c439 100644 --- a/docs/source/stage1.rst +++ b/docs/source/stage1.rst @@ -4,6 +4,9 @@ Stage 1 The following steps are part of the ``Coron1Pipeline`` custom version of the JWST pipeline's ``Detector1Pipeline``. +Note, the ``coron1pipeline.run_obs()`` function provides a succinct interface to create and invoke Coron1Pipeline on all +observations in a spaceKLIP Database; that's the recommended way to invoke Coron1Pipeline. + group_scale ----------- diff --git a/docs/source/stage2.rst b/docs/source/stage2.rst index 76abd427..9022b83b 100644 --- a/docs/source/stage2.rst +++ b/docs/source/stage2.rst @@ -1,6 +1,28 @@ .. _stage2: Stage 2 -------- +======= + +The following steps are part of the ``Coron2Pipeline`` custom version of the JWST pipeline's ``Detector2Pipeline``. + +Note, the ``coron2pipeline.run_obs()`` function provides a succinct interface to create and invoke Coron2Pipeline on all +observations in a spaceKLIP Database; that's the recommended way to invoke Coron2Pipeline. + + +All regular pipeline stage2 steps, mostly +------------------------------------------ + + +The stage 2 pipeline steps from regular ``Image2Pipeline`` are mostly invoked. + +Background subtraction is skipped, to be performed later using functions in spaceKLIP image analyses. + + +OutlierDetection +---------------- + +An outlier detection step is added. This is a step from the regular pipeline, so what's different here? Not sure, may be that this is run as part +of stage 2 instead of stage 3? **TBC, write better docs of this** + +This step is skipped for images with exptype = NRC_TA or NRC_TACONFIRM. -TBD: Detailed documentation of spaceKLIP Stage 2 processing will go here. \ No newline at end of file diff --git a/docs/source/tutorials/tutorial_MIRI_contrast_analyses.ipynb b/docs/source/tutorials/tutorial_MIRI_contrast_analyses.ipynb index 0e104822..d04bcf94 100644 --- a/docs/source/tutorials/tutorial_MIRI_contrast_analyses.ipynb +++ b/docs/source/tutorials/tutorial_MIRI_contrast_analyses.ipynb @@ -318,7 +318,7 @@ "## Extract measurements of the companion HD 65426 b\n", "\n", "\n", - "We need to know where the companion is. Using [whereistheplanet.org](https://whereistheplanet.org) for 2022 Aug 15:\n", + "We need to know where the companion is. Using [whereistheplanet.com](https://whereistheplanet.com) for 2022 Aug 15:\n", "```\n", "RA Offset = 419.183 +/- 7.853 mas \n", "Dec Offset = -697.621 +/- 6.277 mas \n", diff --git a/docs/source/tutorials/tutorial_NIRCam_contrast_analyses.ipynb b/docs/source/tutorials/tutorial_NIRCam_contrast_analyses.ipynb index 153a381c..a0a6cf34 100644 --- a/docs/source/tutorials/tutorial_NIRCam_contrast_analyses.ipynb +++ b/docs/source/tutorials/tutorial_NIRCam_contrast_analyses.ipynb @@ -251,7 +251,7 @@ "This means that we can and should mask that companion out of the contrast calculations. \n", "\n", "\n", - "Using [whereistheplanet.org](https://whereistheplanet.org) for the date of observations, 2022 Aug 15:\n", + "Using [whereistheplanet.com](https://whereistheplanet.com) for the date of observations, 2022 Aug 15:\n", "```\n", "RA Offset = 416.618 +/- 0.045 mas \n", "Dec Offset = -703.443 +/- 0.051 mas \n", diff --git a/spaceKLIP/imagetools.py b/spaceKLIP/imagetools.py index 41987d26..436ada67 100644 --- a/spaceKLIP/imagetools.py +++ b/spaceKLIP/imagetools.py @@ -1530,6 +1530,13 @@ def update_nircam_centers(self): center from Jarron and update the current reference pixel position to reflect the true mask center. Account for filter-dependent distortion. Might not be required for simulated data. + + This step uses lookup tables of information derived from NIRCam + commissioning activities CAR-30 and CAR-31, by J. Leisenring and J. Girard, + and subsequent reanalyses using additional data from PSF SGD observations. + + This information will eventually be applied as updates into the SIAF, + after which point this step will become not necessary. Returns ------- From 6db46edc79fbf943687f47645238294c7e440eca Mon Sep 17 00:00:00 2001 From: Marshall Perrin Date: Wed, 20 Mar 2024 17:05:30 -0400 Subject: [PATCH 3/3] More documentation! --- docs/source/Installation-and-dependencies.rst | 2 +- docs/source/index.rst | 1 + docs/source/recommendations.rst | 22 +++++++++++++++++++ 3 files changed, 24 insertions(+), 1 deletion(-) create mode 100644 docs/source/recommendations.rst diff --git a/docs/source/Installation-and-dependencies.rst b/docs/source/Installation-and-dependencies.rst index e73df3a5..fab16560 100644 --- a/docs/source/Installation-and-dependencies.rst +++ b/docs/source/Installation-and-dependencies.rst @@ -28,4 +28,4 @@ With the anaconda environment created, move to the cloned directory and install cd where/you/installed/the/git/repo pip install -r requirements.txt -Finally, and very importantly, you will need to download reference files to support the functioning of the :code:`webbpsf` and :code:`webbpsf_ext`. Instructions to do this can be found at the respective package websites (`WebbPSF `_, `webbpsf_ext `_). Ensure that if you edit your .bashrc file that you reopen and close your terminal to fully apply the changes (:code:`source ~/.bashrc` or :code:`source ~/.zshrc` may also work) \ No newline at end of file +Finally, and very importantly, you will need to download reference files to support the functioning of the :code:`webbpsf` and :code:`webbpsf_ext`. Instructions to do this can be found at the respective package websites (`WebbPSF `_, `webbpsf_ext `_). Ensure that if you edit your .bashrc file that you reopen and close your terminal to fully apply the changes (:code:`source ~/.bashrc` or :code:`source ~/.zshrc` may also work) diff --git a/docs/source/index.rst b/docs/source/index.rst index 38a81ea9..19cc28a1 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -53,6 +53,7 @@ Compatible Simulated Data: `Here