Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved 1/f noise correction performance #109

Merged
merged 5 commits into from
Dec 7, 2023
Merged

Conversation

JarronL
Copy link
Collaborator

@JarronL JarronL commented Oct 14, 2023

Add multiprocessing capability to speed up performance as well as robust removal of additional 1/f noise from averaged ramp data used to obtain residuals at the group or integration level:

  • Add multiprocessing for more efficient 1/f noise modeling. Defaults to 4 processors (diminishing returns for higher numbers) and of course scales to few processes depending on number of available CPUs.
  • Improved handling of full frame data, such as accounting for the slowaxis being along the x-direction
  • Subarray datasets also corrects 1/f noise along the appropriate axis depending on value of slowaxis header info. For instance, NIRISS detector orientation is rotated 90 degrees relative to NIRCam, with slow scan direction being horizontal along x-axis.
  • More robust extraction of 1/f noise model for single frames where one cannot work on residuals (see next bullet for details in that method).
  • When determining the average slope and building a ideal ramp data or averaged INTS, these averaged data will still have 1/f noise signatures that then get subtracted from the individual integrations when calculating the residuals, so the 1/f noise model never gets to model them. Now, we fit a 1/f noise model to the averaged dataset to first remove it and utilize a "flattening" routine to subtract large variations along the orthogonal direction of the model that are due to over-fitting flux from astrophysical sources. The result is a much cleaner image of the averaged slope image used to determine residuals at the groups and INTs level.
  • Testing on multiple coronagraphic datasets indicates that modeling on the rateints has slightly better noise performance than at the group level. Models for rateints is now the default via the pipeline.stage_1overf attribute, but can be set to 'groups' if desired. Also means there is less processing time from 1/f correction.

 - Add multiprocessing for more efficient 1/f noise modeling. Defaults to 4 processors (diminishing returns after?) and scales with few CPUs.
 - Improved handling of full frame data
 - More robust extraction of 1/f noise model for single frames where cannot work on residuals (this also helps with getting residuals for multiple INTS)
 - Testing on multiple coronagraphic datasets indicates that modeling on the rateints has slightly better noise performance than on the group level. Models for rateints is now the default via the stage_1overf attribute, but can be set to 'groups' if desired
 - account for data in which slowaxis is in x-direction
stage_1overf  indicates where in the pipeline process to perform 1/f noise removal, either during group level or at ints
Copy link
Collaborator

@kammerje kammerje left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I briefly reviewed the changes and think that they can be merged.

@kammerje kammerje merged commit 205d83d into develop Dec 7, 2023
4 checks passed
@JarronL JarronL deleted the OoFN_correction branch January 30, 2024 23:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants