Improvements to MIRI Reductions

[AarynnCarter]

This PR provides significant additions and updates to various MIRI reduction procedures.

• Implemented an alternate background subtraction routine to handle the MIRI glowstick, based on code provided by Nicolas Godoy, which avoids trimming of the first integration by dynamically matching the background on an integration by integration basis.
• Implemented the new jump detection and ramp fitting algorithm developed by Tim Brandt, which has had significant improvements in reducing both spatially correlated and white noise in the images. Particularly for MIRI, and particularly for ramps with a large number of groups.
• Implemented options to mask specific groups/pixels within a ramp to better combat the brighter fatter effect seen for the MIRI PSF. This is particularly useful when the reference observations reach a larger number of detector counts than the science. Can be performed uniformly on entire groups (basic) or dynamically on a pixel-by-pixel basis (advanced), in both cases the number of groups to trim can be estimated automatically by the code.
• Adjusted steps that are run by default for MIRI data, following the pipeline the IPC and charge migration steps should not be performed and negatively impact data quality.

Note that the implemented jump detection and ramp fitting algorithm will eventually be incorporated into the JWST pipeline, so I will open an issue now so that we can eventually switch over from my custom written step.

[wbalmer]

I think the logic here is faulty. I ran with 'mask_groups': {'skip': True} in order to test that inner disk science data and got the following error:

File [~/spaceKLIP/spaceKLIP/coron1pipeline.py:921](http://localhost:8888/lab/workspaces/auto-v/tree/reductions/jwst/exoalma/~/spaceKLIP/spaceKLIP/coron1pipeline.py#line=920), in run_obs(database, steps, subdir, overwrite, quiet, verbose, **kwargs)
    919 if not steps['mask_groups']['skip']:
    920     if steps['mask_groups']['mask_method'] == 'basic':
--> 921         steps = prepare_group_masking_basic(steps, 
    922                                             database.obs[key], 
    923                                             quiet)
    924     elif steps['mask_groups']['mask_method'] == 'advanced':
    925         fitstype = database.obs[key]['TYPE'][j]

File [~/spaceKLIP/spaceKLIP/coron1pipeline.py:986](http://localhost:8888/lab/workspaces/auto-v/tree/reductions/jwst/exoalma/~/spaceKLIP/spaceKLIP/coron1pipeline.py#line=985), in prepare_group_masking_basic(steps, observations, quiet)
    984 if observations['TYPE'][j] == 'SCI':
    985     with fits.open(os.path.abspath(observations['FITSFILE'][j])) as hdul:
--> 986         sci_frame = hdul['SCI'].data[:, -1, :, :].astype(float)
    988         # Subtract a median so we focus on brightest pixels
    989         sci_frame -= np.nanmedian(sci_frame, axis=(1,2), keepdims=True)

IndexError: too many indices for array: array is 3-dimensional, but 4 were indexed

I think if not steps['mask_groups']['skip'] doesn't evaluate in the way we want it to.

[wbalmer]

Turns out this error is occurring because the mask_groups step fails for me no matter what, and then the skip keyword doesn't skip the step. When running with either 'advanced' or 'basic' and 'skip'=False, I get the above IndexError.

[AarynnCarter]

Strange, does your data have more than one integration? I'm not sure why the science frame would be 3-dimensional for an uncal file instead of 4-dimensional.

[spacegal-spiff]

Curiously, I haven't run into the same error as William, but I might be encountering something similar. When I run the original coron1pipeline call from the tutorial notebooks (which does not include mask_groups under the existing steps), I run into group masking errors because the mask_array attribute does not get set for the MaskGroupsStep object. I assume this would just be skipped if group masking is not included as a step, so I'm not sure why it's dependent on the existence of the attribute. Not entirely sure if this is an issue specifically related to this PR, but wondering whether addressing this might also address the error William sees?

File ~/Software/spaceKLIP/spaceKLIP/coron1pipeline.py:942, in run_obs(database, steps, subdir, overwrite, quiet, verbose, **kwargs)
    940 else:
    941     if not quiet: log.info('  --> Coron1Pipeline: processing ' + tail)
--> 942     _ = run_single_file(fitspath, output_dir, steps=steps, 
    943                         verbose=verbose, **kwargs)
    945 if (j == jtervals[-1]) and (groupmaskflag == 1):
    946     '''This is the last file for this concatenation, and the groupmaskflag has been
    947     set. This means we need to reset the mask_array back to original state, 
    948     which was that it didn't exist, so that the routine is rerun. '''

File ~/Software/spaceKLIP/spaceKLIP/coron1pipeline.py:733, in run_single_file(fitspath, output_dir, steps, verbose, **kwargs)
    731         res = pipeline.run(fitspath)
    732 except Exception as e:
--> 733     raise RuntimeError(
    734         'Caught exception during pipeline processing.'
    735         '\nException: {}'.format(e)
    736     )
    737 finally:
    738     pipeline.closeout()

RuntimeError: Caught exception during pipeline processing.
Exception: 'MaskGroupsStep' object has no attribute 'mask_array'

Incidentally, when I set mask_groups 'skip=True' it seems to run without any issues, so the following call to coron1pipeline works with either flag for me:

spaceKLIP.coron1pipeline.run_obs(database=database,
                   steps={'mask_groups': {'skip': False, 
                                          'mask_method': 'advanced',
                                          'save_results':True, 
                                          'types':['REF', 'REF_BG']},
                          'experimental_jumpramp': {'use': True, 'nproc':4}},
                   subdir='stage1',
                   verbose=True)

[AarynnCarter]

@wbalmer @spacegal-spiff Thanks for the comments!

I just took a look and there was a lot of faulty logic to how the function was deciding to skip the step, or prep things for the step. I think both of your situations should be resolved now, and you shouldn't need to specify 'skip' anymore. The only small catch is that if you only put 'mask_groups': {}, it will skip by default.

Would you mind quickly rerunning the scripts to see if it resolved things?

[wbalmer]

This patch allowed me to skip the mask_groups step, so I'd say its good to go for now. Not sure what's up with my dataset, the SCI is one file with dims 30x500xsubarray.

[wbalmer]

The experimental jump/ramp step works a-okay (though I'm not sure if I could tell a big difference on my data).

The group masking step fails no matter how I try to use it (with the error appended to my comments on coron1pipeline.py, though perhaps I need to be testing using a different dataset? ERS maybe? Right now I'm testing using a set of SCI/REF/ SCI_BG/REF_BG from GO 3524.

[spacegal-spiff]

Did not encounter any other issues beyond the bug relating to mask_groups in coron1pipeline. Everything else seems to work well! Running all the way through forward modeling of the companion in our MIRI data actually ended up with slightly higher SNR in the companion detection (SNR of 39 vs. 37 from the previous reduction vs. 30 using the MAST pipelines stage2 products), so that seems great! Visually the backgrounds also appear cleaner, and some of the faint background sources are more easily detected.

[spacegal-spiff]

Curiously, I haven't run into the same error as William, but I might be encountering something similar. When I run the original coron1pipeline call from the tutorial notebooks (which does not include mask_groups under the existing steps), I run into group masking errors because the mask_array attribute does not get set for the MaskGroupsStep object. I assume this would just be skipped if group masking is not included as a step, so I'm not sure why it's dependent on the existence of the attribute. Not entirely sure if this is an issue specifically related to this PR, but wondering whether addressing this might also address the error William sees?

File ~/Software/spaceKLIP/spaceKLIP/coron1pipeline.py:942, in run_obs(database, steps, subdir, overwrite, quiet, verbose, **kwargs)
    940 else:
    941     if not quiet: log.info('  --> Coron1Pipeline: processing ' + tail)
--> 942     _ = run_single_file(fitspath, output_dir, steps=steps, 
    943                         verbose=verbose, **kwargs)
    945 if (j == jtervals[-1]) and (groupmaskflag == 1):
    946     '''This is the last file for this concatenation, and the groupmaskflag has been
    947     set. This means we need to reset the mask_array back to original state, 
    948     which was that it didn't exist, so that the routine is rerun. '''

File ~/Software/spaceKLIP/spaceKLIP/coron1pipeline.py:733, in run_single_file(fitspath, output_dir, steps, verbose, **kwargs)
    731         res = pipeline.run(fitspath)
    732 except Exception as e:
--> 733     raise RuntimeError(
    734         'Caught exception during pipeline processing.'
    735         '\nException: {}'.format(e)
    736     )
    737 finally:
    738     pipeline.closeout()

RuntimeError: Caught exception during pipeline processing.
Exception: 'MaskGroupsStep' object has no attribute 'mask_array'

Incidentally, when I set mask_groups 'skip=True' it seems to run without any issues, so the following call to coron1pipeline works with either flag for me:

spaceKLIP.coron1pipeline.run_obs(database=database,
                   steps={'mask_groups': {'skip': False, 
                                          'mask_method': 'advanced',
                                          'save_results':True, 
                                          'types':['REF', 'REF_BG']},
                          'experimental_jumpramp': {'use': True, 'nproc':4}},
                   subdir='stage1',
                   verbose=True)