Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove units from the ref files #490

Merged
merged 5 commits into from
Jan 16, 2025
Merged

Remove units from the ref files #490

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a67d270
Remove units from the ref files
WilliamJamieson Oct 18, 2024
e7457eb
Fix #489
WilliamJamieson Oct 21, 2024
a57713b
Move ref_common check to rad from roman_datamodels
WilliamJamieson Jan 13, 2025
e3cf85d
Add change log
WilliamJamieson Jan 13, 2025
2b7a04a
Document input/output units in schema
WilliamJamieson Jan 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changes/490.feature.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Remove units from the reference file schemas.
62 changes: 16 additions & 46 deletions src/rad/resources/schemas/reference_files/dark-1.0.0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,21 +20,6 @@ properties:
exposure:
type: object
properties:
ngroups:
title: Number of Resultants
description: |
The number of resultants averaged according to
Multi-Accumulation (MA) Table read pattern.
type: integer
nframes:
title: Number of frames per resultant # should be removed
type: integer
groupgap:
title: Number of Skips Between Resultants
description: |
The number of frames skipped between resultants according to
MA Table read pattern.
type: integer
ma_table_name:
title: Multi-Accumulation Table Name
description: |
Expand All @@ -48,7 +33,7 @@ properties:
Table that keeps the same name will have a new
ma_table_number.
type: integer
required: [ngroups, nframes, groupgap, ma_table_name, ma_table_number]
required: [ma_table_name, ma_table_number]
required: [exposure]
- $ref: asdf://stsci.edu/datamodels/roman/schemas/reference_files/ref_exposure_type-1.0.0
- $ref: asdf://stsci.edu/datamodels/roman/schemas/reference_files/ref_optical_element-1.0.0
Expand All @@ -57,16 +42,11 @@ properties:
description: |
The dark current array represents the integrated number of counts due to
the accumulation of dark current electrons in the pixels.
tag: tag:stsci.edu:asdf/unit/quantity-1.*
properties:
value:
tag: tag:stsci.edu:asdf/core/ndarray-1.*
datatype: float32
exact_datatype: true
ndim: 3
unit:
tag: tag:astropy.org:astropy/units/unit-1.*
enum: ["DN"]
tag: tag:stsci.edu:asdf/core/ndarray-1.*
datatype: float32
exact_datatype: true
ndim: 3
unit: DN
dq:
title: 2-D Data Quality Array
description: |
Expand All @@ -81,31 +61,21 @@ properties:
The dark current rate array represents the slope of the integrated number
of counts due to the accumulation of dark current electrons in the pixels
calculated from slope fitting the Dark Current Array.
tag: tag:stsci.edu:asdf/unit/quantity-1.*
properties:
value:
tag: tag:stsci.edu:asdf/core/ndarray-1.*
datatype: float32
exact_datatype: true
ndim: 2
unit:
tag: tag:astropy.org:astropy/units/unit-1.*
enum: ["DN / s"]
tag: tag:stsci.edu:asdf/core/ndarray-1.*
datatype: float32
exact_datatype: true
ndim: 2
unit: DN / s
dark_slope_error:
title: Dark Current Rate Uncertainty Array
description: |
The uncertainty calculated from the slope fitting of the Dark Current
Array.
tag: tag:stsci.edu:asdf/unit/quantity-1.*
properties:
value:
tag: tag:stsci.edu:asdf/core/ndarray-1.*
datatype: float32
exact_datatype: true
ndim: 2
unit:
tag: tag:astropy.org:astropy/units/unit-1.*
enum: ["DN / s"]
tag: tag:stsci.edu:asdf/core/ndarray-1.*
datatype: float32
exact_datatype: true
ndim: 2
unit: DN / s
required: [meta, data, dq, dark_slope, dark_slope_error]
flowStyle: block
propertyOrder: [meta, data, dq, dark_slope, dark_slope_error]
Expand Down
19 changes: 5 additions & 14 deletions src/rad/resources/schemas/reference_files/distortion-1.0.0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,26 +13,17 @@ properties:
allOf:
- $ref: asdf://stsci.edu/datamodels/roman/schemas/reference_files/ref_common-1.0.0
- type: object

input_units: "pixel"
output_units: "arcsec"

properties:
reftype:
type: string
enum: [DISTORTION]
input_units:
title: Input Model Units
description: |
The pixel input units of the detector coordinate model.
tag: tag:stsci.edu:asdf/unit/unit-1.*
enum: ["pixel"]
output_units:
title: Output Model Units
description: |
The V2/V3 coordinates output units after the model is applied.
tag: tag:stsci.edu:asdf/unit/unit-1.*
enum: ["arcsec"]
required: [output_units, input_units]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PaulHuwe Was there a decision to remove input_units`` and output_units`? I think especially the output units can be important.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right that these two should remain.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we actually going to use this information in the pipeline or is it simply informational?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's informational.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can go in the description but it seems easier to be in a separate key. It will also show in info() this way.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll add it like we added the unit descriptors for quantities

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

- $ref: asdf://stsci.edu/datamodels/roman/schemas/reference_files/ref_optical_element-1.0.0
coordinate_distortion_transform:
title: Distortion Transform Model
title: Distortion Transform Model with inputs in "pixel" and outputs in "arcsec"
description: |
The astropy.modeling.Model instance of of the distortion transform model.
type: object
Expand Down
15 changes: 5 additions & 10 deletions src/rad/resources/schemas/reference_files/gain-1.0.0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,16 +22,11 @@ properties:
description: |
The Gain Data Array represents the pixel to pixel conversion from digital
numbers (DN) to electrons (e). The units are e/DN.
tag: tag:stsci.edu:asdf/unit/quantity-1.*
properties:
value:
tag: tag:stsci.edu:asdf/core/ndarray-1.*
datatype: float32
exact_datatype: true
ndim: 2
unit:
tag: tag:astropy.org:astropy/units/unit-1.*
enum: ["electron / DN"]
tag: tag:stsci.edu:asdf/core/ndarray-1.*
datatype: float32
exact_datatype: true
ndim: 2
unit: electron / DN
required: [meta, data]
flowStyle: block
propertyOrder: [meta, data]
Expand Down
17 changes: 4 additions & 13 deletions src/rad/resources/schemas/reference_files/inverselinearity-1.0.0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,23 +13,14 @@ properties:
allOf:
- $ref: asdf://stsci.edu/datamodels/roman/schemas/reference_files/ref_common-1.0.0
- type: object

input_units: "DN"
output_units: "DN"

properties:
reftype:
type: string
enum: [INVERSELINEARITY]
input_units:
title: Inverse Linearity Input Units
description: |
Units of the input to the inverse linearity polynomial.
tag: tag:astropy.org:astropy/units/unit-1.*
enum: ["DN"]
output_units:
title: Inverse Linearity Output Units
description: |
Units of the output of the inverse linearity polynomial.
tag: tag:astropy.org:astropy/units/unit-1.*
enum: ["DN"]
required: [output_units, input_units]
coeffs:
title: Inverse Linearity Coefficients
description: |
Expand Down
17 changes: 4 additions & 13 deletions src/rad/resources/schemas/reference_files/linearity-1.0.0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,23 +13,14 @@ properties:
allOf:
- $ref: asdf://stsci.edu/datamodels/roman/schemas/reference_files/ref_common-1.0.0
- type: object

input_units: "DN"
output_units: "DN"

properties:
reftype:
type: string
enum: [LINEARITY]
input_units:
title: Units of the Input to the Linearity Polynomial
description: |
Units of the input to the linearity polynomial.
tag: tag:astropy.org:astropy/units/unit-1.*
enum: ["DN"]
output_units:
title: Units of the Output of the Linearity Polynomial
description: |
Units of the output to the linearity polynomial.
tag: tag:astropy.org:astropy/units/unit-1.*
enum: ["DN"]
required: [output_units, input_units]
coeffs:
title: Linearity Coefficients
description: |
Expand Down
18 changes: 4 additions & 14 deletions src/rad/resources/schemas/reference_files/pixelarea-1.0.0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,26 +25,16 @@ properties:
description: |
The nominal pixel area in steradians.
anyOf:
- tag: tag:stsci.edu:asdf/unit/quantity-1.*
properties:
datatype:
enum: ["float64"]
unit:
tag: tag:stsci.edu:asdf/unit/unit-1.*
enum: ["sr"]
- type: number
unit: sr
- type: "null"
pixelarea_arcsecsq:
title: Pixel Area (arcsec^2)
description:
The nominal pixel area in arcec^2.
anyOf:
- tag: tag:stsci.edu:asdf/unit/quantity-1.*
properties:
datatype:
enum: ["float64"]
unit:
tag: tag:stsci.edu:asdf/unit/unit-1.*
enum: ["arcsec**2"]
- type: number
unit: arcsec**2
- type: "null"
required: [pixelarea_steradians, pixelarea_arcsecsq]
required: [photometry]
Expand Down
15 changes: 5 additions & 10 deletions src/rad/resources/schemas/reference_files/readnoise-1.0.0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,16 +22,11 @@ properties:
description: |
The pixel-by-pixel map read noise data array is used in estimating the
expected noise in each pixel.
tag: tag:stsci.edu:asdf/unit/quantity-1.*
properties:
value:
tag: tag:stsci.edu:asdf/core/ndarray-1.*
datatype: float32
exact_datatype: true
ndim: 2
unit:
tag: tag:astropy.org:astropy/units/unit-1.*
enum: ["DN"]
tag: tag:stsci.edu:asdf/core/ndarray-1.*
datatype: float32
exact_datatype: true
ndim: 2
unit: DN
required: [meta, data]
flowStyle: block
propertyOrder: [meta, data]
Expand Down
13 changes: 4 additions & 9 deletions src/rad/resources/schemas/reference_files/refpix-1.0.0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,19 +15,14 @@ properties:
allOf:
- $ref: asdf://stsci.edu/datamodels/roman/schemas/reference_files/ref_common-1.0.0
- type: object

input_units: "DN"
output_units: "DN"

properties:
reftype:
type: string
enum: [REFPIX]
input_units:
title: Units of the input to the linearity polynomial.
tag: tag:astropy.org:astropy/units/unit-1.*
enum: ["DN"]
output_units:
title: Units of the output of the linearity polynomial.
tag: tag:astropy.org:astropy/units/unit-1.*
enum: ["DN"]
required: [output_units, input_units]

gamma:
title: Left column correction coefficients
Expand Down
15 changes: 5 additions & 10 deletions src/rad/resources/schemas/reference_files/saturation-1.0.0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,16 +22,11 @@ properties:
description: |
The pixel level threshold for determining saturation before non-linearity
corrections are applied.
tag: tag:stsci.edu:asdf/unit/quantity-1.*
properties:
value:
tag: tag:stsci.edu:asdf/core/ndarray-1.*
datatype: float32
exact_datatype: true
ndim: 2
unit:
tag: tag:astropy.org:astropy/units/unit-1.*
enum: ["DN"]
tag: tag:stsci.edu:asdf/core/ndarray-1.*
datatype: float32
exact_datatype: true
ndim: 2
unit: DN
dq:
title: 2-D Data Quality Array
description: |
Expand Down
24 changes: 3 additions & 21 deletions src/rad/resources/schemas/reference_files/wfi_img_photom-1.0.0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,39 +32,21 @@ properties:
description: |
Surface brightness, in MJy / steradian.
anyOf:
- tag: tag:stsci.edu:asdf/unit/quantity-1.*
properties:
value:
type: number
unit:
tag: tag:stsci.edu:asdf/unit/unit-1.*
enum: [MJy.sr**-1]
- type: number
- type: "null"
uncertainty:
title: Surface Brightness Uncertainty
description: |
Uncertainty of surface brightness, in MJy / steradian.
anyOf:
- tag: tag:stsci.edu:asdf/unit/quantity-1.*
properties:
value:
type: number
unit:
tag: tag:stsci.edu:asdf/unit/unit-1.*
enum: [MJy.sr**-1]
- type: number
- type: "null"
pixelareasr:
title: Pixel Area
description: |
The nominal pixel area, in steradian.
anyOf:
- tag: tag:stsci.edu:asdf/unit/quantity-1.*
properties:
value:
type: number
unit:
tag: tag:stsci.edu:asdf/unit/unit-1.*
enum: [sr]
- type: number
- type: "null"
required: [photmjsr, uncertainty, pixelareasr]
additionalProperties: false
Expand Down
14 changes: 14 additions & 0 deletions tests/test_schemas.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
asdf.schema.load_schema("asdf://stsci.edu/datamodels/roman/schemas/wfi_optical_element-1.0.0")["enum"]
)
EXPOSURE_TYPE_ELEMENTS = list(asdf.schema.load_schema("asdf://stsci.edu/datamodels/roman/schemas/exposure_type-1.0.0")["enum"])
EXPECTED_COMMON_REFERENCE = {"$ref": "asdf://stsci.edu/datamodels/roman/schemas/reference_files/ref_common-1.0.0"}


@pytest.fixture(scope="session", params=SCHEMA_URIS)
Expand Down Expand Up @@ -263,6 +264,19 @@ def test_reftype_tag(ref_file_uris):
assert asdf.util.uri_match(f"asdf://stsci.edu/datamodels/roman/schemas/reference_files/*{reftype}-*", schema_uri)


def test_ref_file_meta_common(ref_file_schema):
"""
Test that the meta for all reference files contains a reference to `ref_common`
"""
all_of = ref_file_schema["properties"]["meta"]["allOf"]

for item in all_of:
if item == EXPECTED_COMMON_REFERENCE:
break
else:
raise ValueError("ref_common not found in meta")


# don't test tvac or fps schemas as they are static
@pytest.mark.parametrize(
"uri",
Expand Down
Loading