Fix IODA observations for the 2024052700 case

[HuiLiu-NOAA]

I checked the ioda adpsfc.nc in the 2024052700/data/obs prepared by GSL team. It looks that the variables are quite different/inconsistent to the JEDI's standard format. For example, the variable ObsError is set for variable "winds", not for "windEastward" and "windNorthward". The variable "specificHumidity" is not fully set up for its metadata as well. So, It may need to fix the associated yaml files and re-generate observation IODA files for this new case for consistent testing and future applications.

[guoqing-noaa]

@HuiLiu-NOAA Thanks for looking at this.
These obs were generated using Sijie's committed yaml files in the rrfs-workflow.
https://github.com/NOAA-EMC/rrfs-workflow/blob/rrfs-mpas-jedi/parm/prepbufr_adpsfc.yaml

I have not gotten into the details yet. So I will tag @spanNOAA for this issue.

Some superficial comments:

1. What is required by the mpasjedi data assimilation?
2. It looks to me it is more reasonable to set one obs error for wind, instead of separate obs error for u and v. But if mpasjedi needs separate errors, we have to do it that way.

[HuiLiu-NOAA]

@guoqing-noaa : It may be helpful to look at the mpas-jedi/test/ and ufo/test/ sample observation input files for the desired variable standards by JEDI. There are other differences (in addition to the wind errors) between the ioda obs files generated by GSL and the standard JEDI observations.

[delippi]

The ObsErrors provided in the ioda file don't have to be used—in fact, they probably shouldn’t be. Instead, you should specify the ObsErrors directly in your JEDI yaml file. This allows you to set the errors for individual variables like u and v separately. If you're performing any kind of validation work, it's essential to set your own ObsErrors in the yaml to match the values in GSI's errtable. The ObsErrors in the ioda file are unlikely to match what GSI uses, which is why I recommend not including ObsErrors in the ioda file at all and requiring users specify them during assimilation.

For example, I’ve specifically requested that ObsErrors not be included in Mesonet data within ioda files. ObsErrors, along with other settings, can vary significantly between systems like RDAS, HDAS, RTMA, and so on. Each of these systems has its own yaml configurations to handle these differences—hence why each system will have its own App. The simplest approach is to define ObsErrors directly in your JEDI yaml file to ensure consistency.

[HuiLiu-NOAA]

Thanks @delippi.
I would prefer the ObsErr is set correctly in the ioda files, whether it will be used or not later. I also saw other inconsistencies in the ioda file, not just ObsErr.

[guoqing-noaa]

@delippi I agree with all the considerations here!
Just one another perspectives, we may use RDASApp to do some MPAS global data assimilation tests needed by some of our projects. We don't have a clear picture at the moment, but it looks like the global applications prefer to use ObsErrors for individual observations. So it will be good if we have options to use either ObsErrors in ioda files or from external tables/settings in yaml files.

[delippi]

@HuiLiu-NOAA Well, what specifically is set incorrectly in ObsError? Just that there is one error "winds" for u and v? I don't think GSI specifies an error for u and a separate error for v. Having two separate errors for windsEastward and windsNorthward would just be duplicate info and take up extra space. Do you have trouble assimilating winds because of this? What other differences are you seeing?

@guoqing-noaa, I might need more convincing that ObError in the ioda is better just specifying it during assimilation. Do you know where those would get set? I thought it would just be the errors from the bufr files--which might not be what we even want to use for our application. Also, how can you ensure that you're using the same errors between GSI and JEDI?

[HuiLiu-NOAA]

@delippi :

1. I think JEDI requires ObsErr set for windEastward and windNorthward, not "winds".

2. I guess "ensure that you're using the same errors between GSI and JEDI" is just for the initial testing. Later, in operational testing, the GSI errors actually should not be used directly any more, and the errors should be adjusted for the MPAS model accordingly.

[guoqing-noaa]

@delippi Good question. NOTE, I am talking about a global application which is a side-topic for RDASApp at the moment. The obs errors come from the GDAS prepbufr files.

The hofx validation is very helpful to understand the differences between GSI and JEDI. But in the mpas+jedi world, we don't have to repeat GSI exactly. Matching or not, we will need to move forward as NCAR did. We can also use the Desroziers method to fine tune obs error (we were aware that the current RAP/HRRR/RRFS error table is not optimal and has lots of tuning space).

[delippi]

@HuiLiu-NOAA, can you take a look at the new ioda_adpsfc.nc that I just generated? Give it a try and let me know if there are any issues. I have data currently here:
/scratch2/NCEPDEV/fv3-cam/Donald.E.Lippi/RRFSv2/ioda_processing/ioda/

My plan is to have two versions of each ioda file. One will be the full data set. The other (with _dc) in the name will be the offline domain checked versions in case that is needed for testing things that the online domain check can't handle yet.

@guoqing-noaa I am using the yamls that @spanNOAA worked on that you linked above, but I'm making the necessary updates to them. I do think they work well!

[guoqing-noaa]

@delippi Are your changes on yam files related to the offline domain check?

[delippi]

@guoqing-noaa, no. The changes are things like ObsError/wind --> ObsError/windEastward and for windNorthward too, changing data type from integer to float in some instances where I knew that was a problem, or renaming superficial things like units for better consistency. Also, I am reorganizing the yamls a little so that the order of most things are all consistent between the different yamls, when possible; the point of this is so I can use vimdiff (vi -d) more efficiently to compare differences or inconsistencies in various things.

I update the yamls with the changes preivously noted. Then I run the converter and it produces the ioda with all the obs. Then I run the offline domain check and it creates a separate ioda file with the _dc name.

[guoqing-noaa]

@delippi Thanks for the explanation. Could you commit your changes to the rrfs-mpas-jedi branch of rrfs-workflow?

[delippi]

@guoqing-noaa, yes, I can do that. Would we also want them in RDASApp? Let me know if you have a different idea, but I'll probably put them in rrfs-test/IODA/yaml? That is where Praveen has a few yamls already.

[ShunLiu-NOAA]

@guoqing-noaa Let's hold on JEDI update in rrfs-workflow. As ISSUE[https://github.com//issues/178], Sam and Donnie will work on an "unified" ioda test data and yamls. We will add JEDI modules to rrfs-workflow after this effort.

[HuiLiu-NOAA]

@delippi : I just tested your new ioda obs file /scratch2/NCEPDEV/fv3-cam/Donald.E.Lippi/RRFSv2/ioda_processing/ioda/ioda_adpsfc.nc. I got a complaint about missing variable "PreQC":

Exception: Reason: An exception occurred inside ioda while opening a variable.
91: name: PreQC/windEastward
91: source_column: 0

[delippi]

@HuiLiu-NOAA, thanks for giving that a try. I believe QualityMarker is what is supposed to replace PreQC. There are certain functions (e.g., ObsFunction/ObsErrorFactorConventional) in JEDI that expect PreQC though (I think because people use the GSI-IODA and that's what it is called there). If you can, for now, just comment out that part of your JEDI yaml. OR you might be able to replace PreQC with QualityMarker in your JEDI yaml. We do need to either get that fixed or decide to use PreQC inside IODA instead.

[HuiLiu-NOAA]

One more place to look at:

The specificHumidity and relativeHumidity are mixed in the ioda file, and e.g., ObsType is defined for specificHumidity, but not for "relativeHumidity".

[HuiLiu-NOAA]

"TunedObsError" is defined for relativeHUmidity, but not for "specificHumidity" etc...

[delippi]

@HuiLiu-NOAA, I was wondering about that part too. I know in GSI we specify RH oberrors and it is then scaled to specific humidity. When setting the oberrors directly in the JEDI yaml, I've been using the exact RH oberrors from GSI and I can confirm that they do get scaled to specific humidity when I've compared them in GSI vs FV3-JEDI. So my thinking is that the RH ob errors should be the correct ones to use--or at least they need to be in units of RH.

[delippi]

@HuiLiu-NOAA, actually that wasn't exactly true. I use the RH oberrors from GSI/10 because GSI errtable is units of RH %/10. So for example, a value of 0.7 in the errtable would need to be 0.07 in JEDI.

[HuiLiu-NOAA]

thanks @delippi . I think these ioda files need/deserve examined for every piece carefully.

[delippi]

I agree. This is also why I would just define the errors in the JEDI yaml instead of relying on the IODA-defined errors to be correct. I don't know how/where/when we plan to retune the ob errors in the future. I don't know if those will be calculated during the IODA step each time or estimated before hand. But I would argue that the IODA-define ob errors only be used for very preliminary tests but otherwise require users to define it themselves within the JEDI yaml. That is more easily verified to check against GSI since you can ensure they are set the same across systems. I don't know how you can verify that the ob errors are correct/reliable that are currently in the IODA file.

I'm curious how you use the IODA-defined obserror. I still haven't gotten that to work for me. Can you share an example yaml? I'd like to try it myself.

[delippi]

@HuiLiu-NOAA @guoqing-noaa @ShunLiu-NOAA @spanNOAA @hu5970
I have a few updates on ObsError in IODA. I believe the JEDI yaml can be set to use the IODA-defined ObsError via something like:

       obs filters:
         - filter: Perform Action
           filter variables:
           - name: airTemperature
           action:
             name: assign error
             error function:
               name: ObsError/airTemperature

HOWEVER, I am getting zero increments! I checked and this is actually because all the ObsError information in the original IODA files, the IODA files I regenerated, and the prepbufr files are missing! I'm still very much in favor for removing this information from the IODA file and just requiring users to specify it directly in JEDI. I believe this also means that you should be able to specify ObsError/wind for both windEast/Northward if those errors weren't missing. I wanted to get more opinions on removing the ObsError info before doing anything else.

[spanNOAA]

@HuiLiu-NOAA @guoqing-noaa @ShunLiu-NOAA @spanNOAA @hu5970 I have a few updates on ObsError in IODA. I believe the JEDI yaml can be set to use the IODA-defined ObsError via something like:

       obs filters:
         - filter: Perform Action
           filter variables:
           - name: airTemperature
           action:
             name: assign error
             error function:
               name: ObsError/airTemperature

HOWEVER, I am getting zero increments! I checked and this is actually because all the ObsError information in the original IODA files, the IODA files I regenerated, and the prepbufr files are missing! I'm still very much in favor for removing this information from the IODA file and just requiring users to specify it directly in JEDI. I believe this also means that you should be able to specify ObsError/wind for both windEast/Northward if those errors weren't missing. I wanted to get more opinions on removing the ObsError info before doing anything else.

I recommend leaving the error section as is. This is because the prepbufr file currently used by RRFS lacks error information, while some other prepbufr files do contain it.

[HuiLiu-NOAA]

I would prefer to keep ObsErr from bufr files as well. I also suspect that the JEDI default is to use the ObsError from ioda files, which does not require explicit specification in yaml files as I see from my experiments.

[delippi]

I would prefer to keep ObsErr from bufr files as well. I also suspect that the JEDI default is to use the ObsError from ioda files, which does not require explicit specification in yaml files as I see from my experiments.

@HuiLiu-NOAA, I just tried this by artificially setting the ObsError in the IODA file so that it isn't missing. It does seem to be able to use the ObsError without specification. I also checked that I could specify any of the other ObsError/variable to use for airTemperature. This confirms that you could specify ObsError/wind for use with both windEast/Northward cases, but it would have to be specified. The goal is to not have to specify the ObsError, so having it separated out is what is needed. Now I'm not sure what to do with ObsError/relativeHumidity for use with specificHumidity obs. I believe the errors need to be in terms of RH as they will get scaled to specific humidity later, but JEDI can't directly use those errors without being named ObsError/specificHumidity. The only way I see to not have to specify this is to change the name to ObsError/specificHumidity but leave the longName and units in terms of RH, but I don't particularly like that idea. Are there any suggestions?