New ESACCI AEROSOL CMORizer (Python version) #3629
Conversation
There was a problem hiding this comment.
I tested the downloader and formatter for two years (since otherwise it would have taken hours to wait for the data to download). Both worked fine for the two years. The files look ok, and I checked if the values are in a meaningful range. Looks all fine.
Well done, @axel-lauer!
I have a few comments regarding the description in some of the files.
| @@ -420,11 +420,14 @@ datasets: | |||
| ESACCI-AEROSOL: | |||
| tier: 2 | |||
| source: ftp://anon-ftp.ceda.ac.uk/neodc/esacci/aerosol/data/ | |||
| last_access: 2019-01-24 | |||
| # last_access: 2019-01-24 | |||
There was a problem hiding this comment.
Do you leave this line in (just commented out) to show how the cmorizer would work with the older version of the data? This is not clear here.
| ATSR2_SU/L3/v4.21/MONTHLY/ (1997-2002) | ||
| AATSR_SU/L3/v4.21/MONTHLY/ (2003-2011) | ||
| # ATSR2_SU/L3/v4.21/MONTHLY/ (1997-2002) | ||
| # AATSR_SU/L3/v4.21/MONTHLY/ (2003-2011) |
There was a problem hiding this comment.
Same here. I think you commented it out to so that it could be used again for the older version, but it would be nice if this would be described somewhere.
| @@ -156,8 +156,23 @@ diagnostics: | |||
| scripts: null | |||
|
|
|||
|
|
|||
| ESACCI-AEROSOL: | |||
| description: ESACCI-AEROSOL check | |||
| # ESACCI-AEROSOL: | |||
There was a problem hiding this comment.
I think also here it would be nice to have an additional comment that describes why the collection of lines were commented out.
There was a problem hiding this comment.
nice one @axel-lauer many thanks, mate! I reviewed only the Python bits (apols for me not speaking NCL), found just a couple minor points, so am moving this to "in technical review" 🍺
| dataset_time_calender) | ||
| nan_cube.coord('time').points = float(newtime) | ||
|
|
||
| return nan_cube |
There was a problem hiding this comment.
note however that those values are not NaNs in the strict data type sense, they are numpy.Masked objects (that look like -- and have actual values of fill_value which I think it's 1.e+20 here)
| cube.add_dim_coord(time_coord, 0) | ||
|
|
||
| for attr in drop_attrs: | ||
| if attr in cube.attributes.keys(): |
There was a problem hiding this comment.
| if attr in cube.attributes.keys(): | |
| if attr in cube.attributes: |
| loop_date += relativedelta.relativedelta(months=1) | ||
|
|
||
| iris.util.unify_time_units(full_list) | ||
| cube = full_list.concatenate_cube() |
There was a problem hiding this comment.
any chance this may fail (differing attributes etc)? Could try use the more accommodating esmvalcore.preprocessor._io.concatenate if it fails
valeriupredoi
added
in technical review
and removed
looking for technical reviewer
labels
Description
This PR provides new downloading and formatting scripts for ESACCI AEROSOL data (monthly and daily values) in Python replacing the old NCL version supporting monthly values only.
Checklist
It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.
New or updated data reformatting script