0.10.0

Changes in 0.10.0

Incompatible Changes

• The configuration DataStores for xcube serve changed in an
  incompatible way with xcube 0.9.x: The value of former Identifier
  must now be assigned to Path, which is a mandatory parameter.
  Path may contain wildcard characters **, *, ?.
  Identifier is now optional, the default is
  "${store_id}~${data_id}. If given, it should only be used to
  uniquely identify single datasets within a data store
  pointed to by Path. (#516)

Enhancements

• It is now possible to use environment variables in most
  xcube configuration files. Unix bash syntax is used, i.e.
  ${ENV_VAR_NAME} or $ENV_VAR_NAME. (#580)

  Supported tools include

  • xcube gen --config CONFIG
  • xcube gen2 --stores STORES_CONFIG --service SERVICE_CONFIG
  • xcube serve -c CONFIG

• Changed the xcube gen tool to extract metadata for pre-sorting inputs
  from other than NetCDF inputs, e.g. GeoTIFF.

• Optimized function xcube.core.geom.rasterize_features().
  It is now twice as fast while its memory usage dropped to the half. (#593)

Fixes

• xcube serve now also serves datasets that are located in
  subdirectories of filesystem-based data stores such as
  "file", "s3", "memory". (#579)

• xcube serve now accepts datasets whose spatial
  resolutions differ up to 1%. (#590)
  It also no longer rejects datasets with large dimension
  sizes. (Formerly, an integer-overflow occurred in size
  computation.)

• DatasetChunkCacheSize is now optional in xcube serve
  configuration. (Formerly, when omitted, the server crashed.)

• Fixed bug that would cause that requesting data ids on some s3 stores would
  fail with a confusing ValueError.

• Fixed that only last dataset of a directory listing was published via
  xcube serve when using the DataStores configuration with
  filesystem-based datastores such as "s3" or "file". (#576)

Other

• Pinned Python version to < 3.10 to avoid import errors caused by a
  third-party library.

• Values obs and local for the FileSystem parameter in xcube
  configuration files have been replaced by s3 and file, but are kept
  temporarily for the sake of backwards compatibility.

All Issues addressed

• Martin's gen support for geotiff inputs by @martin-boettcher in #568
• naming of secrets GH PAT in the GH actions workflow by @dzelge in #575
• Pinning openssl to fix failing unittest on mac by @AliceBalfanz in #578
• Use env vars in configs by @forman in #582
• Restrict python version by @TonioF in #584
• Retrieval of Data IDs fails on some s3 stores by @TonioF in #587
• Only last configured Identifier is published via xcube serve by @TonioF in #589
• now also serve nested datasets by @forman in #588
• Revise spatial x,y resolutions check by @forman in #591
• extract CLI: ensure CSV times are timezone-naive by @pont-us in #596
• Daskify geometry operations, Part 1 by @forman in #594
• Fixing tests for accessing data from xcube server via python api by @AliceBalfanz in #592
• Toniof 598 gridmapping test fail by @TonioF in #599
• Unify Dataset IO by @TonioF in #535
• Tile server: ensure timezone-naive time selector by @pont-us in #602
• Replaced file system values obs and local with s3 and file by @TonioF in #600
• Corrected retrieval of store params by @TonioF in #606
• adapt to zarr 2.11 by @TonioF in #607

New Contributors

• @martin-boettcher made their first contribution in #568

Full Changelog: v0.9.2...v0.10.0

0.9.2

Changes in 0.9.2

Fixes

• A xcube.core.store.fs.impl.FSDataStore no longer raises exceptions when
  root directories in data store configurations do not exist. Instead, they
  are created when data is written.

0.9.2.dev1

Changes in 0.9.2 (in development)

• Use xcube cci v0.9.2.dev0 in dockerfile

0.9.2.dev0

Changes in 0.9.2 (in development)

• Be more robust with respect to non-existing data store root directories.

0.9.1

Changes in 0.9.1

New features

• The xcube.core.maskset.MaskSet class no longer allocates static numpy
  arrays for masks. Instead, it uses lazy dask arrays. (#556)

• Function xcube.core.geom.mask_dataset_by_geometry has a new parameter
  all_touched: If True, all pixels intersected by geometry outlines will
  be included in the mask. If False, only pixels whose center is within the
  polygon or that are selected by Bresenham’s line algorithm will be included
  in the mask. The default value is set to False.

Other

• Updated Dockerfile: Removed the usage of a no-longer-maintained base image.
  Ensured that the version tag 'latest' can be used with installation mode
  'release' for xcube plugins.

• The xcube package now requires xarray >= 0.19, zarr >= 2.8,
  pandas >= 1.3.

Full Changelog: v0.9.0...v0.9.1

0.9.0

Changes in 0.9.0

New features

• The implementations of the default data stores s3, directory,
  and memory have been replaced entirely by a new implementation
  that utilize the fsspec
  Python package. The preliminary filesystem-based data stores
  are now s3, file, and memory. All share a common implementations
  and tests. Others filesystem-based data stores can be added easily
  and will follow soon, for example hdfs.
  All filesystem-based data stores now support xarray
  datasets (type xarray.Dataset) in Zarr and NetCDF format as
  well as image pyramids (typexcube.core.multilevel.MultiLevelDataset)
  using a Zarr-based multi-level format. (#446)

• Several changes became necessary on the xcube Generator
  package xcube.core.gen2 and CLI xcube gen2.
  They are mostly not backward compatible:

  • The only supported way to instantiate cube generators is the
    CubeGenerator.new() factory method.
  • CubeGenerator.generate_cube() and CubeGenerator.get_cube_info()
    both now receive the request object that has formerly been passed
    to the generator constructors.
  • The CubeGenerator.generate_cube() method now returns a
    CubeGeneratorResult object rather than a simple string
    (the written data_id).
  • Empty cubes are no longer written, a warning status is
    generated instead.
  • The xcube gen2 CLI xcube gen2 has a new option --output RESULT
    to write the result to a JSON file. If it is omitted,
    the CLI will dump the result as JSON to stdout.

• Numerous breaking changes have been applied to this version
  in order to address generic resampling (#391), to support other
  CRS than WGS-84 (#112), and to move from the struct data cube
  specification to a more relaxed cube convention (#488):

  • The following components have been removed entirely
    • module xcube.core.imgeom with class ImageGeom
    • module xcube.core.geocoding with class GeoCoding
    • module xcube.core.reproject and all its functions
  • The following components have been added
    • module xcube.core.gridmapping with new class GridMapping
      is a CF compliant replacement for classes ImageGeom and GeoCoding
  • The following components have changed in an incompatible way:
    • Functionxcube.core.rectify.rectify_dataset() now uses
      source_gm: GridMapping and target_gm: GridMapping instead of
      geo_coding: GeoCoding and output_geom: ImageGeom.
    • Functionxcube.core.gen.iproc.InputProcessor.process() now uses
      source_gm: GridMapping and target_gm: GridMapping instead of
      geo_coding: GeoCoding and output_geom: ImageGeom.
  • xcube no longer depends on GDAL (at least not directly).

• Added a new feature to xcube called "BYOA" - Bring your own Algorithm.
  It is a generic utility that allows for execution of user-supplied
  Python code in both local and remote contexts. (#467)
  The new xcube.core.byoa package hosts the BYOA implementation and API.
  The entry point to the functionality is the xcube.core.byoa.CodeConfig
  class. It is currently utilized by the xcube Cube Generator that can now
  deal with an optional code_config request parameter. If given,
  the generated data cube will be post-processed by the configured user-code.
  The xcube Cube Generator with the BYOA feature is made available through the

  1. Generator API xcube.core.gen2.LocalCubeGenerator and
     xcube.core.gen2.service.RemoteCubeGenerator;
  2. Generator CLI xcube gen2.

• A dataset's cube subset and its grid mapping can now be accessed through
  the xcube property of xarray.Dataset instances. This feature requires
  importing the xcube.core.xarraypackage. Let dataset be an
  instance of xarray.Dataset, then

  • dataset.xcube.cube is a xarray.Dataset that contains all cube
    variables of dataset, namely the ones with dimensions
    ("time", [...,], y_dim_name, x_dim_name), where y_dim_name,
    x_dim_name are determined by the dataset's grid mapping.
    May be empty, if dataset has no cube variables.
  • dataset.xcube.gm is a xcube.core.gridmapping.GridMapping that
    describes the CF-compliant grid mapping of dataset.
    May be None, if dataset does not define a grid mapping.
  • dataset.xcube.non_cube is a xarray.Dataset that contains all
    variables of dataset that are not in dataset.xcube.cube.
    May be same as dataset, if dataset.xcube.cube is empty.

• Added a new utility module xcube.util.temp that allows for creating
  temporary files and directories that will be deleted when the current
  process ends.

• Added function xcube.util.versions.get_xcube_versions()
  that outputs the versions of packages relevant for xcube.
  Also added a new CLI xcube versions that outputs the result of the
  new function in JSON or YAML. (#522)

Other

• The xcube cube generator (API xcube.core.gen2, CLI xcube gen2)
  will now write consolidated Zarrs by default. (#500)

• xcube now issues a warning, if a data cube is opened from object
  storage, and credentials have neither been passed nor can be found,
  and the object storage has been opened with the default anon=False. (#412)

• xcube no longer internally caches directory listings, which prevents
  the situation where a data cube that has recently been written into object
  storage cannot be found.

• Removed example notebooks that used hard-coded local file paths. (#400)

• Added a GitHub action that will run xcube unit tests, and build and
  push Docker images. The version tag of the image is either latest when
  the master changed or equals the release tag.

• Removed warning module 'xcube_xyz' looks like an xcube-plugin but lacks a callable named 'init_plugin.

• Fixed an issue where xcube serve provided wrong layer source options for
  OpenLayers XYZ
  when latitude coordinates where increasing with the coordinate index. (#251)

• Function xcube.core.normalize.adjust_spatial_attrs() no longer removes
  existing global attributes of the form geospatial_vertical_<property>.

• Numerous classes and functions became obsolete in the xcube 0.9
  code base and have been removed, also because we believe there is
  quite rare outside use, if at all.

  Removed from xcube.util.tiledimage:

  • class DownsamplingImage
  • class PilDownsamplingImage
  • class NdarrayDownsamplingImage
  • class FastNdarrayDownsamplingImage
  • class ImagePyramid
  • function create_pil_downsampling_image()
  • function create_ndarray_downsampling_image()
  • function downsample_ndarray()
  • functions aggregate_ndarray_xxx()

  Removed from xcube.util.tilegrid:

  • functions pow2_2d_subdivision()
  • functions pow2_1d_subdivision()

0.9.0.dev2

Changes in 0.9.0 (in development)

New features

• The implementations of the default data stores s3, directory,
  and memory have been replaced entirely by a new implementation
  that utilize the fsspec
  Python package. The preliminary filesystem-based data stores
  are now s3, file, and memory. All share a common implementations
  and tests. Others filesystem-based data stores can be added easily
  and will follow soon, for example hdfs.
  All filesystem-based data stores now support xarray
  datasets (type xarray.Dataset) in Zarr and NetCDF format as
  well as image pyramids (typexcube.core.multilevel.MultiLevelDataset)
  using a Zarr-based multi-level format. (#446)

Fixes

• xcube versions will not throw an error anymore if a package does not provide __version__. It accepts now version as well or reports version 'unkown'

Other

• The xcube cube generator (API xcube.core.gen2, CLI xcube gen2)
  will now write consolidated Zarrs by default. (#500)

0.9.0.dev1

Changes in 0.9.0 (in development)

New features

• The implementations of the default data stores s3, directory,
  and memory have been replaced entirely by a new implementation
  that utilize the fsspec
  Python package. The preliminary filesystem-based data stores
  are now s3, file, and memory. All share a common implementations
  and tests. Others filesystem-based data stores can be added easily
  and will follow soon, for example hdfs.
  All filesystem-based data stores now support xarray
  datasets (type xarray.Dataset) in Zarr and NetCDF format as
  well as image pyramids (typexcube.core.multilevel.MultiLevelDataset)
  using a Zarr-based multi-level format. (#446)

• Several changes became necessary on the xcube Generator
  package xcube.core.gen2 and CLI xcube gen2.
  They are mostly not backward compatible:

  • The only supported way to instantiate cube generators is the
    CubeGenerator.new() factory method.
  • CubeGenerator.generate_cube() and CubeGenerator.get_cube_info()
    both now receive the request object that has formerly been passed
    to the generator constructors.
  • The CubeGenerator.generate_cube() method now returns a
    CubeGeneratorResult object rather than a simple string
    (the written data_id).
  • Empty cubes are no longer written, a warning status is
    generated instead.
  • The xcube gen2 CLI xcube gen2 has a new option --output RESULT
    to write the result to a JSON file. If it is omitted,
    the CLI will dump the result as JSON to stdout.

• Numerous breaking changes have been applied to this version
  in order to address generic resampling (#391), to support other
  CRS than WGS-84 (#112), and to move from the struct data cube
  specification to a more relaxed cube convention (#488):

  • The following components have been removed entirely
    • module xcube.core.imgeom with class ImageGeom
    • module xcube.core.geocoding with class GeoCoding
    • module xcube.core.reproject and all its functions
  • The following components have been added
    • module xcube.core.gridmapping with new class GridMapping
      is a CF compliant replacement for classes ImageGeom and GeoCoding
  • The following components have changed in an incompatible way:
    • Functionxcube.core.rectify.rectify_dataset() now uses
      source_gm: GridMapping and target_gm: GridMapping instead of
      geo_coding: GeoCoding and output_geom: ImageGeom.
    • Functionxcube.core.gen.iproc.InputProcessor.process() now uses
      source_gm: GridMapping and target_gm: GridMapping instead of
      geo_coding: GeoCoding and output_geom: ImageGeom.
  • xcube no longer depends on GDAL (at least not directly).

• A dataset's cube subset and its grid mapping can now be accessed through
  the xcube property of xarray.Dataset instances. This feature requires
  importing the xcube.core.xarraypackage. Let dataset be an
  instance of xarray.Dataset, then

  • dataset.xcube.cube is a xarray.Dataset that contains all cube
    variables of dataset, namely the ones with dimensions
    ("time", [...,], y_dim_name, x_dim_name), where y_dim_name,
    x_dim_name are determined by the dataset's grid mapping.
    May be empty, if dataset has no cube variables.
  • dataset.xcube.gm is a xcube.core.gridmapping.GridMapping that
    describes the CF-compliant grid mapping of dataset.
    May be None, if dataset does not define a grid mapping.
  • dataset.xcube.non_cube is a xarray.Dataset that contains all
    variables of dataset that are not in dataset.xcube.cube.
    May be same as dataset, if dataset.xcube.cube is empty.

Other

• xcube now issues a warning, if a data cube is opened from object
  storage, and credentials have neither been passed nor can be found,
  and the object storage has been opened with the default anon=False. (#412)
• xcube no longer internally caches directory listings, which prevents
  the situation where a data cube that has recently been written into object
  storage cannot be found.
• Removed example notebooks that used hard-coded local file paths. (#400)
• Added a GitHub action that will run xcube unit tests, and build and
  push Docker images. The version tag of the image is either latest when
  the master changed or equals the release tag.
• Removed warning module 'xcube_xyz' looks like an xcube-plugin but lacks a callable named 'init_plugin.
• Fixed an issue where xcube serve provided wrong layer source options for
  OpenLayers XYZ
  when latitude coordinates where increasing with the coordinate index. (#251)
• Updated plugin version in docker image to:
  • XCUBE_SH_VERSION=0.9.0.dev0
  • XCUBE_CCI_VERSION=0.9.0.dev0
  • XCUBE_CDS_VERSION=0.9.0

0.8.2

Changes in 0.8.2

• Fixed the issue that xcube gen2 would not print tracebacks to stderr when
  raising CubeGeneratorErrors (#448).
• Enhanced xcube.core.normalize.normalize_dataset() function to also
  normalize datasets with latitudes given as
  latitude_centers and to invert decreasing latitude coordinate values.
• Introduced xcube.core.normalize.cubify_dataset() function to normalize a
  dataset and finally assert the result complies to the
  xcube dataset conventions.
• Fixed that data stores directory and s3 were not able to handle data
  identifiers that they had assigned themselves during write_data(). (#450)
• Pinned dependencies
  • dask =2021.3.1 because of memory leaks in versions >=2021.4
  • xarray =0.18.2 because a lot of tests fail with versions >= 0.19
  • pandas >1.1,<1.3 because 1.3+ doesn't work with xarray < 0.19

0.8.2.dev4

Note: Release build from branch forman-xxx-0.8.2.dev1

Changes in 0.8.2 (in development)

• Fixed the issue that xcube gen2 would not print tracebacks to stderr when raising
  CubeGeneratorErrors (#448).
• Enhanced xcube.core.normalize.normalize_dataset() function to also normalize datasets with latitudes given as
  latitude_centers and to invert decreasing latitude coordinate values.
• Introduced xcube.core.normalize.cubify_dataset() function to normalize a dataset
  and finally assert the result complies to the xcube dataset conventions.
• Fixed that data stores directory and s3 were not able to handle data identifiers that they
  had assigned themselves during write_data(). (#450)
• Pinned dependencies
  • dask =2021.3.1 because of memory leaks in versions >=2021.4
  • xarray =0.18.2 because a lot of tests fail with versions >= 0.19
  • pandas >1.1,<1.3 because 1.3+ doesn't work with xarray < 0.19