Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Balinus/patch 1 savedoc #438

Merged
merged 3 commits into from
Sep 22, 2024
Merged

Balinus/patch 1 savedoc #438

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f139643
Update write.md
Balinus Sep 13, 2024
fa90f4a
Update write.md
Balinus Sep 13, 2024
96d46c2
Update docs/src/UserGuide/write.md
Balinus Sep 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/src/UserGuide/write.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,17 @@ savedataset(ds, path="ds.zarr", driver=:zarr)
nothing # hide
````

Save a dataset to Zarr format with compression:

````@example write
using Blosc
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need to add Blosc to the Project toml, explicitly. Or, loaded it from whatever other package is also available.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding Blosc to YAXArrays's Project.toml? or perhaps you mean YAXArrays/docs/Project.toml ?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to docs 😄 .

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can access the compressor within Zarr.jl directly. I will adapt the code

n = 7 # compression level, number between 0 (no compression) and 9 (max compression)
compression = Zarr.BloscCompressor(clevel=n)

savedataset(ds, path="ds.zarr", driver=:zarr, compressor=compression)
nothing # hide
````

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should have more than 1 compression level, and compare them both at the end, namely, what their disk sizes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean, compare on my computer and report sizes in the documentation?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here, directly. Let's say a ds_level_1.zarr and ds_level4.zarr. And then query their sizes and print them.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had never use this argument, so..., also I don't know what are the defaults, and hence how different things actually are.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, I see I didn't even thought that the code were actually run! We'll look into it

Copy link
Contributor Author

@Balinus Balinus Sep 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Humm, strangely, the compression does not seems to work as expected. There is little effect on Zarr folders and there is some on netcdf files...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

76M test1.nc
103M test_default.nc
75M test9.nc

84M test1.zarr
83M test_default.zarr
82M test9.zarr

## Write NetCDF

Save a single YAXArray to a directory:
Expand All @@ -45,6 +56,14 @@ savedataset(ds, path="ds.nc", driver=:netcdf)
nothing # hide
````

Save a dataset to NetCDF format with compression:

````@example write
n = 7 # compression level, number between 0 (no compression) and 9 (max compression)
savedataset(ds, path="ds.zarr", driver=:zarr, compress=n)
Balinus marked this conversation as resolved.
Show resolved Hide resolved
nothing # hide
````

## Overwrite a Dataset
If a path already exists, an error will be thrown. Set `overwrite=true` to delete the existing dataset

Expand Down
Loading