Skip to content

Commit

Permalink
prepare for release v0.27 (#2622)
Browse files Browse the repository at this point in the history
* prepare for release v0.27

* Add tip

* fix tests
  • Loading branch information
hanouticelina authored Oct 22, 2024
1 parent 40a64c8 commit 41b7aaf
Show file tree
Hide file tree
Showing 8 changed files with 8 additions and 785 deletions.
35 changes: 1 addition & 34 deletions docs/source/en/guides/upload.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,41 +234,8 @@ Future(...)
### Upload a folder by chunks

[`upload_folder`] makes it easy to upload an entire folder to the Hub. However, for large folders (thousands of files or
hundreds of GB), it can still be challenging. If you have a folder with a lot of files, you might want to upload
it in several commits. If you experience an error or a connection issue during the upload, you would not have to resume
the process from the beginning.

To upload a folder in multiple commits, just pass `multi_commits=True` as argument. Under the hood, `huggingface_hub`
will list the files to upload/delete and split them in several commits. The "strategy" (i.e. how to split the commits)
is based on the number and size of the files to upload. A PR is open on the Hub to push all the commits. Once the PR is
ready, the commits are squashed into a single commit. If the process is interrupted before completing, you can rerun
your script to resume the upload. The created PR will be automatically detected and the upload will resume from where
it stopped. It is recommended to pass `multi_commits_verbose=True` to get a better understanding of the upload and its
progress.

The example below will upload the checkpoints folder to a dataset in multiple commits. A PR will be created on the Hub
and merged automatically once the upload is complete. If you prefer the PR to stay open and review it manually, you can
pass `create_pr=True`.
hundreds of GB), we recommend using [`upload_large_folder`], which splits the upload into multiple commits. See the [Upload a large folder](#upload-a-large-folder) section for more details.

```py
>>> upload_folder(
... folder_path="local/checkpoints",
... repo_id="username/my-dataset",
... repo_type="dataset",
... multi_commits=True,
... multi_commits_verbose=True,
... )
```

If you want a better control on the upload strategy (i.e. the commits that are created), you can have a look at the
low-level [`plan_multi_commits`] and [`create_commits_on_pr`] methods.

<Tip warning={true}>

`multi_commits` is still an experimental feature. Its API and behavior is subject to change in the future without prior
notice.

</Tip>

### Scheduled uploads

Expand Down
2 changes: 0 additions & 2 deletions docs/source/en/package_reference/hf_api.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,6 @@ models = hf_api.list_models()

[[autodoc]] HfApi

[[autodoc]] plan_multi_commits

## API Dataclasses

### AccessRequest
Expand Down
2 changes: 0 additions & 2 deletions docs/source/ko/package_reference/hf_api.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,6 @@ models = hf_api.list_models()

[[autodoc]] HfApi

[[autodoc]] plan_multi_commits

## API Dataclasses[[api-dataclasses]]

### AccessRequest[[huggingface_hub.hf_api.AccessRequest]]
Expand Down
12 changes: 1 addition & 11 deletions src/huggingface_hub/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@
from typing import TYPE_CHECKING


__version__ = "0.26.0.dev0"
__version__ = "0.27.0.dev0"

# Alphabetical order of definitions is ensured in tests
# WARNING: any comment added in this dictionary definition will be lost when
Expand All @@ -70,10 +70,6 @@
"logout",
"notebook_login",
],
"_multi_commits": [
"MultiCommitException",
"plan_multi_commits",
],
"_snapshot_download": [
"snapshot_download",
],
Expand Down Expand Up @@ -167,7 +163,6 @@
"create_branch",
"create_collection",
"create_commit",
"create_commits_on_pr",
"create_discussion",
"create_inference_endpoint",
"create_pull_request",
Expand Down Expand Up @@ -601,10 +596,6 @@ def __dir__():
logout, # noqa: F401
notebook_login, # noqa: F401
)
from ._multi_commits import (
MultiCommitException, # noqa: F401
plan_multi_commits, # noqa: F401
)
from ._snapshot_download import snapshot_download # noqa: F401
from ._space_api import (
SpaceHardware, # noqa: F401
Expand Down Expand Up @@ -694,7 +685,6 @@ def __dir__():
create_branch, # noqa: F401
create_collection, # noqa: F401
create_commit, # noqa: F401
create_commits_on_pr, # noqa: F401
create_discussion, # noqa: F401
create_inference_endpoint, # noqa: F401
create_pull_request, # noqa: F401
Expand Down
Loading

0 comments on commit 41b7aaf

Please sign in to comment.