feat: skip dataset re-download and ensure safe dataset syncing

[JSabadin]

Skip Redownload Dataset and Fix Potential Race Conditions

A new parameter, update_mode, has been introduced in the LuxonisLoader. This parameter allows more control over when the dataset from the cloud is already downloaded locally.

• If set to UpdateMode.ALWAYS, the loader will always re-download the dataset locally.
• If set to UpdateMode.IF_EMPTY, it will only download the dataset if it does not already exist locally.

Example Usage:

from luxonis_loader import LuxonisLoader, UpdateMode

loader = LuxonisLoader(
    dataset,
    update_mode=UpdateMode.IF_EMPTY
)

This ensures that the loader uses the local dataset if one is already present, helping to avoid unnecessary downloads.

---

FileLock Added for Safe Syncing

To address potential race conditions, a FileLock mechanism has been implemented in the following areas:

1. sync_from_cloud Method: This prevents multiple processes from attempting to sync the dataset simultaneously in a distributed environment (e.g., DDP on GCP).
2. _get_metadata in Dataset Initialization: Ensures safe concurrent access when multiple processes initialize the dataset and loader before setting up the DDP environment.

Additionally, the _load_df_offline method has been updated to:

• Only access local storage.
• Avoid attempting to download dataset annotations from the cloud, as sync_from_cloud already handles this.

These changes help to ensure safe and efficient operation in distributed environments by preventing redundant downloads and race conditions.

[codecov]

Codecov Report

Attention: Patch coverage is 92.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 95.81%. Comparing base (1cbc795) to head (2c49d12).
Report is 1 commits behind head on main.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
luxonis_ml/data/datasets/luxonis_dataset.py	88.23%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #220      +/-   ##
==========================================
- Coverage   95.84%   95.81%   -0.04%     
==========================================
  Files          88       88              
  Lines        4571     4583      +12     
==========================================
+ Hits         4381     4391      +10     
- Misses        190      192       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kozlov721]

Doesn't the force flag already do what the skip_redownload_dataset does?

[JSabadin]

Doesn't the force flag already do what the skip_redownload_dataset does?

They serve different purposes:

• force always triggers a fresh download, ignoring what’s on disk.
• skip_redownload_dataset simply prevents downloads if data already exists locally.

Probably a string parameter like update_mode could encapsulate all scenarios better:

• "always" = force a fresh download
• "if_empty" = only download if there’s no local data
• "never" = skip the download entirely

[kozlov721]

Probably a string parameter like update_mode could encapsulate all scenarios better:

• "always" = force a fresh download
• "if_empty" = only download if there’s no local data
• "never" = skip the download entirely

Merging those two parameters to one would be better. Imho it's not very clear what's the difference between force=False and skip_redownloading_dataset=True.