Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/merge datasets #221

Merged
merged 21 commits into from
Jan 13, 2025
Merged

Feat/merge datasets #221

merged 21 commits into from
Jan 13, 2025

Conversation

JSabadin
Copy link
Contributor

@JSabadin JSabadin commented Jan 9, 2025

Merging and Cloning Datasets in LuxonisDataset

This update adds the following features to LuxonisDataset:

  1. Cloning Datasets: Create a copy of a dataset under a new name.
  2. Merging Datasets: Merge datasets either:
    • In-place: Modify the existing dataset.
    • Out-of-place: Create a new merged dataset.
    • Note: Merging is supported only between cloud + cloud or local + local datasets. Cloud + local merging is not supported.
  3. Faster Dataset Deletion: Improved deletion speed using a thread pool in list_datasets.

Code Examples

Clone a Dataset

cloned_dataset = dataset.clone(new_dataset_name="dataset_cloned")

Merge Datasets

# In-place merge
dataset1.merge_with(dataset2, inplace=True)

# Out-of-place merge
merged_dataset = dataset1.merge_with(dataset2, inplace=False, new_dataset_name="merged_dataset")

@JSabadin JSabadin requested a review from a team as a code owner January 9, 2025 10:16
@JSabadin JSabadin requested review from kozlov721, klemen1999, tersekmatija and conorsim and removed request for a team January 9, 2025 10:16
@github-actions github-actions bot added enhancement New feature or request utils Changes affecting luxonis_ml.utils subpackage labels Jan 9, 2025
Copy link

codecov bot commented Jan 9, 2025

Codecov Report

Attention: Patch coverage is 94.87179% with 8 lines in your changes missing coverage. Please review.

Project coverage is 95.81%. Comparing base (11ee765) to head (6022e67).
Report is 1 commits behind head on main.

✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
luxonis_ml/data/datasets/luxonis_dataset.py 92.59% 8 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main     #221    +/-   ##
========================================
  Coverage   95.81%   95.81%            
========================================
  Files          88       88            
  Lines        4583     4728   +145     
========================================
+ Hits         4391     4530   +139     
- Misses        192      198     +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@klemen1999 klemen1999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM, left some comments

luxonis_ml/data/datasets/luxonis_dataset.py Outdated Show resolved Hide resolved
luxonis_ml/data/datasets/luxonis_dataset.py Outdated Show resolved Hide resolved
luxonis_ml/utils/filesystem.py Outdated Show resolved Hide resolved
tests/test_data/test_dataset.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@kozlov721 kozlov721 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some small comments, otherwise LGTM

tests/test_data/test_dataset.py Show resolved Hide resolved
luxonis_ml/utils/filesystem.py Outdated Show resolved Hide resolved
@github-actions github-actions bot added data Changes affecting luxonis_ml.data subpackage DevOps Changes related to DevOps labels Jan 9, 2025
@github-actions github-actions bot added the tests Adding or changing tests label Jan 10, 2025
@klemen1999 klemen1999 merged commit 539fde4 into main Jan 13, 2025
11 of 12 checks passed
@klemen1999 klemen1999 deleted the feat/merge-datasets branch January 13, 2025 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data Changes affecting luxonis_ml.data subpackage DevOps Changes related to DevOps enhancement New feature or request tests Adding or changing tests utils Changes affecting luxonis_ml.utils subpackage
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants