Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement catalog filter for KedroDataCatalog #4449

Merged
merged 48 commits into from
Feb 7, 2025

Conversation

ElenaKhaustova
Copy link
Contributor

Description

Implemented KedroDataCatalog.filter() to filter datasets by name and type.

Solves #3917

Reasoning: #3917 (comment)

Development notes

Implemented on top of #4448 <-- please review it first.

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

  • Read the contributing guidelines
  • Signed off each commit with a Developer Certificate of Origin (DCO)
  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added a description of this change in the RELEASE.md file
  • Added tests to cover my changes
  • Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

ElenaKhaustova and others added 27 commits January 22, 2025 11:49
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
@ElenaKhaustova ElenaKhaustova marked this pull request as ready for review January 28, 2025 15:50
@astrojuanlu
Copy link
Member

Thanks @ElenaKhaustova , that's what I imagined. In my view, given that our users aren't necessarily software engineers, our APIs shouldn't just be perfect partitions of the set of possible use cases, but allow for some convenience methods that, as @datajoely says, alleviate the burden a bit.

@merelcht
Copy link
Member

merelcht commented Feb 3, 2025

I still feel like filter() is a more descriptive name and makes it more obvious that you can provide filter arguments. Whereas with list I wouldn't really expect to be able to add a regex argument.

So my vote would be: introduce filter now and remove list when we do 1.0.0

@ElenaKhaustova
Copy link
Contributor Author

Thanks @ElenaKhaustova , that's what I imagined. In my view, given that our users aren't necessarily software engineers, our APIs shouldn't just be perfect partitions of the set of possible use cases, but allow for some convenience methods that, as @datajoely says, alleviate the burden a bit.

Could you please clarify whether you mean keeping the existing list implementation?

@ElenaKhaustova
Copy link
Contributor Author

From the feedback and discussion here, we're going to:

  1. Keep filter() name
  2. Extend the current filter() implementation by accepting actual types

@ElenaKhaustova ElenaKhaustova marked this pull request as draft February 5, 2025 18:15
Signed-off-by: Elena Khaustova <[email protected]>
@ElenaKhaustova ElenaKhaustova marked this pull request as ready for review February 5, 2025 19:41
Copy link
Member

@merelcht merelcht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

I really like the addition of the by_type filter on top of the regex for type.

RELEASE.md Outdated Show resolved Hide resolved
kedro/io/kedro_data_catalog.py Outdated Show resolved Hide resolved
Signed-off-by: Elena Khaustova <[email protected]>
Copy link
Member

@astrojuanlu astrojuanlu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯

@ElenaKhaustova ElenaKhaustova merged commit 06d5a69 into main Feb 7, 2025
42 checks passed
@ElenaKhaustova ElenaKhaustova deleted the feature/3917-refactor-catalog-filter branch February 7, 2025 15:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants