Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model validation for FS models #165

Merged
merged 12 commits into from
Jul 25, 2024
Merged

Conversation

aliazzzdat
Copy link
Contributor

FS models cannot predict as pyfunc models yet (issue #70), thus we can't use MLflow Evaluate with FS models.

However, MLflow Evaluate can take a function instead of a pyfunc model_uri.
This means we can call an FS model in the function and use this function in the Evaluate method.

Although this is a good solution, we still cannot enable baseline comparison, as we need to provide a model_uri of a pyfunc model (so this means it's not possible to compare a new FS model with a baseline FS model).

@arpitjasa-db
Copy link
Collaborator

@aliazzzdat thanks for all the changes! Looks like some tests are failing, can we fix those and also do a full instantiation of Stacks to confirm everything is still working as expected? Version changes can be tricky, and want to make sure we don't break anything

@aliazzzdat
Copy link
Contributor Author

Featurization pipeline works on DBR 15.3 and with MLflow 2.11
Screenshot 2024-07-19 at 21 43 00

Model training pipeline works on DBR 15.3 and with MLflow 2.11
Screenshot 2024-07-19 at 21 43 29

Validation works with a Feature Store model
Screenshot 2024-07-19 at 21 43 57
Screenshot 2024-07-19 at 21 44 16
Screenshot 2024-07-19 at 21 44 30

@aliazzzdat
Copy link
Contributor Author

The batch inference works as expected once we create the feature_store_inference_input table
Screenshot 2024-07-19 at 21 54 34

.gitignore Outdated Show resolved Hide resolved
tests/example-project-configs/aws/aws-github-fs.json Outdated Show resolved Hide resolved
Copy link
Collaborator

@arpitjasa-db arpitjasa-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for all the changes @aliazzzdat !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants