Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

explicitly set data security mode to SINGLE_USER #179

Merged

Conversation

shichengzhou-db
Copy link
Contributor

@shichengzhou-db shichengzhou-db commented Nov 5, 2024

Title.

  • Set to SINGLE_USER so that mlops stack cluster creation is unaffected by default values of data_security_mode.
  • Fix broken tests by upgrading github actions versions

Manual tested all workflows on all clouds (aws/azure/gcp), no failures related to data_security_mode. Although the workflows seems to be broken for various issues that need to be looked into separately.

Copy link
Collaborator

@arpitjasa-db arpitjasa-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @shichengzhou-db ! Would you mind also fixing the tests that are failing?

@@ -3,6 +3,7 @@ new_cluster: &new_cluster
num_workers: 3
spark_version: 15.3.x-cpu-ml-scala2.12
node_type_id: {{template `cloud_specific_node_type_id` .}}
data_security_mode: "SINGLE_USER"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we confirm these workflows still run on AWS, Azure, and GCP?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

confirmed this works on dogfood(aws), will test the other in a bit

@shichengzhou-db shichengzhou-db force-pushed the cluster_data_security_mode branch 2 times, most recently from 3badd2a to 9b6faa2 Compare November 7, 2024 05:44
@shichengzhou-db
Copy link
Contributor Author

shichengzhou-db commented Nov 7, 2024

Thanks @shichengzhou-db ! Would you mind also fixing the tests that are failing?

@arpitjasa-db
Fixed broken tests, and tested the feature engineering workflow on all clouds, the gcp one seems to have been failing before this, filed a separate es ticket for that.

(Will test the other workflows on all clouds as well tmmrw and document in the PR description)

Copy link
Collaborator

@arpitjasa-db arpitjasa-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @shichengzhou-db !

@arpitjasa-db arpitjasa-db merged commit dd67dff into databricks:main Nov 8, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants