azurerm_databricks_workspace: managed service/disk support managed hsm key

[wuxu92]

Community Note

• Please vote on this PR by adding a 👍 reaction to the original PR to help the community and maintainers prioritize for review
• Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for PR followers and do not help prioritize for review

Description

DRAFT FOR INTERNAL REVIEW

This PR adds Managed HSM Key support for both managed services and managed disks.

It also deprecates the optional key vault ID property for both, as it is unnecessary. The Databricks server can validate the key's availability, so managed_services_cmk_key_vault_id and managed_disk_cmk_key_vault_id have been removed from the documentation. A deprecated description has been added to them, but they are not being removed from the schema to avoid introducing a breaking change.

PR Checklist

• I have followed the guidelines in our Contributing Documentation.
• I have checked to ensure there aren't other open Pull Requests for the same update/change.
• I have checked if my changes close any open issues. If so please include appropriate closing keywords below.
• I have updated/added Documentation as required written in a helpful and kind way to assist users that may be unfamiliar with the resource / data source.
• I have used a meaningful PR title to help maintainers and other users understand this change and help prevent duplicate work.
  For example: “resource_name_here - description of change e.g. adding property new_property_name_here”

Changes to existing Resource / Data Source

• I have added an explanation of what my changes do and why I'd like you to include them (This may be covered by linking to an issue above, but may benefit from additional explanation).
• I have written new tests for my resource or datasource changes & updated any relevent documentation.
• I have successfully run tests with my changes locally. If not, please provide details on testing challenges that prevented you running the tests.
• (For changes that include a state migration only). I have manually tested the migration path between relevant versions of the provider.

Testing

• My submission includes Test coverage as described in the Contribution Guide and the tests pass. (if this is not possible for any reason, please include details of why you did or could not add test coverage)

Change Log

Below please provide what should go into the changelog (if anything) conforming to the Changelog Format documented here.

• azurerm_databricks_workspace - support managed hsm key as managed service and managed disk encription [azurerm_databricks_workspace - missing support for keys from a Managed HSM Key Vault #27738]

This is a (please select all that apply):

• Bug Fix
• New Feature (ie adding a service, resource, or data source)
• Enhancement
• Breaking Change

Related Issue(s)

Fixes #27738

Note

If this PR changes meaningfully during the course of review please update the title and description as required.

[WodansSon]

Hi @wuxu92, I have given this PR and it mostly looks good, however I have left a few minor comments around the schema and depreciation text. If you can get those fixed up, I think this will be good to move out of draft state. 🚀

[WodansSon]

Shouldn't this conflict with managed_services_cmk_key_vault_key_id?

Suggested change

	"managed_services_cmk_key_vault_id": {
	Type: pluginsdk.TypeString,
	Optional: true,
	ValidateFunc: commonids.ValidateKeyVaultID,
	Deprecated: "Use 'managed_services_cmk_key_vault_key_id' instead, this field is not needed for cross subscription key vaults anymore",
	"managed_services_cmk_key_vault_id": {
	Type: pluginsdk.TypeString,
	Optional: true,
	ValidateFunc: commonids.ValidateKeyVaultID,
	Deprecated: "'managed_services_cmk_key_vault_id' has been deprecated in favor of 'managed_services_cmk_key_vault_key_id' and will be removed in v5.0 of the provider",
	ConflictsWith: []string{"managed_services_cmk_key_vault_key_id"},

[wuxu92]

We cannot add ConflictsWith here because some users may have set both managed_services_cmk_key_vault_id and managed_services_cmk_key_vault_key_id. Do you mean the new managed_disk_cmk_managed_hsm_key_id property should conflict with managed_services_cmk_key_vault_id? That would be reasonable. I updated it.

[WodansSon]

Suggested change

	"managed_disk_cmk_key_vault_id": {
	Type: pluginsdk.TypeString,
	Optional: true,
	ValidateFunc: commonids.ValidateKeyVaultID,
	Deprecated: "Use 'managed_disk_cmk_key_vault_key_id' instead, this field is not needed for cross subscription key vaults anymore",
	"managed_disk_cmk_key_vault_id": {
	Type: pluginsdk.TypeString,
	Optional: true,
	ValidateFunc: commonids.ValidateKeyVaultID,
	Deprecated: "'managed_disk_cmk_key_vault_id' has been deprecated in favor of 'managed_disk_cmk_key_vault_key_id' and will be removed in v5.0 of the provider",
	ConflictsWith: []string{"managed_disk_cmk_key_vault_key_id"},

[WodansSon]

Shouldn't this also conflict with managed_disk_cmk_key_vault_id?

Suggested change

	ConflictsWith: []string{"managed_disk_cmk_managed_hsm_key_id"},
	ConflictsWith: []string{"managed_disk_cmk_managed_hsm_key_id", "managed_disk_cmk_key_vault_id"},

[wuxu92]

Like above, we cannot have it conflict with managed_disk_cmk_key_vault_id here.

[WodansSon]

@wuxu92, thanks for pushing those changes so quickly. This LGTM now! 🚀

[stephybun]

Hey @wuxu92, could you please follow the guidance in our breaking changes guide when deprecating properties? Once those changes have been made can you please also provide testing evidence in both 4.x and 5.0 mode. Thanks!

[wuxu92]

Hi @stephybun, since those two properties are not needed for the business, they are just read from the config and set back to the state even in the original logic. Therefore, I merely marked them as deprecated. If you think it's still necessary, I can add the required work for a breaking change.

[stephybun]

Yes, please still follow the guidance that I linked. This will ensure that the property is actually removed in 5.0 and we don't need to remember to remove it from the schema in the week of the major release. Make sure to update any affected tests as well as the upgrade guide too.