Encryption of User Password in K8S Secret

[kyzrfranz]

Not quite sure if this is entirely within scope of this project but here goes:

Since the general procedure of creating users involves creating a Secret containing the user's password, I'd like to kick off a discussion about ways of making this more secure. Of course there are k8s native solutions like encryption at rest, but they are very global and might not be feasible for all users.

So in that light, would it be off limits to discuss how encryption of credentials might be implemented as an optional feature in strimzi?

[scholzj]

I don't think it is off-limits... but there are some general issues you would need to deal with:

• Encrypting the passwords is useless because you need to store the password to encrypt/decrypt it right next to it
• While there might be tools offering better security than Kubernetes Secrets, there is a lack of standardization in the open source landscape, especially after Vault is not open source anymore.

So you would need to deal with those.

Two other things to keep in mind:

• You can always disable the User Operator and manage users in any way you want (or even use it only to manage ACLs and quotas and manage the user passwords on your own)
• If you have an environment that is strict about security, you should consider using OAuth to manage your users somewhere else completely (although that has many challenges on its own)

[kyzrfranz]

Thanks for your comments.

I’d like to provide a bit more context regarding my proposal. My aim isn’t to achieve perfect security for user passwords—since we all know that if a cluster is compromised, most workloads are at risk—but rather to meet regulatory requirements that discourage storing plaintext credentials.

I realize that encrypting a secret only shifts the problem (because the decryption key must also be stored somewhere accessible). However, I’m considering an approach using asymmetric encryption. In this model, the operator would use a public key (managed via a KMS) to encrypt user passwords before writing them into Kubernetes Secrets. When needed, the operator would use the corresponding private key to decrypt the credentials for updating Kafka. This way, even if an administrator or an accidental log exposes the secret, the sensitive data remains encrypted at rest.

I understand that one alternative is to bypass the User Operator altogether and manage users directly via the Kafka API. However, I’m skeptical about the increased maintenance overhead of that approach—after all, one of Strimzi’s main appeals is its ease of use in automating user management.

Does that make sense?

[scholzj]

I understand that one alternative is to bypass the User Operator altogether and manage users directly via the Kafka API. However, I’m skeptical about the increased maintenance overhead of that approach—after all, one of Strimzi’s main appeals is its ease of use in automating user management.

Please keep in mind that this is a very tricky argument. Logic like this will cause a lot of increased effort and maintenance on Strimzi side - surely higher effort than one user would have to do this manually. More features and a bigger codebase mean also higher risk of bugs, CVEs, etc. which are inconvenient not just for maintainers but also for users. So, this is a valid argument only if you can reasonably enough prove that this feature is needed / will be used by a large part of Strimzi users.

From my experience, most of the regulated users:

• Want to use central user management solutions with full auditing, etc. => therefore the authentication mechanisms such as OAuth are more interesting for them than SCRAM-SHA-512
• Prefer to use their own central secret store (for example Vault)

So I do not think the assumption that every regulated user wants this is somehow automatic.

I realize that encrypting a secret only shifts the problem (because the decryption key must also be stored somewhere accessible). However, I’m considering an approach using asymmetric encryption. In this model, the operator would use a public key (managed via a KMS) to encrypt user passwords before writing them into Kubernetes Secrets. When needed, the operator would use the corresponding private key to decrypt the credentials for updating Kafka. This way, even if an administrator or an accidental log exposes the secret, the sensitive data remains encrypted at rest.

I think before you get into the technical details of how to do the encryption, you would need to explain how an encrypted password with the encryption key setting right next to it is in any way more secure than the unencrypted password. The User Operator needs to know the password because Kafka does not provide any metadata about it. So it will need to have the decryption keys and the would be stored either in the same Secret or in a Secret next to it. This would be different if we did not need the password and could keep only the encryption key. But this would need improvements in Kafka.

Second, the encryption is tricky because you will have algorithms, key sizes, their rotation, etc. And every regulated entity will have its own rules for it as they have for anything security-related.

So I think the most realistic way is to have some pluggable implementation where Strimzi ships the current Secret-based storage implementation, and users can use their own custom (possibly shared between them) implementation, for example, using what you suggest, or Vault, or whatever they prefer. That way, you minimize the effort pushed onto the project as it will only need to maintain the pluggable interface and the current implementation. And you can also show how popular your implementation (based on a number of downloads, GitHub stars, or whatever) is to argue about including it by default. But even if that does not happen, you can just maintain it as a plugin yourself. (Honestly, this is one of the things I would have done differently if I started from scratch again - make things more pluggable out of the box)

[kyzrfranz]

Gotcha, I guess this makes a lot of sense from your end and I really like the idea of that plug-in concept.
I am going to take this into consideration because it seems like a pragmatic and flexible way of handling not only this issue.
thanks a lot for your input!

[scholzj]

Please keep in mind that this would need a proposal (https://github.com/strimzi/proposals) to discuss the details and clarify what exactly the interface would look like. But do not think there should be any major issue to get it approved as this approach would:

• Avoid the discussion about the exact details of how the encryption would be done and who prefers what method etc.
• Pluggability is IMHO a generally desired mechanism and it would be hard to argue against it by anyone

[kyzrfranz]

Are there any concerns wrt making plugins language independent? Thinking about grpc for instance..

[scholzj]

Well, the plugin interface has to be a Java interface and the core of the implementation has to be in Java as that is what Strimzi is written in. But whether the Java part calls a remote system through GRPC or invokes some JNI or WASM plugin, that is up to the implementation of course.