Document requirements/recommended process for updating cluster TLS certs/keys

[jimmycuadra]

If you're running Kubernetes with the master components secured with TLS, eventually you will need to update the certificate and key, and possibly even the CA cert. Right now there is no documentation about how this should be approached. What services need to be restarted when the CA cert, endpoint cert, or private key are changed on disk? If all the master components are running via the kubelet's static manifest directory, is it sufficient to just restart kubelet on the host? Or is it necessary to somehow manually restart each containerized master component that reads those files?

[pwittrock]

@kelseyhightower Do you know how folks normally figure this out?

[jimmycuadra]

I just had to go through this process more or less manually, and learned that restarting the kubelet does not restart any k8s master components that were launched by static manifests the kubelet is observing. In order to get kube-apiserver to restart and pickup the new TLS credentials, I had to move the kube-apiserver manifest out of the directory kubelet was watching, restart kubelet, then move the manifest back in and restart the kubelet again.

This definitely needs to be documented. Hopefully there is a better way of telling the kubelet to restart the master components, too. If not, there really should be.

[jimmycuadra]

Would a maintainer please bring a Kubernetes developer who can answer this into the conversation? Not having a way to restart the kube-system components to pick up TLS credential changes is a blocking issue for my team to roll out Kubernetes in production. Thanks!

[jimmycuadra]

Comment from chancez on Slack which helps with a workaround for the time being:

ive only tried once/twice to test a solution for an issue a user was having, but i basically did what you did, but I did docker kill $container instead of messing with the manifest files

[eugene-chow]

I just had to re-make my cert and deploy it to all the nodes. These are my notes:

1. Create the new certs and deploy it to all the nodes
2. Restart each and every node one-by-one. Alternatively, restart every kubernetes service if you don't want to bring down the node completely.
3. Delete the default service account token, which is linked to the old TLS certs, in every namespace (eg. default-token-b7scp). A new default token will be automatically created.
4. Restart every pod that uses the default token, so that it reads the new token.

[liggitt]

steps 3 and 4 are not required if you keep the old key as an additional valid public key (can pass multiple --service-account-key-file args to the apiserver), or use a dedicated service account token signing key separate from the apiserver's private tls key

[eugene-chow]

Thanks for the heads up. I forgot to mention that my setup isn't a production system but Kelsey's https://github.com/kelseyhightower/kubernetes-the-hard-way. The cluster in this tutorial uses one cert to rule them all.

[xiangpengzhao]

/sig docs

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[ankon]

Clearly this is a still-existing problem, and it is really something that needs to be addressed minimally in documentation, ideally even in code.

[emilhdiaz]

This is major blocker for us to run Kubernetes into production. Any advice would be much appreciated!

I already tried to simulate a certificate rotation in a test environment and couldn't do so without causing downtime for running applications.

/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[ankon]

/remove-lifecycle rotten

[k8s-ci-robot]

@trunet: you can't re-open an issue/PR unless you authored it or you are assigned to it.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sftim]

/reopen

[k8s-ci-robot]

@sftim: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@jimmycuadra: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sftim]

/transfer website

[sftim]

/language en
/remove-area security
/sig security

[k8s-ci-robot]

@sftim: Those labels are not set on the issue: area/security

In response to this:

/language en
/remove-area security
/sig security

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sftim]

/sig auth

[sftim]

/kind feature

[ritazh]

/assign @aramase

[divya-mohan0209]

Hello @aramase : Please may we have an update on whether this is being progressed at the moment and if you have any updates?

[mehabhalodiya]

@aramase I don't see any updates; so unassigning you. Please feel free to assign, if you come back here again and are willing to work on it. Thank you! 🙂
/unassign @aramase

[tomkivlin]

This appears to have been completed in 92b56db and could be closed?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[tomkivlin]

Ah my bad - I am still working on this, but slowly. Will aim to get a PR ready by end of this week.

/remove-lifecycle stale

[sftim]

/lifecycle frozen

[sftim]

Slightly relevant to #39694

[sftim]

@tomkivlin, you made a start on this. How did that go?

[tomkivlin]

@sftim I did, and then forgot about it, sorry. I have blocked out some time in the second half of August to get this done. Apologies. I think the issue I created a branch for was #14725 so I'll reassign that to me as well.

[tomkivlin]

FYI my work so far (that I'd pushed) is here: https://github.com/tomkivlin/website/blob/tomkivlin/issue14725/content/en/docs/concepts/security/kubernetes-encryption.md

[tomkivlin]

/assign

[sftim]

Duplicated by (part of) #42258

[sftim]

Anyone who'd like to help with this issue is very welcome to work on it.

[sftim]

/priority important-longterm

[sftim]

Help is (still) welcome