KEP-4214: Separate super-user kubeconfig for kubeadm

Implementation History
ALPHA Implementable
Created 2023-9-18
Updated 2023-10-19
Latest 0.0
Ownership
Owning SIG
SIG Cluster Lifecycle
Participating SIGs
SIG Cluster Lifecycle
Primary Authors
@neolit123

KEP-4214: Separate super-user kubeconfig for kubeadm

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

  • (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
  • (R) KEP approvers have approved the KEP status as implementable
  • (R) Design details are appropriately documented
  • (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
    • e2e Tests for all Beta API Operations (endpoints)
    • (R) Ensure GA e2e tests meet requirements for Conformance Tests
    • (R) Minimum Two Week Window for GA e2e tests to prove flake free
  • (R) Graduation criteria is in place
  • (R) Production readiness review completed
  • (R) Production readiness review approved
  • “Implementation History” section is up-to-date for milestone
  • User-facing documentation has been created in kubernetes/website , for publication to kubernetes.io
  • Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

During the initial control plane node creation (kubeadm init) an admin.conf file is generated. This file currently contains a cluster-admin credential that is bound to the system:masters group. Create two separate files instead - admin.conf containing a regular Kubernetes cluster-admin credential and a super-admin.conf containing a cluster-admin credential bound to the system:masters group.

Motivation

Binding an admin credential to thesystem:masters group means it can bypass RBAC - i.e. its permissions cannot be removed. At the time of writing this KEP, Kubernetes does not support certificate revocation . This means the only way to revoke access of an admin credential bound to the system:masters group is to rotate the certificate authority of this cluster.

A general purpose admin.conf must be created that does not bind to system:masters. This credential can be shared by kubeadm deployed control plane nodes. In case this admin.conf credential is compromised, its permission must be revocable with RBAC.

A separate “break-glass”, super-user credential can be managed in super-admin.conf. This credential can be used to restore the cluster to a normal state in case of disruptive admin user activity.

Both files must not be shared to new admin users and instead new admin credentials should be signed with the kubeadm kubeconfig user command.

Goals

  • Start managing a separate super-admin.conf that contains a super-user credential.
  • Continue using a file named admin.conf for all its current kubeadm uses today.
  • Improve the kubeadm documentation at the k8s.io website for creating additional admin users.

Non-Goals

  • Do not sign the credentials in the admin.conf and super-admin.conf files to be with expiration time of more than 1 year.

Proposal

To preserve the existing behavior where admin.conf has full cluster access, a new ClusterRoleBinding will be created called kubeadm:cluster-admins. It will bind the ClusterRole cluster-admin to the kubeadm:cluster-admins Group. The credential stored in admin.conf will have the following subject: O = kubeadm:cluster-admins, CN = kubernetes-admin. In case of a compromised credential, the ClusterRoleBinding kubeadm:cluster-admins can be removed or updated. The Group kubeadm:cluster-admins is recommended for internal kubeadm use only.

The new file super-admin.conf will contain the following subject: O = system:masters, CN = kubernetes-super-admin. It will act as the “break-glass”, super-user credential that can bypass RBAC. In case this credential is compromised the cluster certificate authority must be rotated.

Both the admin.conf and super-admin.conf files will be renewable by kubeadm upgrade and kubeadm certs renew. If the super-admin.conf is missing it will not cause an error. That is in case the super-admin has manually moved the file to a safe location - i.e. not keeping it on the primary control plane node, where kubeadm init was called.

The documentation in the Generating kubeconfig files for additional users section will be updated. For each new User or a Group of users the recommendation will be to create a new ClusterRoleBinding. Use the flags --client-name and --org of kubeadm kubeconfig user to control what User or Group this credential belongs to. Revoke access of a single User or a whole Group in case of credential compromise. Using the existing system:masters or kubeadm:cluster-admins Groups will not be recommended.

User Stories (Optional)

Story: Resolving compromised admin credentials

As a super-user administrator I want to be able to use RBAC to remove access of an administrator credential that has been compromised. By removing the kubeadm:cluster-admins ClusterRoleBinding all administrators credentials (admin.conf) on control plane nodes that are signed for the kubeadm:cluster-admins Group will stop working. I can proceed to sign new admin.conf credentials to be used on all nodes by using a new custom Group bound to the cluster-admin ClusterRole.

Alternatively, the certificate authority of this cluster can be rotated, which will allow the kubeadm:cluster-admins ClusterRoleBinding to be restored and used again for new admin.conf credentials signed using the new certificate authority and for the Group kubeadm:cluster-admins.

Story: Keeping the super-user credential in a safe place

As a super-user administrator I want to keep a credential that has super powers and can override RBAC outside of nodes managed by kubeadm. After kubeadm init has finished, I can move the super-admin.conf file to a secure location and only use it in case of an emergency.

Notes/Constraints/Caveats (Optional)

Caveats

Removing the kubeadm:cluster-admins ClusterRoleBinding will drop access to all users in the kubeadm:cluster-admins Group, rendering all admin.conf files on control plane nodes as invalid. However, it will not cause immediate downtime as the admin.conf on control plane nodes is only used when executing kubeadm commands, such as kubeadm upgrade. In such conditions the admin.conf should be populated with credentials for a safe, temporary User that is bound to the cluster-admin ClusterRole.

The super-user admin can then proceed to rotate the cluster certificate authority and eventually restore the kubeadm:cluster-admins ClusterRoleBinding.

Risks and Mitigations

Risk: Implementation complexity during in-place upgrade

During kubeadm upgrade apply the new ClusterRoleBinding kubeadm:cluster-admins must be added. The switch to two separate files must be performed on all control plane nodes. This means that both kubeadm upgrade apply and kubeadm upgrade node must handle the migration of the admin.conf properly. The creation of the new super-admin.conf will be done only on the node where kubeadm upgrade apply is called. On later upgrades, one release after this feature is added, the certificate renewal logic of kubeadm upgrade must be aware that the super-admin.conf file could be missing and should not be rotated.

The mitigation here is detailed unit tests and e2e tests that ensure that the migration for in-place upgrades is handled properly.

Risk: Implementation complexity during re-place upgrade

Users or higher level tools that manage kubeadm re-place upgrades, by removing old control plane nodes and adding new control plane nodes, without calling kubeadm upgrade apply/node must handle this transition manually. The ClusterRoleBinding kubeadm:cluster-admins must be created before the upgrade has started. The kubeadm join process for control plane nodes will create new admin.conf files with certificates that bind to the kubeadm:cluster-admins Group.

Again, tests will be required to ensure that the admin.conf works properly and the ClusterRoleBinding kubeadm:cluster-admins exists.

The super-admin.conf file will not exist at all in such clusters, that were upgraded from older versions of kubeadm. The administrator can sign a super-admin.conf manually by using the kubeadm kubeconfig user --client-name=kubernetes-super-admin --org=system:masters command and store it in a safe location.

For new clusters of this kind, the super-admin.conf will exist on the node where kubeadm init was called. It can be left untouched or manually moved.

Design Details

Test Plan

[x] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

kubeadm will include new unit tests to ensure the new separate admin files are generated properly.

One additional e2e test will be added in the kubernetes/kubeadm repository by using the kinder tool. It can be maintained for one or more releases until more users upgrade to the first release where this feature is available. It can do the following:

  • Creates a 3 control plane node cluster that has the latest kubeadm installed.
  • Calls kubeadm init on one of them.
  • Verifies that kubeconfig files and RBAC are setup properly.
  • Calls kubeadm join on the remaining control plane nodes.
  • Verifies the kubeconfig files on the remaining control plane nodes.
  • Deletes the super-admin.conf file from the first control plane node.
  • Deletes the kubeadm:cluster-admins ClusterRoleBinding.
  • Calls kubeadm upgrade using the same kubeadm version.
  • Ensures that the RBAC and super-admin.conf are recreated.
Prerequisite testing updates

None.

Unit tests

At least the following kubeadm packages will require updates and new unit tests:

  • cmd/kubeadm/app/phases/kubeconfig
  • cmd/kubeadm/app/phases/certs
  • cmd/kubeadm/app/phases/upgrade
Integration tests

NONE

e2e tests

A new e2e test will be added by using the kinder tool.

Graduation Criteria

Once released, the feature will come in effect immediately during upgrade to a particular version or when new cluster creation is done with the kubeadm release when the feature was added. There are no plans for opt-out or opt-in with a feature gate as this is considered a security improvement. The feature will graduate immediately after release.

Upgrade / Downgrade Strategy

On upgrade, the regular kubeconfig and certificate renewal process will be performed. The admin.conf file will be replaced with a file that has the de-escalated privileges. If the super-admin.conf file is present on a node during a future N+1 release, the file will be replaced with updated credentials. If super-admin.conf is not present, no errors will be returned.

The kubeadm upgrade apply command will manage the addition of the new ClusterRoleBinding kubeadm:cluster-admins. One release after the feature was enabled, this logic for the RBAC management can be removed.

kubeadm does not support downgrades.

Version Skew Strategy

The target release of kubeadm will be able to upgrade from N-1 nodes that do not have the feature enabled yet. During cluster creation with the target kubeadm version the feature will become enabled immediately.

The kubeadm upgrade apply command will manage the addition of the new ClusterRoleBinding kubeadm:cluster-admins RBAC. One release after the feature was enabled, this logic for the RBAC management can be removed.

Production Readiness Review Questionnaire

Not applicable for kubeadm. The kubeadm project is considered “out-of-tree”.

Implementation History

  • 18.09.2023: KEP created (1.29).
  • 10.10.2023: Address minor feedback. KEP marked as implementable.
  • 10.19.2023: Adjust test plan and risk / mitigations.

Drawbacks

An estimated drawback is the change in user expectations. The users may expect that the admin.conf file will continue to have the super powers provided by the system:masters Group. This feature will affect this expectation. An action will be required by the same users to sign a new explicit super-admin.conf file with the super powers.

Alternatives

Using a feature gate

A feature gate was considered where users can use it to opt-in into this behavior. The feature could use the standard Alpha-Beta-GA graduation cycle and have the feature gate enabled for the Beta release.

An argument against this behavior is that the feature would not add disruption for the average user. The admin.conf can continue to work as a cluster-admin credential. The feature gate would only add complexity and will have no well established benefits, other than granular feature enablement control.

Another argument is that in practice this is a security improvement and preferably users should not be able to opt-out of similar features.

Integration test vs e2e test

The kinder e2e testing tool is quite flexible and the nodes do include tools such as openssl for certificate inspection and base64 for decoding base64 strings. However, writing such an e2e test must be done in bash or hardcoded in kinder as Go code.

Instead the option to use an Go integration test included in cmd/kubeadm/test seems preferable. It will allow using the Go standard library and existing kubeadm utils for parsing kubeconfig files and x509 certificates.

One downside is that the same integration test will be executed on every change in the kubeadm tree under kubernetes/kubernetes instead of being less frequent - i.e. periodic.

Signing individual kubeconfig files for control plane nodes

Today, control plane nodes use the same admin.conf that is shared via a kube-system/kubeadm-certs Secret and encrypted with a RSA key. This behavior is expected from kubeadm since it treats control plane nodes as setup replicas (more or less).

During joining of control plane nodes, this download of the admin.conf can be skipped. Instead the ca.key and ca.cert pair (already shared) can be used to create a new admin.conf unique for this control-plane node.

For example:

  • Control plane node foo wishes to join the cluster.
  • The ca.key and ca.cert are downloaded from the Secret.
  • The node name is ensured, likely by expecting the kubelet client certificate.
  • A new admin.conf is created that has a subject: O = kubeadm:cluster-admins, CN = kubernetes-admin-foo.
  • This User kubernetes-admin-foo is bound to the cluster-admin ClusterRole with an additional RBAC rule.

Similarly, during kubeadm init the admin.conf must contain the CN with the node name. kubeadm upgrade for this node must ensure to properly maintain the new RBAC rule and CN in the admin.conf.

This would allow to revoke access of individual control plane nodes’ admin.conf users. Since it adds complexity to the current KEP, it could be done in a separate KEP as additional hardening.

However, it also opens some questions about node security. With disk structure in mind, if the admin.conf of a control plane node has leaked, that may also mean the ca.key has leaked which means the entire cluster is compromised.

Infrastructure Needed (Optional)

None.