KEP-4471: Make a control-plane's kubelet point to the local API Server on kubeadm join

Implementation History
STABLE Implementable
Created 2024-2-1
Updated 2025-09-17
Latest 0.0
Ownership
Owning SIG
SIG Cluster Lifecycle
Participating SIGs
SIG Cluster Lifecycle
Primary Authors
@chrischdi

KEP-4471: Make a control-plane’s kubelet point to the local API Server on kubeadm join

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

  • (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
  • (R) KEP approvers have approved the KEP status as implementable
  • (R) Design details are appropriately documented
  • (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
    • e2e Tests for all Beta API Operations (endpoints)
    • (R) Ensure GA e2e tests meet requirements for Conformance Tests
    • (R) Minimum Two Week Window for GA e2e tests to prove flake free
  • (R) Graduation criteria is in place
  • (R) Production readiness review completed
  • (R) Production readiness review approved
  • “Implementation History” section is up-to-date for milestone
  • User-facing documentation has been created in kubernetes/website , for publication to kubernetes.io
  • Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

This KEP proposes to enhance kubeadm to make kubelets on control plane nodes point to the local kube-apiserver. Currently the kubelet always points to the load balanced control plane endpoint which could result in violations of version skew policy.

Motivation

When a control plane node joins a cluster, the kubelet will bootstrap itself against the load balanced kube-apiserver endpoint.

This kube-apiserver endpoint may forward requests coming from the local kubelet to the kube-apiserver running on any of the existing control plane nodes, including the one running on the same node as the kubelet.

When doing a immutable rolling upgrade (as e.g. done when using Cluster API ) the kubelet’s minor Kubernetes version could be newer than the version of kube-apiserver running on the previously existing control plane nodes.

In that case this would lead to a violation of the version skew policy rule:

kubelet must not be newer than kube-apiserver.

Because of that is not guaranteed to work and this .

Goals

  • Add a new code path in kubeadm that can be used to join control plane nodes without potentially violating the version skew policy, by letting the kubelet only communicate with the local kube-apiserver.
  • Also adjust init and upgrade to result in the same configuration.
  • Use a new feature gate ControlPlaneKubeletLocalMode to toggle the feature until graduating to GA.

Non-Goals

  • Support the “old way” and “new way” indefinitely. Once the proposed feature gate graduates to GA it will hardcoded to be active.
  • Touch areas of kubeadm different than kubeadm join, kubeadm init and kubeadm upgrade.

Proposal

The proposal is to implement the required changes to make the kubelet point to the local available kube-apiserver. This change relates to initializing, joining and upgrading control plane nodes and does not affect worker nodes.

The overall change is:

  • for kubeadm join to adjust the file /etc/kubernetes/bootstrap-kubelet.conf to point to the local kube-apiserver, which gets created by kubeadm during the KubeletStartJoinPhase (xref ). This will also affect the kubelet’s kubeconfig.
  • for kubeadm init to adjust the created kubeconfig to point to the local kube-apiserver, which gets created by kubeadm during the kubeconfig phase (xref )
  • for kubeadm upgrade to edit the kubelet config file to point to the local kube-apiserver.

To make this work for kubeadm join, an additional change is required: etcd needs to get started and joined to the etcd cluster before waiting for the kubelet to finish its bootstrap process, instead of the other way around. This requires reordering some of the operations done in different kubeadm phases by extracting the relevant parts into separate phases and changing their order.

Because reordering the phases can be considered a breaking change to the CLI of kubeadm for some users, this should get done behind a feature gate, while preserving the previous behavior when the feature gate is disabled.

User Stories (Optional)

Story 1

As a kubeadm user, I wish no component violates the Kubernetes version skew policy when joining a control plane node.

Story 2

As a kubeadm user, I wish the kubelet of a joining control plane node points to the local kube-apiserver.

Story 3

As a kubeadm user, I wish the CLI of kubeadm to be stable and breaking changes to it to be announced ahead of time.

Story 4

As a kubeadm user, I wish the kubelet of an initializing control plane node points to the local kube-apiserver.

Notes/Constraints/Caveats (Optional)

  • This change is only relevant for the control plane nodes because worker nodes do not have a local kube-apiserver to point to and are not affected by this special case of violating the version skew policy violation. Worker nodes will still have to follow the documented version skew policy.

Caveats

The change needs to get implemented in a way that there is no change for existing users, when the feature gate is not enabled.

Risks and Mitigations

Risk: Duration for joining a control-plane node may change

Joining a node is an asynchronous task which consists of multiple steps where kubeadm also relies on the kubelet to bootstrap its own kubeconfig or start static pods. It is possible to hit timeouts when waiting for the kubelet to succeed. This proposal also changes the order of operations done by the kubelet and may change the time required for single steps where kubeadm waits for the kubelet. At the end the existing time boundaries required for joining a control plane node using kubeadm should not change.

Risk: control-plane node stability

Having the kubelet of a control-plane node pointing directly to the local kube-apiserver only may cause some stability issues. E.g. when the local kube-apiserver is not functional, then the local kubelet won’t be able to report its status back to the kube-apiserver and will become NotReady.

To mitigate this, a user could adjust the kubeconfig for the kubelet to point to the load balanced API Server after all components got upgraded.

As alternative, kubeadm’s configuration in v1beta4 could get an option which makes kubeadm change the endpoint back to the load balanced endpoint after the kubelet bootstrapped itself. However this could still lead to violations to the skew policy.

Design Details

As explained above two minor changes are required to implement the required changes.

1. Make the kubelet point to the local kube-apiserver

For kubeadm init to make the kubelet point to the local apiserver, the kubeconfig which get’s written for the kubelet needs to get adjusted when the kubelet’s kubeconfig file gets written (xref ).

For kubeadm join to make the kubelet point to the local apiserver, the file for kubelet’s bootstrap kubeconfig needs to get adjusted, which gets created by kubeadm during the KubeletStartJoinPhase (xref ).

This creates the following chicken-egg issue:

  • After starting the kubelet, kubeadm waits in the KubeletStartPhase for it to bootstrap itself, before it would continue to start etcd in the ControlPlaneJoinPhase.
  • Due to the above change, the kubelet will never be able to bootstrap itself while pointing at the local kube-apiserver, because the local kube-apiserver will never get ready unless etcd is available locally.

The second change address this issue.

2. Introducing and reordering phases for kubeadm join

To fix the above chicken-egg issue some operations of kubeadm need to get rearranged during kubeadm join.

To be more precise: parts of the KubeletStartPhase need to run after the EtcdLocalSubPhase (which is part of the ControlPlaneJoinPhase).

The current phases of kubeadm join are the following:

  1. PreflightPhase
  2. ControlPlanePreparePhase
  3. CheckEtcdPhase
  4. KubeletStartPhase
  5. ControlPlaneJoinPhase

First the existing KubeletStartPhase gets split up and the code which waits for the kubelet’s bootstrap to complete into a separate phase named KubeletWaitBootstrapPhase gets extracted.

  1. PreflightPhase
  2. ControlPlanePreparePhase
  3. CheckEtcdPhase
  4. KubeletStartPhase
  5. KubeletWaitBootstrapPhase
  6. ControlPlaneJoinPhase

Second the EtcdLocalSubphase which is the first part of the ControlPlanJoinPhase gets moved to a new phase ControlPlaneJoinEtcdPhase and added between the KubeletStartPhase and KubeletWaitBootstrapPhase.

  1. PreflightPhase
  2. ControlPlanePreparePhase
  3. CheckEtcdPhase
  4. KubeletStartPhase
  5. ControlPlaneJoinEtcdPhase
  6. KubeletWaitBootstrapPhase
  7. ControlPlaneJoinPhase

Actions to preserve the old behavior when the feature gate is disabled:

  • The new phases ControlPlaneJoinEtcdPhase and KubeletWaitBootstrapPhase should define a RunIf function to ensure they do not run when the feature gate is disabled.
  • The phases KubeletStartPhase and ControlPlaneJoinPhase should behave as before if the feature gate is disabled.
  • There should be no duplication of code, instead the Run functions of the new phases should get called from the old location if the feature gate is disabled.
  • Nothing should be done during the new phases ControlPlaneJoinEtcdPhase and KubeletWaitBootstrapPhase.
  • Add a description to the new phases which explain their functionality and explicitly mark them as EXPERIMENTAL.

Test Plan

[x] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

Prerequisite testing updates

None.

Unit tests

New unit tests must be added for all code paths that use the ControlPlaneKubeletLocalMode feature gate if applicable.

Integration tests

NONE

e2e tests

A new e2e test will be added in the kubernetes/kubeadm repository by using the kinder tool. It can be maintained for one or more releases until the feature gate graduates to beta and will be enabled by default (and because of that be tested already during other test cases). It can do the following:

  • Create a 3 control plane node cluster
  • Call kubeadm init on one of them, having the feature gate ControlPlaneKubeletLocalMode enabled.
  • Check that the kubelet is pointing to the local apiserver.
  • Call kubeadm join on the remaining control plane nodes.
  • Check that the kubelet’s are pointing to the local apiserver.
  • Adjust the kubelet’s kubeconfig’s to point to the load balanced endpoint.
  • Call kubeadm upgrade on the nodes.
  • Check that all kubelet’s are again pointing to the local apiserver.

Graduation Criteria

Alpha

  • Feature implemented behind a feature gate ControlPlaneKubeletLocalMode.
  • Initial unit and e2e tests completed and enabled.
  • Document the feature gate .
  • Document the upcoming breaking change at the release notes.

Beta

  • Make feature gate to be enabled by default.
  • Gather feedback from developers and surveys.
  • Make unit and e2e test changes.
  • Update the feature gate documentation.
  • Document the new phases .

GA

  • Gather feedback from developers and surveys.
  • Update unit tests.
  • Remove e2e tests as this will be tested by all existing kubeadm e2e tests.
  • Update the feature gate documentation.
  • Update the phases documentation.

Upgrade / Downgrade Strategy

  • N/A -> Alpha: users can patch their ClusterConfiguration in the kube-system/kubeadm-config ConfigMap to enable the ControlPlaneKubeletLocalMode feature gate, before calling kubeadm upgrade apply. This will allow them to join control plane nodes with this feature enabled. This scenario is anticipated as rare, because usually users maintain a stable control plane with 3 or more members before upgrading it. But it is still plausible and can be documented in the feature gate documentation. The kubelet.conf on existing nodes can also be edited to match the new behavior.
  • Alpha -> Beta: similarly to the previous stage users can modify the ClusterConfiguration to disable the feature gate during upgrade. This will allow them to use the “old way”, in case they wish to join more control plane nodes to the cluster while the feature gate is enabled by default.
  • Beta -> GA: users could no longer patch the ClusterConfiguration to opt-out of the feature and it will be locked to be enabled by default.

Version Skew Strategy

Not applicable.

Production Readiness Review Questionnaire

kubeadm is considered an “out of tree” component and PRR is out of scope.

Implementation History

  • 01.02.2024: KEP issue created.
  • 08.02.2024: KEP draft created.
  • 07.05.2024: KEP marked as implementable for 1.31 or later.
  • 12.07.2024: KEP adjusted to match discussed implementation.
  • 15.07.2024: KEP alpha implementation merged.
  • 06.02.2024: PRs to promote the feature gate to Beta.
  • 17.09.2025: PRs to promote the feature gate to GA.

Drawbacks

Moving or introducing new phases are breaking changes to the CLI of kubeadm. Adding new phases may break users which are executing the single phases manually. The changes must be well-documented in release notes for users to adapt the changes.

The reordering of phases may change delays or timeouts when control plane nodes are joined. Modifying this code path may introduce potential for user complains about HA cluster creation and maintenance with kubeadm. Sufficient testing and gathering feedback from users would be mandatory.

Alternatives

Take the risk of violating Kubernetes version skew policy

By not introducing the change, users of kubeadm are on risk to hit a case in the future where a violation of the Kubernetes version skew policy happens and joining a control plane node may fail. We anticipate such cases to be rare.

Use external etcd

When the external etcd mode is used nothing changes because the executed code would stay the same. Just the skipped parts (for self-hosted etcd) will be skipped at other places.

Infrastructure Needed (Optional)

None.