KEP-3614: Use etcd's learner mode in kubeadm

Implementation History
BETA Provisional
Created 2022-10-12
Updated 2023-09-04
Latest 0.0
Ownership
Owning SIG
SIG Cluster Lifecycle
Participating SIGs
SIG Cluster Lifecycle
Primary Authors
@neolit123

KEP-3614: Use etcd’s learner mode in kubeadm

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

  • (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
  • (R) KEP approvers have approved the KEP status as implementable
  • (R) Design details are appropriately documented
  • (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
    • e2e Tests for all Beta API Operations (endpoints)
    • (R) Ensure GA e2e tests meet requirements for Conformance Tests
    • (R) Minimum Two Week Window for GA e2e tests to prove flake free
  • (R) Graduation criteria is in place
  • (R) Production readiness review completed
  • (R) Production readiness review approved
  • “Implementation History” section is up-to-date for milestone
  • User-facing documentation has been created in kubernetes/website , for publication to kubernetes.io
  • Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

This KEP proposes to enhance kubeadm to start using etcd’s learner mode which was introduced in version 3.4. The release notes for etcd 3.4 suggest a number of benefits of using this method. The proposal aims to add the new mode as a standard kubeadm / Kubernetes feature gate that is graduated over the period of one year or more, while collecting feedback from all kubeadm users.

Motivation

Kubeadm currently adds all members in the “old way” that etcd supported, that is to add them as voting members from the beginning. If added as learners instead, such members would not disrupt the cluster quorum if they end up being faulty. The “old way” has proven problematic in cases where kubeadm attempts to add a etcd cluster member from a control plane node running on slower infrastructure. In such cases users have to manually interfere and remove the faulty member, by using tools such as etcdctl.

Goals

  • Add a new code path in kubeadm that can be used to deploy etcd with learner mode enabled.
  • Use a new feature gate EtcdLearnerMode that can be used to toggle the feature until graduation to GA.
  • Deprecate and remove the “old way” of adding members.
  • Keep the number of learners as per the etcd default value (1). Users can customize the value if they wish.

Non-Goals

  • Support both the “old way” and “learner mode” in kubeadm as a toggle in the kubeadm API. Ideally we should support only a single, stable, community approved code path.
  • Attempt to improve support for parallel join of control plane nodes. It is currently supported tentatively in the “old way”.
  • Attempt to improve resilience of control plane node join in case of machine misconfiguration, such as NTP offsets.

Proposal

User Stories (Optional)

Story 1

As a kubeadm user, I wish that my HA cluster is more resilient to etcd member failures during addition of new members at cluster bring up time due to slow infrastructure.

Story 2

As a kubeadm user, I wish that my HA cluster is constructed following the recommendation by etcd maintainers and using the latest features - i.e. to use learner mode instead of adding all new members as voting.

Notes/Constraints/Caveats (Optional)

Risks and Mitigations

Risk: insufficient testing by kubeadm users

Once the new code path is added and the logic is controlled by a feature gate, the feature gate will be in Alpha state or disabled by default. Even if e2e tests are added we need to notify users that we are making this important change to etcd and that they start testing it ASAP during Alpha, but not in production.

Mitigation

Notify users on all possible communication channels: Slack, ML, Reddit, Twitter, etc. Keep umbrella issue as a place for discussion and user feedback. Attempt gathering feedback from parties that build product on top of kubeadm.

Risk: unstable implementation of learner mode

Once the new feature is added we need to test the stability of the new code path. The current “old way” of constructing the etcd cluster has proven stable and is used by all kubeadm HA users of the “stacked etcd” topology. It has also proven to allow concurrent join of control plane nodes with their stacked etcd members. With the addition of learner mode we are introducing the potential that once the feature graduates to Beta it would be enabled by default and might cause unforeseen issues.

Mitigation

Once again the mitigation here would be to notify all possible channels and ask consumers of kubeadm to test the feature before it moves to Beta. Testing on slow infrastructure might be a key point to mitigate possible issues.

Design Details

Currently most of the logic of stacked etcd member support in kubeadm is centralized around a couple of files in the source code. These files contain the etcd client wrapped logic and the logic for maintaining a static pod manifest for the etcd server instance.

With the introduction of the new feature gate EtcdLearnerMode a new code path must be created. Preferably the number of “if EtcdLearnerMode” branches in the code should be minimized. Kubeadm currently has some sensitive timeouts while adding etcd members the “old way”. Waiting for learners to become voting members would require some modifications in kubeadm in terms of how we wait for a member to be added. Some details can be found in the official etcd documentation .

Test Plan

[x] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

Prerequisite testing updates
Unit tests

New unit tests must be added for all code paths that use the EtcdLearnerMode feature gate. Once the feature graduates to GA, these unit tests must be merged as part of the default unit tests for testing the kubeadm “stacked etcd” logic.

Integration tests

N/A

e2e tests

kubeadm-kinder-learner-mode-latest was added as part of the kubeadm dashboard since v1.27. All tests in this dashboard use the kinder tool.

  • During Alpha (disabled by default): add a new e2e test that enables the feature gate EtcdLearnerMode
  • During Beta (enabled by default): modify the e2e test to test the feature gate EtcdLearnerMode as disabled
  • During GA (locked to enabled): remove the e2e test as the logic will be exercised in all existing kubeadm e2e tests

Graduation Criteria

Alpha

  • Feature implemented behind the feature gate EtcdLearnerMode
  • Initial unit and e2e tests completed and enabled
  • Document the feature gate .

Beta

  • Gather feedback from developers and surveys
  • Make unit and e2e test changes
  • Update the feature gate documentation

GA

  • Gather feedback from developers and surveys
  • Update unit tests
  • Remove e2e tests as this will be the only code path for adding etcd members and it will be tested by all existing kubeadm e2e tests
  • Update the feature gate documentation

Upgrade / Downgrade Strategy

  • N/A -> Alpha: users can patch their ClusterConfiguration in the kube-system/kubeadm-config ConfigMap to before calling kubeadm upgrade apply This will allow them to enable learner mode in case they wish to add more etcd members to this cluster. This scenario is anticipated as rare, because usually users maintain a stable control plane with 3 or more members before upgrading it. But it is still plausible and can be documented in the feature gate documentation.
  • Alpha -> Beta: similarly to the previous stage users can modify the ClusterConfiguration to disable the feature gate during upgrade. This will allow them to use the “old way”, in case they wish to add more etcd members to the cluster while the feature gate is enabled by default.
  • Beta -> GA: users could no longer patch the ClusterConfiguration to opt-out of the feature and it will be locked to default.

Version Skew Strategy

One important point to make would be that kubeadm must handle a case where the user locked their etcd server version to version < 3.4. This would mean that they must get a sensible error in the lines of “etcd learner mode is not supported by this etcd version” and the control plane with stacked etcd initialization should fail.

All etcd versions that are > 3.4 should be treated as supported by the EtcdLearnerMode feature gate.

If EtcdLearnerMode goes GA, but the user prefers to stay on etcd version < 3.4, their existing cluster will continue to work but they will not be able to add new stacked etcd members. For new clusters the combination of EtcdLearnerMode (GA) and etcd version < 3.4 will not be supported.

Production Readiness Review Questionnaire

kubeadm is considered an “out of tree” component and PRR is out of scope.

Implementation History

Drawbacks

The implementation and enablement of EtcdLearnerMode by default hides a number of risks around stability. The “old way” has been tested for years and consumed by many users. By modifying this code path we are introducing potential for user complains about HA cluster creation and maintenance with kubeadm. Sufficient testing and gathering feedback from users would be mandatory.

Alternatives

N/A

Infrastructure Needed (Optional)

N/A