KEP-1739: kubeadm customization with patches

Release Signoff Checklist
Summary
Motivation
- Goals
- Non-Goals
Proposal
Design Details
Production Readiness Review Questionnaire
Implementation History
Drawbacks
Alternatives
Infrastructure Needed (optional)

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

(R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
(R) KEP approvers have approved the KEP status as implementable
(R) Design details are appropriately documented
(R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
(R) Graduation criteria is in place
(R) Production readiness review completed
Production readiness review approved
“Implementation History” section is up-to-date for milestone
User-facing documentation has been created in kubernetes/website , for publication to kubernetes.io
Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

This KEP proposes a replacement for the feature introduced by the kubeadm KEP “20190722-Advanced-configurations-with-kubeadm-(Kustomize)”.

It has similar scope and goals, allowing the users to amend manifests and configuration files generated by kubeadm using patches, but not using Kustomize as the backend for applying the patches.

The Kustomize implementation in kubeadm was in an Alpha state and was removed.

Motivation

The kubeadm team has decided that it is more beneficial if the project moves to using raw patches instead of Kustomize. Kustomize introduces an undesired dependency which can be avoided by using patches in a similar way kubectl uses them.

Goals

Implement a solution for applying patches over kubeadm generated manifests and configuration during the common kubeadm commands “init”, “join” and “upgrade” and their phases.
Deprecate and remove the existing Kustomize solution after a grace period of at least one release.

Non-Goals

Validate if the user patches contain a good practice configuration and sane security values.
Allow patching of core addon configuration generated by kubeadm - e.g. CoreDNS, kube-proxy (until further notice).
Replace the kubeadm ComponentConfig. The sane defaults would still be generated by kubeadm.

Proposal

This KEP proposes introducing a new flag called --experimental-patches next to the existing --experimental-kustomize. The flag has the exact same semantics pointing to a directory with patches. Once the feature graduates to Beta the flag will be renamed to --patches. As part of its graduation the functionality can also be considered as an addition to the kubeadm ComponentConfig.

All the relevant kubeadm commands such as “init”, “join” and “upgrade” will support the flag.

A kustomization.yaml file will not be required in the directory and instead it will contain a list of patches named following a specific format.

Once the flag is introduced, the existing flag --experimental-kustomize will be marked as deprecated.

User Stories (optional)

(Based on “20190722-Advanced-configurations-with-kubeadm-(Kustomize)”)

Story 1

As a cluster administrator, I want to add a sidecar to the kube-apiserver Pod for running an authorization web-hooks serving component.

Story 2

As a cluster administrator, I want to set timeouts for the kube-apiserver liveness probes for edge clusters.

Story 3

As a cluster administrator, I want to upgrade my cluster preserving all the patches that were applied during “init”/“join”.

Story 4

As a cluster administrator, I want to be able to apply instance specific configuration to kubelets on different nodes.

Notes/Constraints/Caveats (optional)

NONE

Risks and Mitigations

(Based on “20190722-Advanced-configurations-with-kubeadm-(Kustomize)”)

Confusion between kubeadm ComponentConfig and usage of patches

kubeadm already offers a way to implement cluster settings using ComponentConfig. Adding a new feature for supporting customizations using patches can create confusion in the users.

The kubeadm maintainers need to make it clear to the users in documentation and release notes what is the scope of the new feature.

Users are allowed to pass configuration for components using the kubeadm ComponentConfiguration.
kubeadm applies sane / secure defaults.
The users can patch the generated configuration to their liking.

Misleading expectations on the level of flexibility

Even if the proposed solution is based on the user feedback/issues, the kubeadm maintainers want to be sure the implementation is providing the expected level of flexibility. In order to ensure that, feedback will be examined before moving forward in graduating the feature to beta.

Breaking changes post upgrade

A change in a kubeadm generated configuration can make a patch apply to fail.

The kubeadm maintainers will work on release notes to make potential breaking changes more visible. Additionally, upgrade instructions will be updated adding the recommendation to –dry-run and check expected changes before upgrades.

Patch apply errors

If a patch fails to apply, kubeadm command should exit with a descriptive error.

The users would have to either remove the problematic patch or amend it.

Confusion between the old and new customization features

For a period of time (at least one release) the flags --experimental-kustomize and --experimental-patches need to co-exist, which can confuse the users.

By marking --experimental-kustomize as deprecated, the flag will be hidden from command –helpscreens. The warning that is printing when the deprecated flag is used should be amended to denote that–experimental-patches` should be used instead.

The --experimental-patches patches would apply after the --experimental-kustomize patches are applied.

Design Details

Code organization

The code organization of the new feature will be done in a similar fashion to the existing Kustomize feature. The backend will be stored in a new package called patches. Kubeadm commands would have minimal exposure to the underlying backend and only call a single method that accepts the directory where the user patches are.

Patch formats

The proposed design is supporting the following patch formats similarly to kubectl:

Strategic Merge Patches (default):
- Supported as strategic by kubectl patch.
- Will be implemented using the k8s.io/apimachinery library.
RFC6902 JSON patches:
- Supported as json by kubectl patch.
- Would be implemented using the github.com/evanphx/json-patch library.
RFC7396 JSON merge patches:
- Supported as merge by kubectl patch.
- Would be implemented using the github.com/evanphx/json-patch library.

Note that kubectl already uses the same backends for the above patch formats.

Patches defined in both YAML and JSON will be supported, with the exception of json which must be in JSON only.

Conversion from YAML to JSON will be performed using the sigs.k8s.io/yaml library.

Patch file naming format

Once the user passes a folder with patches to a kubeadm command, kubeadm must be able to determine what format each file is and what is its target.

Parsing the patch contents to determine the patch format is not possible, because the formats strategic and merge are the same, while they do have different apply mechanisms.

The proposal is to have the following naming format: target[suffix][+patchtype].{yaml|json}

target must be a known component name - e.g. kube-apiserver or the name of generated configuration.
suffix is optional and can be used to ensure an order when applying patches from multiple files for the same component.
+patchtype is optional and can be used to pass one of the supported patch types. A missing value implies +strategic.
.{yaml|json} defines a patch file extension. It is required and must be either .yaml or .json.

Examples:

etcd.yaml
etcd+json
etcd+merge.json
etcd2.yaml
etcd2+merge.json

Alpha-numeric order will be used when applying patches from multiple files. This allows adjusting the order between patch types based on suffix.

The format:

Must be documented briefly in --help screens.
Must be fully documented at the kubernetes/website (Beta graduation).
Would allow for basic usage (using the default strategic type), but also allow for advanced usage with patch ordering and the json type.
Would allow having arbitrary files that are ignored in the patch folder - e.g. having a README.md.

Patch targets

The mechanism allows patching the following targets via the target naming property:

Control plane static Pods

Static Pod manifests generated on control plane nodes can be patched with the following targets: kube-apiserver, kube-controller-manager, kube-scheduler and etcd. Their API GroupVersionKind is core/v1.Pod.

Patches for the control plane static Pods are applied over the generated manifests in memory during init/join/upgrade.

Kubelet ComponentConfig

The generated per-node kubelet ComponentConfig can be patched with the kubeletconfiguration target. Its API GroupVersionKind is of type kubelet.config.k8s.io/v1beta1.KubeletConfiguration.

During init patches for the kubelet are applied over the user provided and kubeadm defaulted KubeletConfiguration. During join/upgrade patches are applied over the downloaded from the cluster global KubeletConfiguration.

Support for multiple patches per file

To support multiple patches per file, the proposal is to implement reading of JSON and YAML multi-documents where the patches are separated by ---\n.

Multiple patches in a file will be applied top-first.

This extension is fairly simple to implement and would allow even higher flexibility.

Test Plan

The feature will be tested extensively by unit tests.

Similarly to the existing Kustomize feature, e2e tests will be added in the same cycle when the new feature is added in its Alpha state.

For e2e tests the existing kubeadm testing infrastructure (i.e. Prow jobs with kinder) will be used. The e2e tests will ensure that the kubeadm commands “init”, “join” and “upgrade” support the feature.

Graduation Criteria

Alpha -> Beta Graduation

No major bugs are present.
No UX complains by the users are received.
The feature was tested for at least one release in e2e tests and the tests are stable.
The Alpha flag --experimental-kustomize and the underlying backend are removed.
The Alpha flag --experimental-patches is renamed to --patches.
The feature is documented at kubernetes/website under the kubeadm pages.
The functionality is added as part of the kubeadm ComponentConfig under the NodeRegistrationOptions structure. Requires synchronization with a kubeadm ComponentConfig version increment, so could land in GA graduation.

Beta -> GA Graduation

The feature is widely used and at least 2 cycles have passed since Beta.
The --patches flag is removed (optional). Depends on user feedback and the wider kubeadm plan for removing flags in favor of ComponentConfig.
More patch targets are considered (optional).

Upgrade / Downgrade Strategy

Downgrade is not supported by kubeadm.
For upgrades, similarly to the existing Kustomize feature, the new feature will be supported during the execution of the kubeadm upgrade command.
Once kubeadm generates its upgraded manifest files, the folder with patches will be processed and the patches will be applied to the manifests.
In case a patch fails to apply during upgrade, the user will be informed to fix the issue manually.

Version Skew Strategy

This section covers the corev1.Pod as a example target for version skew.

Once introduced, the feature will support patching at minimum the corev1.Pod object. If at some point the Kubernetes core API graduates to v2, kubeadm would have to support patching both corev2 and corev1 for at least one release cycle.

This is due to the fact kubeadm supports deploying Kubernetes v1.YY and v1.YY-1. And if corev2 is added in v1.YY, kubeadm must be able to handle corev1 that is supported in v1.YY-1.

A similar skew strategy must be applied for other supported patch target API types.

Production Readiness Review Questionnaire

Feature enablement and rollback

This section must be completed when targeting alpha to a release.

How can this feature be enabled / disabled in a live cluster?
- Feature gate (also fill in values in kep.yaml)
  - Feature gate name:
  - Components depending on the feature gate:
- Other
  - Describe the mechanism: When a new kubeadm control-plane Node joins the cluster it can optionally apply custom patches to kubeadm generated configuration.
  - Will enabling / disabling the feature require downtime of the control plane? No, unless the user created patches result in bad configuration and this is done on the primary control-plane Node - i.e. no other control-plane Nodes exist yet, which is not a “live cluster” yet.
  - Will enabling / disabling the feature require downtime or reprovisioning of a node? Potentially, during a mutable kubeadm upgrade on a control-plane Node, if the user created patches result in bad configuration.
Does enabling the feature change any default behavior? Any change of default behavior may be surprising to users or break existing automations, so be extremely careful here.
Can the feature be disabled once it has been enabled (i.e. can we rollback the enablement)? Also set rollback-supported to true or false in kep.yaml. Describe the consequences on existing workloads (e.g. if this is runtime feature, can it break the existing applications?). Yes. Once the patches are applied for a kubeadm init|join|upgrade command, the user can decide to roll-back the changes which will result in a restart of component that was customized by the patches.
What happens if we reenable the feature if it was previously rolled back? This is supported by invoking kubeadm phases. The patches will re-apply and the kubelet on the Node will pick up the changes and restart the locally managed control-plane components.
Are there any tests for feature enablement/disablement? The e2e framework does not currently support enabling and disabling feature gates. However, unit tests in each component dealing with managing data created with and without the feature are necessary. At the very least, think about conversion tests if API types are being modified. The feature will include both e2e and unit tests as its Alpha graduation. Feature gates are not used.

Rollout, Upgrade and Rollback Planning

This section must be completed when targeting beta graduation to a release.

How can a rollout fail? Can it impact already running workloads? Try to be as paranoid as possible - e.g. what if some components will restart in the middle of rollout? The feature can result in malfunctioning control-plane components, due to the fact that it targets patching of configuration with user values. The user is responsible for configuring the control-plane correctly.
What specific metrics should inform a rollback? Not applicable.
Were upgrade and rollback tested? Was upgrade->downgrade->upgrade path tested? Describe manual testing that was done and the outcomes. Longer term, we may want to require automated upgrade/rollback tests, but we are missing a bunch of machinery and tooling and do that now. The feature will include an e2e test for upgrade, but rollback is not planned for testing.
Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? Even if applying deprecation policies, they may still surprise some users. The feature will deprecate an existing feature powered by the kubeadm flag --experimental-kustomize. The existing feature will be removed after one release, even if being Alpha grade. An action-required release note will be filed just in case.

Monitoring requirements

This section must be completed when targeting beta graduation to a release.

How can an operator determine if the feature is in use by workloads? Ideally, this should be a metrics. Operations against Kubernetes API (e.g. checking if there are objects with field X set) may be last resort. Avoid logs or events for this purpose. Not directly applicable as the feature will be used to configure control-plane.
What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
- Metrics
  - Metric name:
  - [Optional] Aggregation method:
  - Components exposing the metric:
- Other (treat as last resort)
  - Details: If a control-plane component on a Node fails to start, the admin must inspect their patches and debug the failure.
What are the reasonable SLOs (Service Level Objectives) for the above SLIs? At the high-level this usually will be in the form of “high percentile of SLI per day <= X”. It’s impossible to provide a comprehensive guidance, but at the very high level (they needs more precise definitions) those may be things like:
- per-day percentage of API calls finishing with 5XX errors <= 1%
- 99% percentile over day of absolute value from (job creation time minus expected job creation time) for cron job <= 10%
- 99,9% of /health requests per day finish with 200 code Not applicable as the cluster-admin would see failures immediately after kubeadm commands.
Are there any missing metrics that would be useful to have to improve observability if this feature? Describe the metrics themselves and the reason they weren’t added (e.g. cost, implementation difficulties, etc.). Not applicable.

Dependencies

This section must be completed when targeting beta graduation to a release.

Does this feature depend on any specific services running in the cluster? Think about both cluster-level services (e.g. metrics-server) as well as node-level agents (e.g. specific version of CRI). Focus on external or optional services that are needed. For example, if this feature depends on a cloud provider API, or upon an external software-defined storage or network control plane.
For each of the dependencies fill in the following, thinking both about running user workloads and creating new ones, as well as about cluster-level services (e.g. DNS):
- [Dependency name]
  - Usage description:
    - Impact of its outage on the feature:
    - Impact of its degraded performance or high error rates on the feature: No, this feature customizes on top of kubeadm generated configuration.

Scalability

For alpha, this section is encouraged: reviewers should consider these questions and attempt to answer them.

For beta, this section is required: reviewers must answer these questions.

For GA, this section is required: approvers should be able to confirms the previous answers based on experience in the field.

Will enabling / using this feature result in any new API calls? Describe them, providing:
- API call type (e.g. PATCH pods)
- estimated throughput
- originating component(s) (e.g. Kubelet, Feature-X-controller) focusing mostly on:
- components listing and/or watching resources they didn’t before
- API calls that may be triggered by changes of some Kubernetes resources (e.g. update of object X triggers new updates of object Y)
- periodic API calls to reconcile state (e.g. periodic fetching state, heartbeats, leader election, etc.) Not applicable.
Will enabling / using this feature result in introducing new API types? Describe them providing:
- API type
- Supported number of objects per cluster
- Supported number of objects per namespace (for namespace-scoped objects) Yes. For the Beta graduation potentially, but that would be a kubeadm ComponentConfig sub-type.
Will enabling / using this feature result in any new calls to cloud provider? No.
Will enabling / using this feature result in increasing size or count of the existing API objects? Describe them providing:
- API type(s):
- Estimated increase in size: (e.g. new annotation of size 32B)
- Estimated amount of new objects: (e.g. new Object X for every existing Pod) No.
Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs ? Think about adding additional work or introducing new steps in between (e.g. need to do X to start a container), etc. Please describe the details. No, unless the user patches kubeadm generated configuration with undesired settings.
Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, …) in any components? Things to keep in mind include: additional in-memory state, additional non-trivial computations, excessive access to disks (including increased log volume), significant amount of data send and/or received over network, etc. This through this both in small and large cases, again with respect to the supported limits . No, unless the user patches kubeadm generated configuration with undesired settings.

Troubleshooting

Troubleshooting section serves the Playbook role as of now. We may consider splitting it into a dedicated Playbook document (potentially with some monitoring details). For now we leave it here though.

This section must be completed when targeting beta graduation to a release.

How does this feature react if the API server and/or etcd is unavailable? This feature can be used to configure an etcd or API server instance, by patching their static Pods (considering etcd is run as a static Pod too).
What are other known failure modes? For each of them fill in the following information by copying the below template:
- [Failure mode brief description]
  - Detection: How can it be detected via metrics? Stated another way: how can an operator troubleshoot without logging into a master or worker node? Not applicable.
  - Mitigations: What can be done to stop the bleeding, especially for already running user workloads? The user needs to debug what patch is causing the failure and rollback the change.
  - Diagnostics: What are the useful log messages and their required logging levels that could help debugging the issue? Patches that failed to apply will result in overall kubeadm command failures, causing an exit > 0 status. Not required until feature graduated to Beta.
  - Testing: Are there any tests for failure mode? If not describe why. Failure mode testing for patches is out-of-scope for e2e tests, but will be tested in unit tests.
What steps should be taken if SLOs are not being met to determine the problem? Not applicable.

Implementation History

2020-05-04: initial provisional KEP created
2020-05-14: addressed feedback, filled PRR questionnaire, KEP marked as implementable
2021-09-09: marked the feature as graduated to Beta. This was done as part of the kubeadm v1beta3 API work. The actions under “Alpha -> Beta” are completed.
2022-05-16: adapted the KEP to be more generic, allowing support for patching any kubeadm generated configuration in the future and not only control plane component manifests. Added enumeration of supported patch targets.

Drawbacks

None.

Alternatives

One alternative is using the existing Kustomize feature, yet as pointed out in Motivation the kubeadm maintainers have reservations for using this approach.

Another alternative is supporting instance specific configuration that is persisted in the kubeadm created cluster. Yet, to fully replace the flexibility that patches enable, it requires that the persisted configuration is stored in the underlying low-level format, such as corev1.Pod.

It is not clear whether the kubeadm ComponentConfig will support such alternatives in the future.

Infrastructure Needed (optional)

None.