|
1 | | -# cli-utils |
| 1 | +# Flux kstatus |
2 | 2 |
|
3 | | -`cli-utils` is a collection of Go libraries designed to facilitate bulk |
4 | | -actuation of Kubernetes resource objects by wraping and enahancing |
5 | | -`kubectl apply` with a more user friendly abstraction. |
| 3 | +[](https://github.com/fluxcd/cli-utils/releases) |
| 4 | +[](https://github.com/fluxcd/cli-utils/blob/main/LICENSE) |
| 5 | +[](https://github.com/fluxcd/cli-utils/actions) |
6 | 6 |
|
7 | | -While the name indicates a focus on CLI utilities, the project has evolved to |
8 | | -encompass a broader scope, including CLI use and server-side use in GitOps |
9 | | -controllers. |
| 7 | +This repository is a hard fork of [kubernetes-sigs/cli-utils](https://github.com/kubernetes-sigs/cli-utils). |
10 | 8 |
|
11 | | -## Features |
| 9 | +We've forked `cli-utils` in 2023 as the upstream repo lagged months behind Kubernetes & Kustomize, |
| 10 | +which blocked our ability to use the latest Kustomize features in Flux. |
| 11 | +We have since made a number of improvements and bug fixes to the [kstatus](pkg/kstatus) package |
| 12 | +specific to Flux's needs, and we intend to maintain this fork for the foreseeable future. |
12 | 13 |
|
13 | | -1. **Pruning** |
14 | | -1. **Status Interpretation** |
15 | | -1. **Status Lookup** |
16 | | -1. **Diff & Preview** |
17 | | -1. **Waiting for Reconciliation** |
18 | | -1. **Resource Ordering** |
19 | | -1. **Explicit Dependency Ordering** |
20 | | -1. **Implicit Dependency Ordering** |
21 | | -1. **Apply Time Mutation** |
22 | | -1. **CLI Printers** |
| 14 | +Besides Flux, the `kstatus` fork is also used by other projects such as Helm, Cilium, Pulumi, Talos and |
| 15 | +many others. The Flux maintainers are committed to maintaining it as a standalone library |
| 16 | +for the benefit of the wider Kubernetes ecosystem. |
23 | 17 |
|
24 | | -### Pruning |
| 18 | +## kstatus |
25 | 19 |
|
26 | | -The Applier automatically deletes objects that were previously applied and then |
27 | | -removed from the input set on a subsequent apply. |
| 20 | +The `kstatus` package provides utilities for computing the status of Kubernetes resources, |
| 21 | +and for polling the cluster to determine when resources have reached a desired status. |
28 | 22 |
|
29 | | -The current implementation of `kubectl apply --prune` uses labels to identify the |
30 | | -set of previously applied objects in the prune set calculation. But the use of labels |
31 | | -has significant downsides. The current `kubectl apply --prune` implementation is alpha, |
32 | | -and it is improbable that it will graduate to beta. `cli-utils` attempts to address |
33 | | -the current `kubectl apply --prune` deficiencies by storing the set of previously |
34 | | -applied objects in an **inventory** object which is applied to the cluster. The |
35 | | -reference implementation uses a `ConfigMap` as an **inventory** object, and references |
36 | | -to the applied objects are stored in the `data` section of the `ConfigMap`. |
| 23 | +This package is used to determine the health and readiness of resources managed by Flux controllers, |
| 24 | +and to provide feedback to users about the state of their GitOps pipelines. |
37 | 25 |
|
38 | | -The following example illustrates a `ConfigMap` resource used as an inventory object: |
| 26 | +Instead of using `kstatus` directly, we recommend using |
| 27 | +the [github.com/fluxcd/pkg/ssa](https://github.com/fluxcd/pkg/tree/main/ssa) package |
| 28 | +that offers a high level API for applying resources and waiting for them to be ready: |
39 | 29 |
|
40 | | -```yaml |
41 | | -apiVersion: v1 |
42 | | -kind: ConfigMap |
43 | | -metadata: |
44 | | - # DANGER: Do not change the inventory object namespace. |
45 | | - # Changing the namespace will cause a loss of continuity |
46 | | - # with previously applied grouped objects. Set deletion |
47 | | - # and pruning functionality will be impaired. |
48 | | - namespace: test-namespace |
49 | | - # NOTE: The name of the inventory object does NOT have |
50 | | - # any impact on group-related functionality such as |
51 | | - # deletion or pruning. |
52 | | - name: inventory-26306433 |
53 | | - labels: |
54 | | - # DANGER: Do not change the value of this label. |
55 | | - # Changing this value will cause a loss of continuity |
56 | | - # with previously applied grouped objects. Set deletion |
57 | | - # and pruning functionality will be impaired. |
58 | | - cli-utils.sigs.k8s.io/inventory-id: 46d8946c-c1fa-4e1d-9357-b37fb9bae25f |
| 30 | +```go |
| 31 | +// WaitForSetWithContext checks if the given ObjMetadataSet has been fully reconciled. |
| 32 | +// The provided context can be used to cancel the operation. |
| 33 | +func (m *ResourceManager) WaitForSetWithContext(ctx context.Context, set object.ObjMetadataSet, opts WaitOptions) error |
59 | 34 | ``` |
60 | 35 |
|
61 | | -### Status Interpretation |
| 36 | +### CEL Extensions |
62 | 37 |
|
63 | | -The `kstatus` library can be used to read an object's current status and interpret |
64 | | -whether that object has be reconciled (aka Current) or not, including whether it |
65 | | -is expected to never reconcile (aka Failed). |
| 38 | +While the `kstatus` package can determine the status of native Kubernetes resources (e.g. a Deployment) |
| 39 | +and Flux custom resources that subscribe to the "abnormal-true" polarity pattern (`Reconciling` and `Stalled` conditions), |
| 40 | +it may not be able to determine the status of other custom resources in a reliable manner. |
66 | 41 |
|
67 | | -### Status Lookup |
| 42 | +To overcome the limitations of kstatus, we have extended it with CEL expressions to enable Flux users |
| 43 | +to define the logic for computing the status which is evaluated against the resource's status fields. |
| 44 | +This allows accurate status computation for custom resources regardless of the conditions they use, |
| 45 | +and it also allows for more complex logic to be implemented in the status computation. |
68 | 46 |
|
69 | | -In addition to performing interpretation of status from an object in-memory, |
70 | | -`cli-utils` can also be used to query status from the server, allowing you to |
71 | | -retrieve the status of previously or concurrently applied objects. |
| 47 | +References: |
72 | 48 |
|
73 | | -### Diff & Preview |
74 | | - |
75 | | -`cli-utils` can be used to compare local object manifests with remote objects |
76 | | -from the server. These can be compared locally with diff or remotely with |
77 | | -preview (aka dry-run). This can be useful for discovering drift or previewing |
78 | | -which changes would be made, if the local manifests were applied. |
79 | | - |
80 | | -### Waiting for Reconciliation |
81 | | - |
82 | | -The Applier automatically watches applied and deleted objects and tracks their |
83 | | -status, blocking until the objects have reconciled, failed, or been fully |
84 | | -deleted. |
85 | | - |
86 | | -This functionality is similar to `kubectl delete <resource> <name> --wait`, in |
87 | | -that it waits for all finalizers to complete, except it also works for creates |
88 | | -and updates. |
89 | | - |
90 | | -While there is a `kubectl apply <resource> <name> --wait`, it only waits for |
91 | | -deletes when combined with `--prune`. `cli-utils` provides an alternative that |
92 | | -works for all spec changes, waiting for reconciliation, the convergence of |
93 | | -status to the desired specification. After reconciliation, it is expected that |
94 | | -the object has reached a steady state until the specification is changed again. |
95 | | - |
96 | | -### Resource Ordering |
97 | | - |
98 | | -The Applier and Destroyer use resource type to determine which order to apply |
99 | | -and delete objects. |
100 | | - |
101 | | -In contrast, when using `kubectl apply`, the objects are applied in alphanumeric |
102 | | -order of their file names, and top to bottom in each file. With `cli-utils`, |
103 | | -this manual sorting is unnecessary for many common use cases. |
104 | | - |
105 | | -### Explicit Dependency Ordering |
106 | | - |
107 | | -While resource ordering provides a smart default user experience, sometimes |
108 | | -resource type alone is not enough to determine desired ordering. In these cases, |
109 | | -the user can use explicit dependency ordering by adding a |
110 | | -`config.kubernetes.io/depends-on: <OBJECT_REFERENCE>` annotation to an object. |
111 | | - |
112 | | -The Applier and Destroyer use these explicit dependency directives to build a |
113 | | -dependency tree and flatten it for determining apply ordering. When deleting, |
114 | | -the order is reversed, ensuring that dependencies are not deleted before the |
115 | | -objects that depend on them (aka dependents). |
116 | | - |
117 | | -In addition to ordering the applies and deletes, dependency ordering also waits |
118 | | -for dependency reconciliation when applying and deletion finalization when |
119 | | -deleting. This ensures that dependencies are not just applied first, but have |
120 | | -reconciled before their dependents are applied. Likewise, dependents are not |
121 | | -just deleted first, but have completed finalization before their dependencies |
122 | | -are deleted. |
123 | | - |
124 | | -Also, because dependency ordering is enforced during actuation, a dependency |
125 | | -cannot be pruned by the Applier unless all its dependents are also deleted. This |
126 | | -prevents accidental premature deletion of objects that are still in active use. |
127 | | - |
128 | | -In the following example, the `config.kubernetes.io/depends-on` annotation |
129 | | -identifies that `pod-c` must be successfully applied prior to `pod-a` |
130 | | -actuation: |
131 | | - |
132 | | -```yaml |
133 | | -apiVersion: v1 |
134 | | -kind: Pod |
135 | | -metadata: |
136 | | - name: pod-a |
137 | | - annotations: |
138 | | - config.kubernetes.io/depends-on: /namespaces/default/Pod/pod-c |
139 | | -spec: |
140 | | - containers: |
141 | | - - name: kubernetes-pause |
142 | | - image: registry.k8s.io/pause:2.0 |
143 | | -``` |
144 | | - |
145 | | -### Implicit Dependency Ordering |
146 | | - |
147 | | -In addition to being able to specify explicit dependencies, `cli-utils` |
148 | | -automatically detects some implicit dependencies. |
149 | | - |
150 | | -Implicit dependencies include: |
151 | | - |
152 | | -1. Namespace-scoped resource objects depend on their Namespace. |
153 | | -2. Custom resource objects depend on their Custom Resource Definition |
154 | | - |
155 | | -Like resource ordering, implicit dependency ordering improves the apply and |
156 | | -delete experience to reduce the need to manually specify ordering for many |
157 | | -common use cases. This allows more objects to be applied together all at once, |
158 | | -with less manual orchestration. |
159 | | - |
160 | | -### Apply-Time Mutation |
161 | | - |
162 | | -The Applier can dynamically modify objects before applying them, performing |
163 | | -field value substitution using input(s) from dependency fields. |
164 | | - |
165 | | -This allows for applying objects together in a set that you would otherwise need |
166 | | -to seperate into multiple sets, with manual modifications between applies. |
167 | | - |
168 | | -Apply-Time Mutation is configured using the |
169 | | -`config.kubernetes.io/apply-time-mutation` annotation on the target object to be |
170 | | -modified. The annotation may specify one or more substitutions. Each |
171 | | -substitution includes a source object, and source field path, and a target |
172 | | -field path, with an optional token. |
173 | | - |
174 | | -If the token is specified, the token is |
175 | | -replaced in the target field value string with the source field value. If the |
176 | | -token is not specified, the whole target field value is replaced with the |
177 | | -source field value. This alternatively allows either templated interpretation or |
178 | | -type preservation. |
179 | | - |
180 | | -The source and target field paths are specified using JSONPath, allowing for |
181 | | -robust navigation of complex resource field hierarchies using a familiar syntax. |
182 | | - |
183 | | -In the following example, `pod-a` will substitute the IP address and port from |
184 | | -the spec and status of the source `pod-b` into the spec of the target `pod-a`: |
185 | | - |
186 | | -```yaml |
187 | | -kind: Pod |
188 | | -apiVersion: v1 |
189 | | -metadata: |
190 | | - name: pod-a |
191 | | - annotations: |
192 | | - config.kubernetes.io/apply-time-mutation: | |
193 | | - - sourceRef: |
194 | | - kind: Pod |
195 | | - name: pod-b |
196 | | - sourcePath: $.status.podIP |
197 | | - targetPath: $.spec.containers[?(@.name=="nginx")].env[?(@.name=="SERVICE_HOST")].value |
198 | | - token: ${pob-b-ip} |
199 | | - - sourceRef: |
200 | | - kind: Pod |
201 | | - name: pod-b |
202 | | - sourcePath: $.spec.containers[?(@.name=="nginx")].ports[?(@.name=="tcp")].containerPort |
203 | | - targetPath: $.spec.containers[?(@.name=="nginx")].env[?(@.name=="SERVICE_HOST")].value |
204 | | - token: ${pob-b-port} |
205 | | -spec: |
206 | | - containers: |
207 | | - - name: nginx |
208 | | - image: nginx:1.21 |
209 | | - ports: |
210 | | - - name: tcp |
211 | | - containerPort: 80 |
212 | | - env: |
213 | | - - name: SERVICE_HOST |
214 | | - value: "${pob-b-ip}:${pob-b-port}" |
215 | | -``` |
216 | | - |
217 | | -The primary reason to do this with Apply-Time Mutation, instead of client-side |
218 | | -manifest templating is that the pod IP is populated by a controller at runtime |
219 | | -during reconciliation, and is not known before applying. |
220 | | - |
221 | | -That said, this is a toy example using built-in types. For pods, you probably |
222 | | -actually want to use DNS for service discovery instead. |
223 | | - |
224 | | -Most use cases for Apply-Time Mutation are actually using custom resources, as a |
225 | | -temporary alternative to building higher level abstractions, modifying |
226 | | -interfaces, or creating dependencies between otherwise independent interfaces. |
227 | | - |
228 | | -### CLI Printers |
229 | | - |
230 | | -Since the original intent of `cli-utils` was to contain common code for CLIs, |
231 | | -and end-to-end testing requires a reference implementation, a few printers are |
232 | | -included to translate from the primary event stream into STDOUT text: |
233 | | - |
234 | | -1. **Event Printer**: The event printer just prints text to STDOT whenever an |
235 | | - event is recieved. |
236 | | -1. **JSON Printer**: The JSON printer converts events into a JSON string per |
237 | | - line, intended for automated interpretation by machine. |
238 | | -1. **Table Printer**: The table printer writes and updates in-place a table |
239 | | - with one object per line, intended for human consumption. |
240 | | - |
241 | | -## Packages |
242 | | - |
243 | | -├── **cmd**: the kapply CLI command |
244 | | -├── **examples**: examples that serve as additional end-to-end tests using mdrip |
245 | | -├── **hack**: hacky scripts used by make |
246 | | -├── **pkg** |
247 | | -│ ├── **apis**: API resources that satisfy the kubernetes Object interface |
248 | | -│ ├── **apply**: bulk applier and destroyer |
249 | | -│ ├── **common**: placeholder for common tools that should probably have their own package |
250 | | -│ ├── **config**: inventory config bootstrapping |
251 | | -│ ├── **errors**: error printing |
252 | | -│ ├── **flowcontrol**: flow control enablement discovery |
253 | | -│ ├── **inventory**: inventory resource reference implementation |
254 | | -│ ├── **jsonpath**: utility for using jsonpath to read & write Unstructured object fields |
255 | | -│ ├── **kstatus**: object status event watcher with ability to reduce status to a single enum |
256 | | -│ ├── **manifestreader**: bolk resource object manifest reading and parsing |
257 | | -│ ├── **multierror**: error composition |
258 | | -│ ├── **object**: library for dealing with Unstructured objects |
259 | | -│ ├── **ordering**: sort functionality for objects |
260 | | -│ ├── **print**: CLI output |
261 | | -│ ├── **printers**: CLI output |
262 | | -│ └── **testutil**: utility for facilitating testing |
263 | | -├── **release**: goreleaser config |
264 | | -├── **scripts**: scripts used by make |
265 | | -└── **test**: end-to-end and stress tests |
266 | | - |
267 | | -## kapply |
268 | | - |
269 | | -To facilitate testing, this repository includes a reference CLI called `kapply`. |
270 | | -The `kapply` tool is not intended for direct consumer use, but may be useful |
271 | | -when trying to determine how to best utilize the `cli-utils` library packages. |
272 | | - |
273 | | -## Community, discussion, contribution, and support |
274 | | - |
275 | | -Learn how to engage with the Kubernetes community on the [community page](http://kubernetes.io/community/). |
276 | | - |
277 | | -You can reach the maintainers of this project at: |
278 | | - |
279 | | -- [Slack channel](https://kubernetes.slack.com/messages/sig-cli) |
280 | | -- [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-cli) |
281 | | - |
282 | | -### Code of conduct |
283 | | - |
284 | | -Participation in the Kubernetes community is governed by the [Kubernetes Code of Conduct](code-of-conduct.md). |
| 49 | +- [Flux Kustomization health expressions docs](https://fluxcd.io/flux/components/kustomize/kustomizations/#health-check-expressions) |
| 50 | +- [Flux HelmRelease health expressions docs](https://fluxcd.io/flux/components/helm/helmreleases/#health-check-expressions) |
| 51 | +- [CEL extensions for kstatus source code](https://github.com/fluxcd/pkg/tree/main/runtime/cel) |
0 commit comments