CSM 1.16.2 changes with [PR-1759] for main (#1763)

Author: mgandharva

content/docs/concepts/authorization/v2.x/_index.md

@@ -11,9 +11,150 @@ tags:
 
 The following diagram shows a high-level overview of Container Storage Modules for Authorization with a `tenant-app` that is using a CSI driver to perform storage operations through the CSM for Authorization `proxy-server` to access the a Dell storage system. All requests from the CSI driver will contain the token for the given tenant that was granted by the Storage Administrator.
 
-![Alt text](../../../../images/authorization/v2.x/image.png)
+> **Important**: CSM Authorization **must** be deployed and configured **before** installing the CSI driver. The driver must be configured from the outset to route all storage operations through the Authorization service. For high availability and security containment, the CSM Authorization Proxy Server should run within a Kubernetes cluster managed by the Storage Administrator.
 
-This is the introduction to a Stateless Architecture for Authorization. The creation of storage, roles, and tenants is done through Custom Resources (CRs) which are tracked and contained within CSM Authorization. The underlying communication is consistent with the previous architecture which makes the creation of volumes and snapshots seamless.
+![Alt text](../../../../images/authorization/v2.x/image_new.png)
+
+### Diagram Explanation
+
+The diagram above illustrates the CSM Authorization architecture:
+
+> **Note**: The Driver Namespace and Authorization Namespace can reside on different Kubernetes clusters or on the same cluster.
+
+#### Driver Namespace (Tenant)
+
+1. An **Application pod** issues a storage request (e.g., create volume) via a PersistentVolumeClaim.
+2. The **CSI Driver Pod** contains the **CSI Driver Container** (with Controller and Node components) and an **Authorization sidecar**. The sidecar:
+   - Injects a Bearer JWT token into every outgoing storage request.
+   - Adds forward headers for request routing.
+   - Auto-refreshes access tokens using the refresh token before they expire. If the refresh token expires, new tokens must be manually generated and applied.
+   - Presents a self-signed TLS certificate on the localhost endpoint where the driver sends requests.
+3. The sidecar forwards the request over **HTTPS + JWT** to the Authorization Proxy Server.
+
+#### Authorization Namespace (Proxy)
+
+4. The **Authorization Proxy Server** receives the request and performs **JWT Validation** to authenticate the tenant.
+5. The **Dispatch Handler** routes the request to the appropriate storage backend handler (PowerMax, PowerFlex, PowerStore, or PowerScale).
+6. The following services support the proxy server:
+   - **Tenant Service** — manages tenant configurations and generates JSON Web Tokens.
+   - **Role Service** — manages roles that tenants are bound to.
+   - **Storage Service** — manages backend storage array configurations and makes **Storage API calls** to the arrays.
+7. **Redis (Sentinel)** stores tenant data, enforces quota limits, and handles SDC approval.
+8. **Open Policy Agent (OPA)** evaluates RBAC policies and issues Allow/Deny decisions for each request.
+9. The **CR Controller** manages Custom Resources for Storage, CSM Tenant, and CSM Role definitions.
+10. The **Credentials Manager** retrieves array credentials via the **Kubernetes Secrets Store CSI driver** (supporting Secrets, Vault, or Conjur).
+
+#### Backend Storage
+
+11. Once authorized, the proxy server makes **Driver API calls** to the backend Dell storage arrays (PowerMax, PowerFlex, PowerStore, or PowerScale). The storage array processes the request and returns the response back through the proxy to the CSI driver.
+
+> **Key point**: The storage array is unaware that authorization is taking place. The authorization proxy transparently handles and validates all requests.
+
+This is a Stateless Architecture for Authorization. The creation of storage, roles, and tenants is done through Custom Resources (CRs) which are tracked and contained within CSM Authorization. The underlying communication is consistent with the previous architecture which makes the creation of volumes and snapshots seamless.
+
+## Token Types and Lifecycle
+
+CSM Authorization uses three types of JSON Web Tokens (JWTs):
+
+### Admin Token
+
+- **Purpose**: Used by the Storage Administrator to authenticate with the CSM Authorization Proxy Server for administrative operations (e.g., generating tenant tokens).
+- **Generated by**: `dellctl admin token` command, using the JWT signing secret configured during CSM Authorization installation (the `web.jwtsigningsecret` in `karavi-config-secret`).
+- **Default expiration**: Access token = 1 minute, Refresh token = 720 hours (30 days).
+- **Who can generate it**: Only a user who knows the JWT signing secret (i.e., the Storage Administrator who installed CSM Authorization).
+- **Access requirements**: The `dellctl` CLI communicates with the **CSM Authorization Server**, not directly with the storage array. It can be run from any machine that has network access to the Authorization Server and a valid kubeconfig for the cluster. Direct storage array or Unisphere access is **not** required for token generation.
+
+### Access Token (Tenant)
+
+- **Purpose**: A short-lived token attached to every storage request by the Authorization sidecar proxy. The Proxy Server validates this token to authorize the request.
+- **Default expiration**: 1 minute (configurable via `--access-token-expiration` during token generation).
+- **Behavior on expiry**: When the access token expires, the sidecar proxy automatically requests a new access token using the refresh token. **No manual intervention is required.**
+
+### Refresh Token (Tenant)
+
+- **Purpose**: A longer-lived token used to obtain new access tokens without requiring the tenant to re-authenticate.
+- **Default expiration**: 720 hours (30 days) (configurable via `--refresh-token-expiration` during token generation).
+- **Behavior on expiry**: The refresh token will expire after its configured lifetime regardless of whether the CSI driver pods are active or not. The refresh token is **not** automatically refreshed. When it expires, **the Storage Administrator must generate a new token pair** using `dellctl generate token` and the Kubernetes Tenant Administrator must re-apply the new `proxy-authz-tokens` secret in the driver namespace.
+
+### Automatic Token Refresh
+
+The Authorization sidecar proxy handles access token refresh **automatically**:
+
+1. The sidecar attaches the access token to each storage request sent to the Proxy Server.
+2. If the Proxy Server returns HTTP 401 (indicating the access token has expired), the sidecar sends the refresh token to request a new access token.
+3. The Proxy Server validates the refresh token, checks that the tenant has not been revoked, and issues a new access token.
+4. Kubernetes retries the original CSI operation, and the sidecar attaches the new access token to the retried request.
+
+> **Important**: Only the **access token** is refreshed automatically. The **refresh token** itself is never refreshed — it will expire after its configured lifetime regardless of driver activity. When the refresh token expires, new tokens must be manually generated and applied.
+
+## Kubernetes Secrets Reference
+
+When configuring a CSI driver with CSM Authorization, several Kubernetes secrets are created in the driver namespace. The following table explains each secret:
+
+| Secret Name | Purpose | Contents | How It Is Created |
+| ----------- | ------- | -------- | ----------------- |
+| `proxy-authz-tokens` | Stores the tenant's access and refresh tokens used by the Authorization sidecar to authenticate with the Proxy Server. | Base64-encoded `access` and `refresh` JWT tokens. | Generated by `dellctl generate token` and applied via `kubectl apply -f token.yaml -n [CSI_DRIVER_NAMESPACE]`. Alternatively, the output of `dellctl generate token` can be piped directly to `kubectl`. |
+| `proxy-server-root-certificate` | Contains the Root CA certificate used to validate TLS connections between the CSI driver's Authorization sidecar and the CSM Authorization Proxy Server. | A PEM-encoded root CA certificate under the key `rootCertificate.pem`. | Created manually. See [proxy-server-root-certificate details](#proxy-server-root-certificate-details) below. |
+
+### proxy-server-root-certificate Details
+
+The `proxy-server-root-certificate` secret enables **secure TLS communication** between:
+- The **CSM Authorization sidecar** (running alongside the CSI driver)
+- The **CSM Authorization Proxy Server** (exposed via Ingress)
+
+**Where to get `rootCertificate.pem`:**
+
+- If CSM Authorization was installed with a **self-signed certificate** (via cert-manager), you can extract the CA certificate from the cert-manager CA secret (e.g., `karavi-selfsigned-tls`) in the `authorization` namespace.
+- If CSM Authorization was installed with **your own certificate**, provide the **Root CA certificate that signed it**. This is the root of the certificate chain that the Proxy Server's TLS certificate was issued from.
+- If running in **insecure mode** (not recommended for production), create the secret with empty data and set `skipCertificateValidation` to `true` in the driver configuration.
+
+**Relationship with `skipCertificateValidation`:**
+
+In the **driver secret** (e.g., `vxflexos-config`, `isilon-creds`, `powermax-creds`, `powerstore-config`), `skipCertificateValidation` **must always be set to `true`** when CSM Authorization is enabled. This is because the Authorization sidecar generates a self-signed certificate on the fly for the localhost endpoint, and the driver does not have this certificate.
+
+The `SKIP_CERTIFICATE_VALIDATION` setting in the CSI driver Helm values or CSM Operator CR is a **separate** setting that controls whether the Authorization sidecar validates the Proxy Server's TLS certificate:
+
+- `SKIP_CERTIFICATE_VALIDATION: true` causes the sidecar to **skip TLS verification** of the Proxy Server's certificate. In this case, the `proxy-server-root-certificate` secret can be empty.
+- `SKIP_CERTIFICATE_VALIDATION: false` requires the `proxy-server-root-certificate` secret to contain a valid Root CA so the sidecar can verify the Proxy Server's TLS certificate.
+
+> **Note**: The `skipCertificateValidation` parameter in the **Storage CR** (under `spec.skipCertificateValidation`) is yet another separate setting that controls certificate validation between the **Proxy Server and the backend storage array**.
+
+## Why CSI Driver Configuration Is Still Required (Step 6)
+
+Even when using the CSM Authorization proxy, the CSI driver still requires configuration (Helm values or CSM Operator CR, and Kubernetes secrets) because:
+
+1. **Endpoint redirection**: The driver must be configured to send requests to `https://localhost:<port>` (e.g., `https://localhost:9400`) instead of the actual storage array endpoint. This localhost address is where the Authorization sidecar listens and handles requests.
+2. **Sidecar enablement**: The driver's Helm values or CSM Operator CR must enable the Authorization module and specify the sidecar image, proxy host, and certificate validation settings.
+3. **Reverse proxy configuration** (PowerMax only): PowerMax requires the CSI Reverse Proxy to be configured as a sidecar and pointed to the localhost endpoint.
+
+The driver credentials in the driver secret (e.g., `username`/`password`) are read from a Kubernetes secret or secret provider class resource. When Authorization is enabled, the Proxy Server uses its own credentials to communicate with the storage array. However, the driver credential fields must still be present in the configuration (they can be left as placeholder values).
+
+### PowerMax Traffic Flow
+
+For PowerMax, the request path includes an additional CSI Reverse Proxy component:
+
+```
+CSI Driver → CSI Reverse Proxy (sidecar) → Authorization Sidecar (localhost) → Authorization Proxy Server (via Ingress) → PowerMax Unisphere
+```
+
+- The **CSI Reverse Proxy** is deployed as a sidecar in the driver pod and forwards requests to a localhost endpoint where the Authorization Sidecar listens. The port is configurable and may differ per array in multi-array configurations.
+- The **Authorization Sidecar** (also in the driver pod) injects the JWT token and forwards the request to the Authorization Proxy Server.
+- The **Authorization Proxy Server** validates the token, applies RBAC/quota policies, and proxies the request to Unisphere using its own credentials.
+
+> **Note**: When Authorization is enabled, the `certSecret` field in the driver secret (used for direct Unisphere TLS validation without Authorization) is no longer relevant, since the driver communicates with the local sidecar, not directly with Unisphere. The `certSecret` value can be left empty or set to a placeholder.
+
+## Exposing the CSM Authorization Proxy Server
+
+The CSM Authorization Proxy Server is exposed outside the Kubernetes cluster via an Ingress controller.
+
+- **Kubernetes**: Both the Helm chart and CSM Operator support deploying an optional NGINX Ingress Controller. In Helm, set `nginx.enabled: true` (or `false` if you already have an Ingress Controller). In the CSM Operator, configure the corresponding NGINX component in the Custom Resource.
+- **OpenShift**: OpenShift provides a default Ingress controller (OpenShift Router). Set the `openshift` parameter accordingly in the Helm chart or CSM Operator Custom Resource.
+
+**Additional notes:**
+
+- **cert-manager**: The Helm chart can optionally deploy cert-manager to generate a self-signed TLS certificate for the Authorization Ingress. If cert-manager is already installed and managed separately in your cluster, set `cert-manager.enabled: false` in the Helm values.
+- **Hostname resolution**: The default hostname `csm-authorization.com` is a **placeholder** and must be replaced with a real hostname that is resolvable via **DNS** from the CSI driver pods. Configure your organization's DNS to resolve the chosen hostname to the Ingress controller's external IP address.
+- **`proxyHost`**: The `proxyHost` value configured in the CSI driver Helm values or CSM Operator CR is the hostname of the Authorization Proxy Server. The Authorization sidecar reads this value to know where to forward requests. This is **not** the same as the `endpoint` field in the driver secret, which is always `https://localhost:<port>` (the sidecar's local listener).
 
 ## Container Storage Modules for Authorization Capabilities
 {{<table "table table-striped table-bordered table-sm">}}

content/docs/concepts/authorization/v2.x/configuration/powerflex/_index.md

@@ -21,6 +21,17 @@ Given a setup where Kubernetes, a storage system, and the Authorization Proxy Se
 
 2. Create the proxy-server-root-certificate secret.
 
+    > **Purpose**: This secret contains the Root CA certificate used to establish **secure TLS communication between the Authorization sidecar** (running alongside the CSI driver) **and the CSM Authorization Proxy Server** (exposed via Ingress). It is **not** related to the TLS certificate of the backend storage array.
+
+    **Where to get `rootCertificate.pem`:**
+    - If CSM Authorization was installed with a **self-signed certificate** (via cert-manager), extract the CA certificate from the cert-manager CA secret (e.g., `karavi-selfsigned-tls`) in the `authorization` namespace.
+    - If CSM Authorization was installed with **your own certificate**, provide the **Root CA certificate that signed it** (the root of the certificate chain that the Proxy Server's TLS certificate was issued from).
+    - If running in **insecure mode** (not recommended for production), create the secret with empty data and set `skipCertificateValidation` to `true` in the driver configuration.
+
+    **Relationship with `skipCertificateValidation`:**
+    - When `SKIP_CERTIFICATE_VALIDATION` is set to `true` in the Authorization sidecar configuration, the sidecar skips TLS verification of the Proxy Server, and this secret can be empty.
+    - When `SKIP_CERTIFICATE_VALIDATION` is set to `false`, this secret must contain a valid Root CA certificate.
+
     If running in *insecure* mode, create the secret with empty data:
 
       ```bash

content/docs/concepts/authorization/v2.x/configuration/powermax/_index.md

@@ -22,6 +22,17 @@ Given a setup where Kubernetes, a storage system, and the Container Storage Modu
 
 2. Create the proxy-server-root-certificate secret.
 
+    > **Purpose**: This secret contains the Root CA certificate used to establish **secure TLS communication between the Authorization sidecar** (running alongside the CSI driver) **and the CSM Authorization Proxy Server** (exposed via Ingress). It is **not** related to the TLS certificate of the backend storage array.
+
+    **Where to get `rootCertificate.pem`:**
+    - If CSM Authorization was installed with a **self-signed certificate** (via cert-manager), extract the CA certificate from the cert-manager CA secret (e.g., `karavi-selfsigned-tls`) in the `authorization` namespace.
+    - If CSM Authorization was installed with **your own certificate**, provide the **Root CA certificate that signed it** (the root of the certificate chain that the Proxy Server's TLS certificate was issued from).
+    - If running in **insecure mode** (not recommended for production), create the secret with empty data and set `skipCertificateValidation` to `true` in the driver configuration.
+
+    **Relationship with `skipCertificateValidation`:**
+    - When `SKIP_CERTIFICATE_VALIDATION` is set to `true` in the Authorization sidecar configuration, the sidecar skips TLS verification of the Proxy Server, and this secret can be empty.
+    - When `SKIP_CERTIFICATE_VALIDATION` is set to `false`, this secret must contain a valid Root CA certificate.
+
     If running in *insecure* mode, create the secret with empty data:
 
       ```bash

content/docs/concepts/authorization/v2.x/configuration/powerscale/_index.md

@@ -22,6 +22,17 @@ Given a setup where Kubernetes, a storage system, and the Container Storage Modu
 
 2. Create the proxy-server-root-certificate secret.
 
+    > **Purpose**: This secret contains the Root CA certificate used to establish **secure TLS communication between the Authorization sidecar** (running alongside the CSI driver) **and the CSM Authorization Proxy Server** (exposed via Ingress). It is **not** related to the TLS certificate of the backend storage array.
+
+    **Where to get `rootCertificate.pem`:**
+    - If CSM Authorization was installed with a **self-signed certificate** (via cert-manager), extract the CA certificate from the cert-manager CA secret (e.g., `karavi-selfsigned-tls`) in the `authorization` namespace.
+    - If CSM Authorization was installed with **your own certificate**, provide the **Root CA certificate that signed it** (the root of the certificate chain that the Proxy Server's TLS certificate was issued from).
+    - If running in **insecure mode** (not recommended for production), create the secret with empty data and set `skipCertificateValidation` to `true` in the driver configuration.
+
+    **Relationship with `skipCertificateValidation`:**
+    - When `SKIP_CERTIFICATE_VALIDATION` is set to `true` in the Authorization sidecar configuration, the sidecar skips TLS verification of the Proxy Server, and this secret can be empty.
+    - When `SKIP_CERTIFICATE_VALIDATION` is set to `false`, this secret must contain a valid Root CA certificate.
+
     If running in *insecure* mode, create the secret with empty data:
 
       ```bash