Use the Azure Key Vault Provider for Secrets Store CSI Driver for Azure Kubernetes Service secrets - Azure Kubernetes Service (2023)

  • Article
  • 10 minutes to read

The Azure Key Vault Provider for Secrets Store CSI Driver allows for the integration of an Azure key vault as a secret store with an Azure Kubernetes Service (AKS) cluster via a CSI volume.

Limitations

  • A container using subPath volume mount will not receive secret updates when it is rotated. See

Prerequisites

  • If you don't have an Azure subscription, create a free account before you begin.
  • Before you start, ensure that your version of the Azure CLI is 2.30.0 or later. If it's an earlier version, install the latest version.
  • If restricting Ingress to the cluster, ensure Ports 9808 and 8095 are open.

Supported Kubernetes versions

The minimum recommended Kubernetes version is based on the rolling Kubernetes version support window. Ensure that you're running version N-2 or later.

Features

  • Mounts secrets, keys, and certificates to a pod by using a CSI volume
  • Supports CSI inline volumes
  • Supports mounting multiple secrets store objects as a single volume
  • Supports pod portability with the SecretProviderClass CRD
  • Supports Windows containers
  • Syncs with Kubernetes secrets
  • Supports auto rotation of mounted contents and synced Kubernetes secrets

Create an AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver support

First, create an Azure resource group:

az group create -n myResourceGroup -l eastus2

To create an AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver capability, use the az aks create command with the azure-keyvault-secrets-provider add-on.

az aks create -n myAKSCluster -g myResourceGroup --enable-addons azure-keyvault-secrets-provider --enable-managed-identity

A user-assigned managed identity, named azurekeyvaultsecretsprovider-*, is created by the add-on for the purpose of accessing Azure resources. The following example uses this identity to connect to the Azure key vault where the secrets will be stored, but you can also use other identity access methods. Take note of the identity's clientId in the output:

..., "addonProfiles": { "azureKeyvaultSecretsProvider": { ..., "identity": { "clientId": "<client-id>", ... } }

Upgrade an existing AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver support

To upgrade an existing AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver capability, use the az aks enable-addons command with the azure-keyvault-secrets-provider add-on:

az aks enable-addons --addons azure-keyvault-secrets-provider --name myAKSCluster --resource-group myResourceGroup

As mentioned in the preceding section, the add-on creates a user-assigned managed identity that you can use to authenticate to your Azure key vault.

Verify the Azure Key Vault Provider for Secrets Store CSI Driver installation

The preceding command installs the Secrets Store CSI Driver and the Azure Key Vault Provider on your nodes. Verify that the installation is finished by listing all pods that have the secrets-store-csi-driver and secrets-store-provider-azure labels in the kube-system namespace, and ensure that your output looks similar to the output shown here:

kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver, secrets-store-provider-azure)'NAME READY STATUS RESTARTS AGEaks-secrets-store-csi-driver-4vpkj 3/3 Running 2 4m25saks-secrets-store-csi-driver-ctjq6 3/3 Running 2 4m21saks-secrets-store-csi-driver-tlvlq 3/3 Running 2 4m24saks-secrets-store-provider-azure-5p4nb 1/1 Running 0 4m21saks-secrets-store-provider-azure-6pqmv 1/1 Running 0 4m24saks-secrets-store-provider-azure-f5qlm 1/1 Running 0 4m25s

Be sure that a Secrets Store CSI Driver pod and a Secrets Store Provider Azure pod are running on each node in your cluster's node pools.

Create or use an existing Azure key vault

In addition to an AKS cluster, you'll need an Azure key vault resource that stores the secret content. Keep in mind that the key vault's name must be globally unique.

az keyvault create -n <keyvault-name> -g myResourceGroup -l eastus2

Your Azure key vault can store keys, secrets, and certificates. In this example, you'll set a plain-text secret called ExampleSecret:

az keyvault secret set --vault-name <keyvault-name> -n ExampleSecret --value MyAKSExampleSecret

Take note of the following properties for use in the next section:

  • The name of the secret object in the key vault
  • The object type (secret, key, or certificate)
  • The name of your Azure key vault resource
  • The Azure tenant ID that the subscription belongs to

Provide an identity to access the Azure key vault

The Secrets Store CSI Driver allows for the following methods to access an Azure key vault:

  • An Azure Active Directory pod identity (preview)
  • An Azure Active Directory workload identity (preview)
  • A user-assigned or system-assigned managed identity

Follow the instructions in Provide an identity to access the Azure Key Vault Provider for Secrets Store CSI Driver for your chosen method.

Note

The rest of the examples on this page require that you've followed the instructions in Provide an identity to access the Azure Key Vault Provider for Secrets Store CSI Driver, chosen one of the identity methods, and configured a SecretProviderClass. Come back to this page after completed those steps.

Validate the secrets

After the pod starts, the mounted content at the volume path that you specified in your deployment YAML is available.

## show secrets held in secrets-storekubectl exec busybox-secrets-store-inline -- ls /mnt/secrets-store/## print a test secret 'ExampleSecret' held in secrets-storekubectl exec busybox-secrets-store-inline -- cat /mnt/secrets-store/ExampleSecret

Obtain certificates and keys

The Azure Key Vault design makes sharp distinctions between keys, secrets, and certificates. The Key Vault service’s certificates features were designed to make use of its key and secret capabilities. When a key vault certificate is created, an addressable key and secret are also created with the same name. The key allows key operations, and the secret allows the retrieval of the certificate value as a secret.

A key vault certificate also contains public x509 certificate metadata. The key vault stores both the public and private components of your certificate in a secret. You can obtain each individual component by specifying the objectType in SecretProviderClass. The following table shows which objects map to the various resources associated with your certificate:

ObjectReturn valueReturns entire certificate chain
keyThe public key, in Privacy Enhanced Mail (PEM) formatN/A
certThe certificate, in PEM formatNo
secretThe private key and certificate, in PEM formatYes

Disable the Azure Key Vault Provider for Secrets Store CSI Driver on an existing AKS cluster

Note

Before you disable the add-on, ensure that no SecretProviderClass is in use. Trying to disable the add-on while SecretProviderClass exists will result in an error.

To disable the Azure Key Vault Provider for Secrets Store CSI Driver capability in an existing cluster, use the az aks disable-addons command with the azure-keyvault-secrets-provider flag:

az aks disable-addons --addons azure-keyvault-secrets-provider -g myResourceGroup -n myAKSCluster

Note

If the add-on is disabled, existing workloads will have no issues and will not see any updates in the mounted secrets. If the pod restarts or a new pod is created as part of scale-up event, the pod will fail to start because the driver is no longer running.

Additional configuration options

Enable and disable autorotation

Note

When the Azure Key Vault Provider for Secrets Store CSI Driver is enabled, it updates the pod mount and the Kubernetes secret that's defined in the secretObjects field of SecretProviderClass. It does so by polling for changes periodically, based on the rotation poll interval you've defined. The default rotation poll interval is 2 minutes.

Note

When a secret is updated in an external secrets store after initial pod deployment, the Kubernetes Secret and the pod mount will be periodically updated depending on how the application consumes the secret data.

Mount the Kubernetes Secret as a volume: Use the auto rotation and Sync K8s secrets features of Secrets Store CSI Driver. The application will need to watch for changes from the mounted Kubernetes Secret volume. When the Kubernetes Secret is updated by the CSI Driver, the corresponding volume contents are automatically updated.

Application reads the data from the container’s filesystem: Use the rotation feature of Secrets Store CSI Driver. The application will need to watch for the file change from the volume mounted by the CSI driver.

Use the Kubernetes Secret for an environment variable: Restart the pod to get the latest secret as an environment variable.Use a tool such as Reloader to watch for changes on the synced Kubernetes Secret and perform rolling upgrades on pods.

To enable autorotation of secrets, use the enable-secret-rotation flag when you create your cluster:

az aks create -n myAKSCluster2 -g myResourceGroup --enable-addons azure-keyvault-secrets-provider --enable-secret-rotation

Or update an existing cluster with the add-on enabled:

az aks addon update -g myResourceGroup -n myAKSCluster2 -a azure-keyvault-secrets-provider --enable-secret-rotation

To specify a custom rotation interval, use the rotation-poll-interval flag:

az aks addon update -g myResourceGroup -n myAKSCluster2 -a azure-keyvault-secrets-provider --enable-secret-rotation --rotation-poll-interval 5m

To disable autorotation, first disable the addon. Then, re-enable the addon without the enable-secret-rotation flag.

Sync mounted content with a Kubernetes secret

You might sometimes want to create a Kubernetes secret to mirror the mounted content.

When you create a SecretProviderClass, use the secretObjects field to define the desired state of the Kubernetes secret, as shown in the following example.

Note

The YAML examples here are incomplete. You'll need to modify them to support your chosen method of access to your key vault identity. For details, see Provide an identity to access the Azure Key Vault Provider for Secrets Store CSI Driver.

The secrets will sync only after you start a pod to mount them. To rely solely on syncing with the Kubernetes secrets feature doesn't work. When all the pods that consume the secret are deleted, the Kubernetes secret is also deleted.

apiVersion: secrets-store.csi.x-k8s.io/v1kind: SecretProviderClassmetadata: name: azure-syncspec: provider: azure secretObjects: # [OPTIONAL] SecretObjects defines the desired state of synced Kubernetes secret objects - data: - key: username # data field to populate objectName: foo1 # name of the mounted content to sync; this could be the object name or the object alias secretName: foosecret # name of the Kubernetes secret object type: Opaque # type of Kubernetes secret object (for example, Opaque, kubernetes.io/tls)

Note

Make sure that the objectName in the secretObjects field matches the file name of the mounted content. If you use objectAlias instead, it should match the object alias.

Set an environment variable to reference Kubernetes secrets

After you've created the Kubernetes secret, you can reference it by setting an environment variable in your pod, as shown in the following example code:

Note

The example here demonstrates access to a secret through env variables and through volume/volumeMount. This is for illustrative purposes; a typical application would use one method or the other. However, be aware that in order for a secret to be available through env variables, it first must be mounted by at least one pod.

kind: PodapiVersion: v1metadata: name: busybox-secrets-store-inlinespec: containers: - name: busybox image: k8s.gcr.io/e2e-test-images/busybox:1.29-1 command: - "/bin/sleep" - "10000" volumeMounts: - name: secrets-store01-inline mountPath: "/mnt/secrets-store" readOnly: true env: - name: SECRET_USERNAME valueFrom: secretKeyRef: name: foosecret key: username volumes: - name: secrets-store01-inline csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "azure-sync"

Metrics

The Azure Key Vault Provider

Metrics are served via Prometheus from port 8898, but this port isn't exposed outside the pod by default. Access the metrics over localhost by using kubectl port-forward:

kubectl port-forward -n kube-system ds/aks-secrets-store-provider-azure 8898:8898 &curl localhost:8898/metrics

The following table lists the metrics that are provided by the Azure Key Vault Provider for Secrets Store CSI Driver:

MetricDescriptionTags
keyvault_requestThe distribution of how long it took to get from the key vaultos_type=<runtime os>, provider=azure, object_name=<keyvault object name>, object_type=<keyvault object type>, error=<error if failed>
grpc_requestThe distribution of how long it took for the gRPC requestsos_type=<runtime os>, provider=azure, grpc_method=<rpc full method>, grpc_code=<grpc status code>, grpc_message=<grpc status message>

The Secrets Store CSI Driver

Metrics are served from port 8095, but this port is not exposed outside the pod by default. Access the metrics over localhost by using kubectl port-forward:

kubectl port-forward -n kube-system ds/aks-secrets-store-csi-driver 8095:8095 &curl localhost:8095/metrics

The following table lists the metrics provided by the Secrets Store CSI Driver:

MetricDescriptionTags
total_node_publishThe total number of successful volume mount requestsos_type=<runtime os>, provider=<provider name>
total_node_unpublishThe total number of successful volume unmount requestsos_type=<runtime os>
total_node_publish_errorThe total number of errors with volume mount requestsos_type=<runtime os>, provider=<provider name>, error_type=<error code>
total_node_unpublish_errorThe total number of errors with volume unmount requestsos_type=<runtime os>
total_sync_k8s_secretThe total number of Kubernetes secrets syncedos_type=<runtime os, provider=<provider name>
sync_k8s_secret_duration_secThe distribution of how long it took to sync the Kubernetes secretos_type=<runtime os>
total_rotation_reconcileThe total number of rotation reconcilesos_type=<runtime os>, rotated=<true or false>
total_rotation_reconcile_errorThe total number of rotation reconciles with erroros_type=<runtime os>, rotated=<true or false>, error_type=<error code>
total_rotation_reconcile_errorThe distribution of how long it took to rotate secrets-store content for podsos_type=<runtime os>

Troubleshooting

You can find generic troubleshooting steps for the Azure Key Vault Provider for Secrets Store CSI Driver here

Next steps

Now that you've learned how to use the Azure Key Vault Provider for Secrets Store CSI Driver with an AKS cluster, see Enable CSI drivers for Azure Disks and Azure Files on AKS.

Top Articles
Latest Posts
Article information

Author: Nicola Considine CPA

Last Updated: 01/24/2023

Views: 5916

Rating: 4.9 / 5 (69 voted)

Reviews: 92% of readers found this page helpful

Author information

Name: Nicola Considine CPA

Birthday: 1993-02-26

Address: 3809 Clinton Inlet, East Aleisha, UT 46318-2392

Phone: +2681424145499

Job: Government Technician

Hobby: Calligraphy, Lego building, Worldbuilding, Shooting, Bird watching, Shopping, Cooking

Introduction: My name is Nicola Considine CPA, I am a determined, witty, powerful, brainy, open, smiling, proud person who loves writing and wants to share my knowledge and understanding with you.