Container Storage Interface (CSI) Secrets Store Driver

Prior to the introduction of the Container Storage Interface (CSI), Kubernetes included a powerful volume plugin mechanism. It did, however, provide certain challenges. These “in-tree” plugins were integrated into the main Kubernetes source, thus making it cumbersome for vendors to add or amend storage plugins without adhering to the Kubernetes release schedule. This not only restricted flexibility, but also raised concerns regarding the core Kubernetes system’s reliability and security. Additionally, maintaining this “in-tree” code was often a challenging endeavor for Kubernetes maintainers.

The Container Storage Interface (CSI) was developed as a response to these challenges. This standard provides a consistent method to integrate arbitrary block and file storage systems with containerized workloads across various Container Orchestration Systems, including Kubernetes, Mesos, and Cloud Foundry. The beauty of CSI lies in its extensibility. Vendors are now able to seamlessly integrate their systems with Kubernetes without tampering with its core code. This enhancement conveys end users with a wider array of storage choices while boosting the overall reliability and security of the system.

By the time Kubernetes version v1.13 was released, the CSI implementation was upgraded to General Availability (GA). For anyone interested in further exploring these, a comprehensive list of CSI drivers is accessible here.

Furthermore, this separation enables storage vendors to maintain jurisdiction over their release and feature cycles. They may focus on API development without having to worry about backward incompatibility issues. Supporting their plugin is as simple as deploying a few pods.

Vault CSI Driver

The Secrets Store CSI Driver project for Vault Provider began with a simple discussion on GitHub to determine how much interest there was in utilizing CSI to reveal secrets in a Kubernetes pod’s volume.

The Secrets Store CSI driver interacts with the Vault CSI provider via gRPC to retrieve secret data. This driver enables us to integrate numerous secrets, keys, and certificates from Vault, making them available as a volume in our pods. It makes use of a custom resource definition (CRD) named SecretProviderClass, which identifies Vault as the source and provides configuration settings for both Vault and the paths to your secrets.

When a pod is started, it uses the Kubernetes Auth Method for authentication against the service account identity specified in the pod blueprint. Once authenticated, the secret paths defined in the SecretProviderClass are retrieved and written to a tmpfs volume, and subsequently mounted to the pod. This configuration makes it possible for the applications to access and use the secrets from the attached volume as needed.

Hands-on Time

Prerequisites

Vault Helm Chart

Installing the Vault Helm chart with the injector service disabled and CSI enabled:

$ helm -n vault get values vault | grep -A1 -E '^(csi|injector)' 
csi:
  enabled: true
--
injector:
  enabled: false

Vault kvv2 Secrets Engine

We’ve already enabled a kvv2 Secrets Engine at the unfriendlygrinch path in the previous blog entry and created a secret:

$ vault kv get unfriendlygrinch/app/config 
========== Secret Path ==========
unfriendlygrinch/data/app/config

======= Metadata =======
Key                Value
---                -----
created_time       2023-07-31T14:57:00.611356951Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    dretda1rus

Kubernetes Auth Method

Secrets Store CSI Driver

Installing the Secrets Store CSI Driver secrets-store.csi.k8s.io using Helm:

$ helm -n vault ls
NAME                    	NAMESPACE	REVISION	UPDATED                                 	STATUS  	CHART                         	APP VERSION
secrets-store-csi-driver	vault    	1       	2023-08-15 18:55:39.838682762 +0300 EEST	deployed	secrets-store-csi-driver-1.3.3	1.3.3      

For the moment, the installation uses the default values:

$ helm -n vault get values secrets-store-csi-driver 
USER-SUPPLIED VALUES:
null

The SecretProviderClass Resource

The SecretProviderClass is a Kubernetes custom resource that is used to provide driver configurations and provider-specific parameters to the CSI driver. A SecretProviderClass resource for the Vault provider would typically look like this:

kind: SecretProviderClass
metadata:
  name: vault
  namespace: unfriendlygrinch
spec:
  provider: vault
  parameters:
    roleName: "unfriendlygrinch"
    vaultAddress: "http://<MY_VAULT_ADDR>"
    objects: |
      - objectName: "password"
        secretPath: "unfriendlygrinch/data/app/config"
        secretKey: "password"

Whereas:

• provider: the name of the secrets store. For HashiCorp Vault, this is vault.
• vaultAddress: the URL of the Vault server.
• roleName: The Vault role the driver will use.
• objects: An array of secrets to retrieve from Vault. Each object has:
  • objectName: An alias used within the SecretProviderClass to refer to a specific secret. The name of the file (inside the pod) that contains the secret.
  • secretPath: The path in Vault where that secret is stored.
  • secretKey: The secret key to extract the value from.

After creating the SecretProviderClass, we can reference it the pod’s definition using a volume of csi type. When the pod starts, the driver ensures that the secrets defined by the SecretProviderClass are retrieved from Vault and written to the pod’s filesystem.

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: nginx
  name: nginx
  namespace: unfriendlygrinch
spec:
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    - name: secrets-store
      mountPath: "/mnt/secrets-store"
      readOnly: true
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  serviceAccountName: unfriendlygrinch
  volumes:
  - name: secrets-store
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: vault

$ k -n unfriendlygrinch exec -it nginx -- cat /mnt/secrets-store/password && echo 
dretda1rus

Secrets Store CSI Driver: Sync as Kubernetes secret

The Secrets Store CSI Driver also provides the option to sync these secrets to native Kubernetes secrets, allowing them to be managed and accessed using Kubernetes-native methods.

This is not enabled by default, thus we need to explicitly do so:

$ helm -n vault get values secrets-store-csi-driver 
USER-SUPPLIED VALUES:
syncSecret:
  enabled: "true"

Let’s now adjust the previous example in order to further explore the Sync as Kubernetes secret functionality.

The updated SecretProviderClass resource would look like:

kind: SecretProviderClass
metadata:
  name: vault
  namespace: unfriendlygrinch
spec:
  provider: vault
  parameters:
    roleName: "unfriendlygrinch"
    vaultAddress: "http://<MY_VAULT_ADDR>"
    objects: |
      - objectName: "password"
        secretPath: "unfriendlygrinch/data/app/config"
        secretKey: "password"
  secretObjects:
  - data:
    - key: password
      objectName: password
    secretName: my-password
    type: Opaque

As you may notice in the above example, we employed the optional secretObjects parameter to specify the intended state of the synchronized Kubernetes secret objects:

• data is an array consisting of key, namely the data field to populate, and objectName, the name of the object to sync.
• secretName: the name of the Kubernetes secret object.
• type: the type of Kubernetes secret object.

As such, we are able to create Kubernetes Secrets to mirror the mounted content.

It’s worth noting that secrets will only sync after we start a pod mounting them. Thus, relying solely on the synchronization with Kubernetes secrets functionality does not work. When all of the pods that consume the secret are deleted, the Kubernetes secret is also removed. Next we only need to use the Kubernetes secret in an environment variable in the pod. Well, that’s noting fancy, right?

$ k -n unfriendlygrinch exec -it nginx -- env | grep ^MY
MY_PASSWORD=dretda1rus

Conclusions

The Vault CSI Driver is deployed as a daemonset and uses the Secret Provider Class specified and the pod’s service account to retrieve the secrets from Vault and mount them into the pod’s CSI volume. It’s compatible solely with the Kubernetes auth method.

The Vault CSI driver offers the capability to render Vault secrets into Kubernetes secrets as well as environment variables. However, if Secret synchronization is not absolutely necessary, it’s recommended to disable it. Having secrets accessible both in Vault and within etcd defeats the purpose of using Vault in the first place, as secrets are still stored as base64 encoded data in etcd. Given this scenario, it’s wise to think about encrypting the stored data using one of the various available service providers. It’s important to note, however, that doing so may introduce additional overhead.