I recently had the incredible opportunity to speak at Hashidays Paris. My session focused on how Vault Agent can be used to harden Kubernetes clusters. This event served as an opportunity to discuss the current challenges in cloud-native security and explore the potential of Vault agent to help organizations protect their systems. I’m passionate about how Vault and Kubernetes can be used together, and believe that it is a topic that deserves more attention. With that in mind, I decided to create a blog series dedicated to this very subject. It will explore the various ways that Vault and Kubernetes can be used together, as well as the advantages of each approach, and I think it will be a great resource for anyone looking to better understand the integration of these two powerful tools. I’m looking forward to sharing my thoughts and insights on the subject - stay tuned!

Let’s start our journey! Secrets in Kubernetes maintain sensitive information such as OAuth tokens, passwords, and Secure Shell (SSH) keys. By default, these are stored in Kubernetes as unencrypted base64-encoded strings that can be retrieved by anyone with API access. Secrets can be encrypted by enabling data-at-rest encryption on the API server or by leveraging an external key management service (KMS), which may be provided by a cloud provider.

Enabling and configuring encryption of secret data at-rest is not enough; we must also ensure data is protected in-transit…

HashiCorp Vault, a tool for securely accessing secrets, and Kubernetes, an open-source platform designed to automate deploying, scaling, and managing containerized applications, have become integral parts of the DevOps Landscape. They address key requirements in our ever-changing world: security and scalability.

This is a journey in which we are going to explore the point of convergence of HashiCorp Vault and Kubernetes.

Kubernetes Adoption

Kubernetes has emerged as a key technology for DevOps teams, supporting the adoption of cloud-native architecture and microservices. The use of Kubernetes has expanded beyond container orchestration to include storage, networking, and security. Kubernetes has become a critical technology for cloud-native security, with Kubernetes-native security tools gaining in popularity. (Gartner’s 2022 DevOps Hype Cycle)

Over the past years, Kubernetes has seen significant adoption as more organizations have recognized the benefits of containerization and container orchestration.

Gartner’s 2022 DevOps Hype Cycle report predicts Kubernetes will continue to be essential for DevOps teams, driven by increased adoption of cloud-native architecture and microservices. It also highlights its emerging role in cloud-native security, with Kubernetes-native security tools gaining traction.

The State of Kubernetes 2023 report points out that 60% of the survey respondents agree that Kubernetes enhances their developers’ productivity.

Simultaneously, there’s a rising emphasis on security within containerized applications. According to Red Hat’s State of Kubernetes Security Report 2023, 38% of the respondents perceive the amount of security investment in containerized operations to be insufficient, a figure up by 7% from 2022. Despite the steady growth in Kubernetes adoption, it appears that security efforts are not keeping up, signaling a prospective area of concern.

Given these studies and predictions, it’s hardly surprising that Kubernetes’ adoption rates continue to rise, indicating its escalating importance in modern software development.

Kubernetes Secrets – Not really secret!

Kubernetes secrets provide a solution for storing sensitive data like passwords, API keys, and certificates, and make it available to containers running in a Kubernetes cluster. While they server as a valuable approach for managing sensitive information, it is important to consider their limitations:

1. Limited access control: Kubernetes secrets lack fine-grained access control mechanisms. Once a user has access to a Kubernetes cluster, they can access all stored secrets. Consequently, anyone with access to the cluster may potentially retrieve sensitive information contained within secrets.

2. Limited encryption options: By default, secrets are stored unencrypted in etcd, but can be configured to be encrypted at rest. This means that data is encrypted while stored in etcd and is only decrypted when accessed via a Kubernetes API request. While there are sone encryption providers available, such as IdentityConfiguration, AESConfiguration, KMSConfiguration, SecretboxConfiguration, the use of any of these may not meet our security requirements. Furthermore, enabling data encryption at-rest in Kubernetes is relatively complex; this process requires several actions to be taken by the Kubernetes cluster administrator.

3. Limited key rotation: Kubernetes secrets are not designed for frequent key rotation. Updating a secret requires restarting all pods that use it, which might cause service outages.

4. Limited audit trail: Kubernetes secrets do not provide a comprehensive audit trail that records access. This makes it difficult to identify potential security breaches or unauthorized access.

Overall, Kubernetes secrets are a useful tool for managing sensitive information within a Kubernetes cluster. However, their limitations should be taken into account when designing a security strategy for applications based on Kubernetes.

Why HashiCorp Vault?

HashiCorp Vault is an open-source tool designed to securely store and manage sensitive information, such as passwords, encryption keys, certificates, API keys, and nonetheless Kubernetes secrets It provides a centralized platform for secrets management, access control, and encryption across different applications, systems, and infrastructure. Here are a few reasons that make Vault an advantageous choice for managing Kubernetes secrets:

1. Centralized secrets management: Vault provides a centralized location for managing secrets, which Kubernetes pods and other applications can use. This streamlines secrets management and reduces the likelihood of secrets being stored in multiple places or managed independently by different teams.

2. Fine-grained access control: Vault includes a versatile access control mechanism that may be used to restrict access to secrets on an individual or application-specific basis.

3. Dynamic Secrets: Vault is capable of generating short-lived, dynamically created secrets. This feature contributes to mitigating the risk of secrets being breached, given their limited validity period.

4. Encryption: Vault encrypts data in transit and at rest across clouds and datacenters. Consequently, the security of Kubernetes secrets is improved.

5. Audit logging: Vault comes with exhaustive audit logging, which may be used to track who has accessed secrets and when. This increases awareness of potential security breaches or unauthorized access.

The Convergence of HashiCorp Vault and Kubernetes

From the earliest days, HashiCorp has committed to make Kubernetes a first-class supported experience. In terms of Vault, its objective was to offer a range of solutions for efficiently integrating HashiCorp Vault and Kubernetes, aiming to securely inject secrets into application stacks.

Applications have the flexibility to interact directly with the Vault API, use a Kubernetes service account for Vault authentication, or employ Kubernetes Sidecar Injection for introducing secrets from Vault into applications that don’t have inherent Vault logic.

In 2017, Vault version 0.8.3 introduced a native Kubernetes auth backend, empowering Kubernetes pods to receive and use Vault auth tokens directly, bypassing the need for additional integration elements.

Following that, in 2019, the official Helm Chart for Vault has been released, and shortly after, a Helm chart specifically for Vault Enterprise. This allowed practitioners to start a Vault cluster on Kubernetes in a matter of minutes, significantly simplifying the task of operating Vault on Kubernetes and offering a streamlined deployment process.

Moreover, HashiCorp has collaborated with Microsoft and the Kubernetes SIG community to introduce the Kubernetes Secrets Store Container Storage Interface (CSI) driver. This driver facilitates the use of Vault to securely transfer secrets into a Kubernetes environment via the Container Storage Interface.

It’s crucial to note that currently, there are three distinct integrations designed to help Kubernetes workloads smoothly consume secrets from Vault, eliminating the need for the application to engage directly with Vault. These are:

1. Vault CSI Provider
2. Agent Injector
3. Vault Secrets Operator

Before unlocking the potential of data stored in Vault, it is critical to understand how to access it directly through Vault’s HTTP API. This will provide you with a more complete understanding of the context surrounding the content stored in Vault and unlock the value found within it. The API enables secure, reliable access to data stored within Vault, making it easier to create applications that leverage the skills your teams already has in terms of programming languages to handle the data stored inside. With the API, developers can access data directly without relying on integration, and build advanced applications that utilize the data in more meaningful ways. For me, it was an exciting opportunity to explore the possibilities that Vault’s HTTP API unlocks, and I hope you feel the same way.

But, first things first, let’s briefly introduce the integrations:

The Vault CSI Provider

The Vault CSI Provider is a Container Storage Interface (CSI) plugin that facilitates the smooth provisioning and management of storage volumes from Vault in a Kubernetes environment. It acts as a bridge between Kubernetes and Vault, allowing Kubernetes pods to use Vault as a storage backend.

The Vault CSI Provider works by authenticating with Vault using the Kubernetes authentication method and then creating and managing Kubernetes persistent volumes (PVs) and persistent volume claims (PVCs) backed by Vault. As standard Kubernetes resources, these PVs and PVCs are readily accessible to Kubernetes pods.

The Vault Agent Sidecar Injector

The Vault Sidecar Agent Injector uses the sidecar design pattern to modify pod definitions, integrating a Vault Agent container that processes Vault secrets onto a shared memory volume. This allows containers in the pod to access Vault secrets, even without explicit Vault knowledge. The injector functions as a Kubernetes mutating webhook controller. The controller intercepts pod events and applies mutations to the pod if annotations exist within the request.

In this context, the the sidecar pattern is a key concept. It involves deploying components of an application into a separate process or container to provide a centralized service. This technique is extremely handy in circumstances like this, where we want the main application (in this example, the pods) to be able to access a service (Vault) without having to instruct it expressly. This separation of concerns improves application architecture modularity and adaptability.

The Vault Secrets Operator

The Vault Secrets Operator, a novel approach for integration between Vault and Kubernetes, was officially launched in June 2023 during HashiDays.

The operator handles the synchronization of various types of secrets such as static, dynamic, and PKI-based, originating from one or multiple Vault server instances. Moreover, it has the ability to trigger actions post-rotation, for instance, directly notifying an application by starting a rolling update of a deployment.

The Vault Secrets Operator includes a controller that employs Custom Resource Definitions (CRDs) in Kubernetes. These resources manage the secrets required by various applications. Vault then manages these secrets, which are orchestrated in Kubernetes via custom resources and used by applications in a manner that’s consistent with Kubernetes’ native practices, thus ensuring a distinct separation of concerns.

Conclusions

HashiCorp Vault and Kubernetes are two key cornerstones of modern development and operations. With the steady increase in Kubernetes usage, it has become of paramount importance to acknowledge the security implications associated with running applications in containerized environments, one of which involves Secrets Management.

Moving forward, we are going to dive deeper into the previously outlined strategies, starting with Vault’s HTTP API and then continue with the integrations: Vault CSI Provider, Agent Injector, and Vault Secrets Operator. And we’ll do this by engaging in hands-on experiences with every approach, or, in other words, by getting our hands dirty.