Kubernetes: Five ways to tell you a secret

Saturday, April 18, 2020

GitOps requires all Kubernetes manifests in git, but how exactly do you manage secrets? There are plenty solutions available which makes getting started with security a hassle. Below is an overview of five tools to help you make an informed decision about keeping your secrets a secret.

Sealed Secrets. A Kubernetes operator which introduces a custom resource definition called sealed secret. A new secret can be created by letting kubeseal take a regular Kubernetes secret and encrypting it. Afterwards, the sealed secret manifest can simply be deployed to a cluster. Figure 1 illustrates this flow in detail. Sealed secrets is my personal favourite as it is the complete package: a simple tool that solves secret management in a Kubernetes-native way that is agnostic to cloud providers.

Figure 1: Sealed secrets overview
Figure 1: Sealed secrets overview

Standard Kubernetes secrets. The secret manifest as provided by Kubernetes, nothing more, nothing less. Although discouraged, standard Kubernetes secrets are possible with private repositories that have strict access control. Be warned that you often need third-party integrations, which can now easily get your secrets disclosed. Even if, for some reason, you decide to trust the third-party, you have no control over keeping attackers out of their systems. The other side of the coin is the unmatched simplicity.

Secret Operations (SOPS). A tool which conveniently delegates encryption and decryption to a cloud provider of choice, such as Google or Amazon's key management service. These services are designed with robust security in mind. For example, leaking private keys is nearly impossible. Unfortunately, the secret has to be decrypted before applying it to the cluster. This adds complications when integrating with GitOps operators. The dependency on a cloud provider also means you are no longer cloud agnostic when using this tool.

Vault. The enterprise solution for Kubernetes secrets. Definitely check it out if you want detailed audit logs and tight access control. Be cautious as even the documentation mentions that vault is a complex system. It is a mistake to believe that you don't need to understand how a system works, as with any system you will run into issues which you need to troubleshoot.

Cloud-based Secret Manager. This tool protects secrets by moving them out the git repository. These services allows you to store, manage, and access secrets as binary blobs or text strings. With the appropriate permissions, your application can retrieve the contents of the secret during runtime. It supports versioned secrets and is an alternative to storing secrets as an environment variable. For starters, you need a cloud provider or this approach will not work. Security is increased as the cluster only keeps the secret in-memory at the cost of no longer having the complete overview by having all configuration in a single commit.

My call to action is pretty straight forward: Decide on a tool and get hands-on with your production-ready Kubernetes clusters. Be sure to check out the official homepages of Sealed Secrets, Kubernetes secrets, SOPS, Vault or Secret Manager.

A word of caution: These tools only solve the problem of securely storing secrets in a repository. Once deployed to the cluster, they can be accessed as standard Kubernetes secrets. Be sure to correctly set strict access and rbac in the Kubernetes cluster.

Sign up for the newsletter and I'll email you a fresh batch of insights once every while.