Kubernetes Secrets Management Using Kubernetes Sealed Secrets

2024-05-23 857 words 5 minutes

Contents

Security is a major concern in continuous integration (CI), especially when managing sensitive information like API keys, passwords, and other secrets. For Kubernetes resources, Sealed Secrets offer an effective solution for securely managing sensitive information within your repository. In this blog we will explore what are Sealed Secrets, how to use them and some common management tasks around Sealed Secrets.

What are Sealed Secrets?

Sealed Secrets is a set of Kubernetes resources, controller and custom resource definition, that enables secure storage of secrets in your version control system (VCS). Unlike standard Kubernetes Secrets, which are base64-encoded and easily decoded, Sealed Secrets use asymmetric encryption to ensure your secrets remain encrypted until deployed to your Kubernetes cluster. Once deployed, the Sealed Secrets controller decrypts them into regular Kubernetes secrets, making them accessible to your applications.

Key Benefits of Sealed Secrets

Security: Encrypted secrets can only be decrypted by the controller running in your Kubernetes cluster.
Version Control: Encrypted secrets can be safely stored in your VCS.
Automation: Simplifies the management of secrets in CI/CD pipelines.

Working with Sealed Secrets

Sealed Secrets leverage public and private key pairs. The public key, used by the kubeseal utility, encrypts the secrets and creates the corresponding Sealed Secret.

Note that Sealed Secrets are not a substitute for proper access control in the cluster. Once decrypted by the controller, anyone with appropriate access can view the secret’s content.

If the cluster is compromised or deleted, decrypting Sealed Secrets may become impossible. Later, we will discuss a proper key backup strategy to remedy this problem.

High-Level Overview of Working with Sealed Secrets:

Encrypt Secrets: Admin uses the kubeseal CLI tool to encrypt the secrets.
Store Encrypted Secrets: Admin sends encrypted secrets, known as Sealed Secrets, to the developers, and they are stored in the repository.
Deploy Sealed Secrets: Sealed Secrets are deployed like any other Kubernetes object as part of your CI/CD workflow.

Installing Sealed Secrets

Before starting, ensure you have:

A running Kubernetes cluster with proper access.
kubectl installed and configured.
Helm installed and configured.

Install the kubeseal utility using your package manager. Next, find a version of Sealed Secrets compatible with your Kubernetes cluster.

1
2
3


helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm repo update
helm search repo sealed-secrets --versions

Install Sealed Secrets resources. Setting fullnameOverride parameter to the given value is expected by kubeseal utility.

1

helm install sealed-secrets --namespace kube-system --set-string 'fullnameOverride=sealed-secrets-controller' --version <version sealed-secrets/sealed-secrets

Verify the Sealed Secrets controller is properly deployed and a new encryption key is generated.

1
2


helm list -n kube-system
kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets

Encrypting Secrets

The easiest way to create Sealed Secrets is from a secrets file. Create an example secret file:

1
2
3
4
5


kubectl create secret generic mysecret \
  --dry-run=client \
  --from-literal=username=myuser \
  --from-literal=password=mypassword \
  -o yaml > mysecret.yaml

Then, create a Sealed Secret file:

1

kubeseal -f mysecret.yaml -w mysealedsecret.yaml

An example of a Sealed Secrets file is shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: mysecret
  namespace: default
spec:
  encryptedData:
    password: AgB7FswQSdwcMs+tKtWdoB3wTTraATxg5AVrloZB5xwaoJ+5Uy1EHG2vOWfcNchVyPzy+720hbq7rM9Hi++iveAuyZi3c+67DFvK7A65sKliRBDKCFSMt5v9BVF3yEW3goolzBH4PWhCtYrxEKSD4X2/SR7OFzVfguCKlvdd9sWegcwAKYK+ZU7cm/Jt29n+4tMTLQastk6UnOErYb8t6TTqJDtTIVsFGgXTwQtF5UkCdVRTWe61OaV7Ne1s6F69O3qd2SJggkOr9dKWeDMJEQLm1UMD8kNuWkkD3VugQpuvqAqaUmFKvYP7qTBNaGlzTRbrUmtYxU2XatyqQmV+JDcOTthMQ/Zrm2H40CKznu7Se+YTdZpuljBLkQBdGk6VYAdToCHWmgxo/83ENsc39RNjW+yBpMKN5zxtOxq7aeV0L+gNyQ9DvT6pRk47GVecOYhbZyViQQrkNz74YoMCsZEUD+ddvsls1TFosDT/Xj386zJNbcGr+coknRUGBBdsxajUpHVaBvmJWXyJbjMP7CqUNgrV4qAtzQsEPmivHBEgexJcZNkeOBotHrH0K6leTreVDr4MnkIwfoqvmoPIrcKHCTMfO/9rIp4J5WL3j3NI/ig8ZxZkhE5iEaFbLxs97U4+/iprOokvKf7PyzJVFPQckEBo7cuLJng4EkpKS9Q5WEZ+5AHjB3PYpJ0/pRLfiObAK1mGU78wMP3h
    username: AgDPR8GV5pVqUzPVD+F7A+BJbhjQH89oB9/dxFuYjyv5h2Gx6HoaFK8yH6T3jSlwK9ad7tXfQOPMy2ppa2XNMk4OpDFErBFPi617Wd2axqUmXax8FP2jMp1Fj+arPd5w39mKYSQFH+5HyZStg9YCngIYaALOoCFeybg3a33VKMuBmcMGTLUEzXOES9sOCZh2PIjvKTCKIXyyKaCtVwnmxABI9csGhLupQV6IfkhMv50WHPupeuD+kDSorkzOj5ZynHvhSwxvenQP7aQDwIW236Vhhdh/mPfyKFB0A7EAqs8bYfxRk7M7STsWJSv4pMVr3ikIXnI14p+auzWfGq9cwhRHnU6s2t3EP8pUBxTSyVGDvusIGBVw3aGe/98Mz9znsxHZM1DJX23moADmUDM0bCw98H+ZND4sBfga1JKtfdM7O26Mk+J3D0wrc/VG+a7Pqwy8+LX7rAr+ZJLD2O2yLm4Ha4sXeo2xfvOpjEhumNsZxbPZwYzlFpNm4UmlTfBKVrLv/2jbeggSvcLRoiWM0zvwwBwmIqOyM2S8rxKRfr+vXVglm1Gs6UCGAYNA4/cMoXSJug3PCexP9v2mBz8o6IjzrpTvA/4Nng0ML9/mRYI8nDU+uy7cjgF58JzZ7O/KfkIVnmvrVDi1uG3kM5agGd91AiMwzhWHfDZrovYu3TzBH1lVqb5gpfgiU60vJCTGuQS1CMLYpQU=
  template:
    metadata:
      creationTimestamp: null
      name: mysecret
      namespace: default

Once created, the Sealed Secret is safe to share publicly. Store it in your repository with the rest of your code. Deploy them like any other Kubernetes object.

1

kubectl apply -f mysealedsecret.yaml

Verify that the Sealed Secret is decrypted correctly and mysecret is created:

1

kubectl get secret

Understanding Key Management

When the controller is first deployed, it generates encryption keys. It generates a new key every 30 days thereafter. The latest key is the active one used to encrypt new secrets, while old keys are retained for decrypting old Sealed Secrets.

To see all encryption keys, use:

1
2
3


kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml

It’s good practice to re-encrypt your Sealed Secrets with the latest key, either for compliance reasons or to maintain good security posture.

1
2


kubeseal --re-encrypt -w tmp.yaml < mysealedsecret.yaml
mv tmp.yaml mysealedsecret.yaml

How to Backup Encryption Keys

Backing up the private key is crucial for disaster recovery. If you lose the key, the Sealed Secrets in your repository will become unreadable, and you won’t be able to recover the plaintext secrets.

Backup your encryption key pairs with the following command:

1
2
3


kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > sealed-secrets-encryption-keys.yaml

To restore key pairs to a new cluster, deploy the Sealed Secrets Controller and apply the encryption keys definition file. Then restart the Sealed Secrets controller.

1
2


kubectl apply -f sealed-secrets-encryption-keys.yaml
kubectl delete pod -n kube-system -l app.kubernetes.io/name=sealed-secrets

Verify that you can deploy sealed secret from previous step:

1
2


kubectl apply -f mysealedsecret.yaml
kubectl get secret

Conclusion

In this post, we explored how to use Sealed Secrets to simplify the CI/CD process without compromising security. We covered how to encrypt secrets and create Sealed Secrets, the role of the Sealed Secrets controller, key rotations, and re-encryption of secrets. Finally, we discussed how to properly back up and restore encryption keys.

Happy engineering!