Published on

k8s encrypted secrets with flux, sops, and age

Authors

Introduction

While flux has a very detailed guide on managing secrets with SOPS, particularly when using GPG for the encryption, I personally like to use age for encrypting things like Kubernetes Secrets. age is a bit simpler to use, is more opinionated on the encryption algorithms and parameters used (influenced by best practices) and doesn't have all of the legacy use cases that GPG has to support. One of the downsides is that GPG, since it's more established and widely used, has more audits and reviews that could surface potential issues than a younger project like age is able to do. However, given that I'm using this on a local kubernetes cluster used mostly for developing and testing personal projects and not for critical public-facing infrastructure, the tradeoffs are worth it for me. Also, given the multi-cluster support that flux provides, it's possible to use a more mature encryption tool like GPG, or one of the various managed or cloud provided encryption tools on a production cluster with a minimal amount of effort.

Installing SOPS and age

Before we can configure flux to use sops and age to encrypt/decrypt secrets, we have to install the respective binaries. The most recent release of SOPS at the time of writing this article was 3.8.1 and can be installed with

# Download the binary
curl -LO https://github.com/getsops/sops/releases/download/v3.8.1/sops-v3.8.1.linux.amd64
# make it executable
chmod +x sops-v3.8.1.linux.amd64
# move the binary to /usr/local/bin/sops - this may require sudo depending on your permissions structure
sudo mv sops-v3.8.1.linux.amd64 /usr/local/bin/sops
# view sops help
sops -h
# view sops version
sops -v

age can also be installed from its release page or alternatively, from apt. Since the latest version available for Ubuntu 22.04 was 1.0.0-1, and the most recent release on GitHub was 1.1.1 at the time of writing, age was installed via

# Download the binary
curl -LO https://github.com/FiloSottile/age/releases/download/v1.1.1/age-v1.1.1-linux-amd64.tar.gz
# Extract the archive
tar -xvzf age-v1.1.1-linux-amd64.tar.gz 
# move the binaries to /usr/local/bin
sudo mv age/{age,age-keygen} /usr/local/bin/
# view age usage instructions
age
# view age-keygen help
age-keygen -h

At this point, sops, age, and age-keygen are installed, and we can move on to creating a secrets repository, and then configuring k8s and flux to use the secrets repository.

Creating a secrets repository

Before we configure flux/k8s to use sops and age for secrets management, we need to create a secrets repository, generate an age key pair, and add an encrypted secret to the repository. I created a public blank GitHub repository flux_secrets for storing the secrets and cloned it to my machine. Before we add any secrets and encrypt them, we need to generate an age key pair with age-keygen

age-keygen -o ~/flux.agekey

If we cat the output file, we would see it contains a public key and a private key. Be extremely careful with the private key and where it's stored - this can decrypt all of your secrets encrypted with the public key. cd into the blank secrets repository (in my case flux_secrets) and run

mkdir -p clusters/microk8s
touch clusters/microk8s/.sops.yaml

The clusters/ directory contains directories corresponding to each of the clusters you want to manage secrets on - in this case we're just managing the local microk8s cluster in the microk8s/. If I were to spin up a production cluster, I could manage it's secrets (including using a different encryption provider) in a production/ directory. Finally, the blank .sops.yaml is where we can put common SOPS configuration options for the microk8s cluster, including the age public key, regex for what files need to be encrypted, and regex for which portion of the file needs to be encrypted. My .sops.yaml looks like this

creation_rules:
	- path_regex: .*.yaml
      encrypted_regex: ^(data|stringData)$
      age: <age public key>

Be sure that your public key is what is put in this file - do NOT put your age private key in this file!

Next, we'll create a Kubernetes secret in the clusters/microk8s directory. In this example, I'm creating a secret to store an API key for api.congress.gov, which will be used in some upcoming projects.

cd clusters/microk8s
kubectl -n congressbot create secret generic api-keys \
--from-literal=congressapi=<api.congress.gov key> \
--dry-run=client \
-o yaml > api-keys.yaml

If you were to cat the newly created api-keys.yaml, you would be able to see the API key in plain text. To encrypt the secret, run

sops --encrypt --in-place api-keys.yaml

After that, cat the file and you'll see that the secret has been encrypted. Add all of the files to the repository, commit it, and push to the remote repository for our next step - configuring flux to use the encrypted secrets repository!

Configuring flux to use encrypted secrets

To configure flux to use sops and age for secrets, we first have to create a Kubernetes Secret in the flux-system namespace - we'll do this by piping the .agekey file generated above into a kubectl command like this

cat ~/flux.agekey |  
kubectl create secret generic sops-age \
--namespace=flux-system \
--from-file=age.agekey=/dev/stdin

It's important that the key specified in the --from-file option ends in .agekey in ordered to be detected properly. Next we'll add the git repository as a source to flux with

flux create source git flux-secrets \
--url=https://github.com/zanycadence/flux_secrets \
--branch=main

And we'll create a kustomization to reconcile the secrets with

flux create kustomization flux-secrets \
--source=flux-secrets \
--path=./clusters/microk8s \
--prune=true \
--interval=10m \
--decryption-provider=sops \
--decryption-secret=sops-age

After that reconciles, we should be able to run kubectl get secrets -n congressbot and see that our api-keys secret is available in the cluster (one note - I already had a congressbot namespace created, you will need to create a corresponding namespace if you're directly replicating my work). Now that we've verified that the git source and kustomization are working, we can export the configuration files by adding the --export flag and piping the output to .yaml files tracked by our main flux repository.

cd flux_cluster/
flux create source git flux-secrets \
--url=https://github.com/zanycadence/flux_secrets \
--branch=main \
--export > infra/base/sources/flux-secrets.yaml

flux create kustomization flux-secrets \
--source=flux-secrets \
--path=./clusters/microk8s \
--prune=true \
--interval=10m \
--decryption-provider=sops \
--decryption-secret=sops-age \
--export > infra/microk8s/flux-secrets.yaml

There are some minor edits that are needed to ensure that flux can pick up on the resources to infra/microk8s/kustomization.yaml and infra/base/sources/kustomization.yaml which you can view at each of the linked files. Now, if for some reason I need to reset my local microk8s cluster, all I have to do is make sure that I add the sops-age secret from above and reconcile the flux installation!

Further thoughts…

While it's not a big deal that the secrets repository is public as everything sensitive should be encrypted, I'll probably move it to a private repository and update the source to add in authentication. Also, I've found that when working with teams it's important to configure a git pre-commit hook to ensure that all of the secrets are encrypted prior to being committed; it's possible that a future post will go over helpful git hooks for flux GitOps management.