Kubernetes Operator
Automatic secrets-sync from Doppler to Kubernetes


The Doppler Kubernetes Operator is a background service that syncs secrets to Kubernetes with automatic deployment updates to ensure applications always have the latest version of secrets.
More specifically, the Operator is a controller deployed inside your Kubernetes cluster within its own dedicated doppler-operator-system
namespace so RBAC policies can tightly restrict access.
The Operator uses the DopplerSecret
custom resources which defines what Doppler config to sync, the name of the associated Kubernetes secret the Operator will manage as well what namespace it should be created in.
The Operator is then solely responsible for continuously syncing secret updates in Doppler to its managed Kubernetes secrets and can optionally automatically reload any deployments that reference a managed secret.
Prerequites
- Experience with deploying applications on Kubernetes
Kubernetes Secret Encryption at Rest
The Doppler Kubernetes Operator uses Kubernetes Secrets to store sensitive data.
Kubernetes Secrets are, by default, stored as unencrypted base64-encoded strings. By default they can be retrieved - as plain text - by anyone with API access, or anyone with access to Kubernetes' underlying data store, etcd. Therefore, Kubernetes recommends enabling encryption at rest to secure this data.
Install
You can install the Operator using Helm or kubectl
.
Using Helm
You can install the latest Helm chart with:
helm repo add doppler https://helm.doppler.com
helm install --generate-name doppler/doppler-kubernetes-operator
Updates can be performed with helm upgrade
.
Using kubectl
kubectl
Alternatively, you can deploy the Operator with kubectl
by applying the latest installation YAML directly from the Doppler Kubernetes GitHub repository:
kubectl apply -f https://github.com/DopplerHQ/kubernetes-operator/releases/latest/download/recommended.yaml
Regardless of the installation method, this will use your locally-configured kubectl
to:
- Create a
doppler-operator-system
namespace - Create the resource definition for a
DopplerSecret
- Setup a service account and RBAC role for the operator
- Create a deployment for the operator inside of the cluster
You can verify that the operator is running successfully in your cluster with ./tools/operator-logs.sh
. This waits for the deployment to roll out and then tails the log. You can leave this command running to keep monitoring the logs or quit safely with Ctrl-C.
Secrets Sync
Creating a secrets sync requires the creation of two resources used by the Operator:
- Doppler Token Secret: A Kubernetes secret containing the Service Token for the config to sync
- DopplerSecret: A CRD that references the
Doppler Token Secret
that provides the name namespace for the Kubernetes secret created and managed by the Operator.
Doppler Token Secret
This token will be used to fetch secrets from your Doppler config. The operator will be looking for the token in the serviceToken
field of this secret.
- A Kubernetes secret where your synced Doppler secrets will be stored (AKA "Managed Secret"). This secret will be created by the operator if it does not already exist.
Note: While these resources can be created in any namespace, it is recommended that you create your Doppler Token Secret and DopplerSecret inside the
doppler-operator-system
namespace to prevent unauthorized access. The managed secret should be namespaced with the deployments which will use the secret.
Generate a Doppler Service Token and use it in this command to create your Doppler token secret:
kubectl create secret generic doppler-token-secret \
--namespace doppler-operator-system \
--from-literal=serviceToken=dp.st.dev.XXXX
If you have the Doppler CLI installed, you can generate a Doppler Service Token from the CLI and create the Doppler token secret in one step:
kubectl create secret generic doppler-token-secret \
--namespace doppler-operator-system \
--from-literal=serviceToken=$(doppler configs tokens create doppler-kubernetes-operator --plain)
DopplerSecret CRD
Next, we'll create a DopplerSecret
that references your Doppler token secret and defines the location of the managed secret.
apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
name: dopplersecret-test # DopplerSecret Name
namespace: doppler-operator-system
spec:
tokenSecret: # Kubernetes service token secret (namespace defaults to doppler-operator-system)
name: doppler-token-secret
managedSecret: # Kubernetes managed secret (will be created if does not exist)
name: doppler-test-secret
namespace: default # Should match the namespace of deployments that will use the secret
You can also create secrets in multiple namespaces by creating a DopplerSecret
in each namespace:
---
apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
name: dopplersecret-default
namespace: doppler-operator-system
spec:
tokenSecret:
name: doppler-default-token-secret
managedSecret:
name: doppler-default-secret
namespace: default
---
apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
name: dopplersecret-staging
namespace: doppler-operator-system
spec:
tokenSecret:
name: doppler-staging-token-secret
managedSecret:
name: doppler-staging-secret
namespace: staging
---
apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
name: dopplersecret-prod
namespace: doppler-operator-system
spec:
tokenSecret:
name: doppler-prod-token-secret
managedSecret:
name: doppler-prod-secret
namespace: prod
If you're following along with these example names, you can apply this sample directly:
kubectl apply -f config/samples/secrets_v1alpha1_dopplersecret.yaml
Check that the associated Kubernetes secret has been created:
# List all Kubernetes secrets created by the Doppler operator
kubectl describe secrets --selector=secrets.doppler.com/subtype=dopplerSecret
The operator continuously watches for secret updates from Doppler and when detected, automatically and instantly updates the associated secret.
Next, we'll cover how to configure a deployment to use the Kubernetes secret and enable auto-reloading for Deployments.
Deployments
To use the secret created by the operator, we can use the managed secret in one of three ways. These methods are also covered in greater detail in the Kubernetes Secrets documentation.
envFrom
envFrom
The envFrom
field will populate a container's environment variables using the secret's Key-Value pairs:
envFrom:
- secretRef:
name: doppler-test-secret # Kubernetes secret name
valueFrom
valueFrom
The valueFrom
field will inject a specific environment variable from the Kubernetes secret:
env:
- name: MY_APP_SECRET # The name of the environment variable exposed in the container
valueFrom:
secretKeyRef:
name: doppler-test-secret # Kubernetes secret name
key: MY_APP_SECRET # The name of the key in the Kubernetes secret
volume
volume
The volume
field will create a volume that is populated with files containing the Kubernetes secret:
volumes:
- name: secret-volume
secret:
secretName: doppler-test-secret # Kubernetes secret name
Your deployment can use this volume by mounting it to the container's filesystem:
volumeMounts:
- name: secret-volume
mountPath: /etc/secrets
readOnly: true
Automatic Redeployments
In order for the operator to reload a deployment, three things must be true:
- The deployment is in the same namespace as the managed secret
- The deployment has the
secrets.doppler.com/reload
annotation set to'true'
(string) - The deployment uses the managed secret
Here's an example of the reload annotation:
annotations:
secrets.doppler.com/reload: 'true'
The Doppler Kubernetes operator reloads deployments by updating an annotation with the name secrets.doppler.com/secretsupdate.<KUBERNETES_SECRET_NAME>
. When this update is made, Kubernetes will automatically redeploy your pods according to the deployment's configured strategy.
Examples
Complete examples of these different deployment configurations can be found below:
If you've named your managed Kubernetes secret doppler-test-secret
in the previous step, you can apply any of these examples directly:
kubectl apply -f config/samples/deployment-envfrom.yaml
kubectl rollout status -w deployment/doppler-test-deployment-envfrom
Once the Deployment has completed, you can view the logs of the test container:
kubectl logs -lapp=doppler-test --tail=-1
Setup is complete! To test the sync behavior, modify a secret in the Doppler dashboard and wait 60 seconds. Run the logs command again (or use the watch
command) to see the pods automatically restart with the new secret data.
Processors
By default, the Operator base64 encodes Doppler secret values when creating the managed Opaque Kubernetes secret as Key / Value pairs. However, instances may arise where you do not wish this base64 encoding to occur, such as when a binary .p12
key file has already been base64 encoded in Doppler and double base64 encoding it would prevent Kubernetes from being mounting it in its original binary format.
Secret processors allow you to customize the default encoding behavior.
To override the default plain
processor, add a processors:
map which specifies the secret name and the processor to apply which in this case, is base64
:
apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
name: doppler-secret-pkcs12
namespace: doppler-operator-system
spec:
tokenSecret:
name: doppler-project-pkcs12-token
managedSecret:
name: doppler-pkcs12
namespace: default
processors:
PKCS12_CERT:
# Prevent double base64 encoding the already base64 encoded value from Doppler
type: base64
For a complete example, check out our tutorial on using the Doppler Kubernetes Operator for Managing PKCS12 Certificates.
Processor Types
At present, only plain
and base64
processors exist. Reach out to [email protected] if there is another processor you'd like us to support.
Failure Strategy and Troubleshooting
Inspecting Status
If the operator fails to fetch secrets from the Doppler API (e.g. a connection problem or invalid service token), no changes are made to the managed Kubernetes secret or your deployments. The operator will continue to attempt to reconnect to the Doppler API indefinitely.
The DopplerSecret
uses status.conditions
to report its current state and any errors that may have occurred.
In this example, our Doppler service token has been revoked and the operator is reporting an error condition:
$ kubectl describe dopplersecrets -n doppler-operator-system
Name: dopplersecret-test
Namespace: doppler-operator-system
Labels: <none>
Annotations: <none>
API Version: secrets.doppler.com/v1alpha1
Kind: DopplerSecret
Metadata:
...
Spec:
...
Status:
Conditions:
Last Transition Time: 2021-06-02T15:46:57Z
Message: Secret update failed: Doppler Error: Invalid Service token
Reason: Error
Status: False
Type: secrets.doppler.com/SecretSyncReady
Last Transition Time: 2021-06-02T15:46:57Z
Message: Deployment reload has been stopped due to secrets sync failure
Reason: Stopped
Status: False
Type: secrets.doppler.com/DeploymentReloadReady
Events: <none>
You can safely modify your token Kubernetes secret or DopplerSecret
at any time. To update our Doppler service token, we can modify our token Kubernetes secret directly and the changes will take effect immediately.
The DopplerSecret
resource manages the managed Kubernetes secret but does not officially own it. Therefore, deleting a DopplerSecret
will not automatically delete the managed secret.
Included Tools
tools/get-secret.sh
- fetch and decode a Kubernetes secrettools/operator-logs.sh
- wait for the operator deployment to roll out, then tail the logs
Uninstall
To uninstall the operator, first delete any DopplerSecret
resources and any referenced Kubernetes secrets that are no longer needed.
kubectl delete dopplersecrets --all --all-namespaces
kubectl delete secret doppler-token-secret -n doppler-operator-system
If you installed the operator with Helm, you can use helm uninstall
to remove the installation resources. Otherwise, run the following command:
kubectl delete -f https://github.com/DopplerHQ/kubernetes-operator/releases/latest/download/recommended.yaml
Development
This project uses the Operator SDK.
When developing locally, you can run the operator using:
make install run
See the Operator SDK Go Tutorial for more information.
Release
This project is released with Github Actions. Adding a Github Release will start an action which builds the operator image and publishes it to DockerHub. Tag names should match the pattern vX.X.X
.
Updated 3 months ago