Doppler Secrets Operator
Kubernetes secrets sync automation with deployment auto-reload on secrets change.
The Doppler Secrets Operator is a background service that syncs secrets to Kubernetes with automatic deployment updates to ensure applications always have the latest version of secrets.
More specifically, the Operator is a controller deployed inside your Kubernetes cluster within its own dedicated
doppler-operator-system namespace so RBAC policies can tightly restrict access.
The Operator uses the
DopplerSecret custom resources which defines what Doppler config to sync, the name of the associated Kubernetes secret the Operator will manage as well what namespace it should be created in.
The Operator is then solely responsible for continuously syncing secret updates in Doppler to its managed Kubernetes secrets and can optionally automatically reload any deployments that reference a managed secret.
- Experience with deploying applications on Kubernetes
Kubernetes Secret Encryption at Rest
The Doppler Kubernetes Operator uses Kubernetes Secrets to store sensitive data.
Kubernetes Secrets are, by default, stored as unencrypted base64-encoded strings. By default they can be retrieved - as plain text - by anyone with API access, or anyone with access to Kubernetes' underlying data store, etcd. Therefore, Kubernetes recommends enabling encryption at rest to secure this data.
You can install the Operator using Helm or
You can install the latest Helm chart with:
helm repo add doppler https://helm.doppler.com helm install --generate-name doppler/doppler-kubernetes-operator
Updates can be performed with
Alternatively, you can deploy the Operator with
kubectl by applying the latest installation YAML directly from the Doppler Kubernetes GitHub repository:
kubectl apply -f https://github.com/DopplerHQ/kubernetes-operator/releases/latest/download/recommended.yaml
Regardless of the installation method, this will use your locally-configured
- Create a
- Create the resource definition for a
- Setup a service account and RBAC role for the operator
- Create a deployment for the operator inside of the cluster
You can verify that the operator is running successfully in your cluster with
./tools/operator-logs.sh. This waits for the deployment to roll out and then tails the log. You can leave this command running to keep monitoring the logs or quit safely with Ctrl-C.
You can upgrade the Operator using Helm or
You can upgrade using the latest helm chart. It's important to note that the CRD is not automatically updated if you just perform a simple helm upgrade. As such, be sure you follow this process:
# Update doppler Helm repo helm repo update # Update the dopplersecrets CRD, which is not automatically upgraded by Helm helm pull doppler/doppler-kubernetes-operator --untar kubectl apply -f doppler-kubernetes-operator/crds/all.yaml # Update the chart helm upgrade <release_name> doppler/doppler-kubernetes-operator
Alternatively, if you deployed the Operator with
kubectl you can upgrade by applying the latest installation YAML directly from the Doppler Kubernetes GitHub repository:
# This operation will also update the dopplersecrets CRD kubectl apply -f https://github.com/DopplerHQ/kubernetes-operator/releases/latest/download/recommended.yaml
Creating a secrets sync requires the creation of two resources used by the Operator:
- Doppler Token Secret: A Kubernetes secret containing the Service Token for the config to sync
- DopplerSecret: A CRD that references the
Doppler Token Secretthat provides the name namespace for the Kubernetes secret created and managed by the Operator.
Doppler Token Secret
This token will be used to fetch secrets from your Doppler config. The operator will be looking for the token in the
serviceToken field of this secret.
- A Kubernetes secret where your synced Doppler secrets will be stored (AKA "Managed Secret"). This secret will be created by the operator if it does not already exist.
Note: While these resources can be created in any namespace, it is recommended that you create your Doppler Token Secret and DopplerSecret inside the
doppler-operator-systemnamespace to prevent unauthorized access. The managed secret should be namespaced with the deployments which will use the secret.
Generate a Doppler Service Token and use it in this command to create your Doppler token secret:
kubectl create secret generic doppler-token-secret \ --namespace doppler-operator-system \ --from-literal=serviceToken=dp.st.dev.XXXX
If you have the Doppler CLI installed, you can generate a Doppler Service Token from the CLI and create the Doppler token secret in one step:
kubectl create secret generic doppler-token-secret \ --namespace doppler-operator-system \ --from-literal=serviceToken=$(doppler configs tokens create doppler-kubernetes-operator --plain)
Next, we'll create a
DopplerSecret that references your Doppler token secret and defines the location of the managed secret.
apiVersion: secrets.doppler.com/v1alpha1 kind: DopplerSecret metadata: name: dopplersecret-test # DopplerSecret Name namespace: doppler-operator-system spec: tokenSecret: # Kubernetes service token secret (namespace defaults to doppler-operator-system) name: doppler-token-secret managedSecret: # Kubernetes managed secret (will be created if does not exist) name: doppler-test-secret namespace: default # Should match the namespace of deployments that will use the secret
You can also create secrets in multiple namespaces by creating a
DopplerSecret in each namespace:
--- apiVersion: secrets.doppler.com/v1alpha1 kind: DopplerSecret metadata: name: dopplersecret-default namespace: doppler-operator-system spec: tokenSecret: name: doppler-default-token-secret managedSecret: name: doppler-default-secret namespace: default --- apiVersion: secrets.doppler.com/v1alpha1 kind: DopplerSecret metadata: name: dopplersecret-staging namespace: doppler-operator-system spec: tokenSecret: name: doppler-staging-token-secret managedSecret: name: doppler-staging-secret namespace: staging --- apiVersion: secrets.doppler.com/v1alpha1 kind: DopplerSecret metadata: name: dopplersecret-prod namespace: doppler-operator-system spec: tokenSecret: name: doppler-prod-token-secret managedSecret: name: doppler-prod-secret namespace: prod
If you're following along with these example names, you can apply this sample directly:
kubectl apply -f config/samples/secrets_v1alpha1_dopplersecret.yaml
Check that the associated Kubernetes secret has been created:
# List all Kubernetes secrets created by the Doppler operator kubectl describe secrets --selector=secrets.doppler.com/subtype=dopplerSecret
The operator continuously watches for secret updates from Doppler and when detected, automatically and instantly updates the associated secret.
Next, we'll cover how to configure a deployment to use the Kubernetes secret and enable auto-reloading for Deployments.
To use the secret created by the operator, we can use the managed secret in one of three ways. These methods are also covered in greater detail in the Kubernetes Secrets documentation.
envFrom field will populate a container's environment variables using the secret's Key-Value pairs:
envFrom: - secretRef: name: doppler-test-secret # Kubernetes secret name
valueFrom field will inject a specific environment variable from the Kubernetes secret:
env: - name: MY_APP_SECRET # The name of the environment variable exposed in the container valueFrom: secretKeyRef: name: doppler-test-secret # Kubernetes secret name key: MY_APP_SECRET # The name of the key in the Kubernetes secret
volume field will create a volume that is populated with files containing the Kubernetes secret:
volumes: - name: secret-volume secret: secretName: doppler-test-secret # Kubernetes secret name
Your deployment can use this volume by mounting it to the container's filesystem:
volumeMounts: - name: secret-volume mountPath: /etc/secrets readOnly: true
In order for the operator to reload a deployment, three things must be true:
- The deployment is in the same namespace as the managed secret
- The deployment has the
secrets.doppler.com/reloadannotation set to
- The deployment uses the managed secret
Here's an example of the reload annotation:
annotations: secrets.doppler.com/reload: 'true'
The Doppler Kubernetes operator reloads deployments by updating an annotation with the name
secrets.doppler.com/secretsupdate.<KUBERNETES_SECRET_NAME>. When this update is made, Kubernetes will automatically redeploy your pods according to the deployment's configured strategy.
Complete examples of these different deployment configurations can be found below:
If you've named your managed Kubernetes secret
doppler-test-secret in the previous step, you can apply any of these examples directly:
kubectl apply -f config/samples/deployment-envfrom.yaml kubectl rollout status -w deployment/doppler-test-deployment-envfrom
Once the Deployment has completed, you can view the logs of the test container:
kubectl logs -lapp=doppler-test --tail=-1
Setup is complete! To test the sync behavior, modify a secret in the Doppler dashboard and wait 60 seconds. Run the logs command again (or use the
watch command) to see the pods automatically restart with the new secret data.
Name Transformers allow secret names to be transformed from Doppler's
UPPER_SNAKE_CASE format into any of the following environment variable compatible formats:
Simply add the
nameTransformer field with any of the above types:
apiVersion: secrets.doppler.com/v1alpha1 kind: DopplerSecret metadata: name: dopplersecret-test namespace: doppler-operator-system spec: tokenSecret: name: doppler-token-secret managedSecret: name: doppler-test-secret namespace: default nameTransformer: dotnet-env
nameTransformer values are also validated prior to admission to prevent transformation failures.
By default, the Operator base64 encodes Doppler secret values when creating the managed Opaque Kubernetes secret as Key / Value pairs. However, instances may arise where you do not wish this base64 encoding to occur, such as when a binary
.p12 key file has already been base64 encoded in Doppler and double base64 encoding it would prevent Kubernetes from being mounting it in its original binary format.
Secret processors allow you to customize the default encoding behavior.
To override the default
plain processor, add a
processors: map which specifies the secret name and the processor to apply which in this case, is
apiVersion: secrets.doppler.com/v1alpha1 kind: DopplerSecret metadata: name: doppler-secret-pkcs12 namespace: doppler-operator-system spec: tokenSecret: name: doppler-project-pkcs12-token managedSecret: name: doppler-pkcs12 namespace: default processors: PKCS12_CERT: # Prevent double base64 encoding the already base64 encoded value from Doppler type: base64
For a complete example, check out our tutorial on using the Doppler Kubernetes Operator for Managing PKCS12 Certificates.
At present, only
base64 processors exist. Reach out to [email protected] if there is another processor you'd like us to support.
Instead of the standard Key / Value pairs, you can download secrets as a single file in the following formats:
format is specified, a single
DOPPLER_SECRETS_FILE key is set in the created secret with the string contents of the downloaded file.
Simply add the
apiVersion: secrets.doppler.com/v1alpha1 kind: DopplerSecret metadata: name: dotnet-webapp-appsettings namespace: doppler-operator-system spec: tokenSecret: name: doppler-token-dotnet-webapp namespace: doppler-operator-system managedSecret: name: dotnet-webapp-appsettings namespace: default format: dotnet-json
You can then configure your deployment spec to mount the file at the desired path:
... spec: containers: - name: dotnet-webapp volumeMounts: - name: doppler mountPath: /usr/src/app/secrets readOnly: true volumes: - name: doppler secret: secretName: dotnet-webapp-appsettings # Managed secret name optional: false items: - key: DOPPLER_SECRETS_FILE # Hard-coded by Operator when format specified path: appsettings.json # Name or path to file name appended to container mountPath
Failure Strategy and Troubleshooting
If the operator fails to fetch secrets from the Doppler API (e.g. a connection problem or invalid service token), no changes are made to the managed Kubernetes secret or your deployments. The operator will continue to attempt to reconnect to the Doppler API indefinitely.
status.conditions to report its current state and any errors that may have occurred.
In this example, our Doppler service token has been revoked and the operator is reporting an error condition:
$ kubectl describe dopplersecrets -n doppler-operator-system Name: dopplersecret-test Namespace: doppler-operator-system Labels: <none> Annotations: <none> API Version: secrets.doppler.com/v1alpha1 Kind: DopplerSecret Metadata: ... Spec: ... Status: Conditions: Last Transition Time: 2021-06-02T15:46:57Z Message: Secret update failed: Doppler Error: Invalid Service token Reason: Error Status: False Type: secrets.doppler.com/SecretSyncReady Last Transition Time: 2021-06-02T15:46:57Z Message: Deployment reload has been stopped due to secrets sync failure Reason: Stopped Status: False Type: secrets.doppler.com/DeploymentReloadReady Events: <none>
You can safely modify your token Kubernetes secret or
DopplerSecret at any time. To update our Doppler service token, we can modify our token Kubernetes secret directly and the changes will take effect immediately.
DopplerSecret resource manages the managed Kubernetes secret but does not officially own it. Therefore, deleting a
DopplerSecret will not automatically delete the managed secret.
tools/get-secret.sh- fetch and decode a Kubernetes secret
tools/operator-logs.sh- wait for the operator deployment to roll out, then tail the logs
To uninstall the operator, first delete any
DopplerSecret resources and any referenced Kubernetes secrets that are no longer needed.
kubectl delete dopplersecrets --all --all-namespaces kubectl delete secret doppler-token-secret -n doppler-operator-system
If you installed the operator with Helm, you can use
helm uninstall to remove the installation resources. Otherwise, run the following command:
kubectl delete -f https://github.com/DopplerHQ/kubernetes-operator/releases/latest/download/recommended.yaml
This project uses the Operator SDK.
When developing locally, you can run the operator using:
make install run
See the Operator SDK Go Tutorial for more information.
This project is released with Github Actions. Adding a Github Release will start an action which builds the operator image and publishes it to DockerHub. Tag names should match the pattern
Updated 5 months ago