The Doppler Kubernetes Operator is a background service that syncs secrets to Kubernetes with automatic deployment updates to ensure applications always have the latest version of secrets.

More specifically, the Operator is a controller deployed inside your Kubernetes cluster within its own dedicated doppler-operator-system namespace so RBAC policies can tightly restrict access.

The Operator uses the DopplerSecret custom resources which defines what Doppler config to sync, the name of the associated Kubernetes secret the Operator will manage as well what namespace it should be created in.

The Operator is then solely responsible for continuously syncing secret updates in Doppler to its managed Kubernetes secrets and can optionally automatically reload any deployments that reference a managed secret.

Prerequites

• Experience with deploying applications on Kubernetes

Kubernetes Secret Encryption at Rest

The Doppler Kubernetes Operator uses Kubernetes Secrets to store sensitive data.

Kubernetes Secrets are, by default, stored as unencrypted base64-encoded strings. By default they can be retrieved - as plain text - by anyone with API access, or anyone with access to Kubernetes' underlying data store, etcd. Therefore, Kubernetes recommends enabling encryption at rest to secure this data.

Install

You can install the Operator using Helm or kubectl.

Using Helm

You can install the latest Helm chart with:

helm repo add doppler https://helm.doppler.com
helm install --generate-name doppler/doppler-kubernetes-operator

Updates can be performed with helm upgrade.

Using kubectl

Alternatively, you can deploy the Operator with kubectl by applying the latest installation YAML directly from the Doppler Kubernetes GitHub repository:

kubectl apply -f https://github.com/DopplerHQ/kubernetes-operator/releases/latest/download/recommended.yaml

Regardless of the installation method, this will use your locally-configured kubectl to:

• Create a doppler-operator-system namespace
• Create the resource definition for a DopplerSecret
• Setup a service account and RBAC role for the operator
• Create a deployment for the operator inside of the cluster

You can verify that the operator is running successfully in your cluster with ./tools/operator-logs.sh. This waits for the deployment to roll out and then tails the log. You can leave this command running to keep monitoring the logs or quit safely with Ctrl-C.

Secrets Sync

Creating a secrets sync requires the creation of two resources used by the Operator:

1. Doppler Token Secret: A Kubernetes secret containing the Service Token for the config to sync
2. DopplerSecret: A CRD that references the Doppler Token Secret that provides the name namespace for the Kubernetes secret created and managed by the Operator.

Doppler Token Secret

This token will be used to fetch secrets from your Doppler config. The operator will be looking for the token in the serviceToken field of this secret.

• A Kubernetes secret where your synced Doppler secrets will be stored (AKA "Managed Secret"). This secret will be created by the operator if it does not already exist.

Note: While these resources can be created in any namespace, it is recommended that you create your Doppler Token Secret and DopplerSecret inside the doppler-operator-system namespace to prevent unauthorized access. The managed secret should be namespaced with the deployments which will use the secret.

Generate a Doppler Service Token and use it in this command to create your Doppler token secret:

kubectl create secret generic doppler-token-secret \
  --namespace doppler-operator-system \
  --from-literal=serviceToken=dp.st.dev.XXXX

If you have the Doppler CLI installed, you can generate a Doppler Service Token from the CLI and create the Doppler token secret in one step:

kubectl create secret generic doppler-token-secret \
  --namespace doppler-operator-system \
  --from-literal=serviceToken=$(doppler configs tokens create doppler-kubernetes-operator --plain)

DopplerSecret CRD

Next, we'll create a DopplerSecret that references your Doppler token secret and defines the location of the managed secret.

apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: dopplersecret-test # DopplerSecret Name
  namespace: doppler-operator-system
spec:
  tokenSecret: # Kubernetes service token secret (namespace defaults to doppler-operator-system)
    name: doppler-token-secret
  managedSecret: # Kubernetes managed secret (will be created if does not exist)
    name: doppler-test-secret
    namespace: default # Should match the namespace of deployments that will use the secret

You can also create secrets in multiple namespaces by creating a DopplerSecret in each namespace:

---
apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: dopplersecret-default
  namespace: doppler-operator-system
spec:
  tokenSecret:
    name: doppler-default-token-secret
  managedSecret:
    name: doppler-default-secret
    namespace: default
---
apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: dopplersecret-staging
  namespace: doppler-operator-system
spec:
  tokenSecret:
    name: doppler-staging-token-secret
  managedSecret:
    name: doppler-staging-secret
    namespace: staging
---
apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: dopplersecret-prod
  namespace: doppler-operator-system
spec:
  tokenSecret:
    name: doppler-prod-token-secret
  managedSecret:
    name: doppler-prod-secret
    namespace: prod

If you're following along with these example names, you can apply this sample directly:

kubectl apply -f config/samples/secrets_v1alpha1_dopplersecret.yaml

Check that the associated Kubernetes secret has been created:

# List all Kubernetes secrets created by the Doppler operator
kubectl describe secrets --selector=secrets.doppler.com/subtype=dopplerSecret

The operator continuously watches for secret updates from Doppler and when detected, automatically and instantly updates the associated secret.

Next, we'll cover how to configure a deployment to use the Kubernetes secret and enable auto-reloading for Deployments.

Deployments

To use the secret created by the operator, we can use the managed secret in one of three ways. These methods are also covered in greater detail in the Kubernetes Secrets documentation.

envFrom

The envFrom field will populate a container's environment variables using the secret's Key-Value pairs:

envFrom:
  - secretRef:
      name: doppler-test-secret # Kubernetes secret name

valueFrom

The valueFrom field will inject a specific environment variable from the Kubernetes secret:

env:
  - name: MY_APP_SECRET # The name of the environment variable exposed in the container
    valueFrom:
      secretKeyRef:
        name: doppler-test-secret # Kubernetes secret name
        key: MY_APP_SECRET # The name of the key in the Kubernetes secret

volume

The volume field will create a volume that is populated with files containing the Kubernetes secret:

volumes:
  - name: secret-volume
    secret:
      secretName: doppler-test-secret # Kubernetes secret name

Your deployment can use this volume by mounting it to the container's filesystem:

volumeMounts:
  - name: secret-volume
    mountPath: /etc/secrets
    readOnly: true

Automatic Redeployments

In order for the operator to reload a deployment, three things must be true:

• The deployment is in the same namespace as the managed secret
• The deployment has the secrets.doppler.com/reload annotation set to 'true' (string)
• The deployment uses the managed secret

Here's an example of the reload annotation:

annotations:
  secrets.doppler.com/reload: 'true'

The Doppler Kubernetes operator reloads deployments by updating an annotation with the name secrets.doppler.com/secretsupdate.<KUBERNETES_SECRET_NAME>. When this update is made, Kubernetes will automatically redeploy your pods according to the deployment's configured strategy.

Examples

Complete examples of these different deployment configurations can be found below:

• deployment-envfrom.yaml
• deployment-valuefrom.yaml
• deployment-volume.yaml

If you've named your managed Kubernetes secret doppler-test-secret in the previous step, you can apply any of these examples directly:

kubectl apply -f config/samples/deployment-envfrom.yaml
kubectl rollout status -w deployment/doppler-test-deployment-envfrom

Once the Deployment has completed, you can view the logs of the test container:

kubectl logs -lapp=doppler-test --tail=-1

Setup is complete! To test the sync behavior, modify a secret in the Doppler dashboard and wait 60 seconds. Run the logs command again (or use the watch command) to see the pods automatically restart with the new secret data.

Processors

By default, the Operator base64 encodes Doppler secret values when creating the managed Opaque Kubernetes secret as Key / Value pairs. However, instances may arise where you do not wish this base64 encoding to occur, such as when a binary .p12 key file has already been base64 encoded in Doppler and double base64 encoding it would prevent Kubernetes from being mounting it in its original binary format.

Secret processors allow you to customize the default encoding behavior.

To override the default plain processor, add a processors: map which specifies the secret name and the processor to apply which in this case, is base64:

apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: doppler-secret-pkcs12 
  namespace: doppler-operator-system
spec:
  tokenSecret:
    name: doppler-project-pkcs12-token
  managedSecret:
    name: doppler-pkcs12
    namespace: default
  processors:
    PKCS12_CERT:
      # Prevent double base64 encoding the already base64 encoded value from Doppler
      type: base64

For a complete example, check out our tutorial on using the Doppler Kubernetes Operator for Managing PKCS12 Certificates.

Processor Types

At present, only plain and base64 processors exist. Reach out to [email protected] if there is another processor you'd like us to support.

Failure Strategy and Troubleshooting

Inspecting Status

If the operator fails to fetch secrets from the Doppler API (e.g. a connection problem or invalid service token), no changes are made to the managed Kubernetes secret or your deployments. The operator will continue to attempt to reconnect to the Doppler API indefinitely.

The DopplerSecret uses status.conditions to report its current state and any errors that may have occurred.

In this example, our Doppler service token has been revoked and the operator is reporting an error condition:

$ kubectl describe dopplersecrets -n doppler-operator-system
Name:         dopplersecret-test
Namespace:    doppler-operator-system
Labels:       <none>
Annotations:  <none>
API Version:  secrets.doppler.com/v1alpha1
Kind:         DopplerSecret
Metadata:
  ...
Spec:
  ...
Status:
  Conditions:
    Last Transition Time:  2021-06-02T15:46:57Z
    Message:               Secret update failed: Doppler Error: Invalid Service token
    Reason:                Error
    Status:                False
    Type:                  secrets.doppler.com/SecretSyncReady
    Last Transition Time:  2021-06-02T15:46:57Z
    Message:               Deployment reload has been stopped due to secrets sync failure
    Reason:                Stopped
    Status:                False
    Type:                  secrets.doppler.com/DeploymentReloadReady
Events:                    <none>

You can safely modify your token Kubernetes secret or DopplerSecret at any time. To update our Doppler service token, we can modify our token Kubernetes secret directly and the changes will take effect immediately.

The DopplerSecret resource manages the managed Kubernetes secret but does not officially own it. Therefore, deleting a DopplerSecret will not automatically delete the managed secret.

Included Tools

• tools/get-secret.sh - fetch and decode a Kubernetes secret
• tools/operator-logs.sh - wait for the operator deployment to roll out, then tail the logs

Uninstall

To uninstall the operator, first delete any DopplerSecret resources and any referenced Kubernetes secrets that are no longer needed.

kubectl delete dopplersecrets --all --all-namespaces
kubectl delete secret doppler-token-secret -n doppler-operator-system

If you installed the operator with Helm, you can use helm uninstall to remove the installation resources. Otherwise, run the following command:

kubectl delete -f https://github.com/DopplerHQ/kubernetes-operator/releases/latest/download/recommended.yaml

Development

This project uses the Operator SDK.

When developing locally, you can run the operator using:

make install run

See the Operator SDK Go Tutorial for more information.

Release

This project is released with Github Actions. Adding a Github Release will start an action which builds the operator image and publishes it to DockerHub. Tag names should match the pattern vX.X.X.