Doppler Kubernetes Operator

The Doppler Secrets Operator is a background service that syncs secrets to Kubernetes with automatic deployment updates to ensure applications always have the latest version of secrets.

More specifically, the Operator is a controller deployed inside your Kubernetes cluster within its own dedicated doppler-operator-system namespace so RBAC policies can tightly restrict access.

The Operator uses the DopplerSecret custom resources which defines what Doppler config to sync, the name of the associated Kubernetes secret the Operator will manage as well what namespace it should be created in.

The Operator is then solely responsible for continuously syncing secret updates in Doppler to its managed Kubernetes secrets and can optionally automatically reload any deployments that reference a managed secret.

Prerequites

Experience with deploying applications on Kubernetes

Kubernetes Secret Encryption at Rest

The Doppler Kubernetes Operator uses Kubernetes Secrets to store sensitive data.

Kubernetes Secrets are, by default, stored as unencrypted base64-encoded strings. By default they can be retrieved - as plain text - by anyone with API access, or anyone with access to Kubernetes' underlying data store, etcd. Therefore, Kubernetes recommends enabling encryption at rest to secure this data.

Install

You can install the Operator using Helm or kubectl.

Using Helm

You can install the latest Helm chart with:

helm repo add doppler https://helm.doppler.com
helm install --generate-name doppler/doppler-kubernetes-operator

Updates can be performed with helm upgrade.

Using `kubectl`

Alternatively, you can deploy the Operator with kubectl by applying the latest installation YAML directly from the Doppler Kubernetes GitHub repository:

kubectl apply -f https://github.com/DopplerHQ/kubernetes-operator/releases/latest/download/recommended.yaml

Regardless of the installation method, this will use your locally-configured kubectl to:

Create a doppler-operator-system namespace
Create the resource definition for a DopplerSecret
Setup a service account and RBAC role for the operator
Create a deployment for the operator inside of the cluster

You can verify that the operator is running successfully in your cluster with ./tools/operator-logs.sh. This waits for the deployment to roll out and then tails the log. You can leave this command running to keep monitoring the logs or quit safely with Ctrl-C.

Upgrade

You can upgrade the Operator using Helm or kubectl.

Using Helm

You can upgrade using the latest helm chart. It's important to note that the CRD is not automatically updated if you just perform a simple helm upgrade. As such, be sure you follow this process:

# Update doppler Helm repo
helm repo update

# Update the dopplersecrets CRD, which is not automatically upgraded by Helm
helm pull doppler/doppler-kubernetes-operator --untar
kubectl apply -f doppler-kubernetes-operator/crds/all.yaml

# Update the chart
helm upgrade <release_name> doppler/doppler-kubernetes-operator

Using `kubectl`

Alternatively, if you deployed the Operator with kubectl you can upgrade by applying the latest installation YAML directly from the Doppler Kubernetes GitHub repository:

# This operation will also update the dopplersecrets CRD
kubectl apply -f https://github.com/DopplerHQ/kubernetes-operator/releases/latest/download/recommended.yaml

Secrets Sync

Creating a secrets sync requires the creation of two resources used by the Operator:

Doppler Token Secret: A Kubernetes secret containing the Service Token for the config to sync
DopplerSecret: A CRD that references the Doppler Token Secret that provides the name and namespace for the Kubernetes secret created and managed by the Operator. This secret will be created by the operator if it does not already exist.

Doppler Token Secret

This token will be used to fetch secrets from your Doppler config. The operator will be looking for the token in the serviceToken field of this secret.

Generate a Doppler Service Token and use it in this command to create your Doppler token secret:

kubectl create secret generic doppler-token-secret \
  --namespace doppler-operator-system \
  --from-literal=serviceToken=dp.st.dev.XXXX

If you have the Doppler CLI installed, you can generate a Doppler Service Token from the CLI and create the Doppler token secret in one step:

🚧
This command will generate a CLI Token, which has the same access permissions as your user. If you're on the Developer plan or just doing a quick test to see how this works, that should be fine. However, if you're on the Team plan or higher and this is for an actual deployment, we recommend using a Service Account token instead. The Kubernetes Operator doesn't require any Workplace level permissions and only requires the View Secrets project permission on configs it interacts with.

kubectl create secret generic doppler-token-secret \
  --namespace doppler-operator-system \
  --from-literal=serviceToken=$(doppler configure get token --plain)

DopplerSecret CRD

Next, we'll create a DopplerSecret that references your Doppler token secret and defines the location of the managed secret.

🚧
By default, DopplerSecret CRDs can only reference secrets in the namespace it's created in. If a DopplerSecret needs to reference secrets in other namespaces, then it needs to be created in the same namespace that the operator is deployed in (i.e., doppler-operator-system). If you don't specify a namespace in the CRD, then it will be created in the default namespace and as such will only be able to reference secrets in that namespace.

apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: dopplersecret-test # DopplerSecret Name
  namespace: doppler-operator-system
spec:
  tokenSecret: # Kubernetes service token secret (namespace defaults to doppler-operator-system)
    name: doppler-token-secret
  managedSecret: # Kubernetes managed secret (will be created if does not exist)
    name: doppler-test-secret
    namespace: default # Should match the namespace of deployments that will use the secret
    type: Opaque # Optional: defaults to `Opaque` so can be left out unless changed

apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: dopplersecret-test # DopplerSecret Name
  namespace: doppler-operator-system
spec:
  tokenSecret: # Kubernetes service token secret (namespace defaults to doppler-operator-system)
    name: doppler-token-secret
  project: your-project-name-here
  config: your-config-name-here
  managedSecret: # Kubernetes managed secret (will be created if does not exist)
    name: doppler-test-secret
    namespace: default # Should match the namespace of deployments that will use the secret

The type of the managedSecret can be any of the following:

Opaque
kubernetes.io/tls
kubernetes.io/service-account-token
kubernetes.io/dockercfg
kubernetes.io/dockerconfigjson
kubernetes.io/basic-auth
kubernetes.io/ssh-auth
bootstrap.kubernetes.io/token

You can also create secrets in multiple namespaces by creating a DopplerSecret in each namespace:

---
apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: dopplersecret-default
  namespace: doppler-operator-system
spec:
  tokenSecret:
    name: doppler-default-token-secret
  managedSecret:
    name: doppler-default-secret
    namespace: default
---
apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: dopplersecret-staging
  namespace: doppler-operator-system
spec:
  tokenSecret:
    name: doppler-staging-token-secret
  managedSecret:
    name: doppler-staging-secret
    namespace: staging
---
apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: dopplersecret-prod
  namespace: doppler-operator-system
spec:
  tokenSecret:
    name: doppler-prod-token-secret
  managedSecret:
    name: doppler-prod-secret
    namespace: prod

If you're following along with these example names, you can apply this sample directly:

kubectl apply -f config/samples/secrets_v1alpha1_dopplersecret.yaml

Check that the associated Kubernetes secret has been created:

# List all Kubernetes secrets created by the Doppler operator
kubectl describe secrets --selector=secrets.doppler.com/subtype=dopplerSecret

The operator continuously watches for secret updates from Doppler and when detected, automatically and instantly updates the associated secret.

Next, we'll cover how to configure a deployment to use the Kubernetes secret and enable auto-reloading for Deployments.

Deployments

To use the secret created by the operator, we can use the managed secret in one of three ways. These methods are also covered in greater detail in the Kubernetes Secrets documentation.

`envFrom`

The envFrom field will populate a container's environment variables using the secret's Key-Value pairs:

envFrom:
  - secretRef:
      name: doppler-test-secret # Kubernetes secret name

`valueFrom`

The valueFrom field will inject a specific environment variable from the Kubernetes secret:

env:
  - name: MY_APP_SECRET # The name of the environment variable exposed in the container
    valueFrom:
      secretKeyRef:
        name: doppler-test-secret # Kubernetes secret name
        key: MY_APP_SECRET # The name of the key in the Kubernetes secret

`volume`

The volume field will create a volume that is populated with files containing the Kubernetes secret:

volumes:
  - name: secret-volume
    secret:
      secretName: doppler-test-secret # Kubernetes secret name

Your deployment can use this volume by mounting it to the container's filesystem:

volumeMounts:
  - name: secret-volume
    mountPath: /etc/secrets
    readOnly: true

Automatic Redeployments

In order for the operator to reload a deployment, three things must be true:

The deployment is in the same namespace as the managed secret
The deployment is of the Deployment resource type (which is currently the only type that our operator supports). If you need automatic redeployments with other resource types, consider using the Reloader controller.
The deployment has the secrets.doppler.com/reload annotation set to 'true' (string)
The deployment uses the managed secret

Here's an example of the reload annotation:

annotations:
  secrets.doppler.com/reload: 'true'

The Doppler Kubernetes operator reloads deployments by updating an annotation with the name secrets.doppler.com/secretsupdate.<KUBERNETES_SECRET_NAME>. When this update is made, Kubernetes will automatically redeploy your pods according to the deployment's configured strategy.

Examples

Complete examples of these different deployment configurations can be found below:

If you've named your managed Kubernetes secret doppler-test-secret in the previous step, you can apply any of these examples directly:

kubectl apply -f config/samples/deployment-envfrom.yaml
kubectl rollout status -w deployment/doppler-test-deployment-envfrom

Once the Deployment has completed, you can view the logs of the test container:

kubectl logs -lapp=doppler-test --tail=-1

Setup is complete! To test the sync behavior, modify a secret in the Doppler dashboard and wait 60 seconds. Run the logs command again (or use the watch command) to see the pods automatically restart with the new secret data.

Name Transformers

Name Transformers allow secret names to be transformed from Doppler's UPPER_SNAKE_CASE format into any of the following environment variable compatible formats:

Type	Default	Transform
camel	API_KEY	apiKey
upper-camel	API_KEY	ApiKey
lower-snake	API_KEY	api_key
tf-var	API_KEY	TF_VAR_api_key
dotnet-env	SMTP__USER_NAME	Smtp__UserName
lower-kebab	API_KEY	api-key

Simply add the nameTransformer field with any of the above types:

apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: dopplersecret-test
  namespace: doppler-operator-system
spec:
  tokenSecret:
    name: doppler-token-secret
  managedSecret:
    name: doppler-test-secret
    namespace: default
  nameTransformer: dotnet-env

The nameTransformer values are also validated prior to admission to prevent transformation failures.

Processors

By default, the Operator base64 encodes Doppler secret values when creating the managed Opaque Kubernetes secret as Key / Value pairs. However, instances may arise where you do not wish this base64 encoding to occur, such as when a binary .p12 key file has already been base64 encoded in Doppler and double base64 encoding it would prevent Kubernetes from being mounting it in its original binary format.

Secret processors allow you to customize the default encoding behavior.

To override the default plain processor, add a processors: map which specifies the secret name and the processor to apply which in this case, is base64:

apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: doppler-secret-pkcs12 
  namespace: doppler-operator-system
spec:
  tokenSecret:
    name: doppler-token-secret
  managedSecret:
    name: doppler-pkcs12
    namespace: default
    type: kubernetes.io/tls
  processors:
    PKCS12_CERT:
      # Prevent double base64 encoding the already base64 encoded value from Doppler
      type: base64
    PKCS12_KEY:
      type: base64

You can also override the default name that's used when creating the secret in Kubernetes. By default, it will use the same uppercase secret name found in Doppler. To adjust this, add the asName parameter to the processor:

processors:
  PKCS12_CERT:
    type: base64
    asName: tls.crt
  PKCS12_KEY:
    type: base64
    asName: tls.key

Note that using the plain type combined with the asName parameter will let you rename any secret during the sync without affecting the value being synced.

For a complete example, check out our tutorial on using the Doppler Kubernetes Operator for Managing PKCS12 Certificates and the README in GitHub.

Processor Types

At present, only plain and base64 processors exist. Reach out to [email protected] if there is another processor you'd like us to support.

Secret Subsets

You can specify a subset of secrets from a config that you'd like to sync by using the secrets field. Note that the DOPPLER_* secrets will still appear in addition to any secrets you specify here.

apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: dopplersecret-test
  namespace: doppler-operator-system
spec:
  tokenSecret:
    name: doppler-token-secret
  secrets:
    - HOSTNAME
    - PORT
  managedSecret:
    name: doppler-test-secret
    namespace: default

Download Formats

Instead of the standard Key / Value pairs, you can download secrets as a single file in the following formats:

json
dotnet-json
env
yaml
docker

When format is specified, a single DOPPLER_SECRETS_FILE key is set in the created secret with the string contents of the downloaded file.

Simply add the format field:

apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: dotnet-webapp-appsettings
  namespace: doppler-operator-system
spec:
  tokenSecret:
    name: doppler-token-dotnet-webapp
    namespace: doppler-operator-system
  managedSecret:
    name: dotnet-webapp-appsettings
    namespace: default
  format: dotnet-json

You can then configure your deployment spec to mount the file at the desired path:

...
    spec:
      containers:
        - name: dotnet-webapp
          volumeMounts:
            - name: doppler
              mountPath: /usr/src/app/secrets 
              readOnly: true
      volumes:
        - name: doppler
          secret:
            secretName: dotnet-webapp-appsettings  # Managed secret name
            optional: false
            items:
              - key: DOPPLER_SECRETS_FILE # Hard-coded by Operator when format specified
                path: appsettings.json # Name or path to file name appended to container mountPath

Adjusting Sync Interval

📘
We recommend leaving this interval at the default value unless you run into API rate limit issues.

By default, the operator will poll for secret changes once every 60 seconds for each DopplerSecret you have. This can be adjusted using the resyncSeconds field in your DopplerSecret CRD.

apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: dopplersecret-test
  namespace: doppler-operator-system
spec:
  tokenSecret:
    name: doppler-token-secret
  managedSecret:
    name: doppler-test-secret
    namespace: default
  resyncSeconds: 120

Failure Strategy and Troubleshooting

Inspecting Status

If the operator fails to fetch secrets from the Doppler API (e.g. a connection problem or invalid service token), no changes are made to the managed Kubernetes secret or your deployments. The operator will continue to attempt to reconnect to the Doppler API indefinitely.

The DopplerSecret uses status.conditions to report its current state and any errors that may have occurred.

In this example, our Doppler service token has been revoked and the operator is reporting an error condition:

$ kubectl describe dopplersecrets -n doppler-operator-system
Name:         dopplersecret-test
Namespace:    doppler-operator-system
Labels:       <none>
Annotations:  <none>
API Version:  secrets.doppler.com/v1alpha1
Kind:         DopplerSecret
Metadata:
  ...
Spec:
  ...
Status:
  Conditions:
    Last Transition Time:  2021-06-02T15:46:57Z
    Message:               Secret update failed: Doppler Error: Invalid Service token
    Reason:                Error
    Status:                False
    Type:                  secrets.doppler.com/SecretSyncReady
    Last Transition Time:  2021-06-02T15:46:57Z
    Message:               Deployment reload has been stopped due to secrets sync failure
    Reason:                Stopped
    Status:                False
    Type:                  secrets.doppler.com/DeploymentReloadReady
Events:                    <none>

You can safely modify your token Kubernetes secret or DopplerSecret at any time. To update our Doppler service token, we can modify our token Kubernetes secret directly and the changes will take effect immediately.

The DopplerSecret resource manages the managed Kubernetes secret but does not officially own it. Therefore, deleting a DopplerSecret will not automatically delete the managed secret.

Included Tools

tools/get-secret.sh - fetch and decode a Kubernetes secret
tools/operator-logs.sh - wait for the operator deployment to roll out, then tail the logs

Uninstall

To uninstall the operator, first delete any DopplerSecret resources and any referenced Kubernetes secrets that are no longer needed.

kubectl delete dopplersecrets --all --all-namespaces
kubectl delete secret doppler-token-secret -n doppler-operator-system

If you installed the operator with Helm, you can use helm uninstall to remove the installation resources. Otherwise, run the following command:

kubectl delete -f https://github.com/DopplerHQ/kubernetes-operator/releases/latest/download/recommended.yaml

Development

This project uses the Operator SDK.

When developing locally, you can run the operator using:

make install run

See the Operator SDK Go Tutorial for more information.

Release

This project is released with Github Actions. Adding a Github Release will start an action which builds the operator image and publishes it to DockerHub. Tag names should match the pattern vX.X.X.

Prerequites

Kubernetes Secret Encryption at Rest

Install

Using Helm

Using kubectl

Upgrade

Using Helm

Using kubectl

Secrets Sync

Doppler Token Secret

🚧

DopplerSecret CRD

🚧

Deployments

envFrom

valueFrom

volume

Automatic Redeployments

Examples

Name Transformers

Processors

Processor Types

Secret Subsets

Download Formats

Adjusting Sync Interval

📘

Failure Strategy and Troubleshooting

Inspecting Status

Included Tools

Uninstall

Development

Release

Using `kubectl`

Using `kubectl`

`envFrom`

`valueFrom`

`volume`