Rotated Secrets

📷

Requires an upgraded subscription

This feature is available with our Team and Enterprise plans. View our plans or book a demo for more details.

Proactively rotating secrets is widely accepted as necessary to maintain a strong security posture and mitigate risk. Given the hassle that can be involved with rotating secrets, it is often avoided.

When secrets need to be rotated immediately, like in the event of a breach or leak, panic often ensues. Building an exhaustive inventory of where a secret is being used is rarely achievable, let alone under pressure. Typically, this becomes an 'ask questions later' moment and the secret is blindly changed (which is often the right strategy).

Luckily, Doppler has developed a set of rotation capabilities that help you do all of this and more. Automatically.

Overview

Secret rotation is the act of updating a secret with a new value, either automatically at a defined cadence (preferred) or by manually triggering it.

Benefits

The benefits of secrets rotation are many, but can be summarized by:

• Mitigate Risk: limiting the lifetime of a secret reduces risk in the event of a leak
• Respond Rapidly: quickly respond from a central location in the event of a leak, breach, or intrusion - a simple button push to rotate a secret (and soon secrets)
• Prevent leaks: prevent off-boarded employees from accessing systems they previously had access to

Difficulty

Historically, it's been difficult to rotate secrets for a number of reasons.

• Effort: the scale of secrets to be rotated can reach into the thousands for enterprise companies
• Downtime: If you're not utilizing a multi-secret strategy, rotating secrets can introduce downtime
• Sprawl: without a centralized SecretOps platform, it is difficult to guarantee the rotation will propagate to all the locations a secret is being used

Doppler Secrets Rotation Types

Doppler's SecretOps platform enables teams to rotate secrets automatically or manually, regardless of system location.

Proxied

Proxied rotation allows you to keep your services closed to the public internet while maintaining the flexibility of using Doppler to manage the rotation. Your cloud provider's serverless product (AWS Lambda or GCP Cloud Run) is utilized as the proxy to facilitate rotation.

• Doppler rotation agents are open source and auditable at any time
• IAM requirements are least-privilege
• The audit history lives in your cloud environment
• Systems are not exposed to the internet

Proxied Rotation Security

Proxied rotation was designed to require the minimum possible entitlements; where entitlements are required, they can be granted with specificity using features such as IAM conditions.

Open Source + Signed Rotation Agents

• All rotation agents are open source
  • If you don't want to use automatic upgrades, you can choose to manually upgrade functions once you've read the code
  • The rotation agent code is signed and validated using Doppler's signing profile

Signed Payloads

• The invocation payload sent to the rotation agent function is signed and validated by the agent using Doppler's public key
• All communications occurs over HTTPS

API

API rotation utilizes a service's publicly available API to perform rotation

Rotation Requirements

For Doppler to be able to rotate a secret within a service - proxied and API - the service must support:

• At least two active secret instances at a time (i.e. two database users)
• Programmatically creating & deleting or updating secrets

Two secret strategy

A two secret strategy refers to alternating between a pair of secrets to eliminate downtime. For any secret you wish to rotate, there will be two instances of the secret - two database users, two API keys, two IAM users, etc. One instance is the active secret instance, the other is the inactive secret instance.

When you make a request to Doppler, the active secret instance is returned. The inactive secret instance is still enabled in your service, but it is not accessible via Doppler; this allows you to discontinue use of it. Halfway through each rotation interval, the active and inactive instances are switched. As well, before being set to active, the secret value is rotated in your service.

Two secret strategy with proxied rotation

The vast majority of services that are rotated utilizing proxied rotation use the updater method, meaning a secret instance's value is updated on a defined cadence. When you create the rotated secret, you provide initial secret values, which are a pair of secrets (typically credentials). This way, Doppler doesn't have to create new users in your service (i.e. database) - Doppler can just update the password.

Two secret strategy with API rotation

Most but not all secrets rotated via API utilize the issuer method. At rotation time, a new secret instance is created and then a previous secret instance is deleted. When creating the rotated secret, a seed is not required - Doppler will facilitate the creation of every instance.

Rotation Types: Issuer and Updater

Depending on the service Doppler is connecting to, Doppler will leverage one of two types of rotation:

The rotation interval is how often, in days, a rotated secret instance is rotated. As mentioned in two secret strategy, when a secret instance transitions from inactive to active, its value is rotated.

• Issuer: at rotation time, a new secret instance is created and the previous one is deleted (or inactivated).
• Updater: at rotation time, the value of the existing secret instance is updated

Your interaction with Doppler doesn't change between the two types - it's purely based on what the integrating service supports. Note: Doppler prefers issuer as it makes auditability easier.

Rotation Interval

The rotation period should be set to the longest time in days, amongst all users of a secret, it takes for an application to restart. It is recommended to add a little buffer as well.

Example: three applications use a database credential. The application that restarts/redeploys the least often does so every four months. The rotation interval shouldn't be any less than four months or downtime could occur.

Zero downtime rotation

Preventing downtime when rotating secrets is achievable with the two secret strategy. What zero downtime means to your organization may vary, but any use-case should be achievable.

Understanding the two secret strategy

The two user strategy can initially seem complex, but it is extremely useful . As a recap:

• Each rotated secret is comprised of two secret instances, but only one secret instance is active at a time
  • Active means that when the secret is requested from Doppler by a user, the active instance is the only one returned
  • The inactive secret instance is still valid, but it is not returned when the rotated secret is requested from Doppler by the user.
    • This gives your applications time to cease using it; this also makes selecting your rotation period critical

Kubernetes Automatic Restarts

Our Kubernetes Operator supports automatically redeploying services when a secret changes. The operator will detect a secret rotation like any other secret change, redeploying services if you've configured it to do so.

Webhook Triggers

Any time a change is made to a config, including a automatic and manual rotation, you can choose to fire a webhook event. You can utilize webhooks to restart the appropriate applications and services.

Integration Syncs

Any time a change is made to a config, including a automatic and manual rotation, we will re-sync your secrets to any configured integrations.

Managing User

Every rotated secret also has a managing user. This user is used to authenticate when connecting to the service to perform rotation. By leveraging a managing user, Doppler is able to reduce the likelihood of an unrecoverable state - that is, we don't support 'updating yourself'.

Doppler highly recommends that the managing user is used only for rotation and no other purpose - that is, Doppler is the only place these credentials should live or be used.

Immediate Rotation

At any time, a rotated secret can be immediately rotated. When performed, all the resulting actions that occur when automatically rotating still take place - webhook triggers, integration syncs, k8s operator detection, etc.

To immediately rotate a secret

1. Navigate to the Doppler config containing the secret you would like to rotate
2. Scroll down to the Rotated Secrets section
3. Locate the secret you want to rotate and click the 3-dot icon in its row
4. Select Rotate now
5. Confirm the rotation.

Note: this will cause the rotation interval to be reset.

Error Handling

We use exponential back-off when we encounter an error trying to perform rotation. After a defined number of failures, we will send an email to your account. Your secrets will still remain active and your systems will not be impacted.

Rotation Notifications

Like most other events in Doppler, rotations trigger an Activity Log item. Depending on which plan you're on, you can receive these events in a number of tools, including Slack, MSFT Teams, Sumo Logic, and Splunk.

As mentioned above, they also trigger webhook events.

State Ownership

You should assume that Doppler 'owns' the state of a secret once you create a rotated secret, especially for database credentials. If you configure a rotated secret in Doppler and then manually mutate the state in your system, Doppler will no longer be returning the 'correct' secret value, can no longer perform the proper assertions, and will pause rotation.

Rotation vs. Dynamic secrets

Secret rotation introduces a host of benefits that reduce risk and improve security posture. However, for organizations that are interested in further reducing risk, Dynamic Secrets can be a superior option. However, there are a number of trade-offs with Dynamic Secrets that must be taken into consideration including application architecture, latency, etc. If you have any questions or would like input, shoot us an email [email protected].