Proactively rotating secrets is widely accepted as necessary to maintain a strong security posture and mitigate risk. Given the hassle that can be involved with rotating secrets, it is often avoided.
When secrets need to be rotated immediately, like in the event of a breach or leak, panic often ensues. Building an exhaustive inventory of where a secret is being used is rarely achievable, let alone under pressure. Typically, this becomes an 'ask questions later' moment and the secret is blindly changed (which is often the right strategy).
Luckily, Doppler has developed a set of rotation capabilities that help you do all of this and more. Automatically.
Secret rotation is the act of updating a secret with a new value, either automatically at a defined cadence (preferred) or by manually triggering it.
The benefits of secrets rotation are many, but can be summarized by:
- Mitigate Risk: limiting the life time of a secret reduces risk in the event of a leak
- Respond Rapidly: quickly respond from a central location in the event of a leak, breach, or intrusion - a simple button push to rotate a secret (and soon secrets)
- Prevent leaks: prevent off-boarded employees from accessing systems they previously had access to
Historically, it's been difficult to rotate secrets for a number of reasons.
- Effort: the scale of secrets to be rotated can reach into the thousands for enterprise companies
- Downtime: If you're not utilizing a multi-secret strategy, rotating secrets can introduce downtime
- Sprawl: without a centralized SecretOps platform, it is difficult to guarantee the rotation will propagate to all the locations a secret is being used
Doppler's SecretOps platform enables teams to rotate secrets automatically or manually, regardless of system location.
Proxied rotation allows you to keep your services closed to the public internet while maintaining the flexibility of using Doppler to manage the rotation. Your cloud provider's serverless product (AWS Lambda or GCP Cloud Run) is utilized as the proxy to facilitate rotation.
- Doppler rotation agents are open source and auditable at any time
- IAM requirements are least-privilege
- The audit history lives in your cloud environment
- Systems are not exposed to the internet
Proxied rotation was designed to require the minimum possible entitlements; where entitlements are required, they can be granted with specificity using features such as IAM conditions.
- All rotation agents are open source
- If you don't want to use automatic upgrades, you can choose to manually upgrade functions once you've read the code
- The rotation agent code is signed and validated using Doppler's signing profile
- The invocation payload sent to the rotation agent function is signed and validated by the agent using Doppler's public key
- All communications occurs over HTTPS
API rotation utilizes a service's publicly available API to perform rotation
For Doppler to be able to rotate a secret within a service - proxied and API - the service must support:
- At least two active secret instances at a time (i.e. two database users)
- Programmatically creating & deleting or updating secrets
A two secret strategy refers to alternating between a pair of secrets to eliminate downtime. For any secret you wish to rotate, there will be two instances of the secret - two database users, two API keys, two IAM users, etc. One instance is the active secret instance, the other is the inactive secret instance.
When you make a request to Doppler, the active secret instance is returned. The inactive secret instance is still enabled in your service, but it is not accessible via Doppler; this allows you to discontinue use of it. Halfway through each rotation interval, the active and inactive instances are switched. As well, before being set to active, the secret value is rotated in your service.
The vast majority of services that are rotated utilizing proxied rotation use the updater method, meaning a secret instance's value is updated on a defined cadence. When you create the rotated secret, you provide initial secret values, which are a pair of secrets (typically credentials). This way, Doppler doesn't have to create new users in your service (i.e. database) - Doppler can just update the password.
Most but not all secrets rotated via API utilize the issuer method. At rotation time, a new secret instance is created and then a previous secret instance is deleted. When creating the rotated secret, a seed is not required - Doppler will facilitate the creation of every instance.
Depending on the service Doppler is connecting to, Doppler will leverage one of two types of rotation:
The rotation interval is how often, in days, a rotated secret instance is rotated. As mentioned in two secret strategy, when a secret instance transitions from inactive to active, its value is rotated.
- Issuer: at rotation time, a new secret instance is created and the previous one is deleted (or inactivated).
- Updater: at rotation time, the value of the existing secret instance is updated
Your interaction with Doppler doesn't change between the two types - it's purely based on what the integrating service supports. Note: Doppler prefers issuer as it makes auditability easier.
The rotation period should be set to the longest time in days, amongst all users of a secret, it takes for an application to restart. It is recommended to add a little buffer as well.
Example: three applications use a database credential. The application that restarts/redeploys the least often does so every four months. The rotation interval shouldn't be any less than four months or downtime could occur.
Preventing downtime when rotating secrets is achievable with the two secret strategy. What zero downtime means to your organization may vary, but any use-case should be achievable.
The two user strategy can initially seem complex, but it is extremely useful . As a recap:
- Each rotated secret is comprised of two secret instances, but only one secret instance is active at a time
- Active means that when the secret is requested from Doppler by a user, the active instance is the only one returned
- The inactive secret instance is still valid, but it is not returned when the rotated secret is requested from Doppler by the user.
- This gives your applications time to cease using it; this also makes selecting your rotation period critical
Our Kubernetes Operator supports automatically redeploying services when a secret changes. The operator will detect a secret rotation like any other secret change, redeploying services if you've configured it to do so.
Any time a change is made to a config, including a automatic and manual rotation, you can choose to fire a webhook event. You can utilize webhooks to restart the appropriate applications and services.
Every rotated secret also has a managing user. This user is used to authenticate when connecting to the service to perform rotation. By leveraging a managing user, Doppler is able to reduce the likelihood of an unrecoverable state - that is, we don't support 'updating yourself'.
Doppler highly recommends that the managing user is used only for rotation and no other purpose - that is, Doppler is the only place these credentials should live or be used.
At any time, a rotated secret can be immediately rotated. When performed, all the resulting actions that occur when automatically rotating still take place - webhook triggers, integration syncs, k8s operator detection, etc.
To immediately rotate a secret
- Navigate to the Doppler config containing the secret you would like to rotate
- Scroll down to the Rotated Secrets section
- Locate the secret you want to rotate and click the 3-dot icon in its row
- Select Rotate now
- Confirm the rotation.
Note: this will cause the rotation interval to be reset.
We use exponential back-off when we encounter an error trying to perform rotation. After a defined number of failures, we will send an email to your account. Your secrets will still remain active and your systems will not be impacted.
Like most other events in Doppler, rotations trigger an Activity Log item. Depending on which plan you're on, you can receive these events in a number of tools, including Slack, MSFT Teams, Sumo Logic, and Splunk.
As mentioned above, they also trigger webhook events.
You should assume that Doppler 'owns' the state of a secret once you create a rotated secret, especially for database credentials. If you configure a rotated secret in Doppler and then manually mutate the state in your system, Doppler will no longer be returning the 'correct' secret value, can no longer perform the proper assertions, and will pause rotation.
Secret rotation introduces a host of benefits that reduce risk and improve security posture. However, for organizations that are interested in further reducing risk, Dynamic Secrets can be a superior option. However, there are a number of trade-offs with Dynamic Secrets that must be taken into consideration including application architecture, latency, etc. If you have any questions or would like input, shoot us an email [email protected].
Updated 1 day ago