AWS Secrets Rotation

Doppler rotates secrets that reside within AWS by utilizing AWS Lambda as a proxy. There's no need to expose your services to the internet or grant Doppler access to your services.

Requirements

Overview

To rotate secrets for any system within AWS, Doppler invokes a Lambda function that you create and own within your cloud account using a minimally scoped IAM role. Each Lambda function is open source, signed, and hosted in S3. Each Lambda invocation payload is signed and verified using our public key, which is also hosted in S3.

Primitives

Each AWS rotated secret, regardless of the type, utilizes the same AWS primitives.

PrimitiveUse
LambdaPerforms the secret rotation
Lambda Execution Role & PolicyAuthorization context the Lambda is executed in
Assumed IAM Role & PolicyRole Doppler assumes into your AWS account with in order to execute the Lambda function
Doppler Code Signing ProfileDoppler signing profile which validates the authenticity of the Lambda code
Doppler Public KeyUsed to validate the payload passed to the Lambda function
S3 Gateway EndpointConnection between the Lambda function and the S3-hosted Doppler public key. Required when a Lambda function is in a VPC.

For each primitive, there may be customization required depending on your architecture - e.g. VPC peering. As well, the service you are rotating may need its own configuration, which will be covered in its own dedicated doc.

Supported Services

  • MySQL (RDS or self-hosted)

Rotated Secret Creation

  1. Navigate to the secrets config that the rotated secret will reside in
  2. Go to the Advanced Secrets tab
  3. Select New Rotated Secret
  1. Select the service you'd like to rotate and then select AWS

Lambda Setup

Executor Role

The Lambda executor role is the authorization context Lambda will execute in; this is not the role Doppler will assume into to invoke the Lambda. We describe the minimum grants below, but there's a chance your role will require additional grants based on what service it is rotating and how you've architected your infrastructure.

  1. In your AWS account, navigate to IAM Roles
  2. Create Role
  3. Use AWS service for Trusted entity type
  4. Use Lambda for Use Case
  5. Under permissions, select AmazonS3ReadOnlyAccess and AWSLambdaVPCAccessExecutionRole
  6. Name your role and optionally provide a description
  7. Create

Lambda Function

Each service that is rotated uses a Lambda function powered by an open source rotator. You can use the same Lambda function to rotate n instances of the same service - i.e. you can use a single MySQL Lambda rotator to rotate as many MySQL secrets on as many MySQL instances as you'd like.

  1. In your AWS account - most likely in the region of the app or service you're rotating - navigate to Lambda
  2. Create a new function
  3. Name your function, ensuring the name begins with Doppler_. This allows Doppler to use more finely scope access.
  4. Set Node.js 16.x as the runtime
  5. Expand the permissions section to expose the executor role configuration. Choose the role you created in the section above .
  6. Expand Advanced settings. Ensure Enable Code signing is checked and configure the settings as follows
    6.1 Code signing configuration: Create new configuration
    6.2 Description: Doppler code signing profile
    6.3 Signing profile ARN: arn:aws:signer:us-east-1:299900769157:/signing-profiles/DopplerSecretAgents/7zfqycFf2B
    6.4 Signature validation policy for code: Enforce
    6.5 Code location: Select the appropriate S3 URL from the table
  1. Enable VPC: You will most likely want your function to live in the same VPC as the resource being rotated. If not, you must ensure it can peer into the applicable VPC to reach the service being rotated.
  2. Create your function
  3. Scroll to the Runtime settings section and click edit. Update the handler with the corresponding value from the table below
ServiceHandler
MySQL (RDS or self-hosted)apps/aws-mysql-rotator/dist/app.handler

IAM Role for Doppler to assume into

Policy

  1. In your AWS account, navigate to IAM Policies
  2. Click Create Policy
  3. Click the JSON tab
  4. Enter the following policy below after adding your Lambda's ARN in the resource field.
    1. Note: lambda:UpdateFunctionCode is optional but necessary if you'd like Doppler to be able to perform automatic updates. Otherwise, you'll need to manually upgrade to new versions.
  5. Optionally add any tags
  6. Name your policy and click create policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "lambda:GetFunction",
        "lambda:InvokeFunction",
        "lambda:UpdateFunctionCode"
      ],
      "Effect": "Allow",
      "Resource": "YOUR_LAMBDA_ARN"
    },
     {
            "Action": [
                "s3:GetObject"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::doppler-keys-*/*"
        }
  ]
}

Role

  1. In your AWS account, navigate to IAM Roles
  2. Click Create Role
  3. Select AWS account for Trusted entity type
  4. Check Another AWS account
  5. For Account ID enter 299900769157
  6. Check Require external ID
    1. Enter your workplace slug for the External ID value. You can obtain your workplace slug by visiting the Doppler dashboard. In the URL, grab the value after /workplace/
  7. Attach the policy you created above.

S3 Gateway Endpoint (Optional)

If you're VPC does not have outbound access enabled, a S3 Gateway Endpoint is required to reach Doppler's public key. An AWS Gateway Endpoint enables your Lambda to communicate with our S3 bucket in order to retrieve our public key for payload signing verification.

  1. In your AWS account, navigate to VPC
  2. Click Endpoints and Create Endpoint
  3. Name your endpoint
  4. Leave Service Category as AWS Services
  5. In the Services section search for S3
  6. Select the endpoint with a type of Gateway. It will include the region in its name, e.g. com.amazonaws.us-east-1.s3
  7. Select the VPC that the Lambda rotator resides in
  8. Select the appropriate route table from your VPC
  9. A policy is not required

Service Configuration

After the Lambda function is setup, you can move on to the next screen, where you'll configure service-specific settings. Select the appropriate page below to carry on

Automatic Lambda function updates

If the role provided to Doppler is given lambda:UpdateFunctionCode, we will automatically apply updates when available. If the lambda:UpdateFunctionCode is omitted, updates will need to be performed manually.

Manual Lambda function updates

If you do not grant lambda:UpdateFunctionCode to the Doppler IAM role, you will be responsible for manually updating your Lambda function.

  1. In the AWS console, navigate to the Lambda function
  2. In the Code Source section, click the Upload from drop-down and select Amazon S3 Location
  3. Enter the appropriate S3 URL for your AWS region from the here