Automate RDS Secret management using AWS Secret Manager

Have you ever tried creating an RDS cluster/instance using terraform?

If yes, then I’m sure the question to manage RDS credentials easily and securely has bogged your mind.

If not, well hang tight I’ve got a simple two-step process to improve the security of your infrastructure in terms of managing RDS credentials and other secrets.

A simple RDS cluster can be created using the following block

resource "aws_rds_cluster" "postgresql" {
  cluster_identifier      = "aurora-cluster-demo"
  engine                  = "aurora-postgresql"
  availability_zones      = ["us-west-2a", "us-west-2b", "us-west-2c"]
  database_name           = "mydb"

  # How should we manage the username and the password? 
  master_username         = "???"
  master_password         = "???"
}

The username and password are required arguments, and can’t be omitted. So, in this blog post, first we will start with briefly comparing different ways to store RDS credentials. Later we will be using AWS Secret manager to store RDS credentials, and will try to automate the entire process through terraform.

Different Ways to Manage RDS Secret

Method 0: Storing secrets in plain text.

Sample code:

resource "aws_rds_cluster" "postgresql" {
  cluster_identifier      = "aurora-cluster-demo"
  engine                  = "aurora-postgresql"
  availability_zones      = ["us-west-2a", "us-west-2b", "us-west-2c"]
  database_name           = "mydb"

  master_username         = "foo"
  master_password         = "bar"
}

Pros: Can be done, if a single person is managing the infra or the code is not checked in any version control system.

Cons: Storing secrets in plain text is a bad idea because

Method 1: Storing secrets in environment variables.

Sample code:


variable "username" {
  type        = string
}
variable "password" {
  type        = string
}

resource "aws_rds_cluster" "postgresql" {
  cluster_identifier      = "aurora-cluster-demo"
  engine                  = "aurora-postgresql"
  availability_zones      = ["us-west-2a", "us-west-2b", "us-west-2c"]
  database_name           = "mydb"

  master_username         = var.username
  master_password         = var.password
}
# Set username and password as environment variables
export TF_VAR_username=foo
export TF_VAR_password=bar

Pros: Safer than storing secrets in plain text.

Cons: This technique helps avoid storing secrets in plain text. But doesn’t provide any managing or rotating mechanism. For that, we need to rely on other third-party password management tools such as 1Password, or LastPass. Which adds another overhead to manage them.

Also, everyone working on the code has to take a few extra steps to either manually set these environment variables or run a wrapper script, to fetch values from the third-party password manager.

Method 2: Storing in Encrypted files.

In this technique, the credentials are encrypted, and the ciphertext is checked in the version control system. The problem here is for encrypting the credentials we need some key. The key is another secret, which needs to be protected. Now we are dealing with two secrets, instead of one.

The secret used for encrypting the credentials can be stored in AWS KMS, GCP KMS, etc.

Pros:

Cons:

Method 3: Storing secrets in AWS Secret Manager

In this method, the secrets are stored in a dedicated secret manager specially designed for managing secrets.

Storing and Managing secrets from the AWS console is pretty straightforward, and a bunch of blogs and articles can be found on the internet. Refer this doc for more information. Manage RDS Secrets through AWS Console

Here, we will go one step ahead and try to automate the entire process as much as possible. As discussed earlier, storing and managing secrets through terraform is a two-step process. In the first step, we will be creating an AWS secret for storing the credentials, and a rotation lambda function for rotating the password. Later in the second step will enable the RDS to read credentials from the secret manager instead of reading it from the tfvars file.

To create the secret, use the following snippet;

resource "aws_secretsmanager_secret" "secret" {
  description         = "Secrets for ${var.rds_name}"
  name                = "${var.rds_name}-postgres-secret"
  rotation_lambda_arn = aws_lambda_function.rotate_code_postgres.arn
  rotation_rules {
  // RDS password will be rotated after 30 days automatically.  
    automatically_after_days = "30"
  }
}

resource "aws_secretsmanager_secret_version" "secret" {
  lifecycle {
    ignore_changes = [
      secret_string
    ]
  }
  secret_id     = aws_secretsmanager_secret.secret.id
  secret_string = <<EOF
{
  "username": "${var.rds_master_username}",
  "password": "${var.rds_master_password}",
  "engine": "postgres",
  "host": "${var.rds_host}",
  "port": 5432,
  "dbClusterIdentifier": "${var.rds_cluster_identifier}",
  "db" : "${var.rds_name}"
}
EOF
}

This creates a secret, storing the following variables

Note: Only the password will be rotated by the lambda function and variables like host, port, etc play a crucial role in rotation thus needs to be stored in the secret itself.

Once the secret is created let’s proceed further by creating a lambda function and attaching it to the same VPC, subnet as of the RDS instance/ cluster. Also, the lambda function needs to be given access to read the created RDS secret from the secret manager.

To create the rotation lambda function, use the following snippet.

resource "aws_lambda_function" "rotate_code_postgres" {
  filename         = "${path.module}/rotate.zip"
  function_name    = "rds-rotation-lambda"
  role             = aws_iam_role.lambda_rotation.arn
  handler          = "lambda_function.lambda_handler"
  source_code_hash = filebase64sha256("${path.module}/rotate.zip")
  runtime          = "python3.7"
  vpc_config {
    subnet_ids         = var.subnet_ids
    security_group_ids = [var.rds_sg]
  }
  timeout     = 30
  description = "Conducts an AWS SecretsManager secret rotation for RDS using single user rotation scheme"
  environment {
    variables = {
      SECRETS_MANAGER_ENDPOINT = "https://secretsmanager.${data.aws_region.current.name}.amazonaws.com"
    }
  }
}

For giving the appropriate permission to the lambda function. Please refer this link.

Once RDS secret, the rotation lambda function is created, and the required permissions are given, we can verify the created secret from the AWS Console. See the image below for reference.

Rotate Secret

Let’s proceed further with rotating the secret from the AWS Console. By Clicking the Rotate secret immediately button. This will rotate the password for the RDS. Else, it will be automatically rotated after 30 days as defined in the rotation rule.

Once, done let’s enable the RDS to read credentials from the secret manager.

variable "enable_rds_secret_rotation" {
  type = bool
}

data "aws_secretsmanager_secret" "by_name" {
  count = var.enable_rds_secrets_rotation ? 1 : 0
  name  = "${var.db_name}-postgres-secret"
}

data "aws_secretsmanager_secret_version" "creds" {
  count     = var.enable_rds_secrets_rotation ? 1 : 0
  secret_id = try(data.aws_secretsmanager_secret.by_name[0].id, "")
}

locals {
  username = try(jsondecode(data.aws_secretsmanager_secret_version.creds[0].secret_string)["username"], var.master_username)
  password = try(jsondecode(data.aws_secretsmanager_secret_version.creds[0].secret_string)["password"], var.master_password)
}

resource "aws_db_instance" "postgres" {
  instance_class          = var.db_instance
  engine                  = var.db_engine
  engine_version          = var.db_engine_version
  multi_az                = var.enable_multi_az
  storage_type            = var.db_storage_type
  allocated_storage       = var.db_allocated_storage
  name                    = var.db_name
  username                = local.username
  password                = local.password
  backup_window           = var.db_backup_window
  backup_retention_period = var.db_backup_retention_period
  db_subnet_group_name    = aws_db_subnet_group.db_subnet.name
  vpc_security_group_ids  = [aws_security_group.rds_sg.id]
  skip_final_snapshot     = var.enable_skip_final_snapshot
  publicly_accessible     = var.enable_public_access
}

Key Points:

Pros:

Rotate Secret

Cons:

Conclusion

Some Useful resources

Github Link

AWS Secret Manager Doc

Discussion and feedback