Secrets Management

The ADMA infrastructure uses AWS Secrets Manager to securely store and inject sensitive credentials into ECS containers at runtime. Secrets are never stored in code or logs.

Secrets Architecture

Overview

┌──────────────────────────────────────────────────────────────┐
│  Terraform                                                   │
│  ├─ Creates secrets in Secrets Manager                      │
│  ├─ Generates random passwords                              │
│  └─ Configures IAM permissions                              │
└────────────────┬─────────────────────────────────────────────┘
                 │
                 ▼
┌──────────────────────────────────────────────────────────────┐
│  AWS Secrets Manager                                         │
│  ├─ /adma/prod/jwt-secret          (64-char random)         │
│  └─ arn:aws:rds:.../adma-prod-postgres (RDS-managed)        │
└────────────────┬─────────────────────────────────────────────┘
                 │
                 │ (Read via IAM role)
                 ▼
┌──────────────────────────────────────────────────────────────┐
│  ECS Task Definition                                         │
│  ├─ Execution Role: Can read secrets at task launch         │
│  └─ Container secrets: Injected as environment variables    │
└────────────────┬─────────────────────────────────────────────┘
                 │
                 ▼
┌──────────────────────────────────────────────────────────────┐
│  Running Container                                           │
│  ├─ JWT_SECRET=base64encodedvalue...                        │
│  └─ DB_PASSWORD=generatedpassword                           │
└──────────────────────────────────────────────────────────────┘

Secrets in Terraform

JWT Secret

The JWT signing secret is created and managed by Terraform:

# infrastructure/terraform/main.tf

# Create the secret container
resource "aws_secretsmanager_secret" "jwt" {
  name                    = "/${var.project_name}/${var.environment}/jwt-secret"
  description             = "JWT signing secret for ${local.name_prefix} backend"
  recovery_window_in_days = 7
}

# Generate a secure random value
resource "random_password" "jwt_secret" {
  length           = 64
  special          = true
  override_special = "_-+=@#%"
}

# Store the value in Secrets Manager
resource "aws_secretsmanager_secret_version" "jwt" {
  secret_id     = aws_secretsmanager_secret.jwt.id
  secret_string = random_password.jwt_secret.result
}

Key features:

recovery_window_in_days = 7 — Allows secret restoration within 7 days of deletion
length = 64 — Exceeds OWASP minimum recommendation (32 bytes)
override_special — Limits special chars to avoid shell escaping issues

Secret Rotation: JWT secret is generated once during Terraform apply. To rotate, use terraform taint random_password.jwt_secret and re-apply.

RDS Database Password

RDS password is managed by AWS using native integration:

# modules/rds/main.tf

resource "aws_db_instance" "this" {
  identifier                  = "${local.name_prefix}-postgres"
  engine                      = "postgres"
  engine_version              = var.db_engine_version
  instance_class              = var.db_instance_class
  
  db_name                     = var.db_name
  username                    = var.db_username
  
  # AWS manages the password automatically
  manage_master_user_password = true
  
  # ... other settings ...
}

How it works:

RDS generates a secure random password
Password is stored in Secrets Manager at: arn:aws:secretsmanager:region:account:secret:rds!db-xxxxx
Password can be rotated automatically via Secrets Manager rotation
Terraform references the secret ARN for ECS injection

Do NOT use password argument: Setting password directly in Terraform stores it in state file. Always use manage_master_user_password = true.

IAM Permissions

The ECS Execution Role needs permission to read secrets:

Execution Role Policy

# modules/iam/main.tf

# Base execution role
resource "aws_iam_role" "execution" {
  name               = "${local.name_prefix}-ecs-execution-role"
  assume_role_policy = data.aws_iam_policy_document.ecs_task_assume.json
}

# Attach AWS managed policy for basic ECS functionality
resource "aws_iam_role_policy_attachment" "execution_managed" {
  role       = aws_iam_role.execution.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

# Custom policy for secrets access
data "aws_iam_policy_document" "execution_secrets" {
  statement {
    sid    = "ReadOnlyRequiredSecrets"
    effect = "Allow"

    actions = [
      "secretsmanager:GetSecretValue",
      "secretsmanager:DescribeSecret",
    ]

    resources = [
      var.jwt_secret_arn,
      var.rds_master_secret_arn,
    ]
  }
}

resource "aws_iam_role_policy" "execution_secrets" {
  name   = "${local.name_prefix}-ecs-execution-secrets"
  role   = aws_iam_role.execution.id
  policy = data.aws_iam_policy_document.execution_secrets.json
}

Least Privilege: The execution role can ONLY read the two specific secrets needed for this application. It cannot list or read other secrets in the account.

Permissions Breakdown

Permission	Why Needed	Used By
`secretsmanager:GetSecretValue`	Read secret value	ECS agent during task launch
`secretsmanager:DescribeSecret`	Get secret metadata	ECS agent validation
Specific `resources` ARNs	Limit access scope	Prevents access to other secrets

Secret Injection in ECS

Backend Task Definition

Secrets are injected as environment variables in the ECS task definition:

# modules/ecs/main.tf

resource "aws_ecs_task_definition" "backend" {
  family                   = "${local.name_prefix}-backend"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = tostring(var.backend_task_cpu)
  memory                   = tostring(var.backend_task_memory)
  execution_role_arn       = var.execution_role_arn
  task_role_arn            = var.task_role_arn

  container_definitions = jsonencode([
    {
      name      = "backend"
      image     = "${var.backend_repository_url}:${var.backend_image_tag}"
      essential = true
      
      # Regular environment variables
      environment = [
        {
          name  = "DB_HOST"
          value = var.db_endpoint
        },
        {
          name  = "DB_PORT"
          value = tostring(var.db_port)
        },
        {
          name  = "DB_NAME"
          value = var.db_name
        },
        {
          name  = "DB_USERNAME"
          value = var.db_username
        },
        # ... other non-sensitive vars ...
      ]
      
      # Secrets from Secrets Manager
      secrets = [
        {
          name      = "DB_PASSWORD"
          valueFrom = "${var.db_password_secret_arn}:password::"
        },
        {
          name      = "JWT_SECRET"
          valueFrom = var.jwt_secret_arn
        }
      ]
      
      # ... logging, health check, etc. ...
    }
  ])
}

Secret Reference Format

Full Secret
JSON Key
Versioned Secret

For simple string secrets (like JWT secret):

{
  "name": "JWT_SECRET",
  "valueFrom": "arn:aws:secretsmanager:region:account:secret:name"
}

ECS retrieves the entire secret value.

For secrets with JSON structure (like RDS-managed passwords):

{
  "name": "DB_PASSWORD",
  "valueFrom": "arn:aws:secretsmanager:region:account:secret:name:password::"
}

Format: <secret-arn>:<json-key>::RDS secrets have structure:

{
  "username": "appuser",
  "password": "generatedpassword",
  "engine": "postgres",
  "host": "adma-prod-postgres.xxxxx.rds.amazonaws.com",
  "port": 5432,
  "dbname": "urlshortener"
}

To pin to a specific secret version:

{
  "name": "JWT_SECRET",
  "valueFrom": "arn:aws:secretsmanager:region:account:secret:name:AWSCURRENT:"
}

Stages:

AWSCURRENT — Latest version (default)
AWSPREVIOUS — Previous version
<version-id> — Specific version UUID

Secret Lifecycle

Initial Creation

# Run Terraform to create secrets
cd infrastructure/terraform
terraform apply

# Verify secrets were created
aws secretsmanager list-secrets --query 'SecretList[?Name==`/adma/prod/jwt-secret`]'

# View secret metadata (not value)
aws secretsmanager describe-secret --secret-id /adma/prod/jwt-secret

Viewing Secret Values

Production Safety: Never log or print secret values. Use these commands only for troubleshooting in secure environments.

AWS CLI
AWS Console
From ECS Container

# Get JWT secret
aws secretsmanager get-secret-value \
  --secret-id /adma/prod/jwt-secret \
  --query SecretString \
  --output text

# Get RDS password (JSON)
aws secretsmanager get-secret-value \
  --secret-id arn:aws:secretsmanager:region:account:secret:rds!db-xxxxx \
  --query SecretString \
  --output text | jq -r .password

Navigate to AWS Secrets Manager
Find secret /adma/prod/jwt-secret
Click “Retrieve secret value”
Click “Show” to reveal plaintext

Note: Console access is logged in CloudTrail.

If you need to verify a secret inside a running container:

# Enable ECS Exec (set enable_ecs_exec=true in tfvars)

# Connect to container
aws ecs execute-command \
  --cluster adma-prod-ecs \
  --task <task-id> \
  --container backend \
  --command "/bin/sh" \
  --interactive

# Inside container, check environment variable
echo $JWT_SECRET | head -c 20
# Output: base64encodedstring...

Secret Rotation

Manual JWT Secret Rotation

To rotate the JWT secret:

# Method 1: Taint and re-apply
terraform taint random_password.jwt_secret
terraform apply

# Method 2: Update manually via AWS CLI
aws secretsmanager update-secret \
  --secret-id /adma/prod/jwt-secret \
  --secret-string "$(openssl rand -base64 64 | tr -d '\n')"

# Force ECS to restart tasks with new secret
aws ecs update-service \
  --cluster adma-prod-ecs \
  --service adma-prod-backend \
  --force-new-deployment

Impact: Rotating JWT secret invalidates all existing user tokens. Users will need to re-authenticate.

Automatic RDS Password Rotation

AWS Secrets Manager can automatically rotate RDS passwords:

# Add to modules/rds/main.tf

resource "aws_secretsmanager_secret_rotation" "rds" {
  secret_id           = aws_db_instance.this.master_user_secret[0].secret_arn
  rotation_lambda_arn = aws_lambda_function.rds_rotation.arn

  rotation_rules {
    automatically_after_days = 30
  }
}

Recommended: Enable automatic rotation for production RDS instances. AWS provides managed Lambda functions for RDS rotation.

Security Best Practices

Secret Storage

Never commit secrets to Git

❌ Bad:

environment = [
  {
    name  = "JWT_SECRET"
    value = "my-hardcoded-secret-key"  # NEVER DO THIS
  }
]

✅ Good:

secrets = [
  {
    name      = "JWT_SECRET"
    valueFrom = aws_secretsmanager_secret.jwt.arn
  }
]

Use encryption at rest

Secrets Manager encrypts all secrets with AWS KMS:

resource "aws_secretsmanager_secret" "jwt" {
  name        = "/adma/prod/jwt-secret"
  kms_key_id  = aws_kms_key.secrets.id  # Optional: use custom KMS key
  
  # Default: Uses AWS-managed key (aws/secretsmanager)
}

Minimize IAM permissions

Grant least privilege access:✅ Do:

Specify exact secret ARNs in IAM policies
Use secretsmanager:GetSecretValue (not *)
Separate execution role from task role

❌ Don’t:

Grant secretsmanager:* permissions
Use Resource: "*" in policies
Share execution roles across unrelated services

Enable audit logging

CloudTrail logs all Secrets Manager API calls:

{
  "eventName": "GetSecretValue",
  "requestParameters": {
    "secretId": "/adma/prod/jwt-secret"
  },
  "userIdentity": {
    "principalId": "AIDAI...:adma-prod-backend-task"
  },
  "eventTime": "2026-03-04T10:30:00Z"
}

Monitor for:

Unauthorized access attempts
Frequent secret retrievals (possible leak)
Manual console access (should be rare)

Secrets in Logs

Critical: ECS logs (awslogs) do NOT automatically redact secrets. Ensure your application code never logs sensitive values.

Backend logging configuration:

# src/main/resources/application.yml
logging:
  pattern:
    console: "%d{yyyy-MM-dd HH:mm:ss} - %msg%n"
  level:
    org.springframework: INFO
    
spring:
  datasource:
    url: jdbc:postgresql://${DB_HOST}:${DB_PORT}/${DB_NAME}?sslmode=require
    username: ${DB_USERNAME}
    password: ${DB_PASSWORD}  # Spring Boot NEVER logs this

Test for secret leaks:

# Check CloudWatch logs for secrets
aws logs filter-log-events \
  --log-group-name /ecs/adma-prod-backend \
  --filter-pattern "password" \
  --max-items 50

# Should return zero matches

Troubleshooting

Common Issues

Task fails with 'ResourceInitializationError: unable to pull secrets'

Symptoms: ECS task stops immediately with secrets errorCauses:

Execution role missing secretsmanager:GetSecretValue permission
Secret ARN is incorrect
VPC endpoint for Secrets Manager not configured
Security group blocks egress to VPC endpoint

Fix:

# Verify execution role policy
aws iam get-role-policy \
  --role-name adma-prod-ecs-execution-role \
  --policy-name adma-prod-ecs-execution-secrets

# Check VPC endpoint exists
aws ec2 describe-vpc-endpoints \
  --filters Name=service-name,Values=com.amazonaws.eu-west-1.secretsmanager

# Verify security group allows egress to endpoint
aws ec2 describe-security-group-rules \
  --filters Name=group-id,Values=sg-backend

Application cannot connect to database (wrong password)

Symptoms: Backend logs show “password authentication failed”Causes:

Wrong JSON key in secret reference (should be :password::)
Secret ARN points to wrong RDS instance
RDS password rotated but ECS task not restarted

Fix:

# Get RDS-managed secret ARN
aws rds describe-db-instances \
  --db-instance-identifier adma-prod-postgres \
  --query 'DBInstances[0].MasterUserSecret.SecretArn'

# Compare with ECS task definition secret ARN
aws ecs describe-task-definition \
  --task-definition adma-prod-backend \
  --query 'taskDefinition.containerDefinitions[0].secrets'

# Force new deployment to pick up rotated secret
aws ecs update-service \
  --cluster adma-prod-ecs \
  --service adma-prod-backend \
  --force-new-deployment

JWT tokens invalid after deployment

Symptoms: Users get 401 errors after infrastructure updateCause: JWT secret changed during Terraform applyPrevention:

# Prevent accidental secret recreation
resource "random_password" "jwt_secret" {
  length = 64
  
  lifecycle {
    ignore_changes = [length, special, override_special]
  }
}

Recovery: Users must re-authenticate to get new tokens signed with new secret.

Alternative: SSM Parameter Store

For non-critical configuration (not passwords), you can use SSM Parameter Store:

# Create parameter
resource "aws_ssm_parameter" "app_base_url" {
  name  = "/${var.project_name}/${var.environment}/app-base-url"
  type  = "String"
  value = var.app_base_url
}

# Reference in task definition
container_definitions = jsonencode([
  {
    secrets = [
      {
        name      = "APP_BASE_URL"
        valueFrom = aws_ssm_parameter.app_base_url.arn
      }
    ]
  }
])

# IAM permission
actions = [
  "ssm:GetParameters",
  "ssm:GetParameter",
]
resources = [aws_ssm_parameter.app_base_url.arn]

When to use SSM vs Secrets Manager:

Feature	SSM Parameter Store	Secrets Manager
Cost	Free (standard), $0.05/param/month (advanced)	$0.40/secret/month +$ 0.05/10k API calls
Rotation	Manual	Automatic with Lambda
Versioning	Limited (max 100 versions)	Unlimited
Encryption	Optional (KMS)	Always encrypted
Best for	Non-sensitive config	Passwords, API keys, certificates

Recommendation: Use Secrets Manager for passwords and credentials. Use SSM Parameter Store for application configuration.

Overview

Getting Started

Deployment

Infrastructure

Operations

Secrets Architecture

Overview

Secrets in Terraform

JWT Secret

RDS Database Password

IAM Permissions

Execution Role Policy

Permissions Breakdown

Secret Injection in ECS

Backend Task Definition

Secret Reference Format

Secret Lifecycle

Initial Creation

Viewing Secret Values

Secret Rotation

Manual JWT Secret Rotation

Automatic RDS Password Rotation

Security Best Practices

Secret Storage

Secrets in Logs

Troubleshooting

Common Issues

Alternative: SSM Parameter Store

Build docs developers (and LLMs) love

Overview

Getting Started

Deployment

Infrastructure

Operations

​Secrets Architecture

​Overview

​Secrets in Terraform

​JWT Secret

​RDS Database Password

​IAM Permissions

​Execution Role Policy

​Permissions Breakdown

​Secret Injection in ECS

​Backend Task Definition

​Secret Reference Format

​Secret Lifecycle

​Initial Creation

​Viewing Secret Values

​Secret Rotation

​Manual JWT Secret Rotation

​Automatic RDS Password Rotation

​Security Best Practices

​Secret Storage

​Secrets in Logs

​Troubleshooting

​Common Issues

​Alternative: SSM Parameter Store

Build docs developers (and LLMs) love

Secrets Architecture

Overview

Secrets in Terraform

JWT Secret

RDS Database Password

IAM Permissions

Execution Role Policy

Permissions Breakdown

Secret Injection in ECS

Backend Task Definition

Secret Reference Format

Secret Lifecycle

Initial Creation

Viewing Secret Values

Secret Rotation

Manual JWT Secret Rotation

Automatic RDS Password Rotation

Security Best Practices

Secret Storage

Secrets in Logs

Troubleshooting

Common Issues

Alternative: SSM Parameter Store