Secret Management

Overview

LiteLLM integrates with multiple secret management systems to securely store and retrieve API keys and sensitive configuration. This eliminates hardcoded credentials and enables centralized secret management.

Supported Secret Managers

AWS Secrets Manager: Full-featured AWS secret storage
Azure Key Vault: Microsoft Azure secret management
Google Cloud Secret Manager: GCP secret storage
HashiCorp Vault: Enterprise secret management
CyberArk: Enterprise privileged access management
Google KMS: Encryption key management
AWS KMS: AWS encryption service
Custom: Build your own integration

Quick Start

Choose Your Secret Manager

Install required dependencies:

# AWS Secrets Manager
pip install boto3>=1.28.57

# Azure Key Vault
pip install azure-identity azure-keyvault-secrets

# Google Secret Manager
pip install google-cloud-secret-manager

# HashiCorp Vault
pip install hvac

Configure Environment

Set up authentication:

# AWS
export AWS_REGION_NAME="us-east-1"
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"

# Azure
export AZURE_KEY_VAULT_URI="https://your-vault.vault.azure.net/"

# Google Cloud
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/credentials.json"

# HashiCorp Vault
export VAULT_ADDR="https://vault.example.com"
export VAULT_TOKEN="your-vault-token"

Initialize Secret Manager

Configure LiteLLM to use your secret manager:

from litellm.secret_managers import AWSSecretsManagerV2
import litellm

# Initialize AWS Secrets Manager
litellm.secret_manager_client = AWSSecretsManagerV2(
    aws_region_name="us-east-1"
)

litellm._key_management_system = "aws_secret_manager"

Use Secrets in API Calls

Reference secrets by name:

# Secret name format: os.environ/SECRET_NAME
response = await litellm.acompletion(
    model="gpt-4",
    api_key="os.environ/OPENAI_API_KEY",  # Fetched from secret manager
    messages=[{"role": "user", "content": "Hello!"}]
)

AWS Secrets Manager

Setup

from litellm.secret_managers.aws_secret_manager_v2 import AWSSecretsManagerV2
import litellm

# Initialize
litellm.secret_manager_client = AWSSecretsManagerV2(
    aws_region_name="us-east-1",
    aws_role_name="optional-role-arn",  # For cross-account access
    aws_session_name="litellm-session"
)

litellm._key_management_system = "aws_secret_manager"

Read Secrets

from litellm.secret_managers.aws_secret_manager_v2 import AWSSecretsManagerV2

sm = AWSSecretsManagerV2(aws_region_name="us-east-1")

# Read single secret
api_key = await sm.async_read_secret(
    secret_name="openai-api-key"
)

print(api_key)

Write Secrets

# Create new secret
response = await sm.async_write_secret(
    secret_name="new-api-key",
    secret_value="sk-1234567890",
    description="OpenAI API key for production",
    tags={"Environment": "Production", "Team": "ML"}
)

print(f"Secret ARN: {response['ARN']}")

Rotate Secrets

# Rotate by creating new version and deleting old
response = await sm.async_rotate_secret(
    current_secret_name="old-api-key",
    new_secret_name="new-api-key",
    new_secret_value="sk-new-value"
)

# Old secret scheduled for deletion (7 day recovery window)

Delete Secrets

# Schedule deletion with recovery window
response = await sm.async_delete_secret(
    secret_name="unused-key",
    recovery_window_in_days=30  # Can be recovered within 30 days
)

print(f"Deletion date: {response['DeletionDate']}")

Azure Key Vault

Setup

from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient
import litellm
import os

# Initialize Azure Key Vault client
kv_uri = os.environ["AZURE_KEY_VAULT_URI"]
credential = DefaultAzureCredential()

litellm.secret_manager_client = SecretClient(
    vault_url=kv_uri,
    credential=credential
)

litellm._key_management_system = "azure_key_vault"

Use with LiteLLM

# Secrets automatically fetched from Key Vault
response = await litellm.acompletion(
    model="gpt-4",
    api_key="os.environ/openai-api-key",  # Retrieved from Azure Key Vault
    messages=[{"role": "user", "content": "Hello!"}]
)

Google Cloud Secret Manager

Setup

from litellm.secret_managers.google_secret_manager import GoogleSecretManager
import litellm

# Initialize
litellm.secret_manager_client = GoogleSecretManager(
    project_id="your-gcp-project"
)

litellm._key_management_system = "google_secret_manager"

Operations

sm = GoogleSecretManager(project_id="your-project")

# Read secret (latest version)
api_key = sm.get_secret_from_google_secret_manager(
    secret_name="openai-api-key"
)

# Read specific version
api_key = sm.get_secret_from_google_secret_manager(
    secret_name="openai-api-key",
    version_id="3"
)

HashiCorp Vault

Setup

from litellm.secret_managers.hashicorp_secret_manager import HashiCorpVault
import litellm
import os

# Initialize
vault = HashiCorpVault(
    vault_url=os.environ["VAULT_ADDR"],
    vault_token=os.environ["VAULT_TOKEN"],
    vault_namespace="litellm",  # Optional
    mount_point="secret"  # Default KV mount point
)

litellm.secret_manager_client = vault
litellm._key_management_system = "hashicorp_vault"

Operations

# Async read
api_key = await vault.async_read_secret(
    secret_name="openai/api-key"  # Path in Vault
)

# Sync read
api_key = vault.sync_read_secret(
    secret_name="openai/api-key"
)

CyberArk

Setup

from litellm.secret_managers.cyberark_secret_manager import CyberArkSecretManager
import litellm

# Initialize
cyberark = CyberArkSecretManager(
    cyberark_url="https://cyberark.company.com",
    cyberark_app_id="litellm-app",
    cyberark_safe="litellm-safe",
    cyberark_cert_path="/path/to/client-cert.pem"
)

litellm.secret_manager_client = cyberark
litellm._key_management_system = "cyberark"

Operations

# Read secret
api_key = await cyberark.async_read_secret(
    secret_name="openai-api-key",
    optional_params={
        "folder": "Root",
        "object": "OpenAI-Key"
    }
)

Custom Secret Manager

Implement Base Class

from litellm.secret_managers.base_secret_manager import BaseSecretManager
from typing import Optional, Dict, Any, Union
import httpx

class MySecretManager(BaseSecretManager):
    def __init__(self, api_url: str, api_token: str):
        super().__init__()
        self.api_url = api_url
        self.api_token = api_token
    
    async def async_read_secret(
        self,
        secret_name: str,
        optional_params: Optional[dict] = None,
        timeout: Optional[Union[float, httpx.Timeout]] = None,
    ) -> Optional[str]:
        """
        Fetch secret from your backend.
        
        Returns:
            Secret value as string, or None if not found
        """
        async with httpx.AsyncClient() as client:
            response = await client.get(
                f"{self.api_url}/secrets/{secret_name}",
                headers={"Authorization": f"Bearer {self.api_token}"},
                timeout=timeout or 30.0
            )
            
            if response.status_code == 404:
                return None
            
            response.raise_for_status()
            data = response.json()
            return data["value"]
    
    def sync_read_secret(
        self,
        secret_name: str,
        optional_params: Optional[dict] = None,
        timeout: Optional[Union[float, httpx.Timeout]] = None,
    ) -> Optional[str]:
        """
        Synchronous version of async_read_secret.
        """
        response = httpx.get(
            f"{self.api_url}/secrets/{secret_name}",
            headers={"Authorization": f"Bearer {self.api_token}"},
            timeout=timeout or 30.0
        )
        
        if response.status_code == 404:
            return None
        
        response.raise_for_status()
        data = response.json()
        return data["value"]
    
    async def async_write_secret(
        self,
        secret_name: str,
        secret_value: str,
        description: Optional[str] = None,
        optional_params: Optional[dict] = None,
        timeout: Optional[Union[float, httpx.Timeout]] = None,
        tags: Optional[Union[dict, list]] = None
    ) -> Dict[str, Any]:
        """
        Write a secret to your backend.
        """
        async with httpx.AsyncClient() as client:
            response = await client.post(
                f"{self.api_url}/secrets",
                json={
                    "name": secret_name,
                    "value": secret_value,
                    "description": description,
                    "tags": tags
                },
                headers={"Authorization": f"Bearer {self.api_token}"},
                timeout=timeout or 30.0
            )
            
            response.raise_for_status()
            return response.json()
    
    async def async_delete_secret(
        self,
        secret_name: str,
        recovery_window_in_days: Optional[int] = 7,
        optional_params: Optional[dict] = None,
        timeout: Optional[Union[float, httpx.Timeout]] = None,
    ) -> dict:
        """
        Delete a secret from your backend.
        """
        async with httpx.AsyncClient() as client:
            response = await client.delete(
                f"{self.api_url}/secrets/{secret_name}",
                headers={"Authorization": f"Bearer {self.api_token}"},
                params={"recovery_days": recovery_window_in_days},
                timeout=timeout or 30.0
            )
            
            response.raise_for_status()
            return response.json()

# Register with LiteLLM
import litellm

litellm.secret_manager_client = MySecretManager(
    api_url="https://secrets.company.com",
    api_token="your-token"
)

litellm._key_management_system = "custom"

Use Custom Secret Manager

# Secrets fetched from your backend
response = await litellm.acompletion(
    model="gpt-4",
    api_key="os.environ/openai-key",
    messages=[{"role": "user", "content": "Hello!"}]
)

Using with LiteLLM Proxy

Configuration

AWS Secrets Manager

config.yaml

general_settings:
  use_aws_secret_manager: true
  key_management_system: "aws_secret_manager"

key_management_settings:
  aws_region_name: us-east-1
  aws_role_name: arn:aws:iam::123456789:role/LiteLLMRole

model_list:
  - model_name: gpt-4
    litellm_params:
      model: gpt-4
      api_key: os.environ/OPENAI_API_KEY  # From AWS Secrets Manager

Azure Key Vault

config.yaml

environment_variables:
  AZURE_KEY_VAULT_URI: https://your-vault.vault.azure.net/

general_settings:
  key_management_system: azure_key_vault

model_list:
  - model_name: gpt-4
    litellm_params:
      model: gpt-4
      api_key: os.environ/openai-api-key

Google Secret Manager

config.yaml

environment_variables:
  GOOGLE_APPLICATION_CREDENTIALS: /path/to/credentials.json

general_settings:
  key_management_system: google_secret_manager
  google_kms_project_id: your-gcp-project

model_list:
  - model_name: gpt-4
    litellm_params:
      model: gpt-4
      api_key: os.environ/openai-api-key

HashiCorp Vault

config.yaml

environment_variables:
  VAULT_ADDR: https://vault.company.com
  VAULT_TOKEN: your-vault-token

general_settings:
  key_management_system: hashicorp_vault

model_list:
  - model_name: gpt-4
    litellm_params:
      model: gpt-4
      api_key: os.environ/openai/api-key

Best Practices

Use IAM Roles Instead of Keys

For AWS, prefer IAM roles over access keys:

# Good: IAM role-based authentication
sm = AWSSecretsManagerV2(
    aws_region_name="us-east-1",
    aws_role_name="arn:aws:iam::123456789:role/LiteLLMRole"
)

# Bad: Hardcoded credentials
sm = AWSSecretsManagerV2(
    aws_region_name="us-east-1",
    aws_access_key_id="AKIAIOSFODNN7EXAMPLE",
    aws_secret_access_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
)

Implement Secret Rotation

Regularly rotate secrets:

import asyncio
from datetime import datetime, timedelta

async def rotate_if_needed(sm, secret_name, max_age_days=90):
    # Check secret age
    metadata = await sm.get_secret_metadata(secret_name)
    created = metadata["CreatedDate"]
    
    if datetime.now() - created > timedelta(days=max_age_days):
        # Generate new key (implement your logic)
        new_value = generate_new_api_key()
        
        # Rotate
        await sm.async_rotate_secret(
            current_secret_name=secret_name,
            new_secret_name=f"{secret_name}-v2",
            new_secret_value=new_value
        )
        
        print(f"Rotated {secret_name}")

Cache Secrets Locally

Reduce API calls with caching:

from functools import lru_cache
from datetime import datetime, timedelta

class CachedSecretManager:
    def __init__(self, sm):
        self.sm = sm
        self.cache = {}
        self.cache_ttl = timedelta(minutes=5)
    
    async def get_secret(self, secret_name):
        # Check cache
        if secret_name in self.cache:
            value, timestamp = self.cache[secret_name]
            if datetime.now() - timestamp < self.cache_ttl:
                return value
        
        # Fetch from secret manager
        value = await self.sm.async_read_secret(secret_name)
        
        # Update cache
        self.cache[secret_name] = (value, datetime.now())
        
        return value

Handle Secret Not Found

Gracefully handle missing secrets:

async def get_secret_with_fallback(sm, secret_name, fallback=None):
    try:
        value = await sm.async_read_secret(secret_name)
        
        if value is None:
            if fallback:
                return fallback
            raise ValueError(f"Secret not found: {secret_name}")
        
        return value
    except Exception as e:
        print(f"Error fetching secret: {e}")
        if fallback:
            return fallback
        raise

Audit Secret Access

Log all secret access for security auditing:

import logging

logger = logging.getLogger("secret_access")

class AuditedSecretManager:
    def __init__(self, sm):
        self.sm = sm
    
    async def async_read_secret(self, secret_name, **kwargs):
        logger.info(f"Accessing secret: {secret_name}")
        
        try:
            value = await self.sm.async_read_secret(secret_name, **kwargs)
            logger.info(f"Successfully retrieved: {secret_name}")
            return value
        except Exception as e:
            logger.error(f"Failed to retrieve {secret_name}: {e}")
            raise

Reference

Source Code

Base secret manager: litellm/secret_managers/base_secret_manager.py:9
AWS Secrets Manager: litellm/secret_managers/aws_secret_manager_v2.py:36
Secret manager handler: litellm/secret_managers/secret_manager_handler.py:24

Environment Variables

# AWS
AWS_REGION_NAME=us-east-1
AWS_ACCESS_KEY_ID=your-key
AWS_SECRET_ACCESS_KEY=your-secret

# Azure
AZURE_KEY_VAULT_URI=https://vault.azure.net/
AZURE_TENANT_ID=your-tenant-id
AZURE_CLIENT_ID=your-client-id
AZURE_CLIENT_SECRET=your-secret

# Google Cloud
GOOGLE_APPLICATION_CREDENTIALS=/path/to/creds.json

# HashiCorp Vault
VAULT_ADDR=https://vault.example.com
VAULT_TOKEN=your-token
VAULT_NAMESPACE=litellm

Get Started

Python SDK

AI Gateway (Proxy)

Core Features

Advanced

Secret Management

Overview

Supported Secret Managers

Quick Start

AWS Secrets Manager

Setup

Read Secrets

Write Secrets

Rotate Secrets

Delete Secrets

Azure Key Vault

Setup

Use with LiteLLM

Google Cloud Secret Manager

Setup

Operations

HashiCorp Vault

Setup

Operations

CyberArk

Setup

Operations

Custom Secret Manager

Implement Base Class

Use Custom Secret Manager

Using with LiteLLM Proxy

Configuration

Best Practices

Reference

Source Code

Environment Variables

Build docs developers (and LLMs) love

Get Started

Python SDK

AI Gateway (Proxy)

Core Features

Advanced

Documentation Index

​Overview

​Supported Secret Managers

​Quick Start

​AWS Secrets Manager

​Setup

​Read Secrets

​Write Secrets

​Rotate Secrets

​Delete Secrets

​Azure Key Vault

​Setup

​Use with LiteLLM

​Google Cloud Secret Manager

​Setup

​Operations

​HashiCorp Vault

​Setup

​Operations

​CyberArk

​Setup

​Operations

​Custom Secret Manager

​Implement Base Class

​Use Custom Secret Manager

​Using with LiteLLM Proxy

​Configuration

​Best Practices

​Reference

​Source Code

​Environment Variables

​Related Documentation

Build docs developers (and LLMs) love

Overview

Supported Secret Managers

Quick Start

AWS Secrets Manager

Setup

Read Secrets

Write Secrets

Rotate Secrets

Delete Secrets

Azure Key Vault

Setup

Use with LiteLLM

Google Cloud Secret Manager

Setup

Operations

HashiCorp Vault

Setup

Operations

CyberArk

Setup

Operations

Custom Secret Manager

Implement Base Class

Use Custom Secret Manager

Using with LiteLLM Proxy

Configuration

Best Practices

Reference

Source Code

Environment Variables

Related Documentation