Dynamic Configuration

Dynamic Configuration enables runtime tuning of Cadence server behavior without requiring restarts. This allows operational flexibility, gradual rollouts, and per-domain customization.

Overview

Dynamic config provides:

Runtime Updates: Change settings without restarting services
Scoped Configuration: Apply settings globally, per-domain, or per-task-list
Gradual Rollout: Test changes on specific domains before global deployment
Operational Safety: Tune throttling, timeouts, and limits during incidents
Multi-Tenancy: Customize behavior per tenant (domain)

Configuration Sources

Cadence supports multiple dynamic config backends:

File-Based Configuration

Default backend: YAML file loaded at startup and refreshed periodically. Configuration (config.yaml):

dynamicConfigClient:
  filepath: "config/dynamicconfig/development.yaml"
  pollInterval: "60s"

Dynamic Config File (development.yaml):

# Global setting
system.enableAuthorization:
  - value: false

# Domain-specific setting
history.defaultWorkflowTaskStartToCloseTimeout:
  - value: "10s"
  - value: "30s"
    constraints:
      domainName: "critical-domain"

# Domain and task list specific
matching.maxTaskListSize:
  - value: 1000
  - value: 5000
    constraints:
      domainName: "high-volume-domain"
      taskListName: "important-tasks"

Database-Based Configuration

Enterprise feature: Store config in Cassandra/SQL for centralized management. Configuration:

dynamicConfigClient:
  configstore:
    pollInterval: "30s"

Requires separate config service deployment.

Environment-Based Overrides

Quick overrides: Use environment variables for specific settings.

DYNAMIC_CONFIG_OVERRIDE_frontend.rps=2000 ./cadence-server start

Configuration Format

Value Structure

Each setting is a list of values with optional constraints:

key:
  - value: <default-value>
  - value: <override-value>
    constraints:
      domainName: "<domain>"
      taskListName: "<task-list>"
      taskType: <0=Decision, 1=Activity>

Constraint Types

domainName: String - Apply to specific domain
taskListName: String - Apply to specific task list
taskType: Integer - Apply to task type (0=Decision, 1=Activity)

Constraint Matching

Value is selected if all constraints match exactly:

# Match: domain=foo, taskList=bar
- value: 100
  constraints:
    domainName: "foo"
    taskListName: "bar"

# Match: domain=foo only (no task list specified)
- value: 50
  constraints:
    domainName: "foo"

# Match: any domain/task list (no constraints)
- value: 10

Query with domain=foo, taskList=bar returns 100 (most specific match).

Common Configuration Keys

Throttling and Rate Limits

Frontend Rate Limits

# Global RPS limit per frontend instance
frontend.rps:
  - value: 2000

# RPS limit per domain
frontend.domainRPS:
  - value: 500
  - value: 2000
    constraints:
      domainName: "high-priority-domain"

# Max concurrent requests
frontend.maxConcurrentRequests:
  - value: 1000

History Service Limits

# Task processing RPS
history.taskProcessRPS:
  - value: 2000

# Transfer queue max reader count
history.transferTaskMaxReadLevel:
  - value: 500

# History event count limit per decision
history.historyCountLimitWarn:
  - value: 10000
history.historyCountLimitError:
  - value: 50000

Matching Service Limits

# Task list max size
matching.maxTaskListSize:
  - value: 1000
  - value: 10000
    constraints:
      domainName: "large-scale-domain"

# Task dispatch RPS
matching.taskDispatchRPS:
  - value: 5000

Timeouts and Retention

# Default workflow execution timeout
frontend.defaultWorkflowExecutionTimeout:
  - value: "72h"

# Default decision task timeout
history.defaultWorkflowTaskStartToCloseTimeout:
  - value: "10s"
  - value: "30s"
    constraints:
      domainName: "long-decision-domain"

# Domain retention period
frontend.domainRetentionDays:
  - value: 7

Visibility and Search

# Max page size for list operations
frontend.visibilityMaxPageSize:
  - value: 1000

# Enable advanced visibility
system.advancedVisibilityWritingMode:
  - value: "dual"  # off, on, dual

# Valid search attributes
frontend.validSearchAttributes:
  - value:
      CustomerId: "Keyword"
      OrderStatus: "Keyword"
      OrderAmount: "Double"
      Passed: "Bool"

Archival Configuration

# Archival worker concurrency
history.numArchiverConcurrency:
  - value: 50

# Archival processing rate
history.archivalProcessRPS:
  - value: 100

Feature Flags

# Enable authorization
system.enableAuthorization:
  - value: false

# Enable gRPC
frontend.enableGRPCOutbound:
  - value: true

# Enable sticky scheduling
history.enableStickyQuery:
  - value: true

Key Categories

Configuration keys are organized by component:

System-Wide

system.* - Global system settings
limit.* - Global rate limits

Service-Specific

frontend.* - Frontend service settings
history.* - History service settings
matching.* - Matching service settings
worker.* - Worker service settings

Feature-Specific

archival.* - Archival configuration
visibility.* - Visibility configuration
authorization.* - Authorization settings

Best Practices

Configuration Management

Version Control: Track dynamic config files in Git
Testing: Test changes in development before production
Documentation: Document non-standard settings and reasoning
Monitoring: Watch for config-related errors in logs

Performance Tuning

Start Conservative: Begin with default values
Measure Impact: Monitor metrics before and after changes
Gradual Rollout: Test on single domain before global deployment
Domain Isolation: Use domain-specific limits for noisy neighbors

Operational Safety

Emergency Response: Keep emergency config changes documented
Rollback Plan: Test reverting to previous config
Throttling: Prefer throttling over rejection during incidents
Validation: Validate config syntax before deployment

Multi-Tenancy

# Isolate tenants with different limits
frontend.domainRPS:
  - value: 100
  - value: 1000
    constraints:
      domainName: "premium-tenant"
  - value: 10
    constraints:
      domainName: "trial-tenant"

matching.maxTaskListSize:
  - value: 1000
  - value: 10000
    constraints:
      domainName: "premium-tenant"

Configuration Discovery

List All Keys

See available dynamic config keys in source code:

# Find all dynamic config keys
grep -r "GetBoolProperty\|GetIntProperty\|GetDurationProperty" \
  common/dynamicconfig/

Key Naming Convention

Keys follow pattern: <component>.<feature><PropertyType> Examples:

frontend.rps - Frontend RPS limit
history.taskProcessRPS - History task processing rate
matching.maxTaskListSize - Matching task list size

Monitoring Dynamic Config

Log Entries

Config changes and errors are logged:

[INFO] Dynamic config reloaded successfully
[WARN] Failed to fetch key: frontend.rps, using default value
[DEBUG] Dynamic config not set for key: custom.setting, use default value

Metrics

Monitor config-related metrics:

# Config fetch errors
sum(rate(cadence_dynamic_config_errors_total[5m]))

# Config reload success rate
sum(rate(cadence_dynamic_config_reloads_total[5m]))

Validation

Verify config is applied:

# Check logs for config reload
grep "Dynamic config reloaded" /var/log/cadence/frontend.log

# Monitor specific metric affected by config
# e.g., after changing frontend.rps

Examples

Gradual Feature Rollout

# Enable new feature for canary domain
feature.newQueryEngine:
  - value: false              # Default: disabled
  - value: true
    constraints:
      domainName: "canary"    # Enable for canary

# After validation, enable for all
feature.newQueryEngine:
  - value: true               # Enable globally

Emergency Throttling

# Reduce load during incident
frontend.domainRPS:
  - value: 500              # Default
  - value: 100              # Emergency: reduce to 20%
    constraints:
      domainName: "noisy-domain"

history.taskProcessRPS:
  - value: 1000             # Reduce task processing

Custom Timeouts for Workflows

history.defaultWorkflowTaskStartToCloseTimeout:
  - value: "10s"
  - value: "60s"
    constraints:
      domainName: "ml-training"  # ML workflows need longer
  - value: "30s"
    constraints:
      domainName: "data-pipeline"

Task List Prioritization

matching.taskDispatchRPS:
  - value: 1000
  - value: 5000
    constraints:
      domainName: "priority-domain"
      taskListName: "critical-tasks"

matching.maxTaskListSize:
  - value: 1000
  - value: 10000
    constraints:
      domainName: "priority-domain"
      taskListName: "critical-tasks"

Troubleshooting

Config Not Applied

Problem: Changes to dynamic config have no effect Solution:

# Check file path in config
grep dynamicConfigClient config.yaml

# Verify file syntax
yaml-lint config/dynamicconfig/development.yaml

# Check poll interval (may take time to apply)
grep pollInterval config.yaml

# Force reload (if supported)
kill -SIGHUP <cadence-pid>

# Check logs for reload events
grep "Dynamic config" /var/log/cadence/*.log

Constraint Not Matching

Problem: Domain-specific config not applying Solution:

# Ensure exact match of ALL constraints
- value: 100
  constraints:
    domainName: "my-domain"     # Must match exactly
    # If taskListName is in query, it must be in constraints too

Default Values Used

Problem: Seeing “dynamic config not set” warnings Solution:

This is normal for optional configs
Only add config entries for values you want to override
Default values are defined in code

Next Steps

Configure Isolation Groups for zone isolation
Set up Async Workflow Queues for offloading
Monitor with Web UI
Test with Canary and Bench

Development Tools

Advanced

Documentation Index

​Overview

​Configuration Sources

​File-Based Configuration

​Database-Based Configuration

​Environment-Based Overrides

​Configuration Format

​Value Structure

​Constraint Types

​Constraint Matching

​Common Configuration Keys

​Throttling and Rate Limits

​Frontend Rate Limits

​History Service Limits

​Matching Service Limits

​Timeouts and Retention

​Visibility and Search

​Archival Configuration

​Feature Flags

​Key Categories

​System-Wide

​Service-Specific

​Feature-Specific

​Best Practices

​Configuration Management

​Performance Tuning

​Operational Safety

​Multi-Tenancy

​Configuration Discovery

​List All Keys

​Key Naming Convention

​Monitoring Dynamic Config

​Log Entries

​Metrics

​Validation

​Examples

​Gradual Feature Rollout

​Emergency Throttling

​Custom Timeouts for Workflows

​Task List Prioritization

​Troubleshooting

​Config Not Applied

​Constraint Not Matching

​Default Values Used

​Next Steps

Build docs developers (and LLMs) love