Skip to main content
ComfyUI supports advanced audio generation models for creating music, sound effects, and voice synthesis.

Stable Audio

Stable Audio generates high-quality music and sound effects from text descriptions.

Architecture

Model Configuration:
  • Audio model: dit1.0
  • Latent format: StableAudio1 (64 channels)
  • Sigma range: 0.03 to 500.0
  • Sample rate: 44.1kHz
Components:
  • DiffusionModel: DiT-based audio generator
  • VAE: Audio compression/decompression
  • Text Encoder: T5-based (SAT5 variant)
  • Seconds Embedders: Start time and duration conditioning

Text Encoder

Model: SAT5 (Stable Audio T5)
  • Tokenizer: SAT5Tokenizer
  • Purpose: Encodes text prompts for audio generation

Usage

Creating Empty Latent:
Conditioning:
VAE Encoding:
VAE Decoding:
Latent Dimensions:
  • Channels: 64
  • Length: Based on duration (44.1kHz / 2048 compression)
  • Device: Intermediate device (for memory efficiency)

Audio Workflow Nodes

Loading Audio:
Recording Audio:
Saving Audio: FLAC (Lossless):
MP3 (Compressed):
Opus (Web):
Previewing Audio:

Audio Processing

Trimming:
Channel Operations: Splitting Stereo:
Joining to Stereo:
Concatenation:
Merging/Mixing:
Volume Adjustment:
Equalization (Experimental):
Creating Empty Audio:

Model Files Location

Examples


ACE Step

ACE Step specializes in music generation with lyrics, tags, and musical structure control.

Architecture

Model Configuration:
  • Audio model: ace
  • Latent format: ACEAudio
  • Shift: 3.0
  • Memory usage factor: 0.5
  • Supported dtypes: BF16, FP32
  • Sample rate: 44.1kHz (ACE Step 1.0) or 48kHz (ACE Step 1.5)

ACE Step 1.0

Text Encoder:
  • Custom ACE T5 model
  • Tokenizer: AceT5Tokenizer
Latent Format:
  • Shape: [batch, 8, 16, length]
  • Length: int(seconds * 44100 / 512 / 8)
Text Encoding:
Empty Latent:

ACE Step 1.5

Enhanced Features:
  • Higher sample rate (48kHz)
  • More musical controls
  • Audio reference support
  • LLM-based audio code generation
Latent Format:
  • Shape: [batch, 64, length]
  • Length: round(seconds * 48000 / 1920)
Text Encoding:
Available Languages:
  • en (English), ja (Japanese), zh (Chinese), es (Spanish)
  • de (German), fr (French), pt (Portuguese), ru (Russian)
  • it (Italian), nl (Dutch), pl (Polish), tr (Turkish)
  • vi (Vietnamese), cs (Czech), fa (Persian), id (Indonesian)
  • ko (Korean), uk (Ukrainian), hu (Hungarian), ar (Arabic)
  • sv (Swedish), ro (Romanian), el (Greek)
Key Scales:
  • All major and minor keys
  • Format: “[C/C#/D/Eb/E/F/F#/G/Ab/A/Bb/B] [major/minor]”
Empty Latent:
Reference Audio (Experimental):

Usage Tips

ACE Step 1.0:
  1. Use descriptive tags (genre, mood, instruments)
  2. Lyrics are optional but improve structure
  3. Adjust lyrics_strength to balance music vs. vocals
ACE Step 1.5:
  1. Set BPM and time signature for rhythmic coherence
  2. Choose appropriate key scale for mood
  3. Enable generate_audio_codes for best quality
  4. Disable audio codes when using reference audio
  5. Adjust temperature for more/less variation
  6. Use CFG scale to control prompt adherence

Model Files Location

Examples


Audio Format Details

Waveform Tensor Format

Structure:
Channel Layouts:
  • Mono: channels = 1
  • Stereo: channels = 2
Sample Rates:
  • 44.1kHz: Standard audio CD quality (Stable Audio, ACE 1.0)
  • 48kHz: Professional audio standard (ACE 1.5)

Latent Compression

Stable Audio:
  • Temporal compression: ~2048x (44.1kHz → ~21.5Hz latent rate)
  • Channel expansion: 1-2 channels → 64 latent channels
ACE Step 1.0:
  • Temporal compression: 512 * 8 = 4096x
  • Spatial/channel layout: [8, 16, length]
ACE Step 1.5:
  • Temporal compression: 1920x (48kHz → 25Hz latent rate)
  • Channel expansion: 1-2 channels → 64 latent channels

Performance Optimization

Memory Management

For Long Audio:
  • Use VAEDecodeAudioTiled instead of VAEDecodeAudio
  • Adjust tile_size and overlap based on VRAM
  • Typical settings: tile_size=512, overlap=64
VRAM Requirements:
  • Stable Audio (47 seconds): ~4-6GB
  • ACE Step 1.0 (120 seconds): ~6-8GB
  • ACE Step 1.5 (120 seconds): ~8-10GB

Speed Optimization

Preprocessing:
  • Use FP16/BF16 for VAE when possible
  • Enable --fp16-vae flag
Generation:
  • Start with shorter durations for testing
  • Use fewer sampling steps initially
  • ACE 1.5: Disable generate_audio_codes for faster preview

Quality Settings

Export Formats: Highest Quality:
  • FLAC lossless
  • Full sample rate (44.1/48kHz)
Balanced:
  • MP3 V0 (variable bitrate, ~245kbps)
  • Opus 192k
Streaming:
  • MP3 128k
  • Opus 96k or 128k
Normalization:
  • ComfyUI automatically normalizes audio to prevent clipping
  • Output is divided by std * 5.0 (capped at 1.0)

Advanced Workflows

Music Production Pipeline

  1. Generate base track (ACE Step with tags and BPM)
  2. Trim to exact length (TrimAudioDuration)
  3. Adjust levels (AudioAdjustVolume)
  4. Apply EQ (AudioEqualizer3Band)
  5. Export (SaveAudio with FLAC or high-quality MP3)

Sound Design

  1. Generate variations (batch_size > 1)
  2. Mix multiple outputs (AudioMerge with “add” or “mean”)
  3. Layer effects (AudioConcat)
  4. Process channels separately (SplitAudioChannels → process → JoinAudioChannels)

Podcast/Voice Workflow

  1. Record or load voice (RecordAudio or LoadAudio)
  2. Trim silence (TrimAudioDuration)
  3. Normalize volume (AudioAdjustVolume)
  4. Add music bed (ACE Step + AudioMerge)
  5. Export for web (SaveAudioOpus)

Resources