Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/modal-labs/modal-client/llms.txt

Use this file to discover all available pages before exploring further.

GPU configuration for Modal functions and classes. You can pass a wide range of str values for the gpu parameter of @app.function and @app.cls.

​
String shortcodes

The recommended way to specify GPUs is using string shortcodes:
  • gpu="H100" - Attach 1 H100 GPU to each container
  • gpu="L40S" - Attach 1 L40S GPU to each container
  • gpu="A100" - Attach 1 A100-40GB GPU to each container
  • gpu="A100-80GB" - Attach 1 A100-80GB GPU to each container
  • gpu="T4:4" - Attach 4 T4 GPUs to each container
  • gpu="A100-80GB:4" - Attach 4 A100-80GB GPUs to each container
Example: 
@app.function(gpu="A100-80GB:4")
def my_gpu_function():
    # This will have 4 A100-80GB GPUs
    pass

​
Available GPU types

​
T4

NVIDIA T4 Tensor Core GPU. A low-cost data center GPU based on the Turing architecture, providing 16GB of GPU memory. 
gpu="T4"      # Single T4
gpu="T4:4"    # 4x T4

​
L4

NVIDIA L4 Tensor Core GPU. A mid-tier data center GPU based on the Ada Lovelace architecture, providing 24GB of GPU memory. Includes RTX (ray tracing) support. 
gpu="L4"      # Single L4
gpu="L4:2"    # 2x L4

​
A10G

NVIDIA A10G Tensor Core GPU. A mid-tier data center GPU based on the Ampere architecture, providing 24 GB of memory. A10G GPUs deliver up to 3.3x better ML training performance compared to T4 GPUs. 
gpu="A10G"    # Single A10G
gpu="A10G:2"  # 2x A10G

​
A100

NVIDIA A100 Tensor Core GPU. The flagship data center GPU of the Ampere architecture. Available in 40GB and 80GB GPU memory configurations. 
gpu="A100"         # Single A100-40GB
gpu="A100-40GB"    # Single A100-40GB (explicit)
gpu="A100-80GB"    # Single A100-80GB
gpu="A100:8"       # 8x A100-40GB
gpu="A100-80GB:4"  # 4x A100-80GB

​
H100

NVIDIA H100 Tensor Core GPU. The flagship data center GPU of the Hopper architecture. Enhanced support for FP8 precision and a Transformer Engine that provides up to 4X faster training over the prior generation for GPT-3 (175B) models. 
gpu="H100"    # Single H100
gpu="H100:8"  # 8x H100

​
L40S

NVIDIA L40S GPU. The L40S is a data center GPU for the Ada Lovelace architecture. It has 48 GB of on-chip GDDR6 RAM and enhanced support for FP8 precision. 
gpu="L40S"    # Single L40S
gpu="L40S:2"  # 2x L40S

​
Any

Selects any one of the GPU classes available within Modal, according to availability. 
gpu="any"     # Any available GPU
gpu="any:2"   # 2x any available GPU

​
Type hints

For type checking purposes, the GPU parameter type is: 
GPU_T = Union[None, str, _GPUConfig]
Where _GPUConfig is a deprecated class-based configuration (see deprecation note below).

​
Deprecated class-based configuration

The class-based GPU configuration is deprecated and will be removed in a future version. Use string shortcodes instead.
Older code may use class-based configuration: 
# Deprecated - use gpu="H100" instead
gpu=modal.gpu.H100()

# Deprecated - use gpu="T4:4" instead
gpu=modal.gpu.T4(count=4)

# Deprecated - use gpu="A100-80GB" instead
gpu=modal.gpu.A100(size="80GB")

Build docs developers (and LLMs) love

Get started for freeTalk to us