Documentation Index
Fetch the complete documentation index at: https://mintlify.com/pytorch/rl/llms.txt
Use this file to discover all available pages before exploring further.
TorchRL is distributed as a PyPI package and as optional CUDA-specific wheels for users who need hardware-accelerated prioritized replay buffers. This page covers every supported install path — stable release, nightly builds, source checkouts, and CUDA wheels — along with the optional environment library dependencies and a quick smoke test to confirm everything is working.
Requirements
| Dependency | Minimum version | Notes |
|---|
| Python | 3.10+ | Earlier versions are not supported |
| PyTorch | 2.1+ | Install from pytorch.org before installing TorchRL |
| TensorDict | 0.13.x | Installed automatically as a dependency |
TorchRL releases are synchronized with PyTorch releases. Always install PyTorch first from the PyTorch installation selector to ensure you get the build that matches your hardware (CPU, CUDA 11.x, CUDA 12.x, etc.), then install TorchRL on top. Install TorchRL
Stable Release
Nightly
From Source
CUDA Wheel
The standard PyPI wheel is the right default for most users. It includes CPU-based prioritized replay buffers and all core TorchRL features.Verify the install:import torchrl
print(torchrl.__version__) # 0.13.0
Nightly builds track the main branch and are published under the torchrl-nightly package name. Use these when working against a nightly PyTorch build or when you need features that have not yet landed in a stable release.pip install --pre tensordict-nightly torchrl-nightly
Nightly builds may contain breaking changes and incomplete features. Prefer stable releases for production workloads.
For local development or to contribute to TorchRL, clone both the tensordict and rl repositories. Use --no-deps with uv to prevent dependency resolution from downgrading a pre-selected PyTorch build (for example, a nightly CUDA wheel you installed manually).git clone https://github.com/pytorch/tensordict
git clone https://github.com/pytorch/rl
uv pip install --no-deps -e tensordict
uv pip install --no-deps -e rl
Use --no-deps when you have already pinned a specific PyTorch version. Without it, pip or uv may re-resolve the dependency graph and replace your PyTorch build with a different one.
Starting with TorchRL 0.13, Linux CUDA wheels are published to the PyTorch wheel index. These wheels include CUDA-accelerated prioritized replay-buffer kernels (CudaSumSegmentTree) that the standard PyPI wheel does not.Replace cu128 with the CUDA version that matches your installed PyTorch runtime:pip install "torchrl==0.13.0+cu128" --extra-index-url https://download.pytorch.org/whl/cu128
The CUDA wheel is optional. If you do not use CUDA-based prioritized replay buffers, or if your prioritized replay buffers run on CPU, the standard pip install torchrl wheel is sufficient. CUDA wheels are currently only published for Linux.
import torch
print(torch.version.cuda) # e.g. "12.8"
| CUDA version | Suffix |
|---|
| 11.8 | cu118 |
| 12.1 | cu121 |
| 12.4 | cu124 |
| 12.8 | cu128 |
Optional Environment Dependencies
TorchRL’s core components have no dependencies beyond PyTorch and TensorDict. Environment library wrappers are guarded by optional extras — install only what you need.
# Hydra configuration, logging utilities, and development helpers
pip install "torchrl[utils]"
# Gymnasium continuous-control environments (MuJoCo, robotics tasks, etc.)
pip install "torchrl[gym_continuous]"
# Atari environments via Gymnasium
pip install "torchrl[atari]"
# Offline RL datasets and data loading helpers
pip install "torchrl[offline-data]"
# Multi-agent environment libraries (VMAS, PettingZoo, etc.)
pip install "torchrl[marl]"
# LLM post-training with vLLM backend (Linux only)
pip install "torchrl[llm-vllm]"
# LLM post-training with SGLang backend (Linux only)
pip install "torchrl[llm-sglang]"
| Library | Install command |
|---|
| Gymnasium (basic) | pip install gymnasium |
| MuJoCo environments | pip install "gymnasium[mujoco]" |
| DM Control | pip install dm_control |
| Brax | pip install brax |
| PettingZoo | pip install pettingzoo |
| VMAS | pip install vmas |
| Jumanji | pip install jumanji |
| OpenSpiel | pip install open_spiel |
| Safety Gymnasium | pip install safety_gymnasium |
| Isaac Lab | See the Isaac Lab documentation |
Some optional libraries (Isaac Lab, robohive, vLLM, SGLang) have platform-specific requirements or need additional system packages. Consult the respective library’s documentation for complete setup instructions. vLLM and SGLang integrations are only supported on Linux.
Verifying the Installation
Run this smoke test to confirm TorchRL is installed correctly and the core components are functional:
import torch
from tensordict.nn import TensorDictModule
from torch import nn
from torchrl.envs import PendulumEnv, StepCounter, TransformedEnv
from torchrl.data import PrioritizedReplayBuffer
# 1. Check that the C++ extension loaded correctly.
rb = PrioritizedReplayBuffer(alpha=0.7, beta=0.5)
print("PrioritizedReplayBuffer OK")
# 2. Check a native environment + transform stack.
env = TransformedEnv(PendulumEnv(), StepCounter(max_steps=10))
rollout = env.rollout(max_steps=5)
assert rollout.batch_size == torch.Size([5])
print("Environment rollout OK — shape:", rollout.batch_size)
# 3. Check a policy module.
policy = TensorDictModule(
nn.Sequential(nn.LazyLinear(32), nn.Tanh(), nn.Linear(32, 1), nn.Tanh()),
in_keys=["observation"],
out_keys=["action"],
)
out = policy(env.reset())
print("Policy forward pass OK — action shape:", out["action"].shape)
print("\nTorchRL installation verified successfully.")
PrioritizedReplayBuffer OK
Environment rollout OK — shape: torch.Size([5])
Policy forward pass OK — action shape: torch.Size([1])
TorchRL installation verified successfully.
undefined symbol errors when importing torchrl._torchrl, the C++ extension was compiled against a different PyTorch version than the one currently installed. Reinstall TorchRL after ensuring your PyTorch version matches, or consult the versioning issues guide.