Troubleshooting

​

Installation issues

​

CUDA version compatibility

1. Make sure to install a compatible version of flash-attn manually
2. For CUDA 11.8, flash-attn==2.8.2 has been confirmed to work
3. For RTX 5090, use CUDA 12.8 with flash-attn==2.8.0.post2 and pytorch-cu128

# Example for CUDA 11.8
uv pip install flash-attn==2.8.2

​

UV version too old

# Upgrade uv
curl -LsSf https://astral.sh/uv/install.sh | sh

​

Submodules not initialized

git submodule update --init --recursive

​

Docker issues

​

GPU not detected in container

1. Verify NVIDIA Container Toolkit is installed:
   nvidia-container-toolkit --version
   
2. Restart Docker daemon:
   sudo systemctl restart docker
   
3. Test GPU access:
   docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi
   

​

Permission errors with Docker

sudo usermod -aG docker $USER
# Log out and log back in for changes to take effect

​

Build failures

1. Check disk space:
   df -h
   
2. Clean Docker cache:
   docker system prune -a
   
3. Rebuild without cache:
   sudo bash build.sh --no-cache
   

​

Policy and inference issues

​

RuntimeError: Cannot connect to policy server

1. Verify the server is running:
   # Check if port is listening
   netstat -an | grep 5555
   
2. Check host and port match between server and client:
   # Server
   --host 0.0.0.0 --port 5555
   
   # Client
   policy = PolicyClient(host="localhost", port=5555)
   
3. Try using IP address instead of hostname:
   policy = PolicyClient(host="127.0.0.1", port=5555)
   

​

Observation format mismatch

# Enable validation during development
policy = Gr00tPolicy(
    model_path="/path/to/checkpoint",
    embodiment_tag=EmbodimentTag.NEW_EMBODIMENT,
    strict=True  # This will validate inputs/outputs
)

# Print expected formats
modality_configs = policy.get_modality_config()

# Check video requirements
video_keys = modality_configs["video"].modality_keys
video_horizon = len(modality_configs["video"].delta_indices)
print(f"Expected cameras: {video_keys}")
print(f"Video frames needed: {video_horizon}")

# Check state requirements
state_keys = modality_configs["state"].modality_keys
state_horizon = len(modality_configs["state"].delta_indices)
print(f"Expected states: {state_keys}, horizon: {state_horizon}")

​

Data type errors

• Videos must be np.uint8 arrays with RGB pixel values in range [0, 255]
• States must be np.float32 arrays
• Language instructions are lists of lists of strings

import numpy as np

# Correct video format
video = np.random.randint(0, 256, size=(1, T, H, W, 3), dtype=np.uint8)

# Correct state format
state = np.random.randn(1, T, D).astype(np.float32)

# Correct language format
language = [["pick up the cube"]]

​

TensorRT errors

1. Verify CUDA is available:
   import torch
   print(torch.cuda.is_available())
   
2. Check TensorRT installation:
   python -c "import tensorrt; print(tensorrt.__version__)"
   
3. Rebuild TensorRT engine if it was built on a different system:
   uv run python scripts/deployment/build_tensorrt_engine.py \
     --onnx-path /path/to/model.onnx \
     --output-path /path/to/engine.trt
   

​

Training issues

​

Out of memory (OOM) errors

1. Reduce batch size:
   --global-batch-size 16  # instead of 32
   
2. Reduce dataloader workers:
   --dataloader-num-workers 2  # instead of 4
   
3. Optimize dataloading parameters:
   --num-shards-per-epoch 50 \
   --shard-size 256 \
   --dataloader-num-workers 2
   
4. Use gradient accumulation (if supported):
   --gradient-accumulation-steps 2
   

​

Training variance

• Non-deterministic operations in image augmentations
• Stochastic components in the training pipeline
• Hardware differences

• Run multiple training runs and select the best checkpoint
• Use validation metrics to track performance
• Keep this inherent variance in mind when comparing to reported benchmarks

​

Slow dataloading

1. Increase number of workers:
   --dataloader-num-workers 4  # or higher
   
2. Increase shard size for better IID sampling:
   --num-shards-per-epoch 100 \
   --shard-size 512
   
3. Reduce episode sampling rate for more IID sampling:
   --episode-sampling-rate 0.05
   

​

Simulation evaluation issues

​

EGL/GLX errors

# Check if EGL/GLX libraries exist
ldconfig -p | grep -i egl
ldconfig -p | grep -i gl

# Install if missing (Ubuntu/Debian)
sudo apt-get update
sudo apt-get install -y libegl1-mesa libegl1-mesa-dev libgl1-mesa-glx

​

Environment prefix not found

ENV_PREFIX_TO_EMBODIMENT_TAG = {
    ...
    "my_new_benchmark": EmbodimentTag.MY_ROBOT,
}

​

Low success rate with ReplayPolicy

• Environment reset state not matching the dataset
• Observation preprocessing differences
• Action space mismatches
• Execution horizon mismatch

# Server
--execution-horizon 8

# Client
--n_action_steps 8

​

Data preparation issues

​

Video decoding errors

1. Verify video file integrity:
   ffprobe -v error /path/to/video.mp4
   
2. Re-encode video with compatible settings:
   ffmpeg -i input.mp4 -c:v libx264 -pix_fmt yuv420p output.mp4
   
3. Check video backend:
   # Try different backends
   from gr00t.utils.video_utils import read_frames
   frames = read_frames("/path/to/video.mp4", backend="pyav")  # or "decord", "torchcodec"
   

​

Dataset conversion issues

1. Verify source dataset structure:
   ls -R /path/to/source/dataset
   
2. Check conversion script output for errors:
   python scripts/lerobot_conversion/convert_v3_to_v2.py \
     --input-dir /path/to/v3 \
     --output-dir /path/to/v2 \
     --verbose
   
3. Validate converted dataset:
   from lerobot.common.datasets.lerobot_dataset import LeRobotDataset
   dataset = LeRobotDataset("/path/to/converted/dataset")
   print(f"Number of episodes: {dataset.num_episodes}")
   print(f"Total frames: {len(dataset)}")
   

​

Getting help

1. Check existing issues: Search GitHub issues to see if your problem has been reported
2. Verify environment: Run the verification script:
   python scripts/eval/check_sim_eval_ready.py
   
3. Enable debug logging: Add verbose flags to scripts:
   --verbose  # or set logging level to DEBUG
   
4. Report a bug: If you’ve found a new issue, open a GitHub issue with:
   • Clear description of the problem
   • Steps to reproduce
   • Error messages and stack traces
   • Environment information (CUDA version, GPU model, OS, etc.)