Skip to main content
This guide shows how to convert your robot data to work with the GR00T-flavored LeRobot dataset format. While GR00T adds additional structure, the schema maintains full compatibility with the upstream LeRobot v2 format.
The TLDR: Add a meta/modality.json file to your LeRobot v2 dataset and follow the schema below.

​
LeRobot v2 requirements

If you already have a dataset in the LeRobot v2 format, you can skip this section. If you have a dataset in the LeRobot v3.0 format, use the conversion script: 
python scripts/lerobot_conversion/convert_v3_to_v2.py
If you have a dataset in another format, convert it to LeRobot v2 format satisfying the following requirements.

​
Structure requirements

The folder should follow this structure: 
.
├─meta 
│ ├─episodes.jsonl
│ ├─modality.json      # GR00T LeRobot specific
│ ├─info.json
│ └─tasks.jsonl
├─videos
│ └─chunk-000
│   └─observation.images.ego_view
│     └─episode_000001.mp4
│     └─episode_000000.mp4
└─data
  └─chunk-000
    ├─episode_000001.parquet
    └─episode_000000.parquet

​
Video observations

The videos folder contains the MP4 files associated with each episode. Requirements:
  • Must be stored as MP4 files
  • Should be named using the format: observation.images.<video_name>
  • Use episode_00000X.mp4 naming where X indicates the episode number

​
Data files

The data folder contains all of the parquet files associated with each episode. Each parquet file contains:
  • State information: stored as observation.state (1D concatenated array of all state modalities)
  • Action: stored as action (1D concatenated array of all action modalities)
  • Timestamp: stored as timestamp (float point number of the starting time)
  • Annotations: stored as annotation.<annotation_source>.<annotation_type>(.<annotation_name>)

​
Example parquet file

Here is a sample from the cube_to_bowl dataset: 
{
    "observation.state": [-0.01147082911843003, ..., 0],
    "action": [-0.010770668025204974, ..., 0],
    "timestamp": 0.04999995231628418,
    "annotation.human.action.task_description": 0,
    "task_index": 0,
    "annotation.human.validity": 1,
    "episode_index": 0,
    "index": 0,
    "next.reward": 0,
    "next.done": false
}

​
Meta files

​
meta/tasks.jsonl

Contains a list of all the tasks in the entire dataset: 
{"task_index": 0, "task": "pick the squash from the counter and place it in the plate"}
{"task_index": 1, "task": "valid"}
You can refer to the task index in the parquet file to get the task description.

​
meta/episodes.jsonl

Contains a list of all the episodes in the entire dataset: 
{"episode_index": 0, "tasks": [...], "length": 416}
{"episode_index": 1, "tasks": [...], "length": 470}
Each episode contains a list of tasks and the length of the episode.

​
GR00T LeRobot specific requirements

​
The meta/modality.json configuration

GR00T requires an additional metadata file meta/modality.json that is not present in the standard LeRobot format. This file provides detailed metadata about state and action modalities, enabling:
  • Separate data storage and interpretation: State and action are stored as concatenated float32 arrays, with metadata to interpret them as distinct fields
  • Video: Stored as separate files, with the configuration allowing them to be renamed to a standardized format
  • Annotations: Keeps track of all annotation fields
  • Fine-grained splitting: Divides the state and action arrays into more semantically meaningful fields
  • Clear mapping: Explicit mapping of data dimensions
  • Sophisticated data transformations: Supports field-specific normalization and rotation transformations during training

​
Schema

{
    "state": {
        "<state_key>": {
            "start": <int>,         // Starting index in the state array
            "end": <int>            // Ending index in the state array
        }
    },
    "action": {
        "<action_key>": {
            "start": <int>,         // Starting index in the action array
            "end": <int>            // Ending index in the action array
        }
    },
    "video": {
        "<new_key>": {
            "original_key": "<original_video_key>"
        }
    },
    "annotation": {
        "<annotation_key>": {}  // Empty dictionary to maintain consistency
    }
}
All indices are zero-based and follow Python’s array slicing convention ([start:end]).

​
Example modality.json

{
    "state": {
        "single_arm": {"start": 0, "end": 5},
        "gripper": {"start": 5, "end": 6}
    },
    "action": {
        "single_arm": {"start": 0, "end": 5},
        "gripper": {"start": 5, "end": 6}
    },
    "video": {
        "front": {"original_key": "observation.images.front"},
        "wrist": {"original_key": "observation.images.wrist"}
    },
    "annotation": {
        "human.task_description": {
            "original_key": "task_index"
        }
    }
}

​
GR00T LeRobot extensions to standard LeRobot

GR00T LeRobot is a flavor of the standard LeRobot format with more opinionated requirements:
  • Computes meta/stats.json and meta/relative_stats.json for each dataset automatically
  • Proprioceptive states must always be included in the observation.state keys
  • Supports multi-channel annotation formats (e.g., coarse-grained, fine-tuned), allowing users to add as many annotation channels as needed via the annotation.<annotation_source>.<annotation_type> key
  • Requires the additional metadata file meta/modality.json

​
Multiple annotation support

To support multiple annotations within a single parquet file, users may add extra columns to the parquet file. These columns should be treated the same way as the task_index column in the original LeRobot v2 dataset: In LeRobot v2, actual language descriptions are stored in a row of the meta/tasks.jsonl file, while the parquet file stores only the corresponding index in the task_index column. GR00T follows the same convention and stores the corresponding index for each annotation in the annotation.<annotation_source>.<annotation_type> column. Although the task_index column may still be used for the default annotation, a dedicated column annotation.<annotation_source>.<annotation_type> is required to ensure it is loadable by the custom data loader.

Build docs developers (and LLMs) love

Get started for freeTalk to us