This guide takes you from a bare Linux machine to a running demo in as few steps as possible. All ROS 2 and Python dependencies — including MoveIt 2, DepthAI, and MediaPipe — are managed by Pixi, so you do not need a pre-installed ROS environment.Documentation Index
Fetch the complete documentation index at: https://mintlify.com/MRRP-lab/arm-demos/llms.txt
Use this file to discover all available pages before exploring further.
The hand tracking demo requires the
hand_landmarker.task MediaPipe model file in your working directory before you launch. Step 5 explains how to download it.Install Pixi
Pixi is the package manager used to install all ROS 2 and Python dependencies in a self-contained environment. If you have not installed it yet, follow the Pixi installation guide.Verify the installation:
Build the workspace
Build all colcon packages with symlink installs so that Python source changes take effect without rebuilding.This runs
colcon build --symlink-install inside the Pixi environment. Build outputs go to install/ and build/. Subsequent runs skip unchanged packages.Run the simulation demo
Launch the UR3e driver against mock hardware (no physical robot required). This brings up the full ros2_control stack with RViz opens and shows the robot model. You can inspect joint states and test MoveIt 2 planning without connecting to a real arm.
use_mock_hardware:=true.Download the MediaPipe hand landmark model
The hand tracking demo requires the This fetches
hand_landmarker.task model file at runtime. Download it to the current directory with:hand_landmarker.task (float16, latest) from Google’s MediaPipe model storage and saves it to the current directory. The arm_demo node expects the file at the path hand_landmarker.task relative to the working directory where you launch the node.Run the hand tracking demo (real robot)
With the This is an alias for:The OAK-D camera is accessed directly through the DepthAI SDK (not via a ROS camera node), so
hand_landmarker.task model present and the UR3e reachable on the LAN, start the hand tracking demo:use_camera:=false disables the separate camera launch. Once running, hold your hand in front of the camera — the arm tracks your index fingertip position in real time. Press q in the OpenCV window or Ctrl+C to stop.Run the pick-and-place demo (real robot)
To run the vision-guided pick-and-place demo with the vacuum gripper:This is an alias for:The arm moves to a scanning pose, detects an AprilTag marker on the target object via the OAK-D camera, plans a collision-free pick trajectory with MoveIt 2, activates the vacuum gripper, and deposits the object at a fixed drop-off point. The loop repeats until you call
ros2 service call /stop std_srvs/srv/Trigger or press Ctrl+C.Quick reference: Pixi tasks
| Task | Command | Description |
|---|---|---|
build | pixi run build | Build all packages with colcon --symlink-install |
sim | pixi run sim | Launch UR3e driver with mock hardware |
arm-demo | pixi run arm-demo | Hand tracking demo (real robot) |
demo-gripper | pixi run demo-gripper | Pick-and-place demo (real robot) |
download-mediapipe-model | pixi run download-mediapipe-model | Download hand_landmarker.task model file |
clean | pixi run clean | Remove build/, install/, and log/ directories |
Next steps
Environment setup
Detailed instructions for configuring your development machine and the Pixi environment.
Hardware setup
Connect and verify the UR3e, OAK-D Pro camera, and vacuum gripper.
Hand tracking demo
Deep dive into how the MediaPipe and MoveIt 2 pipeline works.
Pick and place demo
Learn how AprilTag detection drives autonomous pick-and-place cycles.