Imitation Learning

The SO101 ROS 2 workspace integrates teleoperation, recording, and dataset conversion tools so you can generate demonstrations for downstream imitation learning pipelines.

Prerequisites

  • A working SO101 ROS 2 setup with leader and follower communication verified. Follow the getting started guide to prepare and launch the system.

  • Bridge and camera configurations updated to match your hardware setup.

  • (Optional) NVIDIA Isaac Sim 5.0 installed for simulator-based teleoperation. You can use the source code installation guide.

Real teleoperation

Launch the bridge with the physical leader and follower hardware:

ros2 launch so101_bringup so101_teleoperate.launch.py mode:=real display:=true

Monitor the logs for connection issues and confirm that the follower mirrors the leader motions while RViz visualises the joint states and cameras feeds.

Isaac Sim teleoperation

Use the simulator workflow to stream demonstrations into Isaac Sim.

  1. Bringup IsaacSim in one terminal:

    ${ISAACSIM_PATH}/isaac-sim.sh

  2. Load your ready-made usd file or use the provided example scene located at so101_description/usd/so101_new_calib.usd.

  3. Launch in a second terminal the teleoperation pipeline connected to the Isaac transport topics:

    ros2 launch so101_bringup so101_teleoperate.launch.py mode:=isaac display:=true

  4. Start simulation.

This reuses the teleoperation pipeline while switching the interfaces to the Isaac transport topics so you can stream demonstrations from the leader arm directly into the simulator using Isaac ROS2 Bridge.

You should now be able to move the leader arm and see the follower in IsaacSim mimicking its motions in real time and RViz which visualises the follower cameras and follower state comparing to the leader state.

Record demonstrations with system_data_recorder

The workspace ships with launch files for the system_data_recorder package to capture rosbag2 datasets.

  1. Adjust so101_bringup/config/so101_sdr.yaml to list the topics you want to record, set bag_name_prefix and copy_destination.

  2. Start your teleoperation session (real or Isaac).

  3. Launch the recorder:

    ros2 launch so101_bringup so101_record.launch.py

  4. Drive the lifecycle transitions from another terminal:

    ros2 lifecycle set /sdr configure
ros2 lifecycle set /sdr activate

  5. Stop the recording when the demonstration is complete:

    ros2 lifecycle set /sdr deactivate
ros2 lifecycle set /sdr shutdown

Using the Keyboard Commander

For improved usability, the system_data_recorder package includes an SDRKeyboardCommander node that sends lifecycle transition requests to the /sdr node based on keyboard input.

This allows you to control the recorder without manually typing lifecycle commands.

Running the Commander

  1. In one terminal, launch the recorder:

    ros2 launch so101_bringup so101_record.launch.py

  2. In a second terminal, start the keyboard commander:

    ros2 run system_data_recorder sdr_commander

Keyboard Controls

Once running, the commander listens for key presses and triggers the corresponding lifecycle transitions on the /sdr node.

Key

Action

Lifecycle Transition

c

Configure

CONFIGURE

a

Activate (start recording)

ACTIVATE

d

Deactivate (pause recording)

DEACTIVATE

l

Cleanup

CLEANUP

s

Shutdown

SHUTDOWN

g

Get State

Query and print state

h

Help

Print help menu

q

Quit

Exit commander node

The resulting rosbag2 dataset is saved under the directory specified in copy_destination with the prefix from bag_name_prefix.

You can inspect the produced bag using:

ros2 bag info <bag_path>

For additional options and advanced usage, refer to the system_data_recorder documentation .

Convert rosbag2 datasets

Convert captured rosbag2 files into Lerobot datasets with the so101_rosbag2lerobot_dataset utilities.

  1. First install the package:

conda activate lerobot_ros2
git clone https://github.com/nimiCurtis/so101_rosbag2lerobot_dataset.git
cd so101_rosbag2lerobot_dataset
pip install so101_rosbag2lerobot_dataset

  1. Prepare a YAML configuration describing the topics and output directory. An example config is shown in the package config directory.

so101-rosbag2lerobot --config <path_to_your_config.yaml>

  1. Inspect results with lerobot-dataset-viz or online with Huggingface Lerobot Dataset Visualizer.

Training

Finetune a VLA model on the collected dataset using directly the lerobot package. Check out the official tutorials on imitation learning with lerobot.

Deployment

TBD …

Describe how to deploy the trained policy back to the SO101 hardware.