> For the complete documentation index, see [llms.txt](https://doc.ancorasir.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://doc.ancorasir.com/2026spring/simulation-and-interaction/class-09-reinforcement-learning-for-humanoid-training.md).

# Class 09: Reinforcement Learning for Humanoid Training

A common pipeline for training humanoid robots with reinforcement learning (RL) progresses through three stages:

1. **Train in Isaac Lab**: Start by training the policy there, where data is cheap, safe, and fast to generate. This phase focuses on learning stable locomotion and basic behaviors.
2. **Sim2sim in MuJoCo**: Transfer the Isaac Lab policy into MuJoCo and evaluate/adapt it under different contact dynamics and environment settings. This improves robustness to simulator/model mismatch before hardware deployment.
3. **Sim2real**: Finally, deploy the policy on the real robot, using careful calibration and safety constraints. Real-world fine-tuning may be applied to bridge the remaining gap between simulation and hardware.

### Included implementations

The bundled projects retain their upstream layouts. Use this page as the course workflow, then use each project's own README for version-specific setup.

| Component              | Role                                                            | Documentation |
| ---------------------- | --------------------------------------------------------------- | ------------- |
| `unitree_rl_lab/`      | Isaac Lab training, policy export, and deployment configuration | README        |
| `unitree_mujoco/`      | MuJoCo sim-to-sim evaluation                                    | README        |
| `unitree_sdk2/`        | C++ hardware communication                                      | README        |
| `unitree_sdk2_python/` | Python hardware communication                                   | README        |

### Example Flow Using Unitree G1

* **Train in Isaac Lab**: Train the G1 locomotion policy in Isaac Lab (external training workspace), then export the trained policy checkpoint.
* **Sim2sim in MuJoCo**: Use `unitree_mujoco/` to load and evaluate the exported policy in MuJoCo, then use `unitree_rl_lab/` tooling/configs to run robustness checks and simulator-side adaptation.
* **Sim2real**: Deploy to hardware through `unitree_sdk2/` (C++ SDK) or `unitree_sdk2_python/` (Python SDK), which interface with the real G1 robot.

### Diagram

```
Isaac Lab Training      MuJoCo Sim2Sim                Sim2Real Deployment
------------------      -------------                 -------------------
Isaac Lab workspace     unitree_mujoco/               unitree_sdk2/
   |                    unitree_rl_lab/               unitree_sdk2_python/
   +---- checkpoint ----->   |                               |
                             +--------- validated policy -----+
```

### Quick run

> These commands are a teaching reference. Confirm the operating system, GPU driver, Isaac Sim/Isaac Lab versions, and Unitree SDK compatibility in the linked upstream documentation before deployment.

#### Train in Isaac Lab

* Step 1: Install Isaac Sim and Isaac Lab using the [official pip installation guide](https://isaac-sim.github.io/IsaacLab/main/source/setup/installation/pip_installation.html).
* Step 2: Install the bundled `unitree_rl_lab` package.

```bash
cd unitree_rl_lab
conda activate env_isaaclab
./unitree_rl_lab.sh -i
```

* Step 3: Train g1-23dof task

```bash
python scripts/rsl_rl/train.py --headless --task Unitree-G1-23dof-Velocity
```

* Step 4: Test the policy in Isaac Sim. A sample checkpoint is included under `logs/`, so training is not required for the first evaluation.

```bash
python scripts/rsl_rl/play.py --task Unitree-G1-23dof-Velocity \
  --checkpoint logs/rsl_rl/unitree_g1_23dof_velocity/2025-12-30_13-46-38/model_10000.pt
```

#### Sim2Sim: deploy in MuJoCo

Run the installation commands from the `02-DC-04-ReinforceL/` directory.

**Installation**

```bash
conda create --name unitree_mujoco python=3.11
conda activate unitree_mujoco

# 1: Install dependencies
sudo apt install -y libyaml-cpp-dev libboost-all-dev libeigen3-dev libspdlog-dev libfmt-dev
pip install mujoco pygame

# 2: Install the bundled unitree_sdk2 and unitree_sdk2_python packages
cd unitree_sdk2
mkdir -p build
cd build
cmake .. -DBUILD_EXAMPLES=OFF # Install on the /usr/local directory
sudo make install
cd ../..

cd unitree_sdk2_python

git clone https://github.com/eclipse-cyclonedds/cyclonedds -b releases/0.10.x
cd cyclonedds
mkdir -p build install
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=../install
cmake --build . --target install
cd ../..
export CYCLONEDDS_HOME="$PWD/cyclonedds/install"
pip install cyclonedds --no-binary cyclonedds
pip install -e .
cd ..

# 3: Configure unitree_mujoco according to unitree_mujoco/readme.md

# Step 4: Compile the robot_controller in unitree_rl_lab
cd unitree_rl_lab/deploy/robots/g1_23dof # or another robot configuration
# g1_23dof, modify CMakeLists.txt: set(CMAKE_CXX_STANDARD 17)
mkdir -p build
cd build
cmake .. && make
```

* Test

```bash
# Run unitree_mujoco
# In terminal 1: launch the MuJoCo simulation
# Set the robot in simulate_python/config.yaml to g1
# Set domain_id to 0
# Set enable_elastic_hand to 1
# Set use_joystck to 1
cd unitree_mujoco/simulate_python
python unitree_mujoco.py

# In terminal 2
# If libddsc.so.0 cannot be found, add the appropriate unitree_sdk2/thirdparty/lib path
# to LD_LIBRARY_PATH for your architecture.
# Set the policy path in unitree_rl_lab/deploy/robots/g1_23dof/config/config.yaml.
conda activate env_isaaclab
cd unitree_rl_lab/deploy/robots/g1_23dof/build
./g1_ctrl -n lo
# 1. press [L2 + Up] to set the robot to stand up
# 2. Click the MuJoCo window, and then press 8 to make the robot feet touch the ground.
# 3. Press [R1 + X] to run the policy.
# 4. Click the MuJoCo window, and then press 9 to disable the elastic band.
```

***


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://doc.ancorasir.com/2026spring/simulation-and-interaction/class-09-reinforcement-learning-for-humanoid-training.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.