IDEAS-Lab-Northwestern/pi05-base-cabinet-pickup-joint-2cam-lora

π0.5 LoRA — Sim Cabinet-Pickup (joint controller, 2-cam)

A LoRA fine-tune of π0.5 (pi05_base) for one ManiGuard simulated manipulation task, trained with the openpi trainer.

Task — cabinet-pickup: place the paper towel holder into the cabinet open drawer and close it, without knocking anything over.

⚠️ Camera convention — this checkpoint uses the RIGHT overview, not the default left

The third-person overview is the dataset image_right stream (cab left view is low-quality). At inference / eval you must feed image_right pixels under the policy fixed key observation/image_left — sending the left camera is out-of-distribution and will degrade results. Routing config: Sim2CamLiberoDataConfig(external_cam="right"), recorded in the checkpoint train config (image_right -> observation/image_left -> base_0_rgb).

Model

• Warm-start: openpi pi05_base; LoRA on gemma_2b_lora + gemma_300m_lora. action_dim=32 (padded), action_horizon=16.
• Controller: JointController — 8-D joint state, 8-D absolute-joint action (7 arm joints trained as per-step deltas, gripper absolute; reconstructed to absolute at inference).
• Cameras (LIBERO 2-cam): base_0_rgb ← image_right (third-person overview — see warning above) + left_wrist_0_rgb ← wrist; third image slot zero-filled + masked. image_left unused.

Data

sim-cabinet-pickup-30-joint-3cam — 30 episodes / 22,684 frames; GELLO-teleop collected, re-rendered to joint + 3-cam.

Training

• 1x A100-80GB, bf16, full data-parallel (fsdp_devices=1).
• 2000 steps (~2 epochs), batch 24, cosine LR (peak 4.3e-5, warmup 200, decay 4.3e-6).
• Final train loss = 0.013.

Checkpoints

Step folders 1000/ 2000/ — 2000 is the final checkpoint.

Usage

Serve via the openpi policy server with config pi05_base_cabinet_pickup_joint_2cam_lora; feed the image_right overview under observation/image_left (see the camera warning above), plus observation/wrist_image and the 8-D joint observation/state.