ClothDojo: Learned Data Generation and Benchmarking for Bimanual Garment Flattening and Folding
Privileged policies generate complete trajectories.
Re-rendered into RGB training data.
Beyond demonstrated garments.
Human supervision, generation & held-out assets
The 500 garments form a structurally diverse core pool for trajectory generation and evaluation. Human demonstrations and corrections cover only part of the garment space and are not restricted to this pool.
Privileged policies learn from that human experience, then autonomously generate trajectories on garments without human demonstrations or corrections. This generalization extends the data beyond manually collected experience.
Held-out garments test generalization. The split depends on the experiment: an asset unseen in the privileged policy's human supervision is different from an asset unseen in downstream RGB-policy training. The latter evaluation includes held-out assets from both this core pool and the larger converted library. The atlas shows asset diversity, not a color-coded train/test partition.
From trajectories to RGB training data.
About generation & re-rendering
NOCS-conditioned privileged policies autonomously generate complete, end-to-end task trajectories in simulation. Recorded motion is then re-rendered into RGB training observations. Human demonstrations and corrections teach the generation policies; they are not required to collect each new generated trajectory.
Reported throughput: approximately 1,152 trajectory executions per GPU-day on one RTX 5090 D, excluding RGB re-rendering. This measures rollout executions rather than retained successes.
The learned generator produces trajectories for new RGB training observations and compatible robots. Our open-source dataset contains 7,848 trajectories: 3,992 flattening and 3,856 folding, covering 400 and 399 garments respectively. These are the dataset counts, not the generation capacity of the policies. Additional trajectories can be generated in simulation and re-rendered.
The opening film shows twelve successful model-generated trajectories across six garments, using a fixed camera and two worktables. Each round compares flattening on the left and folding on the right of the same garment identity and colour. These task actions are policy rollouts, rather than human demonstrations; the policies were trained using demonstrations and corrections.
Task motion follows recorded states through successful completion at 2× replay. Both robots return to home before the next garment appears. The home return and short asset dissolves are scripted presentation; the camera stays fixed. Throughput reports trajectory execution, excluding RGB rendering.
The re-rendering pipeline varies cloth, table and floor materials, background scenes and lighting. The RGB preview brings together 128 recorded trajectories, with their original camera views and sampled appearances: flattening on the left and folding on the right. Recordings play at 2× speed in a continuous loop; completed trajectories briefly hold their final frame and restart outside the visible area.
One trajectory. Four robots.
About robot transfer
Four robots replay kinematic retargets of one saved PIPER demonstration: ALOHA 2, X5, Franka dual arms and UR5e. The source robot is omitted from this overview. All share the original cloth-state sequence. This shows motion transfer across embodiments, not autonomous closed-loop policy evaluation on four robots. The opening film shows different model-generated trajectories, rather than this source demonstration.
The complete recorded motion is shown at 2× speed and 25 rendered frames per second. The post-task return to home and short loop dissolve are scripted presentation. Four independent fixed orthographic views are composited in a 2×2 layout at equal scale, with complete worktables and their soft shadows. This is a composition of separate views, rather than one shared physical scene. Robot and cloth states stay unchanged. Cotton material, lighting and contact shadows are presentation choices; appearance randomization is shown in the RGB card.
Robot replay sources & rendering details ↗Start with the code preview.
CPU quickstart
Score a saved cloth state.
git clone --depth 1 https://github.com/clothdojo/clothdojo.github.io.git
cd clothdojo.github.io
uv sync --frozen
uv run --frozen python scripts/score_fold.pyEvaluate a policy ↗
Connect a policy server and evaluate against fixed initial states.
02Generate & render ↗
Collect trajectories and export RGB training episodes.
03Train an RGB policy ↗
Inspect datasets and train with task-matched model bundles.
Requirements & expected output
The code preview includes a CPU scoring example, one garment, and four initial states per task. Full model bundles, datasets and visual assets are forthcoming.
This example runs on CPU, with no simulator or model checkpoint. It checks a single saved state, not a complete rollout.
Expected output: frame_passes_threshold: true
Generation, rendering and training workflows require the relevant model/data/visual resources. See each guide for availability and requirements.