RTFF: Random-to-Target Fabric Flattening Policy using Dual-Arm Manipulator

Kai Tang1,†, Dipankar Bhattacharya2,†, Hang Xu3, Fuyuki Tokuda4, Norman C. Tien1, Kazuhiro Kosuge5
1The University of Hong Kong, 2Imperial College London, 3JD.COM, 4Tohoku University, 5City University of Hong Kong, †Equal contribution.
To appear in IEEE/RSJ IROS 2026, Pittsburgh
arXiv Mesh GAT

Abstract

Robotic fabric manipulation remains challenging due to fabric deformability and occlusions from wrinkles and the manipulator. This paper defines Random-to-Target Fabric Flattening (RTFF) as the task of bringing a randomly wrinkled fabric to an arbitrary user-specified wrinkle-free target pose. RTFF requires simultaneous flattening and pose alignment, where the two objectives are inherently coupled since flattening the fabric displaces its pose, while realigning it tends to introduce wrinkles. To solve this task, this paper anchors both the current and target fabric states to the same template mesh, enabling direct vertex-level wrinkle and pose assessment without registration. Building on this representation, a hybrid Imitation Learning--Visual Servoing (IL--VS) RTFF policy is proposed. A novel Mesh Action Chunking Transformer (MACT) leverages structured mesh observations to achieve goal-conditioned coarse alignment from a compact demonstration set, after which VS ensures precise convergence to the target. The policy is validated on a real dual-arm teleoperation system, demonstrating precise alignment to unseen target poses, fabric types, and scales.

Supplemental Video

Method

RTFF brings a randomly wrinkled fabric to a user-specified wrinkle-free target pose. Both the current state and the target are anchored to the same CAD-derived template mesh, so wrinkle severity and pose error can be read directly at corresponding vertices—without registration.

Task Setup

RTFF task overview with predicted and target meshes
Green predicted mesh vs. Red target mesh, with depth cues and occlusion-aware prediction.

Overall Pipeline

Overview of the RTFF policy framework
Mesh state estimation, then a hybrid IL–VS manipulation policy on a dual-arm platform.
  1. State estimation. RGB-D → segmented depth → GAT mesh prediction → sim-to-real refinement, yielding a wrinkle-aware predicted mesh even under end-effector occlusion.
  2. IL (MACT) for coarse alignment. A mesh-conditioned Action Chunking Transformer flattens wrinkles and roughly aligns the fabric to the target from a compact set of teleoperation demos.
  3. VS for fine alignment. Once the fabric is flat, near the target, and dual-grasped, position-based visual servoing drives precise convergence to the target pose.

MACT Architecture

MACT policy architecture
MACT takes depth, predicted/target meshes with alignment error, and end-effector states, and predicts dual-arm action chunks.

Results

Evaluated on a real dual-arm teleoperation system across 20 initial/target pairs. The full RTFF policy converges reliably; mesh conditioning and VS are both essential.

Key Metrics

0.011 m Terminal alignment error
0.926 Mean IoU
0 / 20 Failures

Execution Sequences

RTFF execution sequences with and without visual servoing
With VS (top) and without VS (bottom). Green: predicted mesh; Red: target.

Ablation Study

Without VS, error rises and variance grows. Without predicted mesh and alignment error—even when segmented depth is added—the policy fails to reach the target.

Method Error (m) IoU Fail.
Ours 0.011 ± 0.002 0.926 ± 0.018 —
w/o VS 0.028 ± 0.010 0.871 ± 0.050 —
w/o mesh & error — — 5/5
w seg. depth, w/o mesh & error — — 5/5
Unseen fabric type 0.0145 ± 0.004 0.911 ± 0.031 —
Unseen scale (80%) 0.021 ± 0.004 0.893 ± 0.022 —

Generalization

Trained on one fabric, the same policy generalizes to unseen materials and an 80% scale without retraining.

Generalization across fabric types and scales
Different fabric types (a–c) and a reduced scale (d).

BibTeX

@misc{tang2026rtffrandomtotargetfabricflattening,
        title={RTFF: Random-to-Target Fabric Flattening Policy using Dual-Arm Manipulator}, 
        author={Kai Tang and Dipankar Bhattacharya and Hang Xu and Fuyuki Tokuda and Norman C. Tien and Kazuhiro Kosuge},
        year={2026},
        eprint={2510.00814},
        archivePrefix={arXiv},
        primaryClass={cs.RO},
        url={https://arxiv.org/abs/2510.00814},
}