ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation

Yunao Huang^1,2 Shiyu Sang^1,2 Haotao Lu¹ Suting Ni¹ Shijie Wu¹ Ziyang Guo¹ Ye Shi^1,2 Jingya Wang^1,*

¹ ShanghaiTech University ² InstAdapt ^* Corresponding author

Abstract

Contact-rich robot manipulation requires physical interaction cues that are often invisible to cameras, making tactile sensing essential for robust control. However, scaling visuo-tactile robot learning remains difficult because real tactile interaction data are expensive to collect, hardware-dependent, and limited in task and scene diversity. We present ViTacWorld, an action-conditioned visuo-tactile world model for scalable contact-rich robot manipulation. ViTacWorld leverages public real tactile datasets and a constructed simulation environment to scale visuo-tactile-action data, exploiting the fact that tactile signals are directly grounded in physical contact and can exhibit a smaller simulation-to-real gap than purely visual observations. The model is first pretrained with large-scale real and simulated visuo-tactile trajectories, and then finetuned with real-world policy rollouts to better match downstream manipulation behaviors. Given robot actions, ViTacWorld predicts temporally aligned visual observations and tactile feedback, enabling visuo-tactile-action rollout generation. To the best of our knowledge, ViTacWorld is the first framework that uses a world model for robot visuo-tactile-action trajectory generation and policy evaluation. It serves two roles: synthesizing rollouts to improve downstream tactile policies, and evaluating policies by predicting action-conditioned visuo-tactile outcomes under controlled action sequences. Experiments on contact-rich manipulation tasks show that ViTacWorld generates physically meaningful rollouts, improves policy performance through scalable data augmentation, and enables action-conditioned policy evaluation.

Video

Method

Overview of the ViTacWorld architecture, pretraining and finetuning pipeline, and its use for policy improvement and evaluation. — **Figure 1: Overview of ViTacWorld.** Right side shows scaling training pipeline, where real visuo-tactile data and task-aligned simulation data are used for pretraining, followed by real robot task data and policy rollout data for target-domain tuning. Left side illustrates the action-conditioned visuo-tactile world model and its rollout generation process. Visual and tactile streams are encoded into latent tokens and modeled with a view-aware DiT to produce policy-conditioned rollouts, which support both dataset augmentation and policy evaluation.

Results

Policy Improvement with Generated Rollouts

ViTacWorld-generated visuo-tactile rollouts provide additional supervision for downstream policies. Repeating the dream-data augmentation process with a stronger rollout policy yields further gains.

42.5% Expert only

67.5% + Round-1 rollouts

80.0% + Round-2 rollouts

Data source	Method	Charger Plugging	Cucumber Peeling	U-Block Insertion	Cuboid Insertion	Avg.
Expert only	ACT + tactile	0	0	30	30	15.0
	π_0.5	10	30	60	40	35.0
	π_0.5 + tactile	20	40	70	40	42.5
Expert + Round-1 rollouts	ACT + tactile	10	20	40	40	27.5
	π_0.5	30	60	60	40	47.5
	π_0.5 + tactile	40	80	80	70	67.5
Expert + Round-2 rollouts	ACT + tactile	20	50	50	50	42.5
	π_0.5	40	80	90	50	65.0
	π_0.5 + tactile	60	90	90	80	80.0

Real-robot success rates (%). Additional dream-data augmentation results. Success rates are reported in percentage. “Round-1 rollouts” denotes the ViTacWorld-generated rollouts used in the main paper. “Round-2 rollouts” further adds a second round of dream rollouts generated by the first-round augmented π_0.5 + tactile policy.

Expert-only and ViTacWorld-augmented tactile policies compared over a real-robot rollout. — **Figure 2: Qualitative comparison of real-robot rollouts using the same π_0.5 + tactile policy architecture.** The top row shows the policy trained only with expert demonstrations, while the bottom row shows the policy trained with the same expert demonstrations augmented by ViTacWorld-generated rollouts.

Additional expert-only and ViTacWorld-augmented policy comparisons across contact-rich robot tasks. — **Figure 5: Additional qualitative comparisons of real-robot rollouts across contact-rich tasks.** For each task, the top row shows the π_0.5 + tactile policy trained only with expert demonstrations, while the bottom row shows the same policy architecture trained with expert demonstrations augmented by ViTacWorld-generated rollouts. The augmented policy produces more reliable pickup, alignment, and contact adjustment behaviors, leading to successful task completion in cases where the expert-only policy fails.

Visuo-Tactile World Model Quality

Pretraining improves action-conditioned prediction across the main view, wrist view, and tactile stream. Task-aligned simulation further improves object geometry, motion consistency, and contact prediction.

Ground-truth and predicted visual, wrist, and tactile observations from ViTacWorld and ablated variants. — **Figure 3: Qualitative world-model predictions on held-out validation clips.** The model receives the first observation frame and the logged action sequence, then predicts future visual and tactile observations.

Policy Evaluation with ViTacWorld

Starting from the same initial conditions, ViTacWorld predicts policy outcomes close to real-robot execution and provides a conservative evaluation signal before deployment.

Evaluation source	Charger Plugging	Cucumber Peeling	U-Block Insertion	Cuboid Insertion	Avg.
Real World	40	80	80	70	67.5
ViTacWorld	30	60	70	70	57.5

Policy evaluation with ViTacWorld. Success rates are reported in percentage. We evaluate the first-round augmented π_0.5 + tactile policy. Real-robot success rates are measured over 10 trials per task. ViTacWorld uses the same initial observations and samples three imagined rollouts per initial condition; the predicted outcome is assigned by majority vote.