ObjectWorld: Scaling Object-Centric World Models with a Large-Scale 3D Full-Object Flow Dataset

From diverse, uncalibrated videos to dense motion over complete object surfaces—and from predicted motion to executable robot trajectories.

Explore the data Paper (PDF)

Supplementary video

Video overview

The idea

A complete motion interface between perception and action

ObjectWorld predicts dense trajectories over an object's complete surface, including regions hidden in the input view. The representation captures both rigid motion and non-rigid deformation while remaining independent of a particular robot embodiment.

ObjectWorld overview: dataset, object-centric world model, and real-world execution
ObjectWorld learns executable full-object flow from large-scale real-world and simulated manipulation data.
1,210.1hours of 3D full-object flow
755.6hours of real-world video
454.5hours of simulation
488K+trajectories and clips

Interactive dataset

Rotate the world. Follow the flow.

Explore 28 curated examples: 12 captured from real-world videos and 16 generated in simulation. Drag to rotate 360°, scroll to zoom, and scrub through time.

Loading scene index…
↔ drag · rotate    scroll · zoom
t000 / 00
RGB scene full-object surface pink → blue flow history camera pose

Data at scale

One pipeline, heterogeneous motion

The automatic annotation pipeline turns in-the-wild manipulation videos into complete 3D object trajectories without requiring robot actions, calibrated cameras, depth measurements, or pre-scanned object meshes.

Automatic full-object flow annotation pipeline
Automatic annotation from raw video to full-object flow.
Object-centric world model architecture
Task-conditioned generative prediction over complete object geometry.

Object-centric prediction

See the task. Reconstruct the object. Predict its future.

The model combines task-conditioned visual-language features with a joint 3D encoding of object and scene geometry. It generates 48 future frames for 512 sampled object points in a single dense trajectory.

512surface points
48future frames
3Ddense trajectories

Key results

Better flow, stronger execution

ObjectWorld improves dense flow prediction and transfers the resulting motion target across scenes, environments, and robot embodiments.

All-frame flow error6.22cm

versus 10.24 cm for the strongest reported baseline.

61.0% within 5 cm
Simulation execution29/ 30

successful trials across in-domain and held-out environments.

Real-world execution13/ 15

successful trials, compared with 10/15 for the flow baseline.

Annotation precision93.3%

for the high-quality automatic filtering gate.

Complete confusion matrices for the annotation filtering gates
Expert validation of the Golden and Silver annotation-filtering gates.
Data scaling results for simulated and real data
Combining real and simulated data performs best.

Execution visualization

From predicted flow to robot motion

Compare the predicted 3D full-object flow with the synchronized rollout. Examples span real-robot executions, LIBERO, and ManiSkill.

Loading execution index…
↔ drag · rotate    scroll · zoom
t000 / 00
RGB scene object surface pink → blue flow history camera poseFlow completes at 48 / 60