wtf is rynnworld-4d?

RynnWorld-4D is a research project from Alibaba's DAMO Academy that builds AI models to help robots understand and predict how the world changes over time, specifically so a robot arm can plan how to grab and move objects. Instead of only predicting what a video will look like next, like many similar AI systems do, this project predicts three things together for every frame: the color image, how far away each part of the scene is, and how each pixel is moving. Combining these lets the system build a rough 3D picture of the scene and how it changes, rather than just a flat 2D guess. The project trains this prediction model in three stages, gradually teaching it to keep the color, depth, and motion predictions consistent with each other rather than developing them separately. Once trained, it can generate matching color, depth, and motion videos for a given starting scene and task. On top of this world model, the project adds a second piece called RynnWorld-4D-Policy, which is meant to actually control a robot with two arms picking up and placing objects. Rather than fully predicting a video and then deciding what to do, this policy reads the model's internal representation directly, which lets it make decisions many times per second so the robot can react quickly during a task. Sample data for training the policy, including short video clips of a robot picking and placing objects, is included so the training code can be tried right after downloading the project. Running any of this requires substantial setup: a Python environment with specific deep learning libraries, a large pretrained video generation model downloaded separately, and a GPU, since the underlying models are large diffusion based systems built for research rather than everyday use. The project is released under the Apache License 2.0, and it builds on several other open source projects for depth estimation and optical flow, whose original licenses are kept alongside the vendored code.