wtf is joyai-video-edit?

JoyAI-Video-Edit is a research project for editing video in real time, using plain language instructions instead of manual video editing tools. A user provides a live camera feed or an uploaded video, along with a written instruction such as changing the background or converting the footage to a particular art style, and the system edits each frame as it arrives. It does not need to see the whole video first, does not require a fixed video length, and does not need to look ahead at future frames before making a decision. Under the hood, the system combines three components: a model that reads and understands the instruction and video content together, a video compression model built for this streaming use case, and a large diffusion transformer with 16 billion parameters that generates the edited frames. The team reports the full pipeline running at just over 30 frames per second at 720p resolution in their testing, which they describe as a step toward interactive, streaming video editing rather than the slower batch processing that is common today. The kinds of edits supported include changing a subject's appearance, editing specific objects, adding or removing things from a scene, replacing the background, applying a different visual style, and matching motion or a reference image. The project includes example clips and side by side comparisons of source and edited footage. The maintainers note this is a preview release: deployment on consumer graphics cards, an improved model version, and the full training pipeline are all still in progress. Code, a paper, and pretrained model weights are already available for anyone who wants to try it, and the license is Apache 2.0.