wtf is avtensor?

avtensor is a library from Runway that decodes video, audio, and images directly into PyTorch tensors, the array format used throughout machine learning code. It is written in Rust with Python bindings, and uses FFmpeg under the hood to do the actual decoding work. It is designed for building data-loading pipelines for training or running machine learning models, where getting media into tensor form quickly and efficiently matters a lot. Instead of decoding a video file and then separately converting it into a tensor, avtensor moves data straight from FFmpeg's decoded frames into a tensor in one step. It can read from local files, from cloud storage locations like Google Cloud Storage or S3 buckets, or from any regular web URL. While decoding, it can also resize video, change its frame rate, resample audio, and normalize loudness, all inside FFmpeg before the data ever becomes a tensor. It correctly converts color information using the video's own metadata, and it can tone-map high dynamic range video down to standard range. The project positions itself as an alternative to torchcodec, PyTorch's own official decoder, noting that both use FFmpeg and produce identical color output on CPU decoding, but avtensor adds native cloud storage support, decodes video and audio together in a single pass, and is noticeably faster when resizing during decode. It also supports NVIDIA GPU-accelerated decoding through NVDEC, including an option where decoded video frames stay entirely on the GPU rather than being copied back to the computer's main memory, which speeds up pipelines that also do their model training on the GPU. Installing avtensor requires Python 3.8 or newer, a compatible version of PyTorch, FFmpeg 7.1 or newer built with shared libraries, and the Rust toolchain at build time, since the Python package compiles against your local environment. It is released under the Apache 2.0 license.