wtf is yume-pdq?

yume-pdq helps platforms quickly check user-uploaded images against a database of known harmful content, such as child exploitation material. It implements PDQ, a hashing method originally from Meta's ThreatExchange project, which creates a compact fingerprint of an image so you can compare images without storing the originals. The library is designed to slot into an existing image-processing pipeline with minimal overhead, adding only microseconds of CPU time per request. The project focuses on making both the hashing and the matching extremely fast. It uses specialized CPU instructions (like AVX512) and can optionally offload the matching work to a GPU via Vulkan. The end result is that on a modern server CPU, it can hash an image and search a database of 10 million known hashes in about 20 milliseconds. It guarantees 100% recall, meaning it will never miss a matching hash, and it automatically checks all eight rotations and reflections of an image in a single pass. A trust-and-safety team at a social media company would use this to screen uploads in near real-time before they reach other users. Because it can process video frames at thousands of frames per second on a single server, it could also be used to scan live video streams or newly uploaded video files as they arrive. It also supports running on mobile hardware, tested on Apple Silicon and recent Google Pixel phones, so the same screening logic could run on-device if an app wanted to check images before uploading them. A few design choices are notable. The project deliberately avoids vendor lock-in, supporting multiple hardware vendors rather than tying itself to one company's chips. It also makes a specific accuracy tradeoff: the optimized hashing code is fast and statistically compatible enough for matching, but the README explicitly warns that you should not use these optimized hashes to populate a shared external database. They're meant for real-time screening, not for long-term hash storage in an external system. The code is also hardened against certain memory-safety attacks, with optional build flags that add security protections with under a 10% performance cost.