wtf is magnet-dht?

Magnet-dht is a tool that collects magnet links from the BitTorrent network automatically. If you've ever used a magnet link to download a file, you know it's a convenient way to find and fetch resources without needing a central server. This project essentially sits on the network and gathers those links by the millions, without relying on any specific website. The tool works by joining the Distributed Hash Table (DHT) network, which is the decentralized system BitTorrent clients use to find each other without a central tracker. The project pretends to be a DHT node, connects to other nodes using UDP, and asks around for resources. As other nodes share information, the tool collects the magnet links they reference. It stores everything in Redis to automatically deduplicate entries, and uses multiple threads and processes to speed things up. The author reports collecting over a million magnet links in about four hours on a decent laptop. Beyond just collecting links, the project can also convert those magnet links into actual torrent files using a tool called aria2, and then parse the torrent files to see what specific resources they contain. This would appeal to someone building a search engine for downloadable content, a researcher studying what's available on the BitTorrent network, or really anyone curious about how decentralized file sharing works under the hood. It's a hands-on way to harvest data directly from the source rather than scraping download links from websites. One notable thing is that the author built this partly out of frustration with existing Python DHT crawlers, which were either undocumented or not compatible with Python 3. The project reflects an interest-driven approach to learning rather than being production-hardened, and the author notes that running it on a low-spec cloud server didn't go as well as on a local machine. You'll need Redis installed and optionally aria2 if you want the magnet-to-torrent conversion feature.