tylertreat/inversebloomfilter — explained in plain English
Analysis updated 2026-07-18 · repo last pushed 2015-02-03
Deduplicate events in a fast-moving data pipeline where duplicates appear close together in time.
Skip duplicate notifications in a messaging system that occasionally sends the same message twice within seconds.
Filter out repeated items in an event stream without storing every item you have ever seen.
Build a lightweight duplicate detector for time-localized repeats in streaming data.
| tylertreat/inversebloomfilter | bunnymq/bunnymq | d4l3k/wikigopher | |
|---|---|---|---|
| Stars | 15 | 15 | 15 |
| Language | Go | Go | Go |
| Last pushed | 2015-02-03 | — | 2019-06-25 |
| Maintenance | Dormant | — | Dormant |
| Setup difficulty | easy | hard | moderate |
| Complexity | 2/5 | 5/5 | 3/5 |
| Audience | developer | ops devops | developer |
Figures from each repo's GitHub metadata at analysis time.
Small Go library with no external dependencies, just import the package and start filtering duplicates.
Inverse Bloom filter is a small tool written in Go that helps programs quickly figure out whether they've already seen a particular piece of data. It's designed for situations where you're processing a fast-moving stream of events and want to skip duplicates without keeping a massive list of everything that's come before. Traditional Bloom filters can tell you "maybe I've seen this" but sometimes guess wrong in the other direction, occasionally saying something is new when it's actually a duplicate. This project flips that trade-off: it will never tell you an item is a duplicate if it truly hasn't been seen before, but it might occasionally miss a duplicate and let it through. Think of it like a security guard who never wrongly accuses an innocent person of being a repeat visitor, but every so often fails to recognize someone they've actually met before. This makes it useful for deduplicating events in a data pipeline, especially when duplicates tend to appear close together in time rather than scattered far apart. For example, if a messaging system accidentally sends the same notification twice within a few seconds, this filter would likely catch the second one and skip it. But if the same event reappeared hours later, the filter might have already forgotten about it and process it again. The implementation is based on an original concept by Jeff Hodges, with one notable change: it uses a faster, simpler hashing method called FNV-1a instead of the heavier MD5 algorithm. Since this tool isn't doing security-related work, the lighter hash makes sense for performance. The project is quite small and focused, doing one specific thing rather than trying to be a general-purpose deduplication library.
A small Go tool that helps programs skip duplicate data in fast-moving event streams. It never wrongly flags new data as a duplicate but may occasionally miss a repeat that appears much later.
Mainly Go. The stack also includes Go.
Dormant — no commits in 2+ years (last push 2015-02-03).
The repository does not include explicit license information, so usage rights are unclear.
Setup difficulty is rated easy, with roughly 5min to a first successful run.
Mainly developer.
This repo across BitVibe Labs
Don't trust strangers blindly. Verify against the repo.