gitwtfhub

wtf is unblock-web?

kevinnft/unblock-web — explained in plain English

Analysis updated 2026-05-18

1PythonAudience · developerComplexity · 3/5LicenseSetup · easy

TL;DR

A Python tool that helps AI agents fetch web pages blocked by JavaScript rendering, bot challenges, or regional censorship, by picking the right method automatically.

Mindmap

Mindmap is still cooking. Refresh in a few.

Code map

Detail Auto

An interactive map of this repo's files and how they connect — its source is parsed live in your browser. Click Visualize to build it.

filefunction / class

Why would anyone build with this?

REASON 1

Let an AI agent reliably fetch pages that fail with a plain HTTP request, such as JavaScript-heavy sites.

REASON 2

Bypass ISP-level DNS blocking or geo-restrictions when scraping content for an agent.

REASON 3

Get around Cloudflare Turnstile challenges without paying for a commercial scraping service.

What's in the stack?

PythonPatchrightDocker

How it stacks up

kevinnft/unblock-weba-bissell/unleash-liteabhiinnovates/whatsapp-hr-assistant
Stars111
LanguagePythonPythonPython
Setup difficultyeasyhardhard
Complexity3/54/53/5
Audiencedeveloperresearcherdeveloper

Figures from each repo's GitHub metadata at analysis time.

How do you spin it up?

Difficulty · easy Time to first run · 5min

Requires Python 3.10 or newer, local Chromium is used instead of a paid scraping service.

Wtf does this do

unblock-web is a Python library and command-line tool that helps AI agents fetch web pages that would normally fail due to various blocking mechanisms. The problem it addresses: many websites return useless content to automated requests, JavaScript-rendered pages that require a real browser, Cloudflare Turnstile challenges that block bots, ISP-level DNS blocking (the README specifically notes it was field-tested against Indonesian internet censorship, known locally as Internet Positif), and login walls on sites like x.com. Rather than trying one approach blindly, the library uses a four-tier decision tree that picks the right method based on what type of block it detects. Tier 0 is a plain HTTP fetch for simple static pages. Tier 1 uses a stealthy browser called Patchright (a modified Chromium that avoids browser fingerprinting) to handle JavaScript-rendered pages and Cloudflare Turnstile challenges. Tier 2 routes requests through TinyFish, a free geo-proxy service, to bypass ISP-level DNS blocks or geo-restricted content. Tier 3 uses public aggregator mirrors for content that won't render without authentication. In code you call a single fetch function and it automatically selects the appropriate tier, or you can force a specific one. For AI agent integration, the README shows examples of wiring the tool's health check into session startup hooks. The library is installable via pip, available as a Docker container for zero-install use, and requires Python 3.10 or newer. It uses local Chromium rather than paid scraping services and is licensed MIT.

Yoink these prompts

Prompt 1
Show me how to call unblock-web's fetch function so it automatically picks the right tier for a blocked page.
Prompt 2
Explain the difference between unblock-web's four fetch tiers and when each one is used.
Prompt 3
Help me wire unblock-web's health check into an AI agent's session startup hook.
Prompt 4
Walk me through running unblock-web as a Docker container instead of installing it locally.

Frequently asked questions

wtf is unblock-web?

A Python tool that helps AI agents fetch web pages blocked by JavaScript rendering, bot challenges, or regional censorship, by picking the right method automatically.

What language is unblock-web written in?

Mainly Python. The stack also includes Python, Patchright, Docker.

How hard is unblock-web to set up?

Setup difficulty is rated easy, with roughly 5min to a first successful run.

Who is unblock-web for?

Mainly developer.

View the repo → Decode another repo

This repo across BitVibe Labs

Don't trust strangers blindly. Verify against the repo.