gitwtfhub

wtf is superscrape?

abirpahlwan/superscrape — explained in plain English

Analysis updated 2026-05-18

1PythonAudience · developerComplexity · 3/5Setup · moderate

TL;DR

SuperScrape scrapes a web page, uses a local Ollama model to extract structured JSON data from it, and saves the result into a Supabase database table.

Mindmap

mindmap
  root((superscrape))
    What it does
      Scrapes a URL
      Extracts JSON fields
      Saves to Supabase
    Tech stack
      Python
      Ollama
      Supabase
      Jina Reader
    Use cases
      Species data extraction
      Custom page scraping
      Upsert by URL
    Design
      Retry on bad JSON
      Raw markdown stored

Code map

Detail Auto

An interactive map of this repo's files and how they connect — its source is parsed live in your browser. Click Visualize to build it.

filefunction / class

Why would anyone build with this?

REASON 1

Scrape a web page about a plant or species and automatically fill in a structured database record for it.

REASON 2

Build a custom scraper for a different kind of page by swapping in a new JSON Schema and table.

REASON 3

Re-scrape the same URL later to update an existing database row instead of creating duplicates.

REASON 4

Keep the raw scraped markdown alongside extracted fields to debug what the AI model saw.

What's in the stack?

PythonOllamaSupabaseJina Reader

How it stacks up

abirpahlwan/superscrape0xustaz/streamgatea-bissell/unleash-lite
Stars111
LanguagePythonPythonPython
Setup difficultymoderatehardhard
Complexity3/54/54/5
Audiencedeveloperdeveloperresearcher

Figures from each repo's GitHub metadata at analysis time.

How do you spin it up?

Difficulty · moderate Time to first run · 1h+

Requires a Jina API key, a running Ollama model, and a Supabase project with a migration applied before first use.

No license is stated in the README, so usage rights are unclear.

Wtf does this do

SuperScrape is a small Python script that turns any web page into structured data saved in a database. It works in three steps: it fetches a URL through a service called Jina Reader and turns the page into clean markdown text, then it sends that markdown to a locally running Ollama model and asks it to pull out specific fields as JSON, and finally it saves or updates that JSON data as a row in a Supabase database table. The example that ships with the project is built around scraping plant species pages. A JSON Schema file describes exactly which fields to extract, things like taxonomy, common names, description, habitat, flowering period, and economic uses, and a matching database migration creates a table with columns for all of that structured information plus the original scraped markdown and a list of any image links found on the page. To run it, a person installs the Python dependencies, copies an example environment file and fills in a few required settings: an API key for Jina, which Ollama model to use, and the URL and service key for a Supabase project. A local Supabase database migration needs to be applied once before the tool is used. After that, running the script with a URL, and optionally a different schema file or table name, scrapes that one page and writes the extracted data into the chosen table. If the same URL is scraped again later, SuperScrape updates the existing row instead of creating a duplicate, since it treats the source URL as a unique key. If the Ollama model returns JSON that does not match the expected schema, the script automatically retries once with a stronger prompt asking it to try again. The full raw markdown from each fetch is kept in the database alongside the structured fields, mainly so a developer can debug what the extraction step actually saw. The README does not mention a software license.

Yoink these prompts

Prompt 1
Explain the pipeline SuperScrape uses to go from a URL to a Supabase database row.
Prompt 2
Walk me through setting up the .env file and running superscrape.py against a URL.
Prompt 3
How would I create a new schema file to scrape a different kind of page with SuperScrape?
Prompt 4
What happens in SuperScrape when Ollama returns invalid JSON during extraction?

Frequently asked questions

wtf is superscrape?

SuperScrape scrapes a web page, uses a local Ollama model to extract structured JSON data from it, and saves the result into a Supabase database table.

What language is superscrape written in?

Mainly Python. The stack also includes Python, Ollama, Supabase.

What license does superscrape use?

No license is stated in the README, so usage rights are unclear.

How hard is superscrape to set up?

Setup difficulty is rated moderate, with roughly 1h+ to a first successful run.

Who is superscrape for?

Mainly developer.

View the repo → Decode another repo

This repo across BitVibe Labs

Don't trust strangers blindly. Verify against the repo.