Scrape a web page about a plant or species and automatically fill in a structured database record for it.
Build a custom scraper for a different kind of page by swapping in a new JSON Schema and table.
Re-scrape the same URL later to update an existing database row instead of creating duplicates.
Keep the raw scraped markdown alongside extracted fields to debug what the AI model saw.
| abirpahlwan/superscrape | 0xustaz/streamgate | a-bissell/unleash-lite | |
|---|---|---|---|
| Stars | 1 | 1 | 1 |
| Language | Python | Python | Python |
| Setup difficulty | moderate | hard | hard |
| Complexity | 3/5 | 4/5 | 4/5 |
| Audience | developer | developer | researcher |
Figures from each repo's GitHub metadata at analysis time.
Requires a Jina API key, a running Ollama model, and a Supabase project with a migration applied before first use.
SuperScrape is a small Python script that turns any web page into structured data saved in a database. It works in three steps: it fetches a URL through a service called Jina Reader and turns the page into clean markdown text, then it sends that markdown to a locally running Ollama model and asks it to pull out specific fields as JSON, and finally it saves or updates that JSON data as a row in a Supabase database table. The example that ships with the project is built around scraping plant species pages. A JSON Schema file describes exactly which fields to extract, things like taxonomy, common names, description, habitat, flowering period, and economic uses, and a matching database migration creates a table with columns for all of that structured information plus the original scraped markdown and a list of any image links found on the page. To run it, a person installs the Python dependencies, copies an example environment file and fills in a few required settings: an API key for Jina, which Ollama model to use, and the URL and service key for a Supabase project. A local Supabase database migration needs to be applied once before the tool is used. After that, running the script with a URL, and optionally a different schema file or table name, scrapes that one page and writes the extracted data into the chosen table. If the same URL is scraped again later, SuperScrape updates the existing row instead of creating a duplicate, since it treats the source URL as a unique key. If the Ollama model returns JSON that does not match the expected schema, the script automatically retries once with a stronger prompt asking it to try again. The full raw markdown from each fetch is kept in the database alongside the structured fields, mainly so a developer can debug what the extraction step actually saw. The README does not mention a software license.
SuperScrape scrapes a web page, uses a local Ollama model to extract structured JSON data from it, and saves the result into a Supabase database table.
Mainly Python. The stack also includes Python, Ollama, Supabase.
No license is stated in the README, so usage rights are unclear.
Setup difficulty is rated moderate, with roughly 1h+ to a first successful run.
Mainly developer.
This repo across BitVibe Labs
Don't trust strangers blindly. Verify against the repo.