Build a Self-Hosted Web Crawler for RAG with Draco

Web scraping breaks constantly. Third-party APIs charge per credit. You need control over your data ingestion pipeline. Draco is a single-binary, self-hostable crawler written in Rust. It converts URLs to markdown ready for LLM context.

1. Deploy the Binary

You do not need Docker. You do not need Node dependencies. Download the latest release from the GitHub repo.

curl -L https://github.com/0xchasercat/draco/releases/latest/download/draco -o draco

chmod +x draco

Run the server on port 3000.

./draco --port 3000

Verify it responds.

curl http://localhost:3000/health

You now have a scraping engine running locally. It handles concurrency and markdown conversion out of the box.

2. Ingest a Target Site

Send a POST request to the crawl endpoint. Pass the target URL and depth.

curl -X POST http://localhost:3000/crawl -H "Content-Type: application/json" -d '{"url": "https://example.com", "depth": 2}'

The response contains raw markdown. Save this to disk. Do not store HTML. LLMs waste tokens on tags. Markdown preserves structure without the noise.

For production, wrap this in a Python script.

import requests

res = requests.post("http://localhost:3000/crawl", json={"url": "https://docs.myapp.com"}) content = res.json()["markdown"]

Queue URLs if you need to crawl thousands of pages. Draco handles the fetching; your script handles the logic.

3. Feed Your RAG Pipeline

Take the markdown output. Chunk it by headers. Embed the chunks. Load them into your vector database.

Because Draco runs locally, latency is network-only. There is no external API roundtrip. This matters when crawling 500+ pages. You avoid rate limits from public scraping services. You also avoid sending proprietary internal docs to third-party processors.

Common pitfalls

Ignoring robots.txt: Draco respects crawl rules by default. Do not disable this for public sites. You will get IP banned.

JavaScript Heavy Sites: Draco fetches static content. If your target relies on client-side rendering, you will get empty markdown. Use a headless browser pre-step or choose targets wisely.

Memory Spikes: Rust is safe, but crawling deep trees consumes RAM. Limit your depth parameter. Start with depth=1 for testing.

Next step

Connect this output to a local embedding model like nomic-embed-text to keep the entire pipeline offline.