Preparing Website Content for RAG: Clean Markdown Pipelines

Why clean Markdown beats raw HTML for RAG, how to strip boilerplate and crawl sitemaps, and how to build the pipeline with or without your own infrastructure.

Why Markdown beats raw HTML for RAG

A RAG pipeline is only as good as the text you embed. Feed it raw HTML and three things go wrong at once:

Token bloat: tags, class attributes, inline styles, and script fragments can multiply the size of a document without adding a word of meaning — you pay for those tokens at embedding time and again at generation time
Noisy embeddings: markup mixed into text shifts vector representations, so similar content embeds less similarly
Bad chunk boundaries: character-count splitters slice HTML mid-tag and mid-sentence, producing chunks that are incoherent on their own

Markdown solves all three. It keeps the semantic skeleton — headings, lists, links, code blocks — and discards presentation. Crucially, the heading hierarchy gives you natural chunk boundaries: header-based splitters (LangChain's MarkdownHeaderTextSplitter, LlamaIndex's Markdown parsers) can cut on ## sections so every chunk is a self-contained topic with its own heading for context.

Step 1: Extract content, not pages

Every page on a site shares navigation menus, footers, sidebars, and cookie banners. If you convert whole pages, that boilerplate is embedded hundreds of times — and because it appears everywhere, it matches everything. Retrieval starts returning chunks of your nav bar.

Strip boilerplate before conversion with a content extractor. Well-established options:

Mozilla Readability — the extraction engine behind Firefox Reader View, available as a JavaScript library
trafilatura — a widely used Python library for extracting main text from web pages
Crawl4AI content filtering — Crawl4AI can prune low-value page regions and emit only the main content as Markdown

Step 2: Crawl at the sitemap level

For anything bigger than a handful of URLs, do not crawl by following links — use the site's sitemap.xml. It enumerates every URL the site owner wants indexed, which means no crawler traps, no duplicate URLs with tracking parameters, and a clean list you can diff against on the next run. Record the URL and fetch date alongside each document; you will want both for citations and for refreshing stale content later.

Step 3: Chunk on structure, then size

Split on Markdown headings first, so chunks align with the author's own topic boundaries
Apply a token-size cap only to oversized sections, not as the primary splitter
Never split inside a code block — half a code sample is worse than none
Attach metadata to every chunk: source URL, page title, and the heading path (for example, Guide > Installation > Docker) so the LLM can cite precisely

The pipeline, end to end

Whatever tools you pick, the shape is the same:

Fetch sitemap.xml and list target URLs
Render each page (headless browser if the site uses JavaScript)
Extract main content, dropping boilerplate
Convert to Markdown
Chunk on headings, attach metadata
Embed and upsert into your vector store

Steps 1–4 are the crawling half, and they are the part most teams underestimate: browser automation, rate limiting, retries, and re-crawls on a schedule. If you want to compare the DIY conversion tools first, see how to convert any website to Markdown.

The no-infrastructure option

Crawl4AI (open source, Python, Playwright underneath) covers the crawling half well if you are happy to host it. If you are not, we built Website to Markdown (Crawl4AI) — an Apify actor that runs Crawl4AI as a hosted service. Give it a sitemap URL, a list of URLs, or a start URL to crawl, and it returns one clean Markdown document per page in a dataset you can pull straight into your chunking step via API. JavaScript rendering and boilerplate filtering are handled for you.

Pricing is pay-per-event: $1 per 1,000 pages plus standard Apify platform usage — no subscription. Fair caveats: it requires an Apify account, the pages are processed on Apify's infrastructure, and if your corpus is a few static pages you can do this for free with pandoc.

Actor page: https://apify.com/bikram07/web-to-markdown-crawl4ai

Calling it from Claude or Cursor via MCP

Apify exposes actors through its MCP server at https://mcp.apify.com. Add it to Claude, Cursor, or any MCP client with your Apify API token, and your agent can run the crawl itself: ask it to run bikram07/web-to-markdown-crawl4ai on a docs sitemap, wait for the run to finish, and read the Markdown results back out of the dataset — a website-to-RAG ingest loop with no code on your side.

Frequently Asked Questions

Why is Markdown better than raw HTML for RAG?

Raw HTML buries content in tags, attributes, scripts, and styling that inflate token counts and add noise to embeddings. Markdown preserves the structure that matters — headings, lists, links, code blocks — at a fraction of the size, and its heading hierarchy gives chunkers natural split points.

Should I remove navigation and boilerplate before embedding website content?

Yes. Navigation menus, footers, and cookie banners repeat on every page, so their text dominates similarity search and pushes real content down in retrieval results. Strip them with a content extractor such as Mozilla Readability, trafilatura, or Crawl4AI's content filtering before you chunk and embed.

How should I chunk Markdown for embeddings?

Split on heading boundaries first — for example with LangChain's MarkdownHeaderTextSplitter — so each chunk is a coherent section, then apply a size cap for oversized sections. Keep code blocks intact, and attach the source URL and heading path as metadata so answers can cite where they came from.

Can I run a website-to-Markdown crawl directly from Claude or Cursor?

Yes, via MCP. Apify exposes its actors through an MCP server at mcp.apify.com. Add it to your MCP client with your Apify API token, then ask the agent to run bikram07/web-to-markdown-crawl4ai with your URL or sitemap — the Markdown results land in a dataset the agent can read back.