Why convert websites to Markdown?
Markdown has quietly become the interchange format for AI workflows. LLMs read it well, it survives copy-paste, and it strips away the markup noise of modern web pages while keeping the structure — headings, lists, links, code blocks — that actually carries meaning. Common reasons to convert a site to Markdown:
- LLM context: pasting a docs page into Claude or ChatGPT as Markdown uses far fewer tokens than raw HTML
- RAG pipelines: Markdown chunks cleanly on heading boundaries (see our guide on preparing website content for RAG)
- Docs migration: moving legacy HTML documentation into a Markdown-based system like Docusaurus, MkDocs, or a Git repo
- Archiving: keeping a plain-text, diffable copy of content you care about
There are three broad approaches: command-line converters, JavaScript libraries, and crawlers with a real browser behind them. Each has a clear breaking point.
Method 1: pandoc (single static pages)
pandoc is the classic universal document converter. Pipe HTML in, get Markdown out:
curl -s https://example.com/docs/page | pandoc -f html -t gfm -o page.md
The -t gfm flag targets GitHub-Flavored Markdown, which handles tables and fenced code blocks better than pandoc's default Markdown dialect.
Where it works: server-rendered pages — blogs, classic documentation, anything where View Source shows the actual content.
Where it breaks: pandoc converts the entire raw HTML, so navigation menus, footers, and cookie banners all end up in your Markdown. And it never executes JavaScript, so client-rendered pages come out nearly empty.
Method 2: Turndown (HTML to Markdown in JavaScript)
Turndown is the standard JavaScript library for HTML-to-Markdown conversion. It runs in Node or the browser:
import TurndownService from 'turndown'
const res = await fetch('https://example.com/docs/page')
const html = await res.text()
const markdown = new TurndownService({ headingStyle: 'atx' }).turndown(html)
Where it works: programmatic conversion inside your own scripts, with full control over custom rules (for example, how to treat images or tables).
Where it breaks: the same two places as pandoc. A plain fetch only sees the initial HTML — no JavaScript rendering — and Turndown faithfully converts boilerplate along with content unless you strip it first. Pairing Turndown with Playwright or Puppeteer fixes the rendering problem, but now you are maintaining browser automation.
Method 3: copy-paste and browser tools
For a single page you read once, the pragmatic option is a browser extension or an online converter: copy the rendered page, convert, done. This inherits the browser's JavaScript rendering for free, which is why it often works where curl-based tools fail. The trade-off is that it is entirely manual — fine for one page, unusable for fifty. You can sanity-check any converted output with our Markdown Previewer.
Where all manual methods break down
- JavaScript-rendered sites: React, Vue, and Svelte apps ship an empty HTML shell and render content client-side. curl + pandoc/Turndown sees the shell, not the content.
- Boilerplate: nav bars, footers, sidebars, and consent banners pollute every converted page and have to be removed somehow.
- Scale: converting 500 documentation pages means crawling, rate limiting, retries, and deduplication — a real engineering task, not a one-liner.
The hosted option: a crawler that outputs Markdown
Crawl4AI is a popular open-source Python crawler built specifically for LLM data preparation. It renders pages in a real browser (Playwright underneath) and emits Markdown with optional content filtering to drop boilerplate. If you are comfortable running Python and Playwright, self-hosting it is a solid choice.
If you would rather not run any infrastructure, we built Website to Markdown (Crawl4AI) — an Apify actor that wraps Crawl4AI as a hosted service:
- Paste a single URL, a list of URLs, or a sitemap URL — or let it crawl a whole site
- JavaScript rendering included; no browser setup on your side
- One clean Markdown document per page, downloadable as a dataset (JSON, CSV, or files)
- Pricing: $1 per 1,000 pages converted, plus standard Apify platform usage — pay-per-event, no subscription
Honest trade-offs: it is a paid hosted service and needs an Apify account; for a single static page, pandoc is free and instant; and your URLs are processed on Apify's infrastructure, so do not use it for pages you are not allowed to send to a third party.
Actor page: https://apify.com/bikram07/web-to-markdown-crawl4ai
Which method should you use?
- One static page: curl + pandoc. Free, instant, offline.
- Conversion inside your own code: Turndown, with Playwright if pages need JavaScript.
- One JS-heavy page, occasionally: a browser extension or online converter.
- Many pages, JS rendering, or a whole site: Crawl4AI self-hosted if you want control, or the hosted actor if you want zero setup.