Let your AI read the pages it can't reach
Reddit blocks datacenter IPs. Skool needs a login. X returns a page with no posts. Your agent isn't bad at reading them — it never receives them.
What a server-side scraper actually gets back
Measured 2026-09-05, fetching each URL with a real browser User-Agent and no session. Text length is after stripping tags and scripts:
| URL | HTTP | Body text | Result |
|---|---|---|---|
| reddit.com/r/LocalLLaMA | 200 | 6 | Empty shell |
| skool.com/games | 200 | 14 | Empty shell |
| x.com/elonmusk | 200 | 1,651 | Login skeleton |
| quora.com | 403 | — | Blocked |
| economist.com | 403 | — | Blocked |
| news.ycombinator.com | 200 | 3,843 | Real content |
HTTP 200 is the trap here: Reddit and Skool both return 200, but the body has no content in it. Anti-bot rules change — re-run the test before quoting these numbers.
Why the browser changes the answer
Web2MD runs as a Chrome extension, so it reads the page after JavaScript has rendered it, inside the session you are already signed into. No datacenter IP to block, no login wall to get past — it sees exactly what you see.
One call, up to 50 pages
Point Claude Code, Cursor, or any MCP-compatible agent at the Agent Bridge. It drives your real browser: opens each URL in a background tab, extracts the content, and returns clean Markdown.
npx web2md-mcp-serverWhere this does not help
The extension has to be installed and Chrome has to be running — this is not a headless server product. Pages you are not signed into stay unreadable, because it uses your session rather than working around one. And for ordinary public pages, a plain server-side fetch works fine and costs you nothing.
FAQ
Is this allowed?
It reads pages in your own browser, in your own logged-in session — the same content you can already see. It does not bypass authentication or share your credentials. You are still bound by each site's terms of service, so do not use it to redistribute content you are not allowed to redistribute.
How is this different from Firecrawl or Jina Reader?
They fetch from a datacenter with no session, which is the right design for public pages and much simpler to run. It just means the sites in the table above return a block page or an empty shell. Jina does get some of them — Quora, for instance — so the honest split is: public pages, use whatever is cheapest; session-gated pages, you need a real browser.
Chrome already has Gemini. Why not just use that?
For reading one page, Gemini is free and unlimited and you should use it. The difference shows up when you need 50 pages in one call, a deterministic parse that does not summarise or reword, or something an agent can invoke over MCP. Gemini is a chat assistant; this is a data pipeline.
How many pages can I convert at once?
Up to 50 URLs per call through the agent_batch_convert MCP tool. Each one opens in a background tab in your own Chrome, so throughput depends on your machine and the sites involved.