Why can't my AI agent read Reddit?

Because the fetch happens on a server. Claude Code, Cursor, Firecrawl, Jina Reader, and Parallel all request from their own datacenter IPs, and Reddit blocks those ranges at the network layer. On top of that Reddit renders comments client-side, so even a request that gets through returns a page shell without the discussion. Switching crawler vendors does not help — they share the same architecture.

How do I give an agent access to sites that block crawlers?

Route the read through the browser you already use. Install the Web2MD MCP server with `npx web2md-mcp-server-install`, fully quit and reopen Chrome, and your agent gets `agent_convert` (one URL) and `agent_batch_convert` (up to 50 URLs). Requests then come from your real session, which is what the anti-bot layer is checking for.

How long does a batch of URLs take?

Measured on five subreddit pages: all five returned complete content in 31 seconds, roughly 6 seconds each. Processing is sequential because it is a real browser opening real tabs, so plan on that rate rather than the parallel throughput a server-side crawler gives you.

Which sites actually work?

Measured directly: Reddit thread and listing pages return complete content (4,000-6,000 characters), Hacker News returns the full front page (42,000+ characters), Lobsters and Product Hunt both return complete pages. Stack Overflow's question index and GitHub trending failed in the same test — extraction error and tab timeout respectively. It varies by site, so test the ones you care about rather than assuming blanket coverage.

Do I need API keys or a paid account on those sites?

No API keys. You need Chrome with the Web2MD extension installed, plus whatever access you already have in that browser. If you are logged in somewhere, your agent sees what you see. That is the entire mechanism — there is no credential sharing and no server-side proxy.

How is this different from Firecrawl or Jina Reader?

Different architecture, different strengths. Firecrawl crawls thousands of public URLs in parallel from its own infrastructure, which browser-based extraction cannot match. But that same infrastructure is what gets blocked on Reddit and anything behind a login. Use a server-side crawler for breadth over public pages; use the browser route where authentication or anti-bot defenses are the obstacle.

Is reading these sites through my browser against their terms?

Reading a page in your own authenticated browser is what these sites' terms allow — it is what the browser is for. Converting an already-loaded page to Markdown for personal research sits in the same space as Reader Mode or saving a page locally. High-volume automated scraping is a different activity governed by different rules, and this is not that.

What happens when one URL in a batch fails?

That item comes back as an error entry and the rest of the batch continues. You get results for everything that worked, so one dead thread or a rate-limited page does not cost you the whole run. Check for error entries rather than assuming every URL produced content.

Does this work if my agent runs on a server?

No. This approach needs a real Chrome running on a machine where you are signed in. If your agent runs headless on a server with no browser, a server-side crawler is the right tool — accepting that it will not reach Reddit or login-walled content.

Let Your Agent Read the Sites That Block Crawlers: Reddit, HN, and More

There is a category of website that AI agents simply cannot read, and it happens to contain most of the content worth reading.

Ask Claude Code to summarize the arguments in an r/LocalLLaMA thread. Ask Cursor to pull the top Hacker News discussion so you can cite it. Point any agent at a Reddit URL and ask what people actually said. It fails, and no rephrasing fixes it.

Short answer

Every server-side crawler fetches from its own datacenter IPs, and that is precisely what these sites block. Firecrawl, Jina Reader, Parallel, and the built-in fetch in Claude Code and Cursor all share this architecture, so switching vendors changes nothing. The fix is architectural: route the read through the browser you are already signed into. An agent can drive that over MCP, up to 50 URLs per call.

The two reasons a server-side fetch fails

Understanding both matters, because fixing only one leaves you stuck.

Network-level blocking. Reddit revised its robots.txt and API rules in 2023 to shut out AI crawlers, and Cloudflare bot detection sits in front of the site. Requests from known datacenter ranges get challenged or refused outright. This is why "use a different crawler" is not a solution — Firecrawl, Jina, and Parallel all fetch from their own infrastructure, so they all arrive at the same closed door.

Client-side rendering. Reddit is a React application. Comments load after the initial page response. A fetcher that does get a 200 often receives the shell without the content, which produces the most damaging failure mode: the agent answers confidently while describing navigation links and a title rather than the discussion. You get a plausible summary of nothing.

Your browser has neither problem. Cookies valid, TLS fingerprint genuine, Cloudflare challenge already passed, JavaScript run to completion. The page in front of you is the finished extraction. The only remaining question is how an agent reads it.

Setup

npx web2md-mcp-server-install

Then fully quit Chrome — Cmd+Q on macOS, system tray on Windows — and reopen. Chrome registers new native-messaging hosts only on a cold start; closing the window is not enough.

Your agent then has:

| Tool | What it does | |---|---| | agent_convert | One URL → Markdown, through your Chrome | | agent_batch_convert | Up to 50 URLs → Markdown each, sequential | | semantic_search | Search across what you have already converted |

What the numbers actually look like

I ran five subreddit pages through agent_batch_convert in a single call:

ok   4332 chars  r/LocalLLaMA/top/
ok   5906 chars  r/MachineLearning/top/
ok   4761 chars  r/ObsidianMD/top/
ok   3817 chars  r/ChatGPT/top/
ok   4138 chars  r/ClaudeAI/top/

5/5 complete — 31 seconds

Roughly 6 seconds per URL. That is the honest rate: a real browser opening real tabs one after another. A server-side crawler doing public pages in parallel is far faster, and if that is your workload you should use one. The browser route is not competing on throughput — it is competing on reach.

Each result carries frontmatter identifying its source:

---
title: "LocalLlama"
source: https://www.reddit.com/r/LocalLLaMA/
date: 2026-09-03T16:26:31.071Z
---

That detail matters more than it looks. When an agent digests twenty threads, the frontmatter is what lets it attribute a specific claim back to a specific thread instead of blending everything into one undifferentiated summary.

Measured coverage

Same test, run directly against the tools:

| Site | Result | Content | |---|---|---| | Reddit — thread and listing pages | Complete | 4,000-6,000 chars | | Hacker News front page | Complete | 42,264 chars | | Lobsters | Complete | 12,238 chars | | Product Hunt | Complete | 4,805 chars | | Stack Overflow question index | Failed | Extraction error | | GitHub trending | Failed | Tab load timeout |

Two of six failed, and I am listing them because a coverage claim you cannot verify is worth nothing. Sites differ in how they render and how aggressively they gate. Test the ones you actually care about before building a workflow on top of them.

A research pass that works

The useful pattern is two steps rather than one.

Convert the listing page first to discover threads. That returns titles and links — exactly what you want at this stage, and not useful as analysis input. Then hand the specific thread URLs to agent_batch_convert in one call.

Read these Reddit threads and tell me which complaints about
local model quantization come up more than once. Quote the comments.

https://www.reddit.com/r/LocalLLaMA/comments/.../
https://www.reddit.com/r/LocalLLaMA/comments/.../
https://www.reddit.com/r/LocalLLaMA/comments/.../

The agent makes one agent_batch_convert call, receives Markdown for all three, and reasons over real comment text. The difference from pasting URLs is not subtle. Instead of a summary assembled from general knowledge about the subreddit, you get quotes you can check against the source.

Where this does not apply

Your agent runs headless on a server. No browser, no session, no access. Use a server-side crawler and accept its reach limits.

You need thousands of public pages. Sequential tab-opening is the wrong tool. Firecrawl exists for this and is better at it.

The site defeats extraction anyway. Two of six sites failed in my test. Verify before you depend on it.

Very large batches take real time. Fifty URLs at ~6 seconds each is about five minutes. Fine unattended, painful if you are watching.

Give your agent Reddit access via MCP — the Reddit-specific walkthrough
Why Claude can't read Reddit — the failure explained in depth
Reddit to Markdown — one-click version, no MCP setup
Hacker News thread to Markdown — the HN-specific workflow

Let Your Agent Read the Sites That Block Crawlers: Reddit, HN, and More