Why can't Claude Code or Cursor read Reddit URLs directly?

Their fetch runs server-side from datacenter IPs. Reddit blocks those at the network layer (Cloudflare + robots.txt) and renders comments client-side, so even when a request gets through, the fetcher sees an empty shell instead of the thread. The same applies to Firecrawl, Jina Reader, and Parallel — they all fetch from their own servers, not your session.

How do I give my agent Reddit access?

Install the Web2MD MCP server with `npx web2md-mcp-server-install`, then fully quit and reopen Chrome. Your agent gets `agent_convert` for a single URL and `agent_batch_convert` for up to 50 URLs. Both drive your real Chrome through a local native-messaging host, so Reddit sees a normal logged-in browser.

How many Reddit threads can an agent read in one call?

Up to 50 URLs per `agent_batch_convert` call. They are processed sequentially through your browser, and each one comes back as clean Markdown with the post body and comment tree. For a research pass over a subreddit, that is usually one call.

Does this work on subreddit listing pages or only individual threads?

Both work, but they return different things. An individual thread gives you the post plus the comment tree, which is what you usually want for research. A subreddit listing page gives you titles and navigation — useful for discovering threads to then convert, less useful as analysis input.

Do I need a Reddit account or API key?

No API key. You need Chrome with the Web2MD extension, and whatever Reddit access you already have in that browser. If you are logged in, your agent sees what you see — including subreddits that require an account. That is the whole point of routing through your session instead of a server.

Is this against Reddit's terms of service?

Reading Reddit in your own authenticated browser is what Reddit's terms allow. Converting an already-loaded page to Markdown for personal research sits in the same space as Reader Mode or saving a page. Large-scale automated scraping is a different activity with different rules — this is not that.

How is this different from just using Reddit's .json endpoint?

Appending .json to a thread URL works well and needs no tooling — if you are comfortable writing the parsing yourself and only need public threads. The MCP route matters when you want the agent to do it unattended across many URLs, want the same path to work on sites that have no JSON endpoint, or need pages behind a login.

What happens if one URL in a batch fails?

That item returns an error and the rest of the batch continues. You get results for the URLs that worked plus an error entry for the one that didn't, so a single dead thread does not cost you the whole run.

Give Your AI Agent Reddit Access: Batch-Read Threads via MCP

Ask Claude Code to summarize the top complaints in an r/LocalLLaMA thread and paste the URL. It fails. Ask Cursor to pull the arguments from a Reddit discussion so you can cite them. Also fails.

This is not a prompt problem. There is no phrasing that fixes it.

Short answer

Server-side fetching cannot read Reddit — not from Claude Code, not from Cursor, not from Firecrawl, Jina, or Parallel. They all request from their own datacenter IPs, which Reddit blocks, and Reddit renders comments client-side so even a successful request returns an empty shell. The fix is to route the read through the browser you are already logged into, which an agent can do over MCP: up to 50 threads per call.

Why every server-side crawler fails here

The failure has two independent causes, and fixing either one alone is not enough.

The first is network-level blocking. Reddit updated its robots.txt and API access rules in 2023 to block AI crawlers, and Cloudflare bot detection sits in front of the site. Requests from known datacenter IP ranges get challenged or refused. This is why "just use a different crawler" does not help — Firecrawl, Jina Reader, and Parallel all fetch from their own infrastructure, so they all hit the same wall.

The second is client-side rendering. Reddit is a React application. The comment tree is fetched and rendered after the initial page load. A server-side fetcher that does get a response often receives the page shell without the content it was after. That produces the worst failure mode: the agent answers confidently, but describes a title and some navigation rather than the discussion.

Your browser has neither problem. Your cookies are valid, your TLS fingerprint is a real one, the Cloudflare challenge already passed, and JavaScript ran to completion. The page you are looking at is the finished extraction. The only question is how an agent reads it.

The setup

One command registers a local native-messaging host that lets MCP clients talk to the Web2MD extension:

npx web2md-mcp-server-install

Then fully quit Chrome — Cmd+Q on macOS, system tray on Windows — and reopen it. Chrome only picks up new native-messaging hosts on a cold start, so a window close is not enough.

After that your agent has these tools:

| Tool | What it does | |---|---| | agent_convert | One URL → Markdown, through your Chrome | | agent_batch_convert | Up to 50 URLs → Markdown for each, sequential | | semantic_search | Search across what you have already converted |

What a research pass looks like

The useful pattern is two calls, not one.

First, convert the subreddit listing to find candidate threads. That returns titles and links rather than discussion, which is exactly what you want at this stage. Then hand the thread URLs you care about to agent_batch_convert in a single call.

A prompt that works:

Read these Reddit threads and tell me which complaints about
Obsidian sync come up more than once. Quote the actual comments.

https://www.reddit.com/r/ObsidianMD/comments/.../
https://www.reddit.com/r/ObsidianMD/comments/.../
https://www.reddit.com/r/ObsidianMD/comments/.../

The agent calls agent_batch_convert once, gets Markdown for all three, and reasons over the real comment text. The difference from pasting URLs is not subtle: instead of a plausible summary assembled from general knowledge about the subreddit, you get quotes you can check.

Each result arrives with frontmatter identifying the source:

---
title: "LocalLlama"
source: https://www.reddit.com/r/LocalLLaMA/
date: 2026-09-03T16:26:31.071Z
---

That matters more than it looks. When an agent processes 20 threads, the frontmatter is what lets it attribute a claim back to a specific thread rather than blending everything into one undifferentiated pile.

Where this fits against the alternatives

Reddit publishes a JSON version of every public thread — append .json to any thread URL and you get the full structure. If you are writing a script, only need public threads, and do not mind handling the parsing, that is the simplest path and it costs nothing.

The MCP route earns its setup in three cases. When you want the agent to work unattended across many URLs without you assembling the JSON calls. When you want one path that also works on sites with no JSON endpoint — X, LinkedIn, Quora, paid Substack all fail the same way Reddit does, for the same reason. And when the content needs a login, where a server-side fetch has no way in at all.

For large-scale crawling of public pages, a server-side crawler is still the better tool. Firecrawl can hit thousands of URLs in parallel; a browser opens tabs sequentially. These are different jobs. The browser route wins specifically where authentication and anti-bot defenses are the obstacle, which happens to describe most of the content people actually want to feed an agent.

Honest limits

Sequential processing means 50 URLs takes real time — this is a browser opening pages, not a fleet of workers.

Chrome has to be running. If your agent runs on a server with no browser, this approach does not apply.

Subreddit listing pages return navigation-heavy output. Use them for discovery, then convert the individual threads.

And a deleted or private thread returns an error for that item. The batch continues, but you should check for error entries rather than assuming every URL produced content.

Why Claude can't read Reddit — the failure explained in more detail
Reddit to Markdown — the one-click version, no MCP setup
Chrome MCP webpage to Markdown — the same tooling for general pages
Let your agent read the sites that block crawlers — measured coverage across Reddit, HN, Lobsters, and more

Give Your AI Agent Reddit Access: Batch-Read Threads via MCP