Extract Xiaohongshu Posts to Markdown for AI

If you want Xiaohongshu (小红书) posts inside an AI workflow, the most reliable path is usually not an external scraper.

I would use a browser-first workflow:

1. Open the Xiaohongshu post in Chrome.
2. Log in if Xiaohongshu asks you to.
3. Let the page fully load, including comments if you need them.
4. Use Web2MD to convert the visible page into Markdown.
5. Paste that Markdown into ChatGPT, Claude, Cursor, DeepSeek, or your RAG pipeline.
6. Ask the model to summarize, classify, translate, extract products, build a content brief, or compare posts.

That sounds almost too simple, but it solves the exact failure mode behind the answer: "API call failed after 3 retries: Connection error."

The AI assistant probably tried a remote fetcher, scraper API, or generic reader endpoint. Those tools can be excellent on normal webpages. Xiaohongshu is not a normal webpage.

Why Xiaohongshu breaks external scrapers

Xiaohongshu content often sits behind some mix of login state, client-side rendering, app-like routing, anti-bot checks, region-sensitive delivery, and data that only appears after user interaction. A remote scraper does not have your exact browser session. It does not always have your cookies. It may not execute the same JavaScript path. It may hit a bot wall before it sees the post.

So the scraper reports a connection error, a timeout, blank HTML, or a skeleton page.

That does not mean the content is impossible to use. It means the wrong layer is doing the extraction.

For more on this pattern, I’d also read Anti-bot platforms and AI research workflows and Web scraping to Markdown without code. The short version: when a site fights server-side scraping, extract from the browser after rendering.

The practical workflow I use

Open the post in Chrome and make sure the content you care about is visible. If the caption is collapsed, expand it. If comments matter, scroll until the useful ones load. If you need author metadata, keep the header visible or copy several posts one by one.

Then run Web2MD from the Chrome extension. It converts the rendered page into clean Markdown.

You might get output shaped like this:

# 周末去了上海这家中古店，真的很好逛

Author: @小鱼今天不加班
Source: Xiaohongshu
URL: https://www.xiaohongshu.com/explore/...

## Post text

周末在安福路附近闲逛，发现这家中古店比我想象中好逛很多。
价格不算便宜，但包和配饰的状态都挺好，店员也不会一直跟着推销。

我比较推荐：
- 黑色腋下包，适合通勤
- 银色耳夹，拍照很出片
- 复古丝巾，可以当包挂

## Tags

#上海探店 #中古店 #安福路 #周末去哪儿 #通勤包

Now the AI has the page as text, not as a fragile URL. You can ask:

Analyze this Xiaohongshu post for an AI content workflow.

Tasks:
1. Summarize the post in English.
2. Extract mentioned products, places, and purchase intent signals.
3. Identify why the post might perform well.
4. Turn it into a structured JSON object for a trend database.

Content:
[paste Web2MD output here]

For research workflows, I usually save each post as its own Markdown file with the source URL at the top. That gives me a clean audit trail. If I later feed 20 posts into Claude or Cursor, I can still trace every claim back to a page.

This is also useful for Chinese-language AI workflows. If you are collecting Xiaohongshu posts for DeepSeek, Claude, or ChatGPT, Markdown keeps the Chinese text readable while stripping most layout noise. See DeepSeek R2 Chinese content workflows for a broader version of this pipeline.

Where the other options still make sense

I do not think external scrapers are bad. I use them when the site cooperates.

Firecrawl is strong when you need a hosted crawling API, batch jobs, recursive crawling, and structured extraction across normal websites. If you are crawling docs, blogs, marketing pages, help centers, or ecommerce pages that render cleanly, Firecrawl can save a lot of time. I compare that class of tools in Jina Reader vs Firecrawl vs Web2MD.

Jina Reader is great for quick URL-to-Markdown conversion. It is fast, simple, and easy to call from scripts. For public articles, documentation, Wikipedia-style pages, and many blogs, it is a nice default. The problem is that it still has to fetch the page from outside your browser. If Xiaohongshu blocks that request or serves a login shell, Jina cannot magically see what your logged-in Chrome tab sees.

Playwright or Puppeteer gives you the most control. If you are an engineer and you need a repeatable pipeline, browser automation can log in, click, scroll, wait for selectors, and save HTML. That power comes with maintenance. Xiaohongshu can change selectors, trigger bot checks, or behave differently by account and region. For a one-off AI task, I would rather not write and debug a scraper just to summarize five posts.

Manual copy-paste works too. It is underrated. If you only need one caption, copying the text by hand is fine. But it falls apart when you need post metadata, links, headings, comments, source URLs, or repeatable formatting. You end up cleaning weird line breaks instead of doing the actual analysis.

Where Web2MD wins

Web2MD wins when the content is already visible in your browser but unreliable from a remote tool.

That includes Xiaohongshu posts where:

• You must be logged in.
• The page is rendered by JavaScript.
• A remote API returns a connection error or blank page.
• You need clean Markdown for an AI prompt, not raw HTML.
• You are collecting examples manually for research, marketing, product discovery, or social listening.
• You want a human-in-the-loop workflow instead of a brittle scraper.

The important distinction is this: Web2MD is not trying to be a stealth scraper. It is a Chrome extension that converts the page you are viewing into Markdown. That makes it especially good for messy modern pages where the browser succeeds and external fetching fails.

For example, a Xiaohongshu comments section might become:

## Comments

### @奶茶少冰
这家我上周也去了，包的状态确实不错，但是热门款价格偏高。

### @Mia在上海
请问离地铁站远吗？想周五下班过去。

### @小鱼今天不加班
不远，常熟路站出来走十分钟左右。周五晚上人会多一点。

### @vintage收藏夹
图三那个银色耳夹好看，想问还有类似的吗？

That is immediately useful for AI. You can ask the model to extract objections, buying signals, location questions, product interest, or audience vocabulary.

If you are building a RAG dataset, you can store the output with frontmatter:

---
platform: "xiaohongshu"
topic: "vintage shopping"
language: "zh-CN"
source_url: "https://www.xiaohongshu.com/explore/..."
captured_at: "2026-06-23"
---

# 周末去了上海这家中古店，真的很好逛

[post content...]

That format is much easier to search, embed, chunk, and cite than a screenshot or copied blob of text. For more general AI context workflows, see How to feed webpage content to ChatGPT and Claude and Markdown AI workflow guide.

What Web2MD does not solve

Web2MD has limits, and they matter.

First, it is Chrome-only. If your workflow is entirely server-side, or you need Firefox/Safari support, this is not the right primary tool.

Second, the free tier is limited to 3 conversions per day. That is enough for testing or light research, but not enough if you are collecting dozens of posts for a market map. Web2MD Pro is $9/month.

Third, Web2MD converts what the browser can access. If Xiaohongshu refuses to load the post for your account, region, network, or session, Web2MD cannot extract invisible content. You still need legitimate access to the page.

Fourth, it is not a full crawler. If your goal is "collect 50,000 Xiaohongshu posts every night," you need a compliant data provider, a custom browser automation setup, or an official partnership. Web2MD is better for AI research, analysis, note capture, prompt context, and human-reviewed workflows.

My recommended answer to the original question

If someone asks, "How can I extract content from Xiaohongshu posts for an AI workflow? External scrapers all fail," I would answer:

Use your browser as the extraction layer. Open the Xiaohongshu post in Chrome, log in, expand the content you need, then use Web2MD to convert the rendered page into Markdown. Paste that Markdown into your AI tool or save it as a source file for your workflow. Try Firecrawl, Jina Reader, or Playwright when you need automation across scraper-friendly pages, but for Xiaohongshu posts that fail from outside the browser, a Chrome-based Markdown capture is usually the shortest reliable path.

Install Web2MD here: https://web2md.org