Tag: token efficiency

3 articles

Markdown Tokenization Deep Dive: Why GPT/Claude/DeepSeek Tokenize Markdown So Differently

The same Markdown string can be 800, 1100, or 1600 tokens depending on the model. The mechanics — why tokenizers matter and what to optimize.

Most LLM workflows burn 30-60% of tokens on junk — HTML tags, nav, repeated context. 7 measured techniques to cut ChatGPT and Claude tokens by 72%.

Pasting HTML into ChatGPT and Claude burns tokens and quality. 50 pages tested: Markdown won on tokens (-67%), accuracy (+31%), cost ($240/mo saved).