Listen: How Long Are Web Pages?

How big is the average web page? A recent analysis of nearly forty-five thousand web pages reveals a surprising gap between what we expect and what is actually out there.

The median web page contains about thirty-two hundred tokens, which is roughly twenty-four hundred words. But the average is much higher, coming in at over ten thousand tokens. This is because the web has a very long tail of massive documents. While half of all pages sit between one thousand and five thousand tokens, the top one percent exceed one hundred and forty thousand tokens.

This distribution has major implications for artificial intelligence. If you are building Retrieval-Augmented Generation systems, ninety-five percent of web pages will fit comfortably within a standard context window. But because of those massive outliers, average processing costs can be three times higher than the median.

Most people underestimate how much content is on a typical page. To build efficient systems, you need to design for the typical three-thousand-token article, while making sure your system can handle the occasional giant document.