DomDistiller
Chrome's Reader Mode engine, whose content-extraction heuristics are a public proxy for how machines separate main content from boilerplate.
If you want to know how search engines and AI systems read your website, look no further than DomDistiller. This is the engine behind Chrome's Reader Mode, and it reveals exactly how Google decides what on a page is real content and what is just boilerplate.
DomDistiller does not look at raw HTML. Instead, it analyzes the fully rendered page, breaking it down into individual text blocks. It then runs these blocks through a scoring system.
Elements with high link density are immediately penalized as menus or footers. Large blocks of continuous text score highly. Semantic tags like article, paragraph, and blockquote are treated as positive signals, while tags like nav, aside, and footer are marked as negative. DomDistiller also uses a blacklist to demote blocks containing class or ID names like "ad," "share," "sidebar," or "promo."
The lesson for creators and technical SEOs is clear. Clean semantic structure and minimal boilerplate are essential. By making your pages easy to parse, you ensure that search engines and AI models can quickly isolate and understand your main content.
DomDistiller is the engine behind Chrome's Reader Mode, and it's a valuable, open proxy for how a sophisticated Google system decides what on a page is real content versus boilerplate. Reverse-engineering it tells you a lot about machine readability.
It works as a multi-stage heuristic pipeline over the rendered DOM (not the raw HTML). It segments the page into text blocks, discards hidden elements, then scores each block: high link density is penalised as navigation or footer; substantial continuous text scores well; semantic tags like <article>, <p> and <blockquote> are positive while <nav>, <aside> and <footer> are negative; and a blacklist of class/ID substrings (comment, ad, share, sidebar, promo, related) further demotes non-content.
The lesson for technical SEO is that clean semantic structure and low boilerplate help machines isolate your main content — the same content that then gets fed to AI systems. It's a close relative of the content fetcher and the structured page representation behind Annotated Page Content.