Engadget: Quantitative Linguistic Analysis

Abstract

This paper profiles Engadget's article corpus using nothing but Shannon entropy and data compression — no language model, no parser, no notion of words or topics. It opens with a data-quality result that the method delivers for free: the raw scrape of 13,147 pages is two populations, not one, and compression alone tells them apart. Roughly 60% of the pages are product-hub templates rather than articles, and removing them leaves 5,214 genuine articles spanning 2006–2023. On that cleaned corpus the measurements record a distinctive editorial arc — article length climbing to a mid-2010s long-form peak and then retreating, a compression signal that largely (but not entirely) shadows that length, and a symbol distribution that drifts only slightly across eighteen years. The figures are laid out plainly and in aggregate; the causes are left to the reader.

1. What Is Measured

Six quantities are extracted per article. Five are stored directly; the sixth is derived across the corpus.

1. Order-0 entropy (\(H_0\)) — the Shannon entropy of the symbol-frequency distribution, in bits per symbol. Higher means a flatter, more even spread of characters; lower means a more concentrated one.
2. Order-1 entropy rate (\(H_1\)) — the conditional entropy of a symbol given the one before it: $$H_1 = H(2) - H(1).$$ Higher means adjacent-symbol pairings are less predictable.
3. Redundancy (\(R\)) — the share of the symbol alphabet's capacity that is predictable rather than informative: $$R = 1 - \frac{H_0}{\log_2 D},$$ where \(D\) is the count of distinct symbols. \(R = 0\) is maximal randomness; values toward 1 indicate a skewed, repetitive distribution.
4. Compressed bits/byte, raw — gzip over the UTF-8 bytes, expressed as $$\text{bpb} = 8 \times \frac{\lvert \text{gzip}(x) \rvert}{\lvert x \rvert}.$$ The harder a document is to compress, the more bits per byte it carries.
5. Length-adjusted bits/byte (\(\text{bpb}_{\text{adj}}\)) — the raw figure minus what the corpus's own length-to-compressibility curve predicts for a document of that size, isolating structure from mere shortness. Zero means "exactly as compressible as its length predicts."
6. Article length — character count per article.

The engine has no linguistic knowledge; "vocabulary" and "phrasing" below are shorthand for these symbol-level statistics, not claims about words or grammar.

One caveat specific to this corpus. Engadget's scraped text retains a layer of site furniture — navigation strings, "Sponsored Links", byline and tag fragments, privacy-policy boilerplate. This furniture is fragmentary and compresses poorly, so Engadget's absolute bits/byte sit higher than a cleaner scrape would show. Absolute levels are therefore not comparable to other publishers; Engadget's own trends over time, measured against a roughly constant furniture overhead, are.

2. The Corpus Is Two Corpora

Before any editorial reading, the method flags a problem. Of 13,147 scraped pages, 7,933 (60%) are /products/ hub pages, not articles — product-database entries whose captured text is the site's generic news rail, identical from page to page, uniformly ~6,700 characters, and undated. The remaining 5,214 are genuine articles, essentially all dated.

Compression separates the two with almost no overlap:

A single threshold at 3 bits/byte assigns 99.7% of articles to one side and 93% of template pages to the other — the compressor tells editorial prose from boilerplate without reading a word. The cost of ignoring the split is large: on the raw scrape, a length-normalized anomaly scan flags 48% of pages; on the cleaned corpus it flags 1.9%. The near-half "anomaly rate" was simply the two populations sitting on top of each other.

Everything below is computed on the cleaned corpus of 5,214 articles (/products/ pages excluded by URL — a metadata filter, consistent with the no-language method).

3. The Clean Corpus at a Glance

The distribution of order-0 entropy for Engadget articles forms a single, right-skewed peak with a median of 4.523.

The order-0 entropy distribution is a single, mildly right-skewed hump — the telltale second peak that appears on the raw scrape belongs to the template pages, and vanishes once they are removed.

4. Chronological Trajectories

The per-year medians across eighteen years:

4.1 Length: a peak, not a climb

Line graph of Engadget article length from 2006 to 2023 shows median length peaking in the mid-2010s.

Engadget's length does not grow monotonically. From short posts in the mid-2000s (median 1,614 characters in 2006), it climbs to an early plateau around 2011 (8,000), dips sharply in 2014 (3,400, the year with the most articles in the corpus at 906), leaps to a long-form peak across 2015–2017 (~10,000–10,500), and then retreats through the early 2020s to ~4,200–4,900. The overall 2006-to-2023 change of +201% understates a fuller shape: Engadget lengthened into a mid-decade long-form era and then pulled back toward shorter pieces.

The spread is wide throughout — in 2017 the middle half of articles ran from 6,959 to 13,851 characters — and the article count itself swings (906 in 2014, 45 in 2022), so length and volume move together in places worth noting rather than smoothing over.

4.2 Compressibility and the length confound

Three line graphs show raw compressibility mirroring median article length over time, with a declining length-adjusted trend.

Raw bits/byte moves as a rough mirror of length, falling from ~4.5 in the short mid-2000s to ~3.6–3.7 once articles lengthen (−18% end to end). The link is strong:

1. Per-article rank correlation of length and bits/byte: Spearman \(\rho = -0.913\).
2. Per-article linear correlation: \(r = -0.692\) (\(r^2 = 0.48\)).
3. Year-median correlation: \(r = -0.835\) (\(r^2 = 0.70\)).

But length does not account for all of it. When the length effect is divided out, the residual \(\text{bpb}_{\text{adj}}\) is not flat: it runs from +0.13 in 2006 to −0.18 in 2023. Early articles were slightly less compressible than their length predicted; recent ones — 2023 most sharply — are more compressible than their length predicts. So Engadget's compression trend is mostly length, with a real length- independent drift on top. (That drift could reflect changing prose, changing amounts of site furniture across redesigns, or both; the method cannot separate them.)

4.3 Entropy and redundancy

A dual-axis line chart plotting Engadget article entropy from 2006 to 2023 reveals generally opposing chronological trends.

Order-0 entropy is broadly stable — a shallow decline from 4.608 to 4.512 (−2.1%) across eighteen years. Order-1 entropy rises, 3.276 to 3.386 (+3.4%), reaching a plateau near 3.46 in the mid-2010s before easing at the end. As at other publishers, the character distribution holds roughly steady while adjacent-symbol variety edges up.

Median redundancy of Engadget articles rises from 2006, peaks in the mid-2010s, and slightly declines by 2023.

Redundancy climbs from 0.225 to 0.262 (+16.5%), most of the rise complete by the early 2010s, then a plateau near 0.27.

4.4 A formative early era

Several of the largest year-to-year moves land in 2006–2009, when the archive is thin and articles are short: order-0 entropy, the entropy rate, redundancy, and raw bits/byte all swing hard across 2007–2009 (the 2008-to-2009 redundancy jump of +20% is the single sharpest move in the corpus). These are the statistics of a young site finding its format, and they settle markedly once article volume grows after 2010.

5. The Static Landscape

Three-dimensional scatter plot mapping article length against gzip bits per byte and entropy, showing shorter articles are less compressible.

Placed by entropy, compression and length, the cleaned articles form one continuous manifold rather than the split blob of the raw scrape. Length-normalized anomaly scoring — flagging articles that compress very differently from what their length predicts — now flags only 1.9%:

1. high-info, 34 articles (0.7%) — median length just 310 characters, ~5.6 bits/byte: tiny, fragmentary stubs that resist compression.
2. low-info, 64 articles (1.2%) — median length ~7,400, ~3.2 bits/byte: long and unusually repetitive.
3. normal, 5,116 (98.1%) — the editorial core.

Scatter plot of article compressibility versus length showing that longer articles are more compressible along a downward curve.

Against length, the green curve is the expected bits/byte for each size; articles above it resist compression more than their length predicts, those below are more repetitive. The steep left edge — short articles compressing poorly — is exactly the confound Section 4.2 divides out.

Scatter plot comparing order-0 entropy and gzip bits per byte for Engadget articles, showing a positive correlation.

6. Year-over-Year Inflection Points

Applying a robust z-score to the change in each yearly median (flagging \(\lvert z \rvert \ge 2\)) locates where the corpus turned:

Two things stand out. The turbulence is front-loaded in the formative years (2006–2009). And the clearest late events are structural rather than about symbols: the 2014→2015 tripling of length (the entry into the long-form era) and the 2022→2023 drop in length-adjusted compressibility.

7. What the Numbers Do and Do Not Say

On the cleaned corpus, the measurements establish that between 2006 and 2023 Engadget's articles:

1. rose to a mid-2010s long-form peak (~10,000 characters) and then retreated toward shorter pieces;
2. grew ~18% more compressible in raw bits/byte, mostly — but not entirely — explained by that length movement (year-median \(r = -0.835\));
3. carried a length-independent residual drift, from denser-than-expected early to more-compressible-than-expected by 2023;
4. held order-0 entropy nearly flat while redundancy and the order-1 rate edged up and then plateaued; and
5. were most volatile in the site's early years, settling after 2010.

What the numbers do not do is name a cause. A corpus that lengthens, peaks, and retreats — with a small residual drift in compressibility — is equally consistent with shifting formats (long features versus short news), template and redesign changes altering the site furniture the scrape captures, changes in staffing or house style, or a drift in topic mix. This method cannot separate those, because it never reads a word; it can only say, precisely, what moved and when. And, as Section 2 showed, it can say with confidence which pages were never articles at all.

8. Conclusion

Measured as symbols and bytes, Engadget's story has two layers. The first is a data-quality finding the method surfaces on its own: most of the scraped corpus was template boilerplate, cleanly separable from articles by compression alone. The second, on the genuine 5,214 articles, is an editorial arc distinct from the steady lengthening seen elsewhere — a rise into a mid-2010s long-form peak followed by a retreat, a compression signal that largely tracks that length while keeping a faint drift of its own, and a symbol distribution that barely moves across eighteen years.

Method: Shannon order-0 and order-1 entropy over Unicode code points, plus gzip compression over UTF-8 bytes, computed per article across the full recovered corpus. /products/ hub pages excluded by URL as non-article boilerplate (13,147 scraped → 5,214 articles; 5,207 dated). No natural-language processing is used at any stage. Length adjustment follows the corpus's own rolling-median length-to-compressibility curve. Dates recovered from URL and byline tokens by pattern match only. Absolute bits/byte reflect retained site furniture and are not comparable across publishers.

Information landscape (3D)
Three-dimensional scatter plot maps article entropy and compression rates, showing a dense central cluster alongside highly anomalous outliers. Each point is an article. x = order-0 entropy, y = gzip bits/byte, z = length. Color = anomaly score. · scatter shows a 6,000-point sample.
Compressibility vs length
The green curve is the expected bits/byte for each length. Points above it are more incompressible than their length predicts (fragmentary); points below are more repetitive. This is the baseline the length-normalized score measures against.

Entropy vs compression
Scatter plot mapping gzip bits per byte against entropy, showing article data clustered by information level flags.

Order-0 entropy
Histogram displays the bimodal distribution of order-zero entropy values across the analyzed text corpus.

1. Articles: 13,147
2. Median entropy: 4.808
3. Median bits/byte: 2.352
4. Low-info flags: 3,188
5. High-info flags: 3,271