Listen: Clustering

Clustering is the art of grouping items by similarity without using pre-defined labels. When you run clustering over vector embeddings, you can turn a massive pile of pages, search queries, or entities into organized groups that nobody had to define in advance. This is how we build topic maps, detect near-duplicates, and group search queries.

To group things, every method needs a way to measure distance. For text embeddings, that is usually cosine similarity, which looks at the direction of the vectors rather than their size.

There are a few ways to actually build the clusters. K-means partitions items into a set number of groups, shifting central points until they settle. Hierarchical clustering merges the closest pairs step-by-step, creating a tree that you can cut at any level. Then there are density-based methods, like DBSCAN. These find clusters of any shape and leave sparse points unassigned, which is incredibly useful for real-world data where some pages simply do not belong anywhere.

Because there is no absolute correct answer, we judge the quality of the clusters internally, often using something called a silhouette score. This score compares how close an item is to its own group versus the nearest neighboring group.

Ultimately, the results depend on the embeddings themselves. Clustering can only find patterns that the embedding space already encodes. This all works because of the manifold hypothesis. This is the idea that high-dimensional real-world data actually sits on a much simpler, low-dimensional surface, where clusters are just the dense neighborhoods.