Listen: Better Vector Clustering With Head Noun Extraction
An exploration of how standard embeddings can create a semantic soup by grouping search queries by adjectives rather than head nouns during clustering.
Transcript
Imagine looking at a list of items that includes blue thermal socks, cheap gaming laptops, and rental bulldozers. If you had to group them, you would probably make three neat piles: socks, laptops, and bulldozers. That is how the human brain naturally categorizes the world.
But what happens when we ask a machine to do the same task? If we convert those search queries into mathematical vectors and cluster them by similarity, we get a very different result.
Instead of grouping by the actual objects, the machine groups them by their adjectives. It puts all the "cheap" things together, all the "blue" things together, and all the "used" things together.
This happens because standard embeddings create a semantic soup. The vector for "cheap laptop" is a mathematical average of "cheap" and "laptop." Because "cheap" is such a strong concept, it pulls the vector toward other cheap items, completely ignoring the physical object itself.
An analysis of search queries reveals a wide variety of these patterns, combining adjectives, nouns, and verbs in complex ways. So, what do we do about this machine learning blind spot? To be continued.