Listen: From Hallucinations to Clicks
An automated method for mapping LLM-hallucinated URLs to valid pages using keyword matching and semantic similarity via vector embeddings and cosine similarity.
Transcript
Large language models often hallucinate URLs that do not exist, leading visitors to dead-end four-oh-four pages on your website. When you are managing millions of pages, manually fixing these broken links is impossible.
By analyzing server logs, we can identify these hallucinated visits, focusing specifically on four-oh-four errors referred by platforms like ChatGPT. We can then automatically map those fake URLs to real, active pages using a combination of keyword matching and semantic similarity.
For semantic matching, vector embeddings are highly effective. Specifically, extracting keywords directly from the hallucinated URL works best. This is because cosine similarity has a known quirk: it naturally favors shorter, more direct text over long, detailed paragraphs.
While we can map these pages automatically, setting up automatic redirects is risky. Semantic algorithms lack common sense, and they might accidentally pair a hallucinated link with something completely inappropriate. A safer solution is to keep the four-oh-four page active, but use your matching data to dynamically display the top recommended pages. This guides the user to the right place without the risk of an unwanted redirect.