Listen: Dynamic per-label thresholds for large-scale search query classification with Otsu’s method

When you're running search-intent classifiers in production, setting a single confidence threshold is a recipe for a headache. Some labels are common and score generously, while others are rare and score conservatively. If you set one global cutoff, you either flood your system with irrelevant results or starve your rare categories. Manually tuning thresholds for dozens of labels is a never-ending game of whack-a-mole.

A clever solution actually comes from image processing: Otsu’s algorithm. Originally built to separate the foreground of an image from its background, it can do the exact same thing for search data. Think of your label scores as a mountain range. On one side, you have likely negatives. On the other side, you have likely positives. Otsu's algorithm slides across this landscape and finds the deepest valley between them. This valley becomes the perfect, custom threshold for each individual label, adapting automatically without any hand-tuning.

To make this production-ready, you just need a couple of guardrails. Set a global minimum floor to block noise, and add a fallback rule that assigns the single highest-scoring label if a query gets left with nothing. This approach eliminates unlabeled rows, keeps noise in check, and scales effortlessly to any taxonomy. It solves dynamic thresholding once and for all, with no manual babysitting required.