Beyond Rank Tracking: Analyzing Brand Perceptions Through Language Model Association Networks

This post is based on the codebase and specifications for AI Rank, an AI visibility and rank tracking framework developed by DEJAN AI team: https://airank.dejan.ai/

Abstract:

Traditional SEO has long relied on rank tracking as a primary metric of online visibility. However, modern search engines, increasingly driven by large language models (LLMs), are evolving beyond simple ranking algorithms. They now construct intricate knowledge graphs and semantic networks that interconnect brands, concepts, and user intent in complex ways. This paper introduces the DEJAN methodology, a novel approach that leverages the power of LLMs to analyze brand perception and positioning in a way that surpasses the limitations of traditional rank tracking. We demonstrate how directly probing LLMs can reveal hidden brand associations, competitive landscapes, and evolving market dynamics, providing a richer, more nuanced understanding of a brand’s online presence. This methodology offers a proactive, data-driven approach to brand management and SEO, shifting the focus from simply monitoring keyword rankings to understanding the broader semantic context in which a brand exists.

1. Introduction: The Limitations of Traditional Rank Tracking

For years, Search Engine Optimization (SEO) practitioners have used keyword rank tracking as a cornerstone of their strategies. The position a website holds in Search Engine Results Pages (SERPs) for specific keywords has been considered a direct indicator of online visibility and a proxy for organic traffic. While rank tracking remains a useful signal, its efficacy is diminishing in the face of evolving search engine technology.

Modern search engines, such as Google, heavily utilize Large Language Models (LLMs) like BERT, LaMDA, and Gemini. These models possess a deep understanding of language, context, and relationships between concepts. They don’t simply match keywords; they interpret user intent, analyze semantic relationships, and construct knowledge graphs that connect entities (brands, products, people, places, etc.) based on their associations and contextual relevance.

This shift presents several challenges to traditional rank tracking:

• Personalization and Context: SERPs are increasingly personalized based on user history, location, and other factors. A single, universal rank for a given keyword becomes less meaningful.
• Zero-Click Searches: Featured snippets, knowledge panels, and other rich results often satisfy user queries directly within the SERP, reducing click-through rates even for top-ranked pages.
• Semantic Understanding: LLMs can understand queries and content in ways that go beyond simple keyword matching. A website might be highly relevant to a user’s query even if it doesn’t explicitly target the specific keywords being tracked.
• Brand Perception: Traditional rank tracking provides no insight into how a brand is perceived. It only indicates visibility for specific keywords, not the associations, sentiment, or overall context surrounding the brand.

These limitations highlight the need for a more sophisticated approach to understanding online visibility – one that accounts for the semantic and contextual understanding of LLMs.

2. Language Models and Brand Associations

LLMs, trained on vast amounts of text and code, develop internal representations of language that capture semantic relationships between words and concepts. They can, for example, understand that “Apple” can refer to both a fruit and a technology company, and they can infer the relevant meaning based on context. Crucially, LLMs can also identify and quantify the strength of associations between different entities.

By directly querying an LLM with prompts designed to elicit these associations, we can gain insights into how a brand is perceived. For example, asking an LLM to “List ten things that you associate with the brand [Brand Name]” can reveal key concepts, products, competitors, and even sentiments linked to that brand. This provides a “brand association network” that goes far beyond what traditional keyword research can uncover.

These associations are not static. LLMs are continuously updated and their internal knowledge graphs evolve. By repeatedly querying LLMs over time, we can track changes in brand perception and identify emerging trends.

3. The DEJAN Methodology: Mapping Brand Perception

The DEJAN methodology provides a structured approach to analyzing brand perception using LLMs. It consists of the following key steps:

1. Project Definition:
   • Define Target Brands: Identify the brand(s) to be analyzed. This could be a single brand, a set of competitors, or a broader category of brands.
   • Define Tracked Phrases (Entities): Select relevant entities, keywords, concepts, or phrases related to the brand’s industry, products, or services.
   • Define locations (optional).
   • Define languages (optional).
2. Prompt Design: Craft prompts that elicit relevant associations from the LLM. Two primary prompt types are used:
   • Brand-to-Entity (B→E): “List ten things that you associate with a brand called [Brand Name].” This reveals the concepts and entities most strongly linked to the brand.
   • Entity-to-Brand (E→B): “List ten brands that you associate with [Entity/Keyword].” This identifies competitors and reveals the brands most strongly associated with a specific concept.
3. Data Collection:
   • Automated Probing: Utilize an API or other automated method to repeatedly query the LLM with the designed prompts. Record the responses, timestamps, and any available metadata (e.g., confidence scores, grounding sources if using a grounded LLM).
   • Multiple LLMs: Employ multiple LLMs (e.g., GPT-4o, Gemini) to provide a more robust and comprehensive view, mitigating potential biases inherent in any single model.
   • Grounded vs. Ungrounded: (For models like Gemini) Collect both grounded (search-backed) and ungrounded responses. Grounded responses reflect information available on the web, while ungrounded responses reflect the LLM’s internal knowledge. Comparing these provides insights into current online visibility versus the LLM’s inherent understanding.
4. Data Normalization:
   • Entity Extraction: Extract individual entities from the LLM responses. This may involve cleaning and standardizing the text (e.g., removing punctuation, handling variations in capitalization).
   • Canonicalization: Group variant forms of the same entity (e.g., “Apple Inc.”, “Apple computers”, “Apple”) under a single canonical representation. This can be done manually, algorithmically, or using a combination of both.
   • Ranking: Assign ranks to the entities based on their position in the LLM’s response. Typically, the first item in a list is considered rank 1, the second rank 2, and so on.
5. Data Analysis and Visualization:
   • Frequency Analysis: Count the number of times each entity appears in the responses. This reveals the most prominent associations.
   • Average Rank Calculation: Calculate the average rank of each entity across all responses. Lower average ranks indicate stronger associations.
   • Weighted Score: Calculate a weighted score combining frequency and average rank to better capture the relative importance of entities.
   • Time Series Analysis: Track changes in entity frequencies, average ranks, and weighted scores over time to identify trends and shifts in brand perception.
   • Network Visualization: Represent the brand association network as a graph, with nodes representing brands and entities, and edges representing the strength of their associations.
   • Competitive Analysis: Compare the brand association networks of multiple brands to identify areas of overlap, differentiation, and potential competitive threats.
   • Grounded vs. Ungrounded Comparison: (For models like Gemini) Analyze the differences between grounded and ungrounded responses to identify gaps between current online visibility and the LLM’s inherent understanding.
6. Reporting and Actionable Insights:
   • Summarize the findings in a clear and concise report, highlighting key associations, trends, and competitive insights.
   • Develop actionable recommendations based on the data. This might include:
     • Identifying new content opportunities based on emerging associations.
     • Refining marketing messaging to reinforce desired associations or address negative ones.
     • Monitoring competitor activities and positioning.
     • Tracking the impact of marketing campaigns on brand perception.

5. Conclusion

The DEJAN methodology offers a significant advancement in understanding online visibility and brand perception. By directly tapping into the knowledge and associative capabilities of LLMs, it provides a more nuanced and dynamic view than traditional rank tracking. This approach empowers brands to:

• Move beyond keywords: Understand the broader semantic context in which their brand exists.
• Uncover hidden associations: Identify unexpected connections and potential brand risks.
• Track perception over time: Monitor how brand associations evolve and respond to market changes or marketing efforts.
• Gain a competitive edge: Analyze competitor positioning and identify opportunities for differentiation.
• Make data-driven decisions: Inform content strategy, marketing campaigns, and overall brand management with concrete insights.

As search engines and LLMs continue to evolve, methodologies like our will become increasingly crucial for navigating the complexities of the modern online landscape and maintaining a strong, relevant brand presence.

Future Work:

• Refining Prompt Engineering: Investigating more sophisticated prompt engineering techniques to elicit even more specific and nuanced associations.
• Sentiment Analysis: Integrating sentiment analysis to quantify the positive, negative, or neutral nature of brand associations.
• Cross-Lingual Analysis: Adapting the methodology for use with multiple languages.
• Automated Anomaly Detection: Developing algorithms to automatically identify significant shifts or anomalies in brand association networks.
• Integration with other Data Sources: Combining LLM-derived insights with traditional SEO data, social media analytics, and other data sources for a holistic view of brand performance.
• User Intent Modeling: Exploring how LLM probing can be used to model user intent and inform content strategy.

This article was drafted by Google’s Gemini model from raw code. Curated, fact checked and edited by Dan Petrovic to form the final published version.