Using the same tech behind AI Rank, we prompted Google’s latest Gemini 2.5 Pro model with search grounding enabled in the API request. A total of 10,000 prompts were collected and analysed to determine the grounding status of the prompt. The resulting data was then used to train a replica of Google’s internal classifier which determines if query deserves grounding.
This is a commercial-grade model we now use as part of our machine learning toolkit and various data processing pipelines. The model’s capability is demonstrated in our QDG tool.
The discovery of Gemini’s grounding in a live production environment matches the official Google documentation for developers. The default dynamic retrieval threshold for determining whether user query requires grounding is 0.3 and the responses to user queries can be drastically different when grounded.
Example Prompts
To see the difference in raw output between the two, click on the example prompts to expand for full detail:
What is the current population of India? [GROUNDED]
candidates=[Candidate(content=Content(parts=[Part(video_metadata=None, thought=None, code_execution_result=None, executable_code=None, file_data=None, function_call=None, function_response=None, inline_data=None, text="As of March 31, 2025, estimates for the current population of India vary slightly depending on the source and methodology:\n\n* **Approximately 1.460 billion:** One source indicates the population is 1,460,579,770 as of March 31, 2025, based on UN data interpolation.\n* **Approximately 1.460 billion:** Another source states the population is 1,460,458,003 as of March 30, 2025, based on Worldometer's elaboration of UN data.\n* **Approximately 1.454 billion:** A different source estimates the current 2025 population at 1,454,606,724.\n* **Approximately 1.461 billion:** An estimate for January 1, 2025, placed the population at 1,461,898,454.\n\n**Mid-year 2025 Projections:**\n\n* Several sources project India's population to be around **1.463 to 1.464 billion** (1,463,865,525) by mid-year 2025.\n* Another estimate projects the population to reach **1.45 billion** in 2025.\n\n**Key Points:**\n\n* India is currently the most populous country in the world, having surpassed China.\n* India's population represents approximately 17.78% of the total world population.\n* The population growth rate for India in 2025 is projected to be around 0.88% or 0.89%.\n* The population is expected to continue growing, potentially reaching 1.5 billion by 2028 or 2030, and peaking around 1.7 billion in the 2060s before starting to decline.")], role='model'), citation_metadata=None, finish_message=None, token_count=None, finish_reason=<FinishReason.STOP: 'STOP'>, avg_logprobs=None, grounding_metadata=GroundingMetadata(grounding_chunks=[GroundingChunk(retrieved_context=None, web=GroundingChunkWeb(title='statisticstimes.com', uri='https://vertexaisearch.cloud.google.com/grounding-api-redirect/AWQVqALIg8FJcmLFZvbG4G87Rmgji-rHRCFaYzOlDms7hKtIbgjw7TrdanldvA1XM9x7SqYQ4XXYxnKAYcGeVX6POX2B-c6obxc0eqRSf01MFM5sc1KUPwfjF7Obf0VQIdTc-px1KP09E8chbPyHT6jSGsCPwUrzg0Y_iP7GN4g=')), GroundingChunk(retrieved_context=None, web=GroundingChunkWeb(title='countrymeters.info', uri='https://vertexaisearch.cloud.google.com/grounding-api-redirect/AWQVqAKKjKIObQWSV5nL3INgeQeIuMn5x3ufFzaXgo6C3QJSU4ynTEDAROuNZycpqgWjkzqKWHUbheRF3EgLdprsBl1MTnd31edoqDoJdhX5vs7X-PU0vRRqR35hLg==')), GroundingChunk(retrieved_context=None, web=GroundingChunkWeb(title='factodata.com', uri='https://vertexaisearch.cloud.google.com/grounding-api-redirect/AWQVqAJkQs1RQmRfuigZ7dwaTDT2lULp_8jpg4agWgGxLE8rXQnbnly66N4LQM_vW99WMhRx2lv1zkrKvl9-inRkV1Hc9MXFzhE9kk-3FzYzyBzIhj8UHUmMKB5Ng0DtNcnt7mYlzeUdyiesxzGb7zdC')), GroundingChunk(retrieved_context=None, web=GroundingChunkWeb(title='worldometers.info', uri='https://vertexaisearch.cloud.google.com/grounding-api-redirect/AWQVqAKYb_U61-k35R0MERwc0xum5rx2xMbk6POyli8zNlZmbE5JeO2NAlMaLJ7KTV7dWlifH56zpM_BmM_IspAfcsVeGZLDO1GA4C1kvYsioA6oWx8Nj7IevrLTAsLJjtdXAq0VrbQLtl237AOZj3VIdfnqURpKNoer')), GroundingChunk(retrieved_context=None, web=GroundingChunkWeb(title='worldometers.info', uri='https://vertexaisearch.cloud.google.com/grounding-api-redirect/AWQVqALLgzdjnnlHn_aSvaQUi_CTazd4ShW1R9MVW8D-Jl6fXj5NfzuRHuNa__0DwYGADwIwIpQkN7_8-10JKtcuPs-cIpSRWBWAESq4jsMoNnHxWkBTo_ErC555HtsHjaqD8Cc0uVPT_K4Jrl3Rd7SLej8vW18s42TIgAMDG_E='))], grounding_supports=[GroundingSupport(confidence_scores=[0.924835], grounding_chunk_indices=[0], segment=Segment(end_index=999, part_index=None, start_index=856, text='* **Approximately 1.460 billion:** One source indicates the population is 1,460,579,770 as of March 31, 2025, based on UN data interpolation.')), GroundingSupport(confidence_scores=[0.67996716], grounding_chunk_indices=[1], segment=Segment(end_index=1381, part_index=None, start_index=1272, text='* **Approximately 1.461 billion:** An estimate for January 1, 2025, placed the population at 1,461,898,454.')), GroundingSupport(confidence_scores=[0.68266284], grounding_chunk_indices=[0], segment=Segment(end_index=1535, part_index=None, start_index=1415, text="* Several sources project India's population to be around **1.463 to 1.464 billion** (1,463,865,525) by mid-year 2025.")), GroundingSupport(confidence_scores=[0.74773, 0.63307124], grounding_chunk_indices=[2, 0], segment=Segment(end_index=1615, part_index=None, start_index=1536, text='* Another estimate projects the population to reach **1.45 billion** in 2025.')), GroundingSupport(confidence_scores=[0.96703804, 0.9120462], grounding_chunk_indices=[0, 2], segment=Segment(end_index=1720, part_index=None, start_index=1634, text='* India is currently the most populous country in the world, having surpassed China.')), GroundingSupport(confidence_scores=[0.86126155, 0.9295672, 0.65202457], grounding_chunk_indices=[0, 3, 4], segment=Segment(end_index=1806, part_index=None, start_index=1721, text="* India's population represents approximately 17.78% of the total world population.")), GroundingSupport(confidence_scores=[0.9009998], grounding_chunk_indices=[0], segment=Segment(end_index=2071, part_index=None, start_index=1898, text='* The population is expected to continue growing, potentially reaching 1.5 billion by 2028 or 2030, and peaking around 1.7 billion in the 2060s before starting to decline.'))], retrieval_metadata=None, retrieval_queries=None, search_entry_point=SearchEntryPoint(rendered_content='<style>\n.container {\n align-items: center;\n border-radius: 8px;\n display: flex;\n font-family: Google Sans, Roboto, sans-serif;\n font-size: 14px;\n line-height: 20px;\n padding: 8px 12px;\n}\n.chip {\n display: inline-block;\n border: solid 1px;\n border-radius: 16px;\n min-width: 14px;\n padding: 5px 16px;\n text-align: center;\n user-select: none;\n margin: 0 8px;\n -webkit-tap-highlight-color: transparent;\n}\n.carousel {\n overflow: auto;\n scrollbar-width: none;\n white-space: nowrap;\n margin-right: -12px;\n}\n.headline {\n display: flex;\n margin-right: 4px;\n}\n.gradient-container {\n position: relative;\n}\n.gradient {\n position: absolute;\n transform: translate(3px, -9px);\n height: 36px;\n width: 9px;\n}\n@media (prefers-color-scheme: light) {\n .container {\n background-color: #fafafa;\n box-shadow: 0 0 0 1px #0000000f;\n }\n .headline-label {\n color: #1f1f1f;\n }\n .chip {\n background-color: #ffffff;\n border-color: #d2d2d2;\n color: #5e5e5e;\n text-decoration: none;\n }\n .chip:hover {\n background-color: #f2f2f2;\n }\n .chip:focus {\n background-color: #f2f2f2;\n }\n .chip:active {\n background-color: #d8d8d8;\n border-color: #b6b6b6;\n }\n .logo-dark {\n display: none;\n }\n .gradient {\n background: linear-gradient(90deg, #fafafa 15%, #fafafa00 100%);\n }\n}\n@media (prefers-color-scheme: dark) {\n .container {\n background-color: #1f1f1f;\n box-shadow: 0 0 0 1px #ffffff26;\n }\n .headline-label {\n color: #fff;\n }\n .chip {\n background-color: #2c2c2c;\n border-color: #3c4043;\n color: #fff;\n text-decoration: none;\n }\n .chip:hover {\n background-color: #353536;\n }\n .chip:focus {\n background-color: #353536;\n }\n .chip:active {\n background-color: #464849;\n border-color: #53575b;\n }\n .logo-light {\n display: none;\n }\n .gradient {\n background: linear-gradient(90deg, #1f1f1f 15%, #1f1f1f00 100%);\n }\n}\n</style>\n<div class="container">\n <div class="headline">\n <svg class="logo-light" width="18" height="18" viewBox="9 9 35 35" fill="none" xmlns="http://www.w3.org/2000/svg">\n <path fill-rule="evenodd" clip-rule="evenodd" d="M42.8622 27.0064C42.8622 25.7839 42.7525 24.6084 42.5487 23.4799H26.3109V30.1568H35.5897C35.1821 32.3041 33.9596 34.1222 32.1258 35.3448V39.6864H37.7213C40.9814 36.677 42.8622 32.2571 42.8622 27.0064V27.0064Z" fill="#4285F4"/>\n <path fill-rule="evenodd" clip-rule="evenodd" d="M26.3109 43.8555C30.9659 43.8555 34.8687 42.3195 37.7213 39.6863L32.1258 35.3447C30.5898 36.3792 28.6306 37.0061 26.3109 37.0061C21.8282 37.0061 18.0195 33.9811 16.6559 29.906H10.9194V34.3573C13.7563 39.9841 19.5712 43.8555 26.3109 43.8555V43.8555Z" fill="#34A853"/>\n <path fill-rule="evenodd" clip-rule="evenodd" d="M16.6559 29.8904C16.3111 28.8559 16.1074 27.7588 16.1074 26.6146C16.1074 25.4704 16.3111 24.3733 16.6559 23.3388V18.8875H10.9194C9.74388 21.2072 9.06992 23.8247 9.06992 26.6146C9.06992 29.4045 9.74388 32.022 10.9194 34.3417L15.3864 30.8621L16.6559 29.8904V29.8904Z" fill="#FBBC05"/>\n <path fill-rule="evenodd" clip-rule="evenodd" d="M26.3109 16.2386C28.85 16.2386 31.107 17.1164 32.9095 18.8091L37.8466 13.8719C34.853 11.082 30.9659 9.3736 26.3109 9.3736C19.5712 9.3736 13.7563 13.245 10.9194 18.8875L16.6559 23.3388C18.0195 19.2636 21.8282 16.2386 26.3109 16.2386V16.2386Z" fill="#EA4335"/>\n </svg>\n <svg class="logo-dark" width="18" height="18" viewBox="0 0 48 48" xmlns="http://www.w3.org/2000/svg">\n <circle cx="24" cy="23" fill="#FFF" r="22"/>\n <path d="M33.76 34.26c2.75-2.56 4.49-6.37 4.49-11.26 0-.89-.08-1.84-.29-3H24.01v5.99h8.03c-.4 2.02-1.5 3.56-3.07 4.56v.75l3.91 2.97h.88z" fill="#4285F4"/>\n <path d="M15.58 25.77A8.845 8.845 0 0 0 24 31.86c1.92 0 3.62-.46 4.97-1.31l4.79 3.71C31.14 36.7 27.65 38 24 38c-5.93 0-11.01-3.4-13.45-8.36l.17-1.01 4.06-2.85h.8z" fill="#34A853"/>\n <path d="M15.59 20.21a8.864 8.864 0 0 0 0 5.58l-5.03 3.86c-.98-2-1.53-4.25-1.53-6.64 0-2.39.55-4.64 1.53-6.64l1-.22 3.81 2.98.22 1.08z" fill="#FBBC05"/>\n <path d="M24 14.14c2.11 0 4.02.75 5.52 1.98l4.36-4.36C31.22 9.43 27.81 8 24 8c-5.93 0-11.01 3.4-13.45 8.36l5.03 3.85A8.86 8.86 0 0 1 24 14.14z" fill="#EA4335"/>\n </svg>\n <div class="gradient-container"><div class="gradient"></div></div>\n </div>\n <div class="carousel">\n <a class="chip" href="https://vertexaisearch.cloud.google.com/grounding-api-redirect/AWQVqALcoI6oeHxZuBVcXRcQ-1YTR07O2TEZB-ZcmnTIBL7CVu6HukX7tIjrIdZBvuj9DoLuyzQY0y7GYJNHFRvicusGGYhF90HbAXsHYbJhEKvCMF_5ZEZvgHuSijPPD8O4le3TtYncn7v0auTPwoeHuigC0z8nhwgeIOznYX9IAHfBvppp51XqJwtjUadaJnZ3Kg3Eq_esbEHIg0By6ZY=">India population 2025 estimate</a>\n <a class="chip" href="https://vertexaisearch.cloud.google.com/grounding-api-redirect/AWQVqAJYdtY-fLKGpk_h20WhJuXC2wvq5XXOGY4Xr7JiGjJjOiM-GrVTBJ9ANvWXLAFhW9JnqERH5E_5tBvNb6IG6Moy4ALvAXD0CJeA9vvFydSq_Qpq88uXeZhsrClYbtZlwUC25qIPvN0vw5iGvflQK3I8bfIVbz-h0wlFYqROJzBXH6UaE4P3O1U_xf0G1jOgPxc4NhF9iTt9d33b9kAdaWbkBUsP2d2Yr1NlAjraK7HO8TE=">What is the current population of India in 2025?</a>\n </div>\n</div>\n', sdk_blob=None), web_search_queries=['What is the current population of India in 2025?', 'India population 2025 estimate']), index=0, logprobs_result=None, safety_ratings=None)] create_time=None response_id=None model_version='gemini-2.5-pro-exp-03-25' prompt_feedback=None usage_metadata=GenerateContentResponseUsageMetadata(cache_tokens_details=None, cached_content_token_count=None, candidates_token_count=444, candidates_tokens_details=None, prompt_token_count=13, prompt_tokens_details=[ModalityTokenCount(modality=<MediaModality.TEXT: 'TEXT'>, token_count=13)], thoughts_token_count=None, tool_use_prompt_token_count=None, tool_use_prompt_tokens_details=None, total_token_count=457) automatic_function_calling_history=[] parsed=None
Tell me a joke about computers. [NOT GROUNDED]
candidates=[Candidate(content=Content(parts=[Part(video_metadata=None, thought=None, code_execution_result=None, executable_code=None, file_data=None, function_call=None, function_response=None, inline_data=None, text='Why did the computer keep sneezing?\nIt had a virus!')], role='model'), citation_metadata=None, finish_message=None, token_count=None, finish_reason=<FinishReason.STOP: 'STOP'>, avg_logprobs=None, grounding_metadata=GroundingMetadata(grounding_chunks=None, grounding_supports=None, retrieval_metadata=None, retrieval_queries=None, search_entry_point=None, web_search_queries=None), index=0, logprobs_result=None, safety_ratings=None)] create_time=None response_id=None model_version='gemini-2.5-pro-exp-03-25' prompt_feedback=None usage_metadata=GenerateContentResponseUsageMetadata(cache_tokens_details=None, cached_content_token_count=None, candidates_token_count=32, candidates_tokens_details=None, prompt_token_count=12, prompt_tokens_details=[ModalityTokenCount(modality=<MediaModality.TEXT: 'TEXT'>, token_count=12)], thoughts_token_count=None, tool_use_prompt_token_count=None, tool_use_prompt_tokens_details=None, total_token_count=44) automatic_function_calling_history=[] parsed=NoneTraining Data
Parsing the raw output enabled us to determine the label for each of the 10,000 prompts and generate a robust training dataset based on the decisions made by Google’s own classifier.
- “What is the current population of India?”,1
- “Tell me a joke about computers”,0
- “How do I bake sourdough bread from scratch?”,0
- “Latest news headlines in South Africa?”,1
- “Write a short story set on Mars.”,0
- “Compare the camera specs of the Google Pixel 9 Pro and the iPhone 16 Pro Max.”,1
Synthetic Dataset
In order to address the class imbalance between grounded and ungrounded responses (0 and 1) in the original dataset we also generated synthetic training data. To do so, an entire classification corpus was supplied to Gemini as a system prompt and it was instructed to generate additional examples in the minority class to pad the training dataset.
Model Training
We fine-tuned Microsoft’s DeBERTaV3 (large) model for binary text classification task using a 90:10 dataset split for training and validation and evaluated model performance by monitoring training loss, validation loss, precision, recall, accuracy and F1, which was also used to select the best model.
The model was trained for 5 epochs on a single RTX4090 using a batch size of 24 samples, checkpointing/validating every 500 steps and logging to Weights and Biases every 10 steps.
# ============ Model ============
# Load config first to modify dropout
from transformers import AutoConfig
config = AutoConfig.from_pretrained(MODEL_NAME)
config.hidden_dropout_prob = 0.1 # Adjust hidden layer dropout
config.attention_probs_dropout_prob = 0.1 # Adjust attention dropout
config.num_labels = 2
model = AutoModelForSequenceClassification.from_pretrained(
MODEL_NAME,
config=config
)
# ============ Metrics ============
def compute_metrics(eval_pred):
logits, labels = eval_pred
preds = torch.argmax(torch.tensor(logits), dim=1)
precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='binary')
acc = accuracy_score(labels, preds)
return {
"accuracy": acc,
"f1": f1,
"precision": precision,
"recall": recall
}
# ============ Trainer Setup ============
args = TrainingArguments(
output_dir=OUTPUT_DIR,
evaluation_strategy="steps",
eval_steps=VAL_EVAL_STEPS,
save_strategy="steps", # Explicitly set save strategy
save_steps=CHECKPOINT_STEPS,
save_total_limit=5,
logging_steps=10, # Less frequent logging to reduce overhead
per_device_train_batch_size=BATCH_SIZE,
per_device_eval_batch_size=BATCH_SIZE,
num_train_epochs=5,
# Learning rate adjustments
learning_rate=5e-6, # Reduced from 1e-5
lr_scheduler_type="cosine", # Cosine scheduler for smoother decay
# Warmup adjustments
warmup_ratio=0.1, # Use ratio instead of fixed steps
# Stability improvements
gradient_accumulation_steps=2, # Simulate larger batch
max_grad_norm=1.0, # Gradient clipping
weight_decay=0.01, # L2 regularization
# Mixed precision for better numerical stability
fp16=True,
# Optimizer configuration
optim="adamw_torch",
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-8,
# Save best model
load_best_model_at_end=True,
metric_for_best_model="f1",
# Reporting
report_to="wandb",
logging_dir=os.path.join(OUTPUT_DIR, "logs"),
)
Leave a Reply