# Gemini answers by user name: data package

Data behind the DEJAN article "Gemini writes better answers when it knows the user is a recognised expert"
(https://dejan.ai/blog/expert-users/) and its interactive dashboard
(https://dejan.ai/media/research/expert-users/dashboard.html).

## What was tested

15 tasks, one per field. Each task was sent 15 times as a single user message to `gemini-4-carbon` on Vertex AI,
thinking level HIGH, default temperature, on 12 November 2026:

- 5 times with the system instruction `User name: <expert>` (a recognised expert in that field)
- 5 times with `User name: <random name>` (an invented name, matched to the expert on gender and name style)
- 5 times with no system instruction

A separate Claude session scored all 225 answers from 1 to 10 and ranked the 15 answers in each topic, without the
names. Before rating, answers were shuffled and given random two-letter codes, invented names inside answers were
replaced with "Example", and opening greeting lines ("Hi Joan,") were removed. Real experts cited as sources were left in.

## Files

| File | Contents |
|---|---|
| `tasks.csv` | The 15 tasks with their random name and expert |
| `answers.csv` | All 225 answers: version, name, system instruction, answer text, Gemini's thinking summary, token counts, latency, blind code, score, rank and the rater's one-sentence reason |
| `key.csv` | Blind code to version and name, per topic |
| `rater_files/topic_NN.md` | The files exactly as the rater received them |
| `rater_instructions.md` | The two messages sent to the rater, in full |
| `persona_bias.db` | SQLite database with everything above (tables `tasks`, `batches`, `single_responses`, `blind_ratings`) |
| `scripts/` | The scripts that ran the test, made the blind files and built the dashboard |

`persona_bias.db` also holds an earlier two-turn pilot on topic 1 (tables `runs` and `responses`): the task was
answered once, then the user said "Just so you know, you are talking to {name}". That pilot is not part of the article's numbers.

## Columns in answers.csv

- `version`: `expert`, `random name` or `no name`
- `thinking_summary`: the thought summary returned by the API with `include_thoughts=True`. It is Google's summary of
  the model's reasoning and sometimes includes the summariser's own instructions; it is not the raw reasoning.
- `output_tokens`, `thinking_tokens`: from the API's usage metadata
- `blind_score`: 1 to 10; `blind_rank_in_topic`: 1 (best) to 15 within the topic

## Scope

- 5 answers per version per topic, and one rater.
- The no-name version has no system instruction, so its gap to the other two versions includes any effect of having a
  system instruction at all. The expert and random name versions have the same system instruction format.
- The same rater session rated topic 1 first, then topics 2 to 15. It was never told what the test was about.

## Reproduce

```bash
pip install google-genai
export GOOGLE_CLOUD_PROJECT=your-project        # Vertex AI, application default credentials
cd scripts
python run_single.py --task 1 --samples 5       # writes to persona_bias.db next to the script
python make_blind.py 1                          # writes /tmp/answer-review/topic_01.md and blind_key_topics.json
python build_dashboard.py                       # needs the blind_ratings table filled with your rater's scores
```

`run_single.py` reads the tasks from `scripts/domains.md`.