Treewalker reconstructs the full response text from individual tokens to track character offsets, locates each parsed item substring within that span, and aggregates the probabilities across all overlapping tokens.
flowchart LR
A[Token Stream + Logprobs] --> B[Reconstruct Text with Character Offsets]
C[Parsed Item Text] --> D[Substring Index Search in Reconstructed Text]
B --> D
D --> E[Collect Overlapping Tokens]
E --> F[Calculate Mean & Min Confidence Probs]Referenced by