Abstract
Evaluating the capability of language models to conduct multi-turn interactions presents significant challenges. Current evaluation frameworks rely on arbitrary text sharding or user simulators, which introduce unwanted noise and require complex behavioral modeling. A methodology for evaluating multi-turn interactions through collaborative private information games is disclosed. Private information, such as images or structured data, is provided to participants. This information is essential for task completion but difficult to efficiently convert into text. A shared understanding must be collaboratively built by the participants through constrained linguistic communication. A fixed communication budget is allocated, and the number of permitted conversation turns is varied to isolate interaction capabilities. A scalable evaluation of multi-turn conversational capabilities is thereby enabled. Proactive communicative abilities are measured objectively without reliance on simulated human users.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Eisenstein, Jacob; Berant, Jonathan; Fisch, Adam; Lapata, Mirella; and Huot, Fantine, "Multi-Turn Evaluation Of Language Models Using Private Information Games", Technical Disclosure Commons, (September 09, 2026)
https://www.tdcommons.org/dpubs_series/11674