Abstract
Evaluating conversational systems is a difficult and unresolved problem. We introduce the Conversational Distribution Score (CDS), which compares distributions of conversational behaviour using human conversations as a reference. CDS describes speech rate, syllabic rhythm, and turn interaction through eight interpretable features plus a separate two-feature semantic baseline. We compare conversations with two reference scales: one based on conversational success within human dialogue and another contrasting human and synthetic dialogue. Using listener judgments from out-of-domain goal--oriented dialogues, we examine system ranking, preferences between conversations, and ranking stability. Composite CDS recovers five of six listener system comparisons while individual features show strong correlation with listener preferences between conversations. We examine how many minutes and conversations are required before rankings stabilize. These findings support distributional comparisons as a complement to specific interactional metrics to evaluate conversations and conversational models while showing their interpretable value.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Satish, S. H. B., Cooper, E., Schmidtová, P., Züfle, M., Székely, É., Sanders, N., & Klejch, O. (2026). Distributional Metrics for Evaluating Spoken Conversational Systems. https://omanscience.com/en/articles/distributional-metrics-for-evaluating-spoken-conversational-systems
MLA 9
Satish, Shree Harsha Bokkahalli, et al. "Distributional Metrics for Evaluating Spoken Conversational Systems." https://omanscience.com/en/articles/distributional-metrics-for-evaluating-spoken-conversational-systems.
Chicago (author–date)
Satish, Shree Harsha Bokkahalli, Erica Cooper, Patrícia Schmidtová, Maike Züfle, Éva Székely, Nicholas Sanders, and Ondřej Klejch. 2026. "Distributional Metrics for Evaluating Spoken Conversational Systems." https://omanscience.com/en/articles/distributional-metrics-for-evaluating-spoken-conversational-systems.
Harvard
Satish, S. H. B., Cooper, E., Schmidtová, P., Züfle, M., Székely, É., Sanders, N. and Klejch, O. (2026) 'Distributional Metrics for Evaluating Spoken Conversational Systems', Available at: https://omanscience.com/en/articles/distributional-metrics-for-evaluating-spoken-conversational-systems.
Vancouver
Satish SHB, Cooper E, Schmidtová P, Züfle M, Székely É, Sanders N, et al. Distributional Metrics for Evaluating Spoken Conversational Systems. https://omanscience.com/en/articles/distributional-metrics-for-evaluating-spoken-conversational-systems
IEEE
S. H. B. Satish, E. Cooper, P. Schmidtová, M. Züfle, É. Székely, N. Sanders, and O. Klejch, "Distributional Metrics for Evaluating Spoken Conversational Systems," https://omanscience.com/en/articles/distributional-metrics-for-evaluating-spoken-conversational-systems.