Objective intelligibility assessment by automated segmental and suprasegmental listening error analysis

Yishan Jiao, Amy LaCross, Visar Berisha, Julie Liss

Research output: Contribution to journalArticlepeer-review

5 Scopus citations


Purpose: Subjective speech intelligibility assessment is often preferred over more objective approaches that rely on transcript scoring. This is, in part, because of the intensive manual labor associated with extracting objective metrics from transcribed speech. In this study, we propose an automated approach for scoring transcripts that provides a holistic and objective representation of intelligibility degradation stemming from both segmental and suprasegmental contributions, and that corresponds with human perception. Method: Phrases produced by 73 speakers with dysarthria were orthographically transcribed by 819 listeners via Mechanical Turk, resulting in 63,840 phrase transcriptions. A protocol was developed to filter the transcripts, which were then automatically analyzed using novel algorithms developed for measuring phoneme and lexical segmentation errors. The results were compared with manual labels on a randomly selected sample set of 40 transcribed phrases to assess validity. A linear regression analysis was conducted to examine how well the automated metrics predict a perceptual rating of severity and word accuracy. Results: On the sample set, the automated metrics achieved 0.90 correlation coefficients with manual labels on measuring phoneme errors, and 100% accuracy on identifying and coding lexical segmentation errors. Linear regression models found that the estimated metrics could predict a significant portion of the variance in perceptual severity and word accuracy. Conclusions: The results show the promising development of an objective speech intelligibility assessment that identifies intelligibility degradation on multiple levels of analysis.

Original languageEnglish (US)
Pages (from-to)3359-3366
Number of pages8
JournalJournal of Speech, Language, and Hearing Research
Issue number9
StatePublished - Sep 2019

ASJC Scopus subject areas

  • Language and Linguistics
  • Linguistics and Language
  • Speech and Hearing


Dive into the research topics of 'Objective intelligibility assessment by automated segmental and suprasegmental listening error analysis'. Together they form a unique fingerprint.

Cite this