Home › Research › Digital health tools
Repeatability of Commonly Used Speech and Language Features for Clinical Applications
Stegmann GM, Hahn S, Liss J, Shefner J, Rutkove SB, Kawabata K, Bhandari S, Shelton K et al.. Digital biomarkers, 2020-09-01
Topic: Risk, prevention and care in population studies
Introduction Changes in speech have the potential to provide important information on the diagnosis and progression of various neurological diseases. Many researchers have relied on open-source speech features to develop algorithms for measuring speech changes in clinical populations as they are convenient and easy to use. However, the repeatability of open-source features in the context of neurological diseases has not been studied. Methods We used a longitudinal sample of healthy controls, individuals with amyotrophic lateral sclerosis, and individuals with suspected frontotemporal dementia, and we evaluated the repeatability of acoustic and language features separately on these 3 data sets. Results Repeatability was evaluated using intraclass correlation (ICC) and the within-subjects coefficient of variation (WSCV). In 3 sets of tasks, the median ICC were between 0.02 and 0.55, and the median WSCV were between 29 and 79%. Conclusion Our results demonstrate that the repeatability of speech features extracted using open-source tool kits is low. Researchers should exercise caution when developing digital health models with open-source speech features. We provide a detailed summary of feature-by-feature repeatability results (ICC, WSCV, SE of measurement, limits of agreement for WSCV, and minimal detectable change) in the online supplementary material so that researchers may incorporate repeatability information into the models they develop.
Key findings
- Results Repeatability was evaluated using intraclass correlation (ICC) and the within-subjects coefficient of variation (WSCV).
- Conclusion Our results demonstrate that the repeatability of speech features extracted using open-source tool kits is low.
- We provide a detailed summary of feature-by-feature repeatability results (ICC, WSCV, SE of measurement, limits of agreement for WSCV, and minimal detectable change) in the online supplementary material so that researchers may incorporate repeatability information into the models they develop.
Why it matters
This study found that commonly used open-source speech features had low repeatability across healthy controls and patients with ALS or suspected frontotemporal dementia. It is a caution about measurement methods and reports no treatment or prevention.
New to this subject? Start with our plain-English guide: Digital health →
Study details
- Study designlongitudinal study
- Evidence strengthModerate
How this study scored 49/100
Published in a peer-reviewed journal, and it is a longitudinal study following the same people over time. Scored zero for sample size, effect reported, studied in people and randomised / blinded because the record does not state it.
| Component | Points | Why |
|---|---|---|
| Journal tier | 7 / 10 | Tier 3 — peer-reviewed journal (Digital biomarkers) |
| Journal's citation record | 0 / 4 | Digital Biomarkers: journal h-index 38 (OpenAlex, 2026-10-29) |
| Study design | 7 / 12 | A longitudinal study following the same people over time |
| Strength of evidence | 3 / 5 | Rated “moderate” in our summary |
| Peer-review validation | 10 / 10 | Peer-reviewed. Published as an article in a named peer-reviewed journal. |
| How recent | 2 / 5 | 2020 — within 10 year(s) of the newest study here |
| Sample size | 0 / 4 | Participant count not stated — scores zero (not stated in the record — scored 0) |
| Effect reported | 0 / 3 | No effect size reported — scores zero (not stated in the record — scored 0) |
| Studied in people | 0 / 3 | Not stated whether this was studied in people — scores zero (not stated in the record — scored 0) |
| Randomised / blinded | 0 / 2 | No randomisation or blinding stated — scores zero (not stated in the record — scored 0) |
| Times cited | 4 / 7 | Cited 47 times (OpenAlex, 2026-10-10) |
| Citations per year | 2 / 5 | 5.9 citations a year over 8 year(s) since 2020 |
| Impact for its field and year | 3 / 6 | Cited 3.4× the average for papers of its field and year (OpenAlex FWCI) |
| Cited by studies here | 0 / 6 | No other study in this library cites it |
| Still being cited | 2 / 4 | Cited 20 times in 2024–2026 (2026 still in progress; OpenAlex) |
| Senior author's track record | 2 / 4 | Last-listed author Visar Berisha: h-index 30 (OpenAlex, whole career) |
| Institution's track record | 3 / 3 | Harvard University: institution h-index 3,235 (OpenAlex) |
| Multi-centre collaboration | 2 / 3 | OpenAlex lists 4 institution(s) in 1 country |
| Indexed in PubMed | 2 / 2 | Indexed in PubMed (PMID 33442573) |
| Free to read | 0 / 2 | Behind a paywall — scores zero |
| Total | 49 / 100 | — |
What each of these 20 components measures, and what this score is not →
Read the original
Full text: free to read — OpenAlex lists this copy as free to read: the link above (checked 6 Feb 1020).
Cite this study APA, Vancouver, BibTeX · download .bib or .ris
Built from the record alzhub holds for this paper, not looked up. Volume, issue and page numbers are not held in the record. The record's author list ends in “et al.”, so not every author is named. Check the citation against the original before you use it.
APA 7th
Stegmann, G. M., Hahn, S., Liss, J., Shefner, J., Rutkove, S. B., Kawabata, K., et al. (2020). Repeatability of Commonly Used Speech and Language Features for Clinical Applications. Digital biomarkers. https://doi.org/10.1159/000511671
Vancouver
Stegmann GM, Hahn S, Liss J, Shefner J, Rutkove SB, Kawabata K, et al. Repeatability of Commonly Used Speech and Language Features for Clinical Applications. Digital biomarkers. 2020. doi:10.1159/000511671. PMID: 33442573.
BibTeX
@article{stegmann2020-33442573,
author = {Stegmann, G. M. and Hahn, S. and Liss, J. and Shefner, J. and Rutkove, S. B. and Kawabata, K. and Bhandari, S. and Shelton, K. and others},
title = {{Repeatability of Commonly Used Speech and Language Features for Clinical Applications}},
journal = {Digital biomarkers},
year = {2020},
doi = {10.1159/000511671},
pmid = {33442573},
url = {https://doi.org/10.1159/000511671}
}
Related studies in Digital health tools
Also in Digital health tools
178 studies in the library are filed under Digital health tools with no topic tag they share with another; these are this study's neighbours in a fixed tour of them. The Digital health tools hub.