Home › Research › Brain tissue and pathology
Detecting Self-Repairs from Spontaneous Speech with Prompt Ablation Across LLMs and Fine-Tuned Encoder
Wu, R.; Pugh, S.; OCOnnor, K. B.; Xie, K.; O'Brien, K.; Johnson, K.. medRxiv (preprint), 2026-08-25
Topic: Brain proteins, microglia and lab models
Self-repairs - in-utterance revisions in which a speaker abandons and reformulates their speech - are a promising interpretable marker for speech-based cognitive screening. Detecting them automatically is difficult because a self-repair is defined by its relationship to surrounding speech rather than by fixed lexical cues. On the DementiaBank ADReSS corpus, we compared the capability of generative LLMs under a five-condition prompt ablation against a fine-tuned DistilBERT token classifier at detecting self-repairs. GPT-5 performed best (test F1 = 0.73) and was largely insensitive to prompt design, whereas the LlaMA (open-weight alternative) was both weaker and far more prompt-sensitive (test F1 = 0.47). DistilBERT, nearly 100 times smaller, matched the open-weight LLM at a fraction of the computational cost. These results suggest that a locally deployable encoder, given sufficient in-domain annotation, is a more plausible route to clinical self-repair detection than scaling model size or prompt complexity.
Key findings
- These results suggest that a locally deployable encoder, given sufficient in-domain annotation, is a more plausible route to clinical self-repair detection than scaling model size or prompt complexity.
Why it matters
This study compared methods for automatically detecting self-repairs in speech on the DementiaBank ADReSS corpus, finding a locally deployable encoder was a more plausible route than larger models or complex prompts. It concerns a potential speech-based screening marker, not treatment, cure or prevention.
Study details
- Evidence strengthLow
How this study scored 16/100
Preprint — not yet peer-reviewed. It scored 20 on the same rubric, but non-peer-reviewed sources are capped at 16, so it shows as 16. Scored zero for study design, sample size, effect reported, studied in people and randomised / blinded because the record does not state it. We have no data on impact for its field and year, so that was left out: it earned 19 of the 94 points we could judge it on, which is 20 out of 100.
| Component | Points | Why |
|---|---|---|
| Journal tier | 2 / 10 | Tier 4 — not a peer-reviewed journal (medRxiv (preprint)) |
| Journal's citation record | 0 / 4 | Not a journal article, so the journal's record does not count |
| Study design | 0 / 12 | No study design recorded — scores zero, not guessed (not stated in the record — scored 0) |
| Strength of evidence | 1 / 5 | Rated “low” in our summary |
| Peer-review validation | 4 / 10 | Preprint — not yet peer-reviewed. Hosted on a preprint server. |
| How recent | 5 / 5 | 2026 — within 1 year(s) of the newest study here |
| Sample size | 0 / 4 | Participant count not stated — scores zero (not stated in the record — scored 0) |
| Effect reported | 0 / 3 | No effect size reported — scores zero (not stated in the record — scored 0) |
| Studied in people | 0 / 3 | Not stated whether this was studied in people — scores zero (not stated in the record — scored 0) |
| Randomised / blinded | 0 / 2 | No randomisation or blinding stated — scores zero (not stated in the record — scored 0) |
| Times cited | 0 / 7 | Cited 0 times (OpenAlex, 2026-10-10) |
| Citations per year | 0 / 5 | 0.0 citations a year over 2 year(s) since 2026 |
| Impact for its field and year | — / 6 | No field-weighted citation impact (OpenAlex has none yet) — left out of this score |
| Cited by studies here | 0 / 6 | No other study in this library cites it |
| Still being cited | 0 / 4 | Cited 0 times in 2024–2026 (2026 still in progress; OpenAlex) |
| Senior author's track record | 2 / 4 | Last-listed author Kevin B. Johnson: h-index 39 (OpenAlex, whole career) |
| Institution's track record | 2 / 3 | University of Pennsylvania: institution h-index 1,859 (OpenAlex) |
| Multi-centre collaboration | 0 / 3 | OpenAlex lists 1 institution(s) in 1 country — scores zero (affiliation lists can be incomplete) |
| Indexed in PubMed | 1 / 2 | Not in PubMed, but has a DOI |
| Free to read | 2 / 2 | Free to read |
| Points | 19 / 94 | Scored on 19 of 20 parts. We have no data for the rest, so they are left out: 19 of the 94 points we could judge it on is 20 out of 100. |
| Total | 20 / 100 | Capped to 16 because this is not a peer-reviewed article. |
What each of these 20 components measures, and what this score is not →
Read the original
Full text: free to read — OpenAlex lists this copy as free to read: the link above (checked 6 Feb 1020).
Cite this study APA, Vancouver, BibTeX · download .bib or .ris
Built from the record alzhub holds for this paper, not looked up. Volume, issue and page numbers are not held in the record. Check the citation against the original before you use it.
APA 7th
Wu, R., Pugh, S., OCOnnor, K. B., Xie, K., O'Brien, K., & Johnson, K. (2026). Detecting Self-Repairs from Spontaneous Speech with Prompt Ablation Across LLMs and Fine-Tuned Encoder. medRxiv (preprint). https://doi.org/10.64898/2026.08.21.26360471
Vancouver
Wu R, Pugh S, OCOnnor KB, Xie K, O'Brien K, Johnson K. Detecting Self-Repairs from Spontaneous Speech with Prompt Ablation Across LLMs and Fine-Tuned Encoder. medRxiv (preprint). 2026. doi:10.64898/2026.08.21.26360471.
BibTeX
@article{wu2026-b062eef4,
author = {Wu, R. and Pugh, S. and OCOnnor, K. B. and Xie, K. and O'Brien, K. and Johnson, K.},
title = {{Detecting Self-Repairs from Spontaneous Speech with Prompt Ablation Across LLMs and Fine-Tuned Encoder}},
journal = {medRxiv (preprint)},
year = {2026},
doi = {10.64898/2026.08.21.26360471},
url = {https://doi.org/10.64898/2026.08.21.26360471}
}
Related studies in Brain tissue and pathology
Also in Brain tissue and pathology
124 studies in the library are filed under Brain tissue and pathology with no topic tag they share with another; these are this study's neighbours in a fixed tour of them. The Brain tissue and pathology hub.