Abstract
Speech quality predictors are increasingly used as rewards, yet no agreed measure of their hackability exists. The usual measurement has two flaws. First, the perturbation reaches the predictor through a processing chain -- here a neural codec -- that shifts the score on its own, which scoring against the raw input charges to the attack. Referencing the unperturbed round trip instead changes measured hackability by up to a factor of four (0.31 to 0.08 for one defence). Second, one trained attacker is a sample, not a measurement: five attackers differing only in random seed reach success rates from 0.00 to 0.38 against one fixed predictor, so a defence claim needs the worst case over several. Under this protocol, four published predictors differ widely: NISQA is hacked on 90% of utterances, SSL-MOS on 21%, DNSMOS on 14% and UTMOS on 6%. We then audit a closed attack-detect-patch loop. It hardens the predictor only in its own attack space, by less than the spread between attackers; a random-perturbation baseline matches it; and it costs up to 0.30 system SRCC out of domain. Enhancers post-trained against patched predictors hack them far less (PESQ -0.03 versus -0.23). Code, preregistration and run outputs are released.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Alavi, A., & Williamson, D. S. (2026). How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buys. https://omanscience.com/en/articles/how-hackable-is-your-speech-quality-metric-a-corrected-protocol-a-benchmark-and-what-patching-buys
MLA 9
Alavi, Ali, and Donald S. Williamson. "How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buys." https://omanscience.com/en/articles/how-hackable-is-your-speech-quality-metric-a-corrected-protocol-a-benchmark-and-what-patching-buys.
Chicago (author–date)
Alavi, Ali, and Donald S. Williamson. 2026. "How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buys." https://omanscience.com/en/articles/how-hackable-is-your-speech-quality-metric-a-corrected-protocol-a-benchmark-and-what-patching-buys.
Harvard
Alavi, A. and Williamson, D. S. (2026) 'How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buys', Available at: https://omanscience.com/en/articles/how-hackable-is-your-speech-quality-metric-a-corrected-protocol-a-benchmark-and-what-patching-buys.
Vancouver
Alavi A, Williamson DS. How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buys. https://omanscience.com/en/articles/how-hackable-is-your-speech-quality-metric-a-corrected-protocol-a-benchmark-and-what-patching-buys
IEEE
A. Alavi, and D. S. Williamson, "How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buys," https://omanscience.com/en/articles/how-hackable-is-your-speech-quality-metric-a-corrected-protocol-a-benchmark-and-what-patching-buys.