Abstract
Transformers provide a state-of-the-art modeling framework, yet poor calibration limits their reliability in safety-critical applications. A promising direction addresses this issue by interpreting attention as a Gaussian process (GP) posterior, which enables principled uncertainty calibration but incurs cubic complexity in sequence length due to the inversion of the kernel; although decoupled GP variants reduced the cost to quadratic, the computation remains prohibitive in practice. In this paper, we propose the plug-and-play random Fourier feature Gaussian process attention (RFF-GPA) module, which represents the attention as a GP with a stationary kernel approximated by random Fourier features. This low-rank approximation results in linear-time complexity for approximating the posterior mean and variance, making it far more scalable compared to previous work. Empirical results on multiple real-world datasets show that our attention module improves calibration while maintaining predictive accuracy, and simultaneously reduces computational complexity to linear in the sequence length.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Mahfoozi, A. M., Yang, Z., Li, Y., & Zhang, M. M. (2026). Random Feature Gaussian Process Attention: Linear-Time Probabilistic Attention with Calibrated Uncertainty. https://omanscience.com/en/articles/random-feature-gaussian-process-attention-linear-time-probabilistic-attention-with-calibrated-uncertainty
MLA 9
Mahfoozi, Amir Mohammad, et al. "Random Feature Gaussian Process Attention: Linear-Time Probabilistic Attention with Calibrated Uncertainty." https://omanscience.com/en/articles/random-feature-gaussian-process-attention-linear-time-probabilistic-attention-with-calibrated-uncertainty.
Chicago (author–date)
Mahfoozi, Amir Mohammad, Zi Yang, Ying Li, and Michael Minyi Zhang. 2026. "Random Feature Gaussian Process Attention: Linear-Time Probabilistic Attention with Calibrated Uncertainty." https://omanscience.com/en/articles/random-feature-gaussian-process-attention-linear-time-probabilistic-attention-with-calibrated-uncertainty.
Harvard
Mahfoozi, A. M., Yang, Z., Li, Y. and Zhang, M. M. (2026) 'Random Feature Gaussian Process Attention: Linear-Time Probabilistic Attention with Calibrated Uncertainty', Available at: https://omanscience.com/en/articles/random-feature-gaussian-process-attention-linear-time-probabilistic-attention-with-calibrated-uncertainty.
Vancouver
Mahfoozi AM, Yang Z, Li Y, Zhang MM. Random Feature Gaussian Process Attention: Linear-Time Probabilistic Attention with Calibrated Uncertainty. https://omanscience.com/en/articles/random-feature-gaussian-process-attention-linear-time-probabilistic-attention-with-calibrated-uncertainty
IEEE
A. M. Mahfoozi, Z. Yang, Y. Li, and M. M. Zhang, "Random Feature Gaussian Process Attention: Linear-Time Probabilistic Attention with Calibrated Uncertainty," https://omanscience.com/en/articles/random-feature-gaussian-process-attention-linear-time-probabilistic-attention-with-calibrated-uncertainty.