Abstract
To reduce the number of parameters and cache memory requirements of transformers we introduce dual-headed transformers instead of three heads. We studied Walsh-Hadamard Transform (WHT), Discrete Cosine Transform (DCT), Discrete Fourier Transform, filterbank based Shearlet Transform, and Multiplication-Avoiding (MA) operators to construct dual heads. We combine spatial patches and their orthogonal transforms (or Shearlet and MA operators) in a structure similar to the attention block. We obtained better results than triple headed transformers in ImageNet. Extensive simulation examples are presented.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Meco, E., Hamdan, E., & Cetin, A. E. (2026). Learned Queries and Keys Are All You Need: Replacing the Value Projection with Structured Transforms. https://omanscience.com/en/articles/learned-queries-and-keys-are-all-you-need-replacing-the-value-projection-with-structured-transforms
MLA 9
Meco, Ene, et al. "Learned Queries and Keys Are All You Need: Replacing the Value Projection with Structured Transforms." https://omanscience.com/en/articles/learned-queries-and-keys-are-all-you-need-replacing-the-value-projection-with-structured-transforms.
Chicago (author–date)
Meco, Ene, Emadeldeen Hamdan, and A. Enis Cetin. 2026. "Learned Queries and Keys Are All You Need: Replacing the Value Projection with Structured Transforms." https://omanscience.com/en/articles/learned-queries-and-keys-are-all-you-need-replacing-the-value-projection-with-structured-transforms.
Harvard
Meco, E., Hamdan, E. and Cetin, A. E. (2026) 'Learned Queries and Keys Are All You Need: Replacing the Value Projection with Structured Transforms', Available at: https://omanscience.com/en/articles/learned-queries-and-keys-are-all-you-need-replacing-the-value-projection-with-structured-transforms.
Vancouver
Meco E, Hamdan E, Cetin AE. Learned Queries and Keys Are All You Need: Replacing the Value Projection with Structured Transforms. https://omanscience.com/en/articles/learned-queries-and-keys-are-all-you-need-replacing-the-value-projection-with-structured-transforms
IEEE
E. Meco, E. Hamdan, and A. E. Cetin, "Learned Queries and Keys Are All You Need: Replacing the Value Projection with Structured Transforms," https://omanscience.com/en/articles/learned-queries-and-keys-are-all-you-need-replacing-the-value-projection-with-structured-transforms.