Abstract
Recent progress in tabular foundation models suggests that training on synthetic tasks can substantially improve in-context learning capabilities, with overall performance largely depending on how well models can infer task-specific predictive relationships from the available context during inference. In this paper, we introduce TAFFY, a tabular foundation model with an In-Context Diversity Prior and a Task-Conditioned Looped Transformer that strengthen this ability. Specifically, to construct each synthetic pretraining context, the In-Context Diversity Prior samples from multiple related environments derived via controlled interventions and distribution shifts on a shared causal process. This in-context diversity encourages the model to learn a more comprehensive and task-specific representation. Moreover, the Task-Conditioned Looped Transformer iteratively and selectively applies a shared group of Transformer blocks to refine contextual representations, with a task-conditioned gate modulating the final hidden-state update. This enables task-adaptive iterative refinement. Together, these components encourage the model to identify predictive relationships from contextual contrasts during pretraining and dynamically modulate context integration for each task. Across six classification and five regression benchmark datasets, TAFFY attains the lowest average rank.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Li, Z., Song, X., Luo, G., Qiao, J., Cai, R., Chen, Z., Dong, X., Feng, F., Chen, G., & Zhang, K. (2026). TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity. https://omanscience.com/en/articles/taffy-a-task-adaptive-tabular-foundation-model-with-in-context-diversity
MLA 9
Li, Zijian, et al. "TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity." https://omanscience.com/en/articles/taffy-a-task-adaptive-tabular-foundation-model-with-in-context-diversity.
Chicago (author–date)
Li, Zijian, Xiangchen Song, Gongxu Luo, Jie Qiao, Ruichu Cai, Zhenhao Chen, Xinshuai Dong, Fan Feng, Guangyi Chen, and Kun Zhang. 2026. "TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity." https://omanscience.com/en/articles/taffy-a-task-adaptive-tabular-foundation-model-with-in-context-diversity.
Harvard
Li, Z., Song, X., Luo, G., Qiao, J., Cai, R., Chen, Z., Dong, X., Feng, F., Chen, G. and Zhang, K. (2026) 'TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity', Available at: https://omanscience.com/en/articles/taffy-a-task-adaptive-tabular-foundation-model-with-in-context-diversity.
Vancouver
Li Z, Song X, Luo G, Qiao J, Cai R, Chen Z, et al. TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity. https://omanscience.com/en/articles/taffy-a-task-adaptive-tabular-foundation-model-with-in-context-diversity
IEEE
Z. Li, X. Song, G. Luo, J. Qiao, R. Cai, Z. Chen, X. Dong, F. Feng, G. Chen, and K. Zhang, "TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity," https://omanscience.com/en/articles/taffy-a-task-adaptive-tabular-foundation-model-with-in-context-diversity.