Abstract
Step-size selection remains a central challenge in large-scale neural network optimization; conservative steps slow convergence, while aggressive steps can destabilize it. We combine \textbf{Z}ero-and-\textbf{F}irst-\textbf{O}rder optimization~(ZFO) and propose a lightweight framework that decouples direction selection from step-size. ZFO uses a trusted first-order optimizer to determine the direction and performs zeroth-order evaluations only along this one-dimensional subspace to choose how far to move. Using the current {gradient information} and two additional objective function evaluations, ZFO instances construct a local model of the objective function along the proposed direction and select a curvature-aware step within a bounded search interval. This yields an adaptive step-selection mechanism that costs less than a full line search. We provide theoretical guarantees to show that shared-sample evaluations produce reliable finite-difference curvature estimates, that the induced local model selects a near-optimal step along the search interval, and that ZFO converges to a neighborhood of a stationary point. Across the evaluated settings, language models and datasets, ZFO frequently improves optimization and final performance relative to fixed-step first-order baselines, with the magnitude and preferred local model depending on the objective. Our code is publicly available at: https://github.com/nizswan/Zeroth-First-Order-Framework.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
McGee, C., Bergou, E. H., & Dutta, A. (2026). Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning. https://omanscience.com/en/articles/trust-the-direction-search-the-step-zero-and-first-order-methods-for-llm-fine-tuning
MLA 9
McGee, Cristian, et al. "Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning." https://omanscience.com/en/articles/trust-the-direction-search-the-step-zero-and-first-order-methods-for-llm-fine-tuning.
Chicago (author–date)
McGee, Cristian, El Houcine Bergou, and Aritra Dutta. 2026. "Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning." https://omanscience.com/en/articles/trust-the-direction-search-the-step-zero-and-first-order-methods-for-llm-fine-tuning.
Harvard
McGee, C., Bergou, E. H. and Dutta, A. (2026) 'Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning', Available at: https://omanscience.com/en/articles/trust-the-direction-search-the-step-zero-and-first-order-methods-for-llm-fine-tuning.
Vancouver
McGee C, Bergou EH, Dutta A. Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning. https://omanscience.com/en/articles/trust-the-direction-search-the-step-zero-and-first-order-methods-for-llm-fine-tuning
IEEE
C. McGee, E. H. Bergou, and A. Dutta, "Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning," https://omanscience.com/en/articles/trust-the-direction-search-the-step-zero-and-first-order-methods-for-llm-fine-tuning.