Preprint Open access
Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization
While Mixture-of-Experts (MoE) models effectively scale model capacity through sparse activation, their deployment is often bottlenecked by prohibitive memory requirements. Extracting a compact subset of experts presents a promising solution. However, existing expert selection heuristics predominantly rely on Top-k ran …