Preprint Open access
How Inefficient Is Natural Gradient Descent? From Exact Optimality to Θ(\sqrt{ \log d }) Divergence
Natural gradient descent (NGD) underlies common methods in ML. For dually flat families, idealized NGD on the forward Kullback--Leibler objective follows the mixture geodesic which is often longer than the shortest Fisher--Rao path. We quantify this overhead by the inefficiency ratio \(R \ge 1\), the Fisher length of t …