Tractable structured natural gradient descent using local parameterizations

Wu Lin (UBC Computer Science) IAM Graduate Seminar
July 27, 2022 11:00 am LSK 306
Natural-gradient descent (NGD) on structured parameter spaces (e.g., low-rank covariances) is computationally challenging due to difficult Fisher-matrix computations. We address this issue by using local-parameter coordinates to obtain a flexible and efficient NGD method that works well for a wide-variety of structured parameterizations. We show four applications where our method (1) generalizes a block-box gradient-free algorithm, (2) recovers existing Newton-like algorithms, (3) yields new structured second-order algorithms via matrix Lie groups, and (4) gives new algorithms to learn covariances of Gaussian and Wishart-based distributions. We show results on a range of problems from deep learning, variational inference, and evolution strategies. Our work opens a new direction for scalable structured geometric methods.
Project page: