On the convergence to the maximum likelihood estimate of multinomial logistic regression under gradient descent

Multinomial logistic regression is one of the simplest types of artificial neural networks. Despite its simplicity, some questions remain about its training dynamics under gradient descent. We study convergence to the maximum likelihood estimate when gradient descent is used as the optimizer. Our approach is based on an operator-theoretic framework which introduces positive definiteness into the problem. Using this approach, we investigate nontrivial bounds on the smallest and largest eigenvalues of the Hessian. In connecting these bounds to the learning rate, we provide conditions under which convergence locally occurs to the maximum likelihood estimate. Our upper bound is complementary to existing literature, while our lower bound contribution appears much newer.