About

In the last few decades, much effort has been devoted to the development of first-order methods. These methods enjoy a low per-iteration cost and have optimal complexity, are easy to implement, and have proven to be effective for most machine learning applications. First-order methods, however, have significant limitations: (1) they require fine hyper-parameter tuning, (2) they do not incorporate curvature information, and thus are sensitive to ill-conditioning, and (3) they are often unable to fully exploit the power of distributed computing architectures.

Higher-order methods, such as Newton, quasi-Newton and adaptive gradient descent methods, are extensively used in many scientific and engineering domains. These methods possess several nice features: they exploit local curvature information to mitigate the effects of ill-conditioning, they avoid or diminish the need for hyper-parameter tuning, and they have enough concurrency to take advantage of distributed computing environments. This workshop will attempt to bring machine learning and optimization researchers closer, in order to facilitate a discussion with regards to underlying questions such as the following:

• Why are they not omnipresent?
• Why are higher-order methods important in machine learning?
• What advantages can they offer? What are their limitations and disadvantages?
• How should (or could) they be implemented in practice?

Previous versions of this workshop have appeared in NeurIPS'19 and ICML'20.