2022-05-03-amid22a.md

title	abstract	software	layout	series	publisher	issn	id	month	tex_title	firstpage	lastpage	page	order	cycles	bibtex_author	author	date	address	container-title	volume	genre	issued	pdf	extras
LocoProp: Enhancing BackProp via Local Loss Optimization	Second-order methods have shown state-of-the-art performance for optimizing deep neural networks. Nonetheless, their large memory requirement and high computational complexity, compared to first-order methods, hinder their versatility in a typical low-budget setup. This paper introduces a general framework of layerwise loss construction for multilayer neural networks that achieves a performance closer to second-order methods while utilizing first-order optimizers only. Our methodology lies upon a three-component loss, target, and regularizer combination, for which altering each component results in a new update rule. We provide examples using squared loss and layerwise Bregman divergences induced by the convex integral functions of various transfer functions. Our experiments on benchmark models and datasets validate the efficacy of our new approach, reducing the gap between first-order and second-order optimizers.	https://github.com/google-research/google-research/tree/master/locoprop	inproceedings	Proceedings of Machine Learning Research	PMLR	2640-3498	amid22a	0	LocoProp: Enhancing BackProp via Local Loss Optimization	9626	9642	9626-9642	9626	false	Amid, Ehsan and Anil, Rohan and Warmuth, Manfred	given family Ehsan Amid given family Rohan Anil given family Manfred Warmuth	2022-05-03		Proceedings of The 25th International Conference on Artificial Intelligence and Statistics	151	inproceedings	date-parts 2022 5 3	https://proceedings.mlr.press/v151/amid22a/amid22a.pdf