First vs. Second-Order Optimization in Shallow Neural Networks: A Systematic Study Across Problem Landscapes
12th International Conference on Control, Decision and Information Technologies, CoDIT 2026, Bari, İtalya, 13 - 16 Temmuz 2026, ss.343-348, (Tam Metin Bildiri)
- Yayın Türü: Bildiri / Tam Metin Bildiri
- Doi Numarası: 10.1109/codit70676.2026.11630821
- Basıldığı Şehir: Bari
- Basıldığı Ülke: İtalya
- Sayfa Sayıları: ss.343-348
- Anahtar Kelimeler: Adam, Gauss-Newton, Ill-conditioned problems, Loss landscape geometry, Optimization, Second-order methods
- Hacettepe Üniversitesi Adresli: Evet
Özet
To ensure the accuracy and convergence of neural network training, it is essential to choose an appropriate optimization algorithm. First-order methods prevail in practical applications because of their computational efficiency, whereas second-order methods provide theoretical benefits in traversing intricate loss landscapes. This paper offers a systematic comparison of five optimization algorithms - Stochastic Gradient Descent (SGD), SGD with Momentum, Adam, Newton's method with diagonal Hessian approximation, and Limited-memory BFGS (L-BFGS) - across three distinct problem landscapes: convex (MNIST), non-convex bottleneck (Fashion-MNIST), and ill-conditioned (unnormalized CIFAR-10). Our experiments indicate that adaptive first-order methods exhibit strong performance in typical scenarios, attaining accuracies of up to 92.7% and 85.8% on convex and non-convex problems, respectively. Conversely, second-order methods demonstrate distinct, context-dependent benefits: Newton's method attains superior accuracy (23.9%) on ill-conditioned problems, markedly surpassing Adam (15.5%). Nonetheless, it performs poorly on non-convex landscapes (9.5% accuracy) because of vanishing gradients and saturating activations in bottleneck architectures, which compromise diagonal Hessian approximations. These findings offer evidence-based criteria for optimizer selection: Adam is a reliable default, momentum-based methods are crucial for non-convex optimization, and second-order methods are particularly advised for ill-conditioned situations lacking feature scaling.