Comparative numerical laboratory
Optimization Algorithms: Gradient Descent versus Newton — Chapter 14
Minimize a quadratic with anisotropic curvature and compare trajectories.
\[
\phi(x,y)=(x-1)^2+10(y+1)^2.
\]
Methods compared
| Method | Convenience | Inconvenience / warning |
|---|---|---|
| Gradient descent | Cheap iterations. | Can zigzag and needs step-size control. |
| Newton | Uses curvature and solves this quadratic in one step. | Requires Hessian solve; can fail without globalization in nonlinear problems. |
| Line search viewpoint | Protects decrease. | Adds algorithmic overhead. |
Illustrative plot / animation
Numerical diagnostics
What the student must compare
- objective decrease
- path on contour plot
- gradient norm
- step-size sensitivity