https://amitrajan012.github.io/tags/optimization/Recent content in Optimization on Amit RajanHugoen-usSun, 04 Oct 2026 06:28:22 +0000https://amitrajan012.github.io/book/blog/the-mistake-that-makes-gradient-descent-work/Sun, 04 Oct 2026 06:28:22 +0000https://amitrajan012.github.io/book/blog/the-mistake-that-makes-gradient-descent-work/<p>Machine learning starts with a simple idea: find the model that makes the fewest mistakes. We measure those mistakes with a loss function, and training is the process of driving that loss downward. The natural conclusion is that we should keep descending until we reach the lowest possible point—the deepest valley in the landscape.</p> <p>There is a problem with this idea. The deepest valley is not always the best place to be.</p>