Optimization

References

• http://videolectures.net/deeplearning2015_goodfellow_network_optimization/ (Ian Goodfellow's tutorial on neural network optimization at Deep Learning Summer School 2015).
• http://int8.io/comparison-of-optimization-techniques-stochastic-gradient-descent-momentum-adagrad-and-adadelta (implementation and comparison of popular methods)
• http://www.deeplearningbook.org/contents/numerical.html (basic intro in 4.3)
• http://www.deeplearningbook.org/contents/optimization.html (8.1 generalization, 8.2 problems, 8.3 algorithms, 8.4 init, 8.5 adaptive lr, 8.6 approx 2nd order, 8.7 meta)
• https://arxiv.org/abs/1412.6544
• http://andrew.gibiansky.com/blog/machine-learning/gauss-newton-matrix/ (great posts on optimization)
• https://www.cs.cmu.edu/~quake-papers/painless-conjugate-gradient.pdf (excellent tutorial on cg, gd, eigens etc)
• http://arxiv.org/abs/1412.6544 (Goodfellow paper)
• https://d396qusza40orc.cloudfront.net/neuralnets/lecture_slides/lec6.pdf (hinton slides)
• https://d396qusza40orc.cloudfront.net/neuralnets/lecture_slides/lec8.pdf (hinton slides)
• http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html
• http://machinelearning.wustl.edu/mlpapers/paper_files/icml2010_Martens10.pdf
• http://arxiv.org/abs/1503.05671
• http://arxiv.org/abs/1412.1193
• http://www.springer.com/us/book/9780387303031 (nocedal and wright)
• http://www.nrbook.com (numerical recipes)
• https://maths-people.anu.edu.au/~brent/pub/pub011.html (without derivatives)
• http://stanford.edu/~boyd/cvxbook/ (only convex optimization)