Learn how Gradient Descent, Stochastic Gradient Descent (SGD), and Adam optimization utilize calculus. Highly Recommended PDFs and Books
: These lecture notes focus specifically on matrix calculus, which is essential for understanding deep learning and large-scale optimization. Direct PDF Link calculus for machine learning pdf link
Written by Terence Parr and Jeremy Howard (co-founder of fast.ai), this paper explains matrix calculus explicitly for neural network practitioners. Machine learning is primarily about finding patterns and
Machine learning is primarily about finding patterns and minimizing errors. Calculus provides the language and tools to achieve this. Follow this accelerated plan: | Function | Derivative
Downloading a PDF and letting it sit on your hard drive does nothing. Follow this accelerated plan:
| Function | Derivative | |----------|-------------| | ( x^n ) | ( n x^n-1 ) | | ( e^x ) | ( e^x ) | | ( \ln x ) | ( 1/x ) | | ( \sigma(x) = \frac11+e^-x ) | ( \sigma(x)(1-\sigma(x)) ) | | ( \tanh(x) ) | ( 1 - \tanh^2(x) ) | | ( \textReLU(x) = \max(0,x) ) | 0 if x<0, 1 if x>0 (undefined at 0, but subgradient 0..1) | | Softmax ( p_i = \frace^z_i\sum_j e^z_j ) | ( p_i(\delta_ij - p_j) ) |