The compendium provides an introduction to the theory of deep learning, from basic principles of neural network modeling and optimization to more advanced topics of neural networks as Gaussian processes, neural tangent and information theory. This unique reference text complements a largely missing theoretical introduction to neural networks without being overwhelmingly technical in a level accessible to upper-level undergraduate engineering students.