Objectives
Objectives: The objective of this course is to cover the fundamentals of neural networks as well as some advanced topics such as recurrent neural networks, long short term memory cells and convolutional neural networks. The course also requires students to implement programming assignments related to these topics.
Syllabus
- Basics: Biological Neuron, Idea of computational units, McCulloch–Pitts unit and Thresholding logic, Linear Perceptron, Perceptron Learning Algorithm, Linear separability. Convergence theorem for Perceptron Learning Algorithm.
- Feedforward Networks: Multilayer Perceptron, Gradient Descent, Backpropagation, Empirical Risk Minimization, regularization, autoencoders.
- Deep Neural Networks: Difficulty of training deep neural networks, Greedy layerwise training.
- Better Training of Neural Networks: Newer optimization methods for neural networks (Adagrad, adadelta, rmsprop, adam, NAG), second order methods for training, Saddle point problem in neural networks, Regularization methods (dropout, drop connect, batch normalization).
- Recurrent Neural Networks: Back propagation through time, Long Short Term Memory, Gated Recurrent Units, Bidirectional LSTMs, Bidirectional RNNs
- Convolutional Neural Networks: LeNet, AlexNet.
- Generative models: Restrictive Boltzmann Machines (RBMs), Introduction to MCMC and Gibbs Sampling, gradient computations in RBMs, Deep Boltzmann Machines.
- Recent trends: Variational Autoencoders, Generative Adversarial Networks, Multi-task Deep Learning, Multi-view Deep Learning
- Applications: Vision, NLP, Speech (just an overview of different applications in 2-3 lectures)
Textbooks
Deep Learning, Ian Goodfellow and Yoshua Bengio and Aaron Courville, MIT Press, 2016.
References:
- Neural Networks: A Systematic Introduction, Raúl Rojas, 1996
- Pattern Recognition and Machine Learning, Christopher Bishop, 2007
