Logo ČVUT
CZECH TECHNICAL UNIVERSITY IN PRAGUE
STUDY PLANS
2021/2022

Deep Learning

Login to KOS for course enrollment Display time-table
Code Completion Credits Range Language
BEV033DLE Z,ZK 6 2P+2C English
Lecturer:
Boris Flach (guarantor), Oleksandr Shekhovtsov
Tutor:
Boris Flach (guarantor), Oleksandr Shekhovtsov, Jan Šochman
Supervisor:
Department of Cybernetics
Synopsis:

The course introduces deep neural networks and deep learning – a branch of machine learning and artificial intelligence. Starting from a recap of generic concepts of machine learning (empirical risk minimisation, linear classifiers and regressions, generalisation bounds), it will introduce deep networks as model classes for prediction (classification) and regression and discuss their model complexity and generalisation bounds. The course aims at a solid understanding of all concepts and algorithms needed to successfully design, implement and learn deep networks in machine learning applications. This includes error back propagation and stochastic gradient methods, weight initialisation and normalisation, deterministic and stochastic regularisation methods, data augmentation as well as adversarially robust learning approaches. The course concludes with an introductory discussion of generative neural networks (VAEs and GANs) as well as recurrent neural networks (GRU and LSTM) for structured output classification.

Students will gain solid knowledge of all related methods and concepts as well as practical skills needed for successfully designing, implementing and learning deep networks for machine learning applications. At the same time, this course will provide a solid fundament for forthcoming courses (e.g. computer vision), which consider specialised and often more complex variants of neural networks, loss functions and learning approaches for solving machine learning task in their respective area.

Requirements:

Fundamentals of mathematics comparable to the following courses: Linear Algebra (B0B01LAG), Calculus (B0B01MA2), Optimization (B0B33OPT) as well as Probability, Statistics, and Theory of Information (B0B01PST ).

Besides proficient knowledge of mathematics as given above, students are expected to have solid knowledge in the following areas of computer science and artificial intelligence: basics of graph theory and related algorithms; basics of pattern recognition, empirical risk minimisation, linear classifiers, support vector machines as in Pattern Recognition and Machine Learning (B4B33RPZ or BE4B33RPZ).

Syllabus of lectures:

1. Recap: linear classifiers, linear regression, logistic regression, loss function, empirical risk minimisation, regularisation

2. Artificial neurons, activation functions, network architectures; sidestep: stochastic neurons; sidestep: biological

neurons

3. Neural networks as classifiers, empirical risk minimisation, loss functions, model complexity and generalisation bounds; neural networks as nonlinear regression models, loss functions

4. Backpropagation for feed-forward networks with arbitrary DAG structure, simplification/modularisation for layered

networks

5. NN loss landscape, Stochastic gradient descent for convex functions, SGD for nonlinear functions, (Nesterov)

momentum

6. Convolutional neural networks, architectures, application examples, side step: visual cortex

7. Training neural networks 0: project pipeline, data collection, training/validation/test set, model selection (architecture), overfitting, early stopping

8. Training neural networks 1: data preprocessing, weight initialisation, batch normalisation

9. Training neural networks 2: Adaptive SGD methods

10. Training neural networks 3: regularisation, L1/L2 weight regularisation, randomised predictors, dropout, data

augmentation

11. Training neural networks 4: adversarial patterns, robust learning approaches

12. Generative models: VAE, GANs (introductory level)

13. Recurrent neural networks: recurrent back-propagation, RNN, GRU, LSTM

14. Reserve, other topics not covered, e.g. graph neural networks, convolutions on graphs

Syllabus of tutorials:

Two types of labs (tutorials) will be proposed for the course (alternating):

• practical labs discussing homework assignments in which students will implement selected methods discussed in the course and experiment with them

• theoretical labs in which students will discuss solutions of theoretical assignments (made available before the class).

Study Objective:

The proposed course aims at providing the relevant algorithmic and theoretical concepts needed for successfully designing and training NNs. At the same time it strives at providing technical and practical skills in this domain.

Study materials:

I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016

Note:
Further information:
https://cw.fel.cvut.cz/wiki/courses/bev033dle/start
Time-table for winter semester 2021/2022:
Time-table is not available yet
Time-table for summer semester 2021/2022:
06:00–08:0008:00–10:0010:00–12:0012:00–14:0014:00–16:0016:00–18:0018:00–20:0020:00–22:0022:00–24:00
Mon
Tue
Wed
roomKN:E-301
Flach B.
Shekhovtsov O.

11:00–12:30
(lecture parallel1)
Karlovo nám.
Šrámkova posluchárna K9
Thu
roomKN:E-128
Flach B.
Shekhovtsov O.

09:15–10:45
(lecture parallel1
parallel nr.101)

Karlovo nám.
Cvičebna K3
roomKN:E-112
Flach B.
Shekhovtsov O.

11:00–12:30
(lecture parallel1
parallel nr.102)

Karlovo nám.
Cvičebna Vyčichlova
Fri
room
Flach B.
Shekhovtsov O.

11:00–12:30
(lecture parallel1
parallel nr.103)

The course is a part of the following study plans:
Data valid to 2022-08-08
For updated information see http://bilakniha.cvut.cz/en/predmet6170206.html