Theory of Neural Networks
Code  Completion  Credits  Range  Language 

MITNN  Z,ZK  4  1P+1C  Czech 
 Lecturer:
 Martin Holeňa (guarantor)
 Tutor:
 Daniel Vašata, Martin Holeňa (guarantor)
 Supervisor:
 Department of Applied Mathematics
 Synopsis:

In this course, we study neural networks from the point of view of the theory of function approximation and from the point of view of probability theory.
At first, we recall basic concepts pertaining to artificial neural Networks, such as neurons and connections between them, types of neurons from the point of view of signal transmission, network topology, somatic and synaptic mappings, network training, and the role of time in neural networks. In connection with network topology, we get acquainted with its transformation into a canonical topology, and in connection with somatic and synaptic mappings, with their composition into mappings computed by the Network, Finally in connection with training, we pay attention to the problem of overtraining and to the fact that training is actually a specific optimization task, recalling the most typical objective functions and the most important optimization methods employed for neural network training. We will see the meaninig of all these concepts in the context of common kinds of forward neural networks.
Within the topic approximation approach to neural networks, we first notice the connection of neural networks to expressing functions of many variables using functions of fewer variables (Kolmogorov theorem, Vituškin theorem). Afterwards, we will see how the universal approximation capacity of neural networks can be mathematically formalized as the sets of mappings computed by neural networks being dense in important Banach spaces of functions, in particular in the spaces of continuous functions, spaces of functions integrable with respect to a finite measure, spaces of functions with continuous derivatives, and Sobolev spaces.
Within the topic probabilistic approach, we first get acquainted with training based on expectation and training based on a random sample, and with probabilistic assumptions about training data with which those two kinds of neural networks can be employed. We will see how it is possible to get an estimate of the conditional expectancy of network outputs conditioned by its inputs using the expectancy based learning. We recall the strong and the weak law of large numbers and get acquainted with an analogy of the strong law of large numbers for neural networks and with the assumptions for its validity. Finally, we recall the central limit theorem, get acquinted with its analogy for neural networks, with the assumptions for its validity and with the hypothesis tests based on it. We will see how those tests can be employed to search for the topology of the network.
 Requirements:
 Syllabus of lectures:

1,2: Basic concept of artificial neural networks.
3,4: Artificial neural networks from the point of view of function approximation theory.
5,6: Artificial neural networks from the point of view of probability theory.
 Syllabus of tutorials:

1: Introduction to Matlab.
2: Survey of Python neural networks libraries.
3,4: Shallow neural networks in Matlab.
5,6: Deep neural networks in Matlab.
 Study Objective:
 Study materials:

[1] M.T. Hagan, H.B. Demuth, and M.H. Beale. Neural Network Design. PWS Publishing, 1996.
[2] T. Hasti, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer, 2001.
[3] M.H. Beale, M.T. Hagan, and H.B. Demuth. Deep Learning Toolbox User's Guide, Version 12. Mathworks, 2018.
[4] H. White. Artificial Neural Networks: Approximation and Learning Theory. Blackwell
Publishers,992.
 Note:
 Further information:
 http://courses.fit.cvut.cz/MITNN
 No timetable has been prepared for this course
 The course is a part of the following study plans:

 Knowledge Engineering, in Czech, Presented in Czech, Version 2016 and and 2017 (elective course)
 Computer Security, Presented in Czech, Version 2016 to 2019 (elective course)
 Computer Systems and Networks, Presented in Czech, Version 2016 to 2019 (elective course)
 Design and Programming of Embedded Systems, in Czech, Version 2016 to 2019 (elective course)
 Specialization Web and Software Engineering, in Czech, Version 2016 to 2019 (elective course)
 Specialization Software Engineering, in Czech, Version 2016 to 2019 (elective course)
 Specialization Web Engineering, Presented in Czech, Version 2016 to 2019 (elective course)
 Master Informatics, Presented in Czech, Version 2016 to 2019 (elective course)
 Specialization System Programming, Presented in Czech, Version 2016 to 2019 (elective course)
 Specialization Computer Science, Presented in Czech, Version 20162017 (elective course)
 Specialization Computer Science, Presented in Czech, Version 2018 to 2019 (elective course)
 Knowledge Engineering, in Czech, Presented in Czech, Version 2018 to 2019 (elective course)