Theory of Neural Networks

Course guarantor:

Lecturer:

Tutor:

Supervisor:

Department of Applied Mathematics

Synopsis:

In this course, we study neural networks from the point of view of the theory of function approximation and from the point of view of probability theory.

At first, we recall basic concepts pertaining to artificial neural Networks, such as neurons and connections between them, types of neurons from the point of view of signal transmission, network topology, somatic and synaptic mappings, network training, and the role of time in neural networks. In connection with network topology, we get acquainted with its transformation into a canonical topology, and in connection with somatic and synaptic mappings, with their composition into mappings computed by the Network, Finally in connection with training, we pay attention to the problem of overtraining and to the fact that training is actually a specific optimization task, recalling the most typical objective functions and the most important optimization methods employed for neural network training. We will see the meaninig of all these concepts in the context of common kinds of forward neural networks.

Within the topic approximation approach to neural networks, we first notice the connection of neural networks to expressing functions of many variables using functions of fewer variables (Kolmogorov theorem, Vituškin theorem). Afterwards, we will see how the universal approximation capacity of neural networks can be mathematically formalized as the sets of mappings computed by neural networks being dense in important Banach spaces of functions, in particular in the spaces of continuous functions, spaces of functions integrable with respect to a finite measure, spaces of functions with continuous derivatives, and Sobolev spaces.

Within the topic probabilistic approach, we first get acquainted with training based on expectation and training based on a random sample, and with probabilistic assumptions about training data with which those two kinds of neural networks can be employed. We will see how it is possible to get an estimate of the conditional expectancy of network outputs conditioned by its inputs using the expectancy based learning. We recall the strong and the weak law of large numbers and get acquainted with an analogy of the strong law of large numbers for neural networks and with the assumptions for its validity. Finally, we recall the central limit theorem, get acquinted with its analogy for neural networks, with the assumptions for its validity and with the hypothesis tests based on it. We will see how those tests can be employed to search for the topology of the network.

Requirements:

Syllabus of lectures:

1,2: Basic concept of artificial neural networks.

3,4: Artificial neural networks from the point of view of function approximation theory.

5,6: Artificial neural networks from the point of view of probability theory.

Syllabus of tutorials:

1: Introduction to Matlab.

2: Survey of Python neural networks libraries.

3,4: Shallow neural networks in Matlab.

5,6: Deep neural networks in Matlab.

Study Objective:

Study materials:

[1] M.T. Hagan, H.B. Demuth, and M.H. Beale. Neural Network Design. PWS Publishing, 1996.

[2] T. Hasti, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer, 2001.

[3] M.H. Beale, M.T. Hagan, and H.B. Demuth. Deep Learning Toolbox User's Guide, Version 12. Mathworks, 2018.

[4] H. White. Artificial Neural Networks: Approximation and Learning Theory. Blackwell

Publishers,992.

Note:

Further information:

http://courses.fit.cvut.cz/MI-TNN

No time-table has been prepared for this course

The course is a part of the following study plans:

• Master branch Knowledge Engineering, in Czech, 2016-2017 (elective course)
• Master branch Computer Security, in Czech, 2016-2019 (elective course)
• Master branch Computer Systems and Networks, in Czech, 2016-2019 (elective course)
• Master branch Design and Programming of Embedded Systems, in Czech, 2016-2019 (elective course)
• Master branch Web and Software Engineering, spec. Info. Systems and Management, in Czech, 2016-2019 (elective course)
• Master branch Web and Software Engineering, spec. Software Engineering, in Czech, 2016-2019 (elective course)
• Master branch Web and Software Engineering, spec. Web Engineering, in Czech, 2016-2019 (elective course)
• Master program Informatics, unspecified branch, in Czech, version 2016-2019 (elective course)
• Master branch System Programming, spec. System Programming, in Czech, 2016-2019 (elective course)
• Master branch System Programming, spec. Computer Science, in Czech, 2016-2017 (elective course)
• Master specialization Computer Science, in Czech, 2018-2019 (elective course)
• Master branch Knowledge Engineering, in Czech, 2018-2019 (elective course)