Theory of Neural Networks

Course guarantor:

Martin Holeňa

Lecturer:

Martin Holeňa

Tutor:

Marek Dědič, Martin Holeňa

Supervisor:

Department of Applied Mathematics

Synopsis:

In this course, we study neural networks from the point of view of the theory of function approximation and from the point of view of probability theory.

At first, we recall basic concepts pertaining to artificial neural Networks, such as neurons and connections between them, types of neurons from the point of view of signal transmission, network topology, somatic and synaptic mappings, network training, and the role of time in neural networks. In connection with network topology, we get acquainted with its transformation into a canonical topology, and in connection with somatic and synaptic mappings, with their composition into mappings computed by the Network, Finally in connection with training, we pay attention to the problem of overtraining and to the fact that training is actually a specific optimization task, recalling the most typical objective functions and the most important optimization methods employed for neural network training. We will see the meaninig of all these concepts in the context of common kinds of forward neural networks.

Within the topic approximation approach to neural networks, we first notice the connection of neural networks to expressing functions of many variables using functions of fewer variables (Kolmogorov theorem, Vituškin theorem). Afterwards, we will see how the universal approximation capacity of neural networks can be mathematically formalized as the sets of mappings computed by neural networks being dense in important Banach spaces of functions, in particular in the spaces of continuous functions, spaces of functions integrable with respect to a finite measure, spaces of functions with continuous derivatives, and Sobolev spaces.

Within the topic probabilistic approach, we first get acquainted with training based on expectation and training based on a random sample, and with probabilistic assumptions about training data with which those two kinds of neural networks can be employed. We will see how it is possible to get an estimate of the conditional expectancy of network outputs conditioned by its inputs using the expectancy based learning. We recall the strong and the weak law of large numbers and get acquainted with an analogy of the strong law of large numbers for neural networks and with the assumptions for its validity. Finally, we recall the central limit theorem, get acquinted with its analogy for neural networks, with the assumptions for its validity and with the hypothesis tests based on it. We will see how those tests can be employed to search for the topology of the network.

Requirements:

Probability and linear algebra knowledge at the bachelor level.

Syllabus of lectures:

1,2: Basic concept of artificial neural networks.

3,4: Artificial neural networks from the point of view of function approximation theory.

5,6: Artificial neural networks from the point of view of probability theory.

Syllabus of tutorials:

1: Introduction to Matlab.

2: Survey of Python neural networks libraries.

3,4: Shallow neural networks in Matlab.

5,6: Deep neural networks in Matlab.

Study Objective:

Study materials:

[1] M.T. Hagan, H.B. Demuth, and M.H. Beale. Neural Network Design. PWS Publishing, 1996.

[2] T. Hasti, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer, 2001.

[3] M.H. Beale, M.T. Hagan, and H.B. Demuth. Deep Learning Toolbox User's Guide, Version 12. Mathworks, 2018.

[4] H. White. Artificial Neural Networks: Approximation and Learning Theory. Blackwell

Publishers,992.

Note:

Further information:

http://courses.fit.cvut.cz/MI-TNN

Time-table for winter semester 2024/2025:

Time-table is not available yet

Time-table for summer semester 2024/2025:

Time-table is not available yet

The course is a part of the following study plans:

• Master specialization Computer Science, in Czech, 2018-2019 (elective course)
• Master specialization Computer Security, in Czech, 2020 (elective course)
• Master specialization Design and Programming of Embedded Systems, in Czech, 2020 (elective course)
• Master specialization Computer Systems and Networks, in Czech, 202 (elective course)
• Master specialization Management Informatics, in Czech, 2020 (elective course)
• Master specialization Software Engineering, in Czech, 2020 (elective course)
• Master specialization System Programming, in Czech, version from 2020 (elective course)
• Master specialization Web Engineering, in Czech, 2020 (elective course)
• Master specialization Knowledge Engineering, in Czech, 2020 (elective course)
• Master specialization Computer Science, in Czech, 2020 (elective course)
• Mgr. programme, for the phase of study without specialisation, ver. for 2020 and higher (elective course)
• Study plan for Ukrainian refugees (elective course)
• Master specialization System Programming, in Czech, version from 2023 (elective course)
• Master specialization Computer Science, in Czech, 2023 (elective course)