Neural Networks, Machine Learning and Randomness

Course guarantor:

Lecturer:

Tutor:

Supervisor:

Department of Applied Mathematics

Synopsis:

Stochastic methods, i.e. methods based on randomness, are extremely important for the construction and training of neural networks as well as a number of other machine learning models. The course „Neural networks, machine learning and randomness“ will discuss in sufficient depth a number of specific types of neural networks that rely substantially on randomness, as well as a number of specific stochastic methods for neural networks and machine learning. In the final two topics, it explains the general stochastic approach to training neural networks and shows that, in addition to the use of randomness in neural networks and machine learning, machine learning models, including neural networks, are used in one of the most important applications of randomness stochastic optimization methods, which include e.g. popular evolutionary algorithms.

Requirements:

Syllabus of lectures:

1. Recalling concepts known from other courses

Artificial neural networks, signal transmission, network architecture. Best known types of neural networks. General models in machine learning. Model training. Model selection. Selection of features. Measures of model quality. Interpretability and explainability. Supervised and unsupervised learning, reinforcement learning. Best known supervised learning methods. Rules learning. Clustering. Random variables and random processes. Probability distributions and moments. Bayesian approach.

2. Artificial neural networks based on randomness

ELM (extreme learning machine) networks. Learning ELM networks, the optimization task for learning ELM networks. ELM networks and random projection. Randomized convolutional neural networks. ESN (echo state network) networks. Evolution of activity in ESN networks. ESN networks with inhibit connections. Bayesian Neural Network (BNN). A priori probability distribution in a BNN. Predictions and estimates in a BNN. BNN with stochastic activation, BNN with bounded stochasticity, hierarchical BNN.

3. Stochastic methods for artificial neural networks

Dropout, Bernoulli dropout, properties of Bernoulli dropout. Dropout and network learning, dropout and regularization. Dropout and neural network teams. Dropout in Boltzmann machines and in linear regression. Gaussian dropout. Stochastic gradient. Stochastic gradient descent (SGD). Assumptions and strategies of the SGD method. Approximation of posterior probability distribution, approximation by components.

4. Stochastic methods for machine learning

Observable and latent variables. Monte Carlo Markov chain (MCMC). MCMC estimation of the posterior distribution of latent variables. Metropolis-Hastings algorithm. Variational inference (VI). VI estimation of the posterior distribution of latent variables. Evidence lower bound. Combining VI with MCMC. VI estimates in generative models, deep Kalman filters.

5. General stochastic approach to artificial neural networks

Assumptions of the general stochastic approach. Spaces of random vectors. Mean-based learning and random-based learning. Specificity of mean-based learning under quadratic error function. Strong law of large numbers for neural network learning, assumptions and assertions. Central Limit Theorem for Learning Neural Networks, assumptions and assertions. Connection with testing the zeroness of connection weights, use in network pruning.

6. Machine learning and neural networks as support for stochastic optimization

Stochastic optimization algorithms, the evolutionary algorithm CMA-ES (covariance matrix adaptation evolution strategy). Disadvantage of stochastic optimization for black-box objective functions with costly evaluation. Surrogate modeling for black-box optimization. Choice of evaluation between black-box function and model. Surrogate models based on artificial neural networks, Gaussian processes, random forests and ordinal regression.

Syllabus of tutorials:

- Basics of machine learning in Python, NumPy, Pandas, Seaborn and PyTorch, generating random numbers, visualizing distributions.

- Algorithms for automatic differentiation, simple neural networks.

- Supervised, unsupervised, reinforcement, self-supervised learning. Linear regression, cost functions, visualizing the regression line, evaluating simple models.

- Logistic regression, binary classification, cost functions. Implementation and visualization of decision strategies. Perceptron, its learning algorithm and implementation.

- Splitting data into a training and testing set, overfitting, bias-variance trade-off. L1, L2 regularization, early stopping, dropout, batching. Cross-validation, its variants and use. Double descent.

- Bayes theorem, maximum likelihood estimation vs. maximum a-posteriori estimation. The Naive Bayes classifier vs. k-Nearest Neighbor classifier.

Study Objective:

Systematic explanation of connections between stochastic methods and training of neural networks or other machine learning models.

Study materials:

1. I. Goodfellow, Y. Bengio, A. Courville. Deep Learning. MIT, Boston.

2. Z.H. Zhou. Machine Learning. Springer Nature, Singapore.

Note:

The course is presented in Czech language. Additional course materials are available at https://courses.fit.cvut.cz/NI-NMS.

Further information:

https://courses.fit.cvut.cz/NI-NMS

No time-table has been prepared for this course

The course is a part of the following study plans:

• Master specialization Computer Security, in Czech, 2020 (elective course)
• Master specialization Design and Programming of Embedded Systems, in Czech, 2020 (elective course)
• Master specialization Computer Systems and Networks, in Czech, 202 (elective course)
• Master specialization Management Informatics, in Czech, 2020 (elective course)
• Master specialization Software Engineering, in Czech, 2020 (elective course)
• Master specialization System Programming, in Czech, version from 2020 (elective course)
• Master specialization Web Engineering, in Czech, 2020 (elective course)
• Master specialization Knowledge Engineering, in Czech, 2020 (elective course)
• Master specialization Computer Science, in Czech, 2020 (elective course)
• Mgr. programme, for the phase of study without specialisation, ver. for 2020 and higher (elective course)
• Study plan for Ukrainian refugees (elective course)
• Master specialization System Programming, in Czech, version from 2023 (elective course)
• Master specialization Computer Science, in Czech, 2023 (elective course)