Neural Networks, Machine Learning and Randomness

Course guarantor:

Lecturer:

Tutor:

Supervisor:

Department of Applied Mathematics

Synopsis:

Stochastic methods, i.e. methods based on randomness, are extremely important for the construction and training of neural networks as well as a number of other machine learning models. The course „Neural networks, machine learning and randomness“ will discuss in sufficient depth a number of specific types of neural networks that rely substantially on randomness, as well as a number of specific stochastic methods for neural networks and machine learning. In the final two topics, it explains the general stochastic approach to training neural networks and shows that, in addition to the use of randomness in neural networks and machine learning, machine learning models, including neural networks, are used in one of the most important applications of randomness stochastic optimization methods, which include e.g. popular evolutionary algorithms.

Requirements:

Syllabus of lectures:

1. Recalling concepts known from earlier courses

Artificial neural networks, signal transmission, network architecture. The most common types of neural networks. General models in machine learning. Model training. Model selection. Feature selection. Measures of model quality. Interpretability and explainability. Supervised, unsupervised, and reinforcement learning. The most common supervised learning methods. Rule learning. Clustering. Random variables and random processes. Probability distributions and moments. The Bayesian approach.

2. Artificial neural networks based on randomness

ELM (Extreme Learning Machine) networks. Training of ELM networks, the optimization problem for training ELM networks. ELM networks and random projection. Randomized convolutional neural networks. ESN (Echo State Network) networks. Evolution of activity in ESN networks. ESN networks with forbidden connections. Bayesian neural networks (BNN). Prior probability distributions in BNNs. Prediction and estimation in BNNs. BNNs with stochastic activation, BNNs with limited stochasticity, hierarchical BNNs.

3. Stochastic methods for artificial neural networks

Dropout, Bernoulli dropout, properties of Bernoulli dropout. Dropout and network training, dropout and regularization. Dropout and ensembles of neural networks. Dropout in Boltzmann machines and in linear regression. Gaussian dropout. Stochastic gradient. Stochastic gradient descent (SGD). Assumptions and strategies of SGD. Approximation of posterior probability distributions, component-wise approximation.

4. Stochastic methods for machine learning

Observable and latent variables. Markov chain Monte Carlo (MCMC) methods. MCMC estimation of posterior distributions of latent variables. The MetropolisHastings algorithm. Variational inference (VI). VI estimation of posterior distributions of latent variables. Lower bound of the marginal distribution of observable variables (ELBO). Combining VI with MCMC. VI estimates in generative models, deep Kalman filters.

5. General stochastic approach to artificial neural networks

Assumptions of the general stochastic approach. Spaces of random vectors. Mean-based learning and sampling-based learning. Specific aspects of mean-based learning under a quadratic loss function. Strong law of large numbers in neural network learning, assumptions and statements. Central limit theorem in neural network learning, assumptions and statements. Connections with testing whether connection weights are zero, applications in network pruning.

6. Machine learning and neural networks as support for stochastic optimization

Stochastic optimization algorithms, evolutionary algorithm CMA-ES (Covariance Matrix Adaptation Evolution Strategy). Drawbacks of stochastic optimization for black-box objective functions with expensive evaluations. Surrogate modeling for black-box optimization. Choosing between evaluating the black-box function and the surrogate model. Surrogate models based on artificial neural networks, Gaussian processes, random forests, and ordinal regression.

Syllabus of tutorials:

1, Basics of machine learning in Python, NumPy, Pandas, Seaborn and PyTorch, generating random numbers, visualizing distributions.

2, Algorithms for automatic differentiation, simple neural networks.

3. Supervised, unsupervised, reinforcement, self-supervised learning. Linear regression, cost functions, visualizing the regression line, evaluating simple models.

4. Logistic regression, binary classification, cost functions. Implementation and visualization of decision strategies. Perceptron, its learning algorithm and implementation.

5. Splitting data into a training and testing set, overfitting, bias-variance trade-off. L1, L2 regularization, early stopping, dropout, batching. Cross-validation, its variants and use. Double descent.

6. Bayes theorem, maximum likelihood estimation vs. maximum a-posteriori estimation. The Naive Bayes classifier vs. k-Nearest Neighbor classifier.

Study Objective:

Systematic explanation of connections between stochastic methods and training of neural networks or other machine learning models.

Study materials:

1. I. Goodfellow, Y. Bengio, A. Courville. Deep Learning. MIT, Boston.

2. Z.H. Zhou. Machine Learning. Springer Nature, Singapore.

Note:

The course is presented in Czech language. Additional course materials are available at https://courses.fit.cvut.cz/NI-NMS.

Further information:

https://courses.fit.cvut.cz/NI-NMS

No time-table has been prepared for this course

The course is a part of the following study plans:

• Master specialization Computer Security, in Czech, 2020 (elective course)
• Master specialization Design and Programming of Embedded Systems, in Czech, 2020 (elective course)
• Master specialization Computer Systems and Networks, in Czech, 2020 (elective course)
• Master specialization Management Informatics, in Czech, 2020 (elective course)
• Master specialization Software Engineering, in Czech, 2020 (elective course)
• Master specialization Web Engineering, in Czech, 2020 (elective course)
• Master specialization Knowledge Engineering, in Czech, 2020 (elective course)
• Mgr. programme, for the phase of study without specialisation, ver. for 2020 and higher (elective course)
• Study plan for Ukrainian refugees (elective course)
• Master specialization System Programming, in Czech, version from 2023 (elective course)
• Master specialization Computer Science, in Czech, 2023 (elective course)
• Quantum Informatics (elective course)
• Master specialization Computer Security, in Czech, 2026 (elective course)
• Master specialization Computer Systems and Networks, in Czech, 2026 (elective course)
• Master specialization Computer Science, in Czech, 2026 (elective course)
• Master specialization Programming Languages, in Czech, 2026 (elective course)
• Master specialization Artificial Intelligence, in Czech, 2026 (elective course)
• Master programme, for the phase of study without specialisation, ver. for 2026 and higher (elective course)