Julia for optimization and learning
- Garant předmětu:
- Lukáš Adam
- Lukáš Adam, Václav Mácha
- Lukáš Adam, Václav Mácha, Michaela Mašková
- Department of Computer Science
Julia programming language is increasingly known by the community for its suitability in the field of numerical calculations. The course consists of two parts. The first part presents the basics of Julia. The second part introduces mathematical optimization and its application in machine learning, statistics and optimal control of differential equations. While the first part shows the individual concepts of Julia, the second part combines them into longer logical sections of code. We explain each application theoretically. Students are encouraged both to write simple functions by themselves and compare them with already existing packages. The course ends with a final project. Students can either choose a topic connected to their theses or join a Kaggle competition with real data.
This course is also part of the inter-university programme prg.ai Minor. It pools the best of AI education in Prague to provide students with a deeper and broader insight into the field of artificial intelligence. More information is available at https://prg.ai/minor.
The course assumes only knowledge of basic programming and linear algebra.
- Syllabus of lectures:
8.Introduction to optimization. Optimality conditions for optimization with and without constraints.
9.Introduction to linear regression and classification problems. Closed-form solutions and iterative methods.
10.Introduction to neural networks. Types of neural networks. Individual layers. Overfitting.
11.Convolutional layers, neural network structure, stochastic gradient descent.
12.Selected parts of statistics. Regularized linear regression. Spectral decomposition. Monte Carlo sampling. Hypothesis testing. Generalized linear models.
13.Introduction to ordinary differential equations. Relation with spectral decomposition. Optimal control as a combination of ODEs and optimization.
- Syllabus of tutorials:
1. Introduction to Julia, advantages and disadvantages compared to Matlab, Python and R. Declarations of variables, basic numerical types, numerical and logical operators. Text strings.
2. Vectors, matrices and multidimensional arrays: basic properties, indexing, application of functions by elements. Other data structures and their properties: Tuple, NamedTuple, Dict.
3. If-elseif-else statements and ternary operator. Simple and nested for/while loops. Basic use of iterators and generators. Overview of the most used iterators: zip, eachcol, eachrow, enumerate. Scopes in loops.
4. Function declarations, mandatory, positional and keyword arguments. Functions as a set of methods. Calling methods according to the number of arguments and their type (multiple-dispatch). Local and global workspace.
5. Overview of the most used standard libraries: LinearAgebra, Statistics, Random. Useful packages: Plots.jl for creating graphs, DataFrames.jl for working with tabular data. Packages for interaction with other languages: PyCall, RCall.
6. Compound types and creating a logical hierarchy using abstract types. Parametric types. Internal and external constructors. Definition of specialized methods for complex and abstract types.
7. Organization of code into scripts and modules. Function import and extending existing code. Organization of installed packages into separate environments. Julia package creation.
8. Gradient visualization. Gradient descent method and step length selection. Projected gradients.
9. Data preparation and visualization. Linear and logistic regression training. Predictions.
10. Data preparation. Implementation of a simple neural network, including gradient calculation and training.
11. Creating neural networks using the Flux.jl package. Working on GPU.
12. Implementation of LASSO and ridge regression according to iterative methods for optimization. Analysis of the required number of samples for good approximation quality. The Distributions.jl, HypothesisTests.jl, and GLM.jl packages.
13. Hand-written implementation of the wave equation. Package ODE.jl.
- Study Objective:
The course aims to teach students to program in Julia. At the same time, we will show the use of Julia in applied fields.
- Study materials:
1.Online skripts https://bit.ly/JuliaML
2.Julia documentation. https://docs.julialang.org/en/v1/manual/documentation/index.html
3.Kochenderfer, M. J. and Wheeler, T. A. Algorithms for optimization. MIT Press, 2019. https://algorithmsbook.com/optimization/
4.Lauwens, B. and Downey, A. B. Think Julia: how to think like a computer scientist. O'Reilly Media, 2019. https://benlauwens.github.io/ThinkJulia.jl/latest/book.html
5.Goodfellow, I. and Bengio, Y. and Courville, A. Deep Learning. MIT Press, 2016. https://www.deeplearningbook.org/
- Further information:
- Time-table for winter semester 2023/2024:
Mon Tue Wed ThuroomKN:E-311
Lab K310 LinuxroomKN:E-126
Trnkova posluchárna K5roomKN:E-311
Lab K310 Linux
- Time-table for summer semester 2023/2024:
- Time-table is not available yet
- The course is a part of the following study plans: