Neural Language Models

Course guarantor:

Lecturer:

Tutor:

Supervisor:

Department of Applied Mathematics

Synopsis:

In this course, students will learn the technical foundations of the Transformer architecture as well as the practical aspects of using language models. The goal of the course is to teach students how to use language models to solve problems, make informed risk assessments, and work critically with the scientific literature.

Requirements:

We assume basic knowledge of neural networks (BI-ML2, NI-MVI) and Python programming (BI-PYT).

Syllabus of lectures:

1. Introduction to natural language processing with neural networks, language modeling.

2. Transformer architecture.

3. Training language models.

4. Text generation, decoding algorithms.

5. Data and evaluation.

6. Advanced prompting methods (chain-of-thought, RAG).

7. Efficiency: quantization of models, low-rank adaptation.

8. Interpretability.

9. Discussion of scientific papers.

10. Multimodal models.

11. Application of language models, ethical and legal aspects.

12. Presentations of semestral projects.

Syllabus of tutorials:

1. Libraries for working with language models.

2. Semestral project assignments.

3. Consultation.

4. Consultation.

5. Discussion of scientific papers.

6. Presentations of semestral projects.

Study Objective:

Study materials:

1. Jurafsky, D., Martin, J. H.: Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models (3rd Edition), Online manuscript, 2025.

2. Kamath U., Keenan K., Somers G., Sorenson S.: Large Language Models: A Deep Dive, Springer Cham, 2024.

3. Vaswani, A.: Attention is all you need. Advances in Neural Information Processing Systems, 2017.

Note:

This course is presented in Czech language.

Further information:

https://courses.fit.cvut.cz/NI-NLM

No time-table has been prepared for this course

The course is a part of the following study plans:

• Master specialization Computer Security, in Czech, 2020 (elective course)
• Master specialization Design and Programming of Embedded Systems, in Czech, 2020 (elective course)
• Master specialization Computer Systems and Networks, in Czech, 202 (elective course)
• Master specialization Management Informatics, in Czech, 2020 (elective course)
• Master specialization Software Engineering, in Czech, 2020 (elective course)
• Master specialization System Programming, in Czech, version from 2020 (elective course)
• Master specialization Web Engineering, in Czech, 2020 (elective course)
• Master specialization Knowledge Engineering, in Czech, 2020 (elective course)
• Master specialization Computer Science, in Czech, 2020 (elective course)
• Mgr. programme, for the phase of study without specialisation, ver. for 2020 and higher (elective course)
• Study plan for Ukrainian refugees (elective course)
• Master specialization System Programming, in Czech, version from 2023 (elective course)
• Master specialization Computer Science, in Czech, 2023 (elective course)
• Quantum Informatics (elective course)