Information Theory

Course guarantor:

Tomáš Hobza

Lecturer:

Tomáš Hobza

Tutor:

Supervisor:

Department of Mathematics

Synopsis:

Information theory explores the fundamental limits of the representation and transmission of information. We will focus on the definition and implications of (information) entropy, the source coding theorem, and the channel coding theorem. These concepts provide a vital background for researchers in the areas of data compression, signal processing, controls, and pattern recognition.

Requirements:

Basic course of Calculus and Probability (in the extent of the courses 01MAA3, 01MAA4 and 01MIP held at the FNSPE CTU in Prague).

Syllabus of lectures:

1. Information source and entropy, joint and conditional entropy, information divergence, informations and their relation to entropy

2. Jensen inequality and the methods of convex analysis, sufficient statistics and data processing theorem

3. Fano inequality and Cramér-Rao inequality, asymptotic equipartition property of memoryless sources

4. Entropy rates of information sources, stationary and Markov sources

5. Data compression, Kraft inequality for instantaneous and uniquely decodable codes, Huffman codes

6. Capacity of noisy channels, Shannon theorem about transmissibility of a source through a channel

Syllabus of tutorials:

Study Objective:

Knowledge:

Basic notions and principles of information theory.

Skills:

Ability of application of acquired knowledge to solution of practical problems such as finding optimal Huffman codes, calculation of stacionary distribution of Markov chains, calculation of information channel capacity.

Study materials:

Key references:

[1] Cover, T. M., Thomas, J. A.: Elements of information theory. John Wiley & Sons, NewYork 2012.

Recommended references:

[2] Stone, J.V.: Information Theory - A Tutorial Introduction. Sebtel Press, Sheffield 2015.

[3] Csiszár, I., Körner, J.: Information theory - coding theorems for discrete memoryless systems. Cambridge University Press, Cambridge 2016.

Note:

Time-table for winter semester 2024/2025:

	06:00–08:0008:00–10:0010:00–12:0012:00–14:0014:00–16:0016:00–18:0018:00–20:0020:00–22:0022:00–24:00
Mon
Tue
Wed
Thu
Fri	roomTR:301 Hobza T. 10:00–11:50 (lecture parallel1) Trojanova 13 posluchárna nad schody

Time-table for summer semester 2024/2025:

Time-table is not available yet

The course is a part of the following study plans:

• Aplikovaná algebra a analýza (elective course)
• Aplikované matematicko-stochastické metody (compulsory course in the program)
• Aplikace informatiky v přírodních vědách (elective course)
• Matematické inženýrství (compulsory course in the program)
• Matematická informatika (compulsory course in the program)
• Kvantové technologie (elective course)