Logo ČVUT
CZECH TECHNICAL UNIVERSITY IN PRAGUE
STUDY PLANS
2024/2025

Information Theory

The course is not on the list Without time-table
Code Completion Credits Range Language
01TIN ZK 2 2+0 Czech
Garant předmětu:
Lecturer:
Tutor:
Supervisor:
Department of Mathematics
Synopsis:

Information theory explores the fundamental limits of the representation and transmission of information. We will focus on the definition and implications of (information) entropy, the source coding theorem, and the channel coding theorem. These concepts provide a vital background for researchers in the areas of data compression, signal processing, controls, and pattern recognition.

Requirements:

Basic course of Calculus and Probability (in the extent of the courses 01MAA3, 01MAA4 and 01MIP held at the FNSPE CTU in Prague).

Syllabus of lectures:

1. Information source and entropy, joint and conditional entropy, information divergence, informations and their relation to entropy

2. Jensen inequality and the methods of convex analysis, sufficient statistics and data processing theorem

3. Fano inequality and Cramér-Rao inequality, asymptotic equipartition property of memoryless sources

4. Entropy rates of information sources, stationary and Markov sources

5. Data compression, Kraft inequality for instantaneous and uniquely decodable codes, Huffman codes

6. Capacity of noisy channels, Shannon theorem about transmissibility of a source through a channel

Syllabus of tutorials:
Study Objective:

Knowledge:

Basic notions and principles of information theory.

Skills:

Ability of application of acquired knowledge to solution of practical problems such as finding optimal Huffman codes, calculation of stacionary distribution of Markov chains, calculation of information channel capacity.

Study materials:

Key references:

[1] Cover, T. M., Thomas, J. A.: Elements of information theory. John Wiley & Sons, NewYork 2012.

Recommended references:

[2] Stone, J.V.: Information Theory - A Tutorial Introduction. Sebtel Press, Sheffield 2015.

[3] Csiszár, I., Körner, J.: Information theory - coding theorems for discrete memoryless systems. Cambridge University Press, Cambridge 2016.

Note:
Further information:
No time-table has been prepared for this course
The course is a part of the following study plans:
Data valid to 2024-05-23
Aktualizace výše uvedených informací naleznete na adrese https://bilakniha.cvut.cz/en/predmet11334205.html