Logo ČVUT
CZECH TECHNICAL UNIVERSITY IN PRAGUE
STUDY PLANS
2020/2021

Information Theory

Login to KOS for course enrollment Display time-table
Code Completion Credits Range Language
01TIN ZK 2 2+0 Czech
Lecturer:
Tomáš Hobza (guarantor)
Tutor:
Supervisor:
Department of Mathematics
Synopsis:

Information theory explores the fundamental limits of the representation and transmission of information. We will focus on the definition and implications of (information) entropy, the source coding theorem, and the channel coding theorem. These concepts provide a vital background for researchers in the areas of data compression, signal processing, controls, and pattern recognition.

Requirements:

Basic course of Calculus and Probability (in the extent of the courses 01MAA3, 01MAA4 and 01MIP held at the FNSPE CTU in Prague).

Syllabus of lectures:

1. Information source and entropy, joint and conditional entropy, information divergence, informations and their relation to entropy

2. Jensen inequality and the methods of convex analysis, sufficient statistics and data processing theorem

3. Fano inequality and Cramér-Rao inequality, asymptotic equipartition property of memoryless sources

4. Entropy rates of information sources, stationary and Markov sources

5. Data compression, Kraft inequality for instantaneous and uniquely decodable codes, Huffman codes

6. Capacity of noisy channels, Shannon theorem about transmissibility of a source through a channel

Syllabus of tutorials:
Study Objective:

Knowledge:

Basic notions and principles of information theory.

Skills:

Ability of application of acquired knowledge to solution of practical problems such as finding optimal Huffman codes, calculation of stacionary distribution of Markov chains, calculation of information channel capacity.

Study materials:

Key references:

[1] Cover, T. M., Thomas, J. A.: Elements of information theory. John Wiley & Sons, NewYork 2012.

Recommended references:

[2] Stone, J.V.: Information Theory - A Tutorial Introduction. Sebtel Press, Sheffield 2015.

[3] Csiszár, I., Körner, J.: Information theory - coding theorems for discrete memoryless systems. Cambridge University Press, Cambridge 2016.

Note:
Time-table for winter semester 2020/2021:
Time-table is not available yet
Time-table for summer semester 2020/2021:
Time-table is not available yet
The course is a part of the following study plans:
Data valid to 2020-09-18
For updated information see http://bilakniha.cvut.cz/en/predmet11334205.html