Information theory

Lecturer:

Tomáš Hobza (gar.)

Tutor:

Tomáš Hobza (gar.)

Supervisor:

Department of Mathematics

Synopsis:

Information theory explores the fundamental limits of the representation and transmission of information. We will focus on the definition and implications of (information) entropy, the source coding theorem, and the channel coding theorem. These concepts provide a vital background for researchers in the areas of data compression, signal processing, controls, and pattern recognition.

Requirements:

Basic course of Calculus and Probability (in the extent of the courses 01MAA3, 01MAA4 and 01PRA1 held at the FNSPE CTU in Prague).

Syllabus of lectures:

1. Information source and entropy, joint and conditional entropy, information divergence, informations and their relation to entropy

2. Jensen inequality and the methods of convex analysis, sufficient statistics and data processing theorem

3. Fano inequality and Cramér-Rao inequality, asymptotic equipartition property of memoryless sources

4. Entropy rates of information sources, stationary and Markov sources

5. Data compression, Kraft inequality for instantaneous and uniquely decodable codes, Huffman codes

6. Capacity of noisy channels, Shannon theorem about transmissibility of a source through a channel

Syllabus of tutorials:

Study Objective:

Knowledge:

Basic notions and principles of information theory.

Skills:

Ability of application of acquired knowledge to solution of practical problems such as finding optimal Huffman codes, calculation of stacionary distribution of Markov chains, calculation of information channel capacity.

Study materials:

Key references:

[1] T. Cover and J. Thomas: Elements of information theory, Wiley, 1994

Recommended references:

[2] D. J. C. Mackay: Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003

Note:

Time-table for winter semester 2011/2012:

Time-table is not available yet

Time-table for summer semester 2011/2012:

Time-table is not available yet

The course is a part of the following study plans: