Information Theory
Code  Completion  Credits  Range  Language 

01TIN  ZK  2  2+0  Czech 
 Lecturer:
 Tomáš Hobza (guarantor)
 Tutor:
 Supervisor:
 Department of Mathematics
 Synopsis:

Information theory explores the fundamental limits of the representation and transmission of information. We will focus on the definition and implications of (information) entropy, the source coding theorem, and the channel coding theorem. These concepts provide a vital background for researchers in the areas of data compression, signal processing, controls, and pattern recognition.
 Requirements:

Basic course of Calculus and Probability (in the extent of the courses 01MAA3, 01MAA4 and 01MIP held at the FNSPE CTU in Prague).
 Syllabus of lectures:

1. Information source and entropy, joint and conditional entropy, information divergence, informations and their relation to entropy
2. Jensen inequality and the methods of convex analysis, sufficient statistics and data processing theorem
3. Fano inequality and CramérRao inequality, asymptotic equipartition property of memoryless sources
4. Entropy rates of information sources, stationary and Markov sources
5. Data compression, Kraft inequality for instantaneous and uniquely decodable codes, Huffman codes
6. Capacity of noisy channels, Shannon theorem about transmissibility of a source through a channel
 Syllabus of tutorials:
 Study Objective:

Knowledge:
Basic notions and principles of information theory.
Skills:
Ability of application of acquired knowledge to solution of practical problems such as finding optimal Huffman codes, calculation of stacionary distribution of Markov chains, calculation of information channel capacity.
 Study materials:

Key references:
[1] Cover, T. M., Thomas, J. A.: Elements of information theory. John Wiley & Sons, NewYork 2012.
Recommended references:
[2] Stone, J.V.: Information Theory  A Tutorial Introduction. Sebtel Press, Sheffield 2015.
[3] Csiszár, I., Körner, J.: Information theory  coding theorems for discrete memoryless systems. Cambridge University Press, Cambridge 2016.
 Note:
 Timetable for winter semester 2020/2021:
 Timetable is not available yet
 Timetable for summer semester 2020/2021:
 Timetable is not available yet
 The course is a part of the following study plans:

 Matematické inženýrství (elective course)
 Aplikované matematickostochastické metody (compulsory course of the specialization)
 Matematická informatika (compulsory course of the specialization)