Probability, Statistics, and Theory of Information

Přednášející:

Kateřina Helisová

Cvičící:

Kateřina Helisová

Předmět zajišťuje:

katedra matematiky

Anotace:

Basics of probability theory, mathematical statistics, information theory, a coding. Includes descriptions of probability, random variables and their distributions, characteristics and operations with random variables. Basics of mathematical statistics: Point and interval estimates, methods of parameters estimation and hypotheses testing, least squares method. Basic notions and results of the theory of Markov chains. Shannon entropy, mutual and conditional information, types of codes. Correspondence between entropy and the optimal code length. Information channels and their capacity, compression.

Požadavky:

Linear Algebra, Calculus, Discrete Mathematics

Osnova přednášek:

1. Basic notions of probability theory. Random variables and their description.

2. Characteristics of random variables. Random vector, independence, conditional probability, Bayes formula.

3. Operations with random variables, mixture of random variables. Chebyshev inequality. Law of large numbers. Central limit theorem.

4. Basic notions of statistics. Sample mean, sample variance.

5. Method of moments, method of maximum likelihood. EM algorithm.

6. Interval estimates of mean and variance. Hypotheses testing.

7. Goodness-of-fit tests, tests of correlation, non-parametic tests.

8. Applications in decision-making under uncertainty and pattern recognition. Least squares method.

9. Discrete random processes. Stationary processes. Markov chains.

10. Classification of states of Markov chains. Overview of applications.

11. Shannon's entropy of a discrete distribution and its axiomatical formulation. Theorem on minimal and maximal entropy. Conditional entropy. Chain rule. Subadditivity. Entropy of a continuous variable.

12. Fano's inequality. Information of message Y in message X. Codes, prefix codes, nonsingular codes. Kraft-MacMillan's inequality.

13. Estimation of the average codelength by means of entropy. Huffman codes. Data compression using the law of large numbers. Typical messages. Entropy speed of stationary sources.

14. Information channel and its capacity. Basic types of information channels. Shannon's coding theorem. Universal compression. Ziv-Lempel codes.

Osnova cvičení:

1. Elementary probability. Random variables and their description.

2. Mean and variance of random variables. Unary operations with random variables.

3. Random vector, joint distribution.

4. Binary operations with random variables. Mixture of random variables. Central limit theorem.

5. Sample mean, sample variance. Method of moments, method of maximum likelihood.

6. Interval estimates of mean and variance.

7. Hypotheses testing.

8. Least squares method.

9. Goodness-of-fit tests.

10. Discrete random processes. Stationary processes. Markov chains.

11. Shannons's entropy of a discrete distribution and its axiomatical formulation. Theorem on minimal and maximal entropy. Conditional entropy. Chain rule. Subadditivity. Entropy of a continuous variable.

12. Fano's inequality. Information of message Y in message X. Codes, prefix codes, nonsingular codes. Kraft-MacMillan's inequality.

13. Estimation of the average codelength by means of entropy. Huffman codes. Data compression using the law of large numbers. Typical messages. Entropy speed of stationary sources.

14. Information channel and its capacity. Basic types of information channels. Shannon's coding theorem. Universal compression. Ziv-Lempel codes.

Cíle studia:

Zvládnutí základů teorie pravděpodobnosti a jejich využití pro statistické odhady a testy.

Využití Markovových řetězců pro modelování.

Základní pojmy teorie informace.

Studijní materiály:

[1] Papoulis, A.: Probability and Statistics, Prentice-Hall, 1990.

[2] Stewart W.J.: Probability, Markov Chains, Queues, and Simulation: The Mathematical Basis of Performance Modeling. Princeton University Press 2009.

[3] David J.C. MacKay: Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003.

Poznámka:

Podmínkou získání zápočtu je aktivní účast na cvičeních a vypracování Rozsah výuky v kombinované formě studia: 28p+6s

Další informace:

http://math.feld.cvut.cz/helisova/01pstimfe.html

Pro tento předmět se rozvrh nepřipravuje

Předmět je součástí následujících studijních plánů:

• Cybernetics and Robotics - Robotics (povinný předmět programu)
• Cybernetics and Robotics - Senzors and Instrumention (povinný předmět programu)
• Cybernetics and Robotics - Systems and Control (povinný předmět programu)
• Electrical Engineering, Power Engineering and Management - Applied Electrical Engineering (volitelný předmět)
• Electrical Engineering, Power Engineering and Management - Electrical Engineering and Management (volitelný předmět)
• Communications, Multimedia and Electronics - Communication Technology (volitelný předmět)
• Communications, Multimedia and Electronics - Multimedia Technology (volitelný předmět)
• Communications, Multimedia and Electronics - Applied Electronics (volitelný předmět)
• Communications, Multimedia and Electronics - Network and Information Technology (volitelný předmět)
• Open Informatics - Computer Systems (povinný předmět programu)
• Open Informatics - Computer and Information Science (povinný předmět programu)
• Open Informatics - Software Systems (povinný předmět programu)
• Electrical Engineering, Power Engineering and Management (volitelný předmět)
• Communications, Multimedia and Electronics (volitelný předmět)
• Cybernetics and Robotics (povinný předmět programu)
• Open Informatics (povinný předmět programu)
• Communications, Multimedia and Electronics - Communications and Electronics (volitelný předmět)