Probability, Statistics, and Theory of Information

Podmínkou zápisu předmětu je, že student získal v předchozích semestrech zápočet z následujících předmětů:

Calculus (AE4B01MA2)

Přednášející:

Mirko Navara

Cvičící:

Mirko Navara

Předmět zajišťuje:

katedra matematiky

Anotace:

Basics of probability theory, mathematical statistics, information theory, a coding. Includes descriptions of probability, random variables and their distributions, characteristics and operations with random variables. Basics of mathematical statistics: Point and interval estimates, methods of parameters estimation and hypotheses testing, least squares method. Basic notions and results of the theory of Markov chains. Shannon entropy, mutual and conditional information, types of codes. Correspondence between entropy and the optimal code length. Information channels and their capacity, compression.

Požadavky:

Linear Algebra, Calculus, Discrete Mathematics

Osnova přednášek:

1. Basic notions of probability theory. Random variables and their description.

2. Characteristics of random variables. Random vector, independence, conditional probability, Bayes formula.

3. Operations with random variables, mixture of random variables. Chebyshev inequality. Law of large numbers. Central limit theorem.

4. Basic notions of statistics. Sample mean, sample variance.

5. Method of moments, method of maximum likelihood. EM algorithm.

6. Interval estimates of mean and variance. Hypotheses testing.

7. Goodness-of-fit tests, tests of correlation, non-parametic tests.

8. Applications in decision-making under uncertainty and pattern recognition. Least squares method.

9. Discrete random processes. Stationary processes. Markov chains.

10. Classification of states of Markov chains. Overview of applications.

11. Shannon's entropy of a discrete distribution and its axiomatical formulation. Theorem on minimal and maximal entropy. Conditional entropy. Chain rule. Subadditivity. Entropy of a continuous variable.

12. Fano's inequality. Information of message Y in message X. Codes, prefix codes, nonsingular codes. Kraft-MacMillan's inequality.

13. Estimation of the average codelength by means of entropy. Huffman codes. Data compression using the law of large numbers. Typical messages. Entropy speed of stationary sources.

14. Information channel and its capacity. Basic types of information channels. Shannon's coding theorem. Universal compression. Ziv-Lempel codes.

Osnova cvičení:

1. Elementary probability. Random variables and their description.

2. Mean and variance of random variables. Unary operations with random variables.

3. Random vector, joint distribution.

4. Binary operations with random variables. Mixture of random variables. Central limit theorem.

5. Sample mean, sample variance. Method of moments, method of maximum likelihood.

6. Interval estimates of mean and variance.

7. Hypotheses testing.

8. Least squares method.

9. Goodness-of-fit tests.

10. Discrete random processes. Stationary processes. Markov chains.

11. Shannons's entropy of a discrete distribution and its axiomatical formulation. Theorem on minimal and maximal entropy. Conditional entropy. Chain rule. Subadditivity. Entropy of a continuous variable.

12. Fano's inequality. Information of message Y in message X. Codes, prefix codes, nonsingular codes. Kraft-MacMillan's inequality.

13. Estimation of the average codelength by means of entropy. Huffman codes. Data compression using the law of large numbers. Typical messages. Entropy speed of stationary sources.

14. Information channel and its capacity. Basic types of information channels. Shannon's coding theorem. Universal compression. Ziv-Lempel codes.

Cíle studia:

Zvládnutí základů teorie pravděpodobnosti a jejich využití pro statistické odhady a testy.

Využití Markovových řetězců pro modelování.

Základní pojmy teorie informace.

Studijní materiály:

[1] David J.C. MacKay: Information Theory, Inference, and Learning

Algorithms, Cambridge University Press, 2003.

[2] T.M.Cover and J.Thomson: Elements of information theory, Wiley,

1991.

[3] Papoulis, A.: Probability and Statistics, Prentice-Hall, 1990.

Mood, A.M., Graybill, F.A., Boes, D.C.: Introduction to the Theory of

Statistics. 3rd ed., McGraw-Hill, 1974.

Poznámka:

Podmínkou získání zápočtu je aktivní účast na cvičeních a vypracování Rozsah výuky v kombinované formě studia: 28p+6s

Rozvrh na zimní semestr 2011/2012:

Rozvrh není připraven

Rozvrh na letní semestr 2011/2012:

Rozvrh není připraven

Předmět je součástí následujících studijních plánů:

• Cybernetics and Robotics - Robotics (povinný předmět programu)
• Cybernetics and Robotics - Senzors and Instrumention (povinný předmět programu)
• Cybernetics and Robotics - Systems and Control (povinný předmět programu)
• Open Informatics - Computer Systems (povinný předmět programu)
• Open Informatics - Computer and Information Science (povinný předmět programu)
• Open Informatics - Software Systems (povinný předmět programu)
• Cybernetics and Robotics (povinný předmět programu)
• Open Informatics (povinný předmět programu)