Neural Image Synthesis

Course guarantor:

Lecturer:

Tutor:

Supervisor:

Department of Computer Graphics and Interaction

Synopsis:

This course introduces modern techniques for image synthesis and the representation of virtual scenes using neural networks in a comprehensible way. It focuses, in particular, on methods for rendering 3D models into 2D images. Students will first learn about transformational and generative methods of 2D image creation, 2.5D depth representation of materials, and full 3D representation of virtual scenes based on neural radiation fields. Then, methods to accelerate scene rendering using neural networks, such as radiance caching, importance sampling, and image de-noising, will be discussed. Each lecture motivates the problem being discussed, outlines its solution using classical computer graphics methods, and then discusses in detail the solution based on the use of neural networks. The aim of the course is to provide an understanding of when it makes sense to use classical approaches versus neural approaches. In addition to theoretical knowledge, students will be introduced to the practical applications of neural networks, for example, in the film industry. The course is suitable not only for computer graphics students but also as a complement to the teaching of machine perception.

Requirements:

Basic knowledge of neural networks is required to successfully complete the course. It is also beneficial if students are already familiar with methods of image representation and synthesis using ray tracing and path tracing. For labs, it is advisable to have knowledge of the Monte Carlo method, ray marching, and material representation using bidirectional functions (BRDF).

Syllabus of lectures:

1. Introduction

2. Applications of Convolutional Neural Networks (CNN)

3. Applications of Generative Adversarial Networks (GAN)

4. Applications of Diffusion Models

5. Neural Denoising

6. Neural Materials 1

7. Neural Materials 2

8. Neural Fields (NeRF) 1

9. Neural Fields (NeRF) 2

10. Neural Light Transport 1

11. Neural Light Transport 2

12. Generative Scene Synthesis

13. Machine Learning Methods vs. Traditional Computer Graphics

14. Reserved

Syllabus of tutorials:

1. Introduction to Labs

2. Example-based Video Stylization (U-net)

3. (assignment submission)

4. Face Synthesis and Editing (StyleGAN)

5. (assignment submission)

6. Diffusion Image Analogies (SD + CLIP)

7. (assignment submission)

8. Neural Materials

9. (work on the assignment)

10. (assignment submission)

11. Neural Fields

12. (work on the assignment)

13. (assignment submission)

14. Reserved

Study Objective:

Study materials:

1. Understanding Deep Learning, Simon J.D. Prince, The MIT Press (https://udlbook.github.io/udlbook/)

2. Generative Deep Learning (2nd Edition), David Foster, O'Reilly Media, Inc.

3. Physically Based Rendering: From Theory to Implementation, Pharr et al. (https://www.pbrt.org/)

4. Neural Fields for Visual Computing, ACM SIGGRAPH 2023 Course, Takikawa et al. (https://neuralfields.cs.brown.edu/)

Note:

Further information:

No time-table has been prepared for this course

The course is a part of the following study plans:

• Open Informatics - Computer Graphics (compulsory course of the specialization)