Neuromorphic Algorithms for Modeling Visual Perception

Course guarantor:

Lecturer:

Tutor:

Supervisor:

Department of Cybernetics

Synopsis:

The course focuses on neuromorphic sensing and introduces neuromorphic computing for robotics and computer vision

applications, with special emphasis on event-based vision and spiking neural networks (SNNs) for vision-based tasks,

exploring the differences with classical ANNs.The final part of the course provides an overview of spiking neural networks,

covering key computational principles and exploring potential implementations and applications.

Requirements:

Syllabus of lectures:

Motivated by the development of efficient, brain-inspired perception and control systems, this course introduces

neuromorphic technology and its specific challenges and opportunities:

(i) Neuromorphic sensing and event-based cameras (e.g., silicon retina, gain modulation, ED: visual odometry, motion

detection, visual attention, stereopsis, active vision), (ii) Other event-driven sensory modalities, including touch and

olfaction, (iii) An overview of neuromorphic computing architectures and their applications in robotics.

By taking this course, students (ideally ranging from 15 to max 25 participants) will gain a general overview of event-

based sensing and neuromorphic computing for computer vision/robotics, including core components, processing

approaches, and learning rules. They will also gain insight into real-world robotic applications, including the

Neuromorphic iCub and developments from companies such as BrainChip, SynSense, Innatera, and Intel. Comparisons

with current standard approaches will be drawn where appropriate; however, this course is not intended to serve as

a comprehensive introduction to conventional methods.

Lectures with an overview and comparison to classical frame-based approaches.

1. Neuromorphic sensing and computing - Carver Mead, Misha Mawohald & Rodney Douglas work at Caltech

University (Infineon - iNivation - Prophesee); motivation, history, and relevance

2. Neuromorphic sensing: The retina - contrast sensitivity, Weber-Fechner law, Mach Bands, lateral inhibition, gain

modulation mechanisms

3. Neuromorphic sensing: The human visual system - retinotopic mapping, retino-cortical processing, cortical

magnification, cortical processing (LGN, V1, V2)

4. Event-based cameras - Visual odometry (motion estimation, contrast maximisation, SLAM); comparison with

current classical approaches

5. Event-based cameras - Navigation and motion detection problems (Reichardt - sEMD)

6. Event-based cameras - Stereopsis, depth perception (Osswald, Marr, Poggio, Mahowald)

7. Event-driven sensing - Event-based sensing (vision, touch, cochlea); Event-based cochlea; Olfactory and smell

circuits & applications

8. Spiking Neural Networks - An overview with a comparison with classical ANNs, analog vs digital (spike

detection and analysis (MFR, ISI), temporal coding and learning rules)

Syllabus of tutorials:

Outline of labs: (Github repo laboratories: https://github.com/GiuliaDAngelo/CTU-EDNeuromorphic)

1: Event-based visualisation and events representations methods.

2: Event-based Object Motion Sensitivity (OMS) and Visual Attention

3: Journal Club - Recent literature and interesting problems.

4: Projects brainstorming based on lectures and the Journal Club (list of possible projects available for the students).

13: Presentation of the projects/demos to the class.

Study Objective:

Study materials:

· Introductory material summary: https://github.com/GiuliaDAngelo/CTU-EDNeuromorphic

· Mead, Carver. „Neuromorphic electronic systems.“ Proceedings of the IEEE 78.10 (1990): 1629-1636.

· Indiveri, Giacomo, and Rodney Douglas. „Neuromorphic vision sensors.“ Science 288.5469 (2000): 1189-1190.

· Bartolozzi, Chiara, Giacomo Indiveri, and Elisa Donati. „Embodied neuromorphic intelligence.“ Nature

communications 13.1 (2022): 1024.

· Gallego, Guillermo, et al. „Event-based vision: A survey.“ IEEE Transactions on Pattern Analysis and Machine

Intelligence 44.1 (2020): 154-180.

· DAngelo, Giulia, et al. „Event driven bio-inspired attentive system for the iCub humanoid robot on SpiNNaker.“

Neuromorphic Computing and Engineering 2.2 (2022): 024008.

Note:

Further information:

No time-table has been prepared for this course

The course is a part of the following study plans:

• Open Informatics - Computer Vision (compulsory elective course)