A dynamic vision sensor (DVS) is a neuromorphic sensor that records per-pixel intensity changes, called events, time-stamped with microsecond resolution. These events are recorded asynchronously as they occur, resulting in substantially enhanced efficiency by promptly capturing vital information. The output of an event camera differs significantly from that of conventional cameras. DVS produces an asynchronous data stream, unlike a conventional camera that captures images sequentially, frame by frame, utilizing a global shutter mechanism. So, new algorithms are necessary to address the lack of absolute brightness measurement and utilise the asynchronous nature of high temporal resolution event data.


In this seminar, we will explore strategies to address the constraints associated with traditional cameras (low dynamic range, motion blur, etc.) and develop algorithms for encoding event data and achieving robust perception using event cameras. The raw events require a preparatory step before feeding into any existing algorithms. Each event lacks inherent spatial structure and provides only limited scene-related information. To address this limitation, a widely employed approach is to aggregate specific events into a 'superframe' or arrange them within a grid-based structure, such as a voxel grid. The aggregation process enhances the signal-to-noise ratio significantly. While the invariant asynchronous event data stream is inherently compressed, one can still harness its spatial and temporal redundancies for additional compression gains. We investigate various encoding techniques suitable for event data to achieve low-level representation.


We propose an encoding framework for event data based on a deep belief autoencoder. Spike events are accumulated over a fixed duration to generate event frames, and they are subsequently segregated according to their polarity to construct superframes. The model harnesses the inherent temporal and spatial redundancies within super-frames to derive a latent code block of significantly reduced dimensions. The latent code block undergoes encoding using an entropy encoder to achieve additional lossless compression. We evaluate the framework's performance on a dataset with various scene complexity and intricate scene motions against state-of-the-art methods. It outperforms them in both compression ratio and reconstruction accuracy.


Despite its advantageous characteristics, such as low latency, high dynamic range, and minimal power consumption, humans cannot interpret event streams directly like high-quality intensity images. Additionally, one can immediately apply well-established vision algorithms to the intensity images to solve various computer vision tasks. Hence, we also introduce a fast and resource-efficient method for reconstructing intensity information from raw event streams. Our approach employs a recurrent network-based model guided by context, ensuring efficient and accurate reconstruction. The recurrent connections play a pivotal role in gradually enhancing the hidden state over time, allowing for the reuse of previously computed results. Compared to existing state-of-the-art techniques, the proposed method produces competitive results while being less computationally inexpensive. The algorithm leverages the capabilities of event-based vision to address downstream tasks requiring high-speed capture, a wide dynamic range, minimal latency, and low power consumption.


Finally, we draw attention to the inherent limitations of the existing event data processing paradigm. We also introduce prospective solutions to enhance the efficiency and robustness of well-established downstream vision applications by integrating event-based vision techniques.