The complete visual signal is described by a 7-dimensional function known as the plenoptic function. Today’s commercial cameras sample densely only in 2 spatial dimensions, ignoring other dimensions such as depth, wavelength, time, etc., which are critical in applications such as remote-sensing, autonomous navigation, AR/VR, etc. High data bandwidth signals sample data from multiple dimensions, and they require the device to process and store a large amount of data in real-time. This increases the design complexity and cost of the device. It becomes desirable to intelligently sample only a low data bandwidth measurement from the full signal. An associated algorithm can is used to reconstruct the original signal from the sampled data. We propose intelligent sampling and reconstruction of two high data-bandwidth signals, high-speed and light-field videos.

We explore two different techniques to sample high-speed videos as low data bandwidth signals. The first technique, known as coded-exposure imaging, temporally multiplexes several high-speed frames into a single frame of a low frame-rate video. We propose a learning-based approach to recover high-speed video from the multiplexed low frame-rate video. The second technique uses a novel neuromorphic event-based sensor. They acquire sparse temporal differences as binary events. And we propose two different techniques to reconstruct high-speed videos from these binary events.

Light-Field (LF) video is another high data bandwidth video that is challenging to acquire. We propose to use a stereo video as our low data bandwidth signal, which can be considered a sparse sample of the LF angular views. Unlike LF videos, commercial image sensors can easily acquire stereo videos as they require only 2× the bandwidth of a monocular video. We propose a self-supervised learning-based algorithm that requires only easy-to-acquire stereo videos for training