Summary:

Conventional photo and video cameras were constructed to capture a view of the world that is similar to the view we capture of the world. It has been found out that these pinhole cameras are not necessarily the optimal cameras for processing visual information using a machine. Inspired by nature's task-specific eye design, we define a framework for camera design with regard to 3D motion estimation.

Main Menu

Topic Menu

Sub-Topic Menu

A Framework for Camera Design

When we think about vision, we usually think of interpreting the images taken by (two) eyes, such as our own, that is images acquired by planar eyes. But these are clearly not the only eyes that exist; the biological world reveals a large variety of designs. An eye or camera is a mechanism that forms images by focusing light onto a light sensitive surface (retina, film, CCD array, etc.). Different eyes or cameras are obtained by controlling three elements:

the geometry of the surface
the geometric distribution and optical properties of photoreceptors, and
the way light is collected and projected onto the surface (single or multiple lenses, tubes as in compound eyes)

Evolutionary considerations tell us that the design of a system's eye is related to the visual tasks the system has to solve. The way images are acquired determines how difficult it is to perform a task and since systems have to cope with limited resources, their eyes should be designed to optimize subsequent image processing as it relates to particular tasks.

We can model a generalized camera as a combination of a filter and sampling pattern in the space of light rays. The filter models the effects of the optical system and the sampling pattern that is determined by the by geometric properties of the camera. Such a model allows us to phrase the problem of camera design in terms of finding the filter and sampling pattern in light ray space that will optimally facilitate the task at hand.

A Metric for Eye Design

To evaluate and compare different eye designs in a scientific sense by using mathematical considerations we chose the recovery of descriptions of space-time models from image sequences as our problem. More specifically, we want to determine how we ought to collect images of a (dynamic) scene to best recover the scene's shapes and actions from video sequences. This problem has wide implications for a variety of applications not only in vision and recognition, but also in navigation, virtual reality, tele-immersion, and graphics. At the core of this capability is the celebrated module of structure from motion, and so our question becomes: What eye should we use, for collecting video, so that we can subsequently facilitate the structure from motion problem in the best possible way?

By examining the differential structure of the space of time varying light rays (as described in the framework of plenoptic video geometry), we relate different known and new camera models to the spatio-temporal structure of the observed scene.