Differential Plenoptic Motion Estimation

Differential Plenoptic Brightness Constancy.

Assuming that the plenoptic function in the neighbourhood of the ray parameterized by the origin x and direction r is smoothly varying, then we can develop the plenoptic function L in the neighbourhood of (x;r,t) into a Taylor series

.

Disregarding the higher-order terms, we have a linear function which relates a local change in view ray position and direction to the differential brightness structure of the plenoptic function. This allows us to use the spatio-temporal brightness derivatives of the light rays captured by an imaging surface to constrain the plenoptic ray flow, that is the change in position and orientation between rays captured by the same imaging element at consecutive time instants, by generalizing the well-known Image Brightness Constancy Constraint to the Plenoptic Brightness Constancy Constraint:

.

Differential Plenoptic Motion Constraint.

Assuming that the imaging sensor undergoes a rigid motion with instantaneous translation t and rotation around the origin of the fiducial coordinate system, we can define the plenoptic ray flow for the ray captured by the imaging element located at location x and looking in direction r as

Combining the last two equations leads to the differential plenoptic motion constraint

which is a linear constraint in the motion parameters and relates them to all the differential image information that a sensor can capture. To our knowledge, this is the first time that the temporal properties of the plenoptic function have been related to the structure from motion problem. In previous work, the plenoptic function has mostly been studied in the context of image-based rendering in computer graphics under the names light field (Levoy and Hanrahan 96) and lumigraph (Gortler etal. 96), and only the 4D subspace of the static plenoptic function corresponding to the light rays in free space was examined. The advantages of multiple centers of projection with regard to the stereo estimation problem had been studied before, for example in (Shum etal. 99).
It is to note, that this formalism can also be applied if we observe a rigidly moving object with a set of static cameras. In this case, we attach the world coordinate system to the moving object and we can relate the relative motion of the image sensors with respect to the object to the spatio-temporal derivatives of the light rays that leave the object.

Plenoptic motion estimation using polydioptric cameras.

It is also important to realize that the derivatives and can be obtained from the image information captured by a polydioptric camera. Recall that a polydioptric camera can be envisioned as a surface where every point corresponds to a pinhole camera, the plenoptic derivative with respect to direction Partial of Lightfield with respect to direction is the derivative with respect to the image coordinates that one finds in a traditional pinhole camera. One keeps the position and time constant and changes direction. The second plenoptic derivative, Partial of Lightfield with respect to position, is obtained by keeping the direction of the ray constant and changing the position along the surface. Thus, one captures the change of intensity between parallel rays. This is similar to computing the derivatives in an affine or orthographic camera. The ability to compute all the plenoptic derivatives depends on the ability to capture light at multiple viewpoints coming from multiple directions. This corresponds to the ability to incorporate stereo information into motion estimation, since multiple rays observe the same part of the world. For single-viewpoint cameras this is inherently impossible, and thus it necessitates nonlinear estimation over both structure and motion to compensate for this lack of multi-view (or equivalently depth) information.