Plenoptic Brightness Constancy

Discrete Plenoptic Motion Constraint.

Let us assume that the albedo of the scene surfaces is constant over time and that we observe a static world under constant illumination. In this case, the radiance of a light ray in the world does not change over time which implies that the total time derivative of this light ray vanishes: d/dt L(x;r,t) = 0. This means now that if we transform the space of light rays by a rigid transformation, for example parameterized by the rotation matrix R and a translation vector t, then we have the exact identity, which we term the discrete plenoptic motion constraint

,

since the rigid motion maps the time-invariant space of light rays upon itself. Thus, the problem of estimating the rigid motion of a sensor has become an image registration problem that is independent of the scene!

Illustration of plenoptic brightness constancy using subsets of an epipolar volume

We can illustrate the basic idea by examing how the image motion flow depends on the scene if we look at the subsets of an epipolar volume that are either corresponding to an image sequence captured by a conventional perspective camera or an image sequence captured by a linear pushbroom camera.

We can form an epipolar volume by translating a camera parallel to the horizontal image axis and stack the frames of the image sequence to form a volume:

Image Sequence by a Translating Camera

Image Sequence by a Translating Camera
Epipolar Volume

Epipolar Volume

Every pixel in an epipolar volume corresponds to a unique ray in space. If a camera is undergoing a rigid motion constrained to a horizontal plane, then we can illustrate the subset of light rays that a camera will capture during its motion by sweeping a plane through the epipolar volume. The top half of each movie shows the image sequence and the bottom half the sweep through an epipolar image. By sweeping through the epipolar volume we can simulate the following four rigidly moving cameras:

Image Sequence by a Translating Camera

Image Sequence by a Translating Perspective Camera (1.8 mb avi)
Image Sequence by a Rotating Push-broom Camera

Image Sequence by a Rotating Linear Push-Broom Camera (3.8mb avi)

We can see (top half) that for a rotating push-broom camera and for a translating perspective camera the image motion depends on the depth of the scene. This is the well-known effect of motion parallax. We also notice (bottom half) that during each frame the cameras capture different light rays. Thus to to estimate the camera motion on the basis of the image sequences, we need to estimate the scene structure so that we can correspond the pixels (light rays) to eachother.

Image Sequence by a Translating Camera

Image Sequence by a Translating Linear Push-broom Camera (570kb avi)
Image Sequence by a Translating Camera

Image Sequence by a Rotating Perspective Camera (1.8mb avi)

In contrast, we see (top half) that for a translating push-broom sequence and a rotating perspective image sequence the optical flow in the images is independent of the scene structure. For a perspective camera this is well-known and has been used to generate panoramic images and the parameterization between the frames is given as a homography. This is because most of the rays that form an image of the image sequence at any given time are also part of the preceding and following frames. Only the image boundaries contain new information. Thus we are able to estimate the rotation (translation) by globally matching images to images without having to compute any scene parameters! The idea of polydioptric motion estimation is now that by matching light rays across view points and view directions we can estimate the full 3D motion of a polydioptric camera similar how we can estimate motion of a pinhole camera that is rotating around its optical center.