Isaac Weiss

Dr. Weiss's research, supported by DARPA and the U.S. Air Force, deals with two subjects which are central to computer vision: application of invariance theory, and robust estimation.

Invariants are very powerful tools for several key areas of vision, such as object recognition and camera calibration. One example of how invariants are used for recognition is provided by the viewpoint problem. Different images of the same object often differ from each other because of the different viewpoints from which the images were taken. Common recognition methods try to match an image of an object, seen from an unknown viewpoint, to an image stored in a database. For this they need to find the correct viewpoint, a difficult problem that can involve search in a large parameter space of all possible viewpoints and/or finding point correspondences. Geometric invariants are shape descriptors, computed from the image, that remain unchanged under transformations such as changing the viewpoint. Thus they can be matched without search. Projective (viewpoint) invariants of curves and surfaces were an active mathematical field in the 19th century. However, in the vision community they have only recently received major attention, after a paper by this author reviewed them and pointed out their importance.

Almost all work on invariants in recent years has involved planar shapes. Much more interesting is the projection of 3D, rather than planar, shapes into a 2D image. There are no true invariants of this projection because the depth information is lost. However, there are invariant constraints that are almost as useful as true invariants for the task of eliminating the unknown viewpoint. A current research project involves developing and applying these 3D invariant constraints to recognizing objects. This involves invariants of various types of features that can be found in objects, including point sets, lines, conics and general curves. The invariant constraints can be used to dramatically reduce the search space for the correct viewpoint and thus make recognition feasible. An example is shown in Figure 1.

Objects can undergo other types of geometric changes, in addition to viewpoint changes. For instance, one can recognize apples even though different apples have slightly different shapes. We have applied invariants of such small deformations to characterize classes of objects such as apples and bananas, and to differentiate these classes from each other.

Another example of non-viewpoint changes that we are actively studying is identifying articulated objects in range images. Here we can have objects such as tanks that can change not only their positions but their shapes, by rotating the turret or raising the gun barrel. Invariance to these changes again reduces a very big search space to a more manageable one.

Invariants in fact have much wider uses than eliminating geometric unknowns. Images also depend on the physical process that creates them, which can involve visible light, infrared, radar, etc. The physical component has even more unknowns then the geometric one. For example, in an ordinary image, the amount of light falling on the image plane depends on physical quantities such as surface reflectance, illumination intensity and its spatial distribution, characteristics of the imaging system, etc. Recovering the original shape requires the elimination of all these unknowns. This is the "Shape from Shading" problem. Infrared images also depend on surface temperature. All these unknowns greatly complicate the recognition task.

Invariants of physical processes have been extensively studied in modern physics and there are various methods of finding them. Simple examples are the laws of conservation of energy and momentum, which are in fact invariants of the laws of motion. In current research some of these methods are being adapted for application to computer vision. Moreover, the fusion of information from both geometric and physics-based invariants can also be achieved through such methods. Such fusion of data from different sources can greatly improve the robustness and reliability of the final results.

Robust estimation techniques are of very general significance, and of particular value in obtaining invariants. This is because some invariants depend on accurate measurements of features such as point coordinates or curve parameters. Part of the progress that we have made is in the theoretical analysis of the problem, for instance in finding higher-order derivatives of curves. Another research direction for increasing robustness is using non-parametric methods, such as bootstrap, which are harder to study analytically. Of special interest here is the problem of removing "outliers" from the given data. These outliers distort the results of simpler methods such as least-squares fitting, and removing them greatly increases the reliability of the information that we extract from the data.

Figure 1. Vehicle recognition using invariants. On the left are two images of the same vehicle. On the right, drawn in black, is a 3D model of the vehicle, represented in a 3D "invariant space". The colored (3D) lines on the right are calculated from the 2D images on the left. The intersection of these lines with the model points implies recognition of the vehicle.