Ronen Basri: Research

Alignment by region correspondence

 

A basic question in vision is how to use features extracted from an image to compute the pose of an observed (2D or 3D) object. Most existing methods use either global features (e.g., shape moments) or local features (points and line segments) to compute a transformation, but global features are sensitive to occlusion and local features require correspondence which is difficult to achieve. In a sequence of papers we examined the use of regions to determine a transformation. Suppose we can identify several regions in the image with corresponding regions (or volumes) in the object, but we refrain from using any point correspondences inside the regions. It can be shown that the problem of finding a transformation is non-convex, but if we modify the problem (in a way that allows occlusions in the image) we can make the problem convex and solve it with dynamic programming. Our derivations uses the fixed point theorem to proof uniqueness for various common transformations including 2D affine and projective as well as to affine projection (i.e., 3D-to-2D affine transformation).

What lies ahead? A system based on these ideas will need to use segmentation to extract the regions and then enumerate potential correspondences. Can this be made reliable? Another challenge is to construct a model by learning the regions from data. There is still room for analysis of the case of 3D-to-2D perspective projection and occlusion.

We seek a projective transformation relating the image on top to either of the two images in the bottom, the right of which is partly occluded.

With three corresponding regions (marked in green) the transformation T₁ can be found (left), but with only two regions the solution is non-unique leading to contraction (right).

Our first paper in this topic appeared in ICCV 2005. This paper dealt mainly with the case of 2D affine transformation and introduced the use of fixed points to establish uniqueness. The journal version was published in
                Ronen Basri and David Jacobs, “Recognition using region correspondences,” International Journal of Computer Vision,                 25(2): 141-162, 1997.

My favorite paper in this line of work is the one that deals with 2D projective transformation. Here eigenvalue analysis and polar relations in quadratic forms are used to either establish or refute uniqueness when respectively three or two region correspondences are available. Examples are shown in the figures above. This analysis was published in
                Ronen Basri and David Jacobs, “Projective alignment with regions,” IEEE Transactions on Pattern Analysis and Machine                 Intelligence, 23(5): 519-527, 2001.

The theory was further extended to dealing with 3D objects undergoing affine projection, where unexpected difficulties arise, in
                David Jacobs and Ronen Basri, “3-D to 2-D pose estimation with regions,” International Journal of Computer Vision,                 34(2/3): 123-145, 1999.

Related theoretical results were derived on the efficiency of reconstructing 2D shapes from images containing partial information in
                Ronen Basri, Adam Grove, and David Jacobs, “Efficient determination of shape from multiple images containing partial                 information,” Pattern Recognition, 31(11): 1691-1703, 1998.

T₁?

T₂?

The alignment transformation is found despite the occlusion.