Department of Computer Science and Applied Mathematics
Computer Vision Lab
Weizmann Institute of Science

Space-Time Scene Manifolds

Yonatan Wexler   and   Denis Simakov

The space of real image patches is known to be a non-linear subspace of all pixel combinations. This paper presents the idea of the Scene Manifold, which lies within this subspace, and is restricted to cut the space-time volume without incurring visual artifacts or distortions. Every local neighborhood within it resembles some image patch. For a broad class of inputs the problem can be posed as a shortest path in a graph and can thus be solved efficiently to produce the globally optimal solution. Constraining appearance rather than geometry gives rise to a variety of new capabilities.

The usefulness of this approach is demonstrated by posing the well studied problem of mosaicing in a new way. Instead of treating it as geometrical alignment, we pose it as an appearance optimization. Since the manifold is constrained to lie in the space of valid image patches, the resulting mosaic is guaranteed to have the least distortions possible. Any small part of it can be seen in some image even though the manifold spans the whole video. It can handle both static and dynamic scenes, with or without parallax.


Paper:  Y.Wexler and D.Simakov, "Space-Time Scene Manifolds". International Conference on Computer Vision, 2005.
Full paper (4Mb pdf) | Reference (BibTeX)
Matlab code with examples (11Mb) | Matlab code only
 


Panoramic mosaic of a highly dynamic scene

Scene manifold in the 3D video volume   Mosaic image "cut out" by the manifold
Beach sequence: space-time manifold   Beach sequence: mosaic
Original video and high resolution images


Party video: dynamics and parallax

Mosaic image
Party video: mosaic
Original video and high resolution images



Synopsis mosaic

Display the same person at different moments


Scene manifold   Mosaic image
Cricket (synopsis): space-time manifold   Cricket (synopsis): mosaic
Original video and high resolution images




Parallax scene

Multiple-viewpoint mosaic without distortions.

Scene manifold   Mosaic image
Parallax: space-time manifold   Parallax: mosaic
Original video and high resolution images



Image of an object from all sides

Scene manifold   Mosaic image
Rotating CocaCola: space-time manifold   Rotating CocaCola: mosaic
Original video and high resolution images

Group photo in Chinese restaurant

Mosaic image
Group photo: mosaic
Original video and high resolution images


Computer Vision Lab
Department of Computer Science and Applied Mathematics
The Weizmann Institute of Science
Rehovot 76100, ISRAEL
Tel: 972-8-934-3540     Fax: 972-8-934-4122

This page was last modified on Jun 10, 2007.