Department of Computer Science and Applied Mathematics
Computer Vision Lab

Space-Time Scene Manifolds

Yonatan Wexler and Denis Simakov

The space of real image patches is known to be a non-linear subspace of all pixel combinations. This paper presents the idea of the Scene Manifold, which lies within this subspace, and is restricted to cut the space-time volume without incurring visual artifacts or distortions. Every local neighborhood within it resembles some image patch. For a broad class of inputs the problem can be posed as a shortest path in a graph and can thus be solved efficiently to produce the globally optimal solution. Constraining appearance rather than geometry gives rise to a variety of new capabilities.

The usefulness of this approach is demonstrated by posing the well studied problem of mosaicing in a new way. Instead of treating it as geometrical alignment, we pose it as an appearance optimization. Since the manifold is constrained to lie in the space of valid image patches, the resulting mosaic is guaranteed to have the least distortions possible. Any small part of it can be seen in some image even though the manifold spans the whole video. It can handle both static and dynamic scenes, with or without parallax.

Paper:

Y.Wexler and D.Simakov, "Space-Time Scene Manifolds". International Conference on Computer Vision, 2005.
Full paper (4Mb pdf) | Reference (BibTeX)
Matlab code with examples (11Mb) | Matlab code only

Panoramic mosaic of a highly dynamic scene