Feature-Based Sequence-to-Sequence Matching
Yaron Caspi, Denis Simakov and Michal Irani

Wide base-line. Basketball sequence.

The setup: scene and cameras


Two cameras are looking at each other, capturing basketball field between them.

Input sequences

Camera 1	Camera 2
Video: MPEG 1.4Mb	Video: MPEG 1.4Mb

Detect moving objects

... using background subtraction:


Camera 1	Camera 2
Video: GIF 250Kb   AVI 440Kb   MPEG 1.4Mb	Video: GIF 240Kb   AVI 410Kb   MPEG 1.4Mb

Extract interest points

... by taking the highest point on each blob, in each frame:


Camera 1	Camera 2

Construct trajectories from these points

Camera 1	Camera 2
Video: GIF 250Kb   AVI 500Kb   MPEG 1.4Mb	Video: GIF 250Kb   AVI 450Kb   MPEG 1.4M
Synchronized video: GIF 450Kb   AVI 710Kb   MPEG 1Mb

Use the trajectories as features for matching algorithm

Camera 1	Camera 2

Each trajectory starts at a green square and stops at a red square.
There are about 50 trajectories on each side, but at most 3 (two players and the ball) at each given time moment.

Recover epipolar geometry (spatial alignment)

Camera 1	Camera 2

Yellow points on the left correspond to the yellow lines on the right (their epipolar lines);
red points on the right correspond to the red lines on the left (their epipolar lines).
Points are manually put and their epipolar lines calculated using recovered fundamental matrix.
White cross marks location of the opposite camera (the ground-truth epipole).

... and sub-frame time shift (temporal alignment)

Camera 1	Camera 2

On the right you see two fused frames that are closest in time to the frame on the left (the ball is falling at high speed).
Red epipolar lines on the left don't fall on the top of the ball, showing that it doesn't corespond to either ball on the right.
Nevertheless, it corresponds to the interpolated aqua ball, at non-integer frame 0.7.

To the top of the page

Last modified: Oct 06, 2006