A Probabilistic Framework for Correspondence and Egomotion
Estimation
Most algorithms for egomotion first estimate correspondences, and then use these correspondences to
find egomotion. This is problematic, because establishing correspondences between
images is an extremely difficult problem. Even the most advanced current
techniques often yield a high fraction of wrong ('outlier')
correspondences, while the correct ('inlier') correspondences are usually
corrupted by small errors. Here, instead of finding
correspondences, we find for each point a probability distribution
over the possible correspondences. Egomotion is computed directly from
these correspondence distributions.
This framework can be extended to the uncalibrated case. Code for computing the Fundamental
Matrix.
Publication
J. Domke, Y. Aloimonos, "A Probabilistic Notion of Correspondence and the Epipolar Constraint", Proc. 3DPVT (International Symposium on 3D Data Processing Visualization and Transmission), 2006 pdf
Videos MapSwimSwim 2PuppetTsukubaSawtoothVenusSofaYosemite
These videos illustrate the process of establishing correspondence distributions. We mark one point in the
first image, and encode the probability that that point corresponds to each possible point in the
second image by color. The videos contain four windows, which should be interpreted as follows:
Upper left: First image. The point we are considering is marked
Upper right: Second image. (Blue channel) Total correspondence distribution (Red channel)
Lower left: Correspondence distribution given by each specific orientation and scale.
Lower right: Total Correspondence distribution
Videos WaveYosemiteSwim
These videos show corresponding epipolar lines for several frames from a movie. Notice that the epipolar
geometry is found very accurately for the stationary background, while being undisturbed by the independently
moving parts of the scene.
Video Shrinkage
This video shows the probabilities over the motions as the number of correspondence distributions is
increased. Theta and ry are as in the paper, while all other parameters are fixed to
the ground truth. With a small number of correspondences, it is essentially impossible to disambiguate the
image motion as being due to translation or rotation, but larger amounts of information does it.
Data Some example images to play around with, including ground truth
motion, calibration, etc:
imagepack.mat (610 kb, matlab .mat file)
Download Egomotion Estimation Matlab Code:zip (Updated 5 Apr, 2006) license
This code takes images as input, and gives the Essential Matrix and/or the
camera motion as output. (Algorithms that solve this problem are sometimes
called Image Registration or Structure From Motion. Notice however, that
this algorithm does not try to find Structure- only Motion!)
An important note to users of this code
Most common problems with this code are due to too large a baseline. When the code computes correspondence probability distributions, it assumes that the correspondence is in a local window. If points are moving really far, this will lead to no end of trouble. Most likely, some points aren't moving too far, so instead of getting terrible results telling you something is amiss, the code will sort of work, but not very well. You have two options: (1) Use short baseline sequences. (2) Change the size of the window in which the code looks for matches. We have experimented with windows as large as the entire image- this works fine, but is predictably slow.
The other problem you might have with wide baseline sequences is that there is significant rotation (about the 'z-axis') or scale changes. The Gabor filters are somewhat robust to these effects, but a rotation of more than, say, 15 degrees will lead to trouble.
Matlab 7 or better is required. If you have an older version it would be relatively easy to adopt the code to work- there is a single use of the anonymous function syntax in motion.m. (Anonymous functions first appeared in Matlab 7.)
The optimization toolbox is required. It would be relatively hard to work around this.
This code uses a mex file to accomplish the most computationally intensive inner loop when searching for
the best motion. (This may cause difficulties for Octave users.)
Currently, we provide the full c source code for this file
(mymotion.c), as well as precompiled binaries for Windows and Linux. To compile for a different system could by as
simple as typing the line "mex mymotion.c" from
inside matlab. If you compile such a binary, please email it to Justin Domke so that it can be included
here.
We are redistributing as part of the code the excellent Gabor filter implementation referenced in the paper. This work was done at the Institute of Optics. Many thanks to Oscar Nestares for permission to do this.
