Project X, Optical Motion Capture

For any questions about this project, see Gutemberg in AVW4470
guerra@cs

MoCap.zip Contains the data for this project.

This project consists of two parts. In Part I, you will execute the 
post-processing step of a Motion Capture System. Part II will be 
concerned with sub-problems related to the Optical Motion Capture 
Process.

Motion Capture (MoCap) is the process of recording real life movement of 
a subject. Usually, humans are the subjects (performers) in motion 
capture. A movement is represented as sequences of Cartesian coordinates 
in 3D space.

Optical Motion Capture uses a set of synchronous cameras to capture 
markers placed in strategic locations on the body of a subject 
(performer). These markers are located by triangulating the various 
camera views according to camera calibration. Our Optical Motion Capture 
System has four main steps: marker detection, spatial correspondence, 
temporal correspondence, and post-processing.

The marker detection process solves a problem similar to feature 
detection. In this case, the feature is a marker located near to joints 
of an actor. The 2D pixel coordinates of all markers are computed for 
each frame in each camera view. The output of this module is a file 
(‘markers.dat’).

The spatial correspondence step performs stereo matching between 
different camera views in the same frame. Using intensity and calibration 
information, the same detected marker is matched in different views and a 
3D point is reconstructed through triangulation. The output of this 
module is a file (‘points.dat’).

The temporal correspondence module tracks reconstructed 3D markers 
between consecutive frames. Each 3D marker in frame f is associated with 
3D markers in frames f+1. The output of this module is a file 
(‘matches.dat’).

 

 

1. Part I: Post-Processing MoCap Data

Deadline: Feb 28th, 2005.

 

The post-processing step of our MoCap system involves labeling, 
finishing, and correcting the MoCap data. The labeling process concerns 
assigning label codes (e.g. 01 for FRONT_HEAD) to detected markers 
tracked over most of the frames. The finishing process involves manually 
finding 3D points corresponding to some markers missed in the detection 
phase, or not spatially/temporally corresponded. The correcting process 
will statistically compute the length of rigid segments in our human 
model and detect possible errors in the distance between rigid 3D points 
(markers). These errors will be corrected by finding manually a new 3D 
point for the corresponding marker.

 

1.1. Input Data

 

You will receive a CD containing two video sequences recording different 
human actions. Each file in the input CD is named as 
‘[actor].[action].[camera].[frame].[format]’, where [actor] is ‘gut’, 
[camera] is [1:8], [frame] is [00000:00047], [format] is ‘pgm’, and 
[action] is the name of a human action performed in a video. These videos 
are the initial input for our Optical Motion Capture System.

For the two actions in the CD, you will execute the post-processing of 
the corresponding Motion Capture Data. The Motion Capture Data is divided 
into two sets: Sublexicon1 and Sublexicon2. The MoCap data is found at 
‘...\MoCap\Sublexicon1\’ and ‘...\MoCap\Sublexicon2\’, respectively. 
Calibration Data for each sublexicon is found at 
‘...\MoCap\Calibration1\’ and ‘...\MoCap\Calibration2\’, respectively. 
The MoCap files associated with a particular [action] will be 
‘[actor].[action].markers.dat’, ‘[actor].[action].points.dat’, and 
‘[actor].[action].matches.dat’. These files are located in the sublexicon 
associated with the number written in the input CD.

 

1.2. Matlab Code

 

Our MoCap System was implemented using Matlab. The source code for the 
post-processing step is found at ‘...\MoCap\Code\’. The main command is 
‘mandacaru(actFile, calFile, task);’. actFile is a string formatted as 
‘[sublexDir][actor].[action]’, where [sublexDir] is the directory 
containing the MoCap data (markers, points, matches files). calFile is a 
string formatted as ‘[calDir]camera’, where [calDir] is the directory 
containing the Calibration data. task is an integer code, where 1 
corresponds to the labeling step, 2 corresponds to the finishing step, 
and 3 corresponds to the correcting step. For example, when [sublexDir] 
is 'D:\MoCap\sublexicon2\’ and [calDir] is 'D:\MoCap\calibration2\’, the 
matlab commands are:

 

actFile = 'D:\MoCap\sublexicon2\gut.jog';

calFile = 'D:\MoCap\calibration2\camera';

mandacaru(actFile, calFile, 1);

 

1.3. Labeling Process

 

The labeling process iteratively displays detected 3D points tracked over 
most of the frames. The points are presented as red circles in the eight 
possible camera views. The user should check at least two camera views in 
order to identify the current point as a marker. The set of markers used 
in our human model and the corresponding code table are in the file 
‘...\MoCap\markersCode.doc’. Once a point is identified, a marker code 
(label) should be assigned by typing the corresponding integer number 
found in the code table. If the 3D point is identified as an outlier 
(false detection), the marker code should be 0 (zero). Outliers appear 
mostly near to the head, consequently, special care should be taken in 
the identification/labeling of points near the head. The user may also 
quit and save data using code –1, or go to the previous 3D point using 
code –2 if a mistaken code was entered. The process ends when all markers 
are labeled.

 

1.4. Finishing Process

 

In the finishing process, manual selection of 3D points is performed for 
markers missed in some frames. If a marker is not labeled at all, this 
marker requires some initialization. The initialization consists in 
selecting two 3D points in consecutive frames. The selection step of a 3D 
point shows one camera view at a time and the process will be pause so 
the user may zoom in properly. The left click ignores the current view 
when the current marker is not in the image. The right click selects the 
point in the image. At most three points are select for each new 3D 
reconstruction. After selection, the new 3D reconstructed point is 
displayed. After a possible initialization, the finishing process 
iterated over each missing frame. The 3D point (marker) for this frame 
may be automatically computed. Otherwise, all the camera views are 
displayed with two 3D point candidates: an inverse kinematics (IK) point 
(red) and a linearly interpolated (LI) point (blue). The user must type a 
finishing code: 0 for the IK point, 1 for the LI point, and 2 for none of 
these points. If none is the option, the user will perform the 3D point 
selection step by clicking in the camera views. After selection, the new 
3D reconstructed point is displayed.

 

1.5. Correcting Process

 

The correcting process computes the length of each rigid segment in our 
human model and compares with the distance between rigid points in order 
to detect possible reconstruction errors of 3D points. For each detected 
error, the current 3D point (red circle) and a linearly interpolated 
point (blue circle) are displayed in all camera views. The user must type 
a correction code: 0 for the current point, 1 for the interpolated point, 
and 2 for none of these points. If none is the option, the user will 
select a new 3D point replacement by clicking in at least two camera 
views. During the point selection, one camera view will be displayed at a 
time and the process will be pause so the user may zoom in properly. The 
left click ignores the current view when the current marker is not in the 
image. The right click selects the point in the image. At most three 
points are select for each new 3D reconstruction. After selection, the 
new 3D reconstructed point is displayed.

 

2. Part II: MoCap Sub-problems (will be posted on Feb 29th, 2005)

Web Accessibility