Time series analysis of video sequences using TimeSearcher

 

By Samah Ramadan

 

Introduction:

Background subtraction is the process of detecting moving objects in front of a stationary camera. This can be achieved by comparing each new frame with a representation of the scene background. Typically, background subtraction forms the first stage in automated visual surveillance systems. Results from background subtraction are used for further processing, such as tracking targets and understanding events.

 

Objective:

The objective of this project is to use a time series visualization tool (such as TimeSearcher), to visualize the pixel values in a video sequence. The dramatic changes in pixel values in the video sequence can be an indicator of the presence of a moving subject.

 

Methodology:

The experiment was done on two different video sequences. The first one represents a static background in initial frames and a moving subject in the following frames. The second sequence used represents a dynamic background of a waving tree in the initial frame then a moving subject in the following frames.

To conduct this application project, two basic phases have been addressed.

 

1-      Data extraction:

The input of this phase is a video sequence at a frame rate of 30 frame/sec and the objective of this step is to extract a time series for each pixel of each frame through the video sequence. The original image size for the first sequence at each frame used is 720 X 480, which leads to a total of 345,600 pixels. For the second sequence the original image size is 160 X 120. I used MATLAB to read the images and to output the corresponding value for each pixel in a TQD file format, which is the format needed for TimeSearcher. As we are handling colored images, each pixel has three variables that we monitor throughout the sequence, R (RED), G (Green) and B (BLUE) values.

 

2-      Data exploration

I used TimeSearcher 2.4 to explore and visualize the pixel variable throughout the sequence. The major challenge during that phase was the enormous number of items that Timeseacher should handle (345,600 pixels). With that number of items the TimeSearcher failed to open the TQD file. As a result I had to downsize the image of the first sequence to be 180 X 120 (which is 1/16 of the original image size). The disadvantage of this downsizing is the loss of information and resolution.

The next challenge was even after downsizing the images, the Timesearcher could not handle this number of items (21,600 pixels) to respond to queries in a real time or even a reasonable response time. The solution for this problem was only to select a smaller sub-image (30 x 30) of the resized image to monitor the pixel values only over this sub-image. This step led to even more loss of information.

For the second sequence a sub-image of 30 X 30 was used to avoid the response time problem for the original data size (160 X 120 =  19,200 pixel)

 

Results:

Initial condition:

We will always assume that the initial frames represent only the background; usually this is done in background subtraction to be able to build the background model.

First sequence:

This sequence has a static background and a moving subject. Figure 1 shows a sample image of initial frames and a sample image of a moving subject.

 

        

Fig.1-a-Background for the first sequence b- moving subject

 

1.1- Detection of a passing by object:

As our initial condition indicates, the RGB values in the initial few frames in the sequence represent solely the background; hence the RGB values represented in the initial frames are for the background. This is obvious from figure 2, as we can see that the RGB values are almost constants from frame 1 to frame 23.

 

Fig. 2. The RGB values for the selected sub-image pixels through the sequence.

 

Starting from frame 24 we can see that the RGB values for the pixels in sub-image start to increase dramatically which indicates that there is an object passing by which was not existed in the initial few frames.

 

1.2 - Determination of the direction of the object

As we can see from Figure 3, we can determine the direction of the passing object by monitoring the order in which the pixels change from background values to the new object values then back again to background values when the object moves out of the sub-image. This can be inferred from the order of change of the pixels along with the values of x axis of the pixels. We noticed that Pixel 30,30 (at x=30, y=30) rises before Pixel.1.30  (x=1, y=30) and that is a pattern for all other pixels. So we can conclude that the pixels to the right change before the pixels to the left. Also in the phase when the pixels return to the background RGB values again all the pixels with higher X return to the background values before the pixels with lower Xs. This can be a key to decide that the passing object was moving from the right to the left.

 

 

Pixel.1.30
 

Pixel.30.30

 
 

Figure 3: shows the R values for the two selected pixels at Pixel.1.30 (X=1 y=30) and Pixel.30.30 (X=30, Y=3

 

1.3- Determination of the time the object occupied the sub-image

This can be inferred from Figure 4, if we noticed that the average cycle length it took for change of the pixels values from the background values to the object values and back to the background values was on average was about 47 frames.  With a frame rate of 30 frame/sec we can decide that the object occupied the sub-image for approximately 1.5 sec.

 

 

 

The object is occupying the pixel
 
Fig. 4.  The Duration in which the object occupied a specific pixel in the sub-image.

 

Second Sequence:

This sequence has a dynamic background in which a tree is waving a moving subject. Figure 5 shows a sample image of the initial frames and a sample image of a moving subject.

 

       

Fig.5-a-Background for the second sequence b- moving subject


 

 2.1- Detection of a dynamic background:

As our initial condition indicates, the RGB values in the initial few frames in the sequence represent solely the background; hence the RGB values represented in the initial frames are for the background. It is obvious from Figure 6 that the pixel values changes dramatically in the selected sub-image, which reflects the quasi-periodical nature of the tree. The pixel content changes between the tree and the sky from frame 1 to frame 144.

 

Dynamic background, Pixel values changes between sky color and tree color
 
 

 

 

 


 

Fig.6. Time series for a pixel in the Sub-image that has oscillate between two values the sky color and the waving tree color.

 

2.2- Detection of a static Background

Along with the dynamic background pixels there are also some pixels that almost static, Figure 7 shows an example of a pixel that had the sky color and is not affected by the waving tree.

 

 

 


Fig.7 Time series for a pixel that belongs to the static part of the background.

 

2.3- Detection of a passing by subject.

 As a subject passes by the selected sub-image the pixel values dramatically change, that is for periodical pixel and for static pixels too.

 

 

 

 

 

 

 

 


Fig. 8 The detection of a moving subject by both the static and the dynamic pixels in the selected sub-image

 

Conclusion:

I have used TimeSearcher to visualize and detect moving objects from stationary visual surveillance video. The experiments showed that time series analysis can be used to perform background subtraction. TimeSearcher is a very powerful tool for time series exploration; however there are some limitations for the scalability of this tool.

  1. Failed to open a data file with 200 X 200 pixel (40,000) item or more.
  2. For even smaller number of items (100 X100 for example), the response time for queries was very long and the tool was not interactive as it is supposed to be.