PhD Proposal: Reranking by Multi-Feature Fusion for Image Retrieval

Talk
Fan Yang
Time: 
01.28.2015 10:00 to 11:30
Location: 

AVW 4424

Content-based image retrieval (CBIR) has been studied for decades due to its importance in web and image search. The task of CBIR focuses on searching for same/similar images from an image database given a query image, which contains a particular object. Most existing approaches adopt a single feature modality, which only captures one "view" of an image. However, by reranking initial retrieval results from multiple features, we may discover both agreement and inconsistency among them to improve retrieval quality.
We first propose a simple yet effective multi-feature fusion approach based on regression models for logo retrieval. We utilize similarities between pairs of images from multiple features, where only annotations of similar/dissimilar pairs of images are needed. For each pair of images, a new vector is constructed by concatenating the similarities from multiple features. A regression model is fitted on the new set of vectors with similar/dissimilar annotations as labels. Similarities from multiple features between the query and database images can then be converted to a new similarity score using the learned regression model to rerank initially retrieved database images. Logo class information from training samples can also be included by learning an ensemble of regression models for individual logo classes.
To reduce human annotation, we further present a reranking algorithm by modeling the statistics of similarities between images rather than learning regression functions. The initial ranked list for a query from each feature is represented as an undirected graph, where edge strength comes from feature-specific image similarity. Graphs from multiple modalities are combined by a mixture Markov model. We propose a probabilistic model based on the statistics of similarities of similar and dissimilar image pairs to determine the weight for each graph. A di ffusion process is then applied to the fused graph to reduce noise and achieve better retrieval performance. Experiments on generic object retrieval demonstrate that our approach significantly improves performance over baseline methods and outperforms many existing competitors.
Lastly, we propose a submodular reranking algorithm to combine multiple ranked lists obtained from multiple features, which is fully unsupervised and does not require any annotations of similar/dissimilar pairs of images. We formulate the reranking problem as maximizing a submodular and non-decreasing objective function that consists of an information gain term and a relative ranking consistency term. The information gain term exploits relationships of initially retrieved images based on a random walk model on a graph, then images similar to the query can be found through their neighboring images. The relative ranking consistency term takes relative relationships of initial ranks between retrieved images into account. It captures both images with similar ranks in the initial ranked lists, and images that are similar to the query but highly ranked by only a small number of modalities. Due to its diminishing returns property, the objective function can be efficiently optimized by a greedy algorithm. Experiments show that our submodular reranking algorithm is effective and efficient in reranking images initially retrieved by multiple features, and can be easily generalized to any generic reranking problems for real-time search engines.
Examining Committee:
Committee Chair: - Dr. Larry Davis
Dept's Representative - Dr. Amol Deshpande
Committee Member(s): - Dr. David Jacobs