Image Retrieval-A Review

Image Retrieval-A Review

1. Malcom Marshall

   M.E-Computer Science and Engineering CIET-Coimbatore,Tamil Nadu,India

   Dr. S. Gunasekaran

   Prof and Head- Computer Science and Engineering CIET-Coimbatore, Tamil Nadu,India

   AbstractThis article provides a survey on image retrieval methodologies. The test based image retrieval methodology-where the images are retrieved by metadata informations and also involves natural language processing procedures such as stop word removal, root words, parsing sentence. The Content based image retrieval methodology involves the retrieval of images as by processing it contents such as shape of the image ,texture and color of the images. The third one discussed is Semantic based image retrieval methodology where the retrieval of image as by its semantic concepts and also involves the scene modeling and the scene classification and scene retrieval by labeling and annotating the various image regions.

   Keywordsimage retrieval; labeling; annoatation; semantic;

   1. INTRODUCTION

      Image retrieval has become an important research area in computer vision where digital picture collections are quickly being created & made obtainable to multitudes of users through the World Wide Web. Tremendous increment in the collection of images from art museums, medical institutes & environmental agencies, to name a few. In the commercial sector, companies have been formed that are making huge collections of photographic images of real-world scenes obtainable to users who require them for illustrations in books, articles, commercials and other media meant for the public at huge. Amazingly, the indexing of these images is all being done manually a human indexer selects & inputs a set of keywords for each picture.

      Each keyword can be augmented by terms from a thesaurus that supplies synonyms & other terms that earlier users have tried in searches that led to related images. Keywords may even be obtained from captions, but these are less reliable. Content-based picture retrieval research has produced numerous search engines, the commercial picture providers, for the most part, are not using these techniques. The main reason is that most CBIR systems require an example picture & then retrieve similar images from their databases. Actual users do not have example images; they start with an idea, not an picture. Some CBIR systems permit users to draw the sketch of the images which they wanted. Such systems require the users to have their objectives in mind first & therefore can only be applied in some specific domains, like trademark matching, & buy of painting. Thus the recognition of generic classes of objects & ideas is essential to provide automated indexing of images for CBIR. However, the task is not simple. Computer programs can extract features from an picture, but there is no simple one-to-one mapping between features & objects. Earlier CBIR systems depend on global picture features, such as color histogram & texture statistics. Global features cannot capture object properties, so local features are favored for object class recognition. For the same reason, higher-level picture features are preferred to lower-level ones. Similar picture elements, like pixels, patches, and lines can be grouped together to form higher-level units, which are more likely to correspond to objects or object parts. Different types of features can be combined to improve the feature discriminability. For example, using color and texture to identify trees is more reliable than using color or texture alone. The context information is also helpful for detecting objects. A boat candidate region more likely corresponds to a boat if it is inside a blue region. While

      improving the ability of our technique by designing higher-level picture features and combining individual ones, to be prepared to apply increasingly features since a limited number of features cannot satisfying the requirement of recognizing plenty of different objects in ordinary photographic images.

   2. IMAGE RETRIEVAL METHODS

      A picture tells over the words that can express. Nowadays immense amount of pictures are shared, moved, edited, modified in social networks, military, search engines etc. Several picture retrieval techniques are available in our day-to-day life. That technique retrieves the images according to their names that it was stored, usually system of working of text based picture retrieval, where it retrieves the picture by their metadata informations, captions. But the user interested correct picture cannot be retrieved. There comes the content based picture retrieval, where images are retrieved as by theircontents such as shape, color, texture.An picture retrieval method is of the significant research area which can be used for browsing, searching & retrieving images from a large database of digital images. Most traditional & common methods of picture retrieval method utilize some technique of adding metadata such as captioning, keywords, or descriptions to the images so that retrieval can be performed over the annotation words. In this work, the picture retrieval classified chiefly in to three types.

      1. Text Based Picture Retrieval

      2. Content Based Picture Retrieval

      3. Semanticannotation Based Picture Retrieval

      1. Text based picture retrieval

         The text based picture retrieval utilizes the technique of adding the metadata, such as keywords, captioning or descriptions to the images. The retrieval employed over the annotation words & it makes the annotation complex & time consuming & also requires large labors to by hand annotate the images. The semantic content is not thought about in TBIR.[25],[38]

      2. Content based picture retireval

         The content based image retrieval involves the retrieval of images as by its content namely the shape, texture, color of the image. It also involves the relevance feedback scheme for retrieval of images. The various algorithms such as SVM, K-means, K -nearest neighbor are employed by several authors in their works.[7],[8][10],[11],[12],[13],[14]

      3. Semantic based picture retrieval

      The semantic denotes meaning, so the retrieval of image by understanding the meaning of the image in human way basically by adding labels , keyword,, natural language to the image. The semantic based picture retrieval is closely associated with the content based picture retrieval. The semantic based picture retrieval addresses the global features such semantic concepts of the image, feature extraction and annotation of the image and image regions. Scene classification involves the image annotation where the keyword is associated with the image.

      The annotation can be performed by labeling the image regions by using connected component labeling algorithm. The keywords are associated with visual data dictionaries or the bag of visual words concept. First by finding the various image regions in the image, then by classifying the image regions includes the annotation of the region and extracting the features and last the retrieval of resultant image is done.[40],[18],[16],[31],[35]

      Text and Image Content[5]	textual and visual content descriptors are generated from the text query and image query. The descriptors are converted into a vector format. 2. Region growing algorithm: Step 1.Initialize a two dimension array of the image size. Step 2.Find a pixel which is not labeled. Label it and store its coordinates on a stack. Step 3.ge a pixel from the stack; Step 4. check its neighbours to see, if they are unlabeled and close to the considered pixel; if so, label them and store them on the stack. Step 3.Repeat the above step 3 & 4 until there are no more pixels on the image.	Let be a image set for the user provided query. Step 1.Compose the terms from various metadata fields such as name of the file , caption etc. Step 2.Filter the repeated terms/words and user query terms and calculate each term occurrence. Step 3.Sort the terms in a descending order based on term occurrence. Step 4.Allow the user to select one or more terms, which are relevant for his/her interest image. Step 5.Perform the simple keyword search technique (natural language processing) for the user selected terms and display the resultant images.
      User experiments with the Eurovision cross-language image retrieval system[6]	Created a multilingual search engine using tiny knowledge of any language other than English, then categorizing images assists the user's search, & differences in the way users search between the proposed search tasks	Involves machine translation and traditional monolingual IR. translation
      CONTENT BASED PICTURE RETRIEVAL
      Advanced Techniques in CBIR Local Descriptors, Visual Dictionaries and Bags of Features[45]	Local descriptors- Local descriptors are computed over local features such as regions, borders or Points of Interest. Visual dictionaries- representation which considers (high-dimensional) descriptor space and split it into multiple regions semantic concepts, for example, vegetation, rocks, clear sky, clouds, corners of buildings, BOW- substituting the text words by the visual words metaphor.	Local descriptor- SIFT+Knn search. Visual dictionary- PCA+Kmeans.
      Towards intelligent image retrieval[22]	Spoke about the automatic scene classification and the automatic object recognition (knowledge based)	Reviewed the various image retrieval for efficient processing.
      Hierarchical clustering algorithm for fast image retrieval[46]	a clustering based indexing technique, where the images in the database are grouped into clusters of images with similar color content using a hierarchical clustering algorithm.
      Content Bases Image Search And Retrieval Using Indexing By KMeans Clustering Technique[28]	Proposed the k-means clustering algorithm for image indexing and retrieval	Proposed Algorithm: 1. Read the image and decorrelation technique is applied. 2.conversion of RGB to L*A*B color space and classification a*b*

      1. controlled vocabulary

      2. machine

      3. bilingual parallel Corpora

      4. bilingual dictionaries

      The following table 1 provides the various list of annotation tools

      TABLE 1 ANNOATATION TOOLS OVERVIEW

      Tool	Metadata format/vocabulary	Annotation type	Annotation form
      K-space	Multimedia metadata ontology (m3o)	Image ,region based	Rectangle/polygon
      Photostuff	Media,technical ontology	Image,/region based	Rectangle/polygon/ci rcle
      Activemedia	OWL/Domain ontology	Image/ region based	Recangle/cirlce
      M- ontomatanno tizer	RDFS/DAML/OWL	Image /region based	Rectangle/ellipse/pol ygon/free hand
      Caliph	RDFS/DAML	Image	–
      Swad	Rdf/free text keywords	Image	–
      Labelme	Free text keywords	Image/region based	Polygon
      Marquee	Web based annotation tool	Image/region based	Rectangle
      Skitch	Free text keywords	Image /region based	Rectangle
      Neuromorph ic smile	Speech annotation tool	Image	–

      1. LITERATURE SURVEY TABLE 2 IMAGE RETRIEVAL OVERVIEW

         Paper title	Theme	Algorithm/module s
         TEXT BASED PICTURE RETRIEVAL
         User experiments with the Eurovision cross-language image retrieval system[1]	Created a multilingual search engine using tiny knowledge of any language other than English, then categorizing images assists the user's search, & differences in the way users search between the proposed search tasks	Involves machine translation and traditional monolingual IR. translation
         Text Based Approaches for Content-Based Image Retrieval on Large Image Collections[2]	Text based IR methods for indexing MPEG-7 visual features (from the MPEG-7 XM) to perform fast subset choice within large picture collections.	Involves inverted index structures and term identification techniques Through variations on n-grams
         An Integrated Approach to Text and Image Retrieval[3]	It is a xml text retrieval system based on scored region algebra algorithm,where xml naturally denotes the image regions	For text modelling uses the statistical language modelsand where the visual data is modeled using Weibull distributions or Gaussian mixture models
         Text-Based and Content-Based Image Retrieval on Flickr: DEMO[4]	Offline mode-The text-based descriptors such as title, description, and tags were extracted from the SAPIR collection. Online mode- the end user gives the query image,a search text, and a weighed distance function for each available picture feature. The distance functions can be metric (lEuclidean distance) or non-metric (DPF and cosine distance).	Offline mode-The feature vectors were calculated using the vectorial model and tf-idf weighing Online mode- the system performs a k- NN search using a weighed combination of distances, normalized by the maximum distance of a feature to the origin.
         Interactive Image Retrieval Using	the user enters the text and/ or sample image as a query. The	1.Refine search algorithm:

         1. controlled vocabulary

         2. machine

         3. bilingual parallel Corpora

         4. bilingual dictionaries

         		colors 3.label pixel and segmentation of image by color and divide the nuclie to form a separate image
         An efficient similarity measure via genetic algorithm for content based image retrieval with extensive features[9]	Proposed the genetic algorithm, where it measures the similarity between query and database image features also applies the squared Euclidean distance.	Fitness function is calculated with the help of Euclidean distance
         Content Based Image Retrieval Methods Using Graphical Image Retrieval Algorithm (GIRA)[20]	Involves the building keywords on visual features (labeling the image regions).auto color correlogram and correlation and color correogram is applied for low level features.	Developed a framework of multi- threading for a joint querying image search scheme.
         A novel approach for image classification in Content based image retrieval using support vector machine[33]	Proposed an svm algorithm to classify the pictures and	Involves the pre- processing ,feature extraction and svm classifier modules.
         SEMANTIC BASED PICTURE RETRIEVAL
         Semantic Image Segmentation and Object Labeling[37]	Involves the object detection and segmentation of the picture simultaneously.	Region growing algorithms such as watershed and recursive shortest spanning tree are used
         Integrated keywords and Image Content features For Image Indexing and Retrieval image within Compressed Domain[40]	Provide the solution to the problem of indexing and retrieval of image from compressed DCT domain.	Used semantic object detection categorization algorithm.
         Fusing Integrated Visual Vocabularies- Based Bag of Visual Words and Weighted Colour Moments on Spatial Pyramid Layout for Natural Scene Image Classification[41]	Developed various approaches for semantic scene Classification and modeling based on BoW	Framework of classification Annotation, Retrieval (CAR) is developed. Bag of visual words features is used .
         Semantic Scene Modeling and Retrieval[23]	Proposed the semantic classification ,by classifying the local image concepts.Scenemodeling,scene classification and scene retrieval are involves	Scene modeling multi class svm. Scene categorization- annotating image regions.
         A framework for group based image retrieval and video Annotation[29]	The two techniques are used, one is ontology in order to reduce the semantic gap and other performs a group based image retrieval using video files. The Automatic Semantic base Annotation algorithm performs annotation in three steps.	GIR algorithm to create similar image group. SIFT features are extracted and the steps used by ASVA algorithm.
         A hierarchical knowledge-based approach for retrieving similar medical images described with semantic annotations	Proposed an image retrieval system that considers the semantic of medical images.	Image similarity is computed using ontological relations. capturing the semantic correlations between image contents

         1. Calculates the similarity using SIFT features, sentence and synonym analysis

         2. find similar meaning annotations

         3. finally the conjunction of the sentences

         [42]
         Image annotation	image annotation tool for	SVM Algorithm
         using SVM[44]	classifying image regions in one	used for
         	of seven classes sky,	classification and
         	skin,vegetation, snow, water,	annoataion of images
         	ground, and buildings or as
         	unknown
         Beat the MTurkers:	Automatic segmentation given	Image labeling
         Automatic Image	annotating three dimensional	enables efficient
         Labeling from Weak	bounding boxes, a collection of	image classification
         3D Supervision[43]	computer aided design models	and retrieval
         	are developed.

         REFERENCES

         1. Clough, P and Sanderson, M. User experiments with the Eurovision cross-language image retrieval system Journal of the American Society of Information Science and Technology, vol.57 no.5 pp.697 – 708,2006

         2. Peter Wilkins, Paul Ferguson, Alan F. Smeaton and Cathal Gurrin , Text Based Approaches for Content-Based Image Retrieval on Large Image Collections EWIMT 05

         3. Thijs Westerveld, Jan C. van Gemert, Roberto Cornacchia, Djoerd Hiemstra, Arjen P. de Vries, An Integrated Approach to Text and Image Retrieval ,The Lowlands Team at Trecvid 2005

         4. Juan Manuel Barrios, Diego D´az-Espinoza, and Benjamin Bustos, Text-Based and Content-Based Image Retrieval on Flickr: DEMOIEEE-Second International Workshop on Similarity Search and Applications, 2009.

         5. Eduardo Valle , Matthieu Cord ,Advanced Techniques in CBIR Local Descriptors, Visual Dictionaries and Bags of Features

         6. Amanbir Sandhu and Aarti Kochhar Content Based Image Retrieval using Texture, Color and Shape for Image Analysis Council for Innovative Research International Journal of Computers & Technology, vol.3, no. 1, pp.2277-3061,2012

         7. Arnold W.M Smeulders, Marcel Worring and Amarnath Gupta

            Content based image retrieval at the end of the early years IEEE Transactions on Pattern Analysis and Machine Intelligence ,vol. 22, no.12 pp. 1349-1380,2000

         8. Arthi.K and Mr. J. Vijayaraghavan Content Based Image Retrieval Algorithm Using Color Models, International Journal of Advanced Research in Computer and Communication Engineering,Vol. 2, Issue 3,pp.1346-1347,2013

         9. Baddeti Syam and Yarravarappu Rao An effective similarity measure via genetic algorithm for content based image retrieval with extensive features International Arab journal information technology vol.10 no.2 pp.143-153,2013

         10. Bai Xue, Liu Wanjun Research of Image Retrieval Based on Color, International Forum on Computer Science-Technology and Applications, pp.283-286, 2009

         11. Chin-Chen Chang and Tzu-Chuen Lu A Color-Based Image Retrieval Method Using Color Distribution and Common Bitmap

             Springer, pp. 5671, 2005

         12. Chuen-Horng Lin, Rong-Tai Chen, Yung-Kuan Chan A smart content-based image retrieval system based on color and texture feature, Image and Vision Computing vol.27 pp. 658665,2009

         13. Deepika Nagthane Content Based Image Retrieval system Using K-Means Clustering Technique International Journal of Computer Applications & Information Technology, vol. 3, no.1 pp.21-30,2013

         14. Dengsheng Zhang, Aylwin Wong, Maria Indrawan, Guojun Lu

             Content-based Image Retrieval Using Gabor Texture Features,

             IEEE Pacific-Rim Conference,2000

         15. Dinakaran.D, J. Annapurna, Ch. Aswani Kumar Interactive Image Retrieval Using Text and Image Content Cybernetics and information technologies, vol. 10, no. 3 pp.20-30,2010

         16. Gupta A and Jain R Visual information retrieval,

             Communications of the ACM 40 (5), 7079. 1997

         17. Gonzalez R. C and Richard E.W. , Digital Image Processing, Prentice Hall. 2001.

         18. Horst Eidenberger and Christian Breiteneder Semantic feature layers in content based image retrieval implementation of world features.2004

         19. Jagadish.H.V Retrieval technique for similar Shapes ACM,1991

         20. Jayaprabha.P and Rm.Somasundaram Content Based Image Retrieval Methods Using Graphical Image Retrieval Algorithm (GIRA) International Journal of Information and Communication Technology Research,vol.2 no.1 pp.22-28,2012

         21. James. H, S. Harpreet, W. Equits, M. Flickner and W. Niblack, Efficient Color Histogram Indexing for Quadratic Form Distance Functions, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 17, No. 7, 1995

         22. John P.Eakins , Towards Intelligent image retrieval patternrecognition vol.35 pp.3-14 ,2002

         23. Julia Vogel and Bernt Scheele Semantic modeling of natural scenes for content based Image Retrieval International Journal of Computer Vision,2006

         24. Junding Sun, Ximin Zhang, Jiangtao Cui, Lihua Zhou Image retrieval based on color distribution entropy Pattern Recognition Letters vol.27 pp.11221126,2006

         25. Katerina Pastra, Horacio Saggion and Yorick Wilks NLP for Indexing and Retrieval of Captioned Photographs pp.143-146,2004

         26. Kekre H.B, Sudeep D. Thepade, Tanuja K. Sarode and Shrikant P. Sanas Image retrieval using texture features extracted using LBG, KPE, KFCG, KMCG, KEVR with assorted Color spaces International Journal of Advances in Engineering & Technology,

             Vol. 2, Issue 1, pp. 520-531,2012

         27. Kristina Lidayova and Elena sikudova Semantic categorization and retrieval of Natural scene Images Proceedings of CESCG 2012: The 16th Central European Seminar on Computer Graphics,2012

         28. Murali Krishna Raja.N.V, K.Shirin Bhanu Content Bases Image Search And Retrieval Using Indexing By K Means Clustering Technique International Journal of Advanced Research in Computer and Communication Engineering vol. 2, no.5 pp.2181-2189,2013

         29. Dr. V. Radha , K.Tamil Selvi A framework for group based image retrieval and video Annotation Journal of Global Research in Computer Science, Volume 4, No. 12, December 2013

         30. Nikhil Rasiwasia, Nuno Vasconcelos and Pedro J.Moreno Bridging the gap: Query by Semantic Example IEEE Transactions on Multimedia vol. 9 no.5, pp.923-938,2007

         31. Pushpa B. Patil and Manesh B. Kokare Interactive Semantic Image Retrieval Journal Information Process Systems, Vol.9, No.3, pp.349-364,2013

         32. Renato O. Stehling, Mario A. Nascimento, Alexandre X. Falc ,

             On "Shapes" of Colors for Content-Based Image Retrieval

             ACM, pp.171-174,2000

         33. Roung Shiunn Wu and Wen Hsien Hsu A Semantic Image Retrieval Framework Based on Ontology and Naïve Bayesian Inference International Journal of Multimedia Technology, vol.2 .no2 pp..36-43,2012

         34. Sonali Jain and Satyam Shrivastav A novel approach for image classification in Content based image retrieval using support vector

             machine International Journal of Computer Science & Engineering Technology. vol. 4 no. 03 pp.223-227,2013

         35. Tatsuya harada, Hideki nakayama, Yasuo kuniyoshi and Nobuyuki otsu Image Annotation and Retrieval for Weakly Labeled Images Using Conceptual Learning New Generation Computing, vol.28 pp.277-298,2010

         36. Thanos Athanasiadis, Yannis Avrithis, Member, and Stefanos Kollias

             Semantic Image Segmentation and Object Labeling IEEE transactions on circuits and systems for video technology,vol.17 no.3 pp.283-311,2007

         37. Tobias Weyand Combining Content-based Image Retrieval with Textual, Information Retrieval, Research project,2005

         38. Tony Rose, David Elworthy, Aaron Kotcheff and Amanda Clare

             ANVIL: a System for the Retrieval of Captioned Images using NLP Techniques Challenge of Image Retrieval,2000

         39. Ying Liu, Dengsheng Zhang, Guojun Lu, Wei-Ying Ma A survey of content based image retrieval with high level semantics Elsevier pattern Recognition vol.40 pp.262-282,2007

         40. S. Y. Irianto, Aamer Mohamed, J. Jiang, Integrated Keywords and Image Content features For Image Indexing and Retrieval image within Compressed Domain, Proceeding of 7th Informatics Workshop for Research.2007,211-214,Bradford , UK.

         41. Alqasrawi Y., Neagu D. and Cowling P.I. (2011): "Fusing Integrated Visual Vocabularies-Based Bag of Visual Words and Weighted Colour Moments on Spatial Pyramid Layout for Natural Scene Image Classification" Signal, Image and Video Processing, Springer.

             http://www.springerlink.com/content/b06h4p10u7t1441/

         42. Kurtz C, Beaulieu CF, Napel S, Rubin DL, A hierarchical knowledge-based approach for retrieving similar medical images described with semantic annotations. Elsevier Inc, 2014

         43. Liang-Chieh Chen, Sanja Fidler,Alan L. Yuille, Raquel Urtasun,

             Beat the MTurkers: Automatic Image Labeling from Weak 3D Supervision

         44. Claudio Cusano; Gianluigi Cioccaand Raimondo Schettini "Image annotation using SVM", Proc. SPIE 5304, Internet Imaging V, 330 (December 22, 2003); doi:10.1117/12.526746; http://dx.doi.org/10.1117/12.526746.

         45. Valle, E.; Cord, M., "Advanced Techniques in CBIR: Local Descriptors, Visual Dictionaries and Bags of Features," Computer Graphics and Image Processing (SIBGRAPI TUTORIALS), 2009 Tutorials of the XXII Brazilian Symposium on , vol., no., pp.72,78, 11-14 Oct. 2009

         46. Santhana Krishnamachari , Mohamed Abdel-mottaleb , Hierarchical clustering algorithm for fast image retrieval,In IS&T/SPIE Conference on Storage and Retrieval for Image and Video databases VII,1999