Abstract Features for Image Retrieval An Experimen

FeaturesforImageRetrieval:AnExperimentalComparisonThomasDeselaers1,DanielKeysers2,andHermannNey11HumanLanguageTechnologyandPatternRecognition,ComputerScienceDepartment,RWTHAachenUniversity,Germany{deselaers,ney}@cs.rwth-aachen.de2ImageUnderstandingandPatternRecognition,GermanResearchCenterforArtificialIntelligence(DFKI),Kaiserslautern,[e-mail]29,2007AbstractAnexperimentalcomparisonofalargenumberofdifferentimagedescriptorsforcontent-basedimageretrievalispre-sented.Manyofthepapersdescribingnewtechniquesanddescriptorsforcontent-basedimageretrievaldescribetheirnewlyproposedmethodsasmostappropriatewithoutgivinganin-depthcomparisonwithallmethodsthatwereproposedearlier.Inthispaper,wefirstgiveanoverviewofalargevari-etyoffeaturesforcontent-basedimageretrievalandcomparethemquantitativelyonfourdifferenttasks:stockphotore-trieval,personalphotocollectionretrieval,buildingretrieval,andmedicalimageretrieval.Fortheexperiments,fivedif-ferent,publiclyavailableimagedatabasesareusedandtheretrievalperformanceofthefeaturesisanalysedindetail.Thisallowsforadirectcomparisonofallfeaturesconsid-eredinthisworkandfurthermorewillallowacomparisonofnewlyproposedfeaturestotheseinthefuture.Additionally,thecorrelationofthefeaturesisanalysed,whichopensthewayforasimpleandintuitivemethodtofindaninitialsetofsuitablefeaturesforanewtask.Thearticleconcludeswithrecommendationswhichfeaturesperformwellforwhattypeofdata.Interestingly,theoftenused,butverysimple,colourhistogramperformswellinthecomparisonandthuscanberecommendedasasimplebaselineformanyapplications.1IntroductionImageretrievalingeneralandcontent-basedimageretrieval(CBIR)inparticulararewell-knownfieldsofresearchinin-formationmanagementinwhichalargenumberofmethodshavebeenproposedandinvestigatedbutinwhichstillnosatisfyinggeneralsolutionsexist.Theneedforadequateso-lutionsisgrowingduetotheincreasingamountofdigitallyproducedimagesinareaslikejournalism,medicine,andpri-vatelife,requiringnewwaysofaccessingimages.Forexam-ple,medicaldoctorshavetoaccesslargeamountsofimagesdaily[1],home-usersoftenhaveimagedatabasesofthousandsofimages[2],andjournalistsalsoneedtosearchforimagesbyvariouscriteria[3,4].Inthepast,severalCBIRsystemshavebeenproposedandallthesesystemshaveonethingincommon:imagesarerepresentedbynumericvalues,calledfeaturesordescriptors,thataremeanttorepresenttheprop-ertiesoftheimagestoallowmeaningfulretrievalfortheuser.OnlyrecentlyhavesomestandardbenchmarkdatabasesandevaluationcampaignsbeencreatedwhichallowforaquantitativecomparisonofCBIRsystems.Thesebench-marksallowforthecomparisonofimageretrievalsystemsunderdifferentaspects:usabilityanduserinterfaces,combi-nationwithtextretrieval,oroverallperformanceofasystem.However,toourknowledge,noquantitativecomparisonofthebuildingblocksofthesystems,thefeaturesthatareusedtocompareimages,hasbeenpresentedsofar.In[5]amethodforcomparingimageretrievalsystemswasproposedrelyingontheCoreldatabase,whichhasrestrictedcopyrights,isnolongercommerciallyavailabletoday,andcanthereforenotbeusedforexperimentsthataremeanttobeabasisforothercomparisons.AnotheraspectofevaluatingCBIRsystemsarethere-quirementsoftheusers.In[3]and[4]studiesofuserneedsinsearchingimagearchivesarepresentedandtheoutcomeinbothstudiesisthatCBIRaloneisveryunlikelytoful-filltheneedsbutthatsemanticinformationobtainedfrommetadataandtextualinformationisanimportantadditionalknowledgesource.Althoughtodaythesemanticanalysisandunderstandingofimagesismuchfurtherdevelopedduetotherecentachievementsinobjectdetectionandrecognition,stillmostoftherequirementsspecifiedarenotsatisfiablefullyautomatically.Therefore,inthispaperwecomparetheper-formanceofalargevarietyofvisualdescriptors.Thesecanthenlaterbecombinedwiththeoutcomeoftextualinforma-tionretrievalasdescribede.g.in[6].1Themainquestionweaddressinthispaperis:Whichfea-turesaresuitableforwhichtaskinimageretrieval?Thisquestionisthoroughlyinvestigatedbyexaminingtheperfor-manceofawidevarietyofdifferentvisualdescriptorsforfourdifferenttypesofCBIRtasks.Thequestionofwhichfeaturesperformhowwelliscloselyrelatedtothequestionwhichfeaturescanbecombinedtoobtaingoodresultsinaparticulartask.Althoughwedonotdirectlyaddressthisquestionhere,theresultsfromthispaperleadtoanewandintuitivemethodtochooseanap-propriatecombinationoffeaturesbasedonthecorrelationoftheindividualfeatures.Fortheevaluationofthefeaturesweusefivedifferentpub-liclyavailabledatabaseswhichareagoodstartingpointtoevaluatetheperformanceofnewimagedescriptors.AlthoughtodayvariousinitiativesforevaluationofCBIRsystemshaveevolved,onlyfewofthemresultedinevaluationcampaignswithparticipantsandresults:Benchathlon1wasstartedin2001andlocatedattheSPIEElectronicImagingconferencebuthasbecomesmallerovertime.TRECVID2isaninitiativebytheTREC(TextRetrievalConference)onvideoretrievalinwhichvideoretrievalsystemsarecom-pared.ImageCLEF3ispartoftheCross-LanguageEvalua-tionFramework(CLEF)andstartedin2003withonlyonetaskaimingatacombinationofmulti-lingualinformationre-trievalwithCBIR.In2004,itcomprisedthreetasks,oneofthemfocusedonvisualqueri