Visual tracking and recognition using probabilisti

VisualtrackingandrecognitionusingprobabilisticappearancemanifoldsKuang-ChihLeea,*,JeffreyHob,Ming-HsuanYangc,DavidKriegmanbaBeckmanInstituteandComputerScienceDepartment,UniversityofIllinoisatUrbana-Champaign,Urbana,IL61801,USAbComputerScienceandEngineeringDepartment,UniversityofCaliforniaatSanDiego,LaJolla,CA92093-0114,USAcHondaResearchInstitute,800CaliforniaStreet,MountainView,CA94041,USAReceived13August2004;accepted28February2005Availableonline12April2005AbstractThispaperpresentsanalgorithmformodeling,tracking,andrecognizinghumanfacesinvideosequenceswithinoneintegratedframework.Conventionalvideo-basedfacerecog-nitionsystemshaveusuallybeenembodiedwithtwoindependentcomponents:thetrackingandrecognitionmodules.Incontrast,ouralgorithmemphasizesanalgorithmicarchitecturethattightlycouplesthesetwocomponentswithinasingleframework.Thisisaccomplishedthroughanovelappearancemodelwhichisutilizedsimultaneouslybybothmodules,evenwiththeirdisparaterequirementsandfunctions.Thecomplexnonlinearappearancemani-foldofeachregisteredpersonispartitionedintoacollectionofsubmanifoldswhereeachmodelsthefaceappearancesofthepersoninnearbyposes.Thesubmanifoldisapproxi-matedbyalow-dimensionallinearsubspacecomputedbyprincipalcomponentanalysisusingimagessampledfromtrainingvideosequences.Theconnectivitybetweenthesubman-ifoldsismodeledastransitionprobabilitiesbetweenpairsofsubmanifolds,andthesearelearneddirectlyfromtrainingvideosequences.Theintegratedtaskoftrackingandrecog-nitionisformulatedasamaximumaposterioriestimationproblem.Withinourframe-work,thetrackingandrecognitionmodulesarecomplementarytoeachother,andthe1077-3142/$-seefrontmatter
2005ElsevierInc.Allrightsreserved.doi:10.1016/j.cviu.2005.02.002*Correspondingauthor.E-mailaddress:[e-mail](K.-C.Lee).(2005)303–331capabilityandperformanceofoneareenhancedbytheother.Ourapproachcontrastssharplywithmorerigidconventionalapproachesinwhichthesetwomodulesworkinde-pendentlyandinsequence.Wereportonanumberofexperimentsandresultsthatdem-onstratetherobustness,effectiveness,andstabilityofouralgorithm.
2005ElsevierInc.Allrightsreserved.Keywords:Facetracking;Facerecognition;PCA;Appearancemanifold1.IntroductionInthepastfewdecades,therehasbeenintensiveresearchandgreatstridesindesigninganddevelopingalgorithmsforfacerecognitionwithstillimages.Onlyuntilrecentlyhastheproblemoffacerecognitionwithvideosequencesstartedtoattracttheattentionoftheresearchcommunity[1–4].Thiscanbepartlyattributedtotherecentadvanceincomputerhardware.Inparticular,withlowcostcamerasandsuf-ficientlypowerfulpersonalcomputers,itisnowpossibletoinexpensivelyimplementareal-timefacetrackingsystem(e.g.[5,6])withgoodperformance.Thiscapabilityistheprerequisitefordevelopingreal-timevideofacerecognitionapplications.Comparedwithconventionalstillimagefacerecognition,videofacerecognitionoffersseveralchallengesandopportunity.First,thereisthe‘‘alignment’’problembe-tweenthetrackingandtherecognitionmodules.Avideo-basedfacerecognitionsys-teminvariablyhastwocomponents,i.e.,trackingandrecognitionmodules.Sincetrackingandrecognitionproblemshavebeenstudiedintensivelybutseparatelyinthepast,thesetwomodulesareusuallyimplementedindependentlyandworkinse-quence.Withoutanyalignmentbetweenthetwomodules,theimagesreturnedbythetrackergenerallyarenotbeingoodagreementwiththeappearancemodelusedbytherecognitionmodule,1i.e.,misalignedimages.Unfortunately,virtuallyallappear-ance-basedrecognitiontechniquesaresensitivetomisalignments.Therefore,somemechanismshouldbeinplacetoensurethattheimagesreturnedbythetrackingmodulecanbecorrectlyprocessedbytherecognitionmodule.Second,thereistheproblemofmodelingappearancevariationoffacesforboththetrackingandrecognitionmodules.Attheheartofanytrackingorrecognitionalgorithmisaninternalrepresentationwhichdefinestheallowablevariationinappearancesoftheobjecttobetrackedorrecognized.Factorssuchaschangesofviewpoint,shape(deformations,articulations),andillumination,individuallyorcombined,cancausesignificantimagevariationsinadynamicenvironment.(SeeFig.1).Forappearance-basedmethods,some(ifnotall)oftheseimagevari-ationsshouldbemodeledinordertoproducerobustresults.However,duetotheirdifferentmissions,trackingandrecognitionmodulesgenerallyplacedifferentemphasisandrequirementontheirinternalmodelorrepresentation.For1Inthispaper,ourmainfocusisonappearance(orimage)-basedrecognitionmethods.Forfacerecognition,ithasbeenargued[7,8]thatfeature-basedtechniquesaregenerallylessstableandaccurate.304K.-C.Leeetal./ComputerVisionandImageUnderstanding99(2005)303–331recognition,themodelisrequiredtoaccuratelycapturesubtledifferencesbetweentheappearancesofdifferentenrolledindividualsinordertocorrectlyrecognizethem.Fortracking,suchfinegranularityindetailisunnecessary.Instead,amodelthatcapturescommonimagefeaturesofhumanfacesispreferred,andperhapsmoreimportantly,themodelshouldbesimpleandefficientsothatthetrackingmodulecancompleteitstaskquickly.Therefore,indevelopingacombinedtrack-ingandrecognitionsystem,ourchallengeistodesigna