Vision based gesture recognition within a linguist

VisionBasedGestureRecognitionwithinaLinguisticsFrameworkKonstantinosG.DerpanisAthesissubmittedtotheFacultyofGraduateStudiesinpartialfulllmentoftherequirementsforthedegreeofMasterofScienceGraduatePrograminComputerScienceYorkUniversityToronto,OntarioMay2003.iiiiiiAbstractAnapproachtorecognizinghumanhandgesturesfromamonoculartemporalse-quenceofcolourimagesispresented.OfparticularconcerninthisthesisistherepresentationandrecognitionofhandmovementsthatareusedinsinglehandedAmericanSignLanguage(ASL).Theapproachpresentedexploitspreviouslinguis-ticanalysisofmanuallanguagesthatdecomposedynamicgesturesintotheirstaticanddynamiccomponents.Therstlevelofdecompositionisintermsofthreesetsofprimitives,handshape,locationandmovement.Furtherlevelsofdecompositioninvolvethelexicalandsentencelevelsandarepartofourplanforfuturework.Weproposeandsubsequentlydemonstratethatgivenamonoculargesturesequence,kine-maticfeaturescanberecoveredfromtheapparentmotionthatprovidedistinctivesignaturesfor14single-handedrigidmovementsofASL.Theapproachhasbeenim-plementedinsoftwareandevaluatedonadatabaseof592gesturesequenceswithanoverallrecognitionrateof97.13%.ivAcknowledgmentsIwouldrstliketoexpressmysincereappreciationtoJohnTsotsosandRichardWildes,mysupervisors,fortheirenthusiasm,supportandinsightfulguidance.Ihopeourfuturecollaborationwillbeasrewardingasithasbeeninthepast.IthankBillKapralos,MarkusLatzel,DanZikovitzandAndrewHoguefortheirfriendshipandstimulatingconversationbothacademicandpersonal.AlsospecialthankstoAntoniaVezosforprovidingtheillustrationscontainedinthisthesisandforherloveandsupport.LastbutnotleastIwouldliketothankmyparentsGeorgeandKathyfortheirlove,supportandguidance.Withoutyounoneofthiswouldbepossible.Thisthesisisdedicatedtoyou.vContentsAbstractivAcknowledgmentsv1Introduction11.1Motivation.................................11.2Relatedresearch..............................21.3Contributions...............................101.4Outlineofthesis..............................112TechnicalApproach122.1Generaltechnicalframework.......................122.2Linguisticsbasis..............................152.3Strategy..................................192.4Idealizedgestureexecutions.......................202.5Coloursegmentation...........................312.6Anemotionestimation.........................352.7Kinematicfeatures............................422.8Prototypegesturesignatures.......................46vi2.9Recapitulation...............................503ExperimentalEvaluation513.1Frontoparallelexperiment........................513.1.1Experimentaldesign.......................513.1.2Results...............................563.1.3Discussion.............................593.2Attitudeexperiment...........................603.2.1Experimentaldesign.......................603.2.2Results...............................633.2.3Discussion.............................693.3OverallDiscussion............................694Conclusion714.1Summary.................................714.2Futurework................................72AVelocityDescription74A.1Worldvelocitydescription........................74A.2ImagevelocitydescriptionI.......................76A.3ImagevelocitydescriptionII.......................78A.4First-orderaccuracy...........................79A.5Kinematicparameterdenitions.....................86BHandLocalization88viiCExampleSequences95Bibliography110viiiListofTables1.1Summaryofautomatedhandgesturerecognitionwork........81.2Summaryofautomatedhandgesturerecognitionwork,continued..92.1Mappingsofthenon-periodicmovementsintheworldspacetokine-maticquantitiesintheimagespace...................292.2Mappingsoftheperiodicmovementsintheworldspacetokinematicquantitiesintheimagespace.......................302.3Gesturesignatures............................463.1Toptwocandidateclassicationtable..................573.2Gesturemovementrecognitionresults..................583.3Toptwocandidateclassicationtable,atattitudes15,15,30and30....................................643.4Gesturemovementrecognitionresultsat15..............653.5Gesturemovementrecognitionresultsat15.............663.6Gesturemovementrecognitionresultsat30..............673.7Gesturemovementrecognitionresultsat30.............68ixListofFigures2.1Systemframework............................142.2Stokoe'sphonemicanalysisofASL...................182.3Cameracoordinatesystem........................212.4SkincolourdistributioninnormalizedRGspace............322.5Binaryskinmap.............................342.6Diagramofthehierarchicalmotionestimationframework.......392.7Geman-McClureerrorfunction.....................413.1Sideviewofourexperimentalsetup...................543.2Circularmovementexample.......................553.3Cameraviewofavolunteeratdierentattitudes............62A.1Aneerror(nodmovement).......................84A.2Aneversusquadraticow.......................85B.1AsummaryofconstructingLaplacianandGaussianpyramids....89B.2Localizationexample1..........................92xB.3Localizationexample2..............