Model-based testing through a GUI

Model-BasedTestingThroughaGUIAnttiKervinen1,MikaMaunumaa1,TuulaPääkkönen2,andMikaKatara11TampereUniversityofTechnology,InstituteofSoftwareSystemsP.O.Box553,FI-33101Tampere,FINLAND{firstname.lastname}@tut.fi2NokiaTechnologyPlatformsP.O.Box68,FI-33721Tampere,FINLANDAbstract.Sofar,model-basedtestingapproacheshavemostlybeenusedintest-ingthroughvariouskindsofAPIs.Inpractice,however,testingthroughaGUIisanotherequallyimportantapplicationarea,whichintroducesnewchallenges.Inthispaper,weintroduceanewmethodologyformodel-basedGUItesting.ThisincludesusingLabeledTransitionSystems(LTSs)inconjunctionwithac-tionwordandkeywordtechniquesfortestmodeling.Wehavealsoconductedanindustrialcasestudywherewetestedamobiledeviceandwereabletondprevi-ouslyunreporteddefects.ThetestenvironmentincludedastandardMSWindowsGUItestingtoolaswellascomponentsimplementingourapproach.Assessmentoftheresultsfromanindustrialpointofviewsuggestsdirectionsforfuturede-velopment.1IntroductionSystemtestingthroughaGUIcanbeconsideredasoneofthemostchallengingtypesoftesting.Itisoftendonebyaseparatetestingteamofdomainexpertsthatcanvalidatethattheclients'requirementshavebeenfullled.However,thedomainexpertsoftenlackprogrammingskillsandrequireeasy-to-usetoolstosupporttheirwork.Comparedtoapplicationprogramminginterface(API)testing,GUItestingismademorecomplexbythevarioususerinterfaceissuesthatneedtobedealtwith.Suchissuesincludein-putofusercommandsandinterpretationoftheoutputresults,forinstance,usingtextrecognitioninsomecases.DevelopersareoftenreluctanttoimplementsystemlevelAPIsonlyforthepurposesoftesting.Moreover,general-purposetestingtoolsneedtobeadaptedtousesuchAPIs.Incontrast,aGUIisoftenavailableandthereareseveralgeneral-purposeGUItest-ingtools,whichcanbeeasilytakenintouse.Amongthetestautomationcommunity,however,GUItestingtoolsarenotconsideredanoptimalsolution.Thisislargelyduetobadexperiencesinusingso-calledcapture/replaytoolsthatcapturekeypresses,aswellasmousemovement,andreplaythoseinregressiontests.Thebadexperiencesaremostlyinvolvedwithhighmaintenancecostsassociatedwithsuchatool[1].TheGUIisoftenthemostvolatilepartofthesystemandpossiblechangestoitaffecttheGUItestautomationscripts.Intheworstcase,theselectedcapture/replaytoolusesbitmapcomparisonstoverifytheresultsofthetestruns.Falsenegativeresultscanthenbeobtainedfromminorchangesinthelookandfeelofthesystem.Inpractice,suchtestautomationneedsmaintenancewhenevertheGUIischanged.ThestateoftheartinGUItestingisrepresentedbyso-calledkeywordandactionwordtechniques[2,3].Theyhelpinmaintenanceproblemsbyprovidingaclearsepa-rationofconcernsbetweenbusinesslogicandtheGUInavigationneededtoimplementthelogic.Keywordscorrespondtokeypressesandmenunavigation,suchas“clicktheOKbutton”,whileactionwordsdescribeusereventsatahigherlevelofabstraction.Forinstance,asingleactionwordcanbedenedtoopenaselectedlewhosenamecanbegivenasaparameter.Theideaisthatdomainexpertscandesignthetestcaseseasilyusingactionwordsevenbeforethesystemimplementationhasbeenstarted.TestautomationengineersthendenethekeywordsthatimplementtheactionwordsusingthescriptinglanguageprovidedbytheGUIautomationtool.Althoughsometoolsusesmartercomparisontechniquesthanpurebitmaps,andprovideadvancedtestdesignconcepts,suchaskeywordsandactionwords,themain-tenancecostscanstillbesignicant.Moreover,suchtoolsseldomndnewbugsandreturntheinvestmentonlywhenthesametestsuitesarerunseveraltimes,suchasinregressiontesting.Thebasicproblemisinthestaticandlinearnatureofthetestcases.Evenifonly10%ofthetestcaseswouldneedtobeupdatedwheneverthesystemundertestchanges,thiscanmeanmodifyingonehundredtestcasesfromthetestsuiteofonethousandregressiontests.OurgoalistoimprovethestatusofGUItestingwithmodel-basedtechniques.Firstly,byusingtestmodelstogeneratetestruns,wewillnotrunintodifcultieswithmaintaininglargetestsuites.Secondly,wehavebetterchancesofndingpreviouslyundetecteddefects,sinceweareabletovarytheorderofevents.Towardstheseends,weproposeatestautomationapproachbasedonLabeledTransitionSystems(LTSs)aswellasactionwordsandkeywords.TheideaistodescribeatestmodelasaLTSwhosetransitionscorrespondtoactionwords.Thisshouldbemadeaseasyaspossibleforalsotesterswithnoprogrammingskills.Themaintenanceeffortshouldlocalizetoasinglemodelorfewcomponentmodels.Theactionmachinesweintroducearecomposedinparallelwithrenementmachinesmappingtheactionwordstosequencesofkeywords.TheresultingcompositeLTSisthenreadintoageneral-purposeGUItestingtoolthatinterpretsthekeywordsandwalksthroughthemodelusingsomeheuristics.Thetoolalsoveriesthetestresultsandhandlesthereporting.Thecontributionsofthispaperareinformalizingtheabovescheme,introducingnoveltestmodelarchitectureandapplyingtheapproachinanindustrialcasestudy.Finally,wehaveassessedtheresultsfromanindustrialpointofview.Therestofthepaperisstructuredasfollows.Sections2and3describeourapproachindetailaswellasthecasestudywehaveconducted.TheassessmentoftheresultsisgiveninSection4.RelatedworkisdiscussedinSection5andconclusionsdrawninSection6.2BuildingaTestModelArchitectureInthefollowing,wewilldevelopal