Generative-Adversarial-Networks-(GANs)--2016-12-04

GenerativeAdversarialNetworks(GANs)IanGoodfellow,OpenAIResearchScientistNIPS2016tutorialBarcelona,2016-12-4(Goodfellow2016)GenerativeModeling•Densityestimation•SamplegenerationTrainingexamplesModelsamples(Goodfellow2016)Roadmap•Whystudygenerativemodeling?•Howdogenerativemodelswork?HowdoGANscomparetoothers?•HowdoGANswork?•Tipsandtricks•Researchfrontiers•CombiningGANswithothermethods(Goodfellow2016)Whystudygenerativemodels?•Excellenttestofourabilitytousehigh-dimensional,complicatedprobabilitydistributions•SimulatepossiblefuturesforplanningorsimulatedRL•Missingdata•Semi-supervisedlearning•Multi-modaloutputs•Realisticgenerationtasks(Goodfellow2016)NextVideoFramePredictionCHAPTER15.REPRESENTATIONLEARNINGGroundTruthMSEAdversarialFigure15.6:Predictivegenerativenetworksprovideanexampleoftheimportanceoflearningwhichfeaturesaresalient.Inthisexample,thepredictivegenerativenetworkhasbeentrainedtopredicttheappearanceofa3-Dmodelofahumanheadataspecificviewingangle.(Left)Groundtruth.Thisisthecorrectimage,thatthenetworkshouldemit.(Center)Imageproducedbyapredictivegenerativenetworktrainedwithmeansquarederroralone.Becausetheearsdonotcauseanextremedifferenceinbrightnesscomparedtotheneighboringskin,theywerenotsufficientlysalientforthemodeltolearntorepresentthem.(Right)Imageproducedbyamodeltrainedwithacombinationofmeansquarederrorandadversarialloss.Usingthislearnedcostfunction,theearsaresalientbecausetheyfollowapredictablepattern.Learningwhichunderlyingcausesareimportantandrelevantenoughtomodelisanimportantactiveareaofresearch.FiguresgraciouslyprovidedbyLotteretal.(2015).recognizableshapeandconsistentpositionmeansthatafeedforwardnetworkcaneasilylearntodetectthem,makingthemhighlysalientunderthegenerativeadversarialframework.Seefigure15.6forexampleimages.Generativeadversarialnetworksareonlyonesteptowarddeterminingwhichfactorsshouldberepresented.Weexpectthatfutureresearchwilldiscoverbetterwaysofdeterminingwhichfactorstorepresent,anddevelopmechanismsforrepresentingdifferentfactorsdependingonthetask.Abenefitoflearningtheunderlyingcausalfactors,aspointedoutbySchölkopfetal.(2012),isthatifthetruegenerativeprocesshasxasaneffectandyasacause,thenmodelingp(x|y)isrobusttochangesinp(y).Ifthecause-effectrelationshipwasreversed,thiswouldnotbetrue,sincebyBayes’rule,p(x|y)wouldbesensitivetochangesinp(y).Veryoften,whenweconsiderchangesindistributionduetodifferentdomains,temporalnon-stationarity,orchangesinthenatureofthetask,thecausalmechanismsremaininvariant(thelawsoftheuniverseareconstant)whilethemarginaldistributionovertheunderlyingcausescanchange.Hence,bettergeneralizationandrobustnesstoallkindsofchangescan545(Lotteretal2016)(Goodfellow2016)SingleImageSuper-Resolution(Ledigetal2016)(Goodfellow2016)iGANyoutube(Zhuetal2016)(Goodfellow2016)IntrospectiveAdversarialNetworksyoutube(Brocketal2016)(Goodfellow2016)ImagetoImageTranslationInputGroundtruthOutputInputGroundtruthOutputFigure13:Exampleresultsofourmethodonday!night,comparedtogroundtruth.InputGroundtruthOutputInputGroundtruthOutputFigure14:Exampleresultsofourmethodonautomaticallydetectededges!handbags,comparedtogroundtruth.(Isolaetal2016)Image-to-ImageTranslationwithConditionalAdversarialNetworksPhillipIsolaJun-YanZhuTinghuiZhouAlexeiA.EfrosBerkeleyAIResearch(BAIR)LaboratoryUniversityofCalifornia,Berkeley{isola,junyanz,tinghuiz,efros}@eecs.berkeley.eduLabelstoFacadeBWtoColorAerialtoMapLabelstoStreetSceneEdgestoPhotoinputoutputinputinputinputinputoutputoutputoutputoutputinputoutputDaytoNightFigure1:Manyproblemsinimageprocessing,graphics,andvisioninvolvetranslatinganinputimageintoacorrespondingoutputimage.Theseproblemsareoftentreatedwithapplication-specificalgorithms,eventhoughthesettingisalwaysthesame:mappixelstopixels.Conditionaladversarialnetsareageneral-purposesolutionthatappearstoworkwellonawidevarietyoftheseproblems.Hereweshowresultsofthemethodonseveral.Ineachcaseweusethesamearchitectureandobjective,andsimplytrainondifferentdata.AbstractWeinvestigateconditionaladversarialnetworksasageneral-purposesolutiontoimage-to-imagetranslationproblems.Thesenetworksnotonlylearnthemappingfrominputimagetooutputimage,butalsolearnalossfunc-tiontotrainthismapping.Thismakesitpossibletoapplythesamegenericapproachtoproblemsthattraditionallywouldrequireverydifferentlossformulations.Wedemon-stratethatthisapproachiseffectiveatsynthesizingphotosfromlabelmaps,reconstructingobjectsfromedgemaps,andcolorizingimages,amongothertasks.Asacommu-nity,wenolongerhand-engineerourmappingfunctions,andthisworksuggestswecanachievereasonableresultswithouthand-engineeringourlossfunctionseither.Manyproblemsinimageprocessing,computergraphics,andcomputervisioncanbeposedas“translating”aninputimageintoacorrespondingoutputimage.JustasaconceptmaybeexpressedineitherEnglishorFrench,ascenemayberenderedasanRGBimage,agradientfield,anedgemap,asemanticlabelmap,etc.Inanalogytoautomaticlanguagetranslation,wedefineautomaticimage-to-imagetranslationastheproblemoftranslatingonepossiblerepresentationofasceneintoanother,givensufficienttrainingdata(seeFig-ure1).Onereas