A diffusion-neural-network for learning from small

Adiffusion-neural-networkforlearningfromsmallsamplesqChongfuHuanga,*,ClaudioMoragabaInstituteofResourcesScience,BeijingNormalUniversity,19,XinjiekouwaiStreet,Beijing100875,ChinabDepartmentofComputerScience,ComputerEngineeringandComputingEducation,UniversityofDortmund,44221Dortmund,GermanyReceived1June2003;accepted1June2003AbstractNeuralinformationprocessingmodelslargelyassumethatthepatternsfortraininganeuralnetworkaresufficient.Otherwise,theremustexistanon-negligibleerrorbetweentherealfunctionandtheestimatedfunctionfromatrainednetwork.Toreducetheerror,inthispaper,wesuggestadiffusion-neural-network(DNN)tolearnfromasmallsampleconsistingofonlyafewpatterns.ADNNwithmorenodesintheinputandlayersistrainedbyusingthederivingpatternsinsteadoforiginalpatterns.Inthispaper,wegiveanexampletoshowhowtoconstructaDNNforrecognizinganon-linearfunction.Inourcase,theDNN’serrorislessthantheerroroftheconventionalBPnetwork,about48%.Tosubstantiatethespecialcasearguments,wealsostudyothertwonon-linearfunctionswithsimulationtechnology.TheresultsshowthattheDNNmodelisveryeffectiveinthecasewherethetargetfunctionhasastrongnon-linearityoragivensampleisverysmall.
2003ElsevierInc.Allrightsreserved.Keywords:Neuralnetwork;Non-linearfunction;Fuzzyinformation;InformationdiffusionqProjectsupportedbyaMercatorVisitingProfessorshipoftheGermanResearchSocietyDFG,grantedtoProf.ChongfuHuangattheUniversityofDortmund.*Correspondingauthor.Fax:[phone].E-mailaddresses:[e-mail],[e-mail](C.Huang).0888-613X/$-seefrontmatter
2003ElsevierInc.Allrightsreserved.doi:10.1016/j.ijar.2003.06.001(2004)137–1611.IntroductionArtificialneuralnetworkshavereceivedextensiveattentionduringthelasttwodecades.Itiswellknownthattheycansolvemanypracticalproblemsaspatternrecognition[21],functionapproximation[24],systemidentification[17],timeseriesforecasting,etc.[5,20].Neuro-fuzzymodelingisconcernedwiththeextractionofmodelsfromnumericaldatarepresentingthebehaviourofasystem.Themodelsinthiscasearerule-basedandusetheformalismoffuzzylogic,i.e.theyconsistsofsetsoffuzzy‘‘if-then’’ruleswithpossiblyseveralpremises[18].Neuralinformationprocessingmodelslargelyassumethat:(i)thepatternsarecompatible;(ii)thelearningpatternsfortraininganeuralnetworkaresufficient.Ifthepatternsarecontradictory,theneuralnetworkdoesnotconvergebecausetheadjustmentsofweightsandthresholdsdonotknowwheretoturn.In1996,HuangandRuan[13]usedtheinformationdiffusionmethod[7,8]andthefallingshadowtheory[25]toconstructaninformationdiffusionnetwork(IDN)basedonBPalgorithmtosolvetheproblemofcontradictorypatterns.AnIDNalwaysconverges.ForeveryresultofIDNmethod,itsreliabilitycanbeanalysedconveniently.In1999,HuangandLeung[12]suggestedahybridfuzzy-neural-networktoestimatetherelationshipbetweenisoseismalareaandearthquakemagnitude.Inthemodel,theinformationdiffusionmethodisemployedtoconstructfuzzy‘‘if-then’’rulesasmanyasthegivenobservations.Integratingtherulestoformaninformation-diffusion-approximate-reasoningestimator(IDAR),wecanchangecontradictorypatternstobecompatiblefortrainingaBPnetwork.Thehybrid-modelestimatorismoreprecisethanthelinear-regressionestimator,andmorestablethantheconventionalBP-neural-networkestimator.Thesetwohybridmodelsputforwardthecasethatcon-tradictorypatternscanbelearnedbyneuralnetworks.Inotherwords,withthehelpoftheinformationdiffusiontechnique[10,14],wehaveresolvedtheproblemrelatedtocontradictorypatterns.Ifthelearningpatternsareinsufficient,itisimpossibletorecognizeanon-linearsystem,i.e,theremustexistanon-negligibleerrorbetweentherealfunctionandtheestimatedfunctionfromatrainednetwork.Developingtheinformationdiffusiontechnique,inthispaper,wesuggestanotherhybridmodeltoreducetheerrorofestimatedfunctionfromaBPnetworktrainedbyasmallsample.2.ConventionalBPnetworkstrainedbysmallsamplesAneuralnetworkcanbeunderstood[16]asamappingf:Rp!Rq,definedby138C.Huang,C.Moraga/Internat.J.Approx.Reason.35(2004)137–161y¼fðxÞ¼uðWxÞ;wherex2Rp(Rp¼Rp,Risthesetofrealnumbers)istheinputvector,y2Rqistheoutputvector.TheweightmatrixWisap
qmatrixanduisanon-linearfunctionthatisoftenreferredtoastheactivationfunction.Thetypicalacti-vationfunctionistheSigmoidfunctionuðxÞ¼11þeax;a0:Themappingfcanbedecomposedintoachainingofmappings;theresultisamulti-layernetworkRp!Rm!!Rn!Rq:ThealgorithmforcomputingWisoftencalledthetrainingalgorithm.Themostpopularneuralnetworksarethemulti-layerback-propagationnetworkswhosetrainingalgorithmisthewell-knowngradientdescentmethod.SuchnetworksarecalledBPnetworks.AconventionalBPnetwork(CBPN)consistsofaninputlayer(thefirstlayer),anoutputlayer(thelastlayer),andsomehiddenlayers.Torecognizeafunctionwithpinputvariablesandqoutputvariable,ingeneral,wesetpnodesintheinputlayerandqnodesintheoutputlayer.Inotherwords,thenumberofnodesinthefirstandlastlayerisjustequaltothenumberofinputandoutputvariables,respectively.Relationshipsbetweenvariablesaremostoftenrecognizedbylearningneuralnetworkswithdataorpatternscollected.Theapproachisalsocalledadaptivepatternrecogniti