Evolutionary learning of rich neural networks in the Bayesian model selection framework

• Autorzy: Matteo Matteucci , Dario Spadoni
• Języki publikacji: EN

Abstrakty

EN

In this paper we focus on the problem of using a genetic algorithm for model selection within a Bayesian framework. We propose to reduce the model selection problem to a search problem solved using evolutionary computation to explore a posterior distribution over the model space. As a case study, we introduce ELeaRNT (Evolutionary Learning of Rich Neural Network Topologies), a genetic algorithm which evolves a particular class of models, namely, Rich Neural Networks (RNN), in order to find an optimal domain-specific non-linear function approximator with a good generalization capability. In order to evolve this kind of neural networks, ELeaRNT uses a Bayesian fitness function. The experimental results prove that ELeaRNT using a Bayesian fitness function finds, in a completely automated way, networks well-matched to the analysed problem, with acceptable complexity.

Słowa kluczowe

EN

Bayesian fitness; Bayesian model selection; genetic algorithms; Rich Neural Networks

• Wydawca: University of Zielona Gora Press
• Czasopismo: International Journal of Applied Mathematics and Computer Science
• Rocznik: 2004
• Tom: 14
• Numer: 3
• Strony: 423-440

Daty

wydano

2004

otrzymano

2004-03-01

poprawiono

2004-06-01

Twórcy

autor

Matteo Matteucci

• Department of Electronics and Information Politecnico di Milano, Piazza L. da Vinci 32, 20133 Milan, Italy

autor

Dario Spadoni

• ALaRI (Advanced Learning and Research Institute), University of Lugano, Lugano, Switzerland

Bibliografia

• Angeline P.J. (1994): Genetic Programming and Emergent Intelligence, In: Advances in Genetic Programming (Jr. Kinnear and E. Kenneth, Eds.). - Cambridge, MA: MIT Press, pp. 75-98.
• Bebis G., Georgiopoulos M. and Kasparis T. (1997): Coupling weight elimination with genetic algorithms to reduce network size and preserve generalization. - Neurocomput., Vol. 17, No. 3-4, pp. 167-194.
• Bernardo J.M. and Smith A.F.M. (1994): Bayesian Theory. - New York: Wiley.
• Bishop C.M. (1995): Neural Networks for Pattern Recognition. - Oxford: Oxford University Press.
• Castellano G., Fanelli A.M. and Pelillo M. (1997): An iterative pruning algorithm for feedforward neural networks. - IEEE Trans. Neural Netw., Vol. 8, No. 3, pp. 519-531.
• Chib S. and Greenberg E. (1995): Understanding the Metropolis-Hastings algorithm. -Amer. Stat., Vol. 49, No. 4, pp. 327-335.
• Denison D.G.T., Holmes C.C., Mallick B.K. and Smith A.F.M. (2002): Bayesian Methods for Nonlinear Classification and Regression. - New York: Wiley.
• Dudzinski M.L. and Mykytowycz R. (1961): The eye lens as an indicator of age in the wild rabbit in Australia. - CSIRO Wildlife Res., Vol. 6, No. 1, pp. 156-159.
• Flake G.W. (1993): Nonmonotonic activation functions in multilayer perceptrons. - Ph.D. thesis, Dept. Comput. Sci., University of Maryland, College Park, MD.
• Fletcher R. (1987): Practical Methods of Optimization. - New York: Wiley.
• Goldberg D.E. (1989): Genetic Algorithms in Search, Optimization, and Machine Learning.Reading, MA: Addison-Wesley.
• Gull S.F. (1989): Developments in maximum entropy data analysis, In: Maximum Entropy and Bayesian Methods, Cambridge 1998 (J. Skilling, Ed.). - Dordrecht: Kluwer, pp. 53-71.
• Hancock P.J.B. (1992): Genetic algorithms and permutation problems: A comparison of recombination operators for neural net structure specification. - Proc. COGANN Workshop, Int. Joint Conf. Neural Networks, Piscataway, NJ, IEEE Computer Press, pp. 108-122.
• Hashem S. (1997): Optimal linear combinations of neural networks. - Neural Netw., Vol 10, No. 4, pp. 599-614.
• Hassibi B. and Stork D.G. (1992): Second order derivatives for network pruning: Optimal Brain Surgeon, In: Advances in Neural Information ProcessingSystems (S.J. Hanson, J.D. Cowan and C. Lee Giles, Eds.). -San Matteo, CA: Morgan Kaufmann, Vol. 5, pp. 164-171.
• Hastings W.K. (1970): Monte Carlo sampling methods using Markov chains and their applications. - Biometrika, Vol. 57, pp. 97-109.
• Haykin S. (1999): Neural Networks. A Comprehensive Foundation (2nd Edition). - New Jersey: Prentice Hall.
• Hoeting J., Madigan D., Raftery A. and Volinsky C. (1998): Bayesian model averaging. - Tech. Rep. No. 9814, Department of Statistics, Colorado State University.
• Hornik K.M., Stinchcombe M. and White H. (1989): Multilayer feedforward networks are universal approximators.- Neural Netw., Vol. 2, No. 5, pp. 359-366.
• Liu Y. and Yao X. (1996): A population-based learning algorithm which learns both architectures and weights of neural networks.- Chinese J. Adv. Softw. Res., Vol. 3, No. 1, pp. 54-65.
• Lovell D. and Tsoi A. (1992): The performance of the neocognitron with various s-cell and c-cell transfer functions. - Tech. Rep., Intelligent Machines Laboratory, Department of Electrical Engineering, University of Queensland.
• MacKay D.J.C. (1992): A practical Bayesian framework for back propagation networks. - Neural Comput., Vol. 4, No. 3, pp. 448-472.
• MacKay D.J.C. (1995): Probable networks and plausible predictions - a review of practical Bayesian methods for supervised neural networks. - Netw. Comput. Neural Syst., Vol. 6, No. 3, pp. 469-505.
• MacKay D.J.C. (1999): Comparison of approximate methods for handling hyperparameters. - Neural Comput., Vol. 11, No. 5, pp. 1035-1068.
• Mani G. (1990): Learning by gradient descent in function space. - Tech. Rep. No. WI 52703, Computer Sciences Department, University of Winsconsin, Madison, WI.
• Matteucci M. (2002a): ELeaRNT: Evolutionary learning of rich neural network topologies. - Tech. Rep. No. CMU-CALD-02-103, Carnegie Mellon University, Pittsburgh, PA.
• Matteucci M. (2002b): Evolutionary learning of adaptive models within a Bayesian framework. - Ph.D. thesis, Dipartimento di Elettronica e Informazione, Politecnico di Milano.
• Montana D.J. and Davis L. (1989): Training feedforward neural networks using genetic algorithms. - Proc. 3rd Int. Conf. Genetic Algorithms, San Francisco, CA, USA, pp. 762-767.
• Pearlmutter B.A. (1994): Fast exact multiplication by the Hessian. - Neural Comput., Vol. 6, No. 1, pp. 147-160.
• Press W.H., Teukolsky S.A., Vetterling W.T. and Flannery B.P. (1992): Numerical Recipes in C: The Art of Scientific Computing.- Cambridge, UK: University Press.
• Ronald E. and Schoenauer M. (1994): Genetic lander: An experiment in accurate neuro-genetic control. - Proc. 3rd Conf. Parallel Problem Solving from Nature, Berlin, Germany, pp. 452-461.
• Rumelhart D.E., Hinton G.E. and Williams R.J. (1986): Learning representations by back-propagating errors. - Nature, Vol. 323, pp. 533-536.
• Stone M. (1974): Cross-validation choice and assessment of statistical procedures. - J. Royal Stat. Soc., Series B, Vol. 36, pp. 111-147.
• Tierney L. and Kadane J.B. (1986): Accurate approximations for posterior moments and marginal densities. - J. Amer. Stat. Assoc., Vol. 81, pp. 82-86.
• Tikhonov A.N. (1963): Solution of incorrectly formulated problems and the regularization method. - Soviet Math. Dokl., Vol. 4, pp. 1035-1038.
• Wasserman L. (1999): Bayesian model selection and model averaging. - J. Math. Psych., Vol. 44, No. 1, pp. 92-107.
• Weigend A.S., Rumelhart D.E. and Huberman B.A. (1991): Generalization by weight elimination with application to forecasting, In: Advances in Neural Information Processing Systems, Vol. 3 (R. Lippmann, J. Moody and D. Touretzky, Eds.). - San Francisco, CA: Morgan-Kaufmann, pp. 875-882.
• Williams P.M. (1995): Bayesian regularization and pruning using a Laplace prior. - Neural Comput., Vol. 7, No. 1, pp. 117-143.