Neural Networks: Tricks of the Trade

Genevieve B. Orr, Klaus-Robert Müller (eds.)3540653112, 9783540653110

The idea for this book dates back to the NIPS’96 workshop “Tips of the Trade” where, for the first time, a systematic attempt was made to make an assessment and evaluation of tricks for efficiently exploiting neural network techniques. Stimulated by the success of this meeting, the volume editors have prepared the present comprehensive documentation. Besides including chapters developed from the workshop contributions, they have commissioned further chapters to round out the presentation and complete the coverage of relevant subareas. This handy reference book is organized in five parts, each consisting of several coherent chapters using consistent terminology. The work starts with a general introduction and each part opens with an introduction by the volume editors. A comprehensive subject index allows easy access to individual topics. The book is a gold mine not only for professionals and researchers in the area of neural information processing, but also for newcomers to the field.

---

Table of contents :
Introduction….Pages 1-5
Front Matter….Pages 7-8
Efficient BackProp….Pages 9-50
Front Matter….Pages 51-53
Early Stopping – But When?….Pages 55-69
A Simple Trick for Estimating the Weight Decay Parameter….Pages 71-92
Controlling the hyperparameter search in MacKay’s Bayesian neural network framework….Pages 93-112
Adaptive Regularization in Neural Network Modeling….Pages 113-132
Large Ensemble Averaging….Pages 133-139
Front Matter….Pages 141-143
Square Unit Augmented Radially Extended Multilayer Perceptrons….Pages 145-163
A Dozen Tricks with Multitask Learning….Pages 165-191
Solving the Ill-Conditioning in Neural Network Learning….Pages 193-206
Centering Neural Network Gradient Factors….Pages 207-226
Avoiding roundoff error in backpropagating derivatives….Pages 227-233
Front Matter….Pages 235-237
Transformation Invariance in Pattern Recognition — Tangent Distance and Tangent Propagation….Pages 239-274
Combining Neural Networks and Context-Driven Search for Online, Printed Handwriting Recognition in the Newton….Pages 275-298
Neural Network Classification and Prior Class Probabilities….Pages 299-313
Applying Divide and Conquer to Large Scale Pattern Recognition Tasks….Pages 315-342
Front Matter….Pages 343-345
Forecasting the Economy with Neural Nets: A Survey of Challenges and Solutions….Pages 347-371
How to Train Neural Networks….Pages 373-423