The nature of statistical learning theory: | Guide books

The nature of statistical learning theoryJune 1995

Author:
Vladimir N. Vapnik
AT&T Bell Labs, Holmdel, NJ

Publisher:

Springer-Verlag
Berlin, Heidelberg

ISBN:978-0-387-94559-0

Published:01 June 1995

Pages:

203

Available at Amazon

Bibliometrics

Abstract

No abstract available.

Cited By

Contributors

Vladimir N Vapnik
Columbia University
- Publication Years1982 - 2021
- Publication counts70
- Citation count21,991
- Available for Download7
- Downloads (cumulative)16,720
- Downloads (12 months)4,807
- Downloads (6 weeks)603
- Average Downloads per Article2,389
- Average Citation per Article314
View Full Profile

Index Terms

The nature of statistical learning theory
1. Computing methodologies

Recommendations

Statistical learning theory: a tutorial

In this article, we provide a tutorial overview of some aspects of statistical learning theory, which also goes by other names such as statistical pattern recognition, nonparametric classification and estimation, and supervised learning. We focus on the ...
Read More
Information Theory and Statistical Learning
Read More
An Elementary Introduction to Statistical Learning Theory
Read More

Reviewer: Ranan B. Banerji

In this research monograph, Vapnik summarizes his results and those of some his colleagues (in Russia and the west) on what is now called computational learning theory (COLT) and the analysis of learning with neural nets. The book is in no way a survey, since the author has neglected to mention the extensive work of the major European and American workers in the field. Nevertheless, both the point of view and some recent results make the book important and useful. The reader is expected to be well-versed in statistics, and should have some acquaintance with COLT. After an introduction, the first chapter describes the general problem class as follows. We are given a class of functions on a sample space (the book considers mainly results that are independent of the distribution on this space, but there is some historical discussion of some cases of standard known distributions). There is also an unknown function. One needs to choose one function from the given class that minimizes the expected discrepancy between the chosen and the unknown functions. Clearly, the case of characteristic functions of classes of sets is included. In the abstract method (the ERM method) that the author discusses extensively, one chooses that function in the class that minimizes the average discrepancy and considers that function to be the solution. He compares the method with the classical “maximum likelihood estimator” parametric method and concludes that ERM is superior. The function obtained by the ERM method may not be the function that minimizes the expected discrepancy. One would hope that, as the sample size increases, the probability of the two functions being different would tend to zero. Whether this happens or not (or, as the author points out, happens trivially) depends on the function class. In chapter 2, the condition for such a “consistency” for the ERM is developed in terms of what the author calls the “random entropy” of samples. One of the major results of this chapter is that the class is consistent if the ratio of the expected value of the logarithm of the entropy and the sample size tends to zero as the sample size increases. In terms of this and other related results in the chapter, the author discusses the relationship between COLT and Kant's problem of demarcation and Popper's criterion of falsifiability. Random entropy is related closely to the VC dimension in that its maximum possible value for any sample size increases exponentially with the sample size until the size reaches the VC dimension and then becomes polynomial. The result of chapter 2 can be understood from this point of view. On the basis of this, chapter 3 develops a theory for the rate of convergence, that is, the exact nature of the decrease of the probability of error with sample size. The COLT community will find the discussion unusual in its approach, and thus interesting. Chapter 4 introduces an important area in the theory of learning, dealing with the fact that the set of functions (“hypotheses” in COLT nomenclature) can be graded into a class of sets of increasing VC dimension and that classes of lower dimension require smaller sample sizes for convergence. Also, if the sample is such that none of the hypotheses fits the data to within a sufficiently small error, one would like to expand the hypothesis set. However, while this might yield a smaller average error, the minimizing hypothesis may not be close to the one that minimizes the expected error (that is, it may not be the solution). Thus, with a small sample size, one has to find a balance between the error due to too small a class of hypotheses and the error due to too small a sample size. The chapter develops a theory for this balance. Chapter 5 deals with the description, analysis, and experimentation with neural network and supporting vector techniques for the recognition of handprinted characters. Vapnik concedes that, to obtain a satisfactory balance between data fitting and error estimation, the circuits have to be designed “to reflect an a priori knowledge of the problem.” The book is unnecessarily difficult to read. The author often does not present trains of thought cogently enough for the reader to follow, so gaps have to be filled by guesswork or by many patient re readings.

Access critical reviews of Computing literature here

Become a reviewer for Computing Reviews.

Comments

Browse Books

Sections

Cited By

Index Terms

Statistical learning theory: a tutorial

Information Theory and Statistical Learning

An Elementary Introduction to Statistical Learning Theory

Access critical reviews of Computing literature here

Save to Binder

Sections

Cited By

Save to Binder

Index Terms

Recommendations

Statistical learning theory: a tutorial

Information Theory and Statistical Learning

An Elementary Introduction to Statistical Learning Theory

Access critical reviews of Computing literature here