Elements of information theory: | Guide books

Elements of information theoryJanuary 1991

Authors:
Thomas M. Cover
Stanford Univ., Stanford, CA
,
Joy A. Thomas
IBM Watson Research Center, Hawthorne, NY

Publisher:

Wiley-Interscience
605 Third Avenue New York, NY
United States

ISBN:978-0-471-06259-2

Published:02 January 1991

Pages:

542

Available at Amazon

Bibliometrics

Abstract

No abstract available.

Cited By

Contributors

Thomas M Cover
Stanford University
- Publication Years1985 - 2010
- Publication counts55
- Citation count8,787
- Available for Download1
- Downloads (cumulative)665
- Downloads (12 months)55
- Downloads (6 weeks)8
- Average Downloads per Article665
- Average Citation per Article160
View Full Profile
Joy A Thomas
IBM Research - Almaden
- Publication Years1986 - 2006
- Publication counts6
- Citation count5,425
- Available for Download2
- Downloads (cumulative)1,271
- Downloads (12 months)85
- Downloads (6 weeks)9
- Average Downloads per Article636
- Average Citation per Article904
View Full Profile

Index Terms

Elements of information theory

Recommendations

Synthesis of reversible sequential elements

To construct a reversible sequential circuit, reversible sequential elements are required. This work presents novel designs of reversible sequential elements such as the D latch, JK latch, and T latch. Based on these reversible latches, we construct the ...
Read More
Using Information Foraging Theory to Understand Search Behavior in Different Environments
CHIIR '17: Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval

The aim of this research proposal is to understand the extent in which Information Scent, part of Information Foraging Theory, influence the perception of relevance of retrieved search engine result pages (SERPs) in different environments (desktop and ...
Read More
A Googol of Information about Google

Timothy P. Chartier reviews Google's PageRank and Beyond: The Science of Search Engine Rankings by Amy Langville and Carl Meyer.

Read More

Reviewer: Prasanna Kumar Sahoo

Information theory originated in Claude Shannon's work in 1948. Shannon was motivated by problems in communication theory. Even today, many engineers and scientists erroneously consider information theory as synonymous with communication theory. Information theory can be considered as a branch of modern applied probability theory. During the last 25 years, several textbooks and monographs have appeared presenting topics that are at the forefront of the research, as well as being important to the theoretical development of information theory. Why do we need another book on information theory__?__ Generally, one does not write a new book on information theory unless one has something new to say, or at least has something to say in a new way, or provides applications of theoretical developments. This book contains many new topics not found in earlier books on information theory. Some of these topics are the relationship of information theory to gambling, the universality of the second law of thermodynamics in the context of Markov chains, the competitive optimality of Huffman codes, the Kolmogorov complexity theory, and the relationship between Fisher information and mutual information. The book comprises 16 chapters. Chapter 1 is an introduction describing the relationship of information theory to communication theory, Kolmogorov complexity theory, thermodynamics, probability theory and mathematical statistics, economics, and philosophy of science. In this chapter, the authors briefly preview the various topics covered in the book. Chapter 2 includes the algebraic relationships of entropy, relative entropy, and mutual information. In chapter<__?__Pub Fmt hardspace>3, the authors describe the asymptotic equipartition property. Using this property, the entropy rate of stochastic processes is studied in chapter 4. In chapter 5, the authors answer one of the two fundamental questions of information <__?__Pub Fmt nolinebreak>theory:<__?__Pub Fmt /nolinebreak> What is the ultimate data compression__?__ This chapter also covers Huffman codes and arithmetic codes. Chapter 6 describes the relationship of information theory to gambling. In chapter 7, the authors describe Kolmogorov complexity theory. This chapter is intellectually more demanding than the rest of the book. Chaitin's mystical and magical number W is investigated. This number, which is the binary expansion of the probability that a Turing machine will halt, reveals many of the secrets of mathematics. In chapter 8, the authors answer another important question of information theory: What is the ultimate transmission rate of communication__?__ The channel capacity is established in this chapter in detail. Differential entropy is developed in chapter 9. Chapter 10 investigates the capacity of the fundamental Gaussian channel. In chapter 11, the authors describe the application of maximum entropy to spectral estimation. Burg's maximum entropy theorem is presented. Chapter 12 explores the relationship between information theory and statistics. The problem of testing hypotheses is considered, and best possible error exponents for such tests are derived. The authors also treat the estimation of the parameters of a distribution and describe the role of Fisher information. Chapter 13 deals with rate distortion theory. In large communication networks such as computer networks, satellite networks, and the phone system, one of the general problems is the following: Given many senders and receivers and a channel transition matrix that describes the effects of the interference and noise in the network, decide whether the sources can be transmitted over the channel. Chapter 14 deals with various special cases of this general problem (which is yet unsolved). Many earlier introductory books on information theory did not deal with this topic. Chapters 15 and 16 deal with applications of information theory to stock markets and solving matrix inequalities, respectively. This book emphasizes theoretical aspects of information theory, and thus most of the results are stated as theorems and proofs. In this way, the elegance of the results speaks for itself. The presentation is lucid, and several examples illustrate the key concepts. At the end of each chapter are exercises and historical notes. This book has evolved from the authors' lecture notes on information theory. Their obvious interest and knowledge of the subject come through. This book is suitable for a beginning graduate course on information theory. Few typographical errors occur, but on page 511 the name Acze´l has been misspelled as Axze´l<__?__Pub Caret>. This book will be a useful and important addition to the resources of practitioners and many others engaged in information theory and related areas. Overall, it is an impressive book. The breadth and number of topics covered, the knowledge and understanding of the authors, and the extent of published materials reviewed combine to ensure that this book will be used frequently.

Access critical reviews of Computing literature here

Become a reviewer for Computing Reviews.

Comments

Browse Books

Sections

Cited By

Index Terms

Synthesis of reversible sequential elements

Using Information Foraging Theory to Understand Search Behavior in Different Environments

A Googol of Information about Google

Access critical reviews of Computing literature here

Save to Binder

Sections

Cited By

Save to Binder

Index Terms

Recommendations

Synthesis of reversible sequential elements

Using Information Foraging Theory to Understand Search Behavior in Different Environments

A Googol of Information about Google

Access critical reviews of Computing literature here