Browse Books

Reviewer: Christian Ronse

Developed by the authors for classes at the University of Western Australia, this is a textbook for information theory and coding that is heavily mathematical, yet wonderfully expository. The authors pack a great deal of information into its 385 pages. In a masterstroke, they follow the introduction of each new concept with multiple examples, worked in detail. Each chapter includes a brief bibliography and an average of over 20 exercises, some starred for difficulty. A modest index completes the book. The authors recommend use of the first half of the text for a one-semester undergraduate course, and the second half for a one-semester graduate course. Readers will need knowledge of calculus and linear algebra to benefit from the presentation. Familiarity with probability theory and abstract algebra would be helpful, but the authors provide good introductions to the essential features of those subjects before using them. The undergraduate material is extensive. From humble beginnings-definitions of probability distributions and expected value, the surprise of an event, and entropy as the expected value of surprise-the text rapidly moves on to joint, marginal, and conditional probability distributions, Bayes' Theorem, and conditional entropy. It rounds out the treatment of foundational concepts with discussions of information sources-memoryless and Markov-and their entropies, and the entropy of probability distributions over infinite sample spaces. Good material follows on channels, specifically binary symmetric and binary erasure channels (BSCs and BECs), where the concept of bit error rate appears. Development of the concepts of equivocation and mutual information leads naturally to a definition of channel capacity as maximum mutual information, and derivation of the capacities of BSCs, BECs, weakly symmetric channels, and Gaussian channels. An introduction to rate distortion theory concludes the treatment of channels. A chapter on source coding (as distinct from channel coding) introduces us to the notions of nonsingular codes, block codes, unique decodability, instantaneous codes, and compact codes, and presents McMillan's Theorem, Shannon's Noiseless Coding Theorem, and Fano and Huffman codes, with extensive material on algorithms for Huffman encoding. It also includes treatment of arithmetic coding, with encoding and decoding algorithms described in detail. Finally, a valuable chapter on data compression applies earlier ideas to explain run-length coding, CCITT standard fax compression, block-sorting compression, dictionary coding, statistical compression, and image compression. The graduate material comprises an extensive treatise on coding theory. It begins with the notions of error detection versus error correction, code rate, minimum-error decoding versus maximum-likelihood decoding, and Hamming distance, with a discussion of using the Hamming distance to perform maximum-likelihood decoding for BSCs. The definition of a block code's minimum distance allows clear characterization of a block code's ability to detect or correct up to a fixed number of errors. Discussion of bounds on the number of code words (for instance, the Hamming and Gilbert bounds), maximal codes, perfect codes, and bit and block error probabilities leads to the presentation of Shannon's Fundamental Theorem for BSCs. With this foundation, the text is at last ready to consider error-correcting codes with syndrome decoding, the algebraic underpinnings of which it presents in detail. Subsequent material covers Hadamard matrix codes, cyclic codes (including the Golay code, Hamming codes, CRC codes, and Reed-Muller codes), burst-correcting codes (including Fire codes, BCH codes, and Reed-Solomon codes), and convolutional codes (including Viterbi and sequential decoding and Turbo codes). No work of this breadth and depth is free from quirks. One in this text that might puzzle some readers is the occasional presentation of results masquerading as definitions, for instance the "definition" of prefix condition. This occasionally affects the effectiveness of examples, for instance the unique decodability example that uses the (unproved) informal definition of unique decodability rather than the formal one given. It is fairly common these days for publishers' Web sites to provide auxiliary material for instructors using their textbooks, such as exercise solutions, slides or handouts, and project suggestions. Some such material is reportedly available from the publisher for this text. Unavailable at this time, however, is an errata sheet, which would probably be slim in this case, but which could at least remedy, for example, the peculiar confusion between AND and XOR, which we find in chapter 6. I recommend this book very highly as a textbook or a resource for the mathematically literate layman. Its wealth of material, excellent motivation and exposition, and copious worked examples make it a landmark work destined to become a classic.