
Department of Electrical Engineering and Computer Science
Department of Mechanical Engineering

6.050J/2.110J – Information, Entropy and Computation –
Spring 2014


Unit 7: Processes
Schedule
Lecture 
Wednesday, Mar 19, 2014, 1:00 PM 
Room 5234 
Recitation 
Friday, Mar 21, 2014, 1:00 PM 
Room 5234 
Lecture Handouts
Students who for any reason did not receive these items can pick them up in
Room 38344. Most of this material is also available on the 6.050J/2.110J
Web site
http://mtlsites.mit.edu/Courses/6.050.
 Unit 7 Resources (this page)

6.050J/2.110J Notes

Robert G. Gallager, “Claude E. Shannon: A Retrospective on His Life,
Work, and Impact,” IEEE Transactions on Information Theory, vol. 47,
no. 7, pp. 26812695; November, 2001
Reading Assignment
Resources
Technical

Claude E. Shannon, “A Mathematical Theory of Communication,”
Bell System Technical Journal, vol. 27, no. 3, pp. 379423, July, 1948
(Parts III); and no. 4, pp. 623656, October, 1948 (Parts IIIV). These
seminal papers are available in several forms

PDF
version of first paper, July, 1948

PDF
version of second paper, October, 1948

Claude E. Shannon and Warren Weaver, “The Mathematical Theory of
Communication,” University of Illinois Press, Urbana, IL; 1949, with
later editions 1963 and 1998 (incorporating a number of modifications and
corrections by Shannon)

Claude E. Shannon, “A Mathematical Theory of Communication,”
50th Anniversary Edition, printed for the 1998 IEEE International
Symposium on Information Theory, MIT, Cambridge, MA; August 1621, 1998
(based on 1949 book, with corrections)

Reprinted in D. Slepian, editor, “Key Papers in the Development of
Information Theory,” IEEE Press, New York; 1974

Reprinted in N. J. A. Sloane and A. D. Wyner, editors, “Claude
Elwood Shannon: Collected Papers,” IEEE Press, New York; 1993
Because he treated a general case, Shannon was able to distinguish loss and
noise, in the way done in 6.050J/2.110J. Many others treat only channels
with similar inputs and outputs in which case often L = N so it
may not be obvious why they are different concepts.
Historical

Claude
E. Shannon (1916–2001)
obituary,
Tech Talk, MIT, Feb 28, 2001

Aaron D. Wyner, The
Significance of Shannon’s Work

Robert G. Gallager, “Claude E. Shannon: A Retrospective on His Life,
Work, and Impact,” IEEE Transactions on Information Theory, vol. 47,
no. 7, pp. 26812695; November, 2001

The process model is also known as Markov Processes or Markov Chains after
Andrei
A. Markov (1856–1922)
General Technical Books
There are many excellent texts on communications, most of which assume a
familiarity with mathematics beyond introductory calculus. Some have
treatments of the discrete memoryless channel, but not with the same
emphasis given here. Here are a few:

Robert G. Gallager, “Information Theory and Reliable
Communications,” John Wiley & Sons, Inc., New York, NY; 1968. One
of the early textbooks, designed for firstyear graduate students

Allan R. Hambley, “An Introduction to Communication Systems,”
Computer Science Press; 1990

Thomas M. Cover and Joy A. Thomas, “Elements of Information
Theory,” John Wiley & Sons, Inc., New York, NY; 1991. Aimed at
university seniors and firstyear graduate students. One of several
excellent books of that era.
Second edition; 2006

Simon Haykin, “Communication Systems,” 4th edition, John Wiley
and Sons, Inc.; 2001

David J. C. MacKay, “Information Theory, Inference, and Learning
Algorithms,” Cambridge University Press; 2003–2005.
Online
viewing.
Contents.
See Chapter 2
and Chapter 10
Help Wanted
6.050J/2.110J students: be the first to suggest a resource, for example a
useful Web site or a good book or article, to add to the list above. Send
your suggestion by email during Spring 2014 to 6.050staff at mit.edu.
6.050J/2.110J home page 
Spring 2014 
Search 
Comments and inquiries
Click here for information on MIT Accessibility