

Department of Electrical Engineering and Computer Science
Department of Mechanical Engineering 
6.050J / 2.110J Information and Entropy
Spring 2007


Unit 7: Processes
Schedule
Lecture 
Tuesday, Mar 20, 2007, 12:00 PM 
Room 2105 
Recitation 
Thursday, Mar 22, 2007, 12:00 PM 
Room 2105 
Lecture Handouts
Students who for any reason did not receive these items can pick up a copy in
Room 38344. Most of this material is also available on the 6.050J/2.110J Web site
http://mtlsites.mit.edu/Courses/6.050.
 Unit 7 Resources (this page)
 6.050J/2.110J Notes
 A summary in concise mathematics (mostly algebra) of many topics in information theory,
coding, communications, by Dr. M. Z. Wang, Hong Kong, October 1996:
http://www.eie.polyu.edu.hk/~enmzwang/adc/lnotes/node4.html.
There is a section on Discrete Memoryless Channels.
 Robert G. Gallager, "Claude E. Shannon: A Retrospective on His Life, Work, and Impact,"
IEEE Transactions on Information Theory, vol. 47, no. 7, pp. 26812695; November, 2001.
Reading Assignment
Resources
Technical
 Claude E. Shannon, "A Mathematical Theory of Communication," Bell System
Technical Journal, vol. 27, pp. 379423 (Part I), 623656 (Part II); July and
October, 1948. These seminal papers are available in several forms (see
bibliographic
notes)
 PDF version
of original papers, with corrections but without Shannon's 1949 modifications
 Claude E. Shannon and Warren Weaver, "The Mathematical Theory of Communication,"
University of Illinois Press, Urbana, IL; 1949, with later editions 1963 and 1998
(incorporating a number of modifications and corrections by Shannon)
 Claude E. Shannon, "A Mathematical Theory of Communication," 50th Anniversary
Edition, printed for the 1998 IEEE International Symposium on Information Theory,
MIT, Cambridge, MA; August 1621, 1998 (based on 1949 book, with corrections)
 Reprinted in D. Slepian, editor, Key Papers in the Development of Information
Theory, IEEE Press, New York; 1974
 Reprinted in N. J. A. Sloane and A. D. Wyner, editors, Claude Elwood Shannon:
Collected Papers, IEEE Press, New York; 1993
Because he treated a general case, Shannon was able to distinguish loss and
noise, in the way done in 6.050J/2.110J. Many others treat only channels with
similar inputs and outputs in which case often L = N so it may not be
obvious why they should be different concepts.
Historical
 Claude
E. Shannon (1916  2001)
obituary,
Tech Talk, MIT
 Aaron D. Wyner, The
Significance of Shannon's Work
 Robert G. Gallager, "Claude E. Shannon: A Retrospective on His Life, Work, and Impact,"
IEEE Transactions on Information Theory, vol. 47, no. 7, pp. 26812695; November, 2001.
 The process model is also known as Markov Processes or Markov Chains after
Andrei
A. Markov (1856  1922).
General Technical Books
There are many excellent texts on communications, most of which assume a
familiarity with mathematics beyond introductory calculus. Some have treatments
of the discrete memoryless channel, but not with the same emphasis given here.
Here are a few:
 Robert G. Gallager, "Information Theory and Reliable Communications," John
Wiley & Sons, Inc., New York, NY; 1968. One of the early textbooks,
designed for firstyear graduate students
 Allan R. Hambley, "An Introduction to Communication Systems," Computer
Science Press; 1990
 Thomas M. Cover and Joy A. Thomas, "Elements of Information Theory," John
Wiley & Sons, Inc., New York, NY; 1991. Aimed at university seniors and
firstyear graduate students. One of several excellent books of that era
 Simon Haykin, "Communication Systems," 4th edition, John Wiley and Sons,
Inc.; 2001.
Help Wanted
6.050J/2.110J students: be the first to suggest a resource, for example a useful Web
site or a good book or article, to add to the list above. If your suggestion is
accepted by the 6.050J/2.110J staff, you will get a $5 icecream gift certificate.
Send your suggestion by email during Spring 2007 to 6.050staff (at) mit.edu.
6.050J/2.110J home page 
Spring 2007 
Site map 
Search 
About this page 
Comments and inquiries