Department of Electrical Engineering and Computer Science Department of Mechanical Engineering 6.050J/2.110J – Information, Entropy and Computation – Spring 2009

Unit 7: Processes

Schedule

Lecture Handouts

Students who for any reason did not receive these items can pick them up in Room 38-344. Most of this material is also available on the 6.050J/2.110J Web site http://mtlsites.mit.edu/Courses/6.050.

• Unit 7 Resources (this page)
• 6.050J/2.110J Notes
  • Chapter 7, Processes
• A summary in concise mathematics (mostly algebra) of many topics in information theory, coding, and communications: http://www.eie.polyu.edu.hk/~enmzwang/adc/l-notes/node4.html, by Dr. M. Z. Wang, Hong Kong, October 1996. There is a section on Discrete Memoryless Channels. This is one of many good summaries available
• Robert G. Gallager, "Claude E. Shannon: A Retrospective on His Life, Work, and Impact," IEEE Transactions on Information Theory, vol. 47, no. 7, pp. 2681-2695; November, 2001

Reading Assignment

• Chapter 7, Processes

Resources

Technical

• Claude E. Shannon, "A Mathematical Theory of Communication," Bell System Technical Journal, vol. 27, pp. 379-423 (Part I), 623-656 (Part II); July and October, 1948. These seminal papers are available in several forms (see bibliographic notes)
  • PDF version of original papers, with corrections but without Shannon's 1949 modifications
  • Claude E. Shannon and Warren Weaver, "The Mathematical Theory of Communication," University of Illinois Press, Urbana, IL; 1949, with later editions 1963 and 1998 (incorporating a number of modifications and corrections by Shannon)
  • Claude E. Shannon, "A Mathematical Theory of Communication," 50th Anniversary Edition, printed for the 1998 IEEE International Symposium on Information Theory, MIT, Cambridge, MA; August 16-21, 1998 (based on 1949 book, with corrections)
  • Reprinted in D. Slepian, editor, Key Papers in the Development of Information Theory, IEEE Press, New York; 1974
  • Reprinted in N. J. A. Sloane and A. D. Wyner, editors, Claude Elwood Shannon: Collected Papers, IEEE Press, New York; 1993
  Because he treated a general case, Shannon was able to distinguish loss and noise, in the way done in 6.050J/2.110J. Many others treat only channels with similar inputs and outputs in which case often L = N so it may not be obvious why they are different concepts.

Historical

• Claude E. Shannon (1916 - 2001)
      obituary, Tech Talk, MIT, Feb 28, 2001
• Aaron D. Wyner, The Significance of Shannon's Work
• Robert G. Gallager, "Claude E. Shannon: A Retrospective on His Life, Work, and Impact," IEEE Transactions on Information Theory, vol. 47, no. 7, pp. 2681-2695; November, 2001
• The process model is also known as Markov Processes or Markov Chains after Andrei A. Markov (1856 - 1922)

General Technical Books

There are many excellent texts on communications, most of which assume a familiarity with mathematics beyond introductory calculus. Some have treatments of the discrete memoryless channel, but not with the same emphasis given here. Here are a few:

• Robert G. Gallager, "Information Theory and Reliable Communications," John Wiley & Sons, Inc., New York, NY; 1968. One of the early textbooks, designed for first-year graduate students
• Allan R. Hambley, "An Introduction to Communication Systems," Computer Science Press; 1990
• Thomas M. Cover and Joy A. Thomas, "Elements of Information Theory," John Wiley & Sons, Inc., New York, NY; 1991. Aimed at university seniors and first-year graduate students. One of several excellent books of that era
• Simon Haykin, "Communication Systems," 4th edition, John Wiley and Sons, Inc.; 2001

Help Wanted

6.050J/2.110J students: be the first to suggest a resource, for example a useful Web site or a good book or article, to add to the list above. Send your suggestion by e-mail during Spring 2009 to 6.050-staff at mit.edu.