Department of Electrical Engineering and Computer Science Department of Mechanical Engineering 6.050J/2.110J – Information, Entropy and Computation – Spring 2015

Unit 7: Processes

Schedule

Lecture Handouts

Students who for any reason did not receive these items can pick them up in Room 38-344. Most of this material is also available on the 6.050J/2.110J Web site http://mtlsites.mit.edu/Courses/6.050.

• Unit 7 Resources (this page)
• 6.050J/2.110J Notes
  • Chapter 7: Processes
• Robert G. Gallager, “Claude E. Shannon: A Retrospective on His Life, Work, and Impact,” IEEE Transactions on Information Theory, vol. 47, no. 7, pp. 2681-2695; November, 2001

Reading Assignment

• Notes, Chapter 7, Processes

Resources

Technical

• Claude E. Shannon, “A Mathematical Theory of Communication,” Bell System Technical Journal, vol. 27, no. 3, pp. 379-423, July, 1948 (Parts I-II); and no. 4, pp. 623-656, October, 1948 (Parts III-V). These seminal papers are available in several forms
  • PDF version of the two papers, July, 1948 and October, 1948
  • Claude E. Shannon and Warren Weaver, “The Mathematical Theory of Communication,” University of Illinois Press, Urbana, IL; 1949, with later editions 1963 and 1998 (incorporating a number of modifications and corrections by Shannon)
  • Claude E. Shannon, “A Mathematical Theory of Communication,” 50th Anniversary Edition, printed for the 1998 IEEE International Symposium on Information Theory, MIT, Cambridge, MA; August 16-21, 1998 (based on 1949 book, with corrections)
  • Reprinted in D. Slepian, editor, “Key Papers in the Development of Information Theory,” IEEE Press, New York; 1974
  • Reprinted in N. J. A. Sloane and A. D. Wyner, editors, “Claude Elwood Shannon: Collected Papers,” IEEE Press, New York; 1993
  Because he treated a general case, Shannon was able to distinguish loss and noise, in the way done in 6.050J/2.110J. Many others treat only channels with similar inputs and outputs in which case often L = N so it may not be obvious why they are different concepts.

Historical

• Claude E. Shannon (1916–2001)
      obituary, Tech Talk, MIT, Feb 28, 2001
• Aaron D. Wyner, The Significance of Shannon’s Work
• Robert G. Gallager, “Claude E. Shannon: A Retrospective on His Life, Work, and Impact,” IEEE Transactions on Information Theory, vol. 47, no. 7, pp. 2681-2695; November, 2001
• The process model is also known as Markov Processes or Markov Chains after Andrei A. Markov (1856–1922)

General Technical Books

There are many excellent texts on communications, most of which assume a familiarity with mathematics beyond introductory calculus. Some have treatments of the discrete memoryless channel, but not with the same emphasis given here. Here are a few:

• Robert G. Gallager, “Information Theory and Reliable Communications,” John Wiley & Sons, Inc., New York, NY; 1968. One of the early textbooks, designed for first-year graduate students
• Allan R. Hambley, “An Introduction to Communication Systems,” Computer Science Press; 1990
• Thomas M. Cover and Joy A. Thomas, “Elements of Information Theory,” John Wiley & Sons, Inc., New York, NY; 1991. Aimed at university seniors and first-year graduate students. One of several excellent books of that era. Second edition; 2006
• Simon Haykin, “Communication Systems,” 4th edition, John Wiley and Sons, Inc.; 2001
• David J. C. MacKay, “Information Theory, Inference, and Learning Algorithms,” Cambridge University Press; 2003–2005. Online viewing. Contents. See Chapter 2 and Chapter 10

Help Wanted

6.050J/2.110J students: be the first to suggest a resource, for example a useful Web site or a good book or article, to add to the list above. Send your suggestion by e-mail during Spring 2015 to 6.050-staff at mit.edu.