Department of Electrical Engineering and Computer Science  Department of Mechanical Engineering  6.050J / 2.110J     Information and Entropy     Spring 2005

Unit 8: Inference

Schedule

Lecture Handouts

Students who for any reason did not receive these items can pick up a copy in Room 38-344. Most of this material is also available on the 6.050J/2.110J Web site http://mtlsites.mit.edu/Courses/6.050.

• Unit 8 Resources (this page)
• Problem Set 8
• 6.50J/2.110J Notes
  • Inference
• Jaynes, E. T., "Information Theory and Statistical Mechanics," Physical Review, vol. 106, no. 4, pp. 620-630; May 15, 1957. PDF (2.2 MB), PS (2.6 MB)

Reading Assignment

• Chapter 8, Inference

Resources

Technical

The person most responsible for use of maximum entropy principles in various fields of science is Edwin T. Jaynes (July 5, 1922 - April 30, 1998).

• Jaynes, E. T., "Information Theory and Statistical Mechanics," Physical Review, vol. 106, no. 4, pp. 620-630; May 15, 1957. PDF (2.2 MB), PS (2.6 MB). This is the seminal paper which really started the modern use of the Principle of Maximum Entropy in physics
• Jaynes, E. T., "Information Theory and Statistical Mechanics. II," Physical Review, vol. 108, no. 2, pp. 171-190; October 15, 1957. PDF (3.9 MB), PS (4.7 MB). Continuation of the previous reference

Historical

• Jaynes Photograph
• Jaynes Biography
• Jaynes Bibliography, list of Unpublished Works, and Book Reviews

Jaynes knew, of course, about Thomas Bayes and when on sabbatical in England sought out and photographed the Bayes tombstone.

Books

• The philosophy of assuming maximum uncertainty is discussed in Chapter 3 of M. Tribus, "Thermostatics and Thermodynamics," D. Van Nostrand Co, Inc., Princeton, NJ; 1961
• Another good explanation, in terms of estimating probabilities of an unfair die is in E. T. Jaynes, "Information Theory and Statistical Mechanics," pp. 181-218 in "Statistical Physics," Brandeis Summer Institute 1962, W. A. Benjamin, Inc., New York, NY; 1963. PDF, PS
• Personal history by Jaynes, Edwin T. Jaynes, "Where Do We Stand on Maximum Entropy?," pp. 15-118, in "The Maximum Entropy Formalism," Raphael D. Levine and Myron Tribus, editors, The MIT Press, Cambridge, MA; 1979. PDF, PS

Help Wanted

6.050J/2.110J students: be the first to suggest a resource, for example a useful Web site or a good book or article, to add to the list above. If your suggestion is accepted by the 6.050J/2.110J staff, you will get a $5 ice-cream gift certificate. Send your suggestion by e-mail during Spring 2005 to 6.050-staff (at) mit.edu.