DOCSLIB.ORG

Mathematical Theory of Claude Shannon (2001)

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Mathematical Theory of Claude Shannon (2001) Mathematical Theory of Claude Shannon A study of the style and context of his work up to the genesis of information theory. by Eugene Chiu, Jocelyn Lin, Brok Mcferron, Noshirwan Petigara, Satwiksai Seshasai 6.933J / STS.420J The Structure of Engineering Revolutions Table of Contents Acknowledgements............................................................................................................................................3 Introduction........................................................................................................................................................4 Methodology ..................................................................................................................................................7 Information Theory...........................................................................................................................................8 Information Theory before Shannon.........................................................................................................8 What was Missing........................................................................................................................................15 1948 Mathematical theory of communication ........................................................................................16 The Shannon Methodology .......................................................................................................................19 Shannon as the architect.............................................................................................................................19 Switching Theory .............................................................................................................................................22 Background and Contemporary Work.....................................................................................................22 Building Blocks to Shannon's Master's Thesis........................................................................................23 A Symbolic Analysis of Relay and Switching Circuits ...........................................................................26 Theorems....................................................................................................................................28 Negation Theorems ..................................................................................................................28 Analogue Between the Calculus of Propositions and the Symbolic Relay Analysis..........................29 An Example of a Synthesis Problem........................................................................................................29 Popular Recognition ...................................................................................................................................31 Genetics.............................................................................................................................................................34 An Algebra for Theoretical Genetics, the beginnings ...........................................................................34 History of population genetics ..................................................................................................................35 Eugenics...................................................................................................................................................35 Genetics in late 1930’s............................................................................................................................37 Genetics in early 1940’s .........................................................................................................................38 Vannevar Bush, Claude Shannon, and genetics......................................................................................38 Shannon’s Ph.D...........................................................................................................................................40 Algebraic details...........................................................................................................................................41 Analysis and Comparison...........................................................................................................................43 A Dead End .................................................................................................................................................43 Cryptography....................................................................................................................................................47 Relation to Information Theory................................................................................................................47 Vernam System............................................................................................................................................50 Link to Information Theory ......................................................................................................................52 Shannon's Style.................................................................................................................................................54 Collaboration................................................................................................................................................54 Advising ........................................................................................................................................................56 The "great gadgeteer" .................................................................................................................................57 Application of his work..............................................................................................................................60 Awards and Recognition ............................................................................................................................61 Conclusion ........................................................................................................................................................63 References.........................................................................................................................................................65 Correspondences.........................................................................................................................................65 Interviews .....................................................................................................................................................65 Publications..................................................................................................................................................66 2 Acknowledgements We would like to thank professors Robert Fano, Bob Gallagher, Chris Kaiser, Charles Vest and Hartley Rogers for their academic and personal insight into Claude Shannon’s work and the fields of switching theory, genetics and information theory. Shannon’s advisees Trenchard More, William Sutherland and Henry Ernst provided us with a unique perspective on working with Dr. Shannon. Betty Shannon graciously discussed her husband’s personality and work. MIT staff Be Blackburn and Laura Mersky helped facilitate our research. The staff of the MIT Archives and Library of Congress were instrumental in locating important documents and correspondences between Shannon and his colleagues. Professor David Mindell and Teaching Assistant Chen-Pang Yeang went beyond their roles as instructors and provided us with insight into their own research on this topic. Finally, we would also like to thank Peter Elias, who before passing away on Dec. 7, 2001 allowed us to search through his extensive archive of books and documents related to Claude Shannon and his fields. This paper is dedicated to his memory and the memory of Claude Shannon, who passed away on February 24, 2001. 3 Introduction When Trenchard More first met Claude Elwood Shannon, he was taking his oral exam for his Ph. D at the Massachusetts Institute of Technology1. The written exam was especially difficult at the time, recalled More, so doing well on the oral portion of the exam was vital. Shannon had agreed to be on More's committee because More was a TA under Sam Caldwell, an advisor for Shannon's own Master's thesis. The questions Shannon asked were drastically different from the rest and concentrated on the mathematical ideas behind the topics being discussed. Shannon was “after the ways of thought,” said More. He cared more about how More was thinking and whether he understood the fundamental mathematical concepts behind his research. Shannon felt that someone who really understood ideas could recreate them before your eyes, related More. Fortunately, despite having stumbled through the technical details in the written and oral exam, More had developed a solid understanding of the mathematical concepts and passed the oral exams by successfully answering Shannon's questions. He remembered another student meeting a different fate with Shannon - despite his perfect GPA, the student could not answer Shannon's questions, and it was revealed that he was simply memorizing concepts. The focus Shannon displayed in his questioning of More was representative of the guiding vision which drove much of his work. The most popular and revolutionary pieces of Shannon's work came very early
Load more

Recommended publications
  • Bernard M. Oliver Oral History Interview
    http://oac.cdlib.org/findaid/ark:/13030/kt658038wn Online items available Bernard M. Oliver Oral History Interview Daniel Hartwig Stanford University. Libraries.Department of Special Collections and University Archives Stanford, California November 2010 Copyright © 2015 The Board of Trustees of the Leland Stanford Junior University. All rights reserved. Note This encoded finding aid is compliant with Stanford EAD Best Practice Guidelines, Version 1.0. Bernard M. Oliver Oral History SCM0111 1 Interview Overview Call Number: SCM0111 Creator: Oliver, Bernard M., 1916- Title: Bernard M. Oliver oral history interview Dates: 1985-1986 Physical Description: 0.02 Linear feet (1 folder) Summary: Transcript of an interview conducted by Arthur L. Norberg covering Oliver's early life, his education, and work experiences at Bell Laboratories and Hewlett-Packard. Subjects include television research, radar, information theory, organizational climate and objectives at both companies, Hewlett-Packard's associations with Stanford University, and Oliver's association with William Hewlett and David Packard. Language(s): The materials are in English. Repository: Department of Special Collections and University Archives Green Library 557 Escondido Mall Stanford, CA 94305-6064 Email: [email protected] Phone: (650) 725-1022 URL: http://library.stanford.edu/spc Information about Access Reproduction is prohibited. Ownership & Copyright All requests to reproduce, publish, quote from, or otherwise use collection materials must be submitted in writing to the Head of Special Collections and University Archives, Stanford University Libraries, Stanford, California 94304-6064. Consent is given on behalf of Special Collections as the owner of the physical items and is not intended to include or imply permission from the copyright owner.
    [Show full text]
  • Harry Nyquist 1889-1976
    Nyquist and His Seminal Papers Harry Nyquist 1889-1976 Karl Johan Åström A Gifted Scientist and Department of Mechanical Engineering Engineer University of California Santa Barbara Johnson-Nyquist noise The Nyquist frequency Nyquist’s Stability Criterion ASME Nyquist Lecture 2005 ASME Nyquist Lecture 2005 Introduction 1.Introduction 2.A Remarkable Career • Thank you for honoring Nyquist • Thank you for inviting me to give this lecture 3.Communications • Nyquist was awarded the Rufus Oldenburger 4.Johnson-Nyquist Noise medal in 1975 5.Stability Theory • The person and his contributions • What we can learn 6.Summary ASME Nyquist Lecture 2005 ASME Nyquist Lecture 2005 A Remarkable Career Born in Nilsby Sweden February 7 1889 6 years in school Emigrated to USA 1907 Farmhand Teachers College University of North Dakota PhD Physics Yale University 1917 AT&T Bell Labs 1917-1954 Consultant 1954-1965 ASME Nyquist Lecture 2005 ASME Nyquist Lecture 2005 Becoming a Teacher is my Dream Rubrik • Emigrated 1907 • Southern Minnesota Normal College, Austin Active as in teaching • Back to SMNC • Exam 1911 valedictarian • High School Teacher 1912 ASME Nyquist Lecture 2005 ASME Nyquist Lecture 2005 A Dream Comes True Academia Pulls University of North Dakota BS EE 1914 MS EE 1915 Very active in student organizations met Johnson Yale University PhD Physics 1917. Thesis topic: On the Stark effect in Helium and Neon. Largely experimental. ASME Nyquist Lecture 2005 ASME Nyquist Lecture 2005 The ATT Early Fax A Career in AT&T Bell commersial from 1925 • 1917 AT&T Engineering Department • 1919 Department of Development and Research • 1935 Bell Labs • World War II • 1952 Assistant director fo Systems Studies • 1954 Retired • 1954-1962 Consulting ASME Nyquist Lecture 2005 ASME Nyquist Lecture 2005 An Unusual Research Lab In His Right Environment Control the telephone monoply • Nyquist thrived, challenging problems, clever Key technologies collegues, interesting.
    [Show full text]
  • Claude Elwood Shannon (1916–2001) Solomon W
    Claude Elwood Shannon (1916–2001) Solomon W. Golomb, Elwyn Berlekamp, Thomas M. Cover, Robert G. Gallager, James L. Massey, and Andrew J. Viterbi Solomon W. Golomb Done in complete isolation from the community of population geneticists, this work went unpublished While his incredibly inventive mind enriched until it appeared in 1993 in Shannon’s Collected many fields, Claude Shannon’s enduring fame will Papers [5], by which time its results were known surely rest on his 1948 work “A mathematical independently and genetics had become a very theory of communication” [7] and the ongoing rev- different subject. After his Ph.D. thesis Shannon olution in information technology it engendered. wrote nothing further about genetics, and he Shannon, born April 30, 1916, in Petoskey, Michi- expressed skepticism about attempts to expand gan, obtained bachelor’s degrees in both mathe- the domain of information theory beyond the matics and electrical engineering at the University communications area for which he created it. of Michigan in 1936. He then went to M.I.T., and Starting in 1938 Shannon worked at M.I.T. with after spending the summer of 1937 at Bell Tele- Vannevar Bush’s “differential analyzer”, the an- phone Laboratories, he wrote one of the greatest cestral analog computer. After another summer master’s theses ever, published in 1938 as “A sym- (1940) at Bell Labs, he spent the academic year bolic analysis of relay and switching circuits” [8], 1940–41 working under the famous mathemati- in which he showed that the symbolic logic of cian Hermann Weyl at the Institute for Advanced George Boole’s nineteenth century Laws of Thought Study in Princeton, where he also began thinking provided the perfect mathematical model for about recasting communications on a proper switching theory (and indeed for the subsequent mathematical foundation.
    [Show full text]
  • Digital Communication Systems 2.2 Optimal Source Coding
    Digital Communication Systems EES 452 Asst. Prof. Dr. Prapun Suksompong [email protected] 2. Source Coding 2.2 Optimal Source Coding: Huffman Coding: Origin, Recipe, MATLAB Implementation 1 Examples of Prefix Codes Nonsingular Fixed-Length Code Shannon–Fano code Huffman Code 2 Prof. Robert Fano (1917-2016) Shannon Award (1976 ) Shannon–Fano Code Proposed in Shannon’s “A Mathematical Theory of Communication” in 1948 The method was attributed to Fano, who later published it as a technical report. Fano, R.M. (1949). “The transmission of information”. Technical Report No. 65. Cambridge (Mass.), USA: Research Laboratory of Electronics at MIT. Should not be confused with Shannon coding, the coding method used to prove Shannon's noiseless coding theorem, or with Shannon–Fano–Elias coding (also known as Elias coding), the precursor to arithmetic coding. 3 Claude E. Shannon Award Claude E. Shannon (1972) Elwyn R. Berlekamp (1993) Sergio Verdu (2007) David S. Slepian (1974) Aaron D. Wyner (1994) Robert M. Gray (2008) Robert M. Fano (1976) G. David Forney, Jr. (1995) Jorma Rissanen (2009) Peter Elias (1977) Imre Csiszár (1996) Te Sun Han (2010) Mark S. Pinsker (1978) Jacob Ziv (1997) Shlomo Shamai (Shitz) (2011) Jacob Wolfowitz (1979) Neil J. A. Sloane (1998) Abbas El Gamal (2012) W. Wesley Peterson (1981) Tadao Kasami (1999) Katalin Marton (2013) Irving S. Reed (1982) Thomas Kailath (2000) János Körner (2014) Robert G. Gallager (1983) Jack KeilWolf (2001) Arthur Robert Calderbank (2015) Solomon W. Golomb (1985) Toby Berger (2002) Alexander S. Holevo (2016) William L. Root (1986) Lloyd R. Welch (2003) David Tse (2017) James L.
    [Show full text]
  • Toward a New Science of Information
    Data Science Journal, Volume 6, Supplement, 7 April 2007 TOWARD A NEW SCIENCE OF INFORMATION D Doucette1*, R Bichler 2, W Hofkirchner2, and C Raffl2 *1 The Science of Information Institute, 1737 Q Street, N.W. Washington, D.C. 20009, USA Email: [email protected] 2 ICT&S Center, University of Salzburg - Center for Advanced Studies and Research in Information and Communication Technologies & Society, Sigmund-Haffner-Gasse 18, 5020 Salzburg, Austria Email: [email protected], [email protected], [email protected] ABSTRACT The concept of information has become a crucial topic in several emerging scientific disciplines, as well as in organizations, in companies and in everyday life. Hence it is legitimate to speak of the so-called information society; but a scientific understanding of the Information Age has not had time to develop. Following this evolution we face the need of a new transdisciplinary understanding of information, encompassing many academic disciplines and new fields of interest. Therefore a Science of Information is required. The goal of this paper is to discuss the aims, the scope, and the tools of a Science of Information. Furthermore we describe the new Science of Information Institute (SOII), which will be established as an international and transdisciplinary organization that takes into consideration a larger perspective of information. Keywords: Information, Science of Information, Information Society, Transdisciplinarity, Science of Information Institute (SOII), Foundations of Information Science (FIS) 1 INTRODUCTION Information is emerging as a new and large prospective area of study. The notion of information has become a crucial topic in several emerging scientific disciplines such as Philosophy of Information, Quantum Information, Bioinformatics and Biosemiotics, Theory of Mind, Systems Theory, Internet Research, and many more.
    [Show full text]
  • 4 Classical Error Correcting Codes
    “Machines should work. People should think.” Richard Hamming. 4 Classical Error Correcting Codes Coding theory is an application of information theory critical for reliable communication and fault-tolerant information storage and processing; indeed, Shannon channel coding theorem tells us that we can transmit information on a noisy channel with an arbitrarily low probability of error. A code is designed based on a well-defined set of specifications and protects the information only for the type and number of errors prescribed in its design. Agoodcode: (i) Adds a minimum amount of redundancy to the original message. (ii) Efficient encoding and decoding schemes for the code exist; this means that information is easily mapped to and extracted from a codeword. Reliable communication and fault-tolerant computing are intimately related to each other; the concept of a communication channel, an abstraction for a physical system used to transmit information from one place to another and/or from one time to another is at the core of communication, as well as, information storage and processing. It should however be clear 355 Basic Concepts Linear Codes Polynomial Codes Basic Concepts Linear Codes Other Codes Figure 96: The organization of Chapter 4 at a glance. that the existence of error-correcting codes does not guarantee that logic operations can be implemented using noisy gates and circuits. The strategies to build reliable computing systems using unreliable components are based on John von Neumann’s studies of error detection and error correction techniques for information storage and processing. This chapter covers topics from the theory of classical error detection and error correction and introduces concepts useful for understanding quantum error correction, the subject of Chapter 5.
    [Show full text]
  • The Intellectual Journey of Hua Loo-Keng from China to the Institute for Advanced Study: His Correspondence with Hermann Weyl
    ISSN 1923-8444 [Print] Studies in Mathematical Sciences ISSN 1923-8452 [Online] Vol. 6, No. 2, 2013, pp. [71{82] www.cscanada.net DOI: 10.3968/j.sms.1923845220130602.907 www.cscanada.org The Intellectual Journey of Hua Loo-keng from China to the Institute for Advanced Study: His Correspondence with Hermann Weyl Jean W. Richard[a],* and Abdramane Serme[a] [a] Department of Mathematics, The City University of New York (CUNY/BMCC), USA. * Corresponding author. Address: Department of Mathematics, The City University of New York (CUN- Y/BMCC), 199 Chambers Street, N-599, New York, NY 10007-1097, USA; E- Mail: [email protected] Received: February 4, 2013/ Accepted: April 9, 2013/ Published: May 31, 2013 \The scientific knowledge grows by the thinking of the individual solitary scientist and the communication of ideas from man to man Hua has been of great value in this respect to our whole group, especially to the younger members, by the stimulus which he has provided." (Hermann Weyl in a letter about Hua Loo-keng, 1948).y Abstract: This paper explores the intellectual journey of the self-taught Chinese mathematician Hua Loo-keng (1910{1985) from Southwest Associ- ated University (Kunming, Yunnan, China)z to the Institute for Advanced Study at Princeton. The paper seeks to show how during the Sino C Japanese war, a genuine cross-continental mentorship grew between the prolific Ger- man mathematician Hermann Weyl (1885{1955) and the gifted mathemati- cian Hua Loo-keng. Their correspondence offers a profound understanding of a solidarity-building effort that can exist in the scientific community.
    [Show full text]
  • Prizes and Awards Session
    PRIZES AND AWARDS SESSION Wednesday, July 12, 2021 9:00 AM EDT 2021 SIAM Annual Meeting July 19 – 23, 2021 Held in Virtual Format 1 Table of Contents AWM-SIAM Sonia Kovalevsky Lecture ................................................................................................... 3 George B. Dantzig Prize ............................................................................................................................. 5 George Pólya Prize for Mathematical Exposition .................................................................................... 7 George Pólya Prize in Applied Combinatorics ......................................................................................... 8 I.E. Block Community Lecture .................................................................................................................. 9 John von Neumann Prize ......................................................................................................................... 11 Lagrange Prize in Continuous Optimization .......................................................................................... 13 Ralph E. Kleinman Prize .......................................................................................................................... 15 SIAM Prize for Distinguished Service to the Profession ....................................................................... 17 SIAM Student Paper Prizes ....................................................................................................................
    [Show full text]
  • Blaschke, Osgood, Wiener, Hadamard and the Early Development Of
    Blaschke, Osgood, Wiener, Hadamard and the Early Development of Modern Mathematics in China Chuanming Zong Abstract. In ancient times, China made great contributions to world civi- lization and in particular to mathematics. However, modern sciences includ- ing mathematics came to China rather too late. The first Chinese university was founded in 1895. The first mathematics department in China was for- mally opened at the university only in 1913. At the beginning of the twenti- eth century, some Chinese went to Europe, the United States of America and Japan for higher education in modern mathematics and returned to China as the pioneer generation. They created mathematics departments at the Chinese universities and sowed the seeds of modern mathematics in China. In 1930s, when a dozen of Chinese universities already had mathematics departments, several leading mathematicians from Europe and USA visited China, including Wilhelm Blaschke, George D. Birkhoff, William F. Osgood, Norbert Wiener and Jacques Hadamard. Their visits not only had profound impact on the mathematical development in China, but also became social events sometimes. This paper tells the history of their visits. 2020 Mathematics Subject Classification: 01A25, 01A60. 1. Introduction and Background In 1895, Peiyang University was founded in Tianjin as a result of the westernization movement. In 1898, Peking University was founded in Beijing. They are the earliest universities in China. On May 28, 1900, Britain, USA, France, Germany, Russia, Japan, Italy and Austria invaded China to suppress the Boxer Rebellion1 which was an organized movement against foreign missionaries and their followers. On August 14, 1900, more than 20,000 soldiers of the allied force occupied the Chinese capital Beijing and the emperor’s family with some high ranking officials and servants escaped to Xi’an.
    [Show full text]
  • Andrew Viterbi
    Andrew Viterbi Interview conducted by Joel West, PhD December 15, 2006 Interview conducted by Joel West, PhD on December 15, 2006 Andrew Viterbi Dr. Andrew J. Viterbi, Ph.D. serves as President of the Viterbi Group LLC and Co- founded it in 2000. Dr. Viterbi co-founded Continuous Computing Corp. and served as its Chief Technology Officer from July 1985 to July 1996. From July 1983 to April 1985, he served as the Senior Vice President and Chief Scientist of M/A-COM Inc. In July 1985, he co-founded QUALCOMM Inc., where Dr. Viterbi served as the Vice Chairman until 2000 and as the Chief Technical Officer until 1996. Under his leadership, QUALCOMM received international recognition for innovative technology in the areas of digital wireless communication systems and products based on Code Division Multiple Access (CDMA) technologies. From October 1968 to April 1985, he held various Executive positions at LINKABIT (M/A-COM LINKABIT after August 1980) and served as the President of the M/A-COM LINKABIT. In 1968, Dr. Viterbi Co-founded LINKABIT Corp., where he served as an Executive Vice President and later as the President in the early 1980's. Dr. Viterbi served as an Advisor at Avalon Ventures. He served as the Vice-Chairman of Continuous Computing Corp. since July 1985. During most of his period of service with LINKABIT, Dr. Viterbi served as the Vice-Chairman and a Director. He has been a Director of Link_A_Media Devices Corporation since August 2010. He serves as a Director of Continuous Computing Corp., Motorola Mobility Holdings, Inc., QUALCOMM Flarion Technologies, Inc., The International Engineering Consortium and Samsung Semiconductor Israel R&D Center Ltd.
    [Show full text]
  • The Human Use of Human Beings
    THE HUMAN USE OF HUl\IAN BEINGS This is one of the fundamental documents of our time, a period characterized by the concepts of 'information' and 'communica­ tions'. Norbert Wiener, a child prodigy and a great mathematician, coined the term 'cybernetics' to characterize a very general science of 'control and communication in the animal and machine'. It brought together concepts from engineering, the study of the nervous system and statistical mechanics (e.g. entropy). From these he developed concepts that have become pervasive through science (especially biology and computing) and common parlance: 'in­ formation', 'message', 'feedback' and 'control'. He wrote, 'the thought of every age is reflected in its technique ... If the seventeenth and early eighteenth centuries are the age of clocks, and the later eighteenth and nineteenth centuries constitute the age of steam engines, the present time is the age of communication and control.' In this volume Norbert Wiener spells out his theories for the general reader and reflects on the social issues raised by the dramatically increasing role of science and technology in the new age - the age in which we are now deeply and problematically embroiled. His cautionary remarks are as relevant now as they were when the book first appeared in the 1950s. Norbert Wiener (1894-1964), Professor of Mathematics at the Massachusetts Institute of Technology from 1919 onwards, wrote numerous books on mathematics and engineering. Having de­ veloped methods useful to the military during World War Two, he later refused to do such work during the Cold War, while proposing non-military models of cybernetics.
    [Show full text]
  • AWAR Volume 24.Indb
    THE AWA REVIEW Volume 24 2011 Published by THE ANTIQUE WIRELESS ASSOCIATION PO Box 421, Bloomfi eld, NY 14469-0421 http://www.antiquewireless.org i Devoted to research and documentation of the history of wireless communications. Antique Wireless Association P.O. Box 421 Bloomfi eld, New York 14469-0421 Founded 1952, Chartered as a non-profi t corporation by the State of New York. http://www.antiquewireless.org THE A.W.A. REVIEW EDITOR Robert P. Murray, Ph.D. Vancouver, BC, Canada ASSOCIATE EDITORS Erich Brueschke, BSEE, MD, KC9ACE David Bart, BA, MBA, KB9YPD FORMER EDITORS Robert M. Morris W2LV, (silent key) William B. Fizette, Ph.D., W2GDB Ludwell A. Sibley, KB2EVN Thomas B. Perera, Ph.D., W1TP Brian C. Belanger, Ph.D. OFFICERS OF THE ANTIQUE WIRELESS ASSOCIATION DIRECTOR: Tom Peterson, Jr. DEPUTY DIRECTOR: Robert Hobday, N2EVG SECRETARY: Dr. William Hopkins, AA2YV TREASURER: Stan Avery, WM3D AWA MUSEUM CURATOR: Bruce Roloson W2BDR 2011 by the Antique Wireless Association ISBN 0-9741994-8-6 Cover image is of Ms. Kathleen Parkin of San Rafael, California, shown as the cover-girl of the Electrical Experimenter, October 1916. She held both a commercial and an amateur license at 16 years of age. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the copyright owner. Printed in Canada by Friesens Corporation Altona, MB ii Table of Contents Volume 24, 2011 Foreword ....................................................................... iv The History of Japanese Radio (1925 - 1945) Tadanobu Okabe .................................................................1 Henry Clifford - Telegraph Engineer and Artist Bill Burns ......................................................................
    [Show full text]