• Abstract and Figures
• Public Full-text

INFORMATION THEORETIC RESULTS ON EXACT DISTRIBUTED SIMULATION AND INFORMATION EXTRACTION A DISSERTATION SUBMITTED TO THE DEPARTMENT OF ELECTRICAL ENGINEERING AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY Gowtham Ramani Kumar August 2014 © 2014 by Gowtham Kumar Ramani Kumar. All Rights Reserved. Re-distributed by Stanford University under license with the author. This work is licensed under a Creative Commons Attribution- 3.0 United States License. http://creativecommons.org/licenses/by/3.0/us/ This dissertation is online at: http://purl.stanford.edu/sy415my4619 ii I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Abbas El Gamal, Primary Adviser I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Ayfer Ozgur Aydin I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Tsachy Weissman Approved for the Stanford University Committee on Graduate Studies. Patricia J. Gumport, Vice Provost for Graduate Education This signature page was generated electronically upon submission of this dissertation in electronic format. An original signed hard copy of the signature page is on file in University Archives. iii Abstract In the first part of the thesis we will introduce the notion of exact common infor- mation, which is the minimum description length of the common randomness needed for the exact distributed generation of two correlated random variables (X,Y ). We introduce the quantity G(X; Y ) = minX→W →Y H(W ) as a natural bound on the ex- act common information. We then introduce the exact common information rate, which is the minimum description rate of the common randomness for the exact gen- eration of correlated sources (X,Y ). We give a multiletter characterization for it as n n the limit G(X; Y ) = limn→∞(1/n)G(X ; Y ). While in general G¯(X; Y ) is greater than or equal to the Wyner common information, we show that they are equal for the Symmetric Binary Erasure Source. We then discuss the computation of G(X; Y ). In the second part, we will introduce a conjecture regarding the maximum mutual information a Boolean function can reveal about noisy inputs. Specifically, let (X,Y ) be a doubly symmetric binary source with cross-over probability α. For any boolean function b : 0, 1 n 0, 1 , we conjecture that I(b(Xn); Y n) 1 H(α). While the { } →{ } ≤ − conjecture remains open, we provide substantial evidence supporting its validity. iv Acknowledgements First of all, I like to thank my parents for all they have done for me. If not for their numerous sacrifices, I won’t be a Stanford student in the first place. I am from a lesser developed city in India. We lacked qualified teachers to help me get in to the Indian Institutes of Technology (IIT), premier engineering colleges in India with an acceptance ratio less than 1%. My parents made for that gap by over-spending on books and creating a personal library of all great science text-books in my home. When I was preparing for the IIT entrance examination, my school teachers in fact discouraged me by stating that they haven’t seen anyone ever clear the exam. In spite of these difficulties, my parents constantly believed in my ability to clear the IIT entrance exam. They were there for financial and moral support throughout my struggle. I would also like to take this opportunity to thank my family. In particular, I like to thank my wife for her moral support and for making a video of my PhD defense. Before I thank my Stanford colleagues, I would love to thank Prof. Andrew Thangaraj from IIT Madras, who served as my undergraduate final year project ad- visor. He believed in me and encourage me to publish early even in my undergraduate years. I would also like to thank all my professors and fellow students at IIT Madras who were instrumental in imparting knowledge through various fundamental courses that would later get me admitted to Stanford university. Next, I would like to give a special thanks to the late Prof. Thomas Cover who was my first PhD advisor at Stanford. I read Tom’s wonderful book on Information theory before I even obtained an admit from Stanford. Tom believes that puzzles lead to interesting research problems. When any new student first approaches Tom v for a PhD position, he is asked to attend the Wednesday puzzle meetings and wait for something interesting to happen. These meetings go from elementary high school puzzles to deep research topics in information theory. It keeps everyone engaged, from novices to experts. In addition to puzzles, Tom loved theory. Tom didn’t care about practical problems as much. His only criterion in evaluating a research problem is if it is an interesting or elegant problem. I like to recall one favorite quote from Tom: “Theory is the first term in the Taylor series for practice”. Finally, I like to comment on Tom’s dedication to teaching. We all remember the unfortunate day when Tom passed away on March 26, 2012. Now that I know he chose to teach until his last moments, even after retirement and even after his health was low, I feel truly gifted to have had him as my advisor and be his 26-th PhD student (if you start counting from 0). Next, I would like to thank my advisor, Prof. Abbas El Gamal. My first in- teraction with Abbas is during my first quarter at Stanford when I took his course on Network Information Theory (NIT). The course taught me the fundamentals of NIT that every researcher needs to know. One only has to read the unreadable in- formation theory papers before 1980 to appreciate the simplicity with which Abbas explains concepts. Later in my PhD career, when my original advisor Tom passed away, I turned to Abbas for support. He then took over as my primary advisor and kept the ball rolling. He helped me through the herculean task of solving my PhD thesis problem with his amazing ability to simplify the questions and ask the most fundamental questions. He took over Tom’s challenging problem and distilled it into the concept of exact common information, one of my primary thesis topics. Abbas is also known for his ability to give great presentations. He spends hours with his students in preparing for presentations and in writing papers. I still remember going through numerous dry runs of our ISIT presentations. We started our rehearsals 2 months before the conference and were doing our 10th rehearsal when the other in- formation theory students haven’t yet made the first draft. We all learnt the value of hard work. No great speaker is born great; great speakers are made. Next, I would like to thank Prof. Tsachy Weissman who served as my associate advisor. He taught me the course on universal schemes in information theory, topics vi that every researcher in the field must be familiar with. I attended a lot of his research meetings and interacted with his students throughout my career. Next, I would like to thank Prof. Ayfer Ozgur, Prof. Thomas Courtade and Prof. Amir Dembo, who served as committee members of my PhD oral defense and provided valuable insight and feedback on my research. Next, I would like to thank Yeow Khiang Chia, whose discussions resulted in my formulation of the Boolean functions conjecture. I would also like to thank Prof. Chandra Nair and Prof. Thomas Courtade for their help in making progress on the conjecture. Next, I would like to thank all my collaborators and everyone from the informa- tion theory community I interacted with, including Lei Zhao, Paul Cuff, Haim Per- muter, Idoia Ochoa Alvarez, Mikel Hernaez, Kartik Venkat, Vinith Misra, Alexandros Manolakos, Albert No, Bernd Bandemer, Cheuk Ting Li, Hyeji Kim, Jiangtao and Young-Han-Kim who helped me at various stages of my PhD career and added to the richness of my PhD experience. Next I would like to thank every professors who taught me interesting courses at Stanford and created a unique Stanford experience. I would especially like to thank Prof. Stephen Boyd for his course on convex optimization, Prof. Andrew Ng for his course on Machine Learning and Matt Vassar for his course on public speaking. I would also love to thank Stanford for teaching me various unique skills, one of which is a circus art named aerial fabrics. I like to thank my aerial fabrics teachers Elizabeth, Rachel, Erica, Kristin and Graham. Next, I would like to thank the support staff at Stanford, including Denise Murphy, Karin Sligar, Katt Clark, Andrea Kuduk, Douglas Chaffee, Vickie Carrilo and IT expert John DeSilva. Their efficiency and expertise saved a lot of precious time and helped me focus on my research. Finally, I would love to thank almighty for guiding me in a successful path. At this point, I would love to point out an incident that explains a butterfly effect in my life. During my early years, my dad wanted to enroll me in a school where the medium of instruction is our local language Tamil rather than English. If he had proceeded with his plan, my life and career would be completely different from what vii it is now.

Load more

  89

Copy Link