FORMAL LANGUAGES AND THEIR RELATION TO AUTOMATA FORMAL LANGUAGES AND THEIR RELATION TO AUTOMATA JOHN E. HOPCROFT Cornell University, Ithaca, New York JEFFREY D. ULLMAN Bell Telephone Laboratories, Murray Hill, New Jersey A VV ADDISON-WESLEY PUBLISHING COMPANY Reading, Massachusetts Menlo Park, California • London • Don Mills, Ontario This book is in the ADDISON-WESLEY SERIES IN COMPUTER SCIENCE AND INFORMATION PROCESSING Consulting Editors MICHAEL A. HARRISON RICHARD S. VARGA Copyright © 1969 by Addison-Wesley Publishing Company, Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher. Printed in the United States of America. Published simultaneously in Canada. Library of Congress Catalog Card No. 69-14297. PREFACE The study of formal languages constitutes an important subarea of computer science. This area sprang to life around 1956 when Noam Chomsky gave a mathematical model of a grammar in connection with his study of natural languages. Shortly afterwards, the concept of a grammar was found to be of great importance to the programmer when the syntax of the programming language ALGOL was defined by a context-free grammar. This development led naturally to syntax-directed compiling and the concept of a compiler compiler. Since then a considerable flurry of activity has taken place, the results of which have related formal languages and automata theory to such an extent that it is impossible to treat the areas separately. By now, no serious study of computer science would be complete without a knowledge of the techniques and results from language and automata theory. This book presents the theory of formal languages as a coherent theory and makes explicit its relationship to automata. The book begins with an explanation of the notion of a finite description of a language. The funda- mental descriptive device--the grammar--is explained, as well as its three major subclasses--regular, context-free, and context-sensitive grammars. The context-free grammars are treated in detail, and such topics as normal forms, derivation trees, and ambiguity are covered. Four types of automata equivalent to the four types of grammars are described. These automata are the finite automaton, the pushdown automaton, the linear bounded autom- aton, and the Turing machine. The Turing machine is covered in detail, and unsolvability of the halting problem shown. The book concludes with certain advanced topics in language theory--closure properties, computational complexity, deterministic pushdown automata, LR(k) grammars, stack automata, and decidability. The emphasis is on ideas and ease of understanding rather than undue formalism. In some cases the details of long and tedious proofs are omitted. However, in all cases sufficient intuitive explanation is given so that the reader may easily provide the rigorous proof if desired. The book is intended primarily as a textbook for a first or second year graduate course in formal languages. It is self-contained and presupposes only the normal level of maturity expected of a beginning graduate student. Although not essential, it has been found that a course in finite state machines or Turing machines is useful preparation. The book is not intended as a research monogram on formal languages, although it covers many of the known results, and much of the material presented previously existed only in journals of mathematics and computer science. The chapters are provided with a guide to references on the material covered and related topics. The problems are an integral part of the text and their difficulty ranges from almost trivial to extremely difficult. The material for this book is based upon class notes for courses in language theory taught by the authors at Princeton, Columbia, and Cornel1. The authors would like to thank the many people who offered their sugges- tions and criticism. In particular, we would like to thank A. V. Aho, S. Amoroso, A. Korenjak, and M. Harrison. Thanks are also due to Bell Telephone Laboratories, and Princeton and Cornell Universities for provid- ing the facilities for the preparation of the work. J. E. H. J. D. U. CONTENTS Chapter 1 Languages and Their Representations 1.1 Alphabets and Languages ........... 1 1.2 Procedures and Algorithms .......... 2 1.3 Representations of Languages .......... 5 Problems ................ 7 Chapter 2 Grammars 2.1 Motivation ............... 8 2.2 The Formal Notion of a Grammar ........ 10 2.3 The Types of Grammars ........... 13 2.4 The Empty Sentence ............ 15 2.5 Recursiveness of Context-Sensitive Grammars ..... 16 2.6 Derivation Trees for Context-Free Grammars ..... 18 Problems ................ 24 References ............... 25 Chapter 3 Finite Automata and Regular Grammars 3.1 The Finite Automaton ............ 26 3.2 Equivalence Relations and Finite Automata ..... 28 3.3 Nondeterministic Finite Automata ........ 30 3.4 Finite Automata and Type 3 Languages ....... 33 3.5 Properties of Type 3 Languages ......... 35 3.6 Solvable Problems Concerning Finite Automata .... 39 3.7 Two-way Finite Automata ........... 41 Problems ................ 44 References ............... 45 Chapter 4 Context-Free Grammars 4.1 Simplification of Context-Free Grammars ...... 46 4.2 Chomsky Normal Form ........... 51 4.3 Greibach Normal Form ........... 53 4.4 Solvability of Finiteness and the "uvwxy Theorem" . 57 4.5 The Self-Embedding Property .......... 61 vii 4.6 ~-Rules in Context-Free Grammars ........ 62 4.7 Special Types of Context-Free Languages and Grammars. 63 Problems ................ 65 References ............... 67 Chapter 5 Pushdown Automata 5.1 Informal Description ............ 68 5.2 Definitions ....... ........ 70 5.3 Nondeterministic Pushdown Automata and Context-Free Languages ............... 74 Problems ................ 79 References ............... 79 Chapter 6 Turing Machines 6.1 Introduction ............... 80 6.2 Definitions and Notation ........... 80 6.3 Techniques for Turing Machine Construction ..... 84 6.4 The Turing Machine as a Procedure ....... 91 6.5 Modifications of Turing Machines ........ 92 6.6 Restricted Turing Machines Equivalent tO the Basic Model . 98 Problems ............... 101 References ............... 101 Chapter 7 Turing Machines: The Halting Problem, Type 0 Languages 7.1 Informal Discussion ............ 102 7.2 A Universal Turing Machine ......... 102 7.3 The Unsolvability of the Halting Problem ...... 108 7.4 The Class of Recursive Sets .......... 109 7.5 Turing Machines and Type 0 Grammars ...... 111 Problems ................ 113 References ............... 114 Chapter 8 Linear Bounded Automata and Context-Sensitive Languages 8.1 Introduction ............... 115 8.2 Relation of Linear Bounded Automata to Context-Sensitive Languages ............... 116 8.3 The Context-Sensitive Languages as a Subclass of the Recursive Sets ................. 117 Problems ................ 118 References ............... 119 Chapter 9 Operations on Languages 9.1 Introduction ............... 120 9.2 Closure Under Elementary Operations ....... 120 9.3 Closure Under Mappings ........... 124 Problems ................ 133 References ............... 134 Chapter 10 Time- and Tape-Bounded Turing Machines 10.1 Introduction ............ , . 135 10.2 Definitions ............... 135 10.3 "Speed Up" and "Tape Reduction" Theorems ..... 137 10.4 Single-Tape Turing Machines and Crossing Sequences . 143 10.5 Lower Bounds on Tape Complexity ........ 147 10.6 Tape and Time Hierarchies .......... 149 Problems ................ 154 References ................ 155 Chapter 11 Time and Space Bounds for Recognizing Context-Free Languages 11.1 Introduction ............... 156 11.2 Time Requirements for Recognition of Context-Free Languages 156 11.3 Space Requirements for Recognition of Context-Free Languages 160 Problems ................ 164 References ............... 165 Chapter 12 Deterministic Pushdown Automata 12.1 Introduction ............... 166 12.2 Complements of Deterministic Languages ...... 167 12.3 Properties of Deterministic Languages ....... 171 12.4 Context-Free Languages That Are Not Deterministic . 180 12.5 LR(k) Grammars . ............ 180 Problems ................ 187 References ............... 188 Chapter 13 Stack Automata 13.1 Definitions ............... 189 13.2 Restricted Types of Stack Automata ........ 192 13.3 The Power of Two-Way Stack Automata ...... 192 13.4 The Power of One-Way Stack Automata ...... 201 13.5 Recursiveness of Stack Automata ........ 207 13.6 Closure Properties ............. 209 Problems ................ 209 References ............... 209 Chapter 14 Decidability 14.1 Decidable and Undecidable Questions ....... 211 14.2 Post's Correspondence Problem ......... 212 14.3 A Question Concerning Context-Sensitive Languages . 219 14.4 Unsolvable Questions for Context-Free Languages .... 219 14.5 Ambiguity in Context-Free Languages ....... 222 14.6 Unsolvable Questions Concerning Deterministic Context-Free Languages ............... 229 14.7 Summary of Decision Problems for Regular, LR(k), Context- Free, Context-Sensitive, and Type 0 Grammars .... 231 Problems ................ 231 References ............... 232 Bibliography .......... .... 233 Index ................. 239 CHAPTER 1 LANGUAGES AND THEIR REPRESENTATIONS 1.1 ALPHABETS AND LANGUAGES What is the theory of languages ? To answer this question we first ask"