Chapter 11 a Hierarchy of Formal Languages and Automata

✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 277 — #1 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

A Hierarchy of © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTIONFormal LanguagesNOT FOR SALE OR DISTRIBUTION and Automata © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

© Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTIONe now return our attention to ourNOT main FOR interest, SALE the studyOR DISTRIBUTION of formal languages. Our immediate goal will be to examine the languages W associated with Turing machines and some of their restrictions. Be- cause Turing machines can perform any kind of algorithmic computation, we expect to ﬁnd that the family of languages associated with © Jones & Bartlett Learning, LLC them isquite broad.© It Jones includesnot & Bartlett only regular Learning, and context-free LLC languages, but also the various examples we have encountered that lie outside NOT FOR SALE OR DISTRIBUTIONthese families. The nontrivialNOT FOR question SALE is whether OR DISTRIBUTION there are any languages that are not accepted by some Turing machine. We will answer this question ﬁrst by showing that there are more languages than Turing machines, so that there must be some languages for which there are no Turing ma- © Joneschines. & Bartlett The proof Learning, is short and LLC elegant, but nonconstructive,© Jones and gives & Bartlett little Learning, LLC insight into the problem. For this reason, we will establish the existence NOT FORof SALE languagesnot OR DISTRIBUTION recognizable by Turing machinesthroughNOT more FOR explicit SALE OR DISTRIBUTION examplesthat actually allow usto identify one such language. Another avenue of investigation will be to look at the relation between Turing ma- chinesand certain typesof grammarsand to establisha connection between © Jones & Bartlett Learning,these grammars LLC and regular and context-free© Jones grammars. & Bartlett This Learning, leads to a LLC hierarchy of grammars and through it to a method for classifying language NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

277

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 278 — #2 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 278 Chapter 11 A Hierarchy of Formal Languages and Automata

© Jonesfamilies. & Bartlett Some set-theoreticLearning, LLC diagrams illustrate the relationships© Jones between & Bartlett Learning, LLC variouslanguage familiesclearly. NOT FOR StrictlySALE speaking,OR DISTRIBUTION many of the arguments in this chapterNOT are FOR valid SALE only OR DISTRIBUTION for languages that do not include the empty string. This restriction arises from the fact that Turing machines, as we have deﬁned them, cannot accept the empty string. To avoid having to rephrase the deﬁnition or having to © Jones & Bartlett Learning,add a repeated LLC disclaimer, we make the tacit© Jones assumption & Bartlett that the languagesLearning, LLC discussed in this chapter, unless otherwise stated, do not contain λ.Itisa NOT FOR SALE ORtrivial DISTRIBUTION matter to restate everything so thatNOTλ isincluded, FOR SALE but weOR will DISTRIBUTION leave thisto the reader.

© Jones & Bartlett Learning, LLC11.1 Recursive© and Jones Recursively & Bartlett Learning, Enumerable LLC NOT FOR SALE OR DISTRIBUTION LanguagesNOT FOR SALE OR DISTRIBUTION We start with some terminology for the languages associated with Turing machines. In doing so, we must make the important distinction between languagesfor which there existsanaccepting Turing machine and languages © Jonesfor & which Bartlett there existsLearning, a membership LLC algorithm. Because a© Turing Jones machine & Bartlett Learning, LLC NOT FORdoes SALE not necessarily OR DISTRIBUTION halt on input that it does not accept,NOT the ﬁrst FOR does SALE not OR DISTRIBUTION imply the second.

Definition 11.1 © Jones & Bartlett Learning,A language L LLCissaidto be recursively enumerable© Jones &if thereBartlett exists Learning, a Turing LLC NOT FOR SALE ORmachine DISTRIBUTION that acceptsit. NOT FOR SALE OR DISTRIBUTION

Thisdefinition impliesonly that there existsaTuring machine M, such that, for every w ∈ L, ∗ © Jones & Bartlett Learning, LLC © Jonesq0w M &x Bartlett1qf x2, Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION with qf a final state. The definition says nothing about what happens for w not in L; it may be that the machine haltsin a nonfinal stateor that it never haltsand goesinto an infinite loop. We can be more demanding and ask that the machine tell us whether or not any given input is in its © Joneslanguage. & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION Definition 11.2 A language L on Σ issaidto be recursive if there exists a Turing machine M that accepts L and that haltson every w in Σ+. In other words, a © Jones & Bartlett Learning,language is recursiveLLC if and only if there© exists Jones a membership & Bartlett algorithm Learning, for LLC NOT FOR SALE OR it.DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 279 — #3 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 11.1 Recursive and Recursively Enumerable Languages 279

© Jones & BartlettIf a language Learning, is recursive, LLC then there exists an easily© Jones constructed & Bartlett enu- Learning, LLC meration procedure. Suppose that M isa Turing machine that determines NOT FORmembership SALE OR in DISTRIBUTION a recursive language L. We first constructNOT another FOR TuringSALE OR DISTRIBUTION machine, say M, that generatesall stringsinΣ + in proper order, let us say w1,w2, .... As these strings are generated, they become the input to M, which is modified so that it writes strings on its tape only if they are in L. © Jones & Bartlett Learning,That thereLLC is also an enumeration© procedure Jones for& everyBartlett recursively Learning, enu- LLC merable language is not as easy to see. We cannot use the previous argument NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION as it stands, because if some wj isnot in L, the machine M, when started with wj on itstape, may never halt and therefore never get to the strings in L that follow wj in the enumeration. To make surethat thisdoesnot happen, the computation isperformed in a different way. We firstget M © Jones & Bartlett Learning, LLC to generate w1 and let©M Jonesexecute & one Bartlett move on Learning, it. Then we letLLCM generate NOT FOR SALE OR DISTRIBUTIONw2 and let M executeNOT one move FOR on wSALE2, followed OR byDISTRIBUTION the second move on w1. After this, we generate w3 and do one step on w3, the second step on w2, the third step on w1, and so on. The order of performance is depicted in Figure 11.1. From this, it is clear that M will never get into an infinite loop. Since any w ∈ L isgenerated by M and accepted by M in a finite © Jonesnumber & Bartlett of steps, Learning, every string LLC in L iseventually produced© byJonesM. & Bartlett Learning, LLC NOT FOR SALEIt is easy OR to DISTRIBUTION see that every language for which an enumerationNOT FOR procedure SALE OR DISTRIBUTION exists is recursively enumerable. We simply compare the given input string against successive strings generated by the enumeration procedure. If w ∈ L, we will eventually get a match, and the process can be terminated. Definitions11.1 and 11.2 give usvery little insight into the nature of © Jones & Bartlett Learning,either recursive LLC or recursively enumerable© Jones languages. & Bartlett These definitions Learning, at- LLC tach names to language families associated with Turing machines, but shed NOT FOR SALE OR DISTRIBUTIONno light on the nature of representativeNOT languages FOR in SALE these families. OR DISTRIBUTION Nor do they tell us much about the relationships between these languages or their connection to the language familieswe have encountered before. We are therefore immediately faced with questions such as “Are there languages that are recursively enumerable but not recursive?” and “Are there lan- © Jones & Bartlett Learning, LLC guages, describable somehow,© Jones that & are Bartlett not recursively Learning, enumerable?” LLC While NOT FOR SALE OR DISTRIBUTIONwe will be able to supplyNOT some FOR answers, SALE we willOR not DISTRIBUTION be able to produce very explicit examples to illustrate these questions, especially the second one.

© Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC w w w w . . . Figure 11.1NOT FOR SALE OR1 DISTRIBUTION2 3 4 NOT FOR SALE OR DISTRIBUTION First move ... . Second move .... © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC Third move NOT FOR SALE OR DISTRIBUTION. . NOT FOR SALE OR DISTRIBUTION . .

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 280 — #4 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 280 Chapter 11 A Hierarchy of Formal Languages and Automata

© JonesLanguages & Bartlett That Learning, Are Not LLC Recursively Enumerable© Jones & Bartlett Learning, LLC NOT FORWe canSALE establish OR DISTRIBUTION the existence of languages that are notNOT recursively FOR SALE enu- OR DISTRIBUTION merable in a variety of ways. One is very short and uses a very fundamental and elegant result of mathematics.

Theorem 11.1 Let S be an inﬁnite countable set. Then its powerset 2S isnot countable. © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC S NOT FOR SALE ORProof: DISTRIBUTIONLet S = {s1,s2,s3, ...}. Then anyNOT element FORt of 2SALEcan be OR represented DISTRIBUTION by a sequence of 0’s and 1’s, with a 1 in position i if and only if si isin t. For example, the set {s2,s3,s6} isrepresented by 01100100..., while S {s1,s3,s5, ...} isrepresented by 10101... . Clearly, any element of 2 can be represented by such a sequence, and any such sequence represents a unique © Jones & Bartlett Learning, LLCelement of 2S. Suppose that© Jones 2S were & countable; Bartlett thenLearning, itselementscould LLC be NOT FOR SALE OR DISTRIBUTIONwritten in some order, sayNOTt1,t2 FOR, ..., and SALE we could OR enter DISTRIBUTION these into a table, as shown in Figure 11.2. In this table, take the elements in the main diagonal, and complement each entry, that is, replace 0 with 1, and vice versa. In the example in Figure 11.2, the elementsare 1100..., so we get 0011 ... as the result. The new sequence along the diagonal represents some element S © Jonesof 2& ,sayBartlettti for Learning, some i. But LLC it cannot be t1 because© it diﬀersJones from & Bartlettt1 Learning, LLC NOT FORthrough SALEs1. ForOR theDISTRIBUTION same reason it cannot be t2, t3, or anyNOT other FOR entry SALE in OR DISTRIBUTION the enumeration. This contradiction creates a logical impasse that can be removed only by throwing out the assumption that 2S iscountable.

© Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTIONThiskind of argument, becauseit involvesaNOT FOR manipulation SALE OR of the DISTRIBUTION diagonal elementsof a table, iscalled diagonalization. The technique is attributed to the mathematician G. F. Cantor, who used it to demonstrate that the setof real numbersisnot countable. In the next few chapters,we will see a similar argument in several contexts. Theorem 11.1 is diagonal- © Jones & Bartlett Learning, LLCization in itspurestform.© Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

Figure 11.2 t1 10000. . .

t2 11000. . .

© Jonest &11010. Bartlett Learning, LLC . . © Jones & Bartlett Learning, LLC NOT FOR3 SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION t 11001. . . 4 ...... © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 281 — #5 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 11.1 Recursive and Recursively Enumerable Languages 281

© Jones & BartlettAs an immediate Learning, consequence LLC of this result, we can© show Jones that, & in Bartlett some Learning, LLC sense, there are fewer Turing machines than there are languages, so that NOT FORthere SALE must OR be someDISTRIBUTION languages that are not recursively enumerable.NOT FOR SALE OR DISTRIBUTION Theorem 11.2 For any nonempty Σ, there exist languages that are not recursively enumerable.

© Jones & Bartlett Learning,Proof: A LLC language is a subset of Σ∗, and© Jones every such & Bartlett subset is aLearning, language. LLC ∗ NOT FOR SALE OR DISTRIBUTIONTherefore, the set of all languages isNOT exactly FOR 2Σ .SALE Since OR Σ∗ isinﬁnite,DISTRIBUTION Theorem 11.1 tellsusthat the setof all languageson Σ isnot countable. But the set of all Turing machines can be enumerated, so the set of all recursively enumerable languages is countable. By Exercise 16 at the end of thissection,thisimpliesthat there mustbe somelanguageson Σ that © Jones & Bartlett Learning, LLC are not recursively enumerable.© Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

This proof, although short and simple, is in many ways unsatisfying. It iscompletely nonconstructive and, while it tellsusof the existenceof © Jonessome & Bartlett languages Learning, that are not LLC recursively enumerable, it© gives Jones us no & feeling Bartlett Learning, LLC NOT FORat SALE all for whatOR DISTRIBUTION these languages might look like. In theNOT next setFOR of results,SALE OR DISTRIBUTION we investigate the conclusion more explicitly.

A Language That Is Not Recursively Enumerable © Jones & Bartlett Learning,Since every LLC language that can be described© Jones in a direct & Bartlett algorithmic Learning, fashion LLC NOT FOR SALE OR DISTRIBUTIONcan be accepted by a Turing machine andNOT hence FOR isrecursively SALE OR enumerable, DISTRIBUTION the description of a language that is not recursively enumerable must be indirect. Nevertheless, it is possible. The argument involves a variation on the diagonalization theme. Theorem 11.3 There exists a recursively enumerable language whose complement is not © Jones & Bartlett Learning, LLC recursively enumerable.© Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION Proof: Let Σ = {a}, and consider the set of all Turing machines with this input alphabet. By Theorem 10.3, this set is countable, so we can associate an order M1,M2, ... with itselements. For each Turing machine Mi, there is an associated recursively enumerable language L (Mi). Conversely, for © Joneseach & Bartlett recursively Learning, enumerable LLC language on Σ, there is some© Jones Turing machine& Bartlett Learning, LLC NOT FORthat SALE acceptsit. OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION We now consider a new language L deﬁned asfollows. For each i ≥ 1, i i the string a isin L if and only if a ∈ L (Mi). It isclear that the language i i L is well deﬁned, since the statement a ∈ L (Mi), and hence a ∈ L, must be either true or false. Next, we consider the complement of L, © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC i i NOT FOR SALE OR DISTRIBUTION L = a : a ∈/NOTL (M iFOR) , SALE OR DISTRIBUTION(11.1)

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 282 — #6 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 282 Chapter 11 A Hierarchy of Formal Languages and Automata

© Joneswhich & Bartlett is also well Learning, deﬁned but, LLC as we will show, is not recursively© Jones enumerable. & Bartlett Learning, LLC We will show this by contradiction, starting from the assumption that NOT FORL is SALE recursively OR enumerable. DISTRIBUTION If this is so, then there mustNOT be some FOR Turing SALE OR DISTRIBUTION machine, say Mk, such that

L = L (Mk) . (11.2) © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC k k ∈ NOT FOR SALE ORConsider DISTRIBUTION the string a . Isit in L or in L?NOT Suppose FOR that SALEa LOR. By DISTRIBUTION (11.2) thisimpliesthat

k a ∈ L (Mk) .

Alternatively, if we assume that ak isin L, then ak ∈/ L and (11.2) implies that © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC k ∈ NOT FOR SALE OR DISTRIBUTIONa / L (Mk) . NOT FOR SALE OR DISTRIBUTION

But then from (11.1) we get that

ak ∈ L. © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE ORThe DISTRIBUTION contradiction is inescapable, and we mustNOT conclude FOR SALE that our OR assumption DISTRIBUTION that L isrecursively enumerable isfalse. To complete the proof of the theorem as stated, we must still show that L is recursively enumerable. For this we can use the known enumeration procedure for Turing machines. Given ai, we first find i by counting the © Jones & Bartlett Learning, LLCnumber of a’s. We then use© Jones the enumeration & Bartlett procedure Learning, for Turing LLC machines i to find Mi. Finally, we give itsdescriptionalong with a to a universal NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTIONi i Turing machine Mu that simulates the action of M on a .Ifa isin L, the computation carried out by Mu will eventually halt. The combined effect of thisisa Turing machine that acceptsevery ai ∈ L. Therefore, by Definition 11.1, L isrecursively enumerable. © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

The proof of thistheorem explicitly exhibits,through (11.1), a well- defined language that isnot recursively enumerable. Thisisnot to say © Jones & Bartlett Learning,that there isan LLC easy, intuitive interpretation© Jones of L; it& wouldBartlett be difficultLearning, to LLC exhibit more than a few trivial membersof thislanguage. Nevertheless, L NOT FOR SALE OR isproperlyDISTRIBUTION defined. NOT FOR SALE OR DISTRIBUTION

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 283 — #7 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 11.1 Recursive and Recursively Enumerable Languages 283

© JonesA & LanguageBartlett Learning, That Is Recursively LLC Enumerable© but Jones Not & Bartlett Learning, LLC NOT FORRecursive SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION Next, we show there are some languages that are recursively enumerable but not recursive. Again, we need do so in a rather roundabout way. We begin by establishing a subsidiary result.

© JonesTheorem & Bartlett 11.4 Learning,If a language LLCL and itscomplement L are© bothJones recursively & Bartlett enumerable, Learning, then LLC NOT FOR SALE OR DISTRIBUTIONboth languagesare recursive. If L isrecursive,NOT thenFOR LSALEis also OR recursive, DISTRIBUTION and consequently both are recursively enumerable.

Proof: If L and L are both recursively enumerable, then there exist Turing © Jones & Bartlett Learning, LLC machines M and M that© Jones serve as& enumerationBartlett Learning, procedures LLC for L and L, respectively. The ﬁrst will produce w1,w2, ... in L, the second w1, w2, ... in NOT FOR SALE OR DISTRIBUTION NOT FOR SALE+ OR DISTRIBUTION L. Suppose now we are given any w ∈ Σ . We ﬁrst let M generate w1 and compare it with w. If they are not the same, we let M generate w1 and compare again. If we need to continue, we next let M generate w2, then + M generate w2, and so on. Any w ∈ Σ will be generated by either M or © JonesM &, Bartlett so eventually Learning, we will get LLC a match. If the matching© string Jones is produced & Bartlett Learning, LLC by M, w belongsto L, otherwise it is in L. The process is a membership NOT FORalgorithm SALE forOR both DISTRIBUTIONL and L, so they are both recursive.NOT FOR SALE OR DISTRIBUTION For the converse, assume that L is recursive. Then there exists a membership algorithm for it. But this becomes a membership algorithm for L by simply complementing its conclusion. Therefore, L isrecursive. Since © Jones & Bartlett Learning,any recursive LLC language is recursively enumerable,© Jones & the Bartlett proof is Learning, completed. LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

From this, we conclude directly that the family of recursively enumerable languagesand the family of recursive languagesare not identical. The © Jones & Bartlett Learning, LLC language L in Theorem© 11.3Jones is in & the Bartlett ﬁrst but notLearning, in the second LLC family. NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION Theorem 11.5 There exists a recursively enumerable language that is not recursive; that is, the family of recursive languages is a proper subset of the family of recursively enumerable languages.

© JonesProof: & BartlettConsider Learning, the language LLCL of Theorem 11.3. Thislanguage© Jones isrecur- & Bartlett Learning, LLC NOT FORsively SALE enumerable, OR DISTRIBUTION but itscomplement isnot. Therefore, byNOT Theorem FOR 11.4,SALE OR DISTRIBUTION it isnot recursive, giving usthe looked-for example.

© Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC We see from this that there are indeed well-deﬁned languages for which NOT FOR SALE OR DISTRIBUTIONone cannot construct a membership algorithm.NOT FOR SALE OR DISTRIBUTION

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 284 — #8 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 284 Chapter 11 A Hierarchy of Formal Languages and Automata

© JonesE XERCISES& Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 1. Prove that the set of all real numbers is not countable. 2. Prove that the set of all languages that are not recursively enumerable is not countable. s © Jones & Bartlett Learning,3. Let L be LLC a ﬁnite language. Show that© then JonesL+ is & recursively Bartlett enumerable. Learning, LLC + NOT FOR SALE OR DISTRIBUTIONSuggest an enumeration procedure for LNOT. FOR SALE OR DISTRIBUTION 4. Let L be a context-free language. Show that L+ is recursively enumerable and suggest an enumeration procedure for it. 5. Show that if a language is not recursively enumerable, its complement cannot be recursive. © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC 6. Show that the family of recursively enumerable languages is closed under NOT FOR SALE OR DISTRIBUTIONunion. s NOT FOR SALE OR DISTRIBUTION 7. Is the family of recursively enumerable languages closed under intersection? 8. Show that the family of recursive languages is closed under union and intersection. © Jones9. &Show Bartlett that the Learning, families of recursively LLC enumerable and recursive© Jones languages & Bartlett are Learning, LLC NOT FOR closedSALE under OR reversal. DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 10. Is the family of recursive languages closed under concatenation? 11. Prove that the complement of a context-free language must be recursive. s

12. Let L1 be recursive and L2 recursively enumerable. Show that L2 − L1 is © Jones & Bartlett Learning,necessarily LLC recursively enumerable. © Jones & Bartlett Learning, LLC NOT FOR SALE OR13. DISTRIBUTIONSuppose that L is such that there existsNOT a Turing FOR machine SALE that OR enumerates DISTRIBUTION the elements of L in proper order. Show that this means that L is recursive. 14. If L is recursive, is it necessarily true that L+ is also recursive? s 15. Choose a particular encoding for Turing machines, and with it, ﬁnd one © Jones & Bartlett Learning, LLC element of the language© LJonesin Theorem & Bartlett 11.3. Learning, LLC ⊂ NOT FOR SALE OR DISTRIBUTION16. Let S1 be a countableNOT set, S2 FORa set thatSALE is not OR countable, DISTRIBUTION and S1 S2. Show that S2 must then contain an inﬁnite number of elements that are not in S1.

17. In Exercise 16, show that in fact S2 − S1 cannot be countable. 18. Why does the argument in Theorem 11.1 fail when S is ﬁnite? s © Jones19. &Show Bartlett that the Learning, set of all irrational LLC numbers is not countable.© Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

11.2 Unrestricted Grammars

© Jones & Bartlett Learning,To investigate LLC the connection between© recursively Jones & enumerable Bartlett languagesLearning, LLC and grammars, we return to the general deﬁnition of a grammar in Chapter NOT FOR SALE OR 1.DISTRIBUTION In Deﬁnition 1.1 the production ruleswereNOT FOR allowed SALE to take OR any DISTRIBUTION form,

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 285 — #9 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 11.2 Unrestricted Grammars 285

© Jonesbut & Bartlett various restrictions Learning, were LLC later made to get speciﬁc© grammar Jones types.& Bartlett If Learning, LLC we take the general form and impose no restrictions, we get unrestricted NOT FORgrammars. SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

Definition 11.3 © Jones & Bartlett Learning,A grammar LLC G =(V,T,S,P) iscalled© Jonesunrestricted & Bartlettif all theLearning, produc- LLC NOT FOR SALE OR DISTRIBUTIONtionsare of the form NOT FOR SALE OR DISTRIBUTION

u → v,

∗ where u isin (V ∪ T )+ and v isin (V ∪ T ) . © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION In an unrestricted grammar, essentially no conditions are imposed on the productions. Any number of variables and terminals can be on the left or right, and these can occur in any order. There is only one restriction: λ isnot allowed asthe left sideof a production. © Jones & BartlettAs we will Learning, see, unrestricted LLC grammars are much more© Jones powerful & Bartlett than Learning, LLC NOT FORrestrictedformslike SALE OR DISTRIBUTION the regular and context-free grammarsweNOT have FOR studied SALE OR DISTRIBUTION so far. In fact, unrestricted grammars correspond to the largest family of languages so we can hope to recognize by mechanical means; that is, unrestricted grammars generate exactly the family of recursively enumerable languages. We show this in two parts; the ﬁrst is quite straightforward, but © Jones & Bartlett Learning,the second LLC involves a lengthy construction.© Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION Theorem 11.6 Any language generated by an unrestricted grammar is recursively enumerable.

Proof: The grammar in eﬀect deﬁnesa procedure for enumerating all strings © Jones & Bartlett Learning, LLC in the language systematically.© Jones For & example, Bartlett we Learning, can list all w LLCin L such that NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION S ⇒ w,

that is, w is derived in one step. Since the set of the productions of the grammar is ﬁnite, there will be a ﬁnite number of such strings. Next, we © Joneslist & Bartlett all w in L Learning,that can be derivedLLC in two steps © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTIONS ⇒ x ⇒ w, NOT FOR SALE OR DISTRIBUTION

and so on. We can simulate these derivations on a Turing machine and, therefore, have an enumeration procedure for the language. Hence it is © Jones & Bartlett Learning,recursively LLC enumerable. © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 286 — #10 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 286 Chapter 11 A Hierarchy of Formal Languages and Automata

© Jones &This Bartlett part of Learning, the correspondence LLC between recursively© enumerable Jones & lan-Bartlett Learning, LLC guages and unrestricted grammars is not surprising. The grammar gener- NOT FORates SALE strings OR by a DISTRIBUTION well-deﬁned algorithmic process, so theNOT derivations FOR SALE can OR DISTRIBUTION be done on a Turing machine. To show the converse, we describe how any Turing machine can be mimicked by an unrestricted grammar. We are given a Turing machine M =(Q, Σ, Γ,δ,q0, ,F) and want to © Jones & Bartlett Learning,produce a grammar LLC G such that L (G)=©L (JonesM). The & ideaBartlett behind Learning, the con- LLC structionisrelatively simple,but itsimplementation becomesnotationally NOT FOR SALE ORcumbersome. DISTRIBUTION NOT FOR SALE OR DISTRIBUTION Since the computation of the Turing machine can be described by the sequence of instantaneous descriptions

∗ © Jones & Bartlett Learning, LLC © Jonesq0w &xq Bartlettf y, Learning, LLC (11.3) NOT FOR SALE OR DISTRIBUTIONwe will try to arrange it soNOT that theFOR corresponding SALE OR grammar DISTRIBUTION has the property that

∗ q0w ⇒ xqf y (11.4)

© Jonesif and& Bartlett only if (11.3) Learning, holds. Thisisnot LLC hard to do; what ismore© Jones diﬃcult & to Bartlett Learning, LLC NOT FORsee isSALE how to OR make DISTRIBUTION the connection between (11.4) and whatNOT we reallyFOR want, SALE OR DISTRIBUTION namely,

∗ S ⇒ w

© Jones & Bartlett Learning,for all w satisfying LLC (11.3). To achieve this,© weJones construct & Bartlett a grammar Learning, which, LLC NOT FOR SALE ORin DISTRIBUTION broad outline, hasthe following properties:NOT FOR SALE OR DISTRIBUTION

+ 1. S can derive q0w for all w ∈ Σ . 2. (11.4) is possible if and only if (11.3) holds.

© Jones & Bartlett Learning, LLC3. When a string xqf y ©with Jonesqf ∈ F &isgenerated, Bartlett Learning, the grammar LLC transforms NOT FOR SALE OR DISTRIBUTIONthisstringinto the originalNOT FORw. SALE OR DISTRIBUTION The complete sequence of derivations is then

∗ ∗ ∗ S ⇒ q0w ⇒ xqf y ⇒ w. (11.5) © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC The third step in the above derivation is the troublesome one. How can the NOT FORgrammar SALE remember OR DISTRIBUTIONw if it is modified during the second step?NOT We FOR solve SALE this OR DISTRIBUTION by encoding strings so that the coded version originally has two copies of w. The first is saved, while the second is used in the steps in (11.4). When a final configuration isentered, the grammar eraseseverything except the © Jones & Bartlett Learning,saved w. LLC © Jones & Bartlett Learning, LLC To produce two copiesof w and to handle the state symbol of M (which NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION eventually hasto be removed by the grammar), we introduce variables Vab

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 287 — #11 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 11.2 Unrestricted Grammars 287

∈ ∪{} ∈ ∈ © Jonesand & BartlettVaib for all Learning,a Σ LLC, b Γ, and all i such that q©i JonesQ. The & variable Bartlett Learning, LLC V encodesthe two symbols a and b, while V encodes a and b aswell as NOT FORab SALE OR DISTRIBUTION aib NOT FOR SALE OR DISTRIBUTION the state qi. The ﬁrst step in (11.5) can be achieved (in the encoded form) by

S → VS |SV| T, (11.6) → | © Jones & Bartlett Learning, LLC T TVaa V©a0a, Jones & Bartlett Learning,(11.7) LLC NOT FOR SALE OR DISTRIBUTIONfor all a ∈ Σ. These productions allow theNOT grammar FOR toSALE generate OR an DISTRIBUTION encoded version of any string q0w with an arbitrary number of leading and trailing blanks. For the second step, for each transition

VaicVpq → VadVpjq, (11.8) for all a, p ∈ Σ ∪{}, q ∈ Γ. For each © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTIONδ (qi,c)=(qj,d,L) NOT FOR SALE OR DISTRIBUTION of M, we include in G

VpqVaic → VpjqVad, (11.9) © Jones & Bartlett Learning,for all a, p LLC∈ Σ ∪{}, q ∈ Γ. © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTIONIf in the second step, M enters aNOT ﬁnal state, FOR the SALE grammar OR mustDISTRIBUTION then get rid of everything except w, which is saved in the ﬁrst indices of the V ’s. Therefore, for every qj ∈ F , we include productions

Vajb → a, (11.10) © Jones & Bartlett Learning, LLC for all a ∈ Σ ∪{}, b©∈ JonesΓ. This & creates Bartlett the Learning, ﬁrst terminal LLC in the string, NOT FOR SALE OR DISTRIBUTIONwhich then causes a rewritingNOT FOR in the SALE rest by OR DISTRIBUTION

cVab → ca, (11.11)

Vabc → ac, (11.12) © Jonesfor & allBartletta, c ∈ Σ Learning,∪{}, b ∈ Γ. LLC We need one more special© production Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION → λ. (11.13)

This last production takes care of the case when M movesoutsidethat part of the tape occupied by the input w. To make thingswork in thiscase,we © Jones & Bartlett Learning,must ﬁrst LLC use (11.6) and (11.7) to generate© Jones & Bartlett Learning, LLC

NOT FOR SALE OR DISTRIBUTION ...q0wNOT ... FOR, SALE OR DISTRIBUTION

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 288 — #12 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 288 Chapter 11 A Hierarchy of Formal Languages and Automata

© Jonesrepresenting & Bartlett all Learning, the tape region LLC used. The extraneous blanks© Jones are removed & Bartlett Learning, LLC at the end by (11.13). NOT FOR TheSALE following OR DISTRIBUTION example illustratesthiscomplicatedconstruction.NOT FOR Care- SALE OR DISTRIBUTION fully check each step in the example to see what the various productions do and why they are needed.

Q = {q0,q1} , Γ={a, b, } , Σ={a, b} , © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC F = {q1} , NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION and

δ (q0,a)=(q0,a,R) ,

δ (q0, )=(q1, ,L) . © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC ∗ NOT FORThismachine SALE OR accepts DISTRIBUTIONL (aa ). NOT FOR SALE OR DISTRIBUTION Consider now the computation

q0aa aq0a aaq0 aq1a, (11.14)

which acceptsthe string aa. To derive thisstringwith G, we ﬁrst use rules © Jones & Bartlett Learning,of the form (11.6)LLC and (11.7) to get the appropriate© Jones & starting Bartlett string, Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION S ⇒ SV ⇒ TV ⇒ TVaaV ⇒ Va0aVaaV.

The last sentential form is the starting point for the part of the derivation that mimicsthe computation of the Turing machine. It containsthe original © Jones & Bartlett Learning, LLCinput aa in the sequence© Jones of ﬁrst & indices Bartlett and Learning, the initial instantaneous LLC NOT FOR SALE OR DISTRIBUTIONdescription q0aa in theNOT remaining FOR indices. SALE Next, OR DISTRIBUTION we apply

Va0aVaa → VaaVa0a,

and

VaaV0 → Va1aV

NOT FOR SALE OR DISTRIBUTIONVa0aVaaV ⇒ VaaVa0aV ⇒ VaaVaaNOTV0 FOR⇒ Vaa SALEVa1aV OR. DISTRIBUTION

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 289 — #13 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 11.2 Unrestricted Grammars 289

© JonesThe & Bartlett sequence Learning, of ﬁrst indices LLC remains the same, always© rememberingJones & Bartlett the Learning, LLC initial input. The sequence of the other indices is NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 0aa,a0a,aa0,a1a,

which is equivalent to the sequence of instantaneous descriptions in (11.14). Finally, (11.10) to (11.13) are used in the last steps © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC

NOT FOR SALE OR DISTRIBUTIONVaaVa1aV ⇒ VaaaV ⇒ VNOTaaa ⇒ FORaa SALE⇒ aa. OR DISTRIBUTION

The construction described in (11.6) to (11.13) is the basis of the proof of the following result. © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION Theorem 11.7 For every recursively enumerable language L, there exists an unrestricted grammar G, such that L = L (G).

then

e (x) ⇒ e (y) , © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTIONwhere e (x) denotesthe encoding of a stringaccordingNOT FOR SALE to the givenOR DISTRIBUTION convention. By an induction on the number of steps, we can then show that

∗ e (q0w) ⇒ e (y)

NOT FOR SALE OR DISTRIBUTION NOT FOR SALE∗ OR DISTRIBUTION q0w y.

We also must show that we can generate every possible starting configuration and that w isproperly reconstructedif and only if M entersa final © Jonesconfiguration. & Bartlett Learning, The details, LLC which are not too difficult,© are Jones left as an& Bartlett exer- Learning, LLC NOT FORcise. SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

© Jones & Bartlett Learning,These LLC two theorems establish what© weJones set out & to Bartlett do. They Learning, show that LLC the family of languages associated with unrestricted grammars is identical NOT FOR SALE OR DISTRIBUTIONwith the family of recursively enumerableNOT languages. FOR SALE OR DISTRIBUTION

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 290 — #14 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 290 Chapter 11 A Hierarchy of Formal Languages and Automata

S → S1B,

S1 → aS1b, © Jones & Bartlett Learning, LLC bB → bbbB,© Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION aS1b → aa, B → λ

derive? s © Jones & Bartlett Learning, LLC2. What difficulties would© ariseJones if we & allowed Bartlett the emptyLearning, string as LLC the left side NOT FOR SALE OR DISTRIBUTIONof a production in anNOT unrestricted FOR grammar? SALE OR DISTRIBUTION 3. Consider a variation on grammars in which the starting point for any derivation can be a finite set of strings, rather than a single variable. Formalize this concept, then investigate how such grammars relate to the unrestricted grammars we have used here. s © Jones4. &In Bartlett Example 11.1,Learning, prove that LLC the constructed grammar cannot© Jones generate & anyBartlett Learning, LLC NOT FOR sentenceSALE withOR aDISTRIBUTIONb in it. NOT FOR SALE OR DISTRIBUTION 5. Give the details of the proof of Theorem 11.7. 6. Construct a Turing machine for L (01 (01)∗), then find an unrestricted grammar for it using the construction in Theorem 11.7. Give a derivation for 0101 © Jones & Bartlett Learning,using the LLC resulting grammar. © Jones & Bartlett Learning, LLC NOT FOR SALE OR 7.DISTRIBUTIONShow that for every unrestricted grammarNOT there FOR exists SALE an equivalent OR DISTRIBUTION unrestricted grammar, all of whose productions have the form

u → v,

with u, v ∈ (V ∪ T )+ and |u|≤|v|,or © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FORA SALE→ λ, OR DISTRIBUTION with A ∈ V . s 8. Show that the conclusion of Exercise 7 still holds if we add the further conditions |u|≤2 and |v|≤2. © Jones9. &Some Bartlett authors Learning, give a definition LLC of unrestricted grammars that© Jones is not quite & Bartlett the Learning, LLC NOT FOR sameSALE as ourOR Definition DISTRIBUTION 11.3. In this alternate definition,NOT the productions FOR SALE of OR DISTRIBUTION an unrestricted grammar are required to be of the form

x → y,

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 291 — #15 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 11.3Context-Sensitive Grammars and Languages 291

© Jones & Bartlettand Learning, LLC © Jones & Bartlett Learning, LLC ∗ NOT FOR SALE OR DISTRIBUTIONy ∈ (V ∪ T ) . NOT FOR SALE OR DISTRIBUTION The difference is that here the left side must have at least one variable. Show that this alternate definition is basically the same as the one we use, in the sense that for every grammar of one type, there is an equivalent grammar of the other type. © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 11.3 Context-Sensitive Grammars and Languages © Jones & Bartlett Learning, LLC Between the restricted,© context-free Jones & grammarsBartlett andLearning, the general, LLC unrestricted NOT FOR SALE OR DISTRIBUTIONgrammars, a great varietyNOT of FOR “somewhat SALE restricted” OR DISTRIBUTION grammars can be defined. Not all cases yield interesting results; among the ones that do, the context-sensitive grammars have received considerable attention. These grammars generate languages associated with a restricted class of Turing machines, linear bounded automata, which we introduced in Section 10.5. © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION Definition 11.4 A grammar G =(V,T,S,P) issaidto be context-sensitive if all productionsare of the form → © Jones & Bartlett Learning, LLC x ©y, Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTIONwhere x, y ∈ (V ∪ T )+ and NOT FOR SALE OR DISTRIBUTION |x|≤|y| . (11.15)

© Jones & Bartlett Learning, LLC This deﬁnition shows© Jones clearly & one Bartlett aspect ofLearning, this type ofLLC grammar; it NOT FOR SALE OR DISTRIBUTIONis noncontracting, inNOT the senseFOR thatSALE the OR length DISTRIBUTION of successive sentential forms can never decrease. It is less obvious why such grammars should be called context-sensitive, but it can be shown (see, for example, Salomaa 1973) that all such grammars can be rewritten in a normal form in which all productionsare of the form © Jones & Bartlett Learning, LLCxAy → xvy. © Jones & Bartlett Learning, LLC NOT FORThisisequivalent SALE OR DISTRIBUTION to saying that the production NOT FOR SALE OR DISTRIBUTION A → v can be applied only in the situation where A occursin a context of the string © Jones & Bartlett Learning,x on the leftLLC and the string y on the right.© Jones While & we Bartlett use the terminologyLearning, LLC arising from this particular interpretation, the form itself is of little interest NOT FOR SALE OR DISTRIBUTIONto ushere, and we will rely entirely onNOT Deﬁnition FOR 11.4. SALE OR DISTRIBUTION

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 292 — #16 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 292 Chapter 11 A Hierarchy of Formal Languages and Automata

© JonesContext-Sensitive & Bartlett Learning, Languages LLC and Linear Bounded© Jones & Bartlett Learning, LLC NOT FORAutomata SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION As the terminology suggests, context-sensitive grammars are associated with a language family with the same name.

© Jones & Bartlett Learning,Definition LLC11.5 © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTIONA language L is said to be context-sensitiveNOT FOR if there SALE exists OR a context-DISTRIBUTION sensitive grammar G, such that L = L (G)orL = L (G) ∪{λ}.

In thisdefinition, we reintroduce the empty string. Definition 11.4 © Jones & Bartlett Learning, LLCimpliesthat x → λ is not© Jones allowed, & so Bartlett that a context-sensitiveLearning, LLC grammar NOT FOR SALE OR DISTRIBUTIONcan never generate a languageNOT FOR containing SALE the OR empty DISTRIBUTION string. Yet, every context-free language without λ can be generated by a special case of a context-sensitive grammar, say by one in Chomsky or Greibach normal form, both of which satisfy the conditions of Definition 11.4. By including the empty string in the definition of a context-sensitive language (but not © Jonesin the& Bartlett grammar), Learning, we can claim LLC that the family of context-free© Jones languagesis & Bartlett Learning, LLC NOT FORa subset SALE of the OR family DISTRIBUTION of context-sensitive languages. NOT FOR SALE OR DISTRIBUTION

Example 11.2 The language L = {anbncn : n ≥ 1} is a context-sensitive language. We show this by exhibiting a context-sensitive grammar for the language. One © Jones & Bartlett Learning,such grammar LLC is © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION S → abc|aAbc, Ab → bA, Ac → Bbcc, © Jones & Bartlett Learning, LLC © JonesbB → &Bb, Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOTaB FOR→ aa SALE|aaA. OR DISTRIBUTION We can seehow thisworksby looking at a derivation of a3b3c3.

S ⇒ aAbc ⇒ abAc ⇒ abBbcc © Jones & Bartlett Learning,⇒ aBbbcc LLC⇒ aaAbbcc ⇒ aabAbcc © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION⇒ aabbAcc ⇒ aabbBbccc NOT FOR SALE OR DISTRIBUTION ⇒ aabBbbccc ⇒ aaBbbbccc ⇒ aaabbbccc.

© Jones & Bartlett Learning,The solution LLC eﬀectively uses the variables© AJonesand B &as Bartlett messengers. Learning, An A LLC iscreated on the left, travelsto the right to the ﬁrst c, where it creates NOT FOR SALE OR anotherDISTRIBUTIONb and c. It then sends the messengerNOT FORB back SALE to the OR left DISTRIBUTION in order

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 293 — #17 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 11.3Context-Sensitive Grammars and Languages 293

© Jonesto & createBartlett the correspondingLearning, LLCa. The process is very similar© Jones to the way& Bartlett one Learning, LLC might program a Turing machine to accept the language L. NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

Since the language in the previousexample isnot context-free, we see that the family of context-free languages is a proper subset of the family © Jones & Bartlett Learning,of context-sensitive LLC languages. Example© Jones 11.2 also & shows Bartlett that itLearning, is not an LLC easy matter to ﬁnd a context-sensitive grammar even for relatively simple NOT FOR SALE OR DISTRIBUTIONexamples. Often the solution is mostNOT easily FOR obtained SALE by startingOR DISTRIBUTION with a Turing machine program, then ﬁnding an equivalent grammar for it. A few examples will show that, whenever the language is context-sensitive, the corresponding Turing machine has predictable space requirements; in © Jones & Bartlett Learning, LLC particular, it can be viewed© Jones asa linear & Bartlett bounded Learning, automaton. LLC NOT FOR SALE ORTheorem DISTRIBUTION 11.8 For every context-sensitiveNOT languageFOR SALEL not OR including DISTRIBUTIONλ, there exists some linear bounded automaton M such that L = L (M).

Proof: If L is context-sensitive, then there exists a context-sensitive grammar for L−{λ}. We show that derivations in this grammar can be simulated © Jonesby & aBartlett linear bounded Learning, automaton. LLC The linear bounded automaton© Jones will& Bartlett have Learning, LLC NOT FORtwo SALE tracks, OR one DISTRIBUTION containing the input string w, the otherNOT containing FOR the SALE sen- OR DISTRIBUTION tential formsderived using G. A key point of thisargument isthat no possible sentential form can have length greater than |w|. Another point to notice is that a linear bounded automaton is, by deﬁnition, nondeterministic. This is necessary in the argument, since we can claim that the correct © Jones & Bartlett Learning,production LLC can always be guessed and© that Jones no unproductive & Bartlett alternativesLearning, LLC NOT FOR SALE OR DISTRIBUTIONhave to be pursued. Therefore, the computationNOT FOR described SALE in OR Theorem DISTRIBUTION 11.6 can be carried out without using space except that originally occupied by w; that is, it can be done by a linear bounded automaton.

© Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE ORTheorem DISTRIBUTION 11.9 If a language L isacceptedNOT byFOR somelinear SALE boundedOR DISTRIBUTION automaton M, then there exists a context-sensitive grammar that generates L.

Proof: The construction here is similar to that in Theorem 11.7. All productionsgenerated in Theorem 11.7 are noncontracting except (11.13), © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC → NOT FOR SALE OR DISTRIBUTION λ. NOT FOR SALE OR DISTRIBUTION But thisproduction can be omitted. It isnecessaryonly when the Turing machine movesoutsidethe boundsof the original input, which isnot the case here. The grammar obtained by the construction without this unnecessary © Jones & Bartlett Learning,production LLC isnoncontracting, completing© theJones argument. & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 294 — #18 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 294 Chapter 11 A Hierarchy of Formal Languages and Automata

© JonesRelation & Bartlett Between Learning, Recursive LLC and Context-Sensitive© Jones & Bartlett Learning, LLC NOT FORLanguages SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION Theorem 11.9 tells us that every context-sensitive language is accepted by some Turing machine and is therefore recursively enumerable. Theorem 11.10 follows easily from this. © JonesTheorem & Bartlett 11.10 Learning,Every context-sensitive LLC language L isrecursive.© Jones & Bartlett Learning, LLC NOT FOR SALE ORProof: DISTRIBUTIONConsider the context-sensitive languageNOT FORL with SALE an associated OR DISTRIBUTION context-sensitive grammar G, and look at a derivation of w

S ⇒ x1 ⇒ x2 ⇒···⇒xn ⇒ w. © Jones & Bartlett Learning, LLCWe can assume without© any Jones loss of generality& Bartlett that Learning, all sentential LLC forms in a NOT FOR SALE OR DISTRIBUTIONsingle derivation are diﬀerent;NOT FOR that is, SALExi = xORj for DISTRIBUTION all i =j. The crux of our argument isthat the number of stepsinany derivation isa bounded function of |w|. We know that

|xj|≤|xj+1| , © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC because G isnoncontracting. The only thing we need to add isthat there NOT FORexist SALE some m OR, depending DISTRIBUTION only on G and w, such that NOT FOR SALE OR DISTRIBUTION

|xj| < |xj+m| ,

for all j, with m = m (|w|) a bounded function of |V ∪ T | and |w|. This © Jones & Bartlett Learning,follows because LLC the finiteness of |V ∪ T | impliesthat© Jones & there Bartlett are only Learning, a finite LLC NOT FOR SALE ORnumber DISTRIBUTION of strings of a given length. Therefore,NOT theFOR length SALE of a OR derivation DISTRIBUTION of w ∈ L isat most |w| m (|w|). Thisobservation givesusimmediately a membershipalgorithm for L. We check all derivationsof length up to |w| m (|w|). Since the set of pro- ductionsof G isfinite, there are only a finite number of these. If any of © Jones & Bartlett Learning, LLCthem give w, then w ∈ L©, otherwiseJones & it Bartlett is not. Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

Theorem 11.11 There exists a recursive language that is not context-sensitive. Proof: Consider the set of all context-sensitive grammars on T = {a, b}. © JonesWe & can Bartlett use a convention Learning, in whichLLC each grammar has a variable© Jones set of& theBartlett Learning, LLC NOT FORform SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

V = {V0,V1,V2, ...} .

Every context-sensitive grammar is completely speciﬁed by its productions; © Jones & Bartlett Learning,we can think LLC of them as written as a single© Jones string & Bartlett Learning, LLC

NOT FOR SALE OR DISTRIBUTION x1 → y1; x2 → y2; ...NOT; xm FOR→ ym .SALE OR DISTRIBUTION

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 295 — #19 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 11.3Context-Sensitive Grammars and Languages 295

© JonesTo & thisBartlett string Learning, we now apply LLC the homomorphism © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTIONh (a) = 010, NOT FOR SALE OR DISTRIBUTION h (b)=0120, h (→)=0130, h (; ) = 0140, © Jones & Bartlett Learning, LLC ©i +5Jones & Bartlett Learning, LLC h (Vi)=01 0. NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION Thus, any context-sensitive grammar can be represented uniquely by a ∗ string from L (011∗0) . Furthermore, the representation is invertible in the sense that, given any such string, there is at most one context-sensitive grammar corresponding to it. © Jones & Bartlett Learning, LLC Let usintroduce a proper© Jones ordering & Bartlett on {0, 1} Learning,+, so we can writeLLC strings in NOT FOR SALE OR DISTRIBUTIONthe order w1, w2, etc.NOT A given FOR string SALEwj may OR not DISTRIBUTION deﬁne a context-sensitive grammar; if it does, call the grammar Gj. Next, we deﬁne a language L by

L = {wi : wi defines a context-sensitive grammar Gi and wi ∈/ L (Gi)} . © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FORL isSALE well defined OR DISTRIBUTION and is in fact recursive. To see this, we constructNOT FOR a member- SALE OR DISTRIBUTION ship algorithm. Given wi, we check it to see if it defines a context-sensitive grammar Gi. If not, then wi ∈/ L. If the string does define a grammar, then L (Gi) is recursive, and we can use the membership algorithm of Theorem 11.10 to find out if wi ∈ L (Gi). If it isnot, then wi belongsto L. © Jones & Bartlett Learning,But L LLCis not context-sensitive. If© it Jones were, there & Bartlett would exist Learning, some wj LLC NOT FOR SALE OR DISTRIBUTIONsuch that L = L (Gj). We can then askNOT if w FORj isin SALEL (Gj). OR If weDISTRIBUTION assume that wj ∈ L (Gj), then by definition wj isnot in L. But L = L (Gj), so we have a contradiction. Conversely, if we assume that wj ∈/ L (Gj), then by definition wj ∈ L and we have another contradiction. We must therefore conclude that L is not context-sensitive. © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION The result in Theorem 11.11 indicates that linear bounded automata are indeed less powerful than Turing machines, since they accept only a proper subset of the recursive languages. It follows from the same re- © Jonessult & Bartlett that linear Learning, bounded automata LLC are more powerful than© Jones pushdown & Bartlett au- Learning, LLC tomata. Context-free languages, being generated by context-free grammars, NOT FORare SALE a subset OR of theDISTRIBUTION context-sensitive languages. As variousNOT examples FOR SALE show, OR DISTRIBUTION they are a proper subset. Because of the essential equivalence of linear bounded automata and context-sensitive languages on one hand, and pushdown automata and context-free languageson the other, we seethat any © Jones & Bartlett Learning,language acceptedLLC by a pushdown automaton© Jones is also & Bartlett accepted byLearning, some lin- LLC ear bounded automaton, but that there are languagesaccepted by some NOT FOR SALE OR DISTRIBUTIONlinear bounded automata for which thereNOT are FOR no pushdown SALE automata.OR DISTRIBUTION

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 296 — #20 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 296 Chapter 11 A Hierarchy of Formal Languages and Automata

© JonesE XERCISES& Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 1. Find context-sensitive grammars for the following languages. (a) L = an+1bncn−1 : n ≥ 1 . n n 2n © Jones & Bartlett Learning,(b) L = LLCa b a : n ≥ 1 . © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION(c) L = {anbmcndm : n ≥ 1,m≥ 1}. sNOT FOR SALE OR DISTRIBUTION (d) L = ww : w ∈{a, b}+ . (e) L = {anbncndn : n ≥ 1, }.

NOT FOR SALE OR DISTRIBUTION (a) L = {w : na (w)=NOTnb (w )=FORnc ( wSALE)}. OR DISTRIBUTION

(b) L = {w : na (w)=nb (w)

3. Show that the family of context-sensitive languages is closed under union. © Jones4. &Show Bartlett that the Learning, family of context-sensitive LLC languages is closed© Jones under reversal. & Bartlett Learning, LLC NOT FOR SALEs OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 5. For m in Theorem 11.10, give explicit bounds for m as a function of |w| and |V ∪ T |.

6. Without explicitly constructing it, show that there exists a context-sensitive grammar for the language L = wuwR : w, u ∈{a, b}+ , |w|≥|u| . s © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

11.4 The Chomsky Hierarchy

© Jones & Bartlett Learning, LLCWe have now encountered© aJones number & of Bartlett language Learning, families, among LLC them the recursively enumerable languages (L ), the context-sensitive languages NOT FOR SALE OR DISTRIBUTION NOT FOR RESALE OR DISTRIBUTION (LCS), the context-free languages(L CF ), and the regular languages( LREG). One way of exhibiting the relationship between these families is by the Chomsky hierarchy. Noam Chomsky, a founder of formal language the- ory, provided an initial classification into four language types, type 0 to © Jonestype & 3.Bartlett Thisoriginal Learning, terminology LLC haspersistedand one findsfrequent© Jones ref- & Bartlett Learning, LLC erencesto it, but the numeric typesare actually different namesfor the NOT FORlanguage SALE families OR DISTRIBUTION we have studied. Type 0 languages areNOT those FOR generated SALE OR DISTRIBUTION by unrestricted grammars, that is, the recursively enumerable languages. Type 1 consists of the context-sensitive languages, type 2 consists of the context-free languages, and type 3 consists of the regular languages. As we © Jones & Bartlett Learning,have seen, each LLC language family of type i©is Jones a proper & subset Bartlett of the Learning, family of LLC type i−1. A diagram (Figure 11.3) exhibitsthe relationshipclearly. Figure NOT FOR SALE OR 11.3DISTRIBUTION shows the original Chomsky hierarchy.NOT We FOR have alsoSALE met OR several DISTRIBUTION other

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 297 — #21 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 11.4 The Chomsky Hierarchy 297

L Figure 11.3© Jones & Bartlett Learning,RE LLC © Jones & Bartlett Learning, LLC L NOT FOR SALE OR DISTRIBUTIONCS NOT FOR SALE OR DISTRIBUTION LCF

LREG

NOT FOR SALE OR DISTRIBUTION NOTLCS FOR SALE OR DISTRIBUTION

LCF

LDCF

LREG © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

language familiesthat can be ﬁtted into thispicture. Including the fami- liesof deterministiccontext-free languages(L DCF ) and recursive languages (LREC ), we arrive at the extended hierarchy shown in Figure 11.4. © Jones & Bartlett Learning, LLC Other language familiescan© Jones be & deﬁnedBartlett and Learning, their place inLLC Figure 11.4 NOT FOR SALE OR DISTRIBUTIONstudied, although theirNOT relationships FOR SALE do not OR always DISTRIBUTION have the neatly nested structure of Figures 11.3 and 11.4. In some instances, the relationships are not completely understood.

L = {w : na (w)=nb (w)}

and shown that it is deterministic, but not linear. On the other hand, the language © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION L = {anbn}∪NOTanb 2FORn SALE OR DISTRIBUTION

✐ ✐ ✐ ✐

“15529˙CH11˙Linz” — 2011/1/12 — 10:03 — page 298 — #22 ✐ ✐ © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION 298 Chapter 11 A Hierarchy of Formal Languages and Automata

L L L © Jones & Bartlett Learning,LIN LLC REG DCF © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION

© Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION islinear, but not deterministic. Thisindicatesthat the relationshipbetween regular, linear, deterministic context-free, and nondeterministic context-free languagesisasshown in Figure 11.5. © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR ThereSALE is OR still DISTRIBUTION an unresolved issue. We introducedNOT the concept FOR SALE of a OR DISTRIBUTION deterministic linear bounded automaton in Exercise 8, Section 10.5. We can now ask the question we asked in connection with other automata: What role does nondeterminism play here? Unfortunately, there is no easy answer. At this time, it is not known whether the family of languages © Jones & Bartlett Learning,accepted by LLC deterministic linear bounded© automataJones & is Bartlett a proper Learning, subset of LLC NOT FOR SALE ORthe DISTRIBUTION context-sensitive languages. NOT FOR SALE OR DISTRIBUTION To summarize, we have explored the relationships between several language families and their associated automata. In doing so, we established a hierarchy of languages and classiﬁed automata by their power as language accepters. Turing machines are more powerful than linear bounded © Jones & Bartlett Learning, LLCautomata. These in turn© are Jones more powerful& Bartlett than Learning, pushdown automata. LLC At NOT FOR SALE OR DISTRIBUTIONthe bottom of the hierarchyNOT are FOR ﬁnite SALE accepters, OR with DISTRIBUTION which we began our study.

EXERCISES © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR1. CollectSALE examples OR DISTRIBUTION given in this book that demonstrate thatNOT all FOR the subset SALE OR DISTRIBUTION relations depicted in Figure 11.4 are indeed proper ones. 2. Find two examples (excluding the one in Example 11.3) of languages that are linear but not deterministic context-free. © Jones & Bartlett Learning,3. Find two LLC examples (excluding the one in© Example Jones 11.3) & Bartlett of languages Learning, that are LLC NOT FOR SALE OR DISTRIBUTIONdeterministic context-free but not linear.NOT FOR SALE OR DISTRIBUTION

✐ ✐