Unit VI
COMPUTATIONAL COMPLEXITY
Jagadish K Kamble (M.Tech, IIT Kharagpur) Assistant Professor, Dept. of Information Technology, Pune Institute of Computer Technology, Pune.
1 Outline • Decidability
• Reducibility
• Time Complexity
2 Undecidability and Complexity
◼ Undecidability
◼ Recursive and recursively enumerable languages.
◼ Universal Turing Machines.
◼ Limitations on our ability to compute; undecidable problems. ◼ Computational Complexity
◼ Decidable problems for which no sufficient algorithms are known.
◼ Polynomial time computability.
◼ The notion of NP-completeness and problem reductions.
◼ Examples of hard problems.
◼ Let’s start with a big-picture overview of these topics Recall that: A language L is Turing-Acceptable if there is a Turing machine M that accepts
Also known as: Turing-Recognizable or Recursively-enumerable languages For any string w :
w L M halts in an accept state
w L halts in a non-accept state or loops forever Definition: A language L is decidable if there is a Turing machine (decider) M which accepts and halts on every input string
Also known as recursive languages For any string w : w L M halts in an accept state
w L halts in a non-accept state
Every decidable language is Turing-Acceptable For a decidable language :
Decision Decider for L On Halt: qaccept Accept Input string Reject qreject
For each input string, the computation halts in the accept or reject state For a Turing-Acceptable language :
Turing Machine for L
qaccept Input string
qreject
It is possible that for some input string the machine enters an infinite loop Problem: Is number x prime?
Corresponding language:
PRIMES = {1, 2, 3, 5, 7, }
We will show it is decidable Decider for PRIMES : On input number x : Divide x with all possible numbers between 2 and x
If any of them divides Then reject Else accept the decider for the language solves the corresponding problem
Decider for PRIMES
qaccept YES (Accept) Input number x is prime? (Input string) NO qreject (Reject) Theorem: If a language L is decidable, then its complement L is decidable too
Proof: Build a Turing machine M that accepts and halts on every input string
( is decider for ) Transform accept state to reject and vice-versa
M M
qaccept qreject
qa q reject qaccept
qr Turing Machine M On each input string w do:
1. Let M be the decider for L
2. Run with input string If M accepts then reject If M rejects then accept
Accepts L and halts on every input string
END OF PROOF Decidability vs. Undecidability
◼ There are two types of TMs (based on halting): (Recursive) TMs that always halt, no matter accepting or non- accepting DECIDABLE PROBLEMS (Recursively enumerable) TMs that are guaranteed to halt only on acceptance. If non-accepting, it may or may not halt (i.e., could loop forever).
◼ Undecidability:
◼ Undecidable problems are those that are not recursive
16 Recursive, RE, Undecidable languages
No TMs exist TMs that always halt LBA Non-RE Languages TMs that may or (all other languages for which may not halt no TMs can be built)
Regular Context-
(DFA) free sensitive
(PDA) Context
Recursively Enumerable (RE) Recursive
“Undecidable” problems “Decidable” problems 17 Recursive Languages & Recursively Enumerable (RE) languages
◼ Any TM for a Recursive language is going to look like this: “accept” w M “reject”
◼ Any TM for a Recursively Enumerable (RE) language is going to look like this: “accept” w M “?” 18 ◼ Languages recognized by a TM are called recognizable/acceptable.
◼ Languages decided by a TM are called decidable.
19 Closure Properties of: - the Recursive language class, and - the Recursively Enumerable language class
20 Recursive Languages are closed under complementation
◼ If L is Recursive, L is also Recursive
M “accept” “accept” w w M “reject” “reject”
21 Are Recursively Enumerable Languages closed under complementation? (NO)
◼ If L is RE, L need not be RE
M “accept” “accept” ? w w M “reject” ?
22 Recursive Langs are closed under Union
◼ Let Mu = TM for L1 U L2 ◼ M construction: Mu u accept M 1. Make 2-tapes and 1 reject copy input w on both OR tapes w accept 2. Simulate M1 on tape 1 M 2 reject 3. Simulate M2 on tape 2
4. If either M1 or M2 accepts, then Mu accepts
5. Otherwise, Mu rejects.
23 Recursive Langs are closed under Intersection
◼ Let Mn = TM for L1 L2 ◼ M construction: Mn n accept M 1. Make 2-tapes and 1 reject copy input w on both ANDAND tapes w accept 2. Simulate M1 on tape 1 M 2 reject 3. Simulate M2 on tape 2
4. If M1 AND M2 accepts, then Mn accepts
5. Otherwise, Mn rejects.
24 Other Closure Property Results
◼ Recursive languages are also closed under:
◼ Concatenation
◼ Kleene closure (star operator)
◼ Homomorphism, and inverse homomorphism
◼ RE languages are closed under:
◼ Union, intersection, concatenation, Kleene closure
◼ RE languages are not closed under:
◼ complementation
25 Decidable and Undecidable Problems
Problem
Solvable Unsolvable
Decidable Undecidable (Algorithm) (Procedure) Solution is definite Sometime yes or sometime no Example of Decidable Problems
◼ Does finite Automata Accept Regular Language?
◼ L1 and L2 are two regular languages, are they closed under following property:
◼ Union
◼ Concatenation
◼ Kleene Closure
◼ If L1 and L2 is CFL then L1 U L2 is CFL?
27 Example of Undecidable Problems
◼ For given context free Grammer G, is L(G) is ambiguous?
◼ For two given CFL L1 and L2 whether L1 ∩ L2 is CFL or not?
◼ A Computational problem is said to be unsolvable if no algorithm to solve it.
◼ A problem is said to be undecidable if it is a decision problem and no algorithm can decide the solution.
28 For a decidable language :
Decision Decider for L On Halt: qaccept Accept Input string Reject qreject
For each input string, the computation halts in the accept or reject state For a Turing-Acceptable language :
Turing Machine for L
qaccept Input string
qreject
It is possible that for some input string the machine enters an infinite loop NLP attendance (present numbers) 26/09/2018 • 4006,4037 Recall
• Decidability Proofs of Decidability
How can you prove a language is decidable? What Decidable Means
A language L is decidable if there exists a TM M such that for all strings w:
– If w L, M enters qAccept.
– If w L, M enters qReject.
To prove a language is decidable, we can show how to construct a TM that decides it.
For a correct proof, need a convincing argument that the TM always eventually accepts or rejects any input. Recall
• Undecidability Proofs of Undecidability
How can you prove a language is undecidable? Proofs of Undecidability
To prove a language is undecidable, need to show there is no Turing Machine that can decide the language.
This is hard: requires reasoning about all possible TMs. Recall
• Recursive Enumerable and Recursive Language • Decidable Language Decidable Problems of Regular Languages Membership Question
Question: Given regular language L and string w how can we check if w L ?
Answer: Take the DFA that accepts L and check if w is accepted DFA w w L
DFA
w L Question: Given regular language L how can we check if is empty: ( L = ) ?
Answer: Take the DFA that accepts L
Check if there is any path from the initial state to an accepting state DFA
L
DFA
L = Question: Given regular language L how can we check if is finite?
Answer: Take the DFA that accepts L
Check if there is a walk with cycle from the initial state to a final state DFA
L is infinite
DFA
is finite Problem: Does DFA M accept the empty language L ( M ) = ?
Corresponding Language: (Decidable)
EMPTYDFA = { M : M is a DFA that accepts empty language }
Description of DFA M as a string (For example, we can represent as a binary string, as we did for Turing machines) Decider for EMPTYDFA :
On input M : Determine whether there is a path from the initial state to any accepting state DFA M DFA
L(M) L(M) =
Decision: Reject M Accept M Problem: Does DFA M accept a finite language?
Corresponding Language:X (Decidable)
FINITEDFA = { M : M is a DFA that accepts a finite language} Decider for FINITEDFA : On input M : Check if there is a walk with a cycle from the initial state to an accepting state
DFA M DFA
infinite finite
Decision: Reject M Accept M (NO) (YES) DFA Acceptance Problem: Given a DFA M and an input w, does M accept w?
Corresponding Language:X (Decidable)
ADFA = { ⟨M, w⟩ : M is a DFA that accepts input string w }
ADFA is a decidable language. Decider for ADFA : On input string :
Run DFA M on input string w
If accepts Then accept M , w (and halt) Else reject (and halt) ◼ Proof.
◼ We construct a TM M that decides ADFA. ◼ On input ⟨M,w⟩ where M is a DFA and w an input
◼ 1. Simulate M on input w
◼ 2. If the simulation ends in an accept state, accept. If it ends in a non-accepting state, reject.
◼ Input would be the 5 tuple description of the DFA – the TM would keep track of the current state and process the input tape based on transition function and finally accept if the string’s final state is accept.
52 A Language about TMs/FAs/PDAs & acceptance
ADFA = { ⟨M, w⟩ : M is a DFA that accepts input string w }
Let ADFA be the language of all strings
1. M is a TM/FA/PDA (coded in binary) with input alphabet also binary
2. w is a binary string
3. M accepts input w.
53 ◼ ACFG = {⟨G,w⟩ | G is a CFG deriving string w}
◼ Theorem. ACFG is decidable. ◼ Proof. T = “On input
◼ 1. Convert G to Chomsky Normal Form.
◼ 2. Test all possible derivations of length 2n-1, where n = |w|.
◼ 3. If any generate w, accept, otherwise reject.”
54 Decidable Languages A language A is decidable, if there is a Turing machine M (decider) that accepts the language A and halts on every input string
Decision Turing Machine M On Halt: YES Accept Input Decider for string NO Reject is Turing-Acceptable
Turing machine that accepts A TM :
1. Run M on input w
M,w 2. If accepts then accept A computational problem is decidable if the corresponding language is decidable
We also say that the problem is solvable Undecidable Languages undecidable language = not decidable language
There is no decider: there is no Turing Machine which accepts the language and makes a decision (halts) for every input string
(machine may make decision for some input strings) For an undecidable language, the corresponding problem is undecidable (unsolvable):
there is no Turing Machine (Algorithm) that gives an answer (yes or no) for every input instance
(answer may be given for some input instances) Which of these are Undecidable?
• Does TM M accept any strings? Undecidable • Does TM M accept all strings? Undecidable • Does TM M accept “Hello”? Undecidable
• Does TM M1 accept more strings than TM M2?
• Does TM M take more than 1000 steps to Undecidable process input w? Decidable
• Does TM M1 take more steps than TM M2 to process input w? Undecidable We have shown before that there are undecidable languages:
L Turing-Acceptable L
Decidable
is Turing-Acceptable and undecidable
Halting Problem Input: a program P in some programming language and input I for P Output: Does P go into an infinite loop when given I? or it finish. true if P terminates; false if P runs forever. We will prove that two particular problems are unsolvable/undecidable:
Membership problem
ATM= {
Halting problem
HALTTM= {
Non-RE Languages
x
Regular Context-
(DFA) free sensitive
(PDA) Context
Recursively Enumerable (RE) Recursive
65 What is the Halting Problem?
Definition of the “halting problem”:
◼ Does a givenTuring Machine M halt on a given input w?
Input w Machine M
66 A Turing Machine simulator The Universal Turing Machine
◼ Given: TM M & its input w ◼ Aim: Build another TM called “H”, that will output:
◼ “accept” if M accepts w, and
◼ “reject” otherwise
◼ An algorithm for H: Implies: H is in RE ◼ Simulate M on w
accept, if M halts on w ◼ H(
Question: If M does not halt on w, what will happen to H? 67 A Claim
◼ Claim: No H that is always guaranteed to halt, can exist!
◼ Proof: (Alan Turing, 1936)
◼ By contradiction, let us assume H exists
“accept”
68 Therefore, if H exists ➔ D also should exist. But can such a D exist? (if not, then H also cannot exist) HP Proof (step 1)
◼ Let us construct a new TM D using H as a subroutine:
◼ On input
1. Run H on input
2. Output the opposite of what H outputs;
D
“accept” “accept”
69 HP Proof (step 2)
◼ The notion of inputing “
◼ A program can be input to itself (e.g., a compiler is a program that takes any program as input) accept, if M does not accept
accept, if D does not accept
A contradiction!!! ==> Neither D nor H can exist. 70 Of Paradoxes & Strange Loops
E.g., Barber’s paradox, Achilles & the Tortoise (Zeno’s paradox) MC Escher’s paintings
A fun book for further reading: “Godel, Escher, Bach: An Eternal Golden Braid” by Douglas Hofstadter (Pulitzer winner, 1980) 71 The Diagonalization Language
Example of a language that is not recursive enumerable
(i.e, no TMs exist)
72 The Diagonalization language
The Halting Problem
Non-RE Languages
x
x
Regular Context-
(DFA) free sensitive
(PDA) Context
Recursively Enumerable (RE) Recursive
73 Why should there be languages that do not have TMs?
We thought TMs can solve everything!!
74 Non-RE languages
How come there are languages here? (e.g., diagonalization language)
Non-RE Languages
Regular Context-
(DFA) free sensitive
(PDA) Context
Recursively Enumerable (RE) Recursive
75 One Explanation
There are more languages than TMs
◼ By pigeon hole principle:
◼ ==> some languages cannot have TMs
◼ But how do we show this?
◼ Need a way to “count & compare” two infinite sets (languages and TMs)
76 How to count elements in a set?
Let A be a set:
◼ If A is finite ==> counting is trivial
◼ If A is infinite ==> how do we count?
◼ And, how do we compare two infinite sets by their size?
77 THE DIAGONALIZATION METHOD
◼ mathematician Georg Cantor in 1873
◼ problem of measuring the sizes of infinite sets.
◼ For finite sets
◼ use the counting method for Infinite????
◼ finite sets have the same size if the elements of one set can be paired with the elements of the other set.
◼ extend this idea to infinite sets
78 Cantor’s definition of set “size” for infinite sets (1873 A.D.)
Let N = {1,2,3,…} (all natural numbers) Let E = {2,4,6,…} (all even numbers)
Q) Which is bigger?
◼ A) Both sets are of the same size
◼ “Countably infinite”
◼ Proof: Show by one-to-one, onto set correspondence from N ==> E n f(n) 1 2 i.e, for every element in N, 2 4 there is a unique element in E, 3 6 and vice versa. . . . . 79 . . Really, really big sets! (even bigger than countably infinite sets) Uncountable sets
Example: ◼ Let R be the set of all real numbers ◼ Claim: R is uncountable
n f(n) 1 3 . 1 4 1 5 9 … Build x s.t. x cannot possibly 2 5 . 5 5 5 5 5 … occur in the table 3 0 . 1 2 3 4 5 … 4 0 . 5 1 4 3 0 … E.g. x = 0 . 2 6 4 4 … . . .
80 Therefore, some languages cannot have TMs…
◼ The set of all TMs is countably infinite
◼ The set of all Languages is uncountable
◼ ==> There should be some languages without TMs
81 Reducibility
86 ◼ Method for proving that problems are computationally unsolvable. It is called reducibility.
◼ A reduction is a way of converting one problem to another problem in such a way that a solution to the second problem can be used to solve the first problem.
87 ◼ Reducibility's come up often in everyday life.
◼ E.g. You want to find your way around a new city.
◼ would be easy if you had a map.
◼ Thus, you can reduce the problem of finding your way around the city to the problem of obtaining a map of the city.
◼ Reducibility always involves two problems, which we call A and B.
◼ If A reduces to B, we can use a solution to B to solve A.
◼ So in our example, A is the problem of finding your way around the city and B is the problem of obtaining a map.
◼ Note that reducibility says nothing about solving A or B alone, but only about the solvability of A in the presence of a solution to B. 88 ◼ The problem of traveling from Pune to Delhi reduces to the problem of buying a plane ticket between the two cities
◼ the problem of measuring the area of a rectangle reduces to the problem of measuring its length and width.
89 ◼ Reducibility plays an important role in classifying problems by decidability, and later in complexity theory as well.
◼ When A is reducible to B, solving A cannot be harder than solving B because a solution to B gives a solution to A.
◼ In terms of computability theory, if A is reducible to B and B is decidable, A also is decidable. Equivalently, if A is undecidable and reducible to B, B is undecidable. 90 Reducability
◼ To prove: Problem P1 is undecidable
◼ Given/known: Problem P2 is undecidable ◼ Reduction idea:
1. “Reduce” P2 to P1:
◼ Convert P2’s input instance to P1’s input instance s.t. i) P2 decides only if P1 decides
2. Therefore, P2 is decidable 3. A contradiction
4. Therefore, P1 has to be undecidable
91 Post Correspondence Problem, or PCP.
◼
93 ◼
94 ◼
95 The Post Correspondence Problem
Input: Two sets of n strings
A = w1, w2,, wn
B = v1, v2,, vn There is a Post Correspondence Solution if there is a sequence i , j , , k such that:
PC-solution: wiwj wk = viv j vk
Indices may be repeated or omitted Example: w1 w2 w3 A: 100 11 111
v1 v2 v3 B : 001
PC-solution: 2,1,3 w2w1w3 = v2v1v3
11100111 Example: w1 w2 w3 A: 00 001 1000
v1 v2 v3 B : 0 11 011
There is no solution
Because total length of strings from B is smaller than total length of strings from A The Modified Post Correspondence Problem
Inputs: A = w1, w2,, wn
B = v1, v2,, vn
MPC-solution: 1,i, j,,k
w1wiwj wk = v1viv j vk Example: w1 w2 w3 A: 11 111 100
v1 v2 v3 B : 001
MPC-solution: 1,3,2 w1w3w2 = v1v3v2
11100111 ◼
102 ◼
103 ◼
Numerator
Denominator
104 PCP is Undecidable
◼
105 q a → x, R b → x, L 0 q1 q2
◼ TM M = (Q, , , , q0 , B, F)
◼ ∑={a,b},
◼ Г={a, b, B, x}
◼ input w=aba
◼ Create Dominos using transitions
106 ◼
...... B a b ......
q0
107 ◼
a b x
q0 q1
q a → x, R b → x, L 0 q1 q2
108 ◼
x b a
q1 q2
Г=a,b,x,B
q a → x, R b → x, L 0 q1 q2
109 ◼
q a → x, R b → x, L 0 q1 q2
110 ◼
q a → x, R b → x, L 0 q1 q2
111 ◼
q a → x, R b → x, L 0 q1 q2
112 ◼ q a → x, R b → x, L 0 q1 q2
113 ◼
114 ◼
Since ATM is undecidable, PCP must be, too.
115 ◼ PCP = {
|P is a collection of dominoes with a match} is undecidable.
◼ Proof
◼ The idea behind showing it undecidable is to reduce
ATM to it. ◼ TM configurations are converted into domino sequences.
◼ For a given M and w, w ∈ L(M) iff a match exists for the dominoes.
◼ Since ATM is undecidable, PCP must be, too. 116 Summary
◼ Problems vs. languages ◼ Decidability
◼ Recursive ◼ Undecidability
◼ Recursively Enumerable
◼ Not RE
◼ Examples of languages ◼ The diagonalization technique ◼ Reducability
117 “Languages” vs. “Problems”
A “language” is a set of strings
Any “problem” can be expressed as a set of all strings that are of the form:
◼ “
118 ◼ Function that is both one-to-one and onto is called a correspondence.
◼ In a correspondence, every element of A maps to a unique element of B and each element of B has a unique element of A mapping to it.
◼ A correspondence is simply a way of pairing the elements of A with the elements of B.
119 ◼ Let N be the set of natural numbers {1, 2, 3, . . .}
◼ Let E be the set of even natural numbers {2, 4, 6,. .}.
◼ Using Cantor’s definition of size, we can see that N and E have the same size.
◼ The correspondence f mapping N to E is simply
◼ f(n) = 2n.
120 Some languages are not Turing-recognizable.
◼ T=Turing Machines are finite.
◼ L=Language is also infinite
◼ so there is no correspondence between T and L.S.T.
◼ F(T)=L
121 ◼ ATM = {
◼ We should call the language HALTTM below the halting problem.
◼ HALTTM = {
◼ We use the acceptance problem to prove HALTTM undecidable. Theorem HALTTM is undecidable.
122 ◼ he comment was referring to the Big-O Notation.
◼ Briefly:
◼ O(1) means in constant time - independent of the number of items.
◼ O(N) means in proportion to the number of items.
◼ O(log N) means a time proportional to log(N)
◼ Basically any 'O' notation means an operation will take time up to a maximum of k*f(N) where:
◼ k is a constant multiplier
◼ f() is a function that depends on N 123 A limitation of Turing Machines:
Turing Machines are “hardwired”
they execute only one program
Real Computers are re-programmable Solution: Universal Turing Machine
Attributes:
• Reprogrammable machine
• Simulates any other Turing Machine Universal Turing Machine simulates any Turing Machine M
Input of Universal Turing Machine:
Description of transitions of M
Input string of • Encoding The Turing Machine to i/p UTM Tape 1 Three tapes
Description of
Universal Tape 2 Turing Machine Tape Contents of M
Tape 3
State of Tape 1
Description of
We describe Turing machine M as a string of symbols:
We encode as a string of symbols Alphabet Encoding
Symbols: a b c d
Encoding: 1 11 111 1111 State Encoding States: q1 q2 q3 q4
Encoding: 1 11 111 1111
Head Move Encoding
Move: L R
Encoding: Transition Encoding
Transition: (q1,a) = (q2,b, L)
Encoding: 10101101101
separator Turing Machine Encoding
Transitions: (q1,a) = (q2,b, L) (q2,b) = (q3,c,R)
Encoding: 10101101101 00 1101101110111011
separator Tape 1 contents of Universal Turing Machine:
binary encoding of the simulated machine M
Tape 1 1 0 1 0 11 0 11 0 10011 0 1 10 111 0 111 0 1100 A Turing Machine is described with a binary string of 0’s and 1’s
Therefore: The set of Turing machines forms a language:
each string of this language is the binary encoding of a Turing Machine Language of Turing Machines
L = { 010100101, (Turing Machine 1)
00100100101111, (Turing Machine 2)
111010011110010101, ……
…… } Universal Turing Machine:
• Simulate M on w accept, if M accepts w • UTM(
◼ TMs == Recursively Enumerable languages ◼ TMs can be used as both:
◼ Language recognizers
◼ Calculators/computers ◼ Basic TM is equivalent to all the below:
1. Multi-track TM
2. Multi-tape TM
3. Non-deterministic TM
4. Enumerator ◼ TMs are like universal computing machines with unbounded storage (Universal Turing Machine) ◼ Comparison of All Automata 138 decidability
139 Recall that: A language L is Turing-Acceptable if there is a Turing machine M that accepts
Also known as: Turing-Recognizable or Recursively-enumerable languages For any string w :
w L M halts in an accept state
w L halts in a non-accept state or loops forever Definition: A language L is decidable if there is a Turing machine (decider) M which accepts and halts on every input string
Also known as recursive languages For any string w : w L M halts in an accept state
w L halts in a non-accept state
Every decidable language is Turing-Acceptable Decidability vs. Undecidability
◼ There are two types of TMs (based on halting): (Recursive) TMs that always halt, no matter accepting or non- accepting DECIDABLE PROBLEMS (Recursively enumerable) TMs that are guaranteed to halt only on acceptance. If non-accepting, it may or may not halt (i.e., could loop forever).
◼ Undecidability:
◼ Undecidable problems are those that are not recursive
144 Recursive, RE, Undecidable languages
No TMs exist TMs that always halt LBA Non-RE Languages TMs that may or (all other languages for which may not halt no TMs can be built)
Regular Context-
(DFA) free sensitive
(PDA) Context
Recursively Enumerable (RE) Recursive
“Undecidable” problems “Decidable” problems 145 Recursive Languages & Recursively Enumerable (RE) languages
◼ Any TM for a Recursive language is going to look like this: “accept” w M “reject”
◼ Any TM for a Recursively Enumerable (RE) language is going to look like this: “accept” w M “?” 146 ◼ Languages recognized by a TM are called recognizable/acceptable.
◼ Languages decided by a TM are called decidable.
147 Closure Properties of: - the Recursive language class, and - the Recursively Enumerable language class
148 Recursive Languages are closed under complementation
◼ If L is Recursive, L is also Recursive
M “accept” “accept” w w M “reject” “reject”
149 Are Recursively Enumerable Languages closed under complementation? (NO)
◼ If L is RE, L need not be RE
M “accept” “accept” ? w w M “reject” ?
150 Recursive Langs are closed under Union
◼ Let Mu = TM for L1 U L2 ◼ M construction: Mu u accept M 1. Make 2-tapes and 1 reject copy input w on both OR tapes w accept 2. Simulate M1 on tape 1 M 2 reject 3. Simulate M2 on tape 2
4. If either M1 or M2 accepts, then Mu accepts
5. Otherwise, Mu rejects.
151 Recursive Langs are closed under Intersection
◼ Let Mn = TM for L1 L2 ◼ M construction: Mn n accept M 1. Make 2-tapes and 1 reject copy input w on both ANDAND tapes w accept 2. Simulate M1 on tape 1 M 2 reject 3. Simulate M2 on tape 2
4. If M1 AND M2 accepts, then Mn accepts
5. Otherwise, Mn rejects.
152 Other Closure Property Results
◼ Recursive languages are also closed under:
◼ Concatenation
◼ Kleene closure (star operator)
◼ Homomorphism, and inverse homomorphism
◼ RE languages are closed under:
◼ Union, intersection, concatenation, Kleene closure
◼ RE languages are not closed under:
◼ complementation
153 Decidable and Undecidable Problems
Problem
Solvable Unsolvable
Decidable Undecidable (Algorithm) (Procedure) Solution is definite Sometime yes or sometime no Example of Decidable Problems
◼ Does finite Automata Accept Regular Language?
◼ L1 and L2 are two regular languages, are they closed under following property:
◼ Union
◼ Concatenation
◼ Kleene Closure
◼ If L1 and L2 is CFL then L1 U L2 is CFL?
155 Example of Undecidable Problems
◼ For given context free Grammer G, is L(G) is ambiguous?
◼ For two given CFL L1 and L2 whether L1 ∩ L2 is CFL or not?
◼ A Computational problem is said to be unsolvable if no algorithm to solve it.
◼ A problem is said to be undecidable if it is a decision problem and no algorithm can decide the solution.
156 For a decidable language :
Decision Decider for L On Halt: qaccept Accept Input string Reject qreject
For each input string, the computation halts in the accept or reject state For a Turing-Acceptable language :
Turing Machine for L
qaccept Input string
qreject
It is possible that for some input string the machine enters an infinite loop Problem: Is number x prime?
Corresponding language:
PRIMES = {1, 2, 3, 5, 7, }
We will show it is decidable Decider for PRIMES : On input number x : Divide x with all possible numbers between 2 and x
If any of them divides Then reject Else accept the decider for the language solves the corresponding problem
Decider for PRIMES
qaccept YES (Accept) Input number x is prime? (Input string) NO qreject (Reject) “Languages” vs. “Problems”
A “language” is a set of strings
Any “problem” can be expressed as a set of all strings that are of the form:
◼ “
Question: Given regular language L and string w how can we check if w L ?
Answer: Take the DFA that accepts L and check if w is accepted DFA w w L
DFA
w L Question: Given regular language L how can we check if is empty: ( L = ) ?
Answer: Take the DFA that accepts L
Check if there is any path from the initial state to an accepting state DFA
L
DFA
L = Problem: Does DFA M accept the empty language L ( M ) = ?
Corresponding Language:X (Decidable)
EMPTYDFA = { M : M is a DFA that accepts empty language }
Description of DFA M as a string (For example, we can represent as a binary string, as we did for Turing machines) Decider for EMPTYDFA :
On input M : Determine whether there is a path from the initial state to any accepting state DFA M DFA
L(M) L(M) =
Decision: Reject M Accept M Question: Given regular language L how can we check if is finite?
Answer: Take the DFA that accepts L
Check if there is a walk with cycle from the initial state to a final state DFA
L is infinite
DFA
is finite Problem: Does DFA M accept a finite language?
Corresponding Language:X (Decidable)
FINITEDFA = { M : M is a DFA that accepts a finite language} Decider for FINITEDFA : On input M : Check if there is a walk with a cycle from the initial state to an accepting state
DFA M DFA
infinite finite
Decision: Reject M Accept M (NO) (YES) DFA Acceptance Problem: Given a DFA M and an input w, does M accept w?
Corresponding Language:X (Decidable)
ADFA = { ⟨M, w⟩ : M is a DFA that accepts input string w }
ADFA is a decidable language. Decider for ADFA : On input string :
Run DFA M on input string w
If accepts Then accept M , w (and halt) Else reject (and halt) ◼ Proof.
◼ We construct a TM M that decides ADFA. ◼ On input ⟨M,w⟩ where M is a DFA and w an input
◼ 1. Simulate B on input w
◼ 2. If the simulation ends in an accept state, accept. If it ends in a non-accepting state, reject.
◼ Input would be the 5 tuple description of the DFA – the TM would keep track of the current state and process the input tape based on transition function and finally accept if the string’s final state is accept.
176 ◼ ACFG = {⟨G,w⟩ | G is a CFG deriving string w}
◼ Theorem. ACFG is decidable. ◼ Proof. T = “On input
◼ 1. Convert G to Chomsky Normal Form.
◼ 2. Test all possible derivations of length 2n-1, where n = |w|.
◼ 3. If any generate w, accept, otherwise reject.”
177 ◼ ACFG = {⟨G,w⟩ | G is a CFG & L ( G ) = }
◼ Theorem. ACFG is decidable. ◼ Proof. T = “On input G Where G is CFG
◼ 1. Mark all terminal symbols in G.
◼ 2. Repeat until no new variables become marked.
◼ 3. Mark any A where A → U1U2 . . . Uk and all Ui have already been marked.
◼ 4. If start symbol marked, reject, else accept
178 Decidable Languages A language A is decidable, if there is a Turing machine M (decider) that accepts the language A and halts on every input string
Decision Turing Machine M On Halt: YES Accept Input Decider for string NO Reject A Language about TMs/FAs/PDAs & acceptance
ADFA = { ⟨M, w⟩ : M is a DFA that accepts input string w }
Let ADFA be the language of all strings
180 is Turing-Acceptable
Turing machine that accepts A TM :
1. Run M on input w
M,w 2. If accepts then accept A computational problem is decidable if the corresponding language is decidable
We also say that the problem is solvable Undecidable Languages undecidable language = not decidable language
There is no decider: there is no Turing Machine which accepts the language and makes a decision (halts) for every input string
(machine may make decision for some input strings) For an undecidable language, the corresponding problem is undecidable (unsolvable):
there is no Turing Machine (Algorithm) that gives an answer (yes or no) for every input instance
(answer may be given for some input instances) We have shown before that there are undecidable languages:
L Turing-Acceptable L
Decidable
is Turing-Acceptable and undecidable
Halting Problem Input: a program P in some programming language and input I for P Output: Does P go into an infinite loop when given I? or it finish. true if P terminates; false if P runs forever. We will prove that two particular problems are unsolvable/undecidable:
Membership problem
ATM= {
Halting problem
HALTTM= {
Non-RE Languages
x
Regular Context-
(DFA) free sensitive
(PDA) Context
Recursively Enumerable (RE) Recursive
189 What is the Halting Problem?
Definition of the “halting problem”:
◼ Does a givenTuring Machine M halt on a given input w?
Input w Machine M
190 A Turing Machine simulator The Universal Turing Machine
◼ Given: TM M & its input w ◼ Aim: Build another TM called “H”, that will output:
◼ “accept” if M accepts w, and
◼ “reject” otherwise
◼ An algorithm for H: Implies: H is in RE ◼ Simulate M on w
accept, if M halts on w ◼ H(
Question: If M does not halt on w, what will happen to H? 191 A Claim
◼ Claim: No H that is always guaranteed to halt, can exist!
◼ Proof: (Alan Turing, 1936)
◼ By contradiction, let us assume H exists
“accept”
192 Therefore, if H exists ➔ D also should exist. But can such a D exist? (if not, then H also cannot exist) HP Proof (step 1)
◼ Let us construct a new TM D using H as a subroutine:
◼ On input
1. Run H on input
2. Output the opposite of what H outputs;
D
“accept” “accept”
193 HP Proof (step 2)
◼ The notion of inputing “
◼ A program can be input to itself (e.g., a compiler is a program that takes any program as input) accept, if M does not accept
accept, if D does not accept
A contradiction!!! ==> Neither D nor H can exist. 194 Of Paradoxes & Strange Loops
E.g., Barber’s paradox, Achilles & the Tortoise (Zeno’s paradox) MC Escher’s paintings
A fun book for further reading: “Godel, Escher, Bach: An Eternal Golden Braid” by Douglas Hofstadter (Pulitzer winner, 1980) 195 The Diagonalization Language
Example of a language that is not recursive enumerable
(i.e, no TMs exist)
196 The Diagonalization language
The Halting Problem
Non-RE Languages
x
x
Regular Context-
(DFA) free sensitive
(PDA) Context
Recursively Enumerable (RE) Recursive
197 Why should there be languages that do not have TMs?
We thought TMs can solve everything!!
198 Non-RE languages
How come there are languages here? (e.g., diagonalization language)
Non-RE Languages
Regular Context-
(DFA) free sensitive
(PDA) Context
Recursively Enumerable (RE) Recursive
199 One Explanation
There are more languages than TMs
◼ By pigeon hole principle:
◼ ==> some languages cannot have TMs
◼ But how do we show this?
◼ Need a way to “count & compare” two infinite sets (languages and TMs)
200 How to count elements in a set?
Let A be a set:
◼ If A is finite ==> counting is trivial
◼ If A is infinite ==> how do we count?
◼ And, how do we compare two infinite sets by their size?
201 THE DIAGONALIZATION METHOD
◼ mathematician Georg Cantor in 1873
◼ problem of measuring the sizes of infinite sets.
◼ For finite sets
◼ use the counting method for Infinite????
◼ finite sets have the same size if the elements of one set can be paired with the elements of the other set.
◼ extend this idea to infinite sets
202 Cantor’s definition of set “size” for infinite sets (1873 A.D.)
Let N = {1,2,3,…} (all natural numbers) Let E = {2,4,6,…} (all even numbers)
Q) Which is bigger?
◼ A) Both sets are of the same size
◼ “Countably infinite”
◼ Proof: Show by one-to-one, onto set correspondence from N ==> E n f(n) 1 2 i.e, for every element in N, 2 4 there is a unique element in E, 3 6 and vice versa. . . . . 203 . . Really, really big sets! (even bigger than countably infinite sets) Uncountable sets
Example: ◼ Let R be the set of all real numbers ◼ Claim: R is uncountable
n f(n) 1 3 . 1 4 1 5 9 … Build x s.t. x cannot possibly 2 5 . 5 5 5 5 5 … occur in the table 3 0 . 1 2 3 4 5 … 4 0 . 5 1 4 3 0 … E.g. x = 0 . 2 6 4 4 … . . .
204 Therefore, some languages cannot have TMs…
◼ The set of all TMs is countably infinite
◼ The set of all Languages is uncountable
◼ ==> There should be some languages without TMs
205 Reducibility
210 ◼ Method for proving that problems are computationally unsolvable. It is called reducibility.
◼ A reduction is a way of converting one problem to another problem in such a way that a solution to the second problem can be used to solve the first problem.
211 ◼ Reducibility's come up often in everyday life.
◼ E.g. You want to find your way around a new city.
◼ would be easy if you had a map.
◼ Thus, you can reduce the problem of finding your way around the city to the problem of obtaining a map of the city.
◼ Reducibility always involves two problems, which we call A and B.
◼ If A reduces to B, we can use a solution to B to solve A.
◼ So in our example, A is the problem of finding your way around the city and B is the problem of obtaining a map.
◼ Note that reducibility says nothing about solving A or B alone, but only about the solvability of A in the presence of a solution to B. 212 ◼ The problem of traveling from Pune to Delhi reduces to the problem of buying a plane ticket between the two cities
◼ the problem of measuring the area of a rectangle reduces to the problem of measuring its length and width.
213 ◼ Reducibility plays an important role in classifying problems by decidability, and later in complexity theory as well.
◼ When A is reducible to B, solving A cannot be harder than solving B because a solution to B gives a solution to A.
◼ In terms of computability theory, if A is reducible to B and B is decidable, A also is decidable. Equivalently, if A is undecidable and reducible to B, B is undecidable. 214 Reducability
◼ To prove: Problem P1 is undecidable
◼ Given/known: Problem P2 is undecidable ◼ Reduction idea:
1. “Reduce” P2 to P1:
◼ Convert P2’s input instance to P1’s input instance s.t. i) P2 decides only if P1 decides
2. Therefore, P2 is decidable 3. A contradiction
4. Therefore, P1 has to be undecidable
215 Post Correspondence Problem, or PCP.
◼
217 ◼
218 ◼
219 The Post Correspondence Problem
Input: Two sets of n strings
A = w1, w2,, wn
B = v1, v2,, vn There is a Post Correspondence Solution if there is a sequence i , j , , k such that:
PC-solution: wiwj wk = viv j vk
Indices may be repeated or omitted Example: w1 w2 w3 A: 100 11 111
v1 v2 v3 B : 001
PC-solution: 2,1,3 w2w1w3 = v2v1v3
11100111 Example: w1 w2 w3 A: 00 001 1000
v1 v2 v3 B : 0 11 011
There is no solution
Because total length of strings from B is smaller than total length of strings from A The Modified Post Correspondence Problem
Inputs: A = w1, w2,, wn
B = v1, v2,, vn
MPC-solution: 1,i, j,,k
w1wiwj wk = v1viv j vk Example: w1 w2 w3 A: 11 111 100
v1 v2 v3 B : 001
MPC-solution: 1,3,2 w1w3w2 = v1v3v2
11100111 ◼
226 ◼
227 ◼
Numerator
Denominator
228 PCP is Undecidable
◼
229 q a → x, R b → x, L 0 q1 q2
◼ TM M = (Q, , , , q0 , B, F)
◼ ∑={a,b},
◼ Г={a, b, B, x}
◼ input w=aba
◼ Create Dominos using transitions
230 ◼
...... B a b ......
q0
231 ◼
a b x
q0 q1
q a → x, R b → x, L 0 q1 q2
232 ◼
x b a
q1 q2
Г=a,b,x,B
q a → x, R b → x, L 0 q1 q2
233 ◼
q a → x, R b → x, L 0 q1 q2
234 ◼
q a → x, R b → x, L 0 q1 q2
235 ◼
q a → x, R b → x, L 0 q1 q2
236 ◼ q a → x, R b → x, L 0 q1 q2
237 ◼
238 ◼
Since ATM is undecidable, PCP must be, too.
239 ◼ PCP = {
|P is a collection of dominoes with a match} is undecidable.
◼ Proof
◼ The idea behind showing it undecidable is to reduce
ATM to it. ◼ TM configurations are converted into domino sequences.
◼ For a given M and w, w ∈ L(M) iff a match exists for the dominoes.
◼ Since ATM is undecidable, PCP must be, too. 240 Summary
◼ Problems vs. languages ◼ Decidability
◼ Recursive ◼ Undecidability
◼ Recursively Enumerable
◼ Not RE
◼ Examples of languages ◼ The diagonalization technique ◼ Reducability
241 ◼ Function that is both one-to-one and onto is called a correspondence.
◼ In a correspondence, every element of A maps to a unique element of B and each element of B has a unique element of A mapping to it.
◼ A correspondence is simply a way of pairing the elements of A with the elements of B.
242 ◼ Let N be the set of natural numbers {1, 2, 3, . . .}
◼ Let E be the set of even natural numbers {2, 4, 6,. .}.
◼ Using Cantor’s definition of size, we can see that N and E have the same size.
◼ The correspondence f mapping N to E is simply
◼ f(n) = 2n.
243 Some languages are not Turing-recognizable.
◼ T=Turing Machines are finite.
◼ L=Language is also infinite
◼ so there is no correspondence between T and L.S.T.
◼ F(T)=L
244 ◼ ATM = {
◼ We should call the language HALTTM below the halting problem.
◼ HALTTM = {
◼ We use the acceptance problem to prove HALTTM undecidable. Theorem HALTTM is undecidable.
245 ◼ he comment was referring to the Big-O Notation.
◼ Briefly:
◼ O(1) means in constant time - independent of the number of items.
◼ O(N) means in proportion to the number of items.
◼ O(log N) means a time proportional to log(N)
◼ Basically any 'O' notation means an operation will take time up to a maximum of k*f(N) where:
◼ k is a constant multiplier
◼ f() is a function that depends on N 246