Computability and computational complexity
Lecture 10: Randomized computation
Ion Petre Computer Science, Åbo Akademi University
Fall 2015 http://users.abo.fi/ipetre/computability/
December 2, 2015 http://users.abo.fi/ipetre/computability/ 1 Content
n Monte Carlo algorithms: examples n Random walk n Randomized complexity classes
q RP
q ZPP
q PP
q BPP n Sources of randomness
December 2, 2015 http://users.abo.fi/ipetre/computability/ 2 MONTE CARLO ALGORITHMS: EXAMPLES
December 2, 2015 http://users.abo.fi/ipetre/computability/ 3 PRIMES
n Very important to be able to generate (very) large primes – e.g., in cryptography (RSA) n No practical techniques to yield large prime numbers RSA scheme n Procedure: generate random odd numbers and test –Key generation whether that integer is prime •Choose primes p,q n Testing whether or not an integer n is a prime is a •Compute n=pq difficult problem (“primality is difficult”) •Choose e, 1 q Answer (2002): YES! –Encryption e q Manindra Agrawal, Neeraj Kayal and Nitin Saxena, •C=M mod n “PRIMES is in P”, Ann. of Math. (2), 160:2 (2004) 781-- –Decryption: 793. •Cd mod n = Mde mod n = M 12 q Drawback: high complexity – O((log n) f(log log n)), where f is a polynomial n RSA typically uses primes on 1024-2048 bits 6 q Subsequently improved to O((log n) f(log log n)) December 2, 2015 http://users.abo.fi/ipetre/computability/ 4 Miller-Rabin primality test n Faster methods of testing primality exist – they are all probabilistic q Such an algorithm can give two answers to the question “Is n prime?” n No, it is not n n is probably prime n The probability can be made arbitrarily large q Other types of randomized algorithms may give a precise answer but with low probability they may take a long time to finish n Most popular primality test: Miller- Rabin, based on Fermat’s little theorem December 2, 2015 http://users.abo.fi/ipetre/computability/ 5 Fermat’s little theorem n Fermat’s little theorem: if p is prime and a is positive integer not divisible by p, then ap-1 º 1 mod p n Corollary: For any positive integer a and prime p, ap º a mod p n Comments: q Fermat’s little theorem provides a necessary condition for an integer p to be prime – the condition is not sufficient n We will turn this theorem into a (probabilistic) test for primality December 2, 2015 http://users.abo.fi/ipetre/computability/ 6 Miller-Rabin primality test Fermat’s little theorem: if p is prime and a is positive integer not divisible by p, then ap-1 º 1 mod p •Question: for how many integers a does the test fail? TEST(n) •Failure: n is not prime but the algorithm return 1. n-1=2kq: compute k and q “probably prime” •Answer: for at most (n-1)/4 integers a with 1£a£n-1 2. Select a random integer a, •Thus, the probability of failure is at most ¼ 1 December 2, 2015 http://users.abo.fi/ipetre/computability/ 7 Miller-Rabin primality test n Fermat’s little theorem: if p is prime and a is positive integer not divisible by p, then ap-1 º 1 mod p n Idea of the Miller-Rabin test: q We need to test if the odd integer n is prime: test the equality in Fermat’s little theorem for n and a random a n A speedup may be done so that we do not have to compute all powers of a – details bellow k q n-1 is even, i.e., of the form n-1=2 q, with k>0, q odd: k and q easy to find q Choose an integer a such that 1 q By Fermat’s theorem, if n is prime, then the last value in the sequence is 1 – the sequence may have some other 1s, consider the first 1 in the sequence n Case 1: the first number in the sequence is 1 – then all other powers are also 1 j j-1 n Case 2: some number a2 q in the sequence is 1 – in this case a2 q = n-1 mod n j j-1 j-1 j-1 j-1 q 0 = (a2 q -1) mod n = (a2 q – 1) (a2 q + 1) mod n, i.e., n divides (a2 q – 1) or (a2 q + 1) j-1 j-1 q Since we took the first 1 in the sequence, it follows that n divides (a2 q + 1): a2 q = n-1 mod n q The test: if either the first element in the sequence is 1, or some element before the last is n-1, then n could be prime. Otherwise n is certainly not prime December 2, 2015 http://users.abo.fi/ipetre/computability/ 8 Monte-Carlo algorithms n Definition: Monte-Carlo algorithms q Polynomial-time randomized algorithms: it finishes in polynomial time for all random choices q No false positives: if the algorithm says “yes”, then the result is certain q There might be false negatives: the algorithm does not answer “no” but rather “probably not” q Probability of false negatives is less than 1 n For random choices we assume the existence of a fair coin that generates perfectly random bits (re-discuss sources of randomness later in this lecture) n Important note: by repeating the algorithm in case of a negative answer we can make the probability of a false negative arbitrarily small q p<1 is the probability of a false negative k q getting a false negative in k consecutive runs of the algorithm is p , which decreases to 0 as k increases q running time remains polynomial even if we repeat the algorithm k times December 2, 2015 http://users.abo.fi/ipetre/computability/ 9 Random walks n A randomized algorithm for SAT q start with an arbitrary truth assignment T and repeat the following r times n if all clauses are satisfied, then reply “satisfiable” and halt n otherwise, take any unsatisfied clause: all of its literals are false under T n pick any of these literals and flip its truth value, updating T q after r iterations, reply “probably unsatisfiable” and halt n This is called a random walk algorithm: random path through the possible truth assignments n If the formula is unsatisfiable, then the algorithm answers well: “probably unsatisfiable” n If the formula is satisfiable: q easy to show that if we allow exponentially many repetitions, then a satisfying truth assignment can be found with high probability q Question: how about if we only try a number of times r that is polynomial in the number of variables? n main problem: the (very) large probability of false negatives q Answer: It does not work well for (some instances of) 3SAT! But it works for 2SAT! December 2, 2015 http://users.abo.fi/ipetre/computability/ 10 Random walks and 2SAT n Theorem. Suppose that the random walk algorithm with r=2n2 is applied to any satisfiable instance of 2SAT with n variables. Then the probability that a satisfying truth assignment will be discovered is at least ½. n Important technical result for the proof of the theorem (even though we skip it here): n Lemma. If x is a random variable taking nonnegative integer values, then for any k>0, P(x³k×E(x))£1/k, where E(x) is the expected value of x. December 2, 2015 http://users.abo.fi/ipetre/computability/ 11 RANDOMIZED COMPLEXITY CLASSES December 2, 2015 http://users.abo.fi/ipetre/computability/ 12 Randomized complexity classes n Several types of randomized algorithms exist q Monte Carlo q Las Vegas q Decision reached by simple majority q Decision reached by strong majority q … n Discuss here several of these concepts and the relationships among them n Also discuss how to implement randomized algorithms in practice December 2, 2015 http://users.abo.fi/ipetre/computability/ 13 Randomized complexity classes: RP n We only consider here polynomial-time bounded nondeterministic TM N standardized as follows: q N is precise (all nondeterministic computations halt after exactly the same number of steps) q there are exactly two nondeterministic choices in each step of N n Definition. Let L be a language. A polynomial Monte Carlo TM for L is a nondeterministic TM standardized as above such that: q each computation on an input of size n halts in exactly p(n) steps p(|x|) q if xÎL, then at least half of the 2 computations halt with “yes” q if xÏL, then all computations halt with “no” q In other words, no false positives and false negatives with a probability not more than ½ n The class of all languages with polynomial Monte Carlo TM is denoted RP (randomized polynomial time) December 2, 2015 http://users.abo.fi/ipetre/computability/ 14 RP n RP q N is precise (all nondeterministic computations halt after exactly the same number of steps) q there are exactly two nondeterministic choices in each step of N q each computation on an input of size n halts in exactly p(n) steps p(|x|) q if xÎL, then at least half of the 2 computations halt with “yes” q if xÏL, then all computations halt with “no” December 2, 2015 http://users.abo.fi/ipetre/computability/ 15 Discussion n The class RP remains the same if we replaced the condition that the probability of acceptance is at least half with a number strictly between 0 and 1 q Argument based on repeating the TM computation enough many times n What is the relationship between RP and P, NP? q clearly somewhere between P and NP n any polynomial Monte Carlo algorithm is by definition a nondeterministic polynomial-time algorithm n any polynomial deterministic algorithm is trivially a polynomial Monte Carlo algorithm n Note: not all nondeterministic TM define languages in RP q they have to accept “by majority” q in fact, it is undecidable whether a given TM has this property December 2, 2015 http://users.abo.fi/ipetre/computability/ 16 Randomized complexity classes: ZPP n How about problems in RPÇcoRP q they have two Monte Carlo algorithms – one has no false positives, the other has no false negatives q run both algorithms enough many times and a certain answer will come with high probability (larger than 1-2-k after k repetitions) n the two algorithms may give contradictory answers for a while q main difference with respect to Monte Carlo algorithms: in the end we know the answer for sure. This is a Las Vegas algorithm – the owner of the algorithm never loses! n Las Vegas algorithms q they always give the correct result, or they inform about the failure (“don’t know”) q probability of failure is less than 1 n Note: Las Vegas algorithms do not “gamble” with the verity of the answer, only with the available resources n Notation. We denote by ZPP (polynomial randomized algorithms with zero-probability of error) the class of languages with Las Vegas algorithms q ZPP=RPÇcoRP December 2, 2015 http://users.abo.fi/ipetre/computability/ 17 Randomized complexity classes: PP n MAJSAT. Given a Boolean expression, is it true that the majority of its 2n truth assignments satisfy it? n-1 q In other words: at least 2 +1truth assignments satisfy it q Not clear that the problem is in NP: its obvious certificate is not at all succint! n Definition. We say that language L is in class PP (probabilistic polynomial time) if there is a nondeterministic polynomial TM N standardized as before such that xÎL iff more than half of the computations of N on x end up accepting q we say that N decides L “by majority” q It can be shown that MAJSAT is PP-complete n Theorem. NPÍPP. Proof. Let LÎNP be decided by the nondeterministic polynomial machine N. We build a new machine N’ deciding L by majority: n a new initial state with two nondeterministic choices: one always accepts, the other simulates N on x n If N on x computes for p(|x|), then it has 2p(|x|) computations. N’ has 2p(|x|)+1 computations, at least half of them accepting x n N’ accepts x “by majority” December 2, 2015 http://users.abo.fi/ipetre/computability/ 18 PP – continued n Big difference between RP, ZPP, PP q RP and ZPP are plausible notions of efficient randomized computation; realistic proposals for practical algorithms; somewhat similar to P q PP is natural to capture certain problems such as MAJSAT, but it has no realistic computational content; not capturing practical algorithms; somewhat similar to NP n Main problem with PP: acceptance “by majority” is in some sense too fragile: q x could be accepted with acceptance majority ½+2-p(|x|), with just two more accepting computations than rejecting (out of exponentially many) q there is no plausible efficient experimentation that can detect such marginally accepting behavior q to understand this, consider the example of a biased coin December 2, 2015 http://users.abo.fi/ipetre/computability/ 19 Randomized complexity classes: BPP n Definition. BPP is the class of languages for which there is a nondeterministic polynomial TM N (standardized as before) with the following property for all inputs x: q if xÎL, then at least ¾ of the computations of N on x accept q if xÏL, then at least ¾ of the computations of N on x reject n Comments q BPP stands for probabilistic polynomial algorithms with bounded probability of error q BPP = accept/reject with clear majority q Any number higher than ½ could replace ¾ in the definition n it can be seen based on the argument of iterating the machine enough many times, plus the Chernoff bound n Clearly, RPÍBPPÍPP q first inclusion: run the algorithm twice to limit the probability of false negatives to at most ¼ q second inclusion: decision “by clear majority” is trivially a decision “by majority” n BPP=coBPP n Open problem: is BPPÍNP? December 2, 2015 http://users.abo.fi/ipetre/computability/ 21 SOURCES OF RANDOMNESS December 2, 2015 http://users.abo.fi/ipetre/computability/ 22 Sources of randomness n Implementing any probabilistic algorithm relies on the availability of a source of random bits n Definition. A perfect random source is a random variable with values that are infinite n sequences (x1,x2,…) of bits such that for all n³0 and all (y1,y2,…,yn)Î{0,1} , P(xi=yi, for all 1£i£n)=2-n. q In other words: each xi is the outcome of an independent experiment, where values 0 and 1 are equally probable n With a perfect random source one can directly implement any Monte Carlo algorithm n Question: Are there perfect random sources in nature? q No clear answer exists. n Note (John von Neumann): even though in the definition of a perfect source we ask independence of the bits and fairness, the only crucial ingredient is independence q a biased source of independent bits (0 has probability p, with 0 q break the bit sequence into pairs; ignore pairs 00 and 11; re-label pair 01 as 0 and pair 10 as 1 December 2, 2015 http://users.abo.fi/ipetre/computability/ 23 Sources of randomness n Main problem with a physical source of randomness: outcomes of physical process tend to be affected by the previous output and the circumstances that led to it n Idea: discover randomness in math and computer science à pseudo- random number generators q generate number that “look” random q many options exist q given that they all start from a seed, they are of course non-random December 2, 2015 http://users.abo.fi/ipetre/computability/ 24 Slightly random sources n A weaker concept of randomness that turns out to be much more plausible physically n Definition. Let d be a number with 0 function. The d-random source Sp is a random variable with values that are infinite sequences (x1,x2,…) of bits such that for all n³0 and all n (y1,y2,…,yn)Î{0,1} , n P(xi=yi, for all 1£i£n)=Σ i=1(yip(y1…yi-1)+(1-yi)(1-p(y1…yi-1)). q p is supposed to be some highly complex function unknown to us th q Note: the probability that the i bit is 1 is exactly p(y1…yi-1) – an arbitrary number in [d,1- d] that depends on all previous bits q previous bits might bias the current outcome, but not more than 1-d<1. q Note: a ½-random source is a perfect random source n A d-random source with d<½ is called slightly random. December 2, 2015 http://users.abo.fi/ipetre/computability/ 25 Slightly random sources n Slightly random sources are allowed to have strong dependencies between bits q much better suited for a physical implementation – less demanding on the physical process to be adopted as a slightly random source n Question: What if we used a slightly random source rather than a perfectly random source of bits in a randomized algorithm? December 2, 2015 http://users.abo.fi/ipetre/computability/ 26 RP and BPP with slightly random sources n Definition. Let N be a precise, polynomial nondeterministic TM with exactly two choices in each step, denoted the 0-choice and 0.6 0.4 the 1-choice. On input x, the computation N(x) is a full binary tree of depth n=p(|x|). The tree 0.9 0.1 0.5 0.5 has 2n+1-1 nodes, of which 2n-1 are internal, and 2n+1-2 edges. Let dÎ(0,½). 0.8 0.2 0.7 0.3 0.1 0.9 0.4 0.6 n A d-assignment F to N(x) is a mapping assigning each edge a value in the interval [d,1-d] such that the two edges leaving each internal node are summing up to 1 q Intuitively, a d-assignment gives a randomized run of N on x, when driven by a d-random source Sp. December 2, 2015 http://users.abo.fi/ipetre/computability/ 27 RP and BPP with slightly random sources n d-assignment F to N(x) q For each leaf l of N(x) define the probability of l to be P(l)=PaÎP(l)F(a), where P(l) is the path from the root to leaf l – this is the probability of reaching the leaf q Define probability P(N(x)=“yes”|F) to be the sum of P(l) for all “yes” leaves l of N(x) n We say that language L is in d-RP if there is a nondeterministic TM N standardized as before such that for all d-assignments F we have: q If xÎL then P(M(x)=“yes”|F)³½ q If xÏL then P(M(x)=“yes”|F)=0 q In other words: no false positives, false negatives with less than half probabilities when driven by any d-random source n We say that language L is in d-BPP if there is a nondeterministic TM N standardized as before such thatfor all d-assignments F we have: q If xÎL then P(M(x)=“yes”|F)³3/4 q If xÏL then P(M(x)=“no”|F)³3/4 q In other words: decide correctly and with a clear margin when driven by any d-random source December 2, 2015 http://users.abo.fi/ipetre/computability/ 28 RP and BPP with slightly random sources n Clearly, 0-RP=0-BPP=P, ½-RP=RP, ½-BPP=BPP n Question: what about 0 q How much power of the randomization do we lose when we settle for slightly random sources rather than perfect random sources? q Answer: no loss. Randomization is practically implementable even with non- perfect sources of randomization! n Theorem. For any d>0, d-BPP=BPP and d-RP=RP q In other words, randomized algorithms can be implemented with only slightly random sources December 2, 2015 http://users.abo.fi/ipetre/computability/ 29 Learning objectives n The concepts of Monte Carlo and Las Vegas algorithms n The definitions of the randomized complexity classes RP, ZPP, PP, BPP n The concepts of perfect and slightly random sources n Randomization can be implemented even with slightly random sources December 2, 2015 http://users.abo.fi/ipetre/computability/ 30