Randomized Algorithms
1. Concepts of randomized algorithms 2. Randomize algorithm analysis 3. Randomized complexity classes
4. Pseudorandom number generation
5. Global min-cut problem 6. Maximum 3-SAT 7. Primality test
CP412 ADAII Introduction # 1 1. Concepts of randomized algorithms
• A randomized algorithm is an algorithm whose working not only depends on the input but also on certain random choices made by the algorithm during the execution – A randomized algorithm depends on random numbers for its operation – Assume that we have a random number generator Random(a, b) that generates for two integers a, b with a < b an integer r with a <=r <= b uniformly at random. In other words. We assume that Random(a, b) runs in O(1) time.
Examples randomized quick sort: randomly choose a pivoting element
CP412 ADAII Introduction # 2 Deterministic vs Randomized algorithms Boolean equivalence problem: given two boolean functions f1(x1, .., xn) and f2(x1, …, xn), question: f1=f2 ?
Deterministic algorithm: for each X in the enumeration of {0, 1}n if f1(X) != f2(X) return false; return true
Randomized algorithm (probabilistic algorithms): for k from 1 to N (or while true), randomly choose X from {0, 1}n if f1(X) != f2(X) return false; return true
Note that “true” output of the above probabilistic algorithm may not be correct. But the “false” output is an correct answer.
CP412 ADAII Introduction # 3 Two types of randomized algorithms
• Las Vegas algorithms – use the randomness to reduce the expected running time or memory usage of computing process, but always terminate with a correct result in a bounded amount of time. e.g. randomized quicksort
• Monte Carlo algorithms (probabilistic algorithms) – depend on the random input, have a chance of producing an incorrect result or fail to produce a result either by signalling a failure or failing to terminate
e.g. Randomized Boolean equivalence algorithm
CP412 ADAII Introduction # 4 Why randomized algorithms? • Use randomized algorithm to find a solution
• Using randomness to improve algorithm performance
• Using randomness to improve a solution to a problem
• Generating random data for testing
• Simulation
CP412 ADAII Introduction # 5 3. Analysis
• Resource usage for randomized algorithms – expected worst-case performance, which is the average amount of time it takes on the worst input of a given size. This average is computed over all the possible outcomes of the random numbers during the execution of the algorithm.
– Generally for Las Vegas algorithms
For randomized algorithm A, when input x and random bits r are given,
it becomes deteminstic algorithm. Let TA (x, r) denote the computing time
of the algorithem. TA (x, r) is a random function of random variable r. The expected running time is
E[TAA (x, r) ] = T (x, r) P[r] r 6 Randomized QuickSort
• Pick a pivot p uniformly at random from the elements of the array
Let T(n) be the expected number of comparisons done on an array of n elements.
We have T(0) = T(1) = 0 and for larger n.
Because there are n equally-likely choices for our pivot (hence the 1/n), and for each choice the expected cost is T(k) + T(n − 1 − k), where k is the number of elements less than the pivot.
7 1 n1 Tn() (() Tk Tn ( 1 k )) n k0 222nnn111 Tk() Tk () ak log k nnnkkk011 22 22log1aannnn kkdklog ( ) nn1 244 On(log) n
more precise: Tn ( ) 2 n log n
8 Expectation
Expectation. Given a discrete random variables X, its expectation E[X] is defined by:
Waiting for a first success. Coin is heads with probability p and tails with probability 1– p. How many independent flips X until first heads?
Linearity of expectation. Given two random variables X and Y defined over the same probability space, E[X + Y] = E[X] + E[Y].
CP412 ADAII Introduction # 9 Guessing cards
Game. Shuffle a deck of n cards; turn them over one at a time; try to guess each card.
Guessing with memory. Guess a card uniformly at random from cards not yet seen.
Claim. The expected number of correct guesses is Θ(log n). Coupon collector Coupon collector. Each box of cereal contains a coupon. There are n different types of coupons. Assuming all boxes are equally likely to contain each coupon, how many boxes before you have ≥ 1 coupon of each type?
Claim. The expected number of steps is Θ(n log n). Monte Carlo algorithm analysis
For Monte Carlo algorithms, when an random input is given, the execution is deterministic, but may get incorrect answer. We are interested in the probability of getting correct answer. To get high probability of correctness, we can do many runs. The overall running time is the number of runs times the time of each run.
E.g. Boolean equivalence problem, we are interested in probability of getting a correct answer of the probabilistic algorithm.
• For decision problems, these algorithms are generally classified as either false-biased or true-biased. . A false-biased Monte Carlo algorithm is always correct when it returns false; . a true-biased algorithm is always correct when it returns true
12 Randomized complexity classes
The class R or RP (Randomized Polynomial time) consists of languages L for which a polynomial-time Turing machine M exists such that P[M(x, r) = 1] ≥ 1/2 if x is in L P[M(x, r) = 1] = 0 if x is not in L
In other words, we can find a witness that x is in L with constant probability.
With n runs: P[Get yes answer] ≥ 1 − 2−n .
The class co-R (co-RP): P[M(x, r) = 1] ≥ 1/2 if x is not L P[M(x, r) = 1] = 0 if x is in L
CP412 ADAII Introduction # 13 Randomized complexity classes
The class ZPP (Zero-error Probabilistic Polynomial time) is defined as RP union co-RP.
The class BPP (Bounded-error Probabilistic Polynomial time): P[M(x, r) = 1] ≥ 2/3 if x is in L P[M(x, r) = 1] ≤ 1/3 if x is not in L
The class PP (Probabilistic Polynomial time): P[M(x, r) = 1] > 1/2 if x is in L P[M(x, r) = 1] ≤ 1/2 if x is not in L
CP412 ADAII Introduction # 14 4. Random number generation
“Anyone who attempts to generate random numbers by deterministic means is, of course, living in a state of sin.” —John von Neuman
• The computer is not capable of generating truly random numbers – The computer can only generate pseudorandom numbers-- numbers that are generated by a formula – Pseudorandom numbers look random, but are perfectly predictable if you know the formula • Pseudorandom numbers are good enough for most purposes, but not all--for example, not for serious security applications – Devices for generating truly random numbers do exist • They are based on physics properties
CP412 ADAII Properties of good pseudorandom number generator 1. The numbers must have a correct distribution. In simulations, it is important that the sequence of random numbers is uncorrelated (i.e. numbers in the sequence are not related in any way). In numerical integration, it is important that the distribution is flat. 2. The sequence of random numbers must have a long period. All random number generators will repeat the same sequence of numbers eventually, but it is important that the sequence is sufficiently long. 3. The sequences should be reproducible. Often it is necessary to test the effect of certain simulation parameters and the exact same sequence of random numbers should be used to run many different simulation runs. It must also be possible to stop a simulation run and then re-continue the same run which means that the state of the RNG must be stored in memory. 4. The RNG should be easy to export from one computing environment to another. 5. The RNG must be fast. Large amounts of random numbers are needed in simulations. 6. The algorithm must be parallelizable. 16 Random Bit/Number Generator RBG: a device or algorithm which outputs a sequence of statistically independent and unbiased binary digits
• Hardware-based – elapsed time between emission of particle during radioactive decay – thermal noise from a semiconductor diode or resistor; – the frequency instability of a free running oscillator; – air turbulence within disk drive which causes random fluctuations – drive sector read latency times – sound from a microphone or video input from a camera. • Software-based – the system clock – elapsed time between keystrokes or mouse movement – content of input/output buffers – user input – operating system values such as system load and network statistics
Adopted from slides of Introduction # 17 Pseudo Random Bit/Number Generator PRBG – Input: a seed i.e. a truly random input sequence of length k (the seed)
– Output: a deterministic sequence of length l >> k that “seems random”
An adversary cannot efficiently distinguish between sequences of PRBG and truly RBG of length l.
Introduction # 18 CP412 ADAII Introduction # 19 CP412 ADAII Introduction # 20 CP412 ADAII Introduction # 21 PRBG test
CP412 ADAII Introduction # 22 CP412 ADAII Introduction # 23 CP412 ADAII Introduction # 24 CP412 ADAII Introduction # 25 CP412 ADAII Linear congruential generator
• The simple way of generating pseudo-random numbers is by the linear congruential generator (LCG):
xi+1 = (a * xi + b) % m, i=0, 1, 2, …., ; where a, b an m are constant numbers, say a, b are large prime numbers m is 232 or 264
– The initial value of x0 is called the seed
LCG(a, b, m) or with mixed b>0, MLCG(a, m)
GGL: 31 xi+1 = (16807xi) mod (2 −1)
RAND: 32 32 xi+1 = ((69069xi+1) mod (2 ) or LCG(69069, 1, 2 )
27 RAND implementation #include
• 20000 random number pairs in two dimensions on a unit square
CP412 ADAII Introduction # 29 Lagged Fibonacci generators
• A lagged Fibonacci generator requires an initial set of elements
x1, x2, …. , xr and then uses
RAN3 is LF(55, 24, -) or The period is 255-1
30 Shift register generators
•
31 5. Global minimum cut
Global min cut. Given a connected/undirected graph G = (V, E), find a cut (A, B) of minimum cardinality.
Applications. to find the bottle neck of a network, partitioning items in a database, identify clusters of related documents, network reliability, network design, circuit design, TSP solvers.
Network flow solution. ・Replace every edge (u, v) with two antiparallel edges (u, v) and (v, u). ・Pick some vertex s and compute min s- v cut separating s from each other vertex v ∈ V.
Time: O(|V|3|E|2)
CP412 ADAII Introduction # 32 Contraction algorithm
Contraction algorithm. [Karger 1995] ・Pick an edge e = (u, v) uniformly at random. ・Contract edge e. - replace u and v by single new super-node w - preserve edges, updating endpoints of u and v to w - keep parallel edges, but delete self-loops ・Repeat until graph has just two nodes v1 and v2. ・Return the cut (all nodes that were contracted to form v1). Contraction algorithm
CP412 ADAII Introduction # 34 Contraction algorithm analysis
Theorem: the contraction algorithm returns a min cut with prob ≥ 2 / n2.
Pf. Consider a global min-cut (A*, B*) of G. Let F* be edges with one endpoint in A* and the other in B*. Let k = | F* | = size of min cut. In first step, algorithm contracts an edge in F* probability k / | E |. Every node has degree ≥ k since otherwise (A*, B*) would not be a min-cut ⇒ | E | ≥ ½ k n. Thus, algorithm contracts an edge in F* with probability ≤ 2 / n.
Let Ej = event that an edge in F* is not contracted in iteration j. Contraction algorithm Amplification. To amplify the probability of success, run the contraction algorithm many times.
Claim. If we repeat the contraction algorithm n2 ln n times, then the probability of failing to find the global min-cut is ≤ 1 / n2.
Pf. By independence, the probability of failure is at most Contraction algorithm: example execution Running time analyses
Remark. Overall running time is slow since we perform Θ(n2 log n) iterations and each takes Ω(m) time.
Improvement. [Karger-Stein 1996] O(n2 log3 n).
Early iterations are less risky than later ones: probability of contracting an edge in min cut hits 50% when n / √2 nodes remain. Run contraction algorithm until n / √2 nodes remain. Run contraction algorithm twice on resulting graph and return best of two cuts.
Extensions. Naturally generalizes to handle positive weights.
Best known. [Karger 2000] O(m log3 n).
CP412 ADAII Introduction # 38 6. Maximum 3-satisfiability
Maximum 3-satisfiability. Given a 3-SAT formula, find a truth assignment that satisfies as many clauses as possible.
Remark. NP-hard search problem.
Probabilistic algorithm: 1. Initialize each variable with false value. 2. Set value true with probability ½ independently for each variable. 3. Output the number of true clauses
Do multiple runs of the algorithm, choose the minimum of the outputs
CP412 ADAII Introduction # 39 Maximum 3-satisfiability: analysis
Claim. Given a 3-SAT formula with k clauses, the expected number of clauses satisfied by a random assignment is 7k / 8.
CP412 ADAII Introduction # 40 The Probabilistic Method
Corollary. For any instance of 3-SAT, there exists a truth assignment that satisfies at least a 7/8 fraction of all clauses.
Pf. Random variable is at least its expectation some of the time.
Probabilistic method. [Paul Erdös] Prove the existence of a non- obvious property by showing that a random construction produces it with positive probability!
CP412 ADAII Introduction # 41 7. Primality test
Primality test problem: determine whether an input number is prime. Complement problem (compositeness test): determine whether an input number is a composite number.
Fast deterministic tests • Naive method: for (i =2; i < sqrt(n); i++) if (n mod I ==0) return false; return true. • Fast algorithm: The AKS primality test runs in Õ((log n)7.5)
Probabilistic primality test • bounded-error probabilistic polynomial time (BBP) algorithm
CP412 ADAII Introduction # 42 Primality test
The basic structure of randomized primality tests is as follows
1. Randomly pick a number a.
2. Check some equality (corresponding to the chosen test) involving a and the given number n. If the equality fails to hold true, then n is a composite number, a is known as a witness for the compositeness, and the test stops.
3. Repeat from step 1 until the required accuracy is achieved.
After one or more iterations, if n is not found to be a composite number, then it can be declared probably prime.
CP412 ADAII Introduction # 43 BPP primality testing
Repeat k times: 1. Pick a in {1,...,n-1} at random. 2. If gcd(a, n) != 1, then output COMPOSITE. 3. If a^{(n-1)/2} is not congruent to +1 or -1 (mod n), then output COMPOSITE.
Now, if we ever got a "-1" above output "PROBABLY PRIME" else output "PROBABLY COMPOSITE".
Theorem: this procedure has error probability at most 1/2^k.
Proof: if n is really prime, then in each iteration we get a^{(n-1)/2} = -1 (mod n) with probability 1/2, so we get a -1 at least once with probability 1 - 1/2^k. If n is composite, the proof of success will follow from the lemma below with t = (n-1)/2.
CP412 ADAII Introduction # 44 Solovay-Strassen algorithm
1. Pick random a in {1,...,n-1} 2. if gcd(a,n) != 1, then COMPOSITE. 3. if [a/n] != a^{(n-1)/2} (mod n) then COMPOSITE else POSSIBLY PRIME.
Theorem: if n is composite, then this algorithm says "composite" with probability at least 1/2. Proof: the proof is much like that of our "key lemma". Let J = {a in Z_n^* : [a/n] = a^{(n-1)/2} (mod n)}.J is a group (e.g., use fact that [ab/n] = [a/n]*[b/n]) so it suffices to show there exists b not in J. By the assumption that n is not a prime power and is not a perfect square, we can write n = p1^e1 * r, where p1 and r are relatively prime, and e1 is odd. Let g be an arbitrary non-residue mod p1, and let b = (g,1), in CRT notation. We can see that b^{(n-1)/2} = (g^{(n-1)/2}, 1) is *not* congruent to -1(mod n), since -1 = (-1, -1). So, it suffices to show that [b/n] = -1.This in turn follows from the basic Jacobi symbol identities: [b/n] = ([b/p1])^e1 * [b/r] = ([g/p1])^e1 * [1/r] = (-1)^e1 * 1 = -1.QED CP412 ADAII Introduction # 45