original German edition by
Uwe Sch oning
translated revised and expanded by
Randall Pruim
Gems of
Theoretical Computer Science
Computer Science Monograph English
August
Springer Verlag
Berlin Heidelb erg New York
London Paris Tokyo
Hong Kong Barcelona Budap est
Preface to Original German Edition
In the summer semester of at Universit at Ulm I tried out a new typ e
of course which I called Theory lab as part of the computer science ma jor
program As in an exp erimental lab oratory with written preparatory ma
terials including exercises as well as materials for the actual meeting in
which an isolated research result is represented with its complete pro of and
all of its facets and false leads the students were supp osed to prove p or
tions of the results themselves or at least to attempt their own solutions The
goal was that the students understand and sense how theoretical research is
done To this end I assembled a numb er of outstanding results highlights
p earls gems from theoretical computer science and related elds in par
ticular those for which some surprising or creative new metho d of pro of was
employed Furthermore I chose several topics which don t represent a solu
tion to an op en problem but which seem in themselves to b e surprising or
unexp ected or place a well known problem in a new unusual context
This b o ok is based primarily on the preparatory materials and worksheets
which were prepared at that time for the students of my course and has
b een subsequently augmented with additional topics This b o ok is not a text
b o ok in the usual sense In a textb o ok one pays attention to breadth and
completeness within certain b ounds This comes however at the cost of
depth Therefore in a textb o ok one nds to o often following the statement of
a theorem the phrase The pro of of this theorem would go b eyond the scop e
of this b o ok and must therefore b e omitted It is precisely this that we do
not do here on the contrary we want to dig in to the pro ofs and hop efully
enjoy it The goal of this b o ok is not to reach an encyclop edic completeness
but to pursue the pleasure of completely understanding a complex pro of
with all of its clever insights It is obvious that in such a pursuit complete
treatment of the topics cannot p ossibly b e guaranteed and that the selection
of topics must necessarily b e sub jective The selected topics come from the
areas of computability logic computational complexity circuit theory and
algorithms
Where is the p otential reader for this b o ok to b e found I b elieve he or
she could b e an active computer scientist or an advanced student p erhaps
sp ecializing theoretical computer science who works through various topics
as an indep endent study attempting to crack the exercises and by this
VI
means learns the material on his or her own I could also easily imagine
p ortions of this b o ok b eing used as the basis of a seminar as well as to
provide a simpli ed intro duction into a p otential topic for a Diplomarbeit
p erhaps even for a Dissertation
A few words ab out the use of this b o ok A certain amount of basic knowl
edge is assumed in theoretical computer science automata languages com
putability complexity and for some topics probability and graph theory
similar to what my students encounter prior to the Vordiplom This is very
brie y recapitulated in the preliminary chapter The amount of knowledge as
sumed can vary greatly from topic to topic The topics can b e read and worked
through largely indep endently of each other so one can b egin with any of the
topics Within a topic there are only o ccasional references to other topics in
the b o ok these are clearly noted References to the literature mostly articles
from journals and conference pro ceedings are made throughout the text at
the place where they are cited The global bibliography includes b o oks which
were useful for me in preparing this text and which can b e recommended for
further study or greater depth The numerous exercises are to b e understo o d
as an integral part of the text and one should try to nd one s own solu
tion b efore lo oking up the solutions in the back of the b o ok However if one
initially wants to understand only the general outline of a result one could
skip over the solutions altogether at the rst reading Exercises which have a
somewhat higher level of di culty but are certainly still doable have b een
marked with
For pro of reading the original German text and for various suggestions
for improvement I want to thank Gerhard Buntro ck Volker Claus Uli Her
trampf Johannes K obler Christoph Meinel Rainer Schuler Thomas Thier
auf and Jacob o Tor an Christoph Karg prepared a preliminary version of
Chapter as part of a course pap er
Uwe Sch oning
Preface to the English Edition
While I was visiting Boston University during the academic year I
noticed a small b o ok written in German on a shelf in Steve Homer s o ce
Curious I b orrowed it for my train ride home and b egan reading one of the
chapters I liked the style and format of the b o ok so much that over the
course of the next few months I frequently found myself reaching for it and
working through one chapter or another This was my intro duction to Perlen
der Theoretischen Informatik
A few of my colleagues had also seen the b o ok They also found it inter
esting but most of them did not read German well enough to read more than
small p ortions of it enjoyably I hop e that the English version will rectify this
situation and that many will enjoy and learn from the English version as
much as I enjoyed the German version
The front matter of this b o ok says that it has b een translated revised
and expanded I should p erhaps say a few words ab out each of these tasks
In translating the b o ok I have tried as much as p ossible to retain the feel
of the original which is somewhat less formal and imp ersonal than a typical
text b o ok yet relatively concise I certainly hop e that the pleasure of the
pursuit of understanding has not gotten lost in the translation
Most of the revisions to the b o ok are quite minor Some bibliography
items have b een added or up dated a numb er of German sources have b een
deleted The layout has b een altered somewhat In particular references now
o ccur systematically at the end of each chapter and are often annotated This
format makes it easier to nd references to the literature while providing a
place to tie up lose ends summarize results and p oint out extensions Sp eci c
mention of the works cited at the end of each chapter is made informally if at
all in the course of the presentation Occasionally I have added or rearranged
a paragraph included an additional exercise or elab orated on a solution but
for the most part I have followed the original quite closely Where I sp otted
errors I have tried to x them I hop e I have corrected more than I have
intro duced
While translating and up dating this b o ok I b egan to consider adding
some additional gems of my own I am thankful to Uwe my colleagues
and Hermann Engesser the sup ervising editor at Springer Verlag for en
couraging me to do so In deciding which topics to add I asked myself two
VI I I
questions What is missing and What is new From the p ossible answers to
each question I picked two new topics
The intro duction to average case complexity presented in Topic seemed
to me to b e a completion more accurately a continuation of some of the
ideas from Topic where the term average case is used in a somewhat dif
ferent manner It was an obvious gap to ll
The chapter on quantum computation Topic covers material that is
for the most part newer than the original b o ok indeed several of the articles
used to prepare it have not yet app eared in print I considered covering Shor s
quantum factoring algorithm either instead or additionally but decided
that Grover s search algorithm provided a gentler intro duction to quantum
computation for those who are new to the sub ject I hop e interested readers
will nd Shor s algorithm easier to digest after having worked through the
results presented here No doubt there are many other eligible topics for this
b o ok but one must stop somewhere
For reading p ortions of the text and providing various suggestions for
improvement I want to thank Drue Coles Judy Goldsmith Fred Green
Steve Homer Steve Kautz Luc Longpr e Chris Pollett Marcus Schaefer
and Martin Strauss each of whom read one or more chapters I also want to
thank my wife Pennylyn Dykstra Pruim who in addition to putting up with
my long and sometimes o dd hours also pro ofread the manuscript her e orts
improved its style and reduced the numb er of typ ographical and grammatical
errors Finally many thanks go to Uwe Sch oning for writing the original b o ok
and collab orating on the English edition
Randall Pruim
July
Table of Contents
Fundamental De nitions and Results
The Priority Metho d
Hilb ert s Tenth Problem
The Equivalence Problem
for LOOP and LOOP Programs
The Second LBA Problem
LOGSPACE Random Walks on Graphs
and Universal Traversal Sequences
Exp onential Lower Bounds
for the Length of Resolution Pro ofs
Sp ectral Problems and Descriptive Complexity Theory
Kolmogorov Complexity the Universal Distribution
and Worst Case vs Average Case
Lower Bounds via Kolmogorov Complexity
PAC Learning and Occam s Razor
Lower Bounds for the Parity Function
The Parity Function Again
The Complexity of Craig Interp olants
Equivalence Problems and Lower Bounds
for Branching Programs
The Berman Hartmanis Conjecture and Sparse Sets
X Table of Contents
Collapsing Hierarchies
Probabilistic Algorithms
Probability Ampli cation
and the Recycling of Random Numb ers
The BP Op erator and Graph Isomorphism
The BP Op erator and the Power of Counting Classes
Interactive Pro ofs and Zero Knowledge
IP PSPACE
P 6 NP with probability
Sup erconcentrators and the Marriage Theorem
The Pebble Game
Average Case Complexity
Quantum Search Algorithms
Solutions
Bibliography
Index
Fundamental De nitions and Results
Before we b egin we want to review brie y the most imp ortant terms def
initions and results that are considered prerequisite for this b o ok More
information on these topics can b e found in the appropriate b o oks in the
bibliography
General
The set of natural numb ers including is denoted by N the integers by Z
and the reals by R The notation log will always b e used to indicate logarithms
base and ln to indicate logarithms base e
If is a nite non empty set sometimes referred to as the alphabet
then denotes the set of all nite strings sequences from including the
empty string which is denoted by A subset L of is called a formal
language over The complement of L with resp ect to some alphab et
is L L For a string x jxj denotes the length of x for a set
A jAj denotes the cardinality of A For any ordering of the alphab et the
lexicographical order on induced by the order on is the linear ordering
in which shorter strings precede longer ones and strings of the same length
are ordered in the usual lexicographic way For f g with this
ordering b egins
We assume that all nite ob jects graphs formulas algorithms etc that
o ccur in the de nition of a language have b een suitably enco ded as strings
over f g Such enco dings are denoted by h i
A polynomial p in variables x x is a function of the form
n
k
X
a a
a
i i
ni
x x x p x x
i n
n
i
where k a N For our purp oses the co e cients will usually also
ij i
b e integers sometimes natural numb ers The largest o ccurring exp onent
a with is the degree of the p olynomial p The total degree of the
ij i
Fundamental De nitions and Results
p olynomial p is the largest o ccurring value of the sum a a a
i i in
with A univariate p olynomial of degree d is uniquely determined by
i
sp ecifying d supp ort p oints x y x y x y with x x
d d
x Interp olation Theorem
d
Graph Theory
Graphs play a role in nearly all of our topics A graph is a structure G
V E consisting of a nite sets V of nodes and E of edges In the case of a
V
directed graph E V V and in the case of an undirected graph E
the set of all two element subsets of V A path from v to v is a sequence
of adjoining edges which b egins at v and ends at v If v v and the
path consists of at least one edge then the path is called a cycle A graph is
connected if there is a path from any no de in the graph to any other no de
in the graph acyclic if it contains no cycles and bipartite if the no de set V
can b e partitioned into V V V in such a way that V V and
every edge of the graph joins a no de in V with a no de in V The degree
of a no de is the numb er of other no des that are joined to it by an edge In
directed graphs we sp eak further of the in degree and out degree of a no de
A no de without predecessors in degree is called a source a no de without
successors out degree is called a sink
Two graphs G V E and G V E are isomorphic if there
is a bijective mapping V V which can b e extended in the obvious
way to edges such that e E e E We denote by G the
graph isomorphic to G that results from applying the p ermutation to the
no des and edges of G The set of automorphisms of a graph G is the set of
all isomorphisms b etween G and G i e the set of all p ermutations such
that G G
Bo olean Formulas and Circuits
n
A boolean function is a function f f g f g Bo olean functions can
b e represented as boolean formulas boolean circuits or branching programs
A b o olean formula is built up in the usual way from the symb ols and
the variables x and parentheses SAT denotes the decision problem of
i
determining for a given b o olean formula whether or not it is satis able that
is whether there is an assignment for the variables in the formula that causes
the formula to b e evaluated as true or
A b o olean circuit is a directed acyclic connected graph in which the in
put no des are lab eled with variables x and the internal no des have in degree
i
either or No des with in degree are lab eled with and no des with
in degree are lab eled with either or Each no de in the graph can b e
Fundamental De nitions and Results
asso ciated with a b o olean function in the obvious way by interpreting as
the NOT function as the AND function and as the NOT function The
function asso ciated with the output no de of the graph is the function com
puted by or represented by the circuit A formula can also b e understo o d
as a circuit in which every no de has out degree The size of a circuit or a
formula is the numb er of and symb ols that o ccur in it For some of
the topics we will consider families of b o olean circuits for which there is a
p olynomial p n such that the size of the nth circuit is b ounded by p n In
this case we will say that the family of functions has polynomial size circuits
For some of the topics we will also consider circuits in which the AND or
OR gates may have unb ounded fan in
Branching programs will b e intro duced in Topic
Quanti er Logic
Formulas with quanti ers o ccur in various contexts In order to evaluate a
formula of the form xF where F is a formula containing function and
predicate symb ols in addition to b o olean op erations one must rst x a
structure which consists of a domain and interpretations of all o ccurring
function and predicate symb ols over that domain The formula xF is valid
if there exists some element of the domain such that if all free o ccurrences
of x in F are replaced with that element then the resulting formula is valid
Formulas with universal quanti ers are evaluated analogously
In this b o ok the following variations o ccur in a quanti ed boolean formula
the domain for the variables is always considered to b e the truth values
and and no function or predicate symb ols are p ermitted other than the
b o olean op erations The problem QBF is the problem of determining for a
given quanti ed b o olean formula with no free variables if it is valid under
this interpretation
A predicate logic formula may have arbitrary function and predicate sym
b ols as well the equality symb ol When such a formula F is valid in a given
structure A then we write A j F In this case A is called a model for F
The formula F is satis able if it has at least one mo del It is a tautology if
for every suitable structure A A j F In this case we write j F
In an arithmetic formula only the two sp ecial functions symb ols and
the equality symb ol and the constant symb ols and are allowed Such
formulas are interpreted in the sp ecial xed structure with domain N and
and interpreted by the usual addition and multiplication in N
Probability
We will only consider nite or countable probability spaces so any subset
of can b e an event P r E is used to denote the probability of event E It
Fundamental De nitions and Results
is always the case that P r P r and P r E P r E
If E E are pairwise mutually exclusive events i e for any pair i and
P
j E E then P r E P r E The inclusion exclusion principle
i j i i i
holds for arbitrary events E E and states that
n
P P
P r E E P r E P r E
i j i i i
i j i
P
P r E E E P r E E
i j k n
i j k
In the equation ab ove the rst term on the right overestimates the correct
value the rst two underestimate it etc Thus the rst term can b e used as
an upp er b ound and the rst two terms as a lower b ound for P r E
i i
By conditional probability P r E jE read the probability of E given E
we mean the quotient P r E E P r E
A set of events fE E g is called pairwise independent if for every
n
i j f ng with i j P r E E P r E P r E and completely inde
i j i j
Q
P r E pendent if for every non empty set I f ng P r E
i i I i
i I
A random variable is a map from the set of events into the set R The
P
expected value of a random variable Z is E Z P r Z a a and its
a
variance is V Z E Z E Z E Z E Z The sum is over all
values a that the random variable Z takes on with non vanishing probabil
ity The exp ected value op erator is linear E aX bY aE X bE Y
Occasionally we will make use of various inequalities when approximating
probabilities
Markov s inequality If Z is a random variable that only takes on p ositive
values then P r Z a E Z a
Chebyshev s inequality P r jZ E Z j a V Z a
One frequently o ccurring probability distribution is the binomial distri
bution In a binomial distribution a random exp eriment with two p ossible
outcomes success and failure is carried out n times indep endently Let
p b e the probability of success in one of these trials and let X b e the random
variable that counts the numb er of successes Then
n
i n i
P r X i p p
i
E X np
V X np p
Computability
The set of partial computable functions over N or dep ending on the
context can b e de ned among other ways by means of Turing machines
One de nes a transition function or a transition relation if the Turing ma
chine is nondeterministic instead of deterministic on the set of con gurations
Fundamental De nitions and Results
of a Turing machine among which are the start con gurations which corre
sp ond uniquely to the p ossible values of the function argument x and end
con gurations from which one can derive the value of the function f x A
con guration is a complete description of the Turing machine at a given time
consisting of state head p osition s and contents of the work tap e s A
sequence of con gurations b eginning with the start con guration corresp ond
ing to input x such that each successive con guration is determined according
to the transition function of the machine is called a computation of the ma
chine on input x M denotes the ith Turing machine which corresp onds to
i
the ith partial computable function and the ith computably enumerable
i
language W L M A language L is computable or decidable if b oth the
i i
language and its complement are computably enumerable Equivalently L
computable if there is a Turing machine M such that L L M and M
halts on all inputs Well known undecidable problems languages include
the halting problem
H fhM xi j M halts on input xg
and the halting problem with empty tap e
H fhM i j M halts when started with an empty tap eg
Other mo dels of computation that are equivalent to the Turing machine
mo del include register machines also called GOTO programs WHILE
programs and recursive functions In each of these one can de ne simi
lar undecidable halting problems A further example of a language that is
undecidable but still computably enumerable is the set of all tautologies in
predicate logic On the other hand the set of all valid arithmetic formulas in
the structure N is not even computably enumerable One says that
arithmetic is not axiomatizable
A language A is many one reducible written A B if there is a
m
total computable function f such that for all x x A f x B The
language A is Turing reducible to B written A B if there is an oracle
T
B
Turing machine M that halts on all inputs and for which A L M where
B
L M denotes the language accepted by M using B as oracle i e as a
sub routine If A is reducible to B by either of these typ es of reducibilities
and B is decidable or computably enumerable then A is also decidable
computably enumerable resp ectively
Complexity Theory
A complexity class is formed by collecting together all languages that can
b e computed by Turing machines with similar restrictions on their resources
or structure The class P consists of all problems that can b e solved with
Fundamental De nitions and Results
deterministic Turing machines whose running time is b ounded by a p olyno
mial in the length of the input The class NP consists of all problems that
are accepted by a nondeterministic Turing machine for which the running
time in this case the depth of the computation tree is b ounded by a p oly
nomial in the length of the input More generally one can de ne the classes
DTIME f n and NTIME f n The di erence is that the running times
of the corresp onding Turing machines are no longer required to b e b ounded
by some p olynomial but instead must b e b ounded by some function that is
O f n The classes DSPACE f n and NSPACE f n can b e de ned anal
ogously in which case the amount of space used on the work tap e but not
the input tap e is b ounded instead of the running time PSPACE denotes the
class fDSPACE f n j f is a p olynomialg
We have the following inclusions
DSPACE log n NSPACE log n P NP PSPACE
Notice that at least the rst two classes are strictly contained in the last
Sometimes L and NL are used to denote DSPACE log n and NSPACE log n
If f and F are two functions such that F is time constructible and grows
signi cantly more rapidly than f for example if
f n log f n
lim
n
F n
then DTIME f n DTIME F n Analogous statements hold for DSPACE
NTIME and NSPACE
For a complexity class C coC denotes the set of all languages whose
complements are in C Some classes are known to b e closed under comple
mentation P coP PSPACE coPSPACE DTIME f n coDTIME f n
DSPACE f n coDSPACE f n and NSPACE f n coNSPACE f n
see Topic For other classes closure under complement is not known and
in fact doubtful NP coNP
A language L is called NP complete if L NP and for all A NP
P P
A L where is de ned analogously to with the di erence that
m
m m
the reduction function must b e computable in time that is b ounded by a
p olynomial in the length of the input The language SAT is NP complete
For every NP complete language L we have
L P P NP
The de nition of NP completeness can b e sensibly extended to other larger
complexity classes For example the language QBF is PSPACE complete Just
P P
as is a p olynomial time b ounded version of can b e de ned as the
m
m
T
P
p olynomial time b ounded version of Instead of A B we sometimes
T T
Fundamental De nitions and Results
B
write A P B or A P and say that A is computable in p olynomial time
relative to B If in this de nition we use a nondeterministic machine instead
B
of a deterministic one then we write A NP These notations can also b e
extended to classes of languages
C B C B
P P NP NP
B C B C
Algorithms and Programming Languages
For the representation of algorithms we use a notation that resembles pro
gramming language MODULA this language contains the usual assignment
of variables branching instructions of the form
IF THEN ELSE END
and the usual lo op constructs
FOR TO DO END
WHILE DO END
REPEAT UNTIL END
Occasionally we will use pro cedures esp ecially when describing recursive al
gorithms
The programs are typically to b e understo o d as informal descriptions of
Turing machines and from time to time we will expand the programming
language to include additional keywords that describ e op erations sp eci c to
Turing machines The instructions
INPUT
OUTPUT
express that the Turing machine is to read from its input tap e or write to its
output tap e The instructions
ACCEPT
REJECT
cause the Turing machine to halt in an accepting or rejection state
Nondeterministic machines guess a string from a nite set and assign
the guess to a program variable The variable then takes on any one of the
p ossible values For this we write
GUESS x S
where S is a nite set
Probabilistic randomized algorithms are similar to nondeterministic al
gorithms but the various p ossibilities of a guess are assigned probabilities
always according to the uniform distribution that is each p ossibility is
equally likely to b e guessed In this case we mo dify the instruction ab ove to
Fundamental De nitions and Results
GUESS RANDOMLY x S
After the execution of such an instruction for every s S it is the case that
P r x s jS j
The Priority Metho d
In the early years of computability theory Emil Post formulated a prob
lem which was rst solved twelve years later indep endently by a Russian
Muchnik and an American Friedb erg The solution intro duced a new
metho d the priority metho d which has proven to b e extremely useful in
computability theory
Let W W W b e an enumeration of all computably enumerable lan
guages Such an enumeration can b e obtained by enumerating all Turing
Machines M and setting W L M the language accepted by M In
i i i i
an analogous way we can also enumerate all oracle Turing Machines For
B B
any language B let W L M b e the ith language that is computably
i i
enumerable relative to B that is with B as oracle
A language A is Turing reducible to B written A B if there is
T
an oracle Turing Machine M that computes the characteristic function of A
relative to B i e with oracle B In particular this implies that M halts on
every input with oracle B
The de nition of can also b e written
T
B B
A W A B i A W and j
T
j i
Two languages A and B are said to b e Turing equivalent if A B and
T
B A in which case we write A B
T T
Post s Problem as it has come to b e known is the question of whether
there exist undecidable computably enumerable sets that are not Turing
equivalent to the halting problem In particular the answer is yes if there are
two computably enumerable languages A and B that are incomparable with
resp ect to i e such that A B and B A
T T T
If there are two such languages then neither one can b e decidable
Exercise Why C
nor can either one b e equivalent to the halting problem
Exercise Why C
Topic
So the picture lo oks like this
p p p p p p p p p
T equivalent to halting problem
p p p p p p p p p p p
computably enumerable languages
A B
p p p p p p p p p p p p p p p p p
decidable languages
It is not di cult to de ne languages by diagonalization so that A B
T
in fact so that A and B are incomparable with resp ect to One selects
T
for each i an x and arranges the de nitions of A and B such that x
i i
B B
The input x is said to i e x A x W A x W
i i i i
i i
A is not computably enumerable in B via W b e a witness for the fact that
i
In the known diagonalization constructions one can usually recognize easily
that the function i x is in fact computable This means that the witness
i
x can b e found e ectively in i
i
The problem with these constructions is that the languages A and B that
they pro duce are not computably enumerable In fact it can b e shown that
it is imp ossible to construct two computably enumerable languages A and B
for which A B and B A and such that the resp ective witnesses can
T T
b e found e ectively Thus some new metho d fundamentally di erent from
the usual diagonalization technique is needed to solve Post s Problem
Before solving Post s Problem however we want to show that the claim
just made is valid Notice rst that a computably enumerable language A
A is computably enumerable in B is computable in B if and only if
Therefore we make the following de nition a language A is e ectively not
Turing reducible to B if there is a total computable function f such that for
B B
that is f i A f i W A f i W all i f i
i i
The following claim is then true
Claim If A and B are computably enumerable languages and A is e ectively
not Turing reducible to B then B is computable
Exercise Why do es it follow from the claim that if there are any com
putably enumerable languages that are incomparable with resp ect to Turing
reducibility that this fact cannot b e demonstrated e ectively C
Proof of the claim Since B is computably enumerable it su ces to show
that the hyp othesis implies that B is computably enumerable For each z let
M b e the following oracle Turing machine z
The Priority Metho d
INPUT x
IF z ORACLE THEN REJECT
ELSE ACCEPT
END
The function
g z Co ding of Machine M
z
is clearly computable and furthermore
N if z B
B
W
g z
if z B
By hyp othesis there is a total computable function f such that
B
f n A f n W
n
Now consider f g z for arbitrary z We obtain
B
f g z A f g z W by choice of f
g z
B
N by choice of g W
g z
z B
B g f A Since A is computably enumerable it follows That is
B that B is computably enumerable ut from this representation of
Exercise Show that the preceding sentence is valid C
On the basis of this observation many researchers were of the opinion
that Post s Problem could not b e solved There is however a solution The
languages A and B must b e Turing incomparable in a certain non e ective
way The metho d by which this is p ossible is now referred to as the priority
method and was develop ed indep endently by Friedb erg USA and Muchnik
Russia in roughly years after Post originally p osed the problem
In at the Computational Complexity Conference in Santa Barbara
P Young considered these years as the p otential length of time needed to
nd a solution to the P NP problem Twelve years after its de nition in
the P NP problem unfortunately remained unsolved as it remains to day
Now we turn our attention to the solution of Post s Problem
Theorem There exist computably enumerable languages A and B that
are Turing incomparable
Proof We present an enumeration pro cedure that enumerates A and B si
multaneously thereby showing that b oth are computably enumerable The
enumeration pro ceeds in stages stage stage stage etc Each stage is
Topic
sub divided into two phases an A phase and a B phase In an A phase an
element of A and in a B phase an element of B can p otentially b e enu
merated that is in each phase one element may b e put into the appropriate
language In addition we maintain during the construction two lists L and
A
L the entries of which are all of the form i x In these pairs i is the index
B
of a Turing machine M and x is an input for which we will try to guarantee
i
B A
that x A x W x B x W resp ectively Each entry in these
i i
lists is either active or inactive Furthermore x x will always b e an
A B
input that is so large that it do es not a ect any of the preceding decisions
Step is used for initialization
Step Phase A and B
Let A B L L and x x
A B A B
The actual construction then pro ceeds as follows
Step n Phase A
L L f n x g n x is active
A A A A
FOR all active i x L in increasing order according to i DO
A
IF M on input x and Oracle B as constructed so far
i
accepts in n or fewer steps
THEN A A fxg Declare i x to b e inactive
Let y b e the largest oracle query in
the computation ab ove
x max x y
B B
j
FOR k y L k i DO
B
L L f k y g f k x j g active
B B B
j j
END
x x j
B B
GOTO Phase B
END
END
GOTO Phase B
Step n Phase B
L L f n x g active
B B B
FOR all active i x L in increasing order according to i DO
B
IF M on input x and Oracle A as constructed so far
i
accepts in at most n steps
THEN B B fxg Declare i x to b e inactive
The Priority Metho d
Let y b e the largest oracle query in
the computation ab ove
x max x y
A A
j
FOR k y L k i DO
A
L L f k y g f k x j g active
A A A
j j
END
x x j
A A
GOTO Step n
END
END
GOTO Step n
We claim that the languages A and B that are enumerated by the
preceding construction have the desired prop erties namely that A B and
T
B A or more precisely that for all i IN there exist x and x with
T
B A
x A x W and x B x W
i i
Notice that the entries i x in the lists L and L can change and
A B
that such changes resp ect the following priority ordering The entry
in List L has highest priority then in list L then in list L
A B A
then in list L etc This means that if at stage n of the construction
B
i x L and M accepts x with oracle B in at most n steps then x
A i
is enumerated into A To prevent subsequent changes to the oracle B that
might alter this computation all of the entries j in the list L that
B
have lower priority than i x are removed from the list and replaced by new
entries which are large enough that they do not a ect the computation On
B
the other hand it can happ en that a requirement x A x W that
i
at some p oint in the construction app ears satis ed is later injured This
happ ens if some x is enumerated into B for the sake of some entry i x
of higher priority i e for which i i It is precisely the determination of
these priorities that is carried out in section of the construction
What is imp ortant is that for each i the entry i in list L or L
A B
is changed only nitely many times nite injury This can b e proven by
induction on i
Exercise Carry out this induction Also determine how often at most
an entry i in L L resp ectively can change C
A B
A picture is useful in understanding how the construction pro ceeds One
can imagine the construction taking place on the following abacus
Topic
x
A
A
B
x
B
The active entries i n in the lists L and L are represented by balls
A B
lab eled with the numb er i and app earing in p osition n of the appropriate
row The arrows indicate the numb ers x and x resp ectively If the values
A B
of entries in L L change then this is caused by one of the balls lab eled
B A
i in the A row B row In this case all of the balls of lower priority that
is with larger index in the B row are slid to the right until they are b eyond
the arrow for x x After the balls have b een slid to the right the arrow
B A
x x is slid to the right b eyond all of the balls
B A
We know that the entry i x in L slides only nitely many times Let
A
i x b e the nal entry in L corresp onding to the lo cation in which ball i
A
comes to rest There are two cases x A or x A If x A then there
is a step n such that during the A phase of stage n x enters A Since i x
is the nal entry in L it is not p ossible that at some later stage anything
A
entered B for the sake of some i x of higher priority that might have
B
injured the requirement x A x W
i
B
since otherwise there would have b een a stage n If x A then x W
i
during which the THEN clause for x i would have b een op erative in which
case x would have b een enumerated into A So in this case it is also true that
B
x A x W
i
By a symmetric argument one can show that for all i there is a x with
A
ut x B x W
i
References
H Rogers Theory of Recursive Functions and E ective Computability
McGraw Hill Chapter
W S Brainerd L H Landweb er Theory of Computation Wiley
Section
R I Soare Recursively Enumerable Sets and Degrees Springer
Chapter VI I
Hilb ert s Tenth Problem
Hilb ert s Tenth Problem go es back to the year and concerns a fun
damental question namely whether there is an algorithmic metho d for
solving Diophantine equations The ultimate solution to this problem was
not achieved until The solution was however a negative one there
is no such algorithm
Among the famous op en problems p osed by the mathematician David
Hilb ert in was one problem numb er ten which p ossesses an esp ecially
close connection to computability theory although this connection was only
apparent later This problem deals with Diophantine equations named
after Diophantus of Alexandria rd century A D which are equations of
the form f x x where f is a p olynomial with integer co e cients
n
Required is a metho d nding integer solutions x x to such an equation
n
i e zero es of f in Z This task is equivalent to the task of nding given
two p olynomials f and g with integer co e cients solutions to the equation
f g since f g if and only if f g
Exercise For what values of a b Z f g do es the Diophantine equation
ax by
have integer solutions C
A p ositive solution to Hilb ert s tenth problem by which we mean the
existence of an algorithm to solve such equations would have had many im
p ortant consequences Many op en problems in Numb er Theory and also in
some other areas of mathematics can b e reformulated as Diophantine equa
tions
Hilb ert s tenth problem was solved in by Y V Matijasevi c after sig
ni cant progress had b een made by J Robinson H Putnam and M Davis
The solution was however of a much di erent sort than Hilb ert had prob
ably conjectured The problem is undecidable which means that there is no
algorithm that takes as input a Diophantine equation and correctly decides
whether it has an integer solution
This result represents a signi cant sharp ening of a variant of G odel s In
completeness Theorem which says that the problem of testing an arithmetic
Topic
formula for validity is undecidable Arithmetic formulas are formulas that are
built up from the basic op erators b o olean op erations constants and
variables over the integers as well as existential and universal quanti ers
Example of an arithmetic formula
x z u u v x x u z u x
Diophantine equations can b e seen as a sp ecial case of arithmetic formulas
namely those in which negation and universal quanti ers do not o ccur That
is all variables that o ccur are existentially quanti ed The following two
exercises show how to eliminate the logical connectives AND and OR
Exercise Show that the problem of simultaneously solving a system of
Diophantine equations
f x x and
n
f x x and
n
f x x
k n
can b e reduced to the case of a single Diophantine equation C
Exercise The last exercise demonstrated in a certain sense the existence
of an AND function for the solvability problem for Diophantine equations
Show that there is a corresp onding OR function that is show that the prob
lem of solving
f x x or
n
f x x or
n
f x x
k n
can b e reduced to the solution of a single Diophantine equation C
Exercise Let Dioph Z denote the decision problem of determining for
a given Diophantine equation whether it has solutions in Z i e precisely the
problem we have b een discussing up to this p oint Analogously let Dioph N
denote the problem of determining if there are solutions in N Note that there
are equations G that are in Dioph Z but not it Dioph N
Show Dioph Z Dioph N
T
Show Dioph Z Dioph N C
m
Exercise Show Dioph N Dioph Z m
Hilb ert s Tenth Problem
Hint By a theorem of Lagrange every natural numb er can b e ex
pressed as a sum of four squares C
Now we turn our attention to the pro of of undecidability This will b e
done by rst reducing the halting problem for register machines to the prob
lem ExpDioph N The problem ExpDioph N di ers from Dioph N in that
y
exp onential terms of the form x where x and y are variables are also al
lowed
The second reduction that we need the reduction from ExpDioph N to
Dioph N is a gem of numb er theory but has little to do with theoretical
computer science For this reason we omit the pro of See the references for
places to nd a pro of
For the following it will b e imp ortant that the so called dominance rela
tion a partial order on N which we will denote by E can b e expressed using
exp onential Diophantine equations The relation x E y means that all the
bits of the binary representation of x are less than or equal to the corresp ond
ing bits in the binary representation of y That is if x has a in a certain
p osition the y must have a there to o We p ostp one for the moment the
problem of expressing E by means of Diophantine equations and simply use
the dominance relation in what follows
For the undecidability pro of we use the following mo del for register ma
chines A register machine has a nite numb er of registers R R which
k
can hold arbitrarily large natural numb ers A register program consists of a
sequence of consecutively numb ered instructions
A
A
m A
m
The p ossible instructions are
INC R resp ectively DEC R
j j
Increases decreases the value in register R by one Negative register
j
values are not allowed
GOTO l
Unconditional jump instruction Jumps to instruction l the computation
pro ceeds from there
IF R GOTO l
j
Conditional jump instruction Jumps to instruction l if the value in register
R is
j
HALT
Ends the program
One might wonder ab out the missing instructions R or R
j j
R which many authors allow in their GOTO programs but it is easy to n
Topic
convince oneself that these instructions can b e simulated by corresp onding
programs that use a kind of counting lo op
Exercise Show that the instructions R and R R can b e
j j n
simulated using the given set of instructions C
For the following pro of however it turns out to b e a signi cant simpli
cation if only the set of instructions presented ab ove is available In addition
it is easy to see that without loss of generality we can assume
The last instruction of the program A is always the HALT instruction
m
and that it is the only HALT instruction in the program
Whenever the program stops due to the HALT instruction all of the reg
isters will have b een previously set to
A DEC R instruction is never executed while the value of register j is
j
This can b e avoided using conditional jumps
Despite these various restrictions this mo del of computation is computa
tionally universal Thus the halting problem H for programs b egun with all
registers initialized to is undecidable
H fP j P is a program for a register machine and this ma
chine stops when it is started with all registers ini
tialized to g
We will reduce this problem to ExpDioph N by giving for each program
P a set of exp onential Diophantine equations which have a simultaneous
solution if and only if P H Using the metho ds discussed in Exercises
and this set of equations can then b e transformed into a single equation
So let P b e a program of the form
A
A
m A
m
and let the registers that are addressed in the program b e R R We
k
assume without loss of generality that the previously mentioned restrictions
are satis ed
The system of Diophantine equations will contain a numb er of variables
the intended meaning of which we will give rst For the sake of readability we
will use capital letters for all variables Certain of the variables are supp osed
to represent nite sequences of natural numb ers n n n We represent
s
such a sequence with the single numb er
Hilb ert s Tenth Problem
s
X
i
n B
i
i
where B is a su ciently large base numb er also a variable Su ciently
large will mean that in arithmetic op erations we will b e using for example
adding two such sequence numb ers we will never need to carry In addition
B will b e a p ower of This will allow us to use the dominance relation
to control individual bits of the binary representation of such a sequence
numb er
Now we describ e the most imp ortant variables and their intended mean
ings All of these variables are understo o d to b e existentially quanti ed
B the base numb er describ ed ab ove
S the numb er of steps in the computation i e the
numb er of instructions executed until the HALT in
struction is reached The numb er S is the length of
the sequences co ded in the following variables
W j k a sequence numb er for each register which repre
j
sents the contents of that register at each step
s in the computation where s is the value
of S
N i m a sequence numb er for each instruction numb er
i
which represents for each step s whether the
instruction was or was not executed at
that step
Example Supp ose B S and the register R takes on the values
as the computation runs Then W If the rst instruc
tion is executed at steps and then N co des the sequence
so N
Now we can give the required Diophantine equations that describ e the
halting problem All the equations are conjunctively connected so we can
use the previous exercises to transform them into a single equation
First we have a requirement on the base numb er B namely B must b e
a p ower of
K
B
K is an additional auxiliary variable In addition B must b e su ciently
large
B k B m B S
The last condition implies that B will b e more than twice as large as any
register value that can b e computed in S steps We will need this fact later
These are not Diophantine equations since we have made use of the less than
Topic
symb ol but by intro ducing another auxiliary variable Z an expression like
X Y can b e easily expressed as an equation namely X Z Y
Another metho d would have b een to de ne B large enough from the start
k m S
say B
The next equations establish certain b oundary conditions for example
that the sequence numb ers N consist solely of and comp onents For this
i
and also in the following it is convenient to have available a variable T
P
s
i
that consists solely of s that is T B T can b e sp eci ed with the
i
equation
s
B T B
The condition on the variables N can now b e formulated as
i
N E T i m
i
Since exactly one instruction is executed at each step of the computation we
have the condition
m
X
N T
i
i
The next equations establish the start and end conditions for the register
machine computation The equation
E N
forces the execution of the program to b egin with the rst instruction The
last instruction must b e the HALT instruction which we are assuming is
instruction m
s
B E N
m
Furthermore initially all registers must b e set to
s
W E B B j k
j
Now we come to the signi cant equations namely those which guarantee
the correct transition b ehavior from one time step to the next For each
instruction of the form i GOTO j we intro duce an equation of the form
B N E N
i j
The multiplication by B causes the s in N which indicate the steps at
i
which instruction i is executed to b e moved over one p osition thus forcing
instruction j to b e executed in step s whenever instruction i is executed
at step s
In instructions of the forms i INC R and i DEC R there is also
j j
a hidden GOTO instruction namely the jump to instruction i So in
these cases we also intro duce
Hilb ert s Tenth Problem
B N E N
i i
The actual function of INC and DEC instructions can b e simulated with
X X
j k W B W N N
j j i i
where the rst sum is over all i for which there is an instruction in the
program of the form i INC R and the second sum is over all i for which
j
there is an instruction in the program of the form i DEC R Once again
j
the factor of B causes the e ect of the instructions to take e ect at the next
time step
The only remaining instructions are the conditional jumps An instruction
of the form i IF R GOTO l implies that execution continues either
j
with instruction l if R or with instruction i So rst we intro duce
j
the equation
B N E N N
i l i
which forces that the next instruction can only b e instruction l or instruc
tion i To test the condition R we use
j
B N E N B T W
i i j
Exercise Explain how this equation works C
All that remains is to show that the dominance relation E can b e ex
pressed using exp onential Diophantine equations For this a theorem of Kum
mer and Lucas is helpful
y
Theorem x E y if and only if is odd ut
x
Exercise Prove Theorem
y y y y y
n n
mo d where x x Hint mo d
n
x x x x x
n n
and y y are the binary representations of x and y C
n
Since the prop erty o dd can b e easily expressed as a Diophantine equa
tion z is o dd if and only if z n for some n it only remains to show
that binomial co e cients can b e expressed using exp onential Diophantine
equations For this we make use of the Binomial Theorem
n
X
n
i n
u u
i
i
n n
Provided u this implies that is just the ith co e cient in the u adic
i i
n
n n n
representation of u Since it is su cient to cho ose u
i
So we can write
Topic
n
n k
m u v w u v u m u and
k
n k k
u w u mu v
This completes the solution to Hilb ert s Tenth Problem ut
The next two exercises present some variations on Hilb ert s Tenth Prob
lem that remain undecidable
Exercise Show that the undecidability of Hilb ert s Tenth Problem i e
the undecidability of Dioph N or Dioph Z implies the undecidability of
the following problem
Given two n variate p olynomials f and g with positive co e
cients determine whether it is the case that f x g x for
n
al l x N C
Exercise Show that Hilb ert s Tenth Problem is already undecidable
if one restricts oneself to p olynomial equations of the form f x x
n
where the total degree of f i e the degree of f x x x is at most four
Hint As in the reduction of SAT to SAT see Garey and Johnson one
intro duces for each sub p olynomial of f a new variable and then expresses
the prop erty that f through the conjunction of a set conditions of the
form f f Each p olynomial f has total degree at most two
k i
and the equation f expresses that the ith new variable has the desired
i
value C
References
For the p ortions of the pro of of Hilb ert s Tenth Problem the following liter
ature was helpful
L Adleman K Manders Diophantine complexity Symposium on Foun
dations of Computing IEEE
J L Bell M Machover A Course in Mathematical Logic North Holland
M Davis Unsolvable Problems in J Barwise ed Handbook of Mathe
matical Logic North Holland
P van Emde Boas Domino es are forever Technical Rep ort Dept
of Mathematics University of Amsterdam
R W Floyd R Beigel The Language of Machines An Introduction to
Computability and Formal Languages Computer Science Press
Exercise app ears here
Hilb ert s Tenth Problem
J P Jones Y V Matijasevi c Pro of of recursive unsolvability of Hilb ert s
tenth problem American Mathematical Monthly Oct
G Rozenb erg A Salomaa Cornerstones of Undecidability Prentice
Hall
The reduction of ExpDioph N to Dioph N can b e found in the article by
Jones and Matijasevi c and also in
M Davis Computability and Unsolvability Dover
Exercise is from
M Hack The equality problem for vector addition systems is undecid
able Theoretical Computer Science
The article ab ove also presents further applications of the undecidability re
sult to questions ab out Petri nets
Topic
The Equivalence Problem
for LOOP and LOOP Programs
In the s b efore the b o om in complexity theory several subrecur
sive classes of functions and languages were investigated extensively One
such hierarchy of functions contained in the primitive recursive functions
considers the depth of nesting of FOR lo ops It turns out that there is
a decided di erence in the complexity of the equivalence problems for
LOOP and LOOP programs the former is co NP complete but the
latter is undecidable
LOOP programs form a very restricted class of programs for manipulating
numb ers in N including which are stored in registers Registers in this
mo del can hold arbitrarily large integers The syntax of LOOP programs
is de ned inductively If X and Y are names of registers then X Y
X X and X are LOOP programs Furthermore if P and Q are
LOOP programs then P Q is a LOOP program as is LOOP X DO P END
Regarding the semantics of LOOP programs that is the manner in which
they are to b e executed the following should b e said Assignment statements
are executed in the obvious way so that the register values are changed in
the appropriate fashion A LOOP program of the form P Q is executed by
executing the program P rst and then with the values P leaves in the
registers remaining intact executing program Q And a program of the form
LOOP X DO P END executes program P as many times as the value of X at
the beginning of the loop
In addition to sp ecifying the LOOP program itself it is necessary to sp ec
ify which of the registers which we will also refer to as variables are to b e
understo o d as the input registers and which as output registers Typically
there is only one output register
n m
The function f N N computed by a LOOP program with n input
registers and m output registers is de ned as follows f a a with
n
a N is the vector of values in the output registers of the machine after the
i
program is run with a in the ith input register and in all other registers
i
at the start of execution It is known that the functions that are computable
by LOOP programs are precisely the primitive recursive functions
Topic
Examples Addition in the sense of the assignment Z X Y can b e
done via
Z Y
LOOP X DO Z Z END
where Z is the output register and X and Y are input registers Multiplication
can b e done with the following program
Z
LOOP X DO
LOOP Y DO
Z Z
END
END
The lo op depth of a LOOP program is the maximum o ccurring depth
of nesting of the for lo ops in the program In our examples the addition
program has lo op depth and the multiplication program has lo op depth
Exercise De ne lo op depth inductively on the formation of LOOP
programs C
We call a LOOP program with lo op depth n a LOOP n program
Exercise Subtraction was not included in our set of instructions for
LOOP programs Show that the instruction X X can b e simulated
by a LOOP program Note x y is de ned by
x y if x y
x y
if x y
C
Exercise Show that the instruction IF X THEN P END can b e
simulated by a LOOP program C
Exercise Show that if k is a constant then the instruction IF X k
THEN P END can b e simulated by a LOOP program C
The equivalence problem for programs of a certain typ e is the problem
of determining for two given programs if they compute the same function
The equivalence problem for Turing machines is easily seen to b e undecidable
since it is at least as di cult as the halting problem On the other hand the
equivalence problem for nite automata is decidable The question then is
this With resp ect to equivalence problems where exactly is the b oundary
b etween undecidability and decidability We want to investigate this question
using LOOP programs
LOOP and LOOP Programs
The result will b e this The equivalence problem for LOOP programs
is decidable but for LOOP programs it is undecidable and hence also
undecidable for LOOP n programs where n
We will b egin with the undecidability result The halting problem for
Turing machines is undecidable GOTO programs are equivalent to Turing
machines so the halting problem for GOTO programs is also undecidable
In a GOTO program every line of the program is numb ered consecutively
b eginning with and contains one of the following instructions
X Y
X X
X
GOTO i
IF X k THEN GOTO i
HALT
The semantics of each is self explanatory
We will fo cus on the sp ecial version of the halting problem for GOTO
programs where the programs are run on themselves Let P P P b e
a systematic enumeration of all GOTO programs with one input register and
one output register Then the language
K fn j P n haltsg
n
is undecidable This problem can b e reduced to a suitable version of the
equivalence problem for LOOP programs as follows
n K P n halts
n
s P n halts in at most s steps
n
s A s B s
n
A B
n
Here B is a xed LOOP program that indep endent of its input always
outputs and A is a LOOP program that computes the following function
n
if P n halts in at most s steps
n
A s
n
otherwise
We will see that A b elongs to LOOP From this it follows immediately
n
that the equivalence problem for LOOP programs is undecidable
What remains then is to give a construction that for each n i e for each
GOTO program P e ectively pro duces a LOOP program A with the
n n
desired prop erty As a rst step we de ne a LOOP program D that can n
Topic
simulate individual transitions from one con guration of P to the next We
n
represent a con guration as a vector a x y z z where a is the num
k
b er of the instruction ab out to b e executed or if the program has halted
x is the value stored in the input register X y the value stored in the out
put register Y and z z the values stored in the remaining registers
k
Z Z used by P The desired program D works with the registers A
k n n
X Y Z Z and p erhaps others to represent such con gurations That
k
k k
is D computes a function f N N where f a x y z z is
n k
the successor con guration of a x y z z
k
Exercise If one is given program D that b ehaves as describ ed how can
n
one build the desired program A
n
Program A must have lo op depth What lo op depth must D have for
n n
this to b e the case C
Now we construct D Let the instructions of the GOTO program P b e
n n
numb ered through r Then D lo oks roughly like
n
IF A THEN END
IF A THEN END
IF A r THEN END
How the ellipses are lled in dep ends on the particular instructions of P
n
Assignment statements X Y X X and X can b e carried over
directly with the addition of A A The instruction HALT yields A
For GOTO i we write A i For IF X k THEN GOTO i we write at rst
IF X k THEN A i ELSE A A
Note that by using conjunction we can program out any nesting of IF
statements that o ccur so that we remain in LOOP For example IF B
THEN IF B THEN P END END is equivalent to IF B AND B THEN P END can
b e simulated by
Z B
Z B
Z Z Z
Z Z
LOOP Z DO P END
The instructions in quotation marks indicate LOOP programs see the
previous exercises
LOOP and LOOP Programs
Altogether D is a LOOP program and A is a LOOP program
n n
so we have
Theorem The equivalence problem for LOOP programs is undecid
able ut
Now we turn our attention to LOOP programs In this case we shall
see that the equivalence problem is decidable in fact it is coNP complete
For this we need to characterize the LOOP programs precisely by giving
a certain normal form for LOOP programs so that we can decide if two
LOOP programs are equivalent by considering only their normal forms
m
De nition A function f N N m is cal led simple if it can
be built from the fol lowing components using composition
s x x
n
z x x
n
n
u x x x
n i
i
x x
x k
x x
w x x
x
x DIV k
x MOD k
where k N is an arbitrary constant
We want to prove the following theorem
Theorem A function is LOOP computable if and only if it is simple
For the direction from right to left we observe rst that functions
ab ove are all clearly LOOP computable
Exercise Show that the function w in item is LOOP computable
C
Exercise Show that for every k N the functions x DIV k and
x MOD k are LOOP computable
Hint Note that k is a constant and not the value of an input register This
means that the numb er of registers used to compute x MOD k or x DIV k
can dep end on k C
Now we must show the converse every LOOP computable function
is simple Let P b e a LOOP program Then P has the form P
P P P The function computed by P is the comp osition of the func
m
tions computed by P P P in various registers It is su cient to show
m
that each P computes a simple function in its registers i
Topic
If P is an assignment statement this is clear Now consider a for lo op of
i
the form
P LOOP X DO Q END
i
where Q is a LOOP program
Exercise Show that Q can only compute functions of the form
x x x k or x x k where k is a constant C
n j n
So the e ect of the LOOP program can b e describ ed with the following
equations
k Z y
i
k Z y
i
k Z y
m m m i
m
where for each i Z is a program register f g k N and y is the
i i i i
value of the register Z b efore execution the for lo op
i
Now we can de ne a directed graph corresp onding to this representation
The set of vertices is fZ Z g and the edges are
m
E f Z Z j and i j or k g
i j j j
That is there is an edge from Z to Z if the variable Z is assigned the value
q j j
of Z p erhaps increased by some constant k
q j
Example The equations b elow
Z y
Z y
Z y
Z y
Z y
Z y
Z y
Z y
Z
LOOP and LOOP Programs
give rise to the following graph
Z Z Z
Z Z Z
Z Z Z
Isolated variables i e variables that do not o ccur within a for lo op or
that are indep endent of lo ops are easily handled In the example ab ove the
for lo op computes in variables Z and Z the following functions
Z y
Z IF x THEN y ELSE
These functions are clearly simple see Exercise and this example can
b e immediately generalized
Exercise Show how functions of the form
IF x THEN f ELSE g
can b e expressed using the functions w and C
The remaining variables are either part of exactly one cycle or dep end on
exactly one cycle since each vertex has at most one predecessor
Now we intro duce lab els on the edges of the graph These corresp ond
exactly to the k s Note that Z and Z can b e considered taken care of and
i
omitted from the graph altogether
Topic
Z Z Z
Z Z Z
Z
Supp ose that variable Z is part of a cycle of length t and that if we
start from Z and traverse the cycle backwards then the edge lab els are
k k k Let M k k If X the register that determines the
t t
numb er of times the lo op is executed contains a multiple of t i e x n t for
some n then the function computed in register Z is M n n n n
M times If x n t l for some l t then Z M n k k
l
The numb ers n and l are just x DIV t and x MOD t so the function that is
computed in this case is
Z IF x THEN z ELSE M x DIV t k k
x MOD t
which is simple since the numb ers M t and k are constants
i
Exercise The argument for the simplicity of this function is still lacking
with regards to the sub function k k Fill in the missing
x MOD t
details C
Now consider the case of a variable Z that dep ends on a cycle but is not
itself in the cycle If this variable is i steps from a cycle variable Z then the
function computed in register Z is equal to the function computed in register
Z but on input x i plus a constant which can b e read o of the path
from Z to Z So such a function is also simple This concludes the pro of of
Theorem ut
Now that we have describ ed an e ective pro cedure for obtaining a simple
function from a LOOP program the equivalence problem for LOOP
programs is reduced to the equivalence problem for simple functions A simple
function is sp eci ed by describing in a systematic way how it is built up from
the basic functions This could b e done for example by giving an expression
LOOP and LOOP Programs
tree with leaves given lab els corresp onding to the basic functions or by
providing a su ciently parenthesized expression
We now want to investigate to what extent a simple function is uniquely
determined by sp ecifying its values at certain supp ort p oints For exam
ple a p olynomial in one variable of degree d is completely determined by
sp ecifying its value at any d p oints This fact will also b e imp ortant in
Topic In the case of simple functions however the situation is a bit more
complicated
n
First we intro duce a relation on N i e on the set of p otential input val
ues for a LOOP program Let M K N We say that the tuple x x
n
is M K comparable with x x if for i n we have
n
x M or x M x x and
i i
i i
x M and x M x x mo d K
i i
i i
M K M K
We denote this by x x x x and note that is an
n
n
n
equivalence relation on N Furthermore we observe that
if M M then
M K M K
x x x x x x x x
n n
n n
and if K is a multiple of K then
M K M K
x x x x x x x x
n n
n n
M K
Exercise Determine the index of the equivalence relation that is
the numb er of distinct equivalence classes C
Lemma Every simple function f can be assigned numbers M and K
M K
such that f is a linear combination on equivalence classes of i e
n
X
x f x x
i i n
i
where each is a rational constant that depends only on the equivalence
i
class
Proof The pro of pro ceeds by structural induction on the simple function
The argument for the base case is contained in the argument for the inductive
step so we will fo cus on the inductive step from the start
Let f b e a simple function that has already b een shown to satisfy the
statement of the lemma with constants M and K Then the statement
of the lemma is true also of f x x with the same constants M
n
and K
Topic
n
Up on applying the function z the lemma holds with M and
K
If f f are simple functions that satisfy the lemma with the con
m
m
stants M K M K then the function u f x f x sat
m m m
i
is es the lemma with the constants M K
i i
If the functions f and f satisfy the lemma with the constants M K
M and K then f x f x satis es the lemma with the constants
max M M and K K
If the function f satis es the lemma with the constants M and K then
the function f x satis es the lemma with M K and K
If the functions f and f satisfy the lemma with constants M K M
and K then the function w f x f x satis es the lemma with con
stants max M M K and K K
If the function f satis es the lemma with constants M and K then
f x DIV k satis es the lemma with the constants M and k K
If the function f satis es the lemma with constants M and K then
f x MOD k satis es the lemma with the constants M and k K ut
Exercise Fill in the details of the pro of ab ove by determining the
constants in cases and C
i
Now the decidability result is at hand Simple functions can b e completely
sp eci ed by giving the appropriate values of M and K and for each of the
nitely many equivalence classes the constants Therefore they can also
i
b e compared on the basis of such sp eci cations But there is an even simpler
way to compare simple functions
Exercise Show that a simple function with constants K and M is
completely sp eci ed by giving in addition to K and M the nitely many
function values ff x g where Q f x x j for i n x
x Q n i
M K g C
From this our theorem follows immediately
Theorem The equivalence problem for LOOP programs is decidable
Proof First one determines the constants M K and M K for the two
programs then one checks to see if the functions agree on the input values x
that have x max M M K K for all i ut
i
Exercise Justify the claims made in the previous pro of C
Now we want to determine more exactly the complexity of the equiv
alence problem for LOOP problems We shall see that this problem is
coNP complete which is the same as saying that the inequivalence problem
is NP complete Most of the work in showing that this problem b elongs to
LOOP and LOOP Programs
NP has already b een done The required nondeterministic algorithm given
LOOP programs as input rst determines the constants K M and
K M for the two programs then nondeterministically guesses an input
vector x with x max M M K K for all i and nally checks to see
i
that P x P x
To verify that this whole pro cedure is in NP we must take a closer lo ok
at p ortions of the computation The e ective pro cedure that pro duces for a
given LOOP program its asso ciated simple function can clearly b e carried
out in deterministic p olynomial time An insp ection of the pro of of Lemma
shows that the constants M and K that are assigned to a simple function can
b e chosen in such a way that K is the pro duct of all the constants previously
o ccuring in DIV and MOD functions An empty pro duct is considered to
have the value The value of M can b e determined by counting the o ccur
rences of w and functions that o ccur if there are m such o ccurrences
then we can cho ose M to b e M m K Notice that the numb er of bits in the
binary representations of M and K is p olynomial in the length of the origi
nal LOOP program so we can guess x in nondeterministic p olynomial time
Finally we need to evaluate the LOOP programs on the input x A step
by step simulation of the LOOP program however do es not work since the
simulation time could b e linear in the values of the inputs which would b e
exp onential in the lengths of their binary representations It is more e cient
to make use of the representations as simple functions since the functions
w DIV and MOD can b e evaluated in p olynomial time
Theorem The inequivalence problem for LOOP programs is NP
complete
Proof It remains to show that SAT can b e reduced to this problem Let F
b e a b o olean formula with n b o olean variables We construct two LOOP
programs such that the rst one do es nothing but output The second
program interprets the input x in such a way that x means FALSE
i i
and x means TRUE Under this interpretation the program evaluates
i
F with the given assignment and outputs if the assignment makes the
formula true and otherwise Clearly the problems are inequivalent if and
only if there is a satisfying assignment for F i e F SAT ut
Exercise Construct the second LOOP program mentioned ab ove
Hint It is su cient to show how to simulate the logical NOR function with
a LOOP program C
Exercise Show that the inequivalence problem for LOOP programs
is already NP complete if it is restricted to include only programs with one
input variable C
Topic
References
V Claus The equivalence problem of lo op programs Rep ort No
Abteilung fur Informatik Universit at Dortmund
A R Meyer D M Ritchie Computation complexity and program struc
ture IBM Research Rep ort RC
D Tsichritzis The equivalence of simple programs Journal of the ACM
The Second LBA Problem
The solution of the so called second LBA problem came unexp ectedly in
and was discovered indep endently by an American Immerman and
a Slovakian Szelep cs enyi Among other things this result says that the
class of context sensitive languages is closed under complementation
The two LBA problems were p osed in by S Y Kuro da The meaning of
these problems the formal de nitions will b e given shortly were rep eatedly
brought up and discussed see for example the article by Hartmanis and
Hunt In the time must have b een rip e the second LBA problem
was solved completely indep endently by an American researcher N Immer
man and a Slovakian student R Szelep cs enyi The amazing part of these
pro ofs is this they are considerable easier than one would have exp ected of a
problem that had remained unsolved for years Furthermore the solution
is precisely the opp osite of the conjecture that was widely held prior to the
pro of
What are the LBA problems Kuro da showed in that the class of lan
guages that are recognized by nondeterministic linear space b ounded Turing
machines LBAs linear b ounded automata is the same as the class of con
text sensitive languages This result is often presented in an undergraduate
course on formal languages In mo dern terminology this result says that
NSPACE n CSL
where CSL is the class of context sensitive languages The rst LBA prob
lem is the question of whether deterministic and nondeterministic LBAs are
equivalent
NSPACE n DSPACE n
The result that comes closest to solving the rst LBA problem is Savitch s
Theorem which says that
NSPACE s n DSPACE s n
for all s n log n So in particular
NSPACE n DSPACE n
Topic
The second LBA problem is the question of whether the class of languages
accepted by nondeterministic LBAs is closed under complement
NSPACE n coNSPACE n
A negative answer to the second LBA problem implies of course a negative
answer to the rst since DSPACE n is closed under complement But from a
p ositive solution to the second LBA problem there is no direct consequence
regarding the rst LBA problem
The second LBA problem has now b een solved the rst LBA problem
remains op en NSPACE n is contrary to the previously generally b elieved
conjecture closed under complementation This solution to the second LBA
problem is actually an instance of a more general result From the pro of it
follows immediately that
NSPACE s n coNSPACE s n
whenever s n log n
Although the pro of is actually relatively easy at least in the case of Im
merman it app ears at the end of a sequence of results which say that certain
hierarchies de ned in terms of the class NSPACE n collapse In all of
these results a certain counting technique is employed which can b e used
to complement classes An overview of these techniques was presented in an
article by U Sch oning
Exercise Supp ose that memb ership in a language A can b e determined
by some nondeterministic algorithm M with certain resource b ounds which
do not interest us at the moment Furthermore supp ose that the numb er
of strings in A of a given length is given by an easily computed function
f N N
Under these assumptions give a nondeterministic algorithm for A C
Exercise Now supp ose that in the previous exercise the the algorithm
M has nondeterministic time complexity t n and space complexity s n
M M
and the the computation of f requires t n time and s n space Determine
f f
A in upp er b ounds for the time and space complexity of the algorithm for
the previous exercise C
Now supp ose that A CSL NSPACE n If f n can also b e com
A NSPACE O n puted in linear space then by the observations ab ove
NSPACE n So to nish the pro of we need to show how to compute f in
linear space
Exercise For the computation of f in linear space we will need to make
use of nondeterminism That is we should b e thinking of a nondeterministic
linear space b ounded Turing machine that computes f
The Second LBA Problem
For such nondeterministic machines there are various imaginable mo dels
of what it means for them to compute functions e g single valued multi
valued Find a de nition that is su cient to work correctly in this context
C
Since A is context sensitive there is a corresp onding grammar G with
variable set V and terminal alphab et that generates A For all n i N
n
de ne the subset T of V as follows
i
i
n
T fx jxj n S xg
G
i
i
where S x means that x can b e derived from the start symb ol S in at
G
n
most i steps according to the rules of grammar G For all n T S For any
context sensitive grammar G it is clear that for each n there is an m such
n n n
that T T Let g n jT j where m is the numb er just mentioned
m m m
Sketch
m
i
S
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
Exercise Show that for the pro of it su ces to show that g can b e
computed in the manner describ ed in Exercise C
Now we show that g can b e computed in this nondeterministic sense in
linear space Herein lies the reason why this entire pro of is often referred to as
the inductive counting method Our plan is to compute nondeterministically
n n n n
in order the numb ers jT j jT j jT j until for some m we have jT j
m
n
j at which p oint we will output this numb er as g n What we need then jT
m
n
j under the is a nondeterministic pro cedure that correctly computes jT
i
n
assumption that the correct value of jT j is known
i
Exercise Provide the algorithm for this
n
Hint In order to compute jT j correctly we must identify and count all the
i
n
elements of T For this we need to rst generate in an inner lo op all
i
n
the elements of T We will b e able to guarantee that we have generated all
i
n
of T since we know how many elements are in the set C
i
It should b e noted that we have intentionally displaced the inductive
counting argument of the preceding pro of from its original context to the
n
context of the sets T The original pro of uses instead the sets and numb er i
Topic
of con gurations that are reachable from the start con guration of an LBA
in at most i steps The original pro of is more easily generalized to other space
b ounds like log n
References
For background to the LBA problems see
J Hartmanis H B Hunt The LBA problem and its imp ortance in the
theory of computing in R Karp ed Complexity of Computation Vol
VI I SIAM AMS Pro ceedings
S Y Kuro da Classes of languages and linear b ounded automata Infor
mation and Control
Savitch s Theorem originally app eared in
J Savitch Relationships b etween nondeterministic and deterministic
tap e complexities Journal of Computer and Systems Sciences
The original solutions to the second LBA problem app eared in
N Immerman NSPACE is closed under complement SIAM Journal on
Computing
R Szelep cs enyi The metho d of forced enumeration for nondeterministic
automata Acta Informatica
Since then pro ofs of this result have found their way into the following text
b o oks including
J Balc azar J Diaz J Gabarr o Structural Complexity II Springer
D P Bovet P Crescenzi Introduction to the Theory of Complexity
Prentice Hall
E Gurari An Introduction to the Theory of Computation Computer
Science Press
C Papadimitriou Computational Complexity Addison Wesley
R Sommerhalder S C van Westrhenen The Theory of Computability
Addison Wesley
An overview of the applications of this sort of counting technique app ears in
U Sch oning The p ower of counting in A L Selman ed Complexity
Theory Retrospective Springer
LOGSPACE Random Walks on Graphs
and Universal Traversal Sequences
There is a surprising at rst glance unexp ected di erence in space com
plexity b etween the problems of nding a path from a start no de to an
end no de in a directed graph and of doing so in an undirected graph In
an undirected graph this is easier to solve in fact in can b e done using a
random walk or a universal traversal sequence
We denote by L the class of all decision problems that can b e solved
using algorithms that use only logarithmically much space i e L
DSPACE O log n In this de nition we only count the storage space used on
the work tap e not on the read only input tap e Similarly one can de ne a
nondeterministic version of this class NL NSPACE O log n Whether the
inclusion L NL is strict is an op en question It has however b een shown
that NL coNL see Topic Just as in the case of the P NP question
one can de ne a notion of completeness for dealing with these classes But
p olynomial time reductions are meaningless in this case since NL P
Exercise Why is the running time of every halting NL computation
p olynomially b ounded Why is NL P C
One must design the notion of reduction with the smal ler of the two
classes in mind in this case L A problem A is said to b e log reducible to
problem B written A B if there is a logarithmically space b ounded
log
Turing machine with designated read only input and write only output
tap es which are not considered when computing the space usage such that
for all x x A M x B
Exercise Show that log reducibility is a transitive relation C
A problem A is called NL complete if A NL and for all A NL A A
log
Consider the following algorithmic problem
PATH f G a b j G is a directed graph and a and b are
no des in G such that there is a path
from a to b in G g
Exercise Show that PATH sometimes called GAP graph accessibility
problem is NL complete
Topic
Hint The graph that represents the p ossible transitions from one con gura
tion to the next in the NL computation has p olynomial size in the length of
the input Solutions to this exercise and the previous one can b e found in
many b o oks on complexity theory C
Now one can ask ab out the situation with undirected graphs instead of
directed graphs We de ne
UPATH f G a b j G is an undirected graph and a and b
are no des in G such that there is a path
from a to b in G g
Often the directed and undirected version of a given problem ab out graphs
are equivalent in the sense of complexity theory For example the Hamilto
nian Circuit Problem is NP complete for directed and for undirected graphs
see Garey and Johnson The graph isomorphism problems for undirected
and directed graphs are also equivalent under p olynomial time reductions
see K obler Sch oning Tor an In this case however there seems to b e a
di erence The NL completeness pro of ab ove do es not work for UPATH
Exercise Why not C
It is still the case that UPATH NL so it is p ossible that UPATH is an easier
problem than PATH We will show that a randomized version of UPATH is
in L
Let RL denote the class of problems that can b e decided by algorithms that
are simultaneously logarithmically space b ounded and p olynomially time
b ounded and are allowed to make use of random decisions in their compu
tations The use of these random decisions causes the output of the machine
M to b e a two valued random variable The computation of a problem A is
to b e understo o d in the following way
x A P r M accepts x
x A P r M accepts x
It is clear that L RL NL
Perhaps the reader is wondering why we had to require p olynomial
running time in the de nition In the exercises ab ove we just showed that L
and NL algorithms only need at most p olynomial time But this is not the
case for randomized log space machines
Exercise Give a randomized logarithmic space b ounded algorithm that
has an exp onential exp ected running time
Hint Have the algorithm rep eatedly simulate a random exp eriment for which
n
each trial succeeds with probability only until there is a successful trial
then halt C
Random Walks on Graphs
In fact it can b e shown that the class of all languages accepted by random
ized log space algorithms without the p olynomial b ound on running time
is exactly NL so the restriction to p olynomial running time is signi cant
The problems UPATH and PATH may not b e in L since a systematic
search for a path from a to b would require keeping track of where one has
b een If the input graph has n no des this would require at least n bits of
stored information to o much for a log space machine
There is however a randomized algorithm that demonstrates that
UPATH RL This algorithm carries out a random walk on the graph
starting at a This is continued until if ever b is reached A random walk re
quires less space than a systematic search of the graph we only need enough
space to store the numb er of the current no de i e log n bits Of course we
can no longer avoid no des b eing visited more than once
Our random algorithm is the following
INPUT G a b
v a
FOR i TO p n DO
Randomly cho ose a no de w adjacent to v using the uniform
distribution
v w
IF v b THEN ACCEPT END
END
REJECT
It remains to show that for an undirected graph G the p olynomial p can b e
chosen in such a way that the de nition of UPATH RL is satis ed namely
so that
G a b UPATH P r M accepts G a b
G a b UPATH P r M accepts G a b
Exercise Show that the given algorithm cannot work for directed graphs
That is show that there are directed graphs G such that the no de b in G is
n
reachable by a random walk but only with probability So the exp ected
n
length of the random walk is hence not p olynomial
Hint This exercise is very similar to Exercise C
By a random walk on a graph G started at a we mean an in nite sequence
W v v v with v a and v chosen randomly under the uni
i
form distribution from among the no des adjacent to v So fv v g must
i i i
b e an undirected edge in G Let W denote the nite subsequence consist
i
ing of the rst i no des in W For a no de v in G we compute the probability
of the o ccurrence of v in W as
jfi n j v v gj
i
P lim
v n n
Topic
From the theory of Markov chains it is clear by the law of large numb ers
that this limit exists and that for a connected graph G P for all v
v
P is the exp ected value of the distance b etween adjacent o ccurrences of
v
the no de v in W that is the mean numb er of steps it takes for a random
walk starting at v to return to v
For the moment we will treat each undirected edge fu v g like two directed
edges u v and v u We claim that each of these directed edges o ccurs with
probability e in a random walk on a connected graph G with e undirected
edges It is clear that the sum of these probabilities is
If we extend the notation ab ove to edges as well as no des then our claim
b ecomes
jfi n j v v u v gj
i i
e P lim
n
u v
n
Supp ose this is not the case If the edges do not all o ccur with the same
probability then there must b e an edge u v that has a larger probability
than the mean of the probabilities of its adjacent edges via no de v That
is
X
P P
v w u v
d v
v w G
where d v is the degree of the no de v
Exercise Justify that this inequality follows from the assumption that
the edges do not all o ccur with equal probability C
Thus if we can show that for all edges u v
X
P P
u v v w
d v
v w G
then by Exercise we can conclude that P e for all edges u v
u v
We can see that this is the case as follows Consider the p oints in the random
walk W when the edge u v o ccurs
W u v u v
After v must come an adjacent edge v w and each such edge o ccurs with
equal probability d v So W lo oks like
W u v w u v w
and the desired equality follows Thus every directed edge u v o ccurs with
probability e and therefore every undirected edge fu v g with probability
e
Every edge of the form u v leads to no de v Every such edge o ccurs with
probability e so the no de v o ccurs with probability d v v i e P v
Random Walks on Graphs
d v e The mean length of a random walk starting at a random no de until
the no de v o ccurs is therefore P e d v
v
Exercise Let X b e a random variable that has an exp ected value and
takes on only non negative values Show the following inequality P r X
a E X a
Hint Lo ok in a b o ok on probability under the topic Markov s inequality
C
Now we only have the small step of showing that the length of the random
walk can b e b ounded by a p olynomial but that each no de in particular our
destination no de b will still o ccur with probability Furthermore we
will show that there are universal traversal sequences that is p olynomially
long sequences of instructions of the form right left left right left
which if followed will cause a walk to visit every no de in any graph that
has n no des Frequent museum visitors will likely b e very interested in this
result
Let E i j denote the exp ected value of the numb er of steps in a random
walk from no de i to no de j We have already shown that the mean length of
time from one o ccurrence of v to the next is P e d v In this notation
v
we can express this as E v v e d v
Exercise Let u v b e an edge in G Show that E u v e C
Now we want an approximation for E a b for any no des a and b which
are not necessarily adjacent Let E a G denote the mean length of a random
walk starting at a until all the no des in G have b een visited at least once
We assume that G is connected so that all no des in G are reachable from a
Then for any a and b in G we have E a b E a G
Let a v v v v b e a path in G that starts at a and visits every
k
no de at least once
Exercise Show that in a connected graph with n no des there is always
such a path of length k n C
Now we can give a very rough approximation for E a G by considering
the mean length of a random walk that rst go es from a to v in one or more
steps then wanders to v then v etc and in this way eventually arrives at
v From this we get
k
k
X
E v v n e en E a G
i i
i
Exercise Show that from this inequality it follows that the probability
that a xed no de b in G is not visited in a random walk that starts at a and
pro ceeds for e steps is at most
Hint Use Exercise C
Topic
Now we put everything together On input G a b where G is an undi
rected graph with n no des and e edges we simulate a random walk of length
e If there is no path from a to b then this algorithm cannot p ossibly nd
one If there is such a path i e G a b UPATH then this algorithm nds
one with probability at least
To reduce the probability that a path exists but is not found this random
exp eriment can b e rep eated or equivalently the the length of the random
walk can b e increased If the length of the random walk is m e i e m
rep etitions of the exp eriment then the probability of overlo oking an existing
m
path is reduced to at most
The ability to drastically increase the probability of success while main
taining a p olynomially long random walk often referred to as probability
ampli cation is the key to the existence of a universal traversal sequence A
d regular graph is a connected graph in which each no de has degree at most
d Clearly any d regular graph has e dn edges We order the d edges
leaving a given no de arbitrarily and assign each a numb er d
Note that each edge u v has two numb ers one with resp ect to each of u
and v By a universal traversal sequence for d regular graphs we mean a
sequence I i i i with i f d g such that for any of
k j
these graphs and any choice of starting no des the graph is traversed via I
in the sense that at each step the choice of the next no de to visit is made
according to I and the numb ering of the edges leaving the current no de and
that all no des of the graph are visited
A randomly chosen sequence I i i i where each i is chosen
k j
indep endently at random under the uniform distribution on f d g
describ es precisely a random walk We have already seen that the probability
that a random walk of length k m en m dn do es not completely
m
traverse a graph G is at most So if g is the numb er of lab eled d
regular graphs with the lab eling describ ed ab ove and we cho ose m so large
m
that g then the probability that a randomly chosen sequence
is a universal traversal sequence is p ositive which means at least one such
sequences must exist But how large must m b e
Exercise Show that the numb er of lab eled d regular graphs is at most
dn
n C
Since
m m dn
g n
m dn log n
m dn log n
there must exist a universal traversal sequence for all d regular graphs with
n no des that has length dn log n dn O n log n
Random Walks on Graphs
References
R Aleliunas R M Karp R K Lipton L Lov asz C Racko Random
walks universal traversal sequences and the complexity of maze prob
lems Proceedings of Symposium on Foundations of Computer Science
IEEE
M Garey D Johnson Computers and Intractability Freeman
J K obler U Sch oning J Tor an The Graph Isomorphism Problem Its
Structural Complexity Birkh auser
M Sipser Lecture Notes MIT
Topic
Exp onential Lower Bounds
for the Length of Resolution Pro ofs
The resolution calculus is one of the most commonly used metho ds in
theorem proving For a long time an exp onential lower b ound for the length
of such pro ofs was sought In this was nally proven by Haken
The resolution calculus op erates on sets of clauses A clause is a disjunction
OR of literals A literal is a variable or its negation A set of clauses is
understo o d to represent a conjunction AND of the clauses Altogether this
corresp onds to a formula in disjunctive normal form
Example The set of clauses
x x g fx x g fx x g fx g g f fx
corresp onds to the formula
x x x x x x x x
Exercise Make a truth table for this formula Is the formula satis able
C
In each resolution step of a resolution pro of two previously derived or
given clauses K and K are combined to derive a new clause K This is
p ossible only if there is a variable x that o ccurs p ositively in one clause and
i
negatively in the other The resolvent K is then K K fx x g The
i i
notation for this is
K
b
b
b
b
K
K
Topic
Example
fx x x x g
Q
Q
Q
Q
x x x g fx x
fx x x x x x g
In the example ab ove the resolution is done on variable x since x o ccurs
x in the lower one After resolution x do es not in the upp er clause and
o ccur in the resulting clause but it may o ccur again later in the resolution
Several resolution steps can b e represented by an acyclic graph The goal
of the resolution pro of is to derive
Example continued from ab ove
fx x x g
Q
Q
Q
x x g f
S
f x x g
S
S
S
S
x g f
P
P
x x g f
P
Q
P
Q
Q
fx g
fx g
The resolution calculus is sound and complete for refutation That is if
a set of clauses F has a resolution pro of that leads to the empty clause
then F is unsatis able soundness and if F is unsatis able then there is a
resolution pro of that leads to the empty clause completeness
Exercise Prove this
Hint The pro of of soundness can b e done by induction on the length of the
resolution pro of the pro of of completeness can b e done by induction on the
numb er of variables that o ccur in the clause C
Now supp ose we are given a resolution pro of Furthermore let b e an
arbitrary assignment of the variables that o ccur Then uniquely determines
a path through the resolution pro of from one of the original clauses to the
empty clause so that for each clause K along this path K
The Length of Resolution Pro ofs
Example Let x x x Then the path determined by
in the resolution pro of ab ove is indicated b elow
x x g fx
Q
Q
Q
f x x g
S
S
S
S
S
f x x g
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
P
f x g
P
P
P
P
P
x x g f P
P
P
P
P
P
Q P
P
P
P
Q
Q
fx g
fx g
Exercise Justify the claim that for each such there is a path through
the resolution pro of from one of the original clauses to the empty clause and
that this path is unique
Hint Construct the path by starting from the empty clause and working
backwards toward one of the original clauses C
A sound and complete pro of system for refutation like resolution can b e
viewed as a nondeterministic algorithm that works in the following way
INPUT F the formula to refute
REPEAT
Nondeterministically cho ose one p ossible pro of step and
add it to the pro of that has b een constructed so far
UNTIL unsatis ability of F is established by the pro of
OUTPUT unsatis able and accept
where the test in the REPEAT UNTIL lo op is required to run in time that
is p olynomial in the length of the pro of Such an algorithm accepts in the
SAT nondeterministic sense precisely the set of unsatis able formulas i e
In the theory of NP completeness one de nes the time complexity of a
nondeterministic algorithm to b e the least p ossible numb er of steps under
some choice of the nondeterministic decisions in which the result in this case
establishing the unsatis ability of a formula can b e reached So in this case
the nondeterministic time complexity corresp onds precisely to the length of
the shortest p ossible pro of in some calculus We call a sound and complete
pro of system a SUPER pro of system if there is a p olynomial p so that the
algorithm ab ove on an input of length n makes more exactly can only make
at most p n passes through the lo op b efore arriving at a result Or formu
lated di erently a pro of system is SUPER if for every unsatis able formula
F of length n there is a pro of of the unsatis ability of F that is at most
p olynomially long in n
Topic
Exercise Show that NP coNP if and only if there is a SUPER pro of
system for refutation
Hint Use the fact that SAT is NP complete C
Is the resolution calculus a SUPER pro of system If so then by the ex
ercise ab ove it would follow that NP coNP which would b e sp ectacular
and contradict most conjectures On the other hand if we can show that
resolution is not a SUPER pro of system then this is an interesting result
but it do es not have any immediate direct implications for the NP coNP
problem In the remainder of this chapter we will prove this interesting result
Theorem The resolution calculus is not a SUPER proof system
Proof First we will de ne for each n a formula i e a set of clauses that
expresses the pigeonhole principle n pigeons do not t into n pigeonholes
in such a way that no hole contains more than one pigeon The variable x
i j
i f ng and j f n g will b e used to express that pigeon j
is in pigeonhole i The set of clauses PHP consists of
n
Typ e clauses
fx x x g fx x x g fx x x g
n n n n n n
and
Typ e clauses
x x g fx x g fx x g fx x g fx x g f
n n n
fx x g fx x g fx x g fx x g fx x g
n n n
x x g fx x g fx x g fx x g fx x g f
n n n n n n n n n n n n n
The typ e clauses express that each pigeon is in at least one hole The typ e
clauses express that each hole contains at most one pigeon literally each
hole do es not contain a pair of distinct pigeons
It is clear that for each n PHP is unsatis able Our goal is to show
n
n
that every resolution pro of of this fact must have length at least c for some
constant c Since the numb er of variables in PHP is quadratic in n
n
and the numb er of clauses is O n it follows from this that the resolution
calculus is not SUPER
For the pro of we use n n matrices to represent the clauses in PHP
n
and their resolvents The n rows represent the n holes and the n columns
Translation note In German the pigeonhole principle is called the Schubfach
prinzip literally the drawer principle and the original examples here included
the pigeon example and the following n so cks do not t in n drawers in such
a way that no drawer contains more than one so ck Also the English expression
pigeonhole probably do es not refer to bird houses but to a certain typ e of
mail slots common in many university department o ces which are also called
pigeonholes So that a b etter example would p erhaps b e to replace pigeons with
letters but we will continue with pigeons as in the original
The Length of Resolution Pro ofs
the n pigeons We will place a in p osition i j if the literal x o ccurs
i j
x o ccurs in the clause in the clause and a in p osition i j if the literal
i j
Exercise Draw the representation of PHP in this scheme C
Analogously we can represent assignments to variables in this rectangle
notation scheme we place a b f g in p osition i j if the variable x
i j
should have the value b We will call an assignment critical if n of the n
pigeons have b een assigned distinct holes so that all n holes contain a pigeon
In other words an assignment is said to b e critical if for every i hole there
is a j pigeon with x and also for every j there is at most one i
i j
with x
i j
Exercise Draw a matrix that represents an arbitrary critical as
signment C
Exercise Let n b e xed How many critical assignments are there C
In the matrix representation of a critical assignment there will b e a column
in which there are only s we will call this column the column of the critical
assignment
Exercise A critical assignment always satis es all clauses of PHP
n
except for one which one C
Assume we have a resolution pro of that demonstrates the unsatis ability
of PHP represented as an acyclic graph Let b e a critical assignment
n
We apply the construction of Exercise which pro duces for each such
assignment a unique path connecting one of the original clauses with the
empty clause such that for each clause K along this path K
Exercise The path asso ciated with can only connect the empty clause
with one of the original clauses in PHP Which one C
n
Now we put everything together Every critical assignment has a
column The unique path through the resolution pro of of PHP determined
n
by connects a clause of PHP with the empty clause This clause in PHP
n n
must b e the typ e clause that has s in the column of
Consider a example so n Let b e the following critical
assignment
The arrow indicates the column of
Topic
The next diagram indicates the path in the resolution pro of asso ciated
with
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
b
H
b
H
b
H
b
H
H H
H
H
b
H
H
b
H
b
b
Z
Z
Z
b
Z
b
b
b
b
b
b
Z
Z b
Z
Z
The leftmost clause contains n s The empty clause at the end has none
A resolution step can only eliminate at most one of these s So there must
b e a clause along this path that has exactly n s in the column of
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
Z
Z
Z
Z
Z
Z
Z
Z
Of course the remaining columns of this clause can and must also have
changed but for the moment we are only interested in the column So we
have shown that for every critical assignment there is a clause K in the
resolution pro of that has exactly n s in the column of and that
K
Now imagine the clauses of the resolution pro of to b e linearly ordered in
such a way that resolvents always o ccur later in the list than their parent
clauses Such an ordering is a top ological ordering of the acyclic graph of
the pro of The last clause of this order must b e the empty clause For every
The Length of Resolution Pro ofs
critical assignment let K b e the rst clause K in this ordering that has
exactly n s in the column of and for which K This clause
is not necessarily the clause constructed ab ove in our pro of of the existence
of such a clause
Exercise Show that the clause K which has n s in the column
of has no s in this column C
Next we want to consider partial assignments i e assignments that do
not assign a value to all of the variables We are only interested in partial
assignments that can b e extended to critical assignments That is the partial
assignments that interest us cannot have two s in the same row or column
In what follows we will consider only partial assignments that in addition to
b eing extendible to a critical assignment also have the prop erty that exactly
n of the p ositions are assigned a For simplicity we will assume that
n is divisible by In order to make these partial assignments notationally
distinguishable from critical assignments we will use capital letters for the
partial assignments Let S b e a partial assignment with the restrictions
S
as mentioned We let K denote the rst clause in the pro of sequence of
the form K such that S can b e extended to In this way every partial
assignment S is assigned a critical assignment namely one for which K
S S
K From now on we will call the clauses of the form K complex clauses
The pro of strategy is now the following We will show that if a resolution
pro of is to o short then it must contain an error Supp ose there is a res
olution pro of P for PHP that is to o short where for the length of the
n
pro of P we only count the numb er of complex clauses that o ccur Supp ose
n
this numb er is less than c for some constant c to b e determined later
Then a certain greedy algorithm which we will give b elow with the set of
complex clauses in P as input will nd a partial assignment S that satis es
all of these complex clauses That is the n s in S will b e in p ositions
such that for each of the complex clauses at least one of these p ositions con
tains a Starting with this partial assignment S as describ ed ab ove we
S S
pro duce a complex clause K K in P for some with K But
this is a contradiction since S makes all of the complex clauses in P true
S
Thus K represents an error in the pro of and all correct pro ofs must b e
long enough
In order to carry out this argument with our greedy algorithm we must
show that every complex clause must necessarily have in addition to the
n s in the column other s in other columns In fact there must b e
so many that we can always assume that there are at least n s The
existence of this many s simpli es the argument that n s can b e xed
so that all complex clauses b ecome true
Topic
Consider an example A partial assignment S must in this case x
S
n Assume that the is in p osition Let K K for some
that extends S Furthermore supp ose that x x The
n n
column in this case is n Unsp eci ed values in the diagram b elow are
S
column
S
The corresp onding complex clause K K has exactly n s in
column n For simplicity we draw them at the b ottom
S
K
Exercise Show that there must b e at least n s in the diagram for
S
that are neither in the same row as a in the column of K nor xed
by S C
Now select an arbitrary from among these n many and change to
at this lo cation At the same time change the in the same row of the
column to a The result is another critical assignment for which the
column is the column in which a was changed to a If we carry this
pro cedure out using the in column of our example we get the following transformation
The Length of Resolution Pro ofs
S
column
Claim Now we claim that all n columns in which s of the typ e describ ed
in the last exercise o ccur are go o d columns In our example this would
include among others column A go o d column is one in which either there
is exactly one or at least n s
S
Proof of the claim First note that no column of K can contain two s
Exercise Why C
Now consider one of the n columns mentioned in the claim and supp ose
that this column is not go o d Let this b e column j Then there must b e no
s and fewer than n s in this column In this case the transformation
from to using this column do es not change the truth value it is still
S
the case that K
Exercise Justify this C
S
Now we can construct a path backwards from the clause K to one of the
original clauses so that for each clause K along this path K As in
Exercise this path must lead to a clause of typ e in which the column j
is lled with s So at the end of this path we have a clause with more than
n s Somewhere strictly b etween there must b e a clause with exactly
n s in column j This is a clause of the same form as K p erhaps
K Since K is de ned to b e the rst clause of this typ e that o ccurs
S
in the pro of P K must come strictly b efore K But since is also an
S
extension of S and K was chosen to b e at the rst p ossible p osition in the
S
pro of this is a contradiction K should have b een chosen to b e K Thus
every complex clause has at least n go o d columns including the
column ut
Now supp ose that P is a resolution pro of for the unsatis ability of PHP
n
and fK K K g are the complex clauses that o ccur in P We mo dify
t
the sequence fK g to a sequence K in every column that contains a this
i
i
will b e a go o d column we strike the and ll the rest of the column
except for the p osition of the original with For any partial assignment
S if S K then S K
i
i
The sequence K K K is the input for the following greedy algo
t
rithm This algorithm tries to construct a partial assignment S such that
Topic
the n s in S are su cient to force that S K S K In particular
i
i
this will also b e the case for any extension of S to a critical assignment But
S
this would b e a contradiction since for each complex clause K which is in
S S
the input list there must b e an for which K K and K
We will see that this contradiction always arises if t is to o small Thus
the numb er of complex clauses in P and therefore the length of P itself
must b e large enough
PROCEDURE Greedy M Set of clauses partial assignment
VAR
S partial assignment
E set of matrix p ositions
k i j CARDINAL
BEGIN
S empty assignment
E all p ositions
FOR k TO n DO
Find the p ositions i j in E where the most clauses
in M have a
Expand S by i j
Strike from M those clauses that have in p osition i j
Strike from E row i and column j
END
RETURN S
END Greedy
Now we analyze the algorithm The initial situation is that M contains
the t complex clauses given as input By the discussion ab ove each of these
complex clauses must have at least n n s E initially contains
all n n p ositions This means that in the rst step the fraction of clauses
that are taken care of i e for which their value under S is determined to
b e is at least
n n n n
n n n
Exercise Justify this C
Of course the fraction of clauses that can b e taken care of decreases
with each step since row i and column j are stricken After k passes through
the lo op the ratio of remaining s to the size of E is
n k n k n k n k
n k n k n k
This quotient is smallest at the last pass through the lo op We get a lower
b ound by taking only this last ratio when k n
The Length of Resolution Pro ofs
n n n n
n n
Let M b e the set of remaining clauses in M after the ith pass through the
i
lo op Then
jM j jM j jM j jM j
i i i i
i
This yields the approximation jM j jM j Now we hop e that the line
i
of argumentation outlined ab ove actually works i e that jM j It is
n
su cient to show that jM j since jM j is an integer This is the case
n n
n n n
provided jM j or equivalently if jM j c where
c
In summary if the numb er of complex clauses that are given as input to
n
the greedy algorithm is less than c then the algorithm succeeds after n
passes through the lo op in constructing a partial assignment S that makes all
clauses in the input true However we have also seen that starting with this
partial assignment we can nd a complex clause that do es not have the value
under S since there is an extension of S that makes the clause false This
contradiction implies that all resolution pro ofs of PHP must have at least
n
n
c complex clauses ut
A computer generated pro of of PHP app ears at the end of this chapter
In its search for a pro of the computer generated clauses including the
input clauses of which are required for the pro of
References
The notion of a SUPER pro of system and its relationship to the NP coNP
problem originated with
S Co ok R Reckhow The relative e ciency of prop ositional pro of sys
tems Journal of Symbolic Logic
After the problem had b een worked on since the s the pro of of an exp o
nential lower b ound for the resolution calculus was rst achieved in
A Haken The Intractability of Resolution Theoretical Computer Science
The pro of given here follows a somewhat more elegant technique of
S Co ok and T Pitassi A Feasible Constructive Lower Bound for Reso
lution Pro ofs Information Processing Letters
but improves on the lower b ound stated there
Topic
PROOF res p p p
p p res p p p
p p res p p p
p p res p p p
p p res p p p
p p res p p
p p res p p
p p res p p
p p res p p
p p res p p
p p res p p
p p res p p
p p res p p
p p res p p
p p res p p
p p res p p
p p res p p
p p res p p
p p res p p
p p p res p p
p p p res p p
p p p res p p
p p p res p p
res p p p res p
res p p p res p p
res p p p res p
res p p p res p p
res p p p res p p
res p p p res p p
res p p p res p p
res p p p res p p
res p p p res p
res p p p res p
res p p p res p
res p p p res p
res p p p res p
res p p p res p
res p p p res p
res p p p res
res p p p
res p p p
end of proof
res p p p
res p p p
res p p p
res p p p
res p p p
res p p p
res p p p
res p p p
res p p p
res p p p
res p p p
res p p p
Sp ectral Problems and Descriptive
Complexity Theory
This chapter b egins with a question from predicate logic namely to deter
mine the set of all sizes of nite mo dels of a given formula It turns out
that there is an amazingly close relationship b etween this question and the
world of P and NP
In this chapter we want to discuss formulas in predicate logic These formulas
are built up from atomic formulas There are two kinds of atomic formulas
One typ e has the form P x x where P is a predicate symbol with arity
k
k and each x is a variable The other p ossibility for an atomic formula is a
i
formula of the form x x
i j
Atomic formulas are the simplest formulas More complex formulas are
built up from the atomic formulas Given two formulas G and H we can
form b o olean combinations for example
G H G H G H G H G
Let x b e a variable then by existential or universal quanti cation over
x we obtain from G the new formulas
x G x G
An o ccurrence of the variable x in a formula G is called bound if it o ccurs in
a subformula of F of the form x G or x G Otherwise an o ccurrence of the
variable x is said to b e free In this topic we are only interested in formulas
in which all o ccurrences of x are b ound Such formulas are called sentences
A sentence can b e assigned a truth value by interpreting the sentence
in a given structure A A structure suitable for interpreting the sentence F
consists of a non empty set the universe of values for the variables that
o ccur in F and concrete predicates for each predicate symb ol that o ccurs in
F i e relations on the universe of the appropriate arity The truth value A F
of the formula F is determined recursively on the structure of the formula
If the formula is a b o olean combination of subformulas G and H then one
determines recursively the truth values A G and A H and combines these
values according to the usual b o olean op erations i e truth tables
Topic
If F has the form x G then A F is true if and only if there is a value w
in the universe such that G interpreted under A with x interpreted as w
is assigned the value true See the instructions b elow regarding assigning
truth values to atomic formulas for more ab out the substitution of w for
x
If the formula F has the form x G then we replace there exists with
for all in the instructions ab ove
If the formula is an atomic formula then in general it contains variables
Note that this case do es not o ccur for sentences themselves since all
variables in a sentence are b ound by quanti ers but it o ccurs inside the
recursion in fact it is the base case of the recursion In the previously
executed recursive steps each variable in a sentence will have b een assigned
a value from the universe Therefore we can evaluate this atomic formula
directly using the interpretations of the predicates given by A The equals
sign is always interpreted as identity on the universe
If A F is true we write A j F In this case A is said to b e a model for F
A particular structure is denoted in tuple form A M R R where
m
M is the universe and each R is a relation on the universe which is used to
i
interpret the predicate P A formula can have nite or in nite mo dels or
i
none at all where the size of a mo del is measured by the size of the universe
jM j We let jAj jM j denote the size of a mo del A with universe M In what
follows we are interested in the set of all sizes of nite mo dels for a given
formula Let
S pectr um F fn N j there is a mo del A with jA j n and A j F g
In H Schulz p osed the problem the so called spectral problem of char
acterizing the set of all such sp ectra And in G Asser p osed the question
of whether the class of all sp ectra is closed under complement
Example Let F b e a predicate logic formula that formalizes the axioms for a
eld In the usual de nition of a eld a eld is sp eci ed by its underlying set
and two functions on that set Since we have not allowed function symb ols in
our language we must mo dify the usual de nition slightly Candidates for a
eld are structures A M R R where R R are each ary relations
expressing the equations x y z and x y z resp ectively A subformula
of F that formalizes the asso ciative law for eld multiplication would then
b e
x y z u v w w R x y u R u z w R y z v R x v w
w w
Other subformulas of F would have to express the other eld axioms as well
as the fact that the relations R and R are functions i e that the values
of the eld op erations are unique The set of all mo dels of F would then b e
Sp ectral Problems
the set of all elds From algebra we know that there is a nite eld of size
n if and only if n is a prime p ower So
S pectr um F fn j n is a prime p ower g
Exercise Give a formula for which S pectr um F N f g C
Exercise Give a formula for which S pectr um F f g C
Generalizing the previous exercise we see immediately that every nite subset
of N is the sp ectrum of some sentence
A nite structure can b e co ded as a string just as is commonly done in
complexity theory with other typ es of ob jects e g graphs For example
the structure A M R R with jA j jM j n can b e co ded as
m
n
r r r
m
k
where r is a string of length n and R is the interpretation of a k
i i
ary predicate P This string describ es the relation R bit for bit as a
i i
characteristic sequence on the universe Note that we have not co ded in the
arity of the predicates it is implicit in the syntax of the formulas but this
could b e easily done
Exercise Use the recursive description for evaluation of A F to give
for every sentence F an algorithm that on input A determines in p olynomial
time whether A j F In other words show that for all sentences F in rst
order predicate logic the set M odel s F fA j A j F and A is nite g is in
P C
On the other hand one can ask if for every language L P there is a
formula F so that the mo dels of F are precisely the strings x L But this is
not the case there are languages in P that cannot b e describ ed in this sense
by sentences in rst order predicate logic In fact Immerman has shown that
the class of languages describ ed by sentences of rst order predicate logic
under certain additional technical assumptions is precisely the class AC
Symb olically this is expressed as FO AC FO stands for rst order For
more information ab out the class AC see Topics and
It follows immediately from Exercise that for every formula F the set
S pectr um F over the alphab et f g i e with all numb ers coded in unary
is in NP If we co de the numb ers in binary then the strings are only logarith
S
cn
mically as long so in this case S pectr um F NEXP NTIME
c
Exercise Justify the last statement C
The set S pectr um F co des the information ab out the sizes of mo dels
for F This is very crude information ab out F In what follows we want to
Topic
use a language that expresses somewhat more information ab out F Observe
rst that F has a mo del A M R R where P P are the
m m
predicate symb ols that o ccur in F if and only if the formula
G P P F
m
has M a mo del consisting only of a universe only the size of which
really matters for a mo del The formula P P F is a formula in
m
second order predicate logic which means that we may quantify not only
over variables but also over predicate symb ols and hence over relations on
the universe The semantics of such a formula are de ned in the exp ected
way relations corresp onding to the predicate symb ols P must exist over the
i
universe that make the formula F true Since the predicate symb ols are now
b ound by the quanti cation they no longer require interpretations as part of
the mo del A for G
Now we mo dify the formula and the underlying question ab out the ex
istence of mo dels in the following way some of the predicate symb ols may
b e b ound by quanti ers others for simplicity we will consider just one may
o ccur freely in G
G P P F P P P
m m
The notation F P P P is intended to indicate that exactly the predi
m
cate symb ols P P P o ccur in F A mo del for G now must have the form
m
A M R where R is a relation over the universe M of the appropriate
arity We assign to this formula G the set of all of its mo dels
Mo dels G fA j A j G A is nite g
which is sometimes called the generalized spectrum of G
Exercise We want to consider the question of what complexity the
language M odel s G has for this typ e of second order formula G where the
second order quanti ers are only existential and o ccur b efore all rst order
quanti ers
Show that M odel s G NP
Hint Make a small mo di cation to the observation ab ove that S pectr um F
co ded in unary is in NP C
As we will show b elow this time the converse is also true A language L
is in NP if and only if there is such a second order existentially quanti ed
formula G such that L fx j A j Gg For this we assign to each string
x
x a structure A which co des x in a certain way In this sense ignoring
x
issues of co ding the set of all mo dels of second order existential formulas
is the class NP This fact is expressed succinctly as NP SO The amazing
thing ab out this result is that it gives an exact characterization of the class
NP solely in terms of expressibility in a certain logic There is no mention of
Sp ectral Problems
Turing machines computations running times or p olynomials Descriptive
complexity theory deals with the question of which complexity classes can b e
characterized in such a way and whether it may b e p ossible to prove that two
such classes are distinct using only metho ds from mo del theory
It is instructive and for what follows useful to reprove the previous
exercise SO NP in a di erent way namely via a p olynomial reduc
tion to SAT For this we must reduce an arbitrary structure A M P
appropriate for a formula G in p olynomial time to a b o olean formula
The construction will of course make use of G But note that G is xed
and A is the input so the algorithm must only b e p olynomial in jAj Let
G P P F P P P and let A M P b e a suitable struc
m m
ture The rst order part of G is the formula F where the predicate symb ols
P P P o ccur The interpretation of P is given by A but the existence
m
of predicates fP g that make F true is the question Let M f ng
i
We replace step by step every subformula of F of the form x F with
F x F x n where F x i denotes that for every free o ccurrence
of x in F we replace x with i This is a purely syntactic pro cess Similarly
every subformula of the form x F is replaced by F x F x n
The resulting formula a substitute for the formula F contains no quanti ers
or variables all of the variables have b een replaced by constants The atomic
formulas in the new formula have three p ossible forms
P i i
l
i j
P i i
i k
l and k are the arities of the predicate symb ols involved The atomic formu
las of the forms marked with can b e evaluated directly using the structure
A This can b e used to simplify the formula Furthermore every atomic for
mula of the form P i i can b e made true or false indep endently of
i k
every other one So each P i i can b e considered as a name for a
i k
boolean variable Now we are just lo oking for a satisfying assignment to a
b o olean formula which will exist if and only if A j G Since the construc
tion requires only p olynomial time what we have describ ed is nothing other
than a reduction to SAT
What is interesting here is that the structure of the formula F is re ected
in a certain way in the resulting formula If we assume that the formula
F has the form F x x H where H is quanti er free and a Horn
l
formula i e H is in conjunctive normal form and each clause contains at
most one p ositive literal then the formula given by the reduction is also
a Horn formula For Horn formulas the satis ability problem is known to
b e solvable in p olynomial time SAT H or n P So
In fact it is su cient in this context if only the p ortions of the formula consisting
of the literals P i i form a Horn formula but the input predicate or the
i
k
order relation do not
Topic
SO H or n P
Now we want to turn our attention to the reverse direction which strongly
resembles the pro of of Co ok s theorem Let L b e a language in NP So there is
a nondeterministic Turing machine M that accepts the language L in p olyno
mial time Let the numb er of nondeterministic steps needed by M on inputs
k
of length n b e b ounded by the p olynomial n Without loss of generality
we can assume that the machine never visits tap e cells to the left of the in
put Let b e the work alphab et of the Turing machine and let the input
alphab et b e f g Furthermore let Z b e the set of states Then con
k
gurations of M can b e describ ed as strings of length n over the alphab et
Z The string
a a a z a a a
k
i i i
n
co des that the tap e contents of M at a given time are precisely a a k
n
the head is lo cated at p osition i and the state is z An accepting computation
of M on input x with jxj n and x x x can b e represented by an
n
k k
n n matrix with entries from
z x x x x
n
a a a a a k z
n n e
n
The rst row represents the start con guration In the last row the halting
state z has b een reached and we will assume that this only happ ens when
e
the read write head is at the left most p osition and reading a blank tap e cell
The symb ol in p osition i j dep ends only on the three symb ols
in p ositions i j i j and i j and the nondeterministic choices
of the machine
i j i j i j
i j
Sp ectral Problems
If we assume that at each step there are always two nondeterministic choices
available then we can describ e this with two nite ary relations
a b c
which list all p ossible allowable tuples or list
d
ing one choice the other
Now we set ab out describing a formula G so that x L if and only
if for a certain structure A A j G The simplest structure that can b e
x x
used to represent a binary string for the moment is A f ng E
x
where E i is true if and only if x But if we use this enco ding we
i
have the following problem All structures that are isomorphic to A will b e
x
indistinguishable to the formula G which we have yet to give So we must
add to the structure an order relation on the universe which represents
our intended ordering of the universe which we have already indicated by
saying that our universe is f ng The unique structure A describing
x
x is now
A f ng E
x
The fact that G can make use of an ordering of the universe will b e seen to
b e very useful A rst comp onent of G based on will b e used to describ e
a successor relation S on k tuples In this way we will in a certain sense b e
k
able to count to n and not just up to n The k ary predicate S will b e
existentially quanti ed in G G S Later in G we will axiomatize the
desired prop erties of the predicate S We will de ne the p ortion of G used
to sp ecify the prop erties of S recursively de ning S S S S where
k
each S is a i ary predicate de ning an ordering on i tuples
i
S x y x y z x z y z y z
S x x y y
i i i
V
i
S x y x y
j j
j
Max x Min y S x x y y
i i i
The two additional predicates used ab ove Min and Max identify the largest
and smallest elements of the universe Min can b e de ned via
Min x Min x y x y x y
Max can b e de ned similarly
For every symb ol a in we intro duce a new k ary predicate symb ol
P which we use to represent the computation of machine M on input x
a
as describ ed ab ove In particular P x y will b e true if and only if the
a
symb ol a is in p osition x y in the matrix given ab ove We are using x as
an abbreviation for x x
k
So the nal formula G will have the form
C F P G S Min Max P
a a
m
Topic
where fa a g The k ary predicate C expresses through its truth
m
value for each time step which of the two nondeterministic choices was made
The Formula F consists of the axiomatization of S Min and Max given
ab ove and the following conditions on the predicates P
a
Let s get right to the heart of the formula G the transition relation for
the machine M This can b e expressed as
V
x x x y y P x y P x y P x y
a b c
a b c d
a b c d
S x x S y y S y y
C y P x y
d
We will also need the same formula with in place of and C y in
place of C y Note this is the only formula in the construction that is not
a Horn formula
In a similar way we can express the transition relation for the column at
the far right and left edges of the matrix
Exercise Give a formula that correctly describ es the rst row of the
matrix i e the start con guration of the machine C
Exercise Give a formula that expresses that at no time and at no
lo cation can more than one of the predicates P b e true C
a
Exercise Give a formula for the last row of the matrix that is one that
checks for an appropriate end con guration C
The desired existentially quanti ed second order formula G now consists
of the conjunction of all of the formulas generated in the construction The
mo dels of this formula characterize precisely the strings x that the Turing
machine M accepts So we have
Theorem Fagin s Theorem NP SO ut
This result can b e immediately generalized to the p olynomial hierarchy
PH cf Topic Let SO denote the set of all mo dels of arbitrary second
order formulas universal quanti cation is allowed now then
Corollary Stockmeyer PH SO ut
Let s return now to the sp ectral problem Here we do not have an input
enco ding like A and therefore also no order relationship on the universe
x
Only the question of the sizes of the mo dels is relevant In this context there
can b e any arbitrary ordering of the universe not just the one that corre
sp onds to the intended order of the input string that causes the construction
n
to work The input is co ded in unary as For this reason we can extend
Sp ectral Problems
the construction ab ove to existentially guess an ordering and x it with
additional axioms G Axioms for
Exercise Give a formula that characterizes as a total ordering That
is every mo del for the formula must interpret as a strict total ordering of
the universe C
From this we get
Theorem Bennett R odding Schwichtenberg Jones Selman Fagin
The set of spectra of rst order formulas coded in unary is exactly the class
of NP languages over a one element alphabet NP
Corollary The set of spectra of rst order formulas in closed under
complement if and only if NP is closed under complement if and only if
NEXP is closed under complement
Exercise Justify the last part of the corollary NP is closed under
complement if and only if NEXP is closed under complement C
In the construction ab ove there was only one place where we made use of
a formula that was not a Horn formula in the sense describ ed on page see
also the fo otnote there If the Turing machine is deterministic then we no
longer need the predicate C which simulated the nondeterministic choices
of the nondeterministic machine or the distinction b etween and The
result is a Horn formula This gives the following result
Theorem Gr adel P SO H or n ut
We note only that once again it is imp ortant for this last result that an
order relation on the universe b e available either in the form of the input
enco ding or as it is often done by enriching the logical language to include
a built in symb ol with a xed interpretation The axioms for the ordering
cannot b e expressed as a purely Horn formula The problem is the totality
condition
References
More detailed discussion of predicate logic can b e found in many intro ductory
b o oks on logic including
H Enderton A Mathematical Introduction to Logic Academic Press
H J Keisler J Robbin Mathematical Logic and Computability McGraw
Hill
For a pro of that SAT H or n P see
Topic
C H Papadimitriou Computational Complexity Addison Wesley
page
The results presented here can b e found in
E B orger Decision problems in predicate logic in G Lolli Florence
Logic Col loquium North Holland
C A Christen Sp ektralproblem und Komplexit atstheorie in E Sp ecker
V Strassen Komplexit at von Entscheidungsproblemen Lecture Notes in
Computer Science Springer
R Fagin Generalized rst order sp ectra and p olynomial time recogniz
able sets in R Karp ed Complexity of Computation SIAM AMS Pro
ceedings Volume
R Fagin Finite mo del theory a p ersonal p ersp ective Theoretical Com
puter Science
E Gr adel The expressive p ower of second order Horn logic Proceed
ings of the th Symposium on Theoretical Aspects of Computer Science
Lecture Notes of Computer Science Springer
E Gr adel Capturing complexity classes by fragments of second order
logic Proceedings of the th Structure in Complexity Theory Conference
IEEE
Y Gurevich Toward logic tailored for computational complexity in M M
Richter et al Computation and Proof Theory Lecture Notes in Mathe
matics Springer
N Immerman Expressibility as a complexity measure results and direc
tions Proceedings of the nd Structure in Complexity Theory Conference
IEEE
N Immerman Descriptive and computational complexity in J Hartma
nis ed Computational Complexity Theory AMS Applied Mathematics
Pro ceedings Vol
N D Jones A L Selman Turing machines and the sp ectra of rst order
formulas The Journal of Symbolic Logic No
C H Papadimitriou Computational Complexity Addison Wesley
L J Sto ckmeyer The p olynomial time hierarchy Theoretical Computer
Science
Kolmogorov Complexity
the Universal Distribution
and Worst Case vs Average Case
An algorithm can exhibit very di erent complexity b ehavior in the worst
case and in the average case with a uniform distribution of inputs One
well known example of this disparity is the QuickSort algorithm But it is
p ossible by means of Kolmogorov Complexity to de ne a probability
distribution under which worst case and average case running time for al l
algorithms simultaneously are the same up to constant factors
What is the di erence b etween the following two bit sequences
The rst sequence exhibits a certain pattern which is easy to notice and
which makes the sequence easy to describ e it consists of rep etitions of
The second sequence do es not have such an obvious pattern In fact the
second sequence was generated by ipping a coin If a pattern were detectable
in the second sequence this would b e merely coincidence In this sense the
second sequence is more random than the rst On the other hand in
the sense of probability theory each sequence is an equally likely result of
ipping a coin times namely each o ccurs with probability Thus
probability theory do es not provide the correct framework within which to
talk meaningfully ab out a random sequence
But consider now algorithms that generate each sequence In the rst
case
FOR i TO DO OUTPUT END
and in the second case
OUTPUT
If we abstract away the sp eci c length of our examples namely and
imagine instead an arbitrary value n then the length of the rst program
as text is O log n since we need log n bits to represent the numb er
n The second program on the other hand has length O n That is in
order to describ e this random sequence we are essentially forced to write
down the entire sequence which takes n bits On the basis of this intuition
Topic
we will de ne a sequence to b e random if any description of the sequence
requires essentially n bits
We will also consider the following variation Consider algorithms that
take an input y and pro duce an output x The minimal length of such a
program that generates x from y is a measure of the relative randomness of
x with resp ect to y or said another way it describ es how much information
ab out x is contained in y If in the examples ab ove we let n the length of the
sequence to b e generated b e the input to the program rather than a constant
within the program for example
INPUT n FOR i TO n DO OUTPUT END
then the rst program has length only O while the second program is still
of length O n
Now x a universal programming language say MODULA or Turing ma
chine Then K x j y denotes the length of the shortest program in this xed
programming language that on input y outputs x in nite time K x j y is
the conditional Kolmogorov complexity of x with resp ect to y The absolute
Kolmogorov complexity of x is K x K x j
log n c Returning to our rst example ab ove we obtain K
z
n times
j n c where c is a constant indep endent of n The and K
z
n times
constant c dep ends on the choice of programming language
In many cases the information y from which x is to b e generated will
b e precisely the length of x as in our example This has the following in
tuitive explanation a string x contains two kinds of information its inner
irregularity or randomness and its length If we reveal the information ab out
the length of x for free then we can concentrate solely on the randomness
of x If n jxj then K x j n is called the length conditioned Kolmogorov
complexity of x
Exercise Show that there are constants c and c such that for all strings
x and y K x j y K x c and K x jxj c C
Exercise Show that there is a constant c such that for all x K x j x c
C
Exercise Let b e the sequence consisting of the rst n binary digits
n
in the representation of the irrational numb er How large is K j n C
n
Since the choice of programming language seems to b e arbitrary one must
consider how this choice a ects the de nition In any universal programming
language one can write an interpreter u a universal Turing machine which
on input p y b ehaves just like program p in programming language P on
input y Let K K denote the Kolmogorov complexity with resp ect to the
P P
Kolmogorov Complexity
programming language P P If we assume that p is the shortest program
x j y jp j Since u p y x we get that generates x from y then K
P
K x j p y juj and since K p jp j c see Exercise we get
P P
K x j y jp j c juj K x j y c juj K x j y O
P P P
So values of K with resp ect to two di erent programming languages di er
by at most an additive constant As long as we are willing to ignore constant
additive factors we can consider the de nition of Kolmogorov complexity to
b e robust and sp eak of the Kolmogorov complexity of a string x
There can t b e to o many strings of low Kolmogorov complexity There
k k
are at most programs of length k so there can b e at most strings with
K x k The same is true of K x j y k for any y Altogether we see
n
that the strings of length n are partitioned as follows
at most string has K complexity
at most strings have K complexity
at most strings have K complexity
n
at most strings have K complexity n
In general the numb er of strings with K complexity k is at most
k k
Considered the other way around this means that
at least string has K complexity n
n
more than half of the strings have K complexity n
n
more than of the strings have K complexity n
n
more than of the strings have K complexity n
Exercise Give a lower b ound for the exp ected value of the K complexity
of a string chosen uniformly at random from among all strings of length n
C
Exercise Show that the function x K x is not computable C
Now we want to de ne the universal probability distribution at least on
strings of length n That is for each length n we de ne a probability distri
bution so that x is the probability of selecting the string x from among
the strings of length n jxj In order for this to b e a probability distribution
P
of course it must b e the case that x We want to de ne
fx jxj ng
K xjn
in such a way that x is prop ortional to i e for some constant
P
K xjn K xjn
d c x c This is p ossible as long as
fx jxj ng
for some constant d since then we can set c d So it su ces to show
P
K xjn
is b ounded ab ove by some constant The in the that
fx jxj ng
exp onent is there to give us convergence
Topic
Exercise Show this C
Many algorithms exhibit di erent b ehavior in the worst case than in the
average case at least when average case is interpreted under the uniform dis
tribution of strings of a given length A naive implementation of the Quick
Sort algorithm is an example in the worst case it requires n time but
on average it requires only O n log n Now we want to study the average
case complexity of QuickSort or any other algorithm under the probability
distribution
An interesting detail in the case of QuickSort is that the worst case o ccurs
when the input list is sorted in ascending or descending order Let the num
b ers to b e sorted b e f ng K n j n O so under the
distribution the probability of a sorted list is esp ecially large in fact it is a
K n jn O
constant indep endent of n n c c
From this it follows that the exp ected running time for QuickSort under
the probability distribution is
X
x T x n T n
QuickSort QuickSort
fx jxj ng
n n
where T x denotes the running time of algorithm A on input x and x is
A
a p ermutation of f ng So under the probability distribution the
average running time of QuickSort is as bad as the worst case namely n
We shall see that the probability distribution exhibits this same malevolent
b ehavior towards all algorithms namely that the average case running time
is within a constant factor of the worst case
In the counting arguments given ab ove to b ound the numb er of strings x
with K x n k or K x j y n k we fo cused our attention on strings
of only one length n Now we want to generalize this into the following very
useful and generally applicable theorem
Exercise Let M b e an arbitrary set of strings and let m jM j Show
k
that for every numb er k and every string y there are at least m
strings x in M for which K x j y log m k C
Next we want to discuss the prop erties of a Kolmogorov random string x
i e a string for which K x j jxj jxj We exp ect a random string in the
usual sense of probability theory to have roughly the same numb er of zero es
and ones This is also the case for a Kolmogorov random string
Exercise Explain why a Kolmogorov random string if it is su ciently
long cannot consist of n ones and n zero es
Hint How would this situation provide a means of describing the string
algorithmically with fewer than n bits C
Kolmogorov Complexity
Now let s take another lo ok at the universal probability distribution and
our QuickSort example QuickSort was intended to b e understo o d as a repre
sentative for an arbitrary algorithm in particular for one where the average
case and worst case running times di er under the uniform distribution
The only imp ortant prop erty of the sequence n for that argument
was that it is an input in fact the lexicographically least input on which
the algorithm exhibits its worst case b ehavior n in the case of Quick
Sort
Now let A b e an arbitrary algorithm that halts on all inputs Consider
the following program which on input n describ es i e outputs a certain
string of length n
INPUT n
w
FOR all y with jy j n in lexicographical order DO
v Running time of A on input y
IF v w THEN w v x y END
END
OUTPUT x
This algorithm has a xed length indep endent of the input n but dep endent
on the algorithm A let s call it c For every n let x b e the output of A on
n
K x jn
n
input n Then K x j n c and so x is prop ortional to
n n
c
This means that for some constant indep endent of n x
n
Furthermore by the construction of the algorithm for generating x the
n
running time of A on input x is maximal among inputs of length n That
n
is A exhibits its worst case complexity on input x
n
Exercise Now nish the pro of that under the universal probability
distribution on inputs of length n every algorithm that halts on all inputs
has an average case complexity that is up to constant factors identical to
its worst case complexity C
References
The idea of Kolmogorov complexity rst app eared in the s in pap ers by
Kolmogorov Solomono and Chaitin A comprehensive overview is given in
M Li and P Vit anyi An Introduction to Kolmogorov Complexity and Its
Applications nd edition Springer
The use of the universal probability distribution originally de ned by Levin
and Solomono to analyze the complexity of algorithms comes from
M Li and P Vit anyi Average case complexity under the universal distri
bution equals worst case complexity Information Processing Letters
Topic
M Li and P M B Vit anyi A theory of learning simple concepts and
average case complexity for the universal distribution Proceedings of the
th Symposium on Foundations of Computer Science IEEE
Further results can b e found in
P B Milterson The complexity of malign ensembles SIAM Journal on
Computing
Lower Bounds via Kolmogorov Complexity
The concept of Kolmogorov complexity can b e used to prove complexity
lower b ounds In many cases the pro ofs obtained in this way are much more
elegant or at least shorter than the original pro ofs In a few cases the
lower b ounds were rst achieved by means of Kolmogorov complexity
The metho d of using Kolmogorov complexity to establish lower b ounds works
as follows Supp ose we want to prove a lower b ound on the running time of
a Turing machine to p erform a certain task or a lower b ound on the size
of some other mathematical or computational ob ject Let x b e a su ciently
long Kolmogorov random string i e K x jxj Now we assume that the
lower b ound we are seeking is violated e g there is a Turing machine that
p erforms the given task more quickly than the stated b ound Then p erhaps
there is a way to use this Turing machine and p ossibly some additional
information to describ e the Kolmogorov random string x with fewer than
n jxj bits This would of course b e a contradiction and establish the lower
b ound
Our rst example of this approach comes from numb er theory rather than
computer science but the result has implications for computer science as well
We will use Kolmogorov complexity to prove that there are in nitely many
prime numb ers In fact the argument we will give can even b e extended to
prove a weak form of the Prime Numb er Theorem
So supp ose that there are only nitely many prime numb ers say p p
p Let n b e a su ciently large numb er so that the Kolmogorov complex
k
ity of the binary representation of n is not compressible i e K bin n
jbin n j log n Every natural numb er has a unique prime factorization
n n n
k
i e n p p p So the numb ers n n n co ded as bit strings
k
k
constitute a unique description of n and thus n or rather bin n can b e
reconstructed from the sequence n n n
k
For simplicity in this example we will do computations assuming that jbin n j
n
log n The exact value is actually jbin n j But since
blog nc n
additive constants do not play a role in this context this simpli cation do es not
change the validity of the argument
Topic
Exercise Use the observations just made to determine an upp er b ound
on K bin n that contradicts the choice of n made ab ove and thus completes
the pro of that there are in nitely many primes C
This argument can b e pushed farther The Prime Number Theorem is a
famous and hard theorem in numb er theory which says that n the numb er
of prime numb ers n is asymptotic to n ln n that is
n
n
ln n
With signi cantly simpler arguments using Kolmogorov complexity we
will show that for in nitely many n a lower b ound of
n
n
log n
holds This lower b ound is su cient for many applications
Let p b e the mth prime numb er It is su cient to show that for in nitely
m
many m p m log m
m
Exercise Show that this is indeed su cient C
Now let n b e a su ciently large natural numb er such that K bin n
log n Let p b e the largest prime numb er that divides n The numb er n can
m
b e algorithmically reconstructed from m and n p So a bit string that
m
co des these two numb ers is a su cient description of n
Exercise Give an algorithm that reconstructs n from the numb ers m
and k n p C
m
Thus
log n K bin n jenco ding of m and n p j
m
The problem is that we cannot simply write down the binary representations
of m and n p one after the other since then we would not know where
m
one numb er ended and the other b egan We must sacri ce some additional
and in this case very costly bits to make our enco ding such that it can b e
correctly deco ded
For a string w a a a a f g let w a a a a
n n n n
By means of this enco ding of w the end of the co de for w can b e recognized
by the nal Such a co de is called self terminating More formally an
enco ding scheme is self terminating if it can recognize its own end in the
following sense From any string of the form co de w v where v is arbitrary
it is p ossible to recover co de w algorithmically
We could co de the pair of numb ers as bin m bin n p But since
m
w j jw j this would cost log m log n p bits Plugging this into the ap j
m
proximation ab ove would yield log n K bin n log m log n log p
m
p
so p m i e n n So this metho d of co ding wastes to o many
m bits
Lower Bounds via Kolmogorov Complexity
If however we let
bin jw j w co de w
then we get another co ding of w that is also self terminating
Exercise Justify the claim just made Furthermore compute the length
of co de w as a function of w Use this to show that p m log m C
m
Exercise The improvement from w to co de w can b e iterated What
self terminating co de do es one get in this way What improved lower b ound
for n do es this yield C
It is interesting to note that any improvement in the length or rather
shortness of self terminating co des can b e translated directly into a sharp er
b ound in our Weak Prime Numb er Theorem
Exercise Show that if n is a su ciently large numb er and its binary
representation has high Kolmogorov complexity i e K bin n log n
then n cannot b e a prime numb er
Hint Use n n ln n C
This Kolmogorov metho d can also b e used in the context of circuit com
n
plexity Consider a b o olean function f from f g to f g The circuit
complexity of such a function is the smallest numb er of b o olean gates that
su ce to build a circuit that has n binary input gates and computes the
n
function f Since a truth table for such a function has rows by giving
n
a binary string of length the evaluation vector such a function can b e
uniquely characterized
Now we select an esp ecially di cult b o olean function f for which the
n
Kolmogorov complexity of this string is high i e K What is the circuit
complexity of the asso ciated b o olean function f Since a b o olean circuit
completely characterizes a function and in turn its enco ding as a binary
string we have the following inequality for the Kolmogorov complexity of
the binary string for f
K jthe shortest description of the circuitj O
How many bits do we need to represent a circuit consisting of g gates The
circuit can b e describ ed by listing all its gates Each gate is describ ed by
giving its typ e i e which b o olean op eration it computes and the numb ers
of the gates or inputs that feed into the gate This requires c log g n
bits for each gate
Topic
Exercise Let size f b e the circuit complexity of f that is the smallest
n
numb er of gates that can b e used to compute f Show that size f n
C
Exercise A formula is a circuit in which every gate has fan out Show
n
that the smallest formula for f has size log n
Hint A formula can b e enco ded esp ecially compactly without the use of any
parentheses in reverse Polish notation C
Finally we want to use the Kolmogorov metho d to prove that any one
tap e Turing machine that accepts the language
jw j
L fw w j w f g g
requires at least quadratic n time From this it follows or at least is
provable with only slight mo di cations that the languages
fw w j w f g g
and
R
fw w j w f g g
R
also require quadratic time In the latter a is the reversal of a
R
a a a a
n n
We use the concept of a crossing sequence Let M b e a Turing machine
and let i b e some xed b oundary b etween two adjacent cells on the tap e On
a given input x the crossing sequence for x and i is the sequence of states
in which the Turing machine nds itself as its read write head crosses p oint
i on the tap e We denote this crossing sequence by CS x i CS x i is
M M
an element of Z where Z is the set of states for the Turing machine
Lower Bounds via Kolmogorov Complexity
Example
i
z
time
z
z
z
CS z z z z
Exercise Why is
X
jCS x i j time x
M M
i
time x is the running time of M on input x C
M
By Exercise in order to show that time x n it is su cient
M
to show that a numb er of crossing sequences CS x i for x sp eci cally
M
n of them each have length at least n We will restrict our attention
to the crossing sequences that o ccur in the middle n p ositions of the input
where n is the total length of the input This is the p ortion that on input
jw j
w w consists entirely of s
Without loss of generality we can assume that all of our Turing machines
only enter the halting state when their read write head is on cell of the
tap e
Exercise Prove the following lemma
Let jxj i If CS xy i CS xz i then xy L M if and only if
M M
xz L M C
Exercise At what p oint in the pro of of the previous exercise do we
make use of the WLOG assumption ab ove C
Topic
Exercise Prove the following lemma
If i jxj jx j and c CS xy i CS x y i then c
M M
CS xy i CS x y i C
M M
jw j
Now let M b e a Turing machine that accepts L Let w w b e an input
for M and let jw j i jw j By the previous lemma we know that for
jw j
distinct w and w with jw j jw j the crossing sequences CS w w i
M
jw j
and CS w w i must also b e distinct
M
Exercise Why C
Exercise Describ e an algorithm that on inputs M m N i N
m i m and crossing sequence c all co ded in binary outputs the string
jw j
w of length m for which CS w w i c By the previous observation
M
w if it exists must b e unique C
We conclude therefore that the Kolmogorov complexity of w must satisfy
K w O log jw j jcj If w is chosen to b e Kolmogorov random here s
where the Kolmogorov argument comes in then K w jw j so jcj
jw j O log n
Exercise Complete the pro of that for every one tap e Turing machine
M that accepts L time x jxj C
M
As an aside we mention that the argument ab ove actually works not
only for a string w that is Kolmogorov random but also for any typical
string w The exp ected value under the usual uniform distribution for the
Kolmogorov complexity is E K w jw j cf Topic So even in the
average case time x jxj
M
References
For more on the Prime Numb er Theorem see a b o ok on numb er theory for
example
I Niven H S Zuckerman An Introduction to the Theory of Numbers
Wiley
The following references contain examples of using Kolmogorov complexity
to obtain lower b ounds
W J Paul Kolmogorov complexity and lower b ounds Foundations of
Computation Theory Akademie Verlag Berlin
W J Paul J I Seiferas J Simon An information theoretic approach to
time b ounds for on line computation Journal of Computer and System
Sciences
Lower Bounds via Kolmogorov Complexity
M Li P Vit anyi Applications of Kolmogorov complexity in the theory
of computation in A L Selman ed Complexity Theory Retrospective
Springer
M Li P Vit anyi An Introduction to Kolmogorov Complexity and its
Applications nd edition Springer
Topic
PAC Learning and Occam s Razor
Many algorithmic learning theories have b een develop ed The one which
is now most often considered originated with L Valiant and is called
PAC learning In this chapter we show an interesting connection b etween
PAC learning and the principal known as Occam s Razor
The philosopher and logician Wilhelm von Occam is credited
with the following principle which is usually referred to as Occam s Razor
If there are several hyp otheses that each explain an observed phe
nomenon then it is most reasonable to assume the simplest of them
i e the one most succinctly formulated
With this razor Occam cut out all sup er uous redundant explanations
This was directed in particular at the scholastics
If lo oked at dynamically the pro cess of formulating a hyp othesis that
explains previously made observations is very much like learning up on pre
sentation with additional observations it may b e necessary to revise the
previously held hyp othesis and replace it with a new one which explains the
new observations as well as the old and so on More precisely we are inter
ested in learning a concept by means of observations or examples which are
provided by a teacher at random along with a statement ab out whether or
not the examples b elong to the class that is to b e learned
The pro cess can b e understo o d as the principle of nding a hyp othesis
in the natural sciences Nature is directed internally by a function f
which is unknown to humans Furthermore examples x x are gener
ated according to some equally unknown probability distribution P As an
outsider one can only observe the pairs x f x x f x After a
while one forms a hyp othesis h which explains the observations made up
until time m i e h x f x h x f x
m m
Topic
Sketch
h
p
ppppppppppppppppppppppppppppppppppppppppppppppppppp
p
p p
p
p
p
p
p
p
p
p
p
p
p
p
p
p
p
p
p
p
p
p
p
p p
p p
p
p
p
p
x
p
p
f f x
i
i
p
p
p p
p
p p
p
p p
p
p p
p
p p
p
p p
p
p p
p
p p
p
p p
p
p p
p
x
p
p
i
p
p
p
p
p
p
randomly chosen
p
p
p
p
B
p
p
B
p
p p
according to P
p p
p p
p ppppppppppppppppppppppppppppppppppppppppppppppppppp
The principle of Occam s Razor suggests that one should cho ose h to b e
the simplest p ossible hyp othesis Simple could b e understo o d in the sense
of Kolmogorov complexity i e cho ose h so that K h is minimal A go o d
hyp othesis h is one that proves to b e valuable for future observations as well
P r h x f x should b e close to where x is chosen randomly according
to P The quintessence of the following investigations will b e that it is worth
lo oking for a simple hyp othesis in the sense of Occam s Razor since with
high probability such a hyp othesis will also b e a go o d hyp othesis
Now we want to formalize these ideas and capture them with a de nition
The x s will simply b e strings of a suitable length n The concept to b e
i
learned f and the hyp otheses h will then b e n place b o olean functions
De nition Valiant Let n A hyp othesis space H is a subset
n
of the set of all n place boolean functions A concept to be learned is any
n
function f H Let P be a probability distribution on f g A set of
n
examples is a nite set of pairs x f x x f x where the x s
m m i
are independently chosen according to the distribution P A hyp othesis h
H is consistent with a set of examples if h x f x h x f x
n m m
The function h H di ers from the concept f by at most if P r h f
n
where h f fx j h x f x g and x is chosen randomly according to P
An algorithm A that on input of a nite set of examples produces a consistent
hypothesis is cal led a learning algorithm
A family of hypothesis spaces H is cal led PAC learnable if there is
n n
a learning algorithm A and a polynomial m in the arguments n and
such that for every n every concept f H every probability
n
n
distribution P on f g every and every A on input of an
example set with m n elements chosen at random according to P
produces a hypothesis h H that with probability di ers from h by at
n
most
The abbreviation PAC stands for probabilistically approximately cor rect
PAC Learning and Occam s Razor
Sketch
hyp othesis space H
n
p
p
p
p
p
p
p
p
p
p
p
p
p
ppppppppppppp
set of all
n place
b o olean
functions
ppppppppppppppppppppppppppppppppppppppppppppppp r
concept to
p
p
p
b e learned f
p
p
p
p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p
p
set of all
p
p
p
p
p
hyp otheses
p
p
pppppppppppppppppppppppppppppppppppppp
set of all
consistent with
hyp otheses
a given
h with
example set
P r h f
Exercise Let A b e a learning algorithm that on input of a su ciently
large example set with probability pro duces a hyp othesis h that
di ers from f the concept b eing learned by at most
n
If a set of examples and a value x f g are chosen at random inde
p endently and according to P what is the probability that the hyp othesis
h pro duced by A on this example set agrees with the concept b eing learned
i e with what probability is h x f x C
In the diagram a small oval represents the set of all hyp otheses that are
consistent with a xed example set Such an example set consisting of m
examples is chosen at random according to P The concept f is of course
always among the consistent hyp otheses We will see that the larger m is the
more likely it is that al l consistent hyp otheses lie in the neighb orho o d of f
so that in the diagram the oval is entirely contained in the circle In this case
any learning algorithm has a high probability of pro ducing a hyp othesis that
di ers from f by no more than since by de nition a learning algorithm
always pro duces a consistent hyp othesis
Let s approximate the probability p that after a random choice of an
example set of size m there is a consistent hyp othesis that is not in the
neighb orho o d of f
X
p P r h is consistent
h H
n
P r h f
X
m
h H
n
P r h f
Topic
m
jH j
n
Exercise Under the assumption that P is the uniform distribution on
n
f g give an upp er b ound for the numb er of h H with P r h f
n
C
m
If jH j then we are guaranteed that every learning algorithm
n
on example sets of size m has probability at least of pro ducing a
hyp othesis that di ers from f by at most We can re express this as a
condition on m
m
jH j
n
m
m log jH j log
n
log
To satisfy this inequality since ln x x it is su cient to cho ose
m so that
ln
log jH j log m
n
This term is p olynomial in and log jH j The dominant term is
n
log jH j If H is the set of all n place b o olean functions on f g then
n n
n
n
jH j so log jH j For the p olynomial b ounds that are required
n n
in the de nition it is necessary to restrict the set of relevant hyp otheses
In cases that o ccur in practice it is often the case that there are many
n
fewer than p otential p ossible hyp otheses Sometimes it is the case that the
learning algorithm is guaranteed to pro duce a hyp othesis that is shorter than
n
the typical hyp othesis which has length as if it were following Occam s
razor In order to sp eak of the length of a hyp othesis we must x some
enco ding scheme for example b o olean circuits In pure form we can identify
the length of the hyp othesis with its Kolmogorov complexity
De nition Blumer Ehrenfeucht Haussler Warmuth A learning
algorithm is said to be an Occam algorithm if for every n with respect to
the hypothesis space H on input of an example set of su ciently large size
n
m the hypothesis produced by the algorithm always has length p n m
where p is a polynomial and
This means that we can restrict our hyp othesis space to include only
p n m
hyp otheses of length p n m and this set has size The
term m in this expression has an interesting interpretation an Occam
algorithm must p erform a certain form of information compression so that
the numb er of examples m is sub linear in the size of the hyp othesis space
p n m
Exercise Show that from jH j it follows that for PAC
learnability it is su cient to cho ose m to b e p olynomial in n C
So we have shown
PAC Learning and Occam s Razor
Theorem Blumer Ehrenfeucht Haussler Warmuth If for a family
H fH ng of hypothesis spaces there is an Occam algorithm then H is
n
PAC learnable
We want to explain this concept using an example from the literature cf
Valiant Let DNF denote the set of all formulas in disjunctive normal
n k
form in the variables x x such that all of the monomials consist of at
n
most k literals That is a formula f in DNF has the form
n k
m k
x x g z with z fx x f x x
n ij ij n n
i j
where z denotes that p osition j of clause i remains un lled so that
ij
clause j contains fewer than k literals In what follows we will identify for
mulas with the functions they de ne
k
n
Exercise Show that in DNF there are at most di erent
n k
functions C
The numb er of functions in the hyp othesis space H DNF is dras
n k
n
tically less than the function log jH j is b ounded by the p olynomial
k
n By the preceding discussion to demonstrate the PAC learnability
of DNF it is su cient to give any learning algorithm and to cho ose the
n k
k
size of the example set to b e m n log The only thing
that remains then is to give an algorithm that for any example set pro duces
a consistent hyp othesis
Although the PAC learnability of DNF is often discussed in the liter
n k
ature in connection with Occam s Razor this example is really to o simple
The hyp othesis space is so small that the length of every hyp othesis h sat
k k
is es K h n p n m with p n n and
So every learning algorithm i e any algorithm as long as it is merely able
to pro duce something consistent with the examples is an Occam algorithm
The requirement that the hyp othesis have length sub linear in m do es not
play a role at all
Here is a simple learning algorithm
INPUT example set f x f x x f x g
m m
h set of all monomials with k literals
FOR i TO m DO
IF f x THEN
i
h h fm j m is a monomial in h with m x g
i
END
END
OUTPUT h
Exercise Show that this algorithm always pro duces a hyp othesis that
is consistent with the example set C
Topic
In this example it is imp ortant that the function f is also in the hyp othesis
space H even though all of the de nitions and theorems and the algorithm
n
are applicable even when f H For example let f b e the parity function
n
The results in Topic will imply that this function cannot b e approximated
by a low degree p olynomial So the naive algorithm just given must necessar
ily pro duce an inconsistent hyp othesis when applied to the parity function
It should b e noted that in the literature the complexity of the learning
algorithm relative to the size of the input example set is also an issue and
is usually included in the de nition The learning algorithm should run in
p olynomial time just as it do es in our example Since the complexity of
the learning algorithm is irrelevant for Theorem we left this out of our
de nition
In a further departure from the literature we have implicitly assumed
that the hyp otheses are b o olean functions This seemed to us to b e consistent
with the usual practice in complexity theory where every nite mathematical
ob ject is co ded as a string
References
L G Valiant A theory of the learnable Communications of the ACM
A Blumer A Ehrenfeucht D Haussler M K Warmuth Occam s Razor
Information Processing Letters
A Blumer A Ehrenfeucht D Haussler M K Warmuth Learnability
and the Vapnik Chervonenkis dimension Journal of the ACM
R Board L Pitt On the necessity of Occam algorithms Proceedings of
the nd Symposium on Theory of Computing ACM
D Angluin Computational learning theory survey and selected bibli
ography Proceedings of the th Symposium on Theory of Computing
ACM
B Natara jan Machine Learning Morgan Kaufmann
M Anthony N Biggs Computational Learning Theory Cambridge Uni
versity Press
M Li P Vit anyi An Introduction to Kolmogorov Complexity and its
Applications nd edition Springer
Lower Bounds for the Parity Function
In their pioneering work of Furst Saxe and Sipser intro duced the
metho d of random restrictions to achieve lower b ounds for circuits The
parity function cannot b e computed by an AND OR circuit of p olynomial
size and constant depth
By the parity function PARITY we mean the in nite sequence of functions
n
par f g f g n with
n
n
X
mo d x par x x
i n
n
i
The question of existence or non existence of combinatorial circuits for the
parity function has b een investigated for a long time This is b ecause many
other functions can b e expressed in terms of the parity function
We are interested here in circuits of a sp eci c kind namely circuits that
consist of AND and OR gates with unbounded fan in and have inputs lab eled
with variables x or their negations x
i i
Example
AND
T
T
T
AND
OR
A
a
A
a
a
A
a
a
a
A
a
a
A
a
x x
x x
Gates of the same typ e that follow one directly after the other can b e
combined into a single gate without changing the function computed by the
circuit For example the circuit ab ove is equivalent to the following circuit
Topic
AND
A
A
A
A
OR
A
a
a
A
a
a
a
A
a
a
A
a
A
a
x x
x x
So circuits of this typ e can b e put in a certain normalized leveled form by
arti cially lling in with gates of fan in
AND
H
H
H
H
H
H
H
OR OR OR OR
a
a
a
a
a
a
a
a
a
x x
x x
Now we have on the rst level only OR gates on the second level only AND
gates and if necessary we can continue alternating b etween levels of OR
and AND gates
Exercise Why can all b o olean functions on n variables b e computed
by a circuit with only levels a depth circuit What is the size numb er
of gates of such a circuit in general C
The distributive laws for b o olean algebra state that
x y z x y x z
x y z x y x z
Exercise Use the distributive laws to transform the depth AND OR
circuit ab ove into a depth OR AND circuit C
Exercise Supp ose we have a depth AND OR circuit in which all the
OR gates in the rst level have fan in c and the AND gate in the second level
Lower Bounds for Parity
has fan in d After using the distributive laws to transform such a circuit into
a depth OR AND circuit what is the fan in of the OR gate What is the
fan in of the AND gates C
Note that the condition fan in d for the AND gate in the previous
exercise can b e replaced by the condition that the numb er of variables up on
which the value of the AND gate dep ends is at most d
Exercise Supp ose we have a circuit for the complement of parity par
n
that has depth t and size g Show that there is a circuit with t levels and g
gates that computes par C
n
Exercise Supp ose we have a circuit for par Now set some of the x s
i
n
to others to and leave the remaining ones as variables Any OR gate
that now has an input with value or any AND gate with an input with
value can b e eliminated and replaced with or etc Show that the
resulting reduced circuit again computes either parity or its complement on
a smaller numb er of variables C
Now we want to investigate whether PARITY can b e computed with cir
cuits of constant depth and polynomial size In other words the question
is Is there a constant t and a p olynomial p such that for all n the b o olean
function par can b e computed with a depth t circuit that has at most p n
n
gates Note that t is not allowed to grow with increasing values of n but must
remain constant
For t at least we can show that this is not p ossible
Exercise Prove that PARITY cannot b e computed by p olynomial size
depth OR AND circuits
Hint First show that every AND gate on level in such a circuit must have
n
exactly n inputs From this conclude that the circuit must have at least
AND gates C
Exercise Show that PARITY can b e computed by p olynomial size cir
cuits of depth O log n
Hint As a rst step construct a p olynomial size O log n depth circuit of
XOR gates C
k
the class of all b o olean functions that can b e computed We denote by AC
k
by p olynomial size depth O log n circuits with AND and OR gates of
We want to show unb ounded fan in Exercise shows that PARITY AC
that PARITY AC Note that O log n O Exercise is a rst
step in that direction the base case of an induction
The pro of of this is due to Furst Saxe and Sipser who made use of
at least in this context a new technique of random restrictions which
has since b een used rep eatedly even in other contexts The result was later
Topic
improved from not p olynomial size to at least exp onential size by Yao
and then by H astad whose pro of is regarded as the signi cant breakthrough
We will discuss here the weaker Furst Saxe Sipser version b ecause it is
somewhat simpler and provides a go o d opp ortunity to work through the
technique of random restrictions For this we will need a bit of probability
theory If we conduct a random exp eriment in which there are two p ossible
outcomes success and failure which o ccur with probability p and q p
and if we rep eat this exp eriment n times indep endently then the probabil
n
k n k
ity of obtaining exactly k successes is just p q This is the binomial
k
distribution
Exercise Let X b e a random variable that counts the numb er of
successes in n trials Compute or lo ok up in a b o ok on probability theory
the exp ected value E X and the variance V X for this random variable
C
Exercise Prove Chebyshev s inequality
P r jX E X j a V X a
Hint Use Markov s inequality See Exercise C
Exercise Supp ose n and p Use Chebyshev s inequality
to give an upp er b ound for P r X C
Exercise Prove another inequality for the binomial distribution
a n
P r X a p
where X is as ab ove This inequality is only useful when the right side is
less than C
In order to show that PARITY cannot b e computed by p olynomial size
constant depth circuits it is su cient to prove the following claim
Claim t c p olynomials p PARITY cannot b e computed using a depth t
circuit of size p n that has input fan in c i e constant fan in on level
Theorem PARITY AC
Exercise Why do es this theorem follow directly from Claim C
Lower Bounds for Parity
Proof of Claim Claim was proven in Exercise for the case t
There it was shown that the input fan in cannot b e constant nor can the size
of the circuit b e p olynomial
Supp ose the claim is false Then there is some t such that parity
can b e computed by p olynomial size depth t circuits with constant fan in on
level Let t b e the least such Let k b e strictly larger than the degree of
the p olynomial that b ounds the size of the circuits and let c b e the constant
that b ounds the input fan in We will use this to show that there is also a
p olynomial size circuit family of depth t with constant input fan in that
computes parity It is worth noticing that b oth the degree of the p olynomial
b ounding the size and the constant b ounding the input fan in will increase
when we reduce the depth This will contradict the minimality of t and
establish the claim
The strategy for pro ducing the new circuits is the following Let
S S S b e the supp osed depth t circuit family for PARITY We will
for S S a new depth t circuit in the circuit family S construct S
n
PARITY by taking an element of the S sequence with more than n variables
say S and then as in Exercise replacing the appropriate n n
n
of the variables with the constants and leaving a circuit with n input
variables This circuit will b e constructed in such a way that we can use the
distributive laws Exercises and to reverse the order of the AND
and OR gates on levels and without increasing the size of the circuit ex
ponential ly as happ ens in general see Exercise For this it is su cient
to show that each gate on level dep ends on only a constant numb er of input
variables This guarantees that after application of the distributive laws the
new circuits will have constant input fan in see Exercise After this
transformation the new circuit will have the same typ e of gates on levels
and so these can b e collapsed to a single level leaving a depth t circuit
will b e quadratic in the original size so The size of the resulting circuit S
n
the degree of the p olynomial that b ounds the size doubles
The word appropriate in the preceding paragraph is loaded Just how
are we to nd an appropriate constant substitution Here is the new idea
try a random substitution usually called a random restriction If we can
show that the probability of getting an appropriate substitution is p ositive
then we can conclude that one exists
We use the following random restriction For each variable x indep endent
i
of the others we p erform the following random exp eriment which has three
p ossible outcomes
p
n the variable x remains with probability
i
p
n
with probability the variable x is set to
i
p
n
with probability the variable x is set to
i
How do es one arrive at these probabilities A b oundary condition is that
the probabilities for and must b e equal so that in the following discus
Topic
sion we will b e able to exchange the roles of AND and OR by symmetry
In addition it turns out to b e useful to set the probability that a variable
remains as small as p ossible On the other hand this probability can only b e
p olynomially smaller than n for reasons which we will discuss b elow
r
Let r denote a random restriction and let x b e the result of this restriction
i
r r
on x so x fx g Let S denote the circuit S after applying the
i i
i
r
random restriction r By Exercise it is clear that S is once again a
parity function or the complement of parity on fewer variables In any
case by Exercise there is a circuit for parity with the same depth and
size
r
By Exercise the exp ected value of the numb er of variables in S is
n
p p
p p p
n The variance is n n The numb er of variables n
n n n
actually remaining must b e in inverse p olynomial relationship to the original
numb er of variables and not decrease exp onentially otherwise we will not b e
able to b ound the size of the resulting circuit with a p olynomial in the numb er
of remaining variables The following exercise shows that this happ ens with
high probability
Exercise Use Chebyshev s inequality see Exercises and
p
r
p
C to show that P r there are fewer than n variables in S O
n
n
p
n variables This means that with high probability there are at least
remaining In what follows we are only interested in random restrictions that
leave at least that many variables
Exercise At rst glance it is not yet clear how to pro duce a new
without any gaps For every n the random restriction S S sequence S
applied to the circuit S pro duces a circuit that has on average n inputs
n
and with high probability at least n inputs Explain how to convert the results
without any gaps C S S of this pro cess into a sequence S
Next we will give upp er b ounds for the probabilities that our random
restriction has certain undesirable prop erties If we succeed in showing as in
Exercise that each of these probabilities can b e b ounded ab ove by a
function that approaches as n then the probability that a random
restriction has any of these nitely many prop erties will for large enough
n b e less than From this it follows that for every large enough n there
must exist a restriction that has only desirable prop erties
First we show that with high probability the gates on level after the
restriction dep end on only a constant numb er of inputs For this argument
we can assume that the gates on the rst level are OR gates and thus the
gates on the second level are AND gates If the situation is reversed then by
duality we can rep eat the argument given b elow exchanging AND and OR
x and and x and
i i
For our probability approximations we will take an arbitrary xed AND
gate on level and show that the random restriction has an undesirable e ect
Lower Bounds for Parity
in this case dep endence on to o many variables with probability at most
k
Since altogether there are only O n gates it follows that the O
k
n
probability of this undesirable e ect o ccurring at any AND gate on level is
k
at most O So with high probability al l of the AND gates n O
k
n
n
on level have the desired prop erty after the random restriction
Now we use induction to prove the following claim
Claim For every AND OR circuit that has input fan in at the OR gates
at most c there is a constant e e dep ending only on c such that the
c
probability that the AND gate after a random restriction dep ends on more
than e variables is at most O
k
n
Proof of Claim The pro of of Claim is by induction on c The base case
is when c In this case there are no OR gates only the one AND gate
We distinguish two cases dep ending on whether the AND gate has large or
small fan in
Case B The fan in of the AND gate is at least k ln n
In this case it is very likely that there is at least one input to the AND
gate that has b een set to by the random restriction in which case the AND
gate do es not dep end on any of the variables that remain after the random
restriction
Exercise Show that in this case
P r AND Gate is not set to O C
k
n
Case B The fan in of the AND gate is less than k ln n
In this case it is very likely that the random restriction sets all but
constantly many variables to constants This is at least plausible since
the exp ected value E X for the numb er of remaining variables satis es
p
E X k ln n n as n see Exercise
Exercise Show that in this case
P r the AND Gate dep ends on more than inputs O
k
n
Note that it do es not matter what constant is inserted in place of and
that this may dep end on k
Hint Our solution works with the constant k Use Exercise C
Now we come to the induction step We assume that e exists and show
c
that e exists Once again there are two cases The inductive hyp othesis only
c
plays a role in the second case
Case I Before the random restriction the AND gate on level has at least
c
d ln n OR gates b elow it with disjoint input variables where d k
Topic
In this case we will show that it is very likely that after the random
restriction one of the OR gates will have had all of its inputs set to which
causes the AND gate to also have the value In this case the AND gate do es
not dep end on any inputs and the claim is established
Exercise Show that in this case
P r the AND Gate is not O
k
n
Hint Rememb er that all of the OR gates on level have by assumption at
ln b ln a
most c inputs Also the following relationships might b e useful a b
and ln x x C
Case I Before the random restriction the AND gate on level has less
c
than d ln n OR gates b elow it with disjoint input variables where d k
In this case cho ose a maximal set of OR gates with disjoint variables Let
H b e the set of variables that o ccur in these OR gates
Exercise How large can jH j b e C
It is imp ortant to note that in each of the OR gates at least one variable
from H o ccurs
Exercise Why is this the case C
jH j
There are l assignments for the variables in H If we plug any one of
these assignments into the original AND OR circuit then at least one input
to each OR gate disapp ears So after such plugging in all of the OR gates
have fan in at most c Now we can apply the induction hyp othesis Let
A A b e the l circuits that arise in this way one for each assignment to
l
r
the variables in H The probability that the function value of A i e after
j
the random restriction dep ends on more than e variables is b ounded
c
ab ove by O
k
n
The function f computed by the AND OR circuit can b e completely sp ec
i ed in terms of the A s As an easy example supp ose that H fx x g
j
so l then
f x x A x x A x x A x x A
From this it follows that the probability that f dep ends on more than l e
c
variables is b ounded ab ove by l O
k
n
Instead of using the AND OR circuit to determine the dep endency of f
after the random restriction on the input variables we nd it advantageous
to work with the equivalent representation in terms of the A s just given
j
With high probability after the random restriction H will only consist of
constantly many variables so that the numb er of remaining terms in our
expression for f will also b e constant Let h b e the random variable that
Lower Bounds for Parity
indicates the numb er of remaining variables in H after the random restriction
p
n and Once again we are dealing with a binomial distribution with p
we can approximate as we did in case of the base case of the induction
Exercise Show that P r h cd k O
k
n
Hint Feel free to use a larger constant if necessary This size of the constant
is not at issue Use Exercise C
Thus with high probability h cd k and when this is the case then
h cd k
our representation of f consists of at most m terms that are
not identically Now we put together all the probability approximations If
we let e m e then we get
c c
P r f dep ends on more than e variables
c
P r h cd k
m P r a xed A dep ends on more than e variables
j c
m O O
k k
n n
O
k
n
This completes the pro of of Claim ut
Now the pro of of Claim is complete as well There must exist a restric
tion that leaves enough variables however each AND OR circuit on levels
and dep ends on only constantly many of these variables So with only
constant cost we can apply the distributive laws and get the second level to
b e an OR level and the rst level an AND level But now the adjacent ORs
on levels and can b e combined leaving us with a circuit of p olynomial
size depth t and constant input fan in contrary to our assumption that
t is the minimal depth for which this was p ossible ut
References
Furst Saxe Sipser Parity circuits and the p olynomial time hierarchy
Mathematical Systems Theory
H astad Almost optimal lower b ounds for small depth circuits Proceed
ings of the th Annual Symposium on Theory of Computing ACM
Topic
The Parity Function Again
The lower b ound theory for circuits received an additional b o ost through
algebraic techniques in combination with probabilistic techniques that
go back to Razb orov and Smolensky
The result of Furst Saxe and Sipser that the parity function cannot b e
computed by AC circuits AND and OR gates constant depth unb ounded
fan in and p olynomial size was later proven in a completely di erent way by
A Razb orov and R Smolensky Now we want to work through their metho d
of pro of and some related results The technique is algebraic in nature but
also uses a probabilistic argument The argument works as follows
First we show that every function f computed by AC circuits can b e
approximated by a p olynomial p of very low degree Approximation in
n
this case means that for almost all n tuples a a f g
n
f a a p a a
n n
Then we show that the parity function cannot b e approximated in this
sense by a p olynomial of low degree
We b egin with step Clearly the AND function
AND
So
S
S
x x
n
Q
n
can b e represented as the p olynomial x x x
n i
i
Exercise Using x to represent the NOT function and DeMorgan s
laws give a p olynomial representation of the OR function C
The problem with this is that in general the p olynomials have degree n
that is they contain monomials that mention all of the x s This can b e
i
greatly improved by a probabilistic metho d that go es back to Valiant and
Vazirani We construct a random p olynomial as follows Let S f ng
Furthermore let S S b e chosen randomly so that each element j S
i i i
Topic
is in S with probability Now consider a sequence S S S
i log n
P
x generated as just explained Let q denote the random p olynomial
j i
j S
i
which has degree
Now if OR x x this means that all x s have the value
n i
Thus all the q s are also and therefore the p olynomial p where
i
Q
log n
p q is also This p olynomial has degree O log n
i
i
If on the other hand OR x x then there is at least one
n
x We will show that in this case the probability is that one of
i
the p olynomials q has the value exactly So in this case P r p
i
Example
S
S
S
S
h hx x h h x h x x h
x x x x x x x x x x
In this example we have variables of which x x x x x have the
value One p ossible realization of the random subsets S is indicated Note
i
that in our example S has exactly one variable with the value
So the p olynomial p approximates the OR function in a certain sense
The success rate is still not very high however it is only But this can b e
signi cantly improved by using new indep endent random numb ers to gener
ate additional p olynomials p say p p p and then using the p olynomial
t
p p p which has degree O t log n for our approximation
t
Exercise What is the error probability of the p olynomial p p p
t
How large must t b e to get an error probability b elow a given constant
C
Exercise Construct the corresp onding p olynomial for the AND
function C
We still need to show that for any choice of a non empty subset T of S
corresp onding to the variables that are true the probability is at least
that there is at least one i f log n g such that the size of T S is
i
exactly To approximate this probability we partition the event space into
various cases and then compute the probability separately for each case
Case For all i f log n g jT S j
i
Exercise Give an upp er b ound for this for us bad case C
Parity Again
Case There is an i f log n g with jT S j
i
Case A jT S j jT j
Case B jT S j jT j and there is an i f log n g
with jT S j
i
Under the assumption that we are in case B let i b e such that
jT S j and jT S j The probability for jT S j
i i i
under the condition that jT S j and jT S j t is
i i
t
t
t
t t
t t
t
Exercise Show that it now follows that
P r there is an i with jT S j C
i
Next we want to show how to simulate an AC circuit with size s and
depth t using our p olynomials so that the error probability is at most For
this we use the p olynomials ab ove for the gates but with error probability
for each gate s
Exercise What is the degree of the resulting p olynomial as a function
of s and t O notation su ces If s is p olynomial in n and a constant
what sort of function is this constant logarithmic p olylogarithmic linear
p olynomial exp onential etc C
In summary for every b o olean function f that can b e computed by AC
circuits a p olynomial p can b e randomly generated that has very small
n
degree and such that for any a a f g the probability is at
n
least for example that f a a p a a From this
n n
we conclude that there must b e at least one choice of a xed p olynomial
p for which f a a p a a for all a a S where
n n n
n
jS j
Exercise Justify the last claim C
Now we want to consider more carefully the p ossible representations of
the b o olean values TRUE and FALSE In the p olynomial approximations
ab ove we have tacitly identifying TRUE with and FALSE with as is
usually done For what we are ab out to do it will b e more advantageous to
use the so called Fourier representation which identi es TRUE with and
FALSE with
Exercise Find a linear function that maps to and to What is
its inverse C
Topic
If we apply this function to our p olynomial p we get a p olynomial
n
q y y p x x that for strings in
n n
n
f g correctly simulates f transformed to use f g and has the
same degree as p
Supp ose now that the parity function is in AC Then there must b e such
n n
a function q for parity So for strings in f g q y y
n
Q
n
y That is after this transformation the parity function corresp onds
i
i
exactly to multiplication
Exercise Why C
Now we prove the following
p
n that correctly repre Lemma There is no polynomial of degree
Q
n
n n
y for strings in f g sents the function
i
i
Corollary PARITY AC ut
p
Proof of the lemma Let q y y b e a p olynomial of degree n that
n
Q
n
n n
y for strings in f g Let correctly represents the function
i
i
Q
n
n n
S f y y f g j y q y y g So jS j
n i n
i
We can assume that the p olynomial q is multi linear that is no variable has
an exp onent larger than
Exercise Why C
The vector space L S over R which consists of all linear combinations
of vectors in S has dimension jS j Similarly POL the set of all n variate
p
n is a vector space with the usual multi linear p olynomials of degree n
p olynomial addition which do es not increase the degree and multiplication
by scalars in R A basis for this vector space is the set of all monomials
Q
p
x with jT j n n Thus the dimension of this vector space is
i
i T
p
P
n n
n
i
i
n
Exercise Show that this sum is strictly smaller than jS j
C
Now we show that L S can b e emb edded by a linear transformation h a
vector space homomorphism as a subspace of POL It is su cient to show
how the basis vectors in L S the elements of S are mapp ed by h Let
s S and let T b e the set of indices in s where a o ccurs If jT j n
Q
y if jT j n then h s is the p olynomial then h s is the monomial
i
i T
Q
p
n and therefore y which has degree at most n q y y
i n
i T
is in POL
Exercise Convince yourself that for all y y S
n
Y Y
y q y y y
i n i
i T
i T
Parity Again
Therefore the p olynomials h s are linearly indep endent in POL C
Since the p olynomials h s are linearly indep endent in POL dim POL
dim L S This yields the contradiction
n n
dim L S dim POL
and completes the pro of of Lemma ut
Exercise Improve the result that PARITY AC to show that any
log n
C p olynomial size circuit for par must have depth at least
n
log log n
We can use the result that parity is not in AC to show that this is also
the case for other b o olean functions for example for majority For this we
intro duce a notion of reducibility that is tailored to the de nition of the class
AC
n
A family of b o olean functions F f f f where f f g
n
f g is AC reducible to a family G g g g if there is a constant
d and a p olynomial p such that for every n there are circuits for f that have
n
depth at most d and size at most p n that may consist of AND gates and
OR gates with unb ounded fan in and also g gates i arbitrary
i
It should b e clear that if F is AC reducible to G and G AC then
F AC as well
Exercise Why C
Examples for such families of functions are
PARITY par par par
and
MAJORITY ma j ma j ma j
where ma j x x if and only if for at least n of the inputs x we
n i
have x
i
Exercise Show that PARITY is AC reducible to MAJORITY C
From this it follows immediately that
Theorem MAJORITY AC ut
In fact we can use this same technique to show that every symmetric
boolean function is reducible to MAJORITY A symmetric function is one
that is invariant under p ermutations of the input variables that is its value
P
n
x Such a function can dep ends only on the sum of the input variables
i
i
b e completely sp eci ed by a value vector of the form f f f where
n
P
n
each f gives the value of the function when x k So there are only
k i
i
n
distinct symmetric functions of n variables
Topic
Furthermore one can show that all symmetric functions are in NC and
that ma jority is not AC reducible to parity In other words ma jority can
not b e computed by circuits with constant depth p olynomial size and un
b ounded fan in over f g
The following sketch shows the situation
p p p p p p p p p p p p p p p p p p p
NC
p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p t
MAJORITY
ppppppppppppppppppppppppppppppppp t
PARITY
p p p p p p p p p p p p p p p p p p p p p p p p p p p p p
symmetric functions
p p p p p p p p p p p p p p p p p p p p p p p
AC
The pro of metho d in which a circuit in this case a so called p erceptron
is describ ed via a p olynomial and the degree of this p olynomial is compared
with the least p ossible degree of a p olynomial that represents the parity
function in order to show that the prop osed circuit cannot compute parity
was rst used by Minsky and Pap ert By a perceptron we mean a depth
circuit that has a threshold gate also called a McCul loch Pitts neuron for
its output gate This means for now that the gates on the rst level may
compute any b o olean functions of the inputs From the binary output values
a f g of these functions a weighted sum is then computed each input
i
to the threshold gate b eing assigned the weight w R Finally this weighted
i
sum is compared with a threshold value t R If the sum is at least as large
as the threshold then the p erceptron outputs otherwise it outputs That
is the value of the p erceptron s output is given by
P
w a t if
i i
i
otherwise
The threshold gate can have an unb ounded numb er of inputs
P
P
P
a
P
x
P
P
P
p p p p p p p p pppppppppppppppppppppppppppppppppppp
p p
P
p
p
x
P
p
p
p
p
P
Pq
w
P
p
p
P
p
p
p
p P
a
p
p
p p
w
p p
p
p
p
p
p
p
p
p
w
k p
p
p
p
p
p
t
p
p
p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p
P
P
x
P
n
a
k
threshold gate
Parity Again
The intended application of such circuits as classi ers in the area of pat
tern recognition and the biological mo del of a neuron makes it reasonable to
consider the case where the gates on level do not dep end on all of the in
puts x x but only on a strict p ossibly very small subset of the inputs
n
The p erceptron is also attractive since one had hop ed that by successively
varying the weights such circuits could learn any b o olean function Minsky
and Pap ert showed however that such circuits are not capable of computing
parity on x x It is this result that we consider next This result has
n
b een credited with bringing to a halt research or at least funding of research
in the area of neural nets which in the s had just b egun to ourish This
lasted for ab out years until recently when despite this negative result
the value of neural nets was again recognized
Exercise Convince yourself that the mo del just describ ed is equiva
lent to a mo del in which all gates on the rst level are AND gates but in
addition to the input variables x x their negations are also available
n
The numb er of inputs to an AND gate is still required to b e less than n
Sketch
And
P
x
P
P
P
pppppppppppppppppppppppppppppppppppp p p p p p p p p
p
P
p
p
P
p
p
p P
p
p
p
Pq
w
p
p
p
p
x
n
p
p
p
p
And
p
p
w
p
p
p
p
p p
x
p
p
p
p
w
m
p
p
p p
p
p
t
p
p
p
p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p
And
x
threshold gate
n
C
Every AND gate can b e describ ed as a p olynomial over f g namely
the pro duct of terms that are x if the variable x is an input to the gate and
i i
x if its negation is an input to the gate
i
Example The AND gate that conjuncts x x and x is represented by the
p olynomial
x x x x x x x x x x x
It is clear that on inputs from f g this multi linear p olynomial rep
resents precisely the correct value of the AND gate Furthermore the total
degree of this p olynomial corresp onds to the numb er of inputs to the AND
gate and must therefore b e less than n Let f i m b e the p oly
i
nomial that represents the ith AND gate of a p erceptron These p olynomials
are then added weighted according to the threshold gate
Topic
m
X
w f
i i
i
This is a multi linear p olynomial in all variables x x and still has
n
total degree n i e no monomial in the sum can mention n literals If
such a circuit were capable of computing parity then there would b e some
real constant t such that
m
X
w f t par x x
i i n
n
i
In other words the sign of the p olynomial p with
m
X
w f t p x x
i i n
i
determines the parity of the input bits x x
n
The next step makes use of the fact that parity is symmetric This means
that for any p ermutation
par x x par x x
n n
n n
Now we build the p olynomial
X
q x x p x x
n n
This is a multi linear p olynomial of total degree n the sign of which also
represents the parity function The sum is over all p ermutations of the
n element set fx x g Furthermore all monomials that have the same
n
numb er of variables must o ccur with the same co e cients
Exercise Justify the last sentence C
So the p olynomial q can b e written as
s
X
q x x t
n i i
i
where s n are appropriate co e cients and the terms t sum up
s i
all monomials with i variables
Y X
x t
j i
j S
S fx x g
n
jS j i
By the previous exercise the p olynomial q dep ends only on x x
n
and not on the particular tuple x x Let r b e a p olynomial in one
n
variable de ned by
Parity Again
r x x q x x
n n
x x
n
Since t is just
i
i
s
X
k
r k
i
i
i
This is a univariate p olynomial of degree s n with the prop erty that
r k exactly when k is even for k f ng
Sketch for n
k
r k
But such a p olynomial that has n zero es must have degree at least n This
is a contradiction which proves that no p erceptron can compute parity ut
References
Valiant Vazirani NP is as easy as detecting unique solutions Theoretical
Computer Science
A Razb orov Lower b ounds on the size of b ounded depth networks over a
complete basis with logical addition Mathematical Notes of the Academy
of Sciences of the USSR
R Smolensky Algebraic metho ds in the theory of lower b ounds for
Bo olean circuit complexity Proceedings of the th Annual Symposium
on Theory of Computing ACM
N Alon J H Sp encer The Probabilistic Method Wiley Chapter
R Beigel The p olynomial metho d in circuit complexity Structure in
Complexity Theory Conference IEEE
M L Minsky S A Pap ert Perceptrons MIT Press
J Aspnes R Beigel M Furst S Rudich The expressive p ower of voting
p olynomials Combinatorica
Topic
The Complexity of Craig Interp olants
The Craig Interp olation Theorem was placed in the context of the
P NP and NP coNP questions in a pap er by Mundici
The Craig Interpolation Theorem of prop ositional logic states that for any
two formulas F and G in prop ositional logic such that F G there is a
formula H which uses only variables o ccurring in b oth formulas such that
F H and H G The formula H is called an interp olant of F and G
Exercise Prove the Craig Interp olation Theorem C
If the formulas F and G have length n then the question arises How long
must H b e It turns out that the answer is in uenced by how the formulas are
represented and that the metho d of enco ding can have a decided impact on
the result Branching programs and b o olean circuits are two length e cient
representations of b o olean functions as formulas
For a given b o olean function F let size F b e the size numb er of gates
of the smallest circuit over that computes F For formulas F and G
of length n let H b e an interp olant of minimal circuit size So size H is the
interpolant complexity of F and G which we denote by int F G For every
n let n b e de ned as
n maxfint F G j jF j jGj ng
Not much is known ab out the growth rate of n From the fact that formulas
of length n can have almost n variables more exactly O n log n variables
and the pro of of the previous exercise we get an upp er b ound for n of
n
O
Exercise Why can t a formula of co ding length n contain more than
O n log n many variables C
The interesting op en question is whether p erhaps n has only a p oly
nomial rate of growth A p ositive answer would have an interesting conse
quence for the class NP coNP
Theorem If n is polynomial ly bounded then al l languages in NP
coNP have polynomial size circuits
Topic
For more on p olynomial circuit complexity see Topics and
At rst glance this result seems very surprising How do es the pro of work
Recall the pro of of Co ok s Theorem that SAT is NP complete see the b o ok
by Garey and Johnson for a pro of The pro of contains a construction that
given any language A NP and n N pro duces in p olynomial time a b o olean
formula F x x y y where p is a p olynomial which we will
n n
p n
n
call F x y so that for all x f g
n
x A y F x y
n
Now let A NP coNP Then for every n there is a Co ok formula F x y
n
A NP Note that y for A NP and a corresp onding formula G x z for
n
and z are distinct variables but that the x variables are common to b oth
formulas
Exercise Show that F G C
n n
By the Craig Interp olation Theorem there must b e an interp olant H so
n
that F H and H G Let C b e the smallest circuit that computes
n n n n n
the b o olean function H If n is p olynomially b ounded then for some
n
p olynomial q and all n jC j q n
n
Exercise Show that C is a circuit for the characteristic function of A
n
on strings of length n Since the size of C is p olynomial in n this implies
n
that all languages in NP coNP have p olynomial size circuits C
Now one can ask if the pro of can b e mo di ed to b etter reveal its
quintessence In particular we want to generalize the right side of the im
plication as much as p ossible while maintaining the truth of the statement
By insp ecting the pro of carefully we arrive at the following formulation
Theorem If the function n is polynomial ly bounded then any dis
joint pair of NP languages A and A is PC separable This means that there
A is a language C with polynomial size circuits such that A C and C
Exercise Why is Theorem a simple corollary to Theorem
C
Exercise Prove Theorem C
Exercise Show that at least one of the following statements is true
P NP
NP coNP
Craig Interp olants
An interp olant of F and G with F G is not in general computable
in time p olynomial in jF j jGj
C
Exercise Show that the hyp othesis that every NP language has
p olynomial size circuits cf Topic implies that n is p olynomially
b ounded C
So if one could show that n is not p olynomially b ounded one would have
shown that P NP
References
A pro of of Co ok s Theorem can b e found in many b o oks including
M Garey D Johnson Computers and Intractability A Guide to the
Theory of NP Completeness Freeman
The Craig Interp olation Theorem is from
W Craig Three uses of the Herbrand Gentzen theorem in relating mo del
theory and pro of theory Journal of Symbolic Logic
Further results ab out interp olants can b e found in
E Dahlhaus A Israeli J A Makowsky On the existence of p olynomial
time algorithms for interp olation problems in prop ositional logic Notre
Dame Journal on Formal Logic
Y Gurevich Toward logic tailored for computational complexity in M M
Richter et al Computation and Proof Theory Lecture Notes in Mathe
matics Springer
D Mundici Tautologies with a unique Craig interp olant uniform vs
nonuniform complexity Annals of Pure and Applied Logic
D Mundici NP and Craig s interp olation theorem in G Lolli G Longo
A Marja ed Logic Col loquium North Holland
From the assumption that n is p olynomially b ounded it also follows that
the class UP has p olynomial size circuits UP is the class of all NP languages
A for which if x A there is exactly one witness for this cf the de nition
of FewP in Topic
U Sch oning J Tor an unpublished manuscript
Exercise was communicated by J Tor an
Topic
Equivalence Problems and Lower Bounds
for Branching Programs
Branching programs are a computational mo del for b o olean functions
which in comparison to circuits have a somewhat restricted express
ibility For a certain further restricted mo del of branching programs
the equivalence problem is solvable in probabilistic p olynomial time For
this mo del explicit exp onential lower b ounds have also b een proven
An interesting algorithmic problem is to determine whether two di erently
constructed circuits are equivalent Do these circuits compute the same func
tion even though they are wired di erently In this case we could replace
the more complicated of the two circuits with the simpler one
It would b e ideal of course if this problem could b e solved e ciently
say in p olynomial time on a computer But it is well known that the satis
ability problem for b o olean formulas or for circuits is NP complete and
there is no known algorithm for any NP complete language that runs in p oly
nomial time Testing two circuits for equivalence is similarly hard since the
inequivalence problem is NP complete The naive metho d of trying out all
p ossible assignments to the variables requires exp onential time But is there
some alternative metho d that is cleverer and faster
In order to investigate this question we will leave the world of circuits
and consider a new representation of b o olean functions namely branching
programs also called binary decision trees or BDDs
De nition Branching Program A branching program B with
boolean variables x x x is a directed acyclic graph G V E with
n
the fol lowing types of nodes
computation no des Every computation node b has exactly two out going
x and k with x for some edges k and k where k is labeled with
i i
i f ng
terminal no des Nodes with out degree are cal led terminal nodes Termi
nal nodes are divided into two categories accepting and rejecting
There is one distinguished computation node with in degree which is cal led
the start no de and denoted v
start
Given an assignment x x f g the graph of B is traversed
n
starting with the start node until a terminal node is reached During this
Topic
traversal edges may only be used if their labels agree with the assignment to
the corresponding variable If the computation ends in an accepting terminal
node then B x x otherwise B x x
n n
We say that a branching program B computes the n place boolean function
f if for al l x x f g
n
B x x f x x
n n
A branching program B is one time only if in every path from the start
node to a terminal node each variable occurs at most once
Two branching programs B and B are equivalent if for al l assignments
x x
n
B x x B x x
n n
Exercise Show that for every b o olean function f that is computable by
a branching program of size s there is a circuit of size O s that computes
f C
The problem of determining whether two di erently constructed circuits
compute the same b o olean function can now b e translated to the world of
branching programs We de ne the language BP INEQ as follows
BP INEQ fhB B i j B and B are not equivalent branch
ing programs g
Unfortunately BP INEQ is NP complete This greatly decreases our
chances of nding an e cient algorithm for BP INEQ If we were to suc
ceed we would have shown that P NP
Exercise Show that BP INEQ is NP complete
Hint Reduce the satis ability problem SAT to BP INEQ C
One sp ecial subproblem of BP INEQ is BP INEQ in which pairs of
one time only branching programs are compared
BP INEQ fhB B i j B and B are not equivalent one
time only branching programs g
Testing two one time only branching programs for equivalence app ears to b e
simpler we will show that BP INEQ RP For more on the class RP see
Topic So BP INEQ is contained in a class that is b elow NP This
means that one time only branching programs must have a simpler structure
than is generally the case One should check that in the pro of of the NP
completeness of BP INEQ it is signi cant that no one time only branching
programs are constructed
The basic idea of the pro of that BP INEQ RP is to guess the assign
ment to the n variables The values assigned to the variables however are
not chosen from f g rather they are chosen from the larger set f ng
Branching Programs
Since branching programs are only set up to handle b o olean functions we
need a pro cedure for turning a branching program into a function prefer
ably a p olynomial that can take n tuples of integers as inputs Of course
the actual information the b o olean function that the branching program is
supp osed to b e computing should not b e lost in this pro cess
So the goal is to asso ciate with any branching program B a p olynomial p
B
such that for all x x f g we have B x x p x x
n n B n
To this end we construct for every no de of B a p olynomial that is built from
the p olynomials of its predecessors
For the start no de v p
start v
start
For a no de v with predecessors v v
l
x if v v is lab eled
l
j i
X
with x
j
where p p
i i v v
i
i
x otherwise
j
Finally the p olynomial p is the sum of the p olynomials for the accepting
B
terminal no des
X
p p
B v
v
where the sum is over accepting no des v
Exercise Let B b e the following branching program
x
x
x
x
x
x
R
x x
Construct the p olynomial p and check if p x x x B x x x for
B B
all x x x f g Use a truth table
Now show that the p olynomial constructed in this way from any branching
program has the same value as the underlying branching program for any
assignment x x f g
n
Hint Prove the following claim by induction on m
Let x x f g and let V b e the set of all no des in B that
n m
are reachable from v in exactly m steps Then for all v V
start m
v is reachable on the mth step
of B x x
n
p x x
v n
otherwise C
Topic
When we sp eak of the p olynomial of a branching program we will always
assume that it has b een constructed by the pro cedure describ ed ab ove Note
that it is easy to e ciently evaluate such a p olynomial p on a particular
B
input x x In contrast it is much more di cult to multiply out such
n
a p olynomial symb olically to compute its co e cients
By the previous exercise two branching programs B and B are equivalent
if and only if for all x x f g p x x p x x
n B n B n
n
Exercise Show that it is not p ossible in the case of general branching
programs to conclude from this that for all x x N p x x
n B n
x x C p
n B
n
Now comes the question how many p oints must b e tested for equality
in order to know whether two such p olynomials are identical How many
distinct supp ort p oints of a p olynomial with n variables and degree a
multi linear polynomial determine the p olynomial uniquely
We are aided here by the following theorem
Theorem If p and q are di erent multi linear polynomials in n vari
ables and S R is an arbitrary nite set with jS j then there are at least
n n
jS j points x x S for which p x x q x x
n n n
Exercise Prove Theorem by induction on n C
Corollary If the n variate multi linear polynomials p and q agree on
n
the points then p and q
are identical
Exercise Why is this a consequence of Theorem C
So if B and B are equivalent one time only branching programs then
are identical the p olynomials p and p
B B
Exercise Why C
We have now gathered the necessary to ols to show that BP INEQ is
contained in the class RP
Theorem BP INEQ RP
Proof Let B and B b e one time only branching programs with asso ciated
A probabilistic Turing machine M for multi linear p olynomials p and p
B B
BP INEQ functions as follows
For every variable x i n a value from S f ng is chosen
i
at random under the uniform distribution
x x then M accepts otherwise the input If p x x p
n B n B
is rejected
Branching Programs
If hB B i BP INEQ then B and B are equivalent In this case
P r M hB B i accepts
Exercise Why C
On the other hand if hB B i BP INEQ then for at least half of the
assignments to x x
n
B x x B x x
n n
Exercise Prove this
Hint Use Theorem C
This concludes the pro of that BP INEQ RP ut
Next we want to compare the complexity of various statements regard
ing branching programs The equivalence problem for general branching pro
grams is coNP complete as we have argued ab ove In contrast the corre
sp onding equivalence problem for one time only branching programs is in
coRP since BP INEQ RP
A related but p otentially more di cult problem for a class of branching
programs is the inclusion problem given two branching programs decide
if the b o olean functions f and f they represent satisfy f x x
n
f x x equivalently f x x f x x
n n n
If the inclusion problem can b e solved e ciently then so can the equiva
lence problem Thus the inclusion problem is harder if the equivalence prob
lem do es not have an e cient solution then the inclusion problem do es not
either This follows from the fact that f f if and only if f f and
f f
For general branching programs the inclusion problems remains coNP
complete This follows from the NP completeness pro of for the inequivalence
problem given ab ove
There are however situations in which there is a di erence b etween the
complexity of the inclusion problem and the complexity of the equivalence
problem for a class For example the inclusion problem for deterministic
context free languages is undecidable The status of the equivalence problem
for deterministic context free languages is an op en question and may even b e
decidable
We showed ab ove that the equivalence problem for one time only branch
ing problems is in coRP The inclusion problem on the other hand is coNP
complete
Theorem The inclusion problem for one time only branching prob
lems is coNP complete
Topic
Exercise Prove Theorem C
Finally we want to give an exp onential lower b ound for one time only
branching programs with resp ect to an explicitly given b o olean function
Note that it is not the exp onential lower b ound itself that is of interest this
follows for example from Exercise rather it is the explicit presentation
of such a function
Let n b e a prime numb er Then the algebraic structure
GF n f n g
mo d n mo d n
is a nite eld Let POL b e the set of all p olynomials in one variable over
dn e
GF n with degree n There are exactly n p olynomials in POL
By the Interp olation Theorem since GF n is a eld sp ecifying dn e zero es
uniquely determines the p olynomial
n
Now we de ne our explicit b o olean function f f g f g The
graph of every p olynomial on GF n can b e represented by an argument tuple
x x x in the following way x if and only if p i j
n n i j
Example Let n The rst square b elow represents the p olynomial p x
x the second represents the function p x x x A black circle
represents the b o olean value
v v v
v v v
v v
v
v
v v v
v
Of course some tuples x do not represent any p olynomial Our b o olean
n
function f f g f g is de ned by
f X X represents a p olynomial in POL
Supp ose that B is a one time only branching program that computes f
Along every path from the start no de to a terminal no de each variable is
queried at most once
Exercise Show that along every path from the start no de to an
accepting no de every variable is also queried at least once C
Every b o olean vector x induces a path through the branching program
We will call this path the x path Let x b e given with f x So x
represents a p olynomial in POL Along the x path there must b e exactly n
variables for which the query is answered since there are exactly n s in
Branching Programs
x Let k x b e the no de where on the x path where for the n th time the
query is answered with a
The following picture sketches the branching program with the x path
the no de k x and other no des of the form k x indicated
h
h
h
h
h
h
m mh
Exercise Show that for distinct x and x with f x f x
k x k x
Hint Use the Interp olation Theorem for p olynomials and a cut and paste
argument to construct a new path x from x and x C
n
From this it follows that in B there are n distinct no des of the form
k X with f X since there are that many p olynomials in POL So
n n
n is a lower b ound for the size of a one time only branching
program that computes f Since the numb er of input values is m n this
m
b ound expressed in terms of m is
References
For the pro of that BP INEQ RP see
M Blum A Chandra M Wegman Equivalence of free b o olean graphs
can b e decided probabilistically in p olynomial time Information Process
ing Letters No
D Kozen The Design and Analysis of Algorithms Springer Lec
ture
For the coNP completeness of the inclusion problem for one time only branch
ing programs see
S Fortune J Hop croft E M Schmidt The complexity of equivalence
and containment for free single variable program schemes Proceedings
of the Symposium on Mathematical Foundations of Computer Science
Lecture Notes in Computer Science Springer
Topic
J Gergov C Meinel On the complexity of analysis and manipulation
of Bo olean functions in terms of decision graphs Information Processing
Letters
The pro of of the exp onential lower b ound given here is new The cut and
paste technique was used in
S Z ak An exp onential lower b ound for one time only branching pro
grams Symposium on Mathematical Foundations of Computer Science
Lecture Notes in Computer Science Springer
M Krause C Meinel S Waack Separating the eraser Turing machine
classes L NL co NL and P Theoretical Computer Science
e e e e
The de nition of our b o olean function is patterned after a function in
N Nisan A Wigderson Hardness versus randomness Journal of Com
puter and System Sciences no
The Berman Hartmanis Conjecture and
Sparse Sets
If all NP complete languages were P isomorphic to each other then it
would follow that P NP This Isomorphism Conjecture has b een the
starting p oint of much research in particular into sparse sets and their
p otential to b e NP complete
In computability theory it is well known that all complete problems for the
class of computably enumerable sets under many one reductions are actu
ally computably isomorphic to each other This means that b etween any two
such problems there is a computable bijection that provides the reduction
Led by the idea that the classes P and NP are p olynomial time analogs
of the classes of computable and computably enumerable languages in the
de nitions of computable and computably enumerable nitely many steps
is replaced by p olynomial in the input length many steps it is natural to
ask whether a similar isomorphism theorem is true for NP complete lan
guages as well Precisely this was conjectured by L Berman and J Hartmanis
in
Berman Hartmanis Conjecture Isomorphism Conjecture
All NP complete languages are pairwise p olynomial time isomorphic P
isomorphic to each other
Two languages A and B are said to b e P isomorphic if there is a p olynomial
time computable bijection f that is p olynomial time invertible i e f is
also p olynomial time computable and is a many one reduction from A to B
This implies that f is also a many one reduction from B to A
The analogy to computability theory suggests that the conjecture should
hold But even if it is true it will probably b e hard to prove since
Berman Hartmanis Conjecture holds P NP
Exercise Prove the claim just made C
In their pap er Berman and Hartmanis showed that all the then known
NP complete problems were pairwise P isomorphic For this they showed that
all the known NP complete languages including SAT have a certain prop
erty P and that any language that is NP complete and has prop erty P is
Topic
P isomorphic to SAT and therefore to every other NP complete language
with prop erty P See the following sketch
pppppppppppppp
NP complete languages
s
SAT
p p p p p p p p p p p p p p p p p
NP complete languages
with prop erty P
P isomorphic to SAT
NP
P
So the Berman Hartmanis Conjecture is equivalent to the question Do
al l NP complete languages have prop erty P If they do then they are all
P isomorphic and we will say that the many one degree maximal set of
many one equivalent languages of an NP complete language col lapses to its
isomorphism degree which is the isomorphism degree of SAT
So what is this prop erty P Roughly it says that all inputs x can b e
systematically extended with some irrelevant information y in such a way
that non memb ership is not altered Furthermore the information y can
b e easily recovered from the version of x that has b een padded with y The
following de nition makes this precise
De nition A language A has prop erty P if there are two
polynomial time computable functions p and d
A A
such that
p does not alter non membership in A i e x A p x y A
A A
p is length increasing i e jp x y j jxj jy j
A A
p is injective and
A
d is the inverse of p with respect to the second argument i e
A A
y if z p x y
A
d z
A
otherwise
Note the function p is often referred to as a padding function
A
Although this de nition app ears to b e quite technical and involved it
turns out that for typical NP complete languages this prop erty is easily
established
Exercise Show that SAT has prop erty P C
The Berman Hartmanis Conjecture
Next we consider two lemmas which establish conditions under which
many one reductions can b e converted into reductions and reductions
into isomorphisms
Lemma Let A and B be languages that each have property P and
P P
are polynomial time many one reducible to each other A B and B
m m
A Then each of A and B can be reduced to the other via a polynomial
time computable injective function reducibility that is length increasing
i e jf x j jxj and has a polynomial time computable inverse ut
Exercise Prove Lemma
Hint Construct the injective function using a suitable comp osition of the
many one reduction and the padding function C
Lemma Let A and B be languages that are polynomial time many one
P P
reducible to each other A B and B A via reductions that are in
m m
jective and length increasing and have polynomial time computable inverses
Then there is a polynomial time computable bijection between A and B that
has a polynomial time computable inverse In other words the languages A
and B are P isomorphic ut
Exercise Prove Lemma C
Combining these two lemmas and the observation that SAT has prop erty
P we get the following theorem
Theorem Every NP complete language with property P is P isomor
phic to SAT ut
From to day s standp oint the evidence seems to indicate that the Isomor
phism Conjecture is probably false See the pap er by P Young But aside
from the status of the conjecture itself the Isomorphism Conjecture seems
to b e even more signi cant for the numb er of imp ortant investigations and
results to which it led In particular this is the case in regard to sparse sets
De nition A set A is cal led sparse if there is a polynomial p such that
for al l n N jfx A jxj ngj p n
So in a sparse language of the exp onentially many p ossible strings of length
less than n only p olynomially many b elong to the language
Exercise Show that SAT is not sparse by showing that there are
n
constants and such that there are at least strings of length
at most n in SAT C
Topic
Exercise Show that no sparse language can b e P isomorphic to SAT
C
In light of the previous exercise one can formulate the following weaker
version of the Isomorphism Conjecture
Second Berman Hartmanis Conjecture If P NP then no sparse lan
guage can b e NP complete
The notion of NP completeness in this conjecture as in the original Isomor
phism Conjecture is p olynomial time many one reducibility But it is also
interesting to investigate this question with resp ect to p olynomial time Tur
ing reducibility
After imp ortant rst steps by P Berman and by S Fortune and with
one eye on a technique used by R M Karp and R J Lipton the Second
Berman Hartmanis Conjecture was nally proven in by S Mahaney
For a long time this result could only b e marginally improved An imp or
tant breakthrough to more general reducibilities namely to b ounded truth
table reducibility came in a pap er by M Ogiwara and O Watanab e A
somewhat more streamlined pro of of this result was given by S Homer and
L Longpr e Finally based on the technique of Homer and Longpr e the most
comprehensive result was achieved by V Arvind et al
We present here the original result of Mahaney using the techniques of
Ogiwara Watanab e and Homer Longpr e
Theorem If P NP then no NP complete language can be sparse
P
Proof Supp ose there is a sparse language S with SAT S Let p b e the
m
p olynomial that witnesses the sparseness of S and let f b e the reduction
We must somehow algorithmically exploit the fact that exp onentially many
strings in SAT of length at most n must b e mapp ed onto only p olynomially
many strings in S to show that SAT P so P NP
If we determine that for some pair of strings x and y f x f y then
the status of x and y with resp ect to memb ership in SAT must b e the same
either they are b oth in SAT or neither of them is And for strings x and
y in SAT we exp ect that this situation namely that f x f y o ccurs
frequently
An imp ortant step in the pro of consists of using a variant of SAT rather
than SAT itself for the rest of the pro of Let C b e the lexicographical ordering
on the set of all strings of length at most n For example for n we have
C C C C C C C C
Now we de ne
Lef tSAT f F a j F is a b o olean formula with n variables a
i
f g i n and there is a satisfying assign
n
ment b f g for F with a C b g
The Berman Hartmanis Conjecture
Imagine the set of all strings of length at most n arranged in a tree
n
Let b f g b e the largest with resp ect to C satisfying assignment for
F Then for all strings a in the strip ed region b elow F a Lef tSAT
E
T
T
E
T
E
T
E
T
E
T
E
T
E
T
E
T
E
T
E
T
E
E
T
n n
b
P P
SAT C Lef tSAT and Lef tSAT Exercise Show that SAT
m m
P
S say via the reduction g Let q b e a p olynomial such So Lef tSAT
m
that jg F a j q jF j Now we design an algorithm for SAT that exploits
this reduction from Lef tSAT to S
INPUT F
f Let n b e the numb er of variables in F g
T f g
FOR i TO n DO T E xtend T END
FOR each b T DO
IF F b THEN ACCEPT END
END
REJECT
In the function Extend the set T of partial assignments is extended in such
a way that after n extensions all strings in T have length n In order for this
to b e a p olynomial time algorithm it is imp ortant that calls to Extend do
not increase the size of T exp onentially but only p olynomially In order for
the algorithm to b e correct at the end of the algorithm T must contain a
satisfying assignment for F if there is one For this it will b e imp ortant that
T always contain a string a that can b e extended to the largest with resp ect
to C satisfying assignment b Then at the end of the algorithm b must b e in
the set T
Here is the pro cedure Extend
PROCEDURE Extend T Set Of Strings
Set Of Strings
VAR U Set Of Strings
Topic
BEGIN
U
FOR each a T DO U U fa a g END
FOR each a a U a C a DO
IF g F a g F a THEN U U fag END
END
f Now let U fa a g where i j a C a g
k i j
f Let m p q jF j g
IF k m THEN U fa a g END
m
RETURN U
END Extend
First every string in T is extended by b oth and Thus U has twice as
many strings as T Two pro cesses are then used to reduce the size of U
If a g value o ccurs twice then the smaller of the two strings giving rise to
this value is removed from U If U still has to o many elements more than
m then only the smallest m are kept Since m is p olynomial in n it is the
maximum p ossible numb er of strings g F a for F SAT it is clear that
the algorithm runs in p olynomial time
Correctness is established by the following exercise ut
Exercise Supp ose F is satis able Let b b b b e the largest
n
satisfying assignment for F Show that after each application of Extend the
initial segment b b of b is in T
i
Hint Be sure you understand the e ects of Extend in terms of the tree of
assignments C
Finally we note that the pro of just given can b e fairly easily extended from
many one reductions to more general reductions such as b ounded truth table
reductions or conjunctive reductions for example Furthermore we did not
make use of the prop erty that S NP so we have actually proven a somewhat
stronger theorem namely
Theorem If P NP then there is no sparse language that is NP hard
with respect to many one reducibility ut
References
For more information ab out computability theory and pro ofs of the isomor
phism result in that setting see
M Machtey P Young An Introduction to the General Theory of Algo
rithms North Holland
P Odifreddi Classical Recursion Theory North Holland
The Berman Hartmanis Conjecture
H Rogers Theory of Recursive Functions and E ective Computability
McGraw Hill
The original statement of the Isomorphism Conjecture app eared in
H Berman and J Hartmanis On isomorphism and density of NP and
other complete sets SIAM Journal on Computing
For an overview of some of the work that was generated by this conjecture
as well as an indication why the conjecture is most likely false see
P Young Juris Hartmanis Fundamental contributions to the isomor
phism problem in A Selman ed Complexity Theory Retrospective
Springer
The pro of of the Second Berman Hartmanis Conjecture built on results found
in
P Berman Relationship b etween density and deterministic complexity
of NP complete languages Symposium on Mathematical Foundations of
Computer Science Lecture Notes in Computer Science Springer
S Fortune A note on sparse complete sets SIAM Journal on Computing
R M Karp and R J Lipton Some connections b etween nonuniform and
uniform complexity classes Proceedings of the th Annual Symposium
on Theory of Computing ACM
and app eared in
S Mahaney Sparse complete sets for NP solution of a conjecture of
Berman and Hartmanis Journal of Computer and System Sciences
Extensions of Mahaney s theorem can b e found in
V Arvind et al Reductions to sets of low information content in K
Amb os Spies S Homer U Sch oning ed Complexity Theory Current
Research Cambridge University Press
S Homer and L Longpr e On reductions of NP sets to sparse sets Pro
ceedings of the th Structure in Complexity Conference IEEE
M Ogiwara and O Watanab e On p olynomial time b ounded truth table
reducibility of NP sets to sparse sets SIAM Journal on Computing
Topic
Collapsing Hierarchies
The p olynomial hierarchy can b e de ned in exact analogy to the arithmetic
hierarchy of computability theory but it is not known if the p olynomial
hierarchy is a strict hierarchy of language classes In fact under certain
assumptions ab out the class NP this hierarchy collapses
NP is the class of all languages that can b e de ned using existential quanti
cation over a p olynomial time predicate The search space of the existential
quanti er must however b e b ounded to include only strings of length p oly
nomial in the length of the input In the following formal de nition NP is
P
of the polynomial time hierarchy which is usually ab precisely the class
breviated PH
P
De nition A language L is in the class of the polynomial time
i
hierarchy if L can be de ned via a language A P and a polynomial q as
fol lows
q q q
L fx j y y Q y hy y y i Ag
i i
In this de nition Q if i is even and Q if i is odd Furthermore al l
quanti ers are polynomial ly bounded in the fol lowing sense
q
z z z jz j q jz j z
q
z z z jz j q jz j z
P P
The classes are de ned to include al l languages L such that L
i i
S
P
ut Final ly PH
i
i
Usually we will drop the sup erscripts on our quanti ers when it is clear from
context that they are p olynomially b ounded
The following diagram shows the inclusion structure of the classes in the
p olynomial hierarchy For more details see the b o oks by Balc azar Diaz and
Gabarro Garey and Johnson K obler Sch oning and Tor an or Bovet and Crescenzi
Topic
p
u u
H
H
H
H
H
p p
H u u
H
H
H
H
H
p p
u u H
H
H
H
H
H
p p
H u u
co NP NP
c
c
c u
p p
P
It is not known whether or not the p olynomial time hierarchy is a strict
P P P
hierarchy i e whether or not The analogously
de ned arithmetic hierarchy from computability theory is a strict
hierarchy The di erence is that in the arithmetic hierarchy the quanti ers
do not have a p olynomial length b ound So is the class of all computably
enumerable languages
P P P P
and PH Exercise Show that the statements
i i i i
P
are all equivalent This implies a downward separation prop erty If
i
P P P P P
C for some i then
i i i
The following diagram shows the inclusion structure of the p olynomial
time hierarchy under the unlikely assumption that it collapses to the class
P
p p
PH
u
c
c
p p
c u u
H
H
H
H
H
p p
u Hu
co NP NP
c
c
c u
p p
P
It is absolutely plausible to take P NP as a working hyp othesis and
to prove theorems under this Co ok Hyp othesis The hyp othesis P NP
P P
in terms of the p olynomial time hierarchy means that Further
more by the previous exercise the hyp othesis NP co NP is equivalent to
P P
This second hyp othesis is stronger than the rst Altogether
the p olynomial time hierarchy provides an in nite reservoir of plausible
hyp otheses which could represent the working hyp othesis of some theorems
Collapsing Hierarchies
p
PH for all k
k
p
PH
p
PH
p
NP co NP
PH
p
P NP
PH
The circuit complexity of a language L is a function cc N N such
L
that cc n gives the minimal necessary numb er of b o olean gates of the typ es
L
AND OR and NOT that can b e used to build a circuit that computes the
characteristic function of L on strings of length n If the NP complete lan
guages are not computable via a deterministic p olynomial time b ounded
algorithm i e P NP then it could still b e the case that they have
p olynomial size circuits This would b e a very interesting situation It would
b e p ossible with a certain exp onential exp enditure to build a circuit with
for example n inputs that is only mo destly sized p olynomial in n
and is capable of e ciently solving all instances of SAT of length up to
This means that although it would require exp onentially much work
done as a prepro cess but then never rep eated one could design a fast
chip for SAT
We want to investigate this question and eventually link this situation
with the unlikely collapse of the p olynomial time hierarchy
Exercise Show that there are languages that are not in P but do have
p olynomial size circuits C
Exercise Show that a language L has p olynomial size circuits if and
S
only if there is a sparse set S such that L P i e L can b e computed in
p olynomial time relative to some sparse oracle C
S
P
we can also write L S and consider this as a Instead of L P
T
p olynomial time Turing reduction of L to S By means of the previous
exercise we can express the question of whether all languages in NP have
p olynomial size circuits equivalently as the question of whether all languages
Topic
in NP or just the NP complete languages are reducible to sparse sets In
this way this question is related to the questions surrounding the Berman
Hartmanis Conjecture of the previous topic
P
A language L in NP has the form
L fx j y hx y i Ag
for some A P and a p olynomially b ounded existential quanti er For every
x L there must b e at least one y with hx y i A We call such a y a witness
or a proof for the fact that x A For example a satisfying assignment for
a formula F is a witness that F SAT
We want now to consider circuits that have not just a single output bit
which expresses whether or not x L but which also have suitably many
additional output gates by means of which the circuit provides a witness y
for x L when this is the case We will call such a circuit for a language
in NP a witness circuit
Exercise Show that if SAT has p olynomial size circuits then SAT
also has p olynomial size witness circuits which pro duce in certain output
gates a satisfying assignment for the b o olean formula if it is satis able
Hint Use the self reducibility of SAT C
Now there is only a small step remaining to prove the following result of
R M Karp and R J Lipton
Theorem If al l languages in NP have polynomial size circuits then
P
the polynomial time hierarchy col lapses to its second level PH
P
Proof Let L b e a language in It su ces to show that from the hyp othesis
P
Let A b e a language in P such that it follows that L
L fx j y z hx y z i Ag
Then the language
L fhx y i j z hx y z i Ag
is in NP Since SAT is NP complete L can b e reduced to SAT by a
p olynomial time computable function f i e f SAT L This implies
that
L fx j y hx y i L g fx j y f hx y i SAT g
We claim now that the following characterization of L is p ossible which shows
P
that L
L fx j c y c is a witness circuit and c f hx y i pro duces
a satisfying assignment for the formula f hx y i g
Exercise Prove that this representation of L is correct C
This completes the pro of of Theorem with a much simpler pro of
than was originally given ut
Collapsing Hierarchies
The boolean hierarchy over NP similar to the p olynomial time hierarchy
arises from the fact or conjecture that NP is not closed under complement
The class BH is the smallest class that contains NP and is closed under the
b o olean op erations of intersection union and complement
A hierarchy of classes b etween NP and BH can b e formed by b eginning
with NP and systematically adding unions and intersections with coNP lan
guages
De nition The classes BH BH of the boolean hierarchy over NP
are de ned as fol lows
BH NP
B j A BH B NPg i BH fA
i i
BH fA B j A BH B NPg i
i i
S
BH C Exercise Show that BH
i
i
The inclusion structure for the b o olean hierarchy is similar to that for the
p olynomial time hierarchy
u u
BH co BH
H
H
H
H
H
u u H
BH co BH
H
H
H
H
H
u u H
BH co BH
H
H
H
H
H
u u H
NP BH co BH co NP
c
c
u c
P
NP
P P
The b o olean hierarchy is contained in the class P We say
that the b o olean hierarchy col lapses to the k th level if BH co BH The
k k
following result due to J Kadin was a sensation at the Structure in
Complexity Theory Conference held at Cornell University
Theorem If the boolean hierarchy col lapses at any level then the poly
nomial hierarchy col lapse at the third level More precisely in this case the
P
polynomial time hierarchy col lapses to the boolean closure of which in
P
turn is contained in
Proof It is p ossible to systematically de ne complete languages for the levels
of the b o olean hierarchy We sketch this b elow for the rst few levels
Topic
L SAT
SAT g L fhF F i j F L F
L fhF F F i j hF F i L F SAT g
L fhF F F F i j hF F F i L F SAT g
L is complete for It is easy to show that L is complete for BH and so
i i i
coBH If BH co BH then L can b e reduced to L and vice versa This is
i i i i i
the starting p oint for the pro of We will give the pro of for the case i The
general case contains the same basic argument but is technically somewhat
more involved
P
Supp ose L L so there is a p olynomial time computable function
m
f that maps pairs of b o olean formulas F F to pairs G G with the
prop erty that
SAT G SAT G SAT F SAT F
As a rst step we will try to show that SAT NP We will not b e successful
but will very nearly achieve this goal How do we design a nondeterministic
SAT The following would b e one attempt algorithm for
INPUT F
Nondeterministically guess a formula F with jF j jF j
and a satisfying assignment for F so F SAT
Compute G G f F F
IF G SAT THEN ACCEPT END
SAT namely This nondeterministic algorithm accepts a certain subset of
the unsatis able formulas F for which there is a satis able formula F of the
same length so that f F F G G with G SAT We will call this
algorithm the easy algorithm and the formulas that it accepts will b e called
easy formulas Clearly easy formulas are unsatis able but p erhaps they do
not make up all unsatis able formulas We will call a formula hard if it is
unsatis able but not easy So
SAT F is hard F
SAT F jF j jF j f F F G G G
We note at this p oint that the prop erty of b eing hard can b e describ ed with
P
a predicate
But how do we get the hard formulas The fact that they are hard is the
key Fix an arbitrary hard formula F of length n With the aid of this one
formula we can correctly nondeterministically accept all other unsatis able
formulas of this length by means of the following algorithm
Collapsing Hierarchies
INPUT F
Compute G G f F F
IF G SAT THEN ACCEPT END
Why is this algorithm correct Since F is hard f F F pro duces a pair of
SAT Since F SAT we have the following formulas G G such that G
equivalence
SAT F SAT G
or equivalently
SAT G SAT F
This demonstrates that the algorithm is correct This algorithm is unusual
in that rst we need to b e given n bits of information F but then param
eterized with this information called advice the algorithm is correct for all
formulas of length n For a given hard formula F we will call this algorithm
the F algorithm
We have come very close to demonstrating that SAT NP but have fallen
just short b ecause of the need for advice in the previous algorithm Never
theless what we have done is su cient to show a collapse to the p olynomial
P
P P
time hierarchy to P Let L L can b e represented as
L fx j u v w hx u v w i Ag
for some language A P Let
L fhx u v i j w hx u v w i Ag
L NP so there exists a reduction g from L to SAT So L can b e
written as
SAT g L fx j u v g hx u v i
SAT with some NP predicate this would If we could replace the reference to
P
amount to SAT NP then we would have a characterization of L As
we have said we are not able to achieve this But consider the following
language
n
B f j there is a hard formula of length ng
P
The language B is in Furthermore let
C fx j u v g hx u v i is not accepted
by the easy algorithm g
P
Then C Now consider
D fx j F F is hard
u v g hx u v i is not accepted by the F algorithm g
Topic
P
D is also in The language L can now b e recognized by the following
deterministic oracle algorithm using the languages B C and D as oracles
In this algorithms m is a suitable p olynomial in jxj that gives the length of
g hx u v i
INPUT x
m jxj
IF B THEN
IF x D THEN ACCEPT END
ELSE
IF x C THEN ACCEPT END
END
REJECT
m jxj
This oracle machine asks exactly two of the three oracle questions B
x D and x C Since this is a constant numb er of oracle queries the
P
language L is in the b o olean closure of the class ut
References
For general information ab out the p olynomial time hierarchy see
L J Sto ckmeyer The p olynomial time hierarchy Theoretical Computer
Science
C Wrathall Complete sets and the p olynomial time hierarchy Theoret
ical Computer Science
Theorem was originally proved in
R M Karp R J Lipton Some connections b etween nonuniform and uni
form complexity classes Proceedings of the th Symposium on Theory
of Computer Science ACM
NP
P
to ZPP see Since then the p oint of collapse has b een improved from
N H Bshouty R Cleve S Kannan C Tamon Oracles and queries that
are su cient for exact learning COLT
For more ab out the class ZPP see Topic
For a more detailed exp osition of the b o olean hierarchy see
Cai Gundermann Hartmanis Hemachandra Sewelson Wagner Wech
sung The Bo olean hierarchy I I I SIAM Journal on Computing
and
Theorem app eared in
Kadin The p olynomial hierarchy collapses if the Bo olean hierarchy col
lapses SIAM Journal on Computing
Collapsing Hierarchies
See also
R Chang J Kadin The Bo olean hierarchy and the p olynomial hierarchy
a closer connection Proceedings of the Structure in Complexity Theory
Conference IEEE
Topic
Probabilistic Algorithms
Probability Ampli cation
and the Recycling of Random Numb ers
Probabilistic algorithms require sto chastically indep endent and uniformly
distributed random numb ers and in general the smaller the probability
of error is supp osed to b e the more random numb ers are required Here
we intro duce a metho d whereby random numb ers already used by an
algorithm can b e recycled and then reused later in the algorithm In
this way it is p ossible to drastically reduce the numb er of random numb ers
required to obtain a sp eci c b ound on the error probability
A probabilistic algorithm can make decisions dep ending on chance This can
b e mo deled by a Turing machine in which each con guration may have several
but only nitely many next con gurations each of which is chosen with
equal probability By intro ducing probability the decision and running time
of an algorithm accept or reject or on input x b ecome random variables
The concept of acceptance is de ned here di erently than it is for non
deterministic machines A string x is accepted by a probabilistic algorithm
if the probability that the algorithm accepts x is More generally one
can x an arbitrary threshold probability value and say that a
string x is accepted if the probability of acceptance is greater than
De nition A language L is in the complexity class PP probabilistic
polynomial time if there is a polynomial time bounded probabilistic algorithm
and a threshold value such that L consists of precisely those strings
that are accepted with probability
An algorithm that demonstrates that a language b elongs to PP will b e called
a PP algorithm
The notion of a p olynomial time b ound in the de nition ab ove means
that for every realization of the random variables T x the running time
of the algorithm on input x is b ounded by p jxj where p n is some xed
p olynomial
In this topic we are primarily interested in probabilistic algorithms that
exhibit a probability gap b etween the accepted and rejected strings This
means that not only is the probability when x L and when x L
but in fact the algorithm ful lls a stronger condition If x L then the
probability of acceptance is and if x L then the probability is
Topic
for some So there is a gap of b etween the probabilities
of acceptance for the strings x L and the strings x L
x L
x L
De nition A language L is in the class BPP bounded error prob
abilistic polynomial time if there is a PP algorithm for L with threshold
probability and a probability gap such that the condition above is satis
ed That is L is accepted in the sense of a PP algorithm and the probability
that a string x is accepted is never in the range
A probabilistic algorithm that ful lls the de nition ab ove is called a BPP
algorithm It is clear that P BPP PP
De nition A language L is in the class RP if there is a BPP algorithm
for L such that That is if x L then the probability that the
algorithm accepts on input x is
Sketch
x L
t
x L
This acceptance criterion is a strengthening of the requirement for an NP
language so it is clear that P RP NP and P RP BPP The inclusion
relationship b etween NP and BPP is an op en question We will come back to
this relationship shortly in particular the inclusion NP BPP is unlikely
since it implies a collapse of the p olynomial hierarchy cf Topic
De nition A language L is in the class ZPP zero error probabilistic
L RP That is ZPP RP polynomial time if and only if L RP and
coRP
Exercise The name zero error comes from the fact that for the
languages L ZPP one can give a p olynomial time b ounded probabilistic
algorithm and a constant so that on input x with probability p the
algorithm answers correctly x L or x L and with probability p
the algorithm answers I don t know So in contrast to the class BPP the
algorithm never outputs a wrong assertion
Show that for any language L ZPP there is such an algorithm with
three output values accept reject don t know C
Exercise Prove that ZPP is precisely the class of languages that
p ossess probabilistic algorithms of the following typ e the algorithm always
Probabilistic Algorithms
outputs the correct answer accept or reject never don t know and the
expected running time is p olynomial C
The following diagram indicates the inclusion structure of these proba
bilistic classes in comparison to P and NP
PP
t
t t
NP co NP
ppppppppppppppppppppppppppppppppppppppppppppppppppp pppppppppppppppppppppp
t
BPP
t t
RP co RP
u
P
The problems b elow the dotted line can b e considered e ciently solvable
for all practical purp oses In particular all of them have p olynomial size
circuits The main reason for this is related to the fact that all of them p ermit
probability ampli cation This will b e our next topic of discussion
The probability gap that exists in the de nitions of the classes BPP RP
and ZPP plays an imp ortant role with regard to probability ampli cation
This means that with only a p olynomial increase in running time we can
mo dify our algorithms for example in the case of BPP so that strings in the
language are almost always accepted and strings not in the language are
almost never accepted
Sketch
x L
x L
In the case of the class RP and hence also for ZPP the probability
ampli cation is relatively easy to obtain Accepting computations are always
correct so in this case we know that x L We only need to mistrust the
rejecting computations Supp ose we have b een given an RP algorithm for a
language L This algorithm accepts strings in the language with a certain
probability By running the algorithm several times on the same input
each time with new random numb ers so that the individual results of this
Topic
exp eriment are indep endent and accepting only if at least one of the indi
vidual results was an accepting computation we obtain an algorithm with a
signi cantly larger probability gap
Exercise How large is the probability gap after t trails To get an error
n
probability of how many times must the algorithm b e rep eated C
The situation is not quite so easy for BPP algorithms Here we must
mistrust b oth the accepting computations and the rejecting computations
since b oth results can b e wrong some of the time Although neither result is a
certain criterion for x L or x L each o ers a certain amount of evidence
evidence which can mount in favor of one decision or the other with rep eated
trials Let M b e a BPP algorithm with threshold value and probability gap
The following algorithm yields the desired probability ampli cation i e a
larger probability gap
INPUT x
s
FOR i TO t DO
Simulate M on x let the result b e y
IF y accept THEN s s END
END
IF s t THEN ACCEPT
ELSE REJECT END
This is a Bernoulli exp eriment since a random exp eriment with a certain
probability p of success is rep eated indep endently t times In our case p
if x L and p if x L So the probability that this
algorithm gives a false reject when x L is at most
t
X
t
i t i
i
i
where
Exercise Show that this function is exp onential in t How large
n
must t b e chosen to bring the probability of error under Give a similar
approximation of the error for the case that x A
Hint Use Cherno b ounds or Bernstein s law of large numb ers C
So as in the simpler case of RP we get a linear relation b etween t and n if
the goal is an exp onentially small probability of error
In summary for every language L BPP and for every p olynomial p
there is a BPP algorithm M for the language L with threshold value and a
p n
probability gap of where n is the length of the input This means
p jxj
that the probability of error on input x is less than
Probabilistic Algorithms
So for example if one cho oses p n n the probability of error is
already so small that for most random choices z made by the algorithm the
algorithm is always correct with this xed z for all x of length n That is
this z serves as advice from which memb ership of all strings of length n
can b e correctly decided cf Topic
Exercise From this it follows that all languages in the class BPP have
p olynomial size circuits cf Topics and Justify this claim C
With the results of Topic we now get the following theorem immedi
ately
Theorem If NP BPP then the polynomial hierarchy col lapses to
P
Exercise Show the following result If NP BPP then NP RP
Hint It su ces to show that SAT RP Use a BPP algorithm for SAT
to with small error rate and the self reducibility of SAT to nd a p otential
satisfying assignment for the input formula If the probabilistically generated
assignment is not a satisfying assignment then reject C
In the last section we saw that in order to improve the probability gap
k
from for example to we needed to rep eat the
algorithm O k times and then make a decision based on ma jority vote This
k
shrinks the probability of error to If the underlying probabilistic
algorithm requires r random bits then our new algorithm requires O r k
random bits
If one takes the p ersp ective that not only computation time and memory
use are costly resources to b e minimized but also the numb er of random
bits then one could attempt to achieve the probability ampli cation with
the smallest p ossible numb er of random bits It seems plausible that the
numb er of random bits could b e reduced when one considers that each blo ck
of r random bits used by the algorithm really only serves to get bit of
information namely whether the probabilistic algorithm accepts some input
with those r random bits That is the relative entropy of r bits on a given
decision of the probabilistic algorithm is still very high We just need to nd a
way to recycle b etter re prepare the r random bits so that no signi cant
dep endency results That this is p ossible is essentially due to the metho d of
universal hashing and an analysis presented in the Leftover Hash Lemma
b elow
The solution to the problem will lo ok like this After using the r random
bits we will use a hash function h to obtain s r recycled bits which are
then augmented with r s fresh random bits to form r random bits for
the next trial This pro cess is then iterated for each trial
Topic
r random bits
h
p
p
p
p
ppppppppppppppppp
p
r s random bits
s Bits
p
p
h
p
p
p
p
p p p p p p p p p p p p p p p p p
p
s Bits
r s random bits
p
p
The sequence of r bit pseudo random numb ers that arises in this way can
then b e used in place of the original k indep endently chosen random numb ers
Now we approximate the numb er of actual random bits used We need
r bits for the rst random numb er and r s for each subsequent pseudo
random numb er The function h will b e chosen randomly from a class of
so called universal hash functions We will see that the selection of h can b e
done with O r bits Altogether we need r O r r s k random
bits We will see that r s can b e chosen so that r s O k So we
need O r k random bits In fact it is p ossible by other metho ds
to further reduce the numb er of random bits required to O r k This is
a drastic improvement over our original probability ampli cation algorithm
which required O r k random bits
r s
De nition A class H of hash functions from f g to f g s r
r
is cal led universal if for every x y f g with x y
s
P r h x h y
The probability above is over h chosen uniformly at random from H In this
case the probability of a col lision is the same as if h x and h y are chosen
s
uniformly at random from f g
r s
A class H of hash functions from f g to f g s r is cal led almost
r
universal if for every x y f g with x y
s r
P r h x h y
r
Let s try the following class of hash functions Let p b e a prime
numb er with bit length O r The hash functions in H are sp eci ed by giving
two numb ers a b p The hash function h h is de ned so that
p a b
h x is the s least signi cant bits of the numb er ax b mo d p This can b e
expressed as
s
h x ax b mo d p mo d
p a b
It is clear that for any x y and randomly chosen values a and b the value
ax b mo d p ay b mo d p is uniformly distributed in p p
Probabilistic Algorithms
This uses the fact that GF p is a eld So this class of functions would b e
universal
The restriction to the last s bits however destroys the uniformity of the
distribution and therefore the universality of the hash functions
Exercise Show that the class H de ned ab ove is an almost universal
class of hash functions C
De nition Let D and D be probability distribution on a nite set S
We say that D and D are similar if for every subset X S
jD X D X j
Here D X is the probability of X under the distribution D so D X
P
P r x
D
x X
A distribution D is almost uniformly distributed if D and the uniform
distribution on S are similar
The collision probability of a distribution D is the probability that two
elements x and y chosen independently according to D are the same
The following lemma shows that for an almost universal class of hash
functions an amazing amount of uniformity holds if we cho ose h randomly
from H but cho ose the argument x at random from an unknown but large
enough set
Lemma Leftover Hash Lemma Let X be an arbitrary subset of
r l
f g with jX j Let H be an almost universal class of hash functions
r s
from f g to f g Suppose r l s and let e l s Then the
e s
distribution of h h x is almost uniform on the set H f g
Proof We show the pro of in two steps
Step The collision probability of the distribution h h x is at most
e s
jH j
Step Let D b e a distribution on a nite set S If the collision probability
of D is at most jS j then D is almost uniformly distributed on
S
It is clear that the statement of the lemma follows immediately from Steps
and which are proven b elow ut
Exercise Prove Step C
Proof of Step Supp ose D is not almost uniformly distributed Then
there is a subset Y of S with D Y jY j jS j Let b e such that
D Y jY j jS j The collision probability of D is given by P r d d
Topic
where d and d are chosen at random according to D We use Y to partition
the probability space The probability for d d under the condition that
d d Y is at least jY j and the probability for d d under the
condition d d Y is at least jS j jY j So the collision probability is
at least
D Y D Y
jY j jS j jY j jS j jY j jS j jY j
which is obtained by substituting jY j jS j for D Y and simplifying This
expression is minimized when jY j jS j In that case the collision probabil
ity is at least jS j jS j which contradicts our assumption
ut
Let M b e a BPP algorithm for a language L that requires exactly r ran
dom bits to pro cess an n bit input Let the probability of error b e somewhere
b etween and For technical reasons we must also place a lower b ound
on the error So we have
P r M x
x L
P r M x
P r M x
x L
P r M x
Let x x x b e the sequence of pseudo random numb ers generated
k
by the metho d and hash functions describ ed ab ove We want to study the
probabilistic b ehavior of the algorithm that uses the x s as random bits in k
i
simulations of M and then decides based on a ma jority vote whether x L
r k
The bit sequence x x x is certainly not uniformly distributed f g it
k
isn t even almost uniformly distributed since only O r k genuine random
bits were used to generate it So most of the sequences cannot even o ccur
and thus have probability What is imp ortant for us is that the bit sequence
b x b x b x where
k
if M with random numb er x accepts
i
b x
i
if M with random numb er x rejects
i
approximates the distribution that one gets by running the algorithm M
k times using new random numb ers for each simulation and noting the bit
sequence b b b
k
Without loss of generality we may assume that the input x L so
P r b p Supp ose there is a small numb er so that the
i
real sequence b b b and the simulated sequence b x b x b x
k k
are similar
Exercise If we use genuine random numb ers we know that the prob
k
ability of error of the ma jority voting algorithm is Approximate how
Probabilistic Algorithms
small the error b ound must b e as a function of k so that using pseudo
random numb ers instead of genuine random numb ers still results in an error
k
probability of at most C
Now we prove the following claim by induction
Claim For all i the distribution of b x b x hx where x is the
i i j
j th pseudo random numb er as describ ed ab ove is similar for some yet
i i
to b e determined to the following distribution The rst i bits are chosen
indep endently at random to b e with probability p and with probability
p then h is chosen uniformly at random from H this is followed by a
r
string chosen uniformly at random from f g
Proof of the claim If i we only need to consider the distribution of
hx In this case this is uniformly distributed so
We also consider the case where i separately since it is instructive for
r
the inductive step Let x b e chosen at random from f g We must show
that the distribution of b x hx is similar to a correct distribution one
as is describ ed in the statement of the claim with i for some small
It is su cient to show that b x hh x is similar to a distribution where
the rst bit is chosen according to the p p distribution and h is chosen
randomly from H and the last s bits are chosen uniformly at randomly from
s
f g indep endent of each other Since x is random b x is distributed
according to p p Under the assumption that b x a f g we
s
must show that hh x is nearly uniformly distributed on H f g This
is where the Leftover Hash Lemma helps Under the given condition we
r r r
still have p or p so at least choices available for x The
statement of the lemma is satis ed if we put l r The lemma then yields
a b ound of
e l s r s
Now comes the induction step By the inductive hyp othesis the claim is
true for i so the distribution of b x b x hx is similar to the
i i i
correct distribution We want to show the claim for i We need to compare
the distribution of b x b x hx with the correct distribution which we
i i
represent as b b hz It su ces to show that b x b x hh x is similar
i i i
to the distribution of b b hv where the b s are chosen indep endently
i j
according to the p p distribution h is chosen uniformly from H and v
s
is chosen uniformly from f g The following claim is useful
Let F G and H b e random variables such that the distribution
of F is similar to the distribution of G Furthermore let t b e a
transformation such that the distribution of t G is similar to H
Then H and t F are similar
Sketch
Topic
p p p p p p p p p p p p p p p p p p
t F t G
H
ppppppppppppppppppppppp
F G
Exercise Prove this claim C
Now we put
F b x b x hx
i i
G b b hz
i
H b b b hv
i i
and de ne a transformation t by
t b b hz b b b z hh z
i i
Then we get
t G b b b v hh v
i
t F b x b x b x hh x
i i i
With this choice of F G H and t F and G are similar by the inductive
i
hyp othesis Furthermore t G and H are similar as in the case i
It follows that H and t F are similar But this is precisely the
i
inductive claim if we set i e i
i i i
From the claim with i k it follows that the bits b x b x are
k
r s
k k similar to the distribution sets each bit to b e with
probability p and with probability p
All that remains is to determine the degrees of freedom s We do this
by setting r s k so s r k Then we get a b ound of
k k
k on the similarity which by Exercise is go o d enough to
obtain the desired probability ampli cation ut
Remarks The pro of ab ove is still missing one detail somehow we need to
nd the prime numb er p used in the de nition of the hash function class H
This requires another probabilistic test which was previously known Fur
thermore it is p ossible we may need to test several n bit strings for primality
b efore a suitable candidate is found This seems to require an unavoidable
numb er of random bits which p erhaps brings our upp er b ound on the num
b er of random bits into question Impagliazzo and Zuckerman to whom this
metho d of recycling random bits is due actually used another class of hash
functions that do es not have this di culty but which is harder to analyze
Probabilistic Algorithms
Another interpretation of the result presented here is that the sequence
of pseudo random numb ers x x app ear to b e genuinely random for
k
every statistical test that can b e p erformed in p olynomial time the sequence
b x b x of test results di er in their probability distribution only
k
minimally from the exp ected distribution b z b z where the z s are
k i
genuine random numb ers The only b oundary condition is that the test
should not output or to o infrequently cf the Leftover Hash Lemma
Sketch
x
b x f g
Test
i
i
References
An intro duction to probabilistic complexity classes app ears in many text
b o oks including b o oks by the following authors Balc azar Diaz and Gabarr o
Brassard and Bratley Bovet and Crescenzi Gurari Welsh Sch oning Com
plexity and Structure and K obler Sch oning and Tor an
For more on universal hashing see
Cormen Leiserson Rivest Introduction to Algorithms MIT Press
McGraw Hill
The metho d of recycling random numb ers comes from
R Impagliazzo D Zuckerman How to recycle random bits Symposium
on Foundations of Computer Science IEEE
The elegant metho d used here to prove the induction step is thanks to
J K obler
Exercise is from
A A
C H Bennett J Gill Relative to a random oracle A P NP
A
co NP with probability SIAM Journal on Computing
Exercise is from
K Ko Some observations on the probabilistic algorithms and NP hard
problems Information Processing Letters
and the solution presented in the back of the b o ok is from
U Sch oning Complexity and Structure Lecture Notes in Computer Sci
ence Springer
Topic
The BP Op erator and Graph Isomorphism
The following results suggest that the Graph Isomorphism problem is not
NP complete since unlike the known NP complete problems Graph Iso
morphism b elongs to a class that can b e de ned by means of the BP
op erator an op erator that has proven useful in many other applications
as well
In this section we want to intro duce a very useful op erator notation that
arises out of the observations ab out probabilistic algorithms made in Topic
and that will make p ossible the main result of this topic namely that the
graph isomorphism problem is not NP complete unless the p olynomial time
hierarchy collapses This result originally came out of a sequence of results
in the area of interactive pro of systems We will not intro duce these pro of
systems here but see Topic instead we will prove this result along a
more or less direct route using the BP op erator and some of its prop erties
The transition from the class P to the class BPP cf Topic amounts
to the addition of probability but retains many of the essential features of
the original class We want to concentrate on this randomization step and
formulate its essential characteristics with the help of an op erator that can
b e applied to any class C to pro duce its probabilistic or randomized gener
alization
De nition Let C be an arbitrary class of languages We denote by
BP C the class of al l languages A for which there is a language B C a
polynomial p and constants and such that for al l strings x
x A P r hx y i B
x A P r hx y i B
The probabilities are over al l y with jy j p jxj chosen uniformly at random
As in Topic will b e called the threshold value and the probability
gap Furthermore it is clear that BPP BP P By applying the BP op erator
to other complexity classes we can now form other classes as well like BP NP
for example
Topic
Exercise Formulate a condition on the class C so that C BP C C
Now that we have generalized the BP op erator we must rethink under
what conditions probability ampli cation as discussed in Topic is
p ossible This is p ossible if C p ossesses a certain closure prop erty
Exercise Formulate a condition on the class C that p ermits probability
ampli cation cf Topic More precisely by probability ampli cation we
mean that for every language A BP C and every p olynomial q there should
b e a language B in C such that
q jxj
x A P r hx y i B and
q jxj
x A P r hx y i B
Again y is chosen uniformly at random from among all strings of some suit
able p olynomial length in jxj
Hint Closure of the class C under p olynomial time Turing reductions would
certainly b e a su cient condition But since we are particularly interested
in the class BP NP and would like to b e able to amplify probabilities for
that class this condition is to o strong NP is probably not closed under
Turing reductions since that would imply NP coNP So the task here is to
nd a su cient but somewhat weaker condition than closure under Turing
reductions that applies to the class NP among other classes C
Exercise Let C b e a class such that probability ampli cation is p ossible
in the class BP C Show that BP BP C BP C C
By applying probability ampli cation we are able to show the following
result ab out swapping the order of two op erations on a class of languages
Lemma Swapping Lemma Let Op be an operator on complexity
classes with the fol lowing property If D is an arbitrary class and A is an
arbitrary language in Op D then there is a polynomial p and a language
B D so that the property x A depends only on the initial segment of
B up to strings of length p jxj Furthermore let C be a class for which
probability ampli cation is possible
Then Op BP C BP Op C
Exercise Prove Lemma C
Such op erators with a p olynomial dep endency region include for ex
ample co P NP and the p olynomial quanti ers and used to de ne
P P
the classes and in Topic Since we will make use of the p olynomial
i i
quanti ers again here we rep eat their formal de nitions
De nition For a class C C denotes the class of al l languages A for
which there is a polynomial p and a language B C such that
Graph Isomorphism
A fx j y jy j p jxj hx y i B g
The class C is de ned analogously using universal quanti cation over
strings of length p jxj
A few of the obvious prop erties of these op erators include
P NP co C co C co C co C
Now that we have studied a few prop erties of the BP op erator we want
to b ecome familiar with a natural problem that is in one of the BP classes
The problem is the graph isomorphism problem GI given two undirected
graphs determine whether or not they are isomorphic that is determine
whether or not there is a p ermutation of the no des no de numb ers such that
when applied to the rst graph it results in the second
Example The following two graphs are isomorphic One p ossible isomor
phism is
m
k k
m m
Q
m m
Q
Q
l
Q
Q l
m m m
It is clear that this problem is in NP Whether this problem is NP complete
remains an op en question but most likely it is not See for example a dis
cussion of this problem in the b o ok by Garey and Johnson On the other
hand no p olynomial time algorithm is known for this problem either So GI
is a candidate for a problem that is in NP but neither NP complete nor in P
If P NP it can b e proven that such problems exists but it will probably
b e di cult to prove that any natural problem has this prop erty
Our pro of of the following theorem unlike the original pro of will b e the
result of a more or less direct attack
GI is Theorem The complement of the graph isomorphism problem
in BP NP
Proof We b egin with a pair of elementary observations ab out isomorphism
of graphs Let Aut G b e the set a group of isomorphisms of a graph G
that is the set of all p ermutations of the no des that map the graph to itself
For example is an automorphism of the rst graph in the example
ab ove
Topic
Exercise If the graph G has exactly n no des and m jAut G j how
many di erent graphs are isomorphic to G C
Exercise Given two graphs G and G each having n no des de ne
a set X X G G of ob jects such that the ob jects in X can b e generated
nondeterministically from the graphs G and G in p olynomial time and
G and G are isomorphic jX j n
G and G are not isomorphic jX j n
C
By forming the cross pro duct
Y X X
we can increase the di erence b etween these two numb ers
G and G are isomorphic jY j n
G and G are not isomorphic jY j n n
Next we need a class H H of universal hash functions This class of
n
hash functions should map elements from a set U for example strings
with Y U to the elements of the set M n f n g
We require of this class of hash functions H the following
Uniformity For every y U and a M if a hash function h is chosen at
random from H then
P r h y a jM j
Pairwise independence For all y y U with y y and all z M if a
hash function h is chosen at random from H then
P r h y z h y z jM j
E cient computation The random selection of a hash function h from
H H and the evaluation of h on an input y should b e p ossible in
n
p olynomial time with p olynomially many random bits
In what follows we will b e interested in an approximation to the proba
bility that for a randomly selected h h Y Let G and G b e two input
graphs with n no des each If they are isomorphic then jY j n and in
this case
P r h Y P r y Y h y
X
P r h y
y Y
jY j jM j
Graph Isomorphism
If the graphs are not isomorphic then jY j n jM j In this case
we can give a lower b ound for the probability by using the rst two terms of
the inclusion exclusion principle cf Kozen page
X X
P r h y h z P r h Y P r h y
y Y
fy z g Y
jY j
jY j jM j jM j
In summary
G and G are isomorphic P r h Y
G and G are not isomorphic P r h Y
Since the predicate h Y is in NP guess a y verify nondeterministi
cally that y Y check that h y this representation shows that the
complement of graph isomorphism is in BP NP The constants are
and
We have yet to discuss the class of hash functions required We could use
the class of hash functions intro duced in Topic but that choice brings with
it a numb er of technical di culties since it requires a nondeterministic check
for primality which is p ossible and b ecause jM j is not necessarily prime
a b
We can de ne another class of hash functions H from f g to f g
as follows A hash function h H is describ ed by a b o olean a b matrix
h Let x x x b e a string of length a Then the j th bit of h x
ij a
is de ned to b e
a
M
h x j b
ij i
i
ab
Each of the such hash functions h can b e chosen uniformly at random
from H so ab bits are required to sp ecify a hash function
Exercise Show that this class H has the pairwise indep endence prop
a b
erty i e for all y y f g with y y and for all z f g if h is
b
chosen uniformly at random then P r h y z h y z C
A small technical problem is now that size of the range of the functions is a
p ower of This relaxation of the desired numb er jM j is acceptable if we rst
further increase the di erence in the p ossible values of jY j for isomorphic vs
non isomorphic pairs of graphs Once this has b een done the pro of follows
easily from what we have done ab ove ut
Topic
Since the complement of the graph isomorphism problem is in BP NP
GI coBP NP BP coNP so it is almost in coNP The following diagram
shows the relative p osition of GI
coBP NP
NP P coNP
r
p
p
p
p
p
p
p
p
p
p
p
p
p
p
p
p
p
p
p
p
p
GI
The diagram suggests that GI is probably not NP complete since the NP
complete problems should b e thought of as b eing at the far left of the diagram
as far away from P as p ossible
Just as no NP complete language can b e in coNP unless NP coNP that
P
we will show now that if there is an NP complete is PH collapses to
P
language in coBP NP then PH collapses to So if GI is NP complete
P
then PH
For this we need a metho d of simulating the BP op erator with sequences
of and
Lemma Let C be a class of languages that is closed under the operator
Pos see page so that in particular BP C permits probability ampli ca
tion Then BP C C and BP C C
Proof Let A b e in BP C We cho ose a language B in C so that for all x
n
x A P r hx y i B
n
x A P r hx y i B
p jxj
where y is chosen uniformly at random from f g and p is a suitable
p olynomial
With the help of the following claim the pro of pro ceeds easily
p n n p n
with jE j jF j Then Claim Let E F f g
u u v u v E u v E and
p n p n
u u v u v F u v F
p n p n
All strings u and v ab ove have length p n and is the bitwise XOR
i
function
Let G fy j hx y i B g Then we can apply the claim to either G or G
dep ending on whether x A or x A to prove the lemma
Graph Isomorphism
x A with E G fy j hx y i B g
G fy j hx y i B g x A with F
and
x A with F G fy j hx y i B g
G fy j hx y i B g x A with E
The statements u v E u v E and u v E
p n
u v E with E fy j hx y i B g are in Pos C and C is closed under
p n
Pos so A C and A C
This completes the pro of of Lemma mo dulo the pro of of the claim
which is done in the next two exercises ut
Exercise Prove part of the claim
Hint Show the existence of u u by means of a probabilistic con
p n
struction that is show that a random choice of u s has the desired prop erty
i
with non zero probability therefore such u s must exist C
i
Exercise Prove part of the claim
Hint Use an indirect pro of and a combinatorial counting argument C
Several inclusion relations are now immediate consequences of this result
P P
Theorem BPP
P P
Proof BPP BP P P P ut
P
Theorem BP NP
P
Proof BP NP NP NP ut
The following lemma along with Theorem is the key to the main
result of this section namely that GI is almost surely not NP complete
P P
BP NP Lemma If coNP BP NP then PH
Exercise Prove Lemma C
Corollary If GI is NP complete then the polynomial hierarchy col
lapses to the second level
Proof If GI is NP complete then GI is coNP complete By Theorem it
follows that coNP BP NP From this the collapse follows by Lemma
ut
Topic
References
For background on probabilistic complexity classes and the BP op erator see
U Sch oning Probabilistic complexity classes and lowness Journal of
Computer and System Sciences
S Zachos Probabilistic quanti ers adversaries and complexity classes
an overview Proceedings of the st Structure in Complexity Theory Con
ference Lecture Notes in Computer Science Springer
More background on the graph isomorphism problem and its structural
prop erties can b e found in
C M Ho mann Group Theoretic Algorithms and Graph Isomorphism
Springer
J K obler U Sch oning J Tor an The Graph Isomorphism Problem Its
Structural Complexity Birkh auser
The original pro of of Theorem was very indirect and followed from results
in
O Goldreich S Micali A Wigderson Pro ofs that yield nothing but their
validity and a metho dology of cryptographic proto col design Proceedings
of the Symposium on Foundations of Computer Science IEEE
S Goldwasser M Sipser Private coins versus public coins in interactive
pro of systems in S Micali ed Advances in Computing Research Vol
Randomness and Computation JAI Press
Theorem is due to
C Lautemann BPP and the p olynomial hierarchy Information Process
ing Letters
which includes the technique we used to prove Lemma and
M Sipser A complexity theoretic approach to randomness Proceedings
of the th Symposium on Theory of Computing ACM
Theorem is from
L Babai Trading group theory for randomness Proceedings of the th
Symposium on Theory of Computing ACM
where the result was expressed in terms of Babai s class AM Arthur Merlin
games which is a sp ecial version of interactive pro of systems AM has b een
shown to b e equivalent to BP NP
Graph Isomorphism
For Lemma and its implications for GI see
R B Boppana J H astad S Zachos Do es co NP have short interactive
pro ofs Information Processing Letters
U Sch oning Graph isomorphism is in the low hierarchy Journal of Com
puter and System Sciences
U Sch oning Probabilistic complexity classes and lowness Journal of
Computer and System Sciences
Topic
The BP Op erator and the Power of
Counting Classes
With the help of the BP op erator To da achieved an astonishing
result the classes P and P are in a sense at least as expressive as the
entire p olynomial hierarchy
The class P read parity P consists of all languages L for which there is
a nondeterministic p olynomial time b ounded machine M so far this sounds
just like the de nition of NP with the following prop erty
x L the numb er of accepting computations of M on input
x is odd
The di erence b etween this and the class NP is that the numb er of accepting
paths is required to b e o dd rather than merely non zero
This class represents a rudimentary form of counting of the numb er of
accepting computations In the full counting task which we will discuss
shortly one would like to know all of the bits of the binary representation
of this numb er the languages in the class P are de ned solely in terms of
the lowest order bit i e the parity
What can one b egin to do with this parity information Or in other words
How p owerful is the class P It is clear that P is in P and that P is closed
under complement
Exercise Why C
But how do es this class compare with the class NP The only immediate
conclusion is that P NP co NP P The relationship b etween NP and
P is unclear Most likely they are incomparable
Topic
Sketch
P
NP co NP
P
Each p ositive instance of an NP problem can have a di erent numb er
in general up to exp onentially many witnesses for memb ership in the
language More concretely for a satis able b o olean formula F i e F SAT
there can b e exp onentially many satisfying assignments If we now restrict
NP to the languages for which each p ositive instance has at most p olynomially
many witnesses we obtain the class FewP That is a language L is in FewP
if there is an NP machine that accepts the language L and has the additional
prop erty that for each input x there are at most p jxj accepting paths
where p is a suitable p olynomial
Clearly P FewP NP It is not clear how much of a restriction it is to
require few witnesses We do however have the following result
Theorem FewP P
Exercise Prove this theorem
Hint Let m b e the correct numb er of accepting computations of the FewP
P
m
m
m
is o dd if and only if m C machine Then
i
i
Sketch
P
p
NP P
p
p
p
p
p
p
p
p
p
p
p
p
p
p
p
FewP
The Power of Counting Classes
It is an interesting coincidence that in the examples known to this p oint
such as the prime numb er problem or the equivalence problem for tourna
ments see the b o ok by K obler Sch oning and Tor an the prop erty of b e
longing to P seems to b e connected with the prop erty that faster algorithms
log n
are known with upp er b ounds something like O n than are known for
the NP complete languages For the problems just mentioned it is not clear
whether they are in P It is unknown if there is a causal link b etween P
log n
and the complexity O n
In this sense one can interpret P has a low complexity class In the
last chapter we applied the BP op erator to various complexity classes such as
P and NP and interpreted this as merely a certain probabilistic generalization
which did not signi cantly alter the prop erties of the underlying complexity
P is a low class is however classes The intuition that therefore BP
P and completely incorrect Valiant and Vazirani showed that NP BP
P This will b e the main To da extended this to show that even PH BP
result of this chapter
We b egin with a few observations ab out the class P This class has com
plete languages under p olynomial time many one reductions for example
SAT fF j F is a b o olean formula with an o dd numb er of satis
fying assignments g
This follows from Co ok s Theorem which demonstrates the NP completeness
of SAT and the observation that the Co ok formula has the same numb er
of satisfying assignments as the underlying Turing machine has accepting
computations
We say that a language L has AND functions if there is a p olynomial time
computable function f such that
x L and and x L f hx x i L
n n
Analogously we can de ne the notion of an OR function and a NOT function
for a language
Exercise Show that SAT has AND functions and OR functions Fur
thermore SAT has NOT functions if and only if NP coNP C
What is the situation with SAT We will see that SAT has AND
OR and NOT functions The AND function for P is demonstrated as for
SAT by the function f hF F i F F where the formulas F
n n i
have disjoint sets of variables Let a b e the numb er of satisfying assignments
i
for F Then the numb er of satisfying assignments for the formula F F
i n
Q
n
a which is o dd exactly when every a is o dd is precisely
i i
i
Topic
A NOT function can b e obtained by adding on a dummy satisfying
y where x x are the assignment g F F y x x
n n
variables that o ccur in F and y is a new variable The formula g F clearly
has exactly one more satisfying assignment than the original formula F
By combining these two formulas and using DeMorgan s laws we imme
diately obtain an OR function
h x x g f g x g x
n n
P Theorem NP BP
Proof It is su cient to show that SAT is in a certain sense probabilistically
reducible to SAT More precisely each input formula F can b e transformed
by a probabilistic p olynomial time algorithm into a formula F with the
prop erty that
F SAT P r F SAT
p jF j
F SAT F SAT
for some p olynomial p
Exercise Show that Theorem follows from the preceding state
ment
Hint For the required probability ampli cation use the fact that the language
SAT has OR functions C
Let the input formula F have n variables x x Let S b e a random
n
subset of f ng i e S is sp eci ed by n random bits the ith bit deter
L
x That mining whether i S We denote by S the b o olean formula
i
i S
is S is a formula that is true precisely when there are an o dd numb er of
i S for which x The probabilistic algorithm now transforms F as
i
follows
INPUT F
GUESS RANDOMLY k f n g
GUESS RANDOMLY subsets S S f ng
k
OUTPUT F F S S
k
Intuitively with each addition of a subformula of the form S to the con
junction the numb er of satisfying assignments is approximately halved since
for each assignment b the probability is that b S and that
b S Without loss of generality assume that b These
events are however not completely indep endent but only pairwise indep en
The Power of Counting Classes
dent This makes the analysis of the probabilities somewhat more di cult
But it seems at least plausible that after one of these halving steps there will
b e a non negligible probability of having exactly one satisfying assignment
left
It is clear that if F is unsatis able then F is also unsatis able and
therefore has an even numb er namely of satisfying assignments so that
F SAT Now supp ose that F has m satisfying assignments With
k k
probability at least n k will b e chosen so that m Now
we show that with this choice of k the probability that F has exactly one
satisfying assignment is at least Thus the probability that F SAT
is at least n n
Let b b e a xed satisfying assignment of F Since the subsets S i
i
k are chosen indep endently the probability that b is also a satisfy
k
ing assignment of F i e that it survives each halving is Under
the condition that b survived the probability of any other satisfying assign
k
ment b for F also survives is also Thus the probability that b survives
but none of the other m satisfying assignments for F survive is at least
k
X
m
k
k k k k k k
b
So the probability that there is such a b that is the only satisfying assign
ment for F is at least
X
k k k k
m
b
With that Theorem has b een proven ut
Exercise Show the following generalization of the preceding theorem
P C P BP
The preceding pro of and exercise actually show not only that NP
P resp ectively but that for any p olynomial q with P BP P BP
a suitable choice of a P language L the probability that F SAT but
q n
F L can b e made less than
To da showed that this inclusion can b e signi cantly strengthened
P Theorem PH BP
Proof We show by induction on k that the following claim holds
P P
P and the error proba Claim For all k is contained in BP
k k
q n
bility can b e made to b e less than for any p olynomial q
Topic
The base case when k is trivial Now we show the claim holds for
k under the assumption that the claim holds for k It su ces to show
P
P is closed under complement P since BP that is contained in BP
k
P P
Let L b e an arbitrary language in let p b e the p olynomial
k k
that b ounds the length of the existentially quanti ed strings and let q b e
q n
an arbitrary p olynomial which gives the error rate that must b e
P and an error rate of at achieved By the inductive hyp othesis L BP
q n p n
most may b e assumed The BP op erator can b e pulled to the
P with an error rate now front see previous chapter so we get L BP
q n
P and of at most The previous exercise implies that L BP BP
that the error rate of the second BP op erator can b e chosen to b e at most
q n
Now we can combine b oth BP op erators into one In the worst
P with an error rate of at case the error rates add So we get that L BP
q n
most as was desired ut
As a corollary to this result in combination with techniques from the
previous chapter we get the following immediately
Corollary If P is contained in the polynomial time hierarchy PH
then PH col lapses ut
Exercise Prove Corollary
Hint Use the fact that P has complete languages like SAT for example
C
P has complete languages This Exercise Prove that the class BP
seems to b e unusual for a BP class although for example BP PSPACE
PSPACE and of course PSPACE also has complete languages
Hint Use the facts that P P P which we have already implicitly
shown and that Theorem relativizes that is for all classes C PH C
P PH P P P C From this we can conclude that BP BP
P C P P BP BP
Let acc b e a function from to N that gives for each input x
M
the numb er of accepting computations of M when run with input x Let P
b e the set of all functions of the form acc for some NP machine M
M
We want to investigate the question of whether one can compute lan
guages in the p olynomial time hierarchy using the information given by a
P function In fact we will show that
P P P Theorem BP
The Power of Counting Classes
P From this it follows that PH P P Indeed for any language A BP
we will show how to construct an NP machine M such that from the value of
acc x we can determine whether or not x A by means of a simple arith
M
metic computation in p olynomial time Consider the following example
P means that A BP
x A for many strings y there is an o dd numb er of strings z
such that hx y z i B
x A for only a few strings y is there an o dd numb er of strings
z such that hx y z i B
where B is a language in P For the sake of concreteness let s assume that
there are p otential y strings and p otential z strings Many could mean
in this case or few or Then the following tree represents a situation
where x A An a in the diagram represents that hx y z i B
x A
H
H
J
J H
H
J H
y
H
J H
H
J H
A A
A A
A A
A A
z
A A
A A
A A
A A
a a a a a
In this case there are y s with an o dd numb er of z values
In the next picture there is only one such y so x A
x A
H
H
J
J H
H
J H
y
H
J H
H
J H
A A
A A
A A
A A
z
A A
A A
A A
A A
a a a a a
If we were to ignore the structural di erence b etween y and z however
and view the entire tree as an NP machine M with input x then we could
not learn anything from the numb er of accepting paths acc x since in
M
b oth cases acc x M
Topic
Now we want to mo dify the z subtrees to z subtrees in such a way that
the following holds
Odd numb er of z s the numb er of z is congruent to mo d
Even numb er of z s the numb er of z is congruent to mo d
The numb er was chosen in this case b ecause and is the numb er of
p otential y s In addition it simpli es matters to cho ose a p ower of
In the rst example ab ove we now get
x A
H
H
J
J H
H
J H
y
H
J H
H
J H
A A A A
A A A A
z
A A A A
A A A A
A A A A
z z z z
mo d mo d mo d mo d
z
mo d
And in the second case
x A
H
H
J
J H
H
J H
y
H
J H
H
J H
A A A A
A A A A
z
A A A A
A A A A
A A A A
z z z z
mo d mo d mo d mo d
z
mo d
Now the total numb er g of accepting computations is su cient to di er
entiate the two cases x A if g or g mo d and x A if g
p jxj p jxj
or g mo d In general x A if and only if g mo d
The Power of Counting Classes
where p is a large enough p olynomial and g is the total numb er of accepting
computations For an appropriate machine N g acc x So it turns out
N
that the determination of whether x A dep ends only on a single bit of the
binary representation of acc x a P function Note the pro of of inclu
N
P P P do es not dep end on the fact that the BP op erator has sion BP
a gap b etween the accepting and rejecting probabilities or that this gap
can b e ampli ed the inclusion would also b e true if we had de ned the BP
op erator without this gap see the previous chapter
Our goal has now b een reached provided we can transform an arbitrary
NP machine M into an NP machine M such that for some su ciently large
p olynomial p we have
p jxj
acc x o dd acc x mo d
M M
p jxj
x mo d acc x even acc
M M
The p olynomial p dep ends on the numb er of y in the preceding example
p n
should b e larger than the numb er of p otential y i e p n must b e
larger than the maximum p ossible length of such a y In order to ful ll
this prop erty it su ces to give a general construction that works for every
polynomial p
For this consider the following magic formula let p p jxj
p p
acc x
M
Now we run through each case acc x even or o dd
M
Case acc x even
M
p p
acc x
M
z
mo d
z
p
mo d
z
p
mo d
z
p
mo d
Topic
Case acc x o dd
M
p p
acc x
M
z
mo d
z
mo d
z
mo d
z
p
mo d
In each case the chain of implications is understo o d to go from top to
b ottom
Exercise In b oth cases we used the implications
p p
a mo d b a mo d b
p
a mo d b a mo d b
Prove them C
Exercise Let M b e an NP machine Show that the function
p jxj p jxj
f x acc x
M
is in P C
P P P ut This completes the pro of that PH BP
The diagram b elow summarizes the known inclusion relationships The
language class PP which app ears in the diagram is closely related to the
function class P Let M b e an NP machine and p b e a p olynomial such
that every computation on every input of length n runs in time p n Then
p n
there are di erent p ossible computations on each input of length n A
language A is in PP if there is such a machine M and p olynomial p for which
p jxj
x A acc x i e at least half of the computations on
M
input x are accepting It is easy to see that NP PP P P P PP and
PP is closed under complementation
The Power of Counting Classes
PSPACE
P P P PP
P BP
l
l
PP
l
l
PH
l
l
P
NP
P
References
The class P was intro duced by C H Papadimitriou and S Zachos
Two remarks on the p ower of counting Proceedings of the th GI Con
ference in Theoretical Computer Science Lecture Notes in Computer
Science Springer
There P P P is proven
The class FewP originated with
E Allender Invertible Functions PhD thesis Georgia Tech
The inclusion FewP P app ears in
J Cai L A Hemachandra On the p ower of parity p olynomial time Pro
ceedings of the Symposium on Theoretical Aspects of Computer Science
Lecture Notes in Computer Science Springer
The BP op erator was rst de ned in
U Sch oning Probabilistic complexity classes and lowness Journal of
Computer and Systems Sciences
P formulated di erently is from The result NP BP
Topic
L G Valiant V V Vazirani NP is as easy as detecting unique solutions
Theoretical Computer Science
The extension of this inclusion to PH and the inclusion PH P P was
achieved by
S To da PP is as hard as the p olynomial time hierarchy SIAM Journal
on Computing
P come Our approximations of the probabilities in the pro of of NP BP
from
C H Papadimitriou Computational Complexity Addison Wesley
page
Our magic formula is a simpli ed version of the formulas found in
R Beigel J Tarui On ACC Proceedings of the Symposium on Founda
tions of Computer Science IEEE
J K obler S To da On the p ower of generalized MOD classes Mathe
matical Systems Theory
Interactive Pro ofs and Zero Knowledge
In Goldwasser Micali and Racko intro duced the notion of an inter
active pro of system Using these systems probabilistic generalizations of
the complexity class NP can b e de ned A zero knowledge proto col is able
to provide convincing evidence that a pro of of a statement exists without
disclosing any information ab out the pro of itself
In Topic among other things the relationship b etween e cient pro ofs and
the class NP was discussed As we noted there the class NP can b e viewed as
the set of all languages that have p olynomially long pro ofs in an appropriate
pro of calculus
Exercise Sketch a pro of of this characterization of NP The de nition
of a pro of calculus is intentionally omitted x it appropriately for the pro of
C
Here we want to consider such pro ofs more generally as a communication
problem b etween a prover who knows the pro of the prover and a veri er who
do es not know the pro of but is supp osed to b e convinced of its correctness
or p erhaps only of its existence In our previous considerations i e in the
de nition of the class NP given ab ove the communication has b een entirely
one sided the prover gives the veri er a complete pro of which the veri er
then checks in p olynomial time
We want to recast this now as a communication problem b etween two
Turing machines prover and veri er The veri er will b e a p olynomial time
b ounded Turing machine Later we will also consider probabilistic veri ers
cf Topic The prover on the other hand will have no complexity re
strictions which in some sense corresp onds to the existential quanti er in the
de nition of NP These two machines communicate over a common communi
cation channel a Turing machine tap e and b oth machines also have access
This idea follows the manner in which pro ofs were often written down in antiq
uity namely in the form of a dialog b etween someone carrying out a pro of and
another party who doubts its validity In more mo dern form such dialog pro ofs
can b e found for example in
P Lorenzen Metamathematik Bibl Inst
Topic
to the input which in some sense represents the statement to b e proven Fur
thermore each machine has a private work tap e for internal computations
input
tap e
R
Prover
Veri er
I
communication
R
tap e
work tap e work tap e
The computation pro ceeds in rounds In each round only one of the par
ticipants is active A round b egins with the reading of the information on
the communication tap e or on the input tap e and ends p erhaps after
some private computations with writing at most p olynomially much new
information on the communication tap e The numb er of rounds is limited
to b e p olynomial in the length of the input Since b oth participants can in
principle keep a complete log of all communication on their work tap es the
newly computed information at each round can b e considered a function of
the input and all previously communicated information In the case of the
veri er this function must b e computable in p olynomial time The prelimi
nary de nition of the language A that is represented by such an interactive
proof system is the following There must b e a veri er algorithm V so that for
any x A there is a prover strategy so that the veri er eventually accepts
If x A then the veri er rejects regardless of what strategy the prover uses
Note that we have made no restrictions on the complexity of the prover
Exercise Show that using this apparently more general interactive
communication mo del nothing more can b e computed than the class NP C
The situation changes drastically at least p otentially if the Turing ma
chines in particular the veri er are allowed to work probabilistically In this
case the participants can make the execution of their computations dep end
on the result of a random exp eriment Furthermore we relax the requirements
for x A and x A somewhat Our nal de nition go es as follows
De nition A language A is in the class IP if there is a probabilistic
polynomial time bounded Turing machine V the veri er such that for al l x
x A Prover P P r P V x
x A Prover P P r P V x
Interactive Pro ofs and Zero Knowledge
In the de nition ab ove P V x means that the result for a given
prover P and veri er V is that x is accepted This de nition was intro duced
by S Goldwasser S Micali and C Racko
The constants and are somewhat arbitrary Because of the p os
sibility of probability ampli cation in this mo del cf Topic the exact
values of the constants do not matter as long as the rst one has the form
and the second one the form for some constant In
jxj
fact one can cho ose the rst constant to b e and the second to b e
jxj
It is clear from the previous discussion that all NP languages have interac
tive pro ofs In fact they have trivial ones in which all of the communication
is from the prover to the veri er So NP IP Since the veri er can compute
the languages in BPP without any communication from the prover it is also
clear that BPP IP
Exercise Show that IP PSPACE C
Exercise A language A is provable by an oracle if there is a p olynomial
time b ounded probabilistic Turing machine M such that for all x
B
x A oracle B P r M x
B
x A oracle B P r M x
Show that A IP implies that A is provable by an oracle Give an argument
why the reverse direction might not b e true C
So just how large is the class IP It is contained in PSPACE and contains
NP How much more of PSPACE is in IP A well known example of a problem
in IP that is not known to b e in NP is the complement of graph isomorphism
GI cf Topic given two graphs G and G prove interactively that
they are not isomorphic One interactive proto col that achieves this is the
following
Prover communication Veri er
Randomly guess i
f g and a p ermuta
tion of f ng
where n is the numb er
of no des in the graphs
G and G Compute
the graph H G
i
H
Determine j f g
so that G and H are
j
isomorphic
j
Accept if i j
In fact even more is true It can b e shown that the rst constant can b e chosen
to b e see the pap er by Goldreich Micali and Sipser
Topic
Now if G and G are not isomorphic then a suitable prover algorithm
can always answer with the correct j rememb er the prover is not computa
tionally limited and so is able to nd any isomorphisms that exist so the
veri er will always accept That is
G not isomorphic to G Prover P P r P V G G
On the other hand if the two graphs are isomorphic then the prover
has at most a chance of selecting the correct value i j So
G isomorphic to G Prover P P r P V G G
Exercise For which provers is the probability exactly What is the
probability in other cases C
The de nition of IP has not yet b een satis ed since the probability of error
is which is greater than
Exercise Mo dify the proto col ab ove slightly without changing the
communication proto col so that probabilities and ab ove are trans
formed to and for some C
Exercise How can the proto col b e mo di ed so that instead of the
k
probabilities and we get the probabilities and C
GI IP It is imp ortant to note that the proto col describ ed ab ove uses So
only a constant numb er of rounds namely So GI IP where IP k is
the subset of IP where only k rounds are allowed If we really make use of the
p olynomially many rounds available then we arrive at IP PSPACE This
was rst shown by A Shamir and will b e the sub ject of Topic
It app ears at rst glance to b e signi cant that the veri er makes his ran
dom choices i and on the private work tap e out of sight of the prover
S Goldwasser and M Sipser showed however that this secrecy is
not necessary Every IP proto col like the one ab ove can b e transformed into
one in which the veri er do es nothing but generate random numb ers and
communicate these directly to the prover without subsequent internal com
putation This kind of proto col was intro duced indep endently by L Babai
who called it an Arthur Merlin game b ecause the prover plays the role of the
wizard Merlin and the veri er the role of King Arthur
Example The metho d used in Topic to demonstrate that GI BP NP
can b e recast as an IP proto col with public coins as follows
Interactive Pro ofs and Zero Knowledge
Prover communication Veri er
Randomly guess a
hash function h H
h
Determine a y Y
with h y Let b b e
a pro of of y Y
y b
Accept if b is correct
and h y
For a discussion of the terminology and probability approximations in this
example see the description given in Topic In fact it is the case that
BP NP IP
Now we come to another interesting concept also intro duced by Gold
wasser Micali and Racko namely zero know ledge This means that an IP
pro of can b e carried out in such a way that the veri er do es not obtain
any information ab out the pro of itself but is nevertheless convinced of the
existence of such a pro of since the IP proto col is still correct
The most famous example of this was given in a pap er by O Goldreich
S Micali A Wigderson and is the graph isomorphism problem GI In order
to convince the veri er that two graphs are isomorphic the simplest thing
the prover could do would b e to give the veri er an isomorphism co ded in
p olynomially many bits which the veri er could then easily check But then
the whole secret would b e revealed the veri er could use this information to
tell a third party etc In certain contexts cryptologists rack their brains
over such contexts it may well b e that it is not desirable that the pro of b e
revealed but that the veri er must nevertheless b e convinced that the prover
knows a pro of a graph isomorphism in our example A zero knowledge
pro of of knowledge is actually able to satisfy these apparently paradoxical
requirements
Consider the following proto col for GI Contrary to our previous exam
ple in this proto col it is the prover who b egins and the prover also uses randomness
Topic
Prover communication Veri er
Randomly guess i
f g and a p ermuta
tion of f ng
where n is the numb er
of no des in the graphs
G and G Compute
the graph H G
i
H
Randomly select a j
f g
j
Determine so that
G H
j
Accept if G H
j
Exercise Show that this is an IP proto col for GI C
This proto col is unusual however in that it do es not reveal anything
ab out the isomorphism if one exists b etween the graphs G and G The
proto col ful lls our de nition of zero knowledge The information that is
transmitted on the communication channel in this case H j and contains
statistically no new information for the veri er The veri er would b e able
with his computational resources to generate exactly the same probability
distribution for the triples H j as would o ccur in a typical realization
of the proto col Therefore the veri er can learn absolutely nothing new by
viewing the communication log
De nition Goldwasser Micali Racko Suppose A IP via a
prover veri er pair P V Then this protocol is a zero knowledge protocol
if there is a probabilistic Turing machine M such that M on any input x
runs in polynomial time and if x A then M outputs a tuple y y
k
so that the distribution of such tuples is exactly the distribution that can be
observed on the communication channel of P and V on input x
Exercise Show that the proto col given ab ove for GI is a zero
knowledge proto col C
Exercise Some authors have criticized that since the prover in the def
inition of IP has unb ounded resources these typ es of IP pro ofs of which the
zero knowledge ones are most interesting are not practically implementable
But let s mo dify the scenario slightly Now the prover like the veri er
will also b e a probabilistic p olynomial time b ounded Turing machine but at
More precisely what we have de ned here is perfect zero know ledge with xed
veri er in contrast to other de nitions of zero knowledge that exist in the liter
ature Under weaker de nitions one can show that al l NP languages have zero
knowledge IP pro ofs but this do es not app ear to b e the case for our de nition
For more on this see the article by Brassard
Interactive Pro ofs and Zero Knowledge
the start of the proto col the prover has some additional information namely
the pro of is written on its work tap e In our example this would b e an
isomorphism b etween the graphs G and G if one exists Show that the
proto col describ ed ab ove can b e used in this situation so that the prover can
in p olynomial time convince the veri er of the existence of an isomorphism
without revealing the isomorphism That is zero knowledge pro ofs are still
p ossible with such mo di cations to the mo del C
References
Zero knowledge pro ofs are an imp ortant topic in cryptology they can b e
used for example to provide secure identi cation or authorization in the
presence of eavesdropping See
A Fiat A Shamir How to prove yourself practical solutions to identi
cation and signature problems CRYPTO Lecture Notes in Computer
Science Springer
Surveys of interactive pro ofs systems and zero knowledge can b e found in
J L Balc azar J Diaz J Gabarr o Structural Complexity II Springer
Chapter
D P Bovet P Crescenzi Introduction to the Theory of Complexity
Prentice Hall Chapter
J Feigenbaum Overview of interactive pro of systems and zero knowl
edge Chapter in G J Simmons ed Contemporary Cryptography The
Science of Information Integrity IEEE
O Goldreich Randomness interactive pro ofs and zero knowledge in
R Herken ed The Universal Turing Machine A Half Century Survey
Oxford University Press
S Goldwasser Interactive pro of systems in J Hartmanis ed Compu
tational Complexity Theory Pro ceedings of the Symp osium in Applied
Mathematics Vol AMS
J K obler U Sch oning J Tor an The Graph Isomorphism Problem Its
Structural Complexity Birkh auser Chapter
C Papadimitriou Computational Complexity Addison Wesley
Section
Primary sources used in writing this chapter include
Babai Trading group theory for randomness Proceedings of the th
Symposium on Theory of Computing ACM
Topic
G Brassard Sorting out zero knowledge Advances in Cryptology
Lecture Notes in Computer Science Springer
O Goldreich S Micali M Sipser Interactive pro of systems provers
that never fail and random selection Proceedings of the Symposium on
Foundations of Computer Science IEEE
O Goldreich S Micali A Wigderson Pro ofs that yield nothing but
their validity or all languages in NP have zero knowledge pro of systems
Journal of the ACM
S Goldwasser S Micali C Racko The knowledge complexity of inter
active pro of systems SIAM Journal on Computing
S Goldwasser M Sipser Private coins versus public coins in interactive
pro of systems Proceedings of the th Symposium on Theory of Com
puting ACM
S Micali ed Randomness and Computation Vol of Advances in Com
puting Research JAI Press
IP PSPACE
We present the surprising result of A Shamir that the classes IP and
PSPACE are the same
For a long time it was unclear how encompassing the class IP really is The
conjecture was that this class represented only a small generalization of the
class NP something like BP NP cf Topic It was not even clear that the
class coNP was contained in IP In fact there were oracle results that sp oke
against this cf Topic
Triggered by the observation of N Nisan that by means of a certain
arithmetization technique and the use of To da s results cf Topic one
could show that PH IP which implies of course coNP IP as well there
ensued a race toward the p otential goal of IP PSPACE Rememb er that
in Topic we saw that IP PSPACE This race to ok place by e mail in
Decemb er among certain insiders see the article by L Babai This
race was eventually won by A Shamir We want to work through the pro of
of this result in this chapter
The strategy is to give an interactive pro of proto col for QBF a PSPACE
complete problem thus proving that QBF IP From this it follows that
PSPACE IP and therefore that IP PSPACE The language QBF consists
of all valid quanti ed b o olean formulas with no free variables Quanti cation
is over boolean variables f g or fFALSE TRUEg
Example The following formula is in QBF
x y x y z x z y z
It will b e imp ortant in what follows that we only allow formulas in which
the negations are only applied to variables Let QBF QBF b e the set of
all such formulas that have no free variables and are valid
P
Exercise Show that QBF QBF so it is su cient to prove that
m
QBF IP C
The goal of the desired interactive pro of proto col is to get the prover to
convince the veri er that the input formula F is valid and if F is not valid
then the veri er should reject with high probability
Topic
The trick is to interpret the formulas arithmetical ly We do this in the
following way We assume that the variables can take on integer values
An AND op eration will b e interpreted as multiplication and an OR
op eration as addition If the variable x is negated we interpret this as
i
x
i
It remains to show how the quanti cation can b e arithmetized For a
formula of the form x every free o ccurrence of x in F is replaced once with
and once with and each of the resulting formulas is then arithmetized
Let a a Z b e the resulting values The value of xF is then a a
For xF we use a a instead Let b o ol F denote the b o olean value of a
formula F and let arith F denote its arithmetic value
Example We can compute the value of the formula in the previous example
as follows
x y x y z x z y z
X Y Y
x y x z y z
z x y
Y Y
x y x y x y
x y
Y
x x x x
x
Y
x x
x
Exercise Show that
b o ol F TRUE arith F
b o ol F FALSE arith F
Hint Use structural induction on the formula F C
Instead of convincing the veri er that the b o olean value of a formula is
TRUE the prover will instead rst communicate to the veri er a value a Z
and then try to convince the veri er that the arithmetic value is exactly a
It is precisely this more di cult task which demands more information from
the prover that makes a correct interactive proto col for QBF p ossible
Now we come to our rst technical problem The arithmetic value a is
supp osed to b e communicated The prover with unb ounded resources can
clearly compute this numb er from the formula F a task that by the previous
exercise is at least as di cult as QBF But the value of a could b e so large
that the binary representation of a do esn t have p olynomial length
IP PSPACE
Exercise Show that for the following formula
F x x y z y z
m
m
arith F C
Exercise Show that if the length of a formula F as a string is n then
n
arith F C
So we must nd a way to reduce the arithmetic value of the formulas
O n
We can do this by computing mo dulo some numb er k of size Then we
can represent a mo d k using only O n bits But we must make sure that
the prop erties a for valid formulas and a for invalid formulas
continue to hold when this mo dular arithmetic is used
n
It is the case however that for all a with a there is a prime
n n
numb er k in the interval such that a mo d k
Exercise Show this You may use without pro of the fact that for every
p p
m there are at least m prime numb ers m i e m m
Hint Chinese Remainder Theorem C
In Topic we gave lower b ounds for n but these were go o d only for
in nitely many n The Prime Numb er Theorem actually says more namely
that n n ln n But for the exercise ab ove the following weaker version
would su ce
p
n C Exercise Show that n
Our desired proto col for QBF IP b egins with the following information
b eing communicated from the prover to the veri er in the rst round
Prover Communication Veri er
Compute n jF j com
pute a arith F and
determine a prime num
n n
b er k if p os
sible with a mo d
k Let a a mo d k
Find a pro of b for the
primality of k
a k b
Verify that a that
n n
k and that b
is a correct pro of that k
is a prime
In the remainder of the proto col the prover has the task of convincing the
veri er that the arithmetic value of F mo dulo k really is a
At this p oint it is not yet clear why the prover must convince the veri er
that k is a prime numb er since even if k is comp osite we can conclude from
Topic
a that a So far the only place primality has played any role has
b een in the context of the Chinese Remainder Theorem ab ove But in what
follows it will b e imp ortant for the veri er to know that f k g
GF k is a eld Otherwise the prover would b e able to cheat as we will
so on see
Nor have we discussed what b the pro of of primality is like It is in fact
p ossible to certify the primality of a numb er via a p olynomially long non
interactive pro of that is PRIMES NP This fact was originally due to
V Pratt but we will not pursue the pro of further here See the references
at the end of the chapter for more information
Another p ossibility is to have the veri er cho ose a prime numb er from a
somewhat larger interval at random in the rst round Metho ds for selecting
a random prime can b e found in the b o ok by Cormen Leiserson and Rivest
If the veri er selects such a prime p at random then with high probability
a mo d p
Exercise In this alternative how large must the interval from which
n
the prime numb er k is chosen b e in order that for every a the
n
probability is at least that a mo d k C
Now we come to the heart of the proto col verifying that arith F mo d
k a where F is for the moment the input formula This will require
multiple rounds Each round will b e asso ciated with some claim of the form
form arith F a mo d k where initially F is the input formula and a
is the value communicated by the veri er in the rst round In subsequent
rounds F will b e some shorter formula that o ccurs as a subformula or
instantiation of a formula F from a previous round and a will b e the value
that the prover claimed was arith F mo d k
If for example F has the form F F F then in the next round
the prover must give two numb ers a and a along with pro ofs that a is the
arithmetic value of F mo dulo k and that a is the arithmetic value of F
mo dulo k The veri er will check to b e sure that a a a mo d k The
pro cedure for F F F is analogous except that the veri er will check
that a a a mo d k
It b ecomes more interesting when F has the form F x G The variable
x o ccurs freely in G so the arithmetic value of G is not a numb er but a
function In fact it is a p olynomial since the only op erations are and
The prover will now b e required to tell the veri er the co e cients of this
p olynomial which we will call p since this task is to o di cult for the veri er
G
The veri er then checks that p p a mo d k selects a random
G G
numb er z f k g GF k and communicates this numb er z to the
prover The prover is now exp ected to provide a pro of that the arithmetic
value of G with the numb er z substituted for the variable x is precisely
IP PSPACE
p z In the case of a formula of the form F x G everything pro ceeds
G
analogously except that the veri er checks that p p a mo d k
G G
Example If F x y x y z x z y z as in our previous
examples then
X Y
x y x z y z p x
G
z y
x x x x
Note that this gives more information than merely giving the values p
G
and p
G
So at this p oint the proto col is doing the following In order to prove
that the value of F is a the prover must give the veri er the p olynomial
p Now the task is to verify that the p olynomial p is correct We will see
G G
b elow that these p olynomials have very low degree in comparison to the size
of the underlying eld GF k For this reason on substitution of a random
value it is very likely that if the prover tried to cheat by giving an incorrect
p olynomial which for the moment will not b e detected by the veri er then
in the next round the prover will b e forced to once again give an incorrect
p olynomial if he ever hop es to cause the the veri er to accept but now for
a smaller formula G z In the end the prover s deception will with very
high probability b e found out
Exercise Let d k b e the degree of the p olynomials p and p
Supp ose the prover tries or is forced to try b ecause of previous rounds to
show that p is the p olynomial asso ciated with G z when in fact it is really
p p What is the probability that given a random choice of z as describ ed
ab ove p z p z C
At the end of the proto col we arrive at the innermost parts of F the
variables which in the previous rounds have b een replaced by random num
b ers Now the veri er need only check that these random numb ers agree with
the arithmetic value
This concludes a description of the proto col by which the prover can
convince the veri er that the arithmetic value of a formula is exactly some
supp osed value Some comments ab out its correctness have already b een
made What is still missing is a rigorous approximation for the probability of
correctness and an argument that the degree of the p olynomials is small As
we saw in the previous exercise the degree of the p olynomial plays a role in
the probability arguments Furthermore the prover must communicate the
p olynomials The co e cients themselves come from GF n and so require
n
only O n bits since k so what we need is a p olynomial b ound on the
Technically G z is no longer a formula since an integer has b een substituted
for one of the variables
Topic
number of co e cients In order for this to b e p ossible the degree should b e
at most p olynomial in n
Without taking further precautions the degree of the p olynomial p could
G
b e exp onentially large It is the universal quanti ers that have the p ossi
bility of drastically increasing the degree If for example G has the form
y y H x y y and H is a quanti er free formula that has due
m m
to nested ANDs and ORs degree c with resp ect to x then the p olynomial
m
p has degree c In order to maintain a low degree for the p olynomials as
G
so ciated with a subformula G of F with some variables already instantiated
with random numb ers it is imp ortant that already in F the numb er of
universal quanti ers b etween an o ccurrence of a variable and the quanti er
that binds it b e small In what follows we will only allow one such intervening
universal quanti er
De nition A quanti ed boolean formula without free variables is said
to be simple if for every variable that occurs the number of universal quanti
ers that lie between any occurrence of the variable and its binding quanti er
is at most
We denote by QBF the set of simple formulas in QBF
Exercise Show that the degree of any p olynomial that comes from a
simple formula F of length n is at most n C
Exercise Show that QBF or QBF is p olynomial time reducible
to QBF Therefore it su ces for our interactive proto col to consider only
simple formulas
Hint For every o ccurrence of the situation Qx y x intro duce a
new variable to b e a place holder for x C
For the correctness of QBF IP we still need to give a b ound on
the probability of error in the proto col If the input formula F is valid then
the prover can satisfy all of the veri er s demands and pro duce the correct
arithmetic value and all the required p olynomials So in this case the veri er
will accept with probability
If on the other hand F is not valid then there is only a very small chance
that the veri er incorrectly accepts This can only happ en if in some round
the veri er randomly cho oses a numb er z such that the incorrect p olynomial
previously given by the prover and the correct p olynomial p have the same
G
value on z By the earlier exercises in any given round this can only happ en
n
with probability at most n Since there are at most n rounds we get the
following approximation for the probability that the prover can prosp er by
cheating
n
P r error n n
So the de nition of IP is satis ed It follows that IP PSPACE as was to
b e shown ut
IP PSPACE
Exercise Use P r error P r no error to get a b etter approxi
mation C
Finally we note that in this proto col the numb ers randomly chosen by
the veri er are given directly to the prover So this proto col is in fact of
Arthur Merlin typ e cf Topic
References
The tale of the e mail race toward IP PSPACE is told in
Babai E mail and the unexp ected p ower of interaction Proceedings of
the th Structure in Complexity Theory Conference IEEE
The original pro of app eared in
A Shamir IP PSPACE Journal of the ACM
Pro ofs can also b e found in the following b o oks
C Papadimitriou Computational Complexity Addison Wesley
D P Bovet P Crescenzi Introduction to the Theory of Complexity
Prentice Hall
C Lund The Power of Interaction MIT Press
The fact that PRIMES NP is from
V Pratt Every prime has a succinct certi cate SIAM Journal of Com
puting
See also
D Knuth The Art of Computer Programming Vol Semi Numerical
Algorithms Addison Wesley page Exercise
E Kranakis Primality and Cryptography John Wiley Sons Sec
tion
For an explanation of how to select prime numb ers at random see
T H Cormen C E Leiserson R L Rivest Introduction to Algorithms
MIT Press McGraw Hill Chapter
Topic
P 6 NP with probability
There are worlds in which P NP and others in which P NP Fur
thermore if a world is chosen at random the probability is that it will
b e a world in which P NP
If one is not familiar with the material that we are ab out to present then
one would probably assume without further thought that if one could show
that P NP then from this fact and a p otential pro of of this fact it
A A
would immediately follow that for any oracle P NP since although the
problem is shifted by the oracle its fundamental structure seems to remain
the same Similarly if we assumed the unlikely case that P NP again we
A A
would exp ect that by a similar argument one could show that that P NP
for all oracles
Such a relativization principle do es seem to hold in recursion theory
Every known theorem can b e relativized by the addition of an oracle mech
anism and is in fact still true with any choice of oracle But in complexity
theory things are di erent T Baker J Gill and R Solovay showed that there
are languages A and B such that
A A B B
P NP and P NP
Exercise Show that any PSPACE complete language can b e used for
B C
Now we want to show that for almost all choices of oracle P NP
More precisely what we mean is this If we generate the oracle language
A according to a random pro cedure in which for every string x f g
indep endently of all other strings is equally likely to b e in or out of A i e
xP r x A then
A A A A
P r P NP i e P r P NP
From this of course it follows that there exists an oracle relative to which
the classes P and NP are di erent
Topic
This result seems esp ecially attractive On a scale from to the needle
p oints all the way to the right Could this b e an indication of how the un
relativized question is answered i e that P NP unrelativized This way
of thinking namely that if a statement holds with probability relative to a
random oracle then one can conclude or b etter conjecture that the unrela
tivized statement also holds is referred to as the Random Oracle Hypothesis
It can b e shown that in statements of this typ e i e comparisons of com
plexity classes the probability will always b e either or Kolmogorov s
Law So the needle can only p oint all the way to the left or right of the
scale This is very suggestive Nevertheless the Random Oracle Hyp othesis
has b een refuted by a numb er of counterexamples
A A
Our goal is to show that for a random A P r P NP Let
M M b e a listing of all p olynomial time b ounded deterministic oracle
A
A
Turing machine That is P fL M j i g Momentarily we will de ne
i
A
an oracle dep endent language L A It will b e easy to see that L A NP
regardless of the oracle A Thus the only issue will b e whether or not L A
A
P We can approximate as follows
A A A
P r P NP P r L A P
A
P r i L M L A
i
X
A
P r L M L A
i
i
X
A
P r x x L M L A
i
i
where A B fx j x A x B g i e A B A B
First let s de ne the language L A For this we imagine that the elements
of f g are arranged in lexicographical order Whether x with jxj n
n
is in L A or not will b e determined by the n strings that follow x in
lexicographical order
x
f g
n n n n
n
n
n
These strings are divided into blo cks of n strings each x is in L A if and
only if there is at least one such blo ck of n strings all of which are in A
Exercise Given a random oracle A what is the probability that a string
x with jxj n is in L A To what value do es this probability converge as n
b ecomes large C
P NP with probability
A
Exercise Show that for any oracle A L A NP C
For every machine M let x x x b e a sequence of widely
i
separated strings For example if we cho ose n jx j jx j n
j j j
j
n
j
strings that determine the then we are guaranteed that the regions of n
j
memb ership of each x in A are all disjoint and therefore that the events
j
x L A j are completely indep endent
j
A
The events x L M i j however are not completely
j
i
indep endent in particular b ecause the machines M can also make very short
i
oracle queries But as we will see shortly we do get su cient indep endence
n
j
In this way if we cho ose the x s to b e so widely separated that n
i j
we can ensure that for large j machine M on input x cannot query strings
i j
of length n
j
Now we can continue the approximation b egun ab ove
A A
P r P NP
X
A
P r x x L M L A
i
i
X
A
P r j x L M L A
j
i
i
X Y
A A
P r x L M L A j x L M L A k j
j k
i i
i j
A
The next step is to prove an upp er b ound on the probability of x L M
j
i
L A or equivalently a lower b ound for the probability of a mistake
A
x L A L M under the condition that
j
i
A
x L M L A for all k j
k
i
We will refer to this condition as C and in the probability considerations
b elow we will always assume condition C which has the e ect of shrinking
the probability space of random oracles that we are considering Essentially
we consider the oracle A to b e xed on all strings that are resp onsible for
condition C all of which will b e much shorter than x
j
We can now sketch a partitioning of the probability space as in the dia
gram b elow
Topic
A A
x L M x L M
j j
i i
x L A
j
x L A
j
By Exercise we have
P r x L A j C P r x L A
j j
P r x L A j C P r x L A
j j
Now we approximate the conditional probability
A
P r x L A j x L M C
j j
i
A
P r x L A j x L M
j j
i
The imp ortant observation here is that in the course of its computation on
input x the machine M can only query p olynomially in m jx j many
j i j
strings of the oracle Let p b e a p olynomial that b ounds the numb er of
i
queries For the purp oses of our approximation we consider these at most
p m oracle strings to b e xed These strings can lie in at most p m di erent
i i
m
blo cks so there are at at least p m blo cks that are available for our
i
m
random exp eriment For large m p m so just as in Exercise
i
we can approximate as follows
m
A m p m
i
P r x L A j x L M
j j
i
m
m
p
m
m
since converges to e
Now we want to show that the probability of error is greater
than For this we consider two cases dep ending on the probability of
Case
Then the value of is at least
P NP with probability
Case
In this case since we have And since
it follows that
A
So in either case we have shown that P r x L A L M
j
i
Putting everything together we see that
X X Y
A A
P r P NP
i i j
A A A A
and so we have proven that P r P NP or P r P NP ut
A A
Exercise Show that P r NP coNP C
References
For oracles separating P from NP see
T Baker J Gill R Solovay Relativizations of the P NP question
SIAM Journal on Computing
A A A
C H Bennett J Gill Relative to a random oracle A P NP coNP
with probability SIAM Journal on Computing
For discussion of the random oracle hyp othesis see
R Chang B Chor O Goldreich et al The random oracle hyp othesis is
false Journal of Computer and System Sciences
S Kurtz On the random oracle hyp othesis Information and Control
Topic
Sup erconcentrators and the Marriage
Theorem
We want to study graphs with sp ecial extreme connectivity prop erties
prove that they exist and approximate their size In the next topic the
existence of these graphs will b e used to obtain certain lower b ounds
De nition An n sup erconcentrator is a directed acyclic graph with
n input nodes and n exit nodes such that for every choice of k input nodes
and k output nodes k n there exist node disjoint paths through the
graph that connect the k input nodes with the k output nodes in any order
The following sketch clari es this for n and a sp eci c choice of input
no des and output no des
g w
g w
w g
w w
w g
It is trivial that there are n sup erconcentrators with O n edges the
bipartite graph with n input no des n output no des and an edge connecting
each input no de with each output no de is an example
Exercise It is also relatively easy to build n sup erconcentrators with
O n log n edges using a recursive divide and conquer algorithm Do it C
It was susp ected that this b ound is optimal see for example Prob
lem in the b o ok by Aho Hop croft and Ullman Interestingly it is
p ossible to construct these graphs with only linear size where the size is the
numb er of edges
Theorem There is a constant c such that for every n there is an n
superconcentrator with cn edges
Proof The remainder of this topic will present one p ossible construction As
in Exercise the construction will b e recursive But instead of building
Topic
an n sup erconcentrator out of two smaller sup erconcentrators this time we
will use only one n sup erconcentrator for some in the recursive
step
g
h
J
J
J
g
h
J
J
J
X
X
J
X
X
J
J
G
G
S
J
J
X
X
X
J
X
The diagram ab ove indicates the construction First each of the n input
no des is connected directly to its corresp onding output no de In addition
there is a detour through the graph G which has n m inputs m out
puts and certain prop erties which we will describ e b elow The m output
no des of G are connected as inputs to the m sup erconcentrator S the out
puts of which are connected to G which is identical to G except that the
direction of the edges is reversed We ignore here and b elow the p ossibility
that m may not b e an integer For our asymptotic approximations this will
not play a role The fo cus of our attention will b e on the graph G It will b e
p ossible to construct such a graph with linear size
Exercise Assume that G has linear size and that the construction is
otherwise correct Show that the size of the resulting n sup erconcentrator is
cn for an appropriate constant c C
In what follows we will always assume that n m The desired graph
G will have the prop erty that any set of k n m input no des can b e
connected with no de disjoint paths to some set of k output no des The graph
G has the dual prop erty that for any choice of k n m outputs there
are always k input no des that can b e connected to them by no de disjoint
paths
Exercise Assuming these prop erties of the graphs G and G show
that the construction is correct
Hint How many inputs of the sup erconcentrator can b e connected directly
to output no des without going through the graphs G S and G C
The graph G will b e a bipartite graph that is the no des fall into two sets
n m input no des and m output no des and the edges connect a no de
from each set We must determine how to arrange linearly many edges to
get the desired prop erty
For a given bipartite graph with edge set E a subset M E is called
a matching if M consists of edges that are pairwise not connected We are
Sup erconcentrators
interested in the maximal size of matchings Before pro ceeding with the con
struction of G we want to consider maximal matchings in isolation
Example
v u
H
H
H
H
H
v u
X
X X
X
X
X
X H X
H
X
Q
X
X X
X
X
X H X
H X
X
X X
X
Q
X
X H X
H
X
X
X X
X
X
Q X H X
H X
X
X X
X
X
X H X
X H
X
X
Q
H
X
v u
X
H
X
Q
X
H
X
X
Q H
X
X
H
X
Q
X
H
X
H
X
Q
v u
Q
Q
Q
v u
The thick lines indicate a matching M with jM j A matching with
edges is not p ossible since u u u are only connected with no des v
and v Therefore in any matching one of the no des u u and u must b e
omitted
Guided by this example we arrive at the following theorem
Theorem Let G be a bipartite graph with n input nodes Let S be a
subset of the input nodes with jS j n Then there is a matching starting
with S and connecting the nodes in S to some jS j exit nodes if and only
if for every subset S of S jS j jN S j where N S is the set of nodes
connected to the nodes in S i e the set of potential partners for the nodes
in S
Exercise Prove Theorem C
Theorem is called the Marriage Theorem after the following interpre
tation The input no des are the women the exit no des the men or vice versa
and every edge means that the corresp onding pair are friendly so that a
marriage is not excluded A maximally sized matching is then a matching
up of the marriage partners that at least seen globally leads to the largest
amount of satisfaction Theorem says exactly when this is p ossible
Now we return to the construction of the graph G From the p ersp ective
of matchings G should have the prop erty that every subset of n m
input no des can b e matched to some n m exit no des The theorem says
that is the case if and only if for every subset S of input no des with jS j m
jS j jN S j
We construct G probabilistically as follows We reserve for each input
no de out going edges and corresp ondingly for each exit no de in coming
edges Altogether this is m m m edges that we may place
Topic
in G We will cho ose the edges randomly under the uniform distribution in
the following sense rst we cho ose for the rst edge leaving no de a partner
p osition on the exit side for example the th edge entering no de So for the
rst choice we have m p ossibilities For the next choice m etc We
will show that the probability is that the resulting graph has the desired
matching prop erty From this we can conclude that such a graph must exist
Exercise How large is the sample space that is how many ways are
there to connect the m input no des to the m output no des in the manner
just describ ed C
We want to show that the numb er of bad graphs graphs which do not
have the desired prop erty is strictly smaller than the numb er determined in
the preceding exercise A graph G is bad if and only if there is a k element
subset of S with the prop erty that jN S j k where k n m
First we note that this can only happ en if k
Exercise Why C
So assume k If G is bad then there is an integer k and sets S and
T such that jS j jT j k and N S T The calculations b elow work out
somewhat b etter however if we also count situations where N S T even
though this do es not imply that the graph G is bad For each k there are
m m
choices for S and choices for T For a xed pair S and T with
k k
k
m k many ways to cho ose the jS j jT j k there are at most k
edge relations for G such that N S T for each of the k edges leaving S
cho ose one of the k edges entering T then cho ose the remaining m k
b
edges arbitrarily We are using here the notation for falling p owers a
a
b
a a a a b So for example b a
b
Thus to prove the existence of a graph G and therefore also of G that
is not bad it is su cient to show that
m
X
m m
k
m k m k
k k
k
So it is su cient to show that
m
m m k
X
k k k
m
k
k
To prove this inequality we will use
m m m m
k k k k
Exercise Prove this formula C
Using this we see that it su ces to show that
Sup erconcentrators
m
k
X
k
m
k
k
for large n
k m
Exercise Let L denote the term and analyze the b ehavior
k
k k
of L L to conclude that L as a function of k with xed m is a convex
k k k
function Thus the largest summand L in the sum ab ove is either the rst
k
term k or the last term k m as in the following sketch
L
k
k
m
C
Finally we show that it is su cient to show that mL and mL are
m
b oth smaller than Note that m is an upp er b ound on the numb er of
summands in the sum
Exercise Carry this out to complete the pro of C
With that we have proven the existence of sup erconcentrators of linear
size and with a constant b ound on the degree of each no de ut
References
Aho Hop croft Ullman The Design and Analysis of Computer Algo
rithms Addison Wesley
N Pipp enger Sup erconcentrators SIAM Journal on Computing
F R K Chung Constructing random like graphs Proceedings of Sympo
sium in Applied Mathematics Vol American Mathematics So ciety
S N Bhatt On concentration and connection networks MIT LCS Tech
nical Rep ort
Topic
One can read ab out additional explicit constructions and application of su
p erconcentrators in the contributed chapter by N Pipp enger in
J van Leeuwen ed Handbook of Theoretical Computer Science Elsevier
MIT Press
The Pebble Game
The Pebble game is a mo del for successive execution of a computation
with the use of an auxiliary storage devise The game can b e used to
study trade o e ects b etween the memory use and running time for a
particular computation We will show a lower b ound originally proved by
Paul Tarjan and Celoni which says that certain graphs based on
sup erconcentrators require many p ebbles
The Pebble game is a one player game on a xed directed acyclic graph In
the course of the game p ebbles are placed on or removed from no des in the
graph according to the following rules
A p ebble may b e placed on an input no de a no de with no predecessors
at any time
If all predecessors of a no de u are marked with p ebbles then a p ebble
may b e placed on no de u
A p ebble may b e removed from a no de at any time
Note that rule subsumes rule but it is nevertheless useful to distinguish
the two cases
A move in this game consists of the placing or removing of one of the
p ebbles in accordance with one of the three rules The goal of the game
is to place a p ebble on some previously distinguished no de v usually an
output no de while minimizing the numb er of p ebbles used by which we
mean minimizing the maximum numb er of p ebbles that at any p oint in the
game are simultaneously on the no des of the graph i e p ebbles that have
b een removed from the graph can b e reused
A strategy for the game is a sequence of legal moves that ends in p ebbling
the distinguished no de v
Topic
Example The following graph
I
AK
A
AK
A
A
A
n
can b e p ebbled by placing p ebbles on no des through in order without
removing any p ebbles This takes only moves but uses p ebbles Another
strategy is represented in the following table
place p ebble
on no de
remove p ebble
from no de
numb er of p ebbles
on the graph
time
This strategy requires moves but only p ebbles
The p ebble game is a mo del for the computation of some result v from
given input data Consider the following machine mo del
CPU
memory
register
In this mo del the use of a register either to retrieve a value from memory
or to store an intermediate result corresp onds to the placing of a p ebble
The Pebble Game
in the p ebble game Rule says that at any time a value may b e loaded
into a register from memory Rule says that whenever all of the required
intermediate results for some op eration are lo cated in registers then the
op eration can b e carried out and the result stored in another register Finally
Rule says that at any time a register may b e cleared
The graph in the game indicates the dep endency structure of the op
erations to b e p erformed Such graphs arise for example in the design of
compilers The use of as few p ebbles as p ossible corresp onds to the use of
as few registers as p ossible With the help of the p ebble game certain time
space trade o s can b e studied As in the example ab ove often one has the
situation that it is p ossible to p erform a p ebbling task in relatively few moves
in a short amount of time but at the cost of using a large numb er of p eb
bles large memory use On the other hand there may b e another strategy
that uses far fewer p ebbles but requires more time since some of the p ebbles
that are removed must later b e recalculated If a p ebbling task cannot b e
simultaneously solved with minimal space p ebbles and time moves then
we say that the task exhibits a time space trade o
We want to investigate how many p ebbles are in general required to p ebble
graphs with n no des We must restrict ourselves to families of graphs with
restricted in degree Otherwise the numb er of p ebbles will dep end directly
on the in degree of the graph
For example consider the family of pyramid graphs
P
d
P
J
J
d d d
P
J J J
J J J
d d d d d d
p p p
P
J J J J J J
J J J J J J
d d d d d d d d d d
P
J J J J J J J J J J
J J J J J J J J J J
c d c d d d d c d d d c d d d
P
k
The pyramid graph P has i k k k no des and k k
k
i
k edges
Exercise Show that it is p ossible to p ebble the pyramid P with k
k
p
n p ebbles with resp ect to the numb er of edges p ebbles Note that this is O
n in the graph C
Exercise Show that every p ebbling strategy for P k must use
k
p
at least k p ebbles Again this is n p ebbles expressed in terms of
the numb er of edges n in the graph C
Now we want to investigate how many p ebbles are required for an arbi
trary graph with restricted in degree We can restrict our attention to graphs
Topic
of in degree since every graph with n edges and b ounded in degree d
that can b e p ebbled with p n p ebbles can b e transformed into a graph with
at most n edges and in degree that can also b e p ebbled with p n p ebbles
Exercise Why C
We will need the following lemma
Lemma Every directed acyclic graph with n edges and in degree can
be partitioned into two subgraphs G and G so that G contains between n
and n edges and al l edges between the two graphs go from G to G
and none in the other direction
We wil l let A denote this set of edges
Sketch
G
edge set A
G
Exercise Prove Lemma C
Now we want to show that every graph with in degree can b e p ebbled
with O n log n p ebbles For this we will analyze the following recursive
p ebbling strategy
If the graph G is small enough fewer than n edges then p ebble the
distinguished no de directly Else continue according to or
If the distinguished no de v is in graph G then apply the recursive
pro cedure to graph G since p ebbling no des in G cannot b e useful
If the distinguished no de v is in G and A is smal l jAj n log n then
recursively p ebble every predecessor no de to A by applying the recursive
strategy in G Leave all of these p ebbles in place but remove all other
p ebbles in G that were used along the way This will allow us to p ebble
any input no des of G so now start a recursive strategy for p ebbling v
in G
If the distinguished no de v is in G and A is big i e jAj n log n
then start a recursive p ebbling strategy for v in G but every time this
strategy requires placing a p ebble on an input no de of G that has pre
decessors in G use the recursive strategy to p ebble these no des in G
rst then continue
For the various cases we get the following recursion relations where P n
is the maximal numb er of p ebbles required to p ebble a graph with n edges
The Pebble Game
P n O for n n
P n P n
P n n log n P n
P n P n n log n P n
We need a solution to this recursion Let s try P n cn log n for a suitably
large constant c
Exercise Con rm that cn log n is a solution for cases and C
Exercise Con rm that cn log n is a solution for case C
Exercise Con rm that cn log n is a solution for case
a
C Hint Use the equality
x a x x x a
With the aid of this result it is easy to demonstrate the following inclusion
relationship b etween two complexity classes
DTIME t n DSPACE t n log t n
Exercise Show that there must b e context sensitive languages that
cannot b e b e accepted in linear time
Hint The class of context sensitive languages is precisely NSPACE n C
Next we want to show that the O n log n b ound on the numb er of
p ebbles needed to p ebble graphs with n no des is optimal This means we
must construct a family of graphs G jI j such that for every
n n I
constant c and every n I at least c jG j log jG j p ebbles are required
n n
to p ebble the graph G Note this implies that the inclusion DTIME t n
n
DSPACE t n log t n cannot b e improved
The sup erconcentrators from Topic will prove very useful in obtain
ing this result The following lemma demonstrates an imp ortant prop erty of
sup erconcentrators with resp ect to p ebbling
Lemma If j pebbles are placed on any j nodes of an n supercon
centrator j n and A is a subset of at least j output nodes
then there are at least n j inputs that are connected to A along pebble free
paths
Exercise Prove Lemma C
n
For every n let C n and C n b e two copies of a sup erconcentrator
n
By the results in Topic there is a constant d with jC n j d We
i
construct the family of graphs fG j n g recursively b eginning with G
n
Topic
which is chosen to b e a sup erconcentrator G is de ned as
n
follows
A A
z z
k k k k
I I I I
C n
G n
G n
C n
I I
k k k k
z z
E E
n
G has inputs and outputs divided into the sets E E and A A
n
n
each of which has size Each input is connected directly to its corresp ond
ing output and is also routed through two copies of G surrounded by two
n
sup erconcentrators of the appropriate size which in a certain sense decou
ple the inputs and outputs The outputs of C n are identi ed with the
inputs of G n similarly for G n and G n and G n and C n In
his lecture at ICALP Pipp enger referred to these graphs as sup er dup er
concentrators
n
Exercise Show that the size of the graph G is asymptotically n
n
So it is su cient in what follows to show that there is some c such that at
n
least c p ebbles are required to p ebble G C
n
n
Exercise Show that a C n a sup erconcentrator and a G n
n
together still form a sup erconcentrator C
The clever part of the following pro of which as one exp ects will b e done
by induction is to formulate an induction hyp othesis that is strong enough
to b e useful for the inductive step It is not su cient to merely assume that
n
c p ebbles are required to p ebble G although this is what matters in the
n
end rather we must also pack into the inductive hyp othesis and therefore
The Pebble Game
also prove in our inductive step some additional prop erties of the p otential
p ebbling strategies on G
n
So what is this hyp othesis
n n
Theorem Let n In order to pebble at least
n outputs of G n in any order beginning with an initial con gu
n
ration in which at most n nodes are pebbled there must be an interval of
time during which at least n nodes remain pebbled and during which time
at least n inputs must be pebbled
We note the outputs can b e p ebbled in any order and that the p ebbles
are not required to remain on the output no des so that the output no des do
not have to b e p ebbled simultaneously
Proof The pro of is by induction on n
Exercise Verify the base case of the induction that is show that the
hyp othesis holds for G
Hint Use Lemma C
Inductive step Assume that the hyp othesis holds for G we must show
n
that it holds for G Consider an initial con guration on G with at
n n
most n n p ebbled no des and assume that in the course of
the moves t at least n n outputs are p ebbled We
will refer to this time p erio d as the time interval t We must show that
within the interval t there is a time interval during which there are always
at least n n p ebbles on the graph and during which at least
n n outputs are p ebbled
We distinguish four cases In case we assume that the p ebble strategy
pro ceeds somehow unevenly In these three cases we are able to carry out
our pro of without using the inductive hyp othesis which is only needed in
case
Case There is a time interval t t t during which there are always
n p ebbles on the graph and during which at least n inputs of G n
are p ebbled
Let t b e the last time b efore t at which not more than n p ebbles
were on the graph We apply Lemma at time t to the following two
subgraphs of G
n
Topic
C n C n
I I
k k k k
z z
E E
n
So there is a time t at which there are at least n n
p ebble free paths from E to the n inputs of G n outputs of C n
that are p ebbled during the interval t t The same b ound holds for p ebble
free paths starting from E So altogether there are n p ebble free paths
and therefore that many inputs that remain to b e p ebbled during the time
interval t t Similarly during this interval there must b e at least n
n p ebbles on the graph This establishes the claim for the interval
t t
Case There is a time interval t t t during which there are always
at least n p ebbles on the graph and during which at least n inputs
of G n are p ebbled
The pro of in this case is analogous to the one given for case but in this
case we apply Lemma to the following two subgraphs of G Note
n
n
that each of these graphs is a sup erconcentrator see Exercise so
the lemma applies
G n G n
C n C n
I I
k k k k
z z
E E
Case There is a time interval t t t during which there are always
n p ebbles on the graph and during which n outputs of G are
n
p ebbled
During t t we can assume without loss of generality that at least n
outputs of A are p ebbled Once again the claim can b e established by a pro of
that is analogous to the one used in case this time applying Lemma
The Pebble Game
n
to the two subgraphs of G depicted b elow each of which is again a
n
sup erconcentrator see Exercise
A A
z z
k k k k
I I I I
C n C n
G n G n
G n G n
C n C n
I I
k k k k
z z
E E
Case None of the cases is applicable
Since we are not in case there must b e a time t t such that fewer
than n outputs of G are p ebbled during the time interval t
n
and at time t fewer than n p ebbles are on the graph This means that
during the time interval t t at least n n n outputs of
G are p ebbled Without loss of generality we may assume that n of
n
these are in A By Lemma applied to C n at time t there are at
n
least n n p ebble free paths b etween these n outputs
of A and the outputs G n Thus during the time interval t t at least
n n outputs of G n must b e p ebbled
By the inductive hyp othesis there is a time interval t t t t during
which at least n inputs of G n outputs of G n are p ebbled and
during which there are always at least n p ebbles on graph G n
Since we are not in case there must b e a time t t t so that
within the interval t t fewer than n inputs of G n are p ebbled and
at time t fewer than n p ebbles are on the graph So during t t at
least n n n inputs of G n outputs of G n must b e
p ebbled b eginning from a con guration with at most n p ebbled no des
at time t
Topic
By the inductive hyp othesis there is a time interval t t t t dur
ing which at least n inputs of G n outputs of C n are p ebbled
and during which there are always at least n p ebbles on graph G n
Since we are not in case there must b e a time t t t such that
during the interval t t fewer than n inputs of G n are p ebbled and
at time t fewer than n p ebbles are on the graph So during t t at
least n n n inputs of G n outputs C n must b e
p ebbled b eginning from a con guration with at most n p ebbled no de at
t
n
By Lemma it follows that at time t there must b e at least
n n inputs of E and corresp ondingly at least n inputs
of E that are connected to these n outputs of C n by p ebble free
paths So this numb er of inputs must b e p ebbled during the time interval
t t Also during this interval there are at least n p ebbles on each of
G n and G n so there must b e at least n p ebbles on G This
n
demonstrates the claim for the interval t t ut
Exercise Show that it follows from Theorem that the O n log n
b ound is optimal C
References
For the upp er b ound on the numb er of p ebbles required see
J Hop croft W J Paul L Valiant On time versus space Journal of the
ACM
K Wagner G Wechsung Computational Complexity VEB Deutscher
Verlag der Wissenschaften
For the lower b ound see
W J Paul R E Tarjan J R Celoni Space b ounds for a game on graphs
Mathematical Systems Theory
Average Case Complexity
In Topic we noticed that the exp ected running time of an algorithm
dep ends up on the underlying distribution of the instances and may in
general b e di erent from the worst case running time Now we want to
lo ok more carefully at the notions of b eing easy or hard on average As it
turns out equating easy on average with p olynomially b ounded exp ected
running time has serious drawbacks But in L Levin prop osed an
alternate de nition and demonstrated its robustness thereby initiating the
study of average case complexity
Many NP hard problems are of practical imp ortance but since there is no
known p olynomial time algorithm for any of these problems other typ es of
solutions have b een considered approximation algorithms probabilistic al
gorithms algorithms that work e ciently on an imp ortant restricted set of
inputs etc Another such alternative is to require that the problem only b e
e ciently solvable on average rather than in the worst case which for some
NP complete problems may b e very rare Indeed reductions to NP complete
languages typically reduce instances of an NP problem to instances of the
NP complete problem that are not very typical The natural question of
course is which NP problems are easy on average and which are hard
But b efore we can hop e to answer these questions we need to say what
is meant by easy and hard on average Presumably this will involve three
comp onents a class of distributions that we allow the results of Topic
indicate that some restriction must b e made a de nition of easy on average
and a notion of reduction b etween distributional problems that can b e
used to de ne hardness on average We will restrict our attention to decision
problems although similar things can b e done for search problems as well
For any strings x and y over some ordered alphab et let x y denote
that x o ccurs b efore y in the standard lexicographical order and let x
b e the predecessor of x in this order We intro duce the following
de nitions for distributions and density functions over
De nition A probability distribution can be described
by giving either
a probability distribution function i e a function that
Topic
is non decreasing x y x y and
converges to lim x
x
or a probability density function i e a function such
that
X
x
x
For any distribution or distribution function the associated density
function is
if x
x
x x otherwise
and for any distribution or density function the associated distribution
function is
X
x y
y x
A distributional problem is a pair A where A is a decision problem
a language and is a distribution For technical reasons we will assume
that It seems natural that we should at least allow
distributions that can b e computed in p olynomial time This leads to the
following de nition
De nition P computable distribution A distribution function
is P computable if there is a deterministic algorithm running in polynomial
time that on input x outputs the binary expansion of x
Note that if a distribution function is P computable then is also
P computable but that the converse may not b e true since to compute a
distribution from a density function requires an exp onentially large sum
Exercise Show that if every P computable density function induces
a P computable distribution function then P NP
Hint It easy to check if a p olynomial time nondeterministic machine accepts
along a xed path but probably hard to tell if there exists a path along
which it accepts Design a distributional problem where and re ect this
di erence C
Nevertheless it is usually more convenient to express things in terms of the
density function and we will write instead of from now on Note however
that for P computability we always mean P computability of not of
If is P computable then j x j is p olynomially b ounded This implies
that it is not p ossible to place to o much weight on the short strings i e
there is a p olynomial p such that for large n the combined weight of all
strings of length n is at least p n Of course the de nition of P computable
distribution also rules out the universal probability distribution of Topic
Later we will discuss a p otentially larger class of distributions as well
We are particularly interested in distributional versions of NP problems
Average Case Complexity
De nition DistNP A distributional problem A belongs to the
class DistNP if A NP and is a P computable distribution Another nota
tion for this class is NP P computable
Typically a distribution for such a DistNP problem is de ned in terms of
a density function which is usually designed to mo del a random exp eriment
of the following general typ e First pick a size length of string an integer
numb er of no des in graph etc for the instance then randomly select an
instance of the given size according to some probabilistic exp eriment Unless
we sp ecify some other distribution we will assume that the selection of a
size is done in such a way that the probability of selecting size m is roughly
P
m prop ortional to m This is an easy and natural choice since
m
P
m converges Of course technically we must scale the diverges and
m
P
m to get a probability distribution probability by c where c
m
but often in the literature this scaling is omitted and one simply works with
probability distributions that converge to something other than
Exercise An alternative to scaling m m is to mo dify slightly
X X
so that the sum is Show that C
m m
m
m m
For ease of notation we will adopt the convention of writing m for this
density function instead of m
Examples Each of the following DistNP problems is de ned by giving the
form of an instance the question and a description of the distribution in
terms of a random exp eriment For the rst example we include an expression
for the density function as well
D HALT
Instance A Turing machine enco ding M a string x and an integer k
written in unary
Question Do es M x halt within k steps
Distribution Randomly pick sizes m n and k Randomly cho ose a string
M of length m and a string x of length n So the probability density
function is
k
M x
jM j jxj
k
jM j jxj
D COL
Instance A graph G
Question Is there a coloring of the graph G A coloring is an as
signment of one of three colors to each of the no des in the graph in such
a way that no pair of adjacent no des is assigned the same color
Distribution Randomly pick a size n Randomly cho ose a graph with
vertices n by selecting each p ossible edge indep endently with
probability
Topic
D HAM
Instance A graph G
Question Is there a Hamiltonian circuit in G
Distribution Randomly pick a size n Randomly cho ose a graph with
vertices n by selecting each p ossible edge indep endently with
probability
Thus far the de nitions have all b een completely natural Unfortunately
the naive de nition for easy on average has serious drawbacks The obvious
de nition would b e the following A is easy on average if there is a deter
ministic algorithm for deciding memb ership in A with p olynomially b ounded
exp ected running time i e the running time t x should satisfy
A
X
k
x t x cn
n A
jxj n
for some integer constants c and k Here x denotes the conditional prob
n
ability
x jxj n if jxj n
n
x x j x
n
otherwise
Unfortunately this de nition has serious drawbacks
Exercise Find a function f R such that f has a p olynomially
b ounded exp ected value but f do es not
Hint Let f b e exp onential linear exp onent on a small set of inputs of each
length Then f will b e exp onential with a quadratic exp onent on those same
inputs C
This implies that a theory of average case based on p olynomially b ounded ex
p ected running time would have several signi cant drawbacks Such a de ni
tion would b e machine dep endent since there is a p olynomial loss of e ciency
when converting b etween some mo dels of p olynomial time computation that
we normally regard as equivalent Thus whether or not a distributional prob
lem is easy on average might dep end on the underlying mo del of computation
Worse still even for a xed mo del of computation the class of problems that
are easy on average would not b e closed under op erations like comp osition
This would mean that an algorithm that is easy on average could not neces
sarily b e used as a sub routine in another algorithm that is easy on average
neglecting the time sp ent on the sub routine calls Using the naive de ni
tion of easy on average this sort of comp osition of two algorithms that are
easy on average might result in an algorithm that is not easy on average
Levin s solution to these problems is the following de nition
r
r r
Throughout this topic we will use the notation f or f x to denote f x
and not the r fold comp osition of f with itself
Average Case Complexity
De nition Polynomial on average A function f is p olynomial
on average if there is a constant such that
X
f x
x
jxj
x
that is the function f x is linear on average
De nition AP A distributional problem A is in the class AP
average polynomial time if there is an algorithm for A with running time
that is polynomial on average
Exercise Show that if f and g are p olynomial on average then so
k
are max f g f f g and f g C
Exercise Show that if the exp ected value of f is p olynomially b ounded
with resp ect to some distribution then f is p olynomial on average
Hint Use the fact that a a for all C
Thus AP has some of the nice closure prop erties that we exp ect of a robust
complexity class
Even though the colorability problem is NP complete its distributional
version at least with the distribution we have presented is in AP
Theorem The distributional version of the colorability problem
D COL is in AP ut
Exercise Prove Theorem
Hint If a graph G has a copy of K the complete graph with four vertices
and six edges as a subgraph then G cannot b e colored C
Now we want to know if there are any DistNP problems that are hard on
average For this we want to de ne a reducibility b etween two distribu
r
tional problems with the following prop erties
Closure If A B and B is in AP then A is as well
r
Transitivity is transitive
r
If H is hard for DistNP under a reducibility with these prop erties then
H AP AP DistNP
It has b een shown that if AP DistNP then NEXP EXP which is consid
ered unlikely
Once again the simplest thing to try do es not have the desired prop erties
Polynomial time many one reductions may map very likely instances of A
Topic
on which an AP algorithm must run quickly to very unlikely instances of
B on which an AP algorithm may run very slowly this would violate
the closure prop erty Thus we must design a reducibility that resp ects the
distributions in some way
De nition Domination Let and be distributions Then
read is dominated by if there is a polynomial p such that x
p jxj x
P
Let A and B be distributional problems and let f A B Then
m
is dominated by with resp ect to f written if there is a distribu
f
P
x tion for A such that and y range f
f x y
De nition Polynomial time reduction A is polynomial
P
time many one reducible to B if A B via a function f such that
m
P
B ut We wil l denote this by f A
f
m
P P
B and is transitive and if A Lemma The reducibility
m m
B AP then A AP
P
is transitive on distributional problems C Exercise Prove that
m
P
Exercise Show that AP is closed under C
m
Now we can give an example of a distributional version of an NP complete
problem that is DistNP hard The problem we will cho ose is D HALT the
distributional halting problem Although it is relatively straightforward to
show that the usual b ounded halting problem is complete for NP that pro of
do es not carry over directly to the distributional setting In fact it is not
immediately clear that there are any DistNP complete problems Certainly
there is a many one reduction from any NP decision problem to the halting
problem but this reduction may not resp ect the distributions In particular
we need to show that D HALT which has a xed distribution is hard for all
NP problems with any P computable distribution To achieve this we will
need to mo dify the reduction on instances of the decision problem for the
sake of resp ecting the distributions
P
Theorem D HALT is complete for DistNP
m
Proof For any NP problem A there is a nondeterministic machine N that
A
decides memb ership in A in time p n for some p olynomial p We have already
noted however that the reduction
p jxj
x N x
A
may not resp ect the distributions involved Instead we will use a reduction
of the form
Average Case Complexity
O
jxj
x N co de x
A
where N is a machine that dep ends only on N and but not on x and
A A
co de x is an enco ding of x
The success of the pro of dep ends up on nding an enco ding satisfying the
following lemma
Lemma Let be a P computable distribution Then there is an en
coding function co de satisfying the fol lowing properties
Compression For every x
jco de x j min jxj log
x
E ciency The function co de can be computed in polynomial time
Uniqueness The function co de is one to one i e if co de x co de y
then x y ut
Exercise Prove Lemma
jxj
Hint Distinguish two cases The case when x is easy In the other
case recall that x x x and de ne co de x based up on
a comparison of the binary representations of x and x C
The reduction is now de ned by
q jxj
f x N co de x
A
where q jxj is a p olynomial that b ounds the sum jxj the time required to
compute co de x and the time required to run N on input x and N is
A A
a nondeterministic machine implementing the following algorithm
INPUT y
GUESS x such that co de x y
IF N x accepts THEN ACCEPT
A
ELSE REJECT
END
This nondeterministic algorithm runs in time q jxj on input y co de x
so f is a p olynomial time computable many one reduction on instances All
that remains is to show that where is the distribution of D HALT
f
Exercise Show that C
f
Distributional versions of several other NP complete problems have also
P
b een shown to b e complete for DistNP including a tiling problem the
m
Post corresp ondence problem word problems for Thue systems and nitely
presented groups and satis ability
Topic
So what distinguishes distributional versions of NP complete problems
that are hard for DistNP from those that are not Y Gurevich provided a par
tial answer to this question when he observed that under a plausible complex
ity theoretic assumption in many cases the distribution of a distributional
problem already determines that the problem cannot b e DistNP complete
regardless of the question This prop erty of distributions that makes distri
butional problems unlikely to b e DistNP complete is atness
De nition Flat distribution A probability density function is
jxj
at if there is an such that for al l x x A dis
tribution function is at if its associated density function is at
So a distribution is at if no long strings are weighted to o heavily none
of them juts out from the others
Exercise Show that the distribution of D HAM is at C
Exercise Show that the distribution of D HALT is not at C
P
Theorem No DistNP problem with a at distribution is complete
m
for DistNP unless NEXP EXP
P
Proof Supp ose H DistNP is at and H is complete for
m
DistNP Since H NP H EXP Let A b e an arbitrary decision problem in
p n
NEXP Then there is a p olynomial p such that A NTIME We want
to de ne a distributional problem A DistNP that is related to A For
p jxj
jxj
this let x x de ne A fx j x Ag and let b e de ned
by
jxj
jxj if z x for some x
z
otherwise
Exercise Show that A NP C
Since A NP A DistNP and since H is complete for DistNP
P
there is a reduction f A H This implies that there is a distri
m
bution and a p olynomial q such that
O q jxj
f x can b e computed in time
x A if and only if f x H
x q jx j x and
X
z f x range f
f z f x
Now we put these pieces together First for notational ease we will as
sume that range f The argument b elow can b e easily mo di ed if
range f c for some constant c
Average Case Complexity
X
z q jx j f x x q jx j x q jx j
f z f x
so
jxj
jxj x
r jxj
f x
jxj p jxj
q jx j q jx j
jxj q
for some p olynomial r But since is at there is an such that
jf x j
f x From this it follows that
jf x j log f x r jxj
so jf x j is p olynomial in jxj Thus A EXP x A if and only if f x H
O q jxj
and we can compute f x in time and then determine whether
poly jf x j poly jxj
f x H in time
The theory of average case complexity has b een extended in several ways
from the results presented here One generalization of the theory consid
ers other less restrictive reductions b etween distributional problems It is
fairly straightforward for example to generalize p olynomial time many one
reducibility to p olynomial on average many one reducibility All the results
presented here remain true in that setting as well Similarly one can con
sider other typ es of reducibilities such as truth table or Turing reducibility
Finally one can de ne a notion of randomized reduction Roughly a random
ized reduction from A to B is given by a probabilistic oracle Turing
machine that on input x with oracle B and random bits r runs in time p oly
nomial on average where is the distribution on the random bits r
correctly decides x A with probability at least and asks queries in a
manner that resp ects the distributions so that the reduction may not query
oracle instances of low weight to o often The random reduction is many one
or truth table if the Turing machine b ehaves in one of these more restrictive
ways There are randomized many one complete problems for DistNP that
have at distributions
A second generalization considers a larger class of distributions
De nition P samplable distribution A distribution is cal led
P samplable if there is a randomized algorithm that takes no input but ips
coins and eventual ly outputs a string x if it halts in such a way that
the probability that the algorithm outputs x is x and
the running time is polynomial ly bounded in jxj
Exercise Show that all P computable distributions are P samplable C
Topic
If certain cryptographic one way functions exist p olynomial time computable
functions that are hard to invert on most instances then there are P
samplable distributions that are not P computable
It can b e shown that a version of the b ounded halting problem with an
other so called universal distribution not to b e confused with the universal
P
probability distribution of Topic is complete for NP P samplable
m
Roughly the universal distribution in this case amounts to randomly select
ing a P samplable distribution from an enumeration and then sampling ac
cording to that distribution Furthermore in the setting of NP search prob
lems where a solution must not only give the answer to an NP predicate
but also provide a witness and reductions must also preserve witnesses
Impagliazzo and Levin have shown that every distributional problem that
is complete for NP search P computable under randomized many one re
ductions is also complete for NP search P samplable under randomized
many one reductions This is a pleasing result since most natural problems
have P computable distributions and it says that in some sense there is no
b etter way to nd hard instances of a problem like SAT than to pick an
instance uniformly at random A similar result holds for decision problems
under randomized truth table reductions
References
Below are a numb er of articles that were used in preparing this topic Those
interested in learning more ab out average case complexity are encouraged to
consult the bibliographies of these pap ers for additional references
O Goldreich Notes on Levin s theory of average case complexity avail
able at http theory lcs mit edu oded surveys html
Y Gurevich Average case completeness Journal of Computer and Sys
tem Sciences
R Impagliazzo A p ersonal view of average case complexity Proceedings
of the th annual Conference on Structure in Complexity Theory IEEE
R Impagliazzo L Levin No b etter ways to generate hard NP instances
than picking uniformly at random Proceedings of the st Symposium
on Foundations of Computer Science IEEE
L Levin Average case complete problems SIAM Journal on Computing
J Wang Average case computational complexity theory Chapter in
Complexity Theory Retrospective II Springer
Quantum Search Algorithms
Widespread interest in quantum computation was sparked by an algo
rithm of P Shor for factoring integers on a quantum computer We inves
tigate here a more recent quantum algorithm of L Grover for searching a
database This algorithm demonstrates a proven sp eed up against the b est
p ossible classical algorithm for the same task
Up to this p oint all of the computing devices and all of the complexity classes
we have considered have b een based on classical physics In the early s
however P Benio prop osed a mo del of computation based on quantum me
chanics The mo del of quantum computation was subsequently formalized
primarily due to the work of D Deutsch Widespread interest in quantum
computation was sparked in when P Shor gave a feasible quantum
algorithm for factoring integers a problem that has no known feasible classi
cal algorithm a fact which underlies many encryption schemes Thus Shor s
algorithm showed that quantum machines are able at least in theory
to p erform an interesting and imp ortant task that classical probabilistic
machines p erhaps cannot There remain of course di culties in imple
menting such an algorithm on a physical device
We want to consider here L Grover s quantum mechanical algorithm for
searching a database This algorithm has the advantages of b eing somewhat
simpler to describ e and analyze than Shor s algorithm since it do es not make
use of as much numb er theory and of demonstrating something that can
provably b e done more e ciently on a quantum machine than on a classical
machine
The database search problem in this context is de ned as follows Supp ose
n
we have an unsorted data set with N entries exactly one of which
matches our selection criteria For example the data set could b e a phone
b o ok from which we want to retrieve the name of a p erson with a particular
phone numb er Note that the usual sorting of a phone b o ok do es not assist
us in this query We will assume that we have a means either by using an
oracle or an easily computed function of telling when we have lo cated the
prop er element in the data set but we have no information ab out where in
the set this element is lo cated
Classically of course there is nothing we can do but lo ok at each entry
in the data set until we happ en to nd the correct one
Topic
Exercise What is the exp ected numb er of queries to the data set re
quired by such an algorithm C
Even randomization do esn t help us here
Exercise If a randomized algorithm queries m elements from the data
set what is the probability that it will nd the target element C
Thus a classical probabilistic algorithm must query the data set in at least
n
N steps to have a probability of success that is greater than
p
Grover s quantum algorithm reduces this numb er to O N demonstrating
a quadratic sp eed up versus any classical algorithm
Before de ning what we mean by quantum computation let s return for a
moment to classical probabilistic computation A classical probabilistic com
putation gives rise to a computation tree Each no de in the tree is lab eled with
a con guration instantaneous description of tap e contents head lo cation s
and internal state of the Turing machine Edges in the tree are lab eled with
real numb ers in the interval which corresp ond to the probability of a
transition from the parent con guration to the child con guration Each level
of the tree represents one time step so the depth of the tree represents the
running time of the machine
Probabilities can b e assigned to a no de by multiplying the probabilities
along the path from the ro ot to that no de The probability of the computation
b eing in con guration c at time t is the sum of the probabilities assigned to
each no de at level t that has b een assigned con guration c
In order for such a tree to represent a probabilistic computation it must
meet two constraints
Locality The probability assigned to the edge from one no de to another
must corresp ond to the action of one step of a probabilistic Turing machine
so in particular
the probability is non zero only if the underlying nondeterministic Turing
machine could actually make such a transition thus for example the
only tap e cells that can change are the ones that were under a head in
the parent con guration and
the probability dep ends only on the part of the con guration that deter
mines the action of the machine and not on the rest of the con guration
or its lo cation in the tree
Classical probability The sum of all probabilities on any level must b e
Exercise Show that if the sum of the probabilities on the edges leaving
any no de equals then the classical probability constraint is satis ed C
For the purp oses of complexity considerations it is usually su cient to con
g sider probabilities from the set f
Quantum Search Algorithms
The computation tree can b e represented by a k k matrix M where k is
the numb er of p ossible con gurations and M the entry at lo cation a b is
ab
the probability of going from con guration a to con guration b in one step
s
M then represents the transitions that o ccur in s steps The probability that
a machine accepts on input x after s steps is
X
Pr con guration c at step s j con guration c at step
c
acc
where is the set of all accepting con gurations and c is the initial
acc
con guration corresp onding to an input x
In a quantum computation instead of assigning real valued probabilities
to the edges we assign complex valued probability amplitudes with norm
at most The amplitude of a no de in the computation tree is again the
pro duct of the amplitudes along the path to that no de and the amplitude
asso ciated with b eing in con guration c at step t is the sum of the amplitudes
of all no des at level t lab eled with c Probability amplitudes corresp ond to
probabilities in the following way The probability is the squared absolute
value of the amplitude
As b efore our lab eling of the tree must satisfy two constraints
Locality This condition is the same as b efore the lab eling of the tree must
corresp ond to the action of a Turing Machine
Quantum probability If one represents the quantum computation by a ma
trix M then M must b e unitary which means that its inverse is equal to
its conjugate transp ose
This quantum probability condition implies that the sum of the proba
P
bilities on any level will b e j j This time however it is not
c
su cient to merely require that the sum of the squares of the amplitudes
leaving any no de b e This is due to the e ects of interference canceling
among the con gurations
Exercise Give an example of a lab eled tree where the sum of the
probabilities leaving each no de is but the sum of the probabilities at some
level is not
Hint Two levels su ce C
Exercise Show that if all of the entries in M are real numb ers in
then M is unitary if and only if it is orthonormal that is the dot pro duct of
any two distinct rows or columns is always and the dot pro duct of a row
or column with itself is
Hint Some notation and de nitions are p erhaps in order here If M is a
matrix we denote the entry in the ith row and j th column by M The
ij
t t
M In the conjugate transp ose of M is denoted by M and de ned by M
j i ij
Topic
transp ose M M is the complex conjugate of M So if the entries are
j i
ij
all real then the transp ose and conjugate transp ose are the same matrix
The inverse of a matrix M is denoted by M and is a matrix such that
M M M M I where I is the identity matrix which consists of
s down the ma jor diagonal and s elsewhere To have an inverse a matrix
must b e square but not all square matrices have inverses C
Note that a unitary matrix M has an inverse which means that unlike
classical computation quantum computation is necessarily reversible
For many purp oses it is not necessary to use a particularly rich set of
complex numb ers usually rationals and square ro ots of rationals are more
than su cient In fact for the de nitions of BQP and NQP de ned b elow the
set f g is su cient for the lo cal transformation amplitudes
Although the de nitions and mental images for probabilistic computation
and quantum computation are in many ways similar there are imp ortant and
p owerful di erences Unlike a probabilistic machine which we think of as b e
ing in one of a set of con gurations with certain probabilities we consider
a quantum machine to b e in a superposition of con gurations This is some
thing like saying that a quantum machine is simultaneously and to varying
degrees in several con gurations at once Up on observation of a quantum
mechanical device the sup erp osition collapses to a single con guration The
probability with which each con guration is observed is determined by its
amplitude in the sup erp osition
We denote a con guration also called a quantum state by jci and a
sup erp osition of such states by
X
j i jci
c
c
where is the amplitude of jci Algebraically the states jci for all con gu
c
rations c form an orthonormal basis in a Hilb ert space Since the basis states
jci are mutually orthonormal the amplitude of jci in a sup erp osition j i
c
is the inner pro duct of jci with j i denoted by hc j i
If is the amplitude of jci in a sup erp osition j i then j j is the probabil
ity of observing c when the machine is in sup erp osition j i The probability of
accepting is de ned as for the probabilistic computation it is the probability
of observing an accepting state at a certain time t
The imp ortant added wrinkle which provides the added p ower in quan
tum computation is the fact that probability amplitudes can cancel see Ex
ercise For example there may b e two paths in the computation tree
that b oth lead to the same con guration c but one may have probability
amplitude and the other So for example after some numb er of steps
the sup erp osition may have the form
X
jci jci jc i
c
c c
Quantum Search Algorithms
In this case the probability of b eing in state jci is The heart of most
quantum algorithms consists in using sup erp osition to try out a numb er of
p ossibilities and using cancellation to reduce the probability of bad p ossi
bilities while increasing the probability of go o d p ossibilities
The classes NQP and BQP can b e de ned analogously to the classes NP
and BPP by replacing the probabilistic machine with a quantum machine
De nition A language L is in NQP if and only if there is a quantum
Turing machine Q and a polynomial p such that
x L P r Q accepts x in p jxj steps
A language L is in BQP if and only if there is a quantum Turing machine Q
and a polynomial p such that
x L P r Q accepts x in p jxj steps
x L P r Q accepts x in p jxj steps
It is not known if BQP is equal to some classical complexity class It
contains BPP but results of Fortnow and Rogers have led them to con
jecture that BQP contains no interesting complexity classes outside BPP
Even if this conjecture holds BQP typ e algorithms will remain interesting
since they can provide a signi cant increase in sp eed for some problems like
searching as we will see shortly and at least have the p otential to solve
interesting problems that are probably neither NP complete nor in P like
Graph Isomorphism see Topic
NQP on the other hand has b een exactly characterized in terms of clas
sical counting classes NQP coC P where C P is the class of languages
where acceptance is de ned by a nondeterministic machine that has an equal
numb er of accepting and rejecting computations
So how do es one use quantum computation to sp eed up a search First
we give a formal de nition to the search problem Let f i b e a function that
tells if the ith item in the data set is the item we seek i e there is exactly
one target t N such that f t otherwise f i We would like
to design a quantum algorithm that after a certain numb er of steps with
high probability greater than in any case will b e in a con guration that
identi es t That is imagine that for each i N there is a con guration
n
jc i that represents that the item sought is item numb er i of N Then we
i
want the probability of jc i to b e high at the time we observe our computation
t
For concreteness let c b e the con guration that has i written as a string of
i
length n on its work tap e and is otherwise uninteresting the head is at the
left of each tap e all other tap es are empty the internal state is some xed
state etc
Topic
The quantum algorithm has two main comp onents The rst is used to
generate all p ossible keys in sup erp osition and the second is used to separate
the value t for which f t from the others and amplify its amplitude
We b egin with the rst comp onent Supp ose for a moment that we were
designing a probabilistic machine instead of a quantum machine Then we
might cho ose to ip a coin n times and use the outcome of the coin to
set the bits of the string on our work tap e More precisely for each bit in
succession with probability we leave the bit as it is initially a and
with probability we change it to a Now we want to do a similar thing
with our quantum machine If we fo cus on a single bit then we are lo oking
for a unitary transformation
j i aj i bj i
j i cj i dj i
a b
i e a unitary matrix M such that jaj jbj
c d
Exercise Find such a matrix
Hint It su ces to use real values C
Let W b e the matrix transformation that is the comp osition of M ap
plied to each of the n bits in succession As a matrix W is roughly blo ck
diagonal with copies of M lo cated at appropriate p ositions near the diagonal
s elsewhere on the diagonal and s lling the rest of the matrix
Exercise What sup erp osition results from applying W to jc i C
Exercise What sup erp osition results from applying W to jc i C
i
The transformation W is known as the Walsh Hadamard transformation
and it or a related op eration known as the Fourier transformation is a
comp onent of many quantum algorithms
Now that we have in some sense generated all the p ossible keys we need
to do something to distinguish the target t This is the second comp onent of
the algorithm For this we will simply ip the sign of the amplitude of the
target t Note that any diagonal matrix with s and s along the diagonal
is unitary so that this kind of sign ipping on any set of states can b e
done provided we know where to place the s and s For this we need
to know f i which we can assume is determined by an oracle call or some
simple deterministic algorithm It is known that deterministic computation
can b e done reversibly and this fact has b een used to show that deterministic
computation can b e simulated on a quantum machine Let F denote the
transformation that determines f i and ips the sign of the amplitude if
f i
Quantum Search Algorithms
Finally these two comp onents are combined to form what Grover refers
to as the di usion transformation D The purp ose of D is to amplify the am
plitude of the state corresp onding to the target Let F b e the transformation
that ips the sign of all the states except jc i that is
if i j
if i j
F
ij
if i j
The di usion transformation is D W FW
Exercise Compute D for arbitrary i and j C
ij
The transformations F and D form the heart of the algorithm and W is
used once at the b eginning as a sort of initialization With this background
laid the description of the algorithm is relatively short
Initialize by applying W to the initial state jc i
p
N times We will determine the constant in Rep eat the following O
the O notation as part of the analysis It will turn out to b e imp ortant
to do this the correct numb er of times
a Apply F Flip the amplitude on the target
b Apply D Di usion
Observe the state of the quantum machine The probability of jc i will
t
b e at least
Now we need to show that the algorithm b ehaves as claimed In particu
lar we need to understand what the di usion transformation is doing The
di usion transformation can b e interpreted as an inversion about the average
in the following sense Supp ose the quantum machine is in the sup erp osition
X
jc i j i
i i
i N
P
b e the average amplitude over all the states Then the result Let
i
i
N
of applying D to j i is to increase or decrease each amplitude so that after
the op eration it is as much ab ove or b elow the average as it was b elow or
ab ove the average prior to the op eration That is
X X
jc i jc i
i i i i
where
i i i
Lemma The di usion transformation D is unitary Furthermore it
performs an inversion about the average as described above
Proof For the pro of it is useful to have another representation for D Let P
b e the pro jection matrix P N Then D I P see Exercise ij
Topic
Exercise Show that D is unitary
Hint First show that P P C
Exercise Show that D p erforms an inversion ab out the average
Hint Let x b e a vector what do es P x compute C
ut
Now we intro duce some notation for the sup erp ositions that o ccur in this
computation Let
X
j k l i k jc i l jc i
t i
i t
We want to express the e ects of Grover s algorithm using this notation By
Exercise step the initialization of the algorithm amounts to
p p
i W jc i j
N N
And by Lemma each iteration of step is
F D
N
N N
k l l k i j k l i j k l i j
N N N N
p
N since the average Note that after one iteration k is still very nearly
p
amplitude prior to applying D was very nearly N but l is approxi
p p
mately N The hop e is that each iteration increases k by N so
p p
p p
N iterations k N that after O
N
Let j k l i denote the sup erp osition after j iterations In the
j j j
pap er where Grover originally presented his algorithm he proved that there
p
was a numb er of iterations j N such that k His pro of followed
j
from a sequence of lemmas ab out the transformation D culminating in the
following lemma
p p
and l then k k k and Lemma If k
j j j j
N
l
j
The problem with this argument however is that Lemma only
provides a lower b ound on k Thus while we know there is an iteration
p
N after which k we don t know what m is In fact after ex m
m
p
actly N iterations the probability is less than that we will observe the
target From there it continues to decrease to a negligible probability b efore
it eventually increases again Thus it is imp ortant to know m explicitly
In order to know precisely when to make our observation we need a
tighter analysis This was provided in a pap er by Boyer H yer and Tapp
where explicit formulas are given for k and l Using standard techniques
j j
and some patience the recurrence for k and l can b e solved giving explicit
formulas for k and l
j j
Quantum Search Algorithms
Lemma Let k and l be de ned as above then
j j
k sin j
j
p
l cos j
j
N
where is chosen so that sin N
Exercise Prove Lemma
Hint Although it is a bit arduous to solve the recurrence relations to get the
explicit formulas of the lemma it is straightforward to prove by induction that
the formulas are indeed correct Readers who have forgotten the trigonometric
identities may need to refer to an undergraduate calculus text C
From this it follows that k when m i e when
m
m Of course this is probably not an integer but the probability
should b e almost if we p erform approximately this many iterations Since
for small sin the numb er of iterations needed is approximately
p
N
In fact after we have generalized the problem slightly we will prove the
following lemma
p
N
Lemma After b c iterations of Grover s algorithm the probability
p
N
of failure is less than N After b c iterations of Grover s algorithm the
probability of failure is at most
p
N
However if we iterate twice as many times ab out iterations the prob
ability of success is negligible
Before proving Lemma we want to generalize the algorithm to handle
more than one target Supp ose there is a set of targets T such that
t T
f i
t T
Grover s algorithm or at least his analysis dealt only with the case that
jT j
What happ ens if we apply the same algorithm when jT j De ne
X X
j k l i k jc i l jc i
i i
i T
i T
Then each iteration of Grover s algorithm has the e ect
Topic
N jT j N jT j jT j N jT j
k l l k i j k l i j
N N N N
Once again this recurrence can b e solved this time yielding
sin j
p
k
j
jT j
cos j
p
l
j
N jT j
where sin jT j N
The probability of success is greatest when l is smallest and l if
m
m but that may not b e an integer Let m b c instead
Note that jm m j So j m m j But by the
de nition of m m so j cos m j j sin j Thus the
probability of failure after m iterations is
N jT j l cos m sin jT j N
m
q
p
N
Since sin the algorithm requires jT j N m
jT j
p
N jT j iterations The exp ected running time until a target is found O
can b e improved slightly by running the algorithm for fewer iterations at a
cost of a lower probability of success and rep eating the entire algorithm if
unsuccessful This is b ecause the sine function is quite at near its extreme
values so that the last iterations do less to increase the probability than the
earlier ones
The case where jT j N is particularly interesting In this case sin
so and
t
l cos
N jT j
This means that the probability of success is after just one iteration It
should b e mentioned that this really involves two queries to the data set
one to determine if there is a phase shift sign ip and one as part of the
uncomputation which makes this reversible
Exercise What is the exp ected numb er of queries required by the
naive classical probabilistic algorithm when jT j N C
Exercise What is the worst case for a classical algorithm Imagine
that an adversary is determining the outcome of your coin tosses C
Thus the quantum algorithm uses only half of the numb er of queries
exp ected for a classical probabilistic algorithm and exp onentially fewer in
n than a classical algorithm uses in the worst case
Quantum Search Algorithms
There have b een other generalizations of Grover s algorithm a few of
which we summarize brie y here The rst two generalizations deal with the
case when jT j is unknown Supp ose rst that we want to lo cate one of the
targets t T but we don t know how many targets there are If we use
p
N iterations we will almost certainly nd the target t if it is unique
But if jT j then the probability of success after this many iterations is
nearly Boyer H yer and Tapp give an algorithm with exp ected running
p
N for nding a target in this case as well The main idea is to time O
randomly select m the numb er of iterations from some set M where initially
M f g and then to run Grover s algorithm for that many iterations If
this is not successful it is done again but with a larger set M
Brassard H yer and Tapp also consider the problem of approximating
jT j rather than nding an element of T In their recent pap er they give an
algorithm that demonstrates a trade o b etween the accuracy of the approx
imation and the running time of the algorithm Furthermore they show that
their algorithm is in some sense optimal This pap er also considers the prob
lem of amplitude ampli cation the key ingredient in Grover s algorithm in
a more general setting This allows them to demonstrate a sp eed up b etween
quantum and classical search algorithms even in the case where b etter than
brute force metho ds exist classically
All of the algorithms ab ove b egin from a sup erp osition with only one non
zero amplitude Another way to generalize the problem is to consider other
p ossible initial sup erp ositions such as might o ccur if quantum searching were
used as a subroutine in a more complicated quantum algorithm Results of
Biham Biham Biron Grassl and Lidar show that the optimal time for
observation can b e determined in this situation as well and that it dep ends
T only on the means and variances of the initial amplitudes on T and
Finally we make two comments on Grover s algorithm One might ask if
there are even b etter quantum search algorithms which require say O log N
queries to the database But this is not p ossible Grover s algorithm is op
timal in the sense that any quantum algorithm for searching must make at
p
least N queries to the database Also the reader may have noticed
that we have assumed throughout that N is a p ower of This assumption
simpli es the arguments and notation somewhat but it is not an essential
restriction It can b e avoided by replacing the Walsh Hadamard transforma
tion by any transformation in a large class of transformations of which the
Walsh Hadamard transformation is the simplest when N is a p ower of
References
Foundational work in the theory of quantum computation can b e found in
Topic
P Benio The computer as a physical system A microscopic quantum
mechanical Hamiltonian mo del of computers as represented by Turing
machines Journal of Statistical Physics
E Bernstein U Vazirani Quantum Complexity Theory Proceedings of
the th Annual Symposium on Theory of Computing ACM
D Deutsch Quantum Theory the Church Turing principle and the uni
versal quantum computer Proceedings of the Royal Society London Series
A
R P Feynman Simulating physics with computers International Journal
of Theoretical Physics
Prior to the results of Shor and Grover there were already indications that
quantum computation is more p owerful than classical computation on some
what contrived problems
D Deutsch R Jozsa Rapid solutions of problems by quantum compu
tation Proceedings of the Royal Society of London
D Simon On the p ower of Quantum Computation SIAM Journal on
Computing
Many pap ers on Grover s algorithm and its generalizations as well as a num
b er of other topics related to quantum computation are available as LANL
preprints at http xxx lanl gov The site is easily searchable so we include
here only those pap ers that were most instrumental in preparing this topic
E Biham O Biham D Biron M Grassl D Lidar Exact solution of
Grover s quantum search algorithm for arbitrary initial amplitude distri
bution LANL preprint quant ph
M Boyer G Brassard P H yer A Tapp Tight b ounds on quantum
searching Proceedings of th Workshop on Physics and Computation
Final version to app ear in Fortschritte der Physik Also
LANL preprint quant ph
G Brassard P H yer A Tapp Quantum Counting LANL preprint
quant ph
L Grover A fast quantum mechanical algorithm for database search
Proceedings of the th Annual Symposium on Theory of Computing
ACM Also LANL preprint quant ph
L Grover A framework for quantum mechanical algorithms Proceedings
of the th Annual Symposium on Theory of Computing ACM
Also LANL preprint quant ph
Quantum Search Algorithms
Shor s algorithm originally app eared in
P Shor Algorithms for quantum computation Discrete log and factoring
Proceedings of the th Symposium on Foundations of Computer Science
IEEE
but see also
A Ekert R Jozsa Quantum computation and Shor s factoring algo
rithm Reviews of Modern Physics
P Shor Polynomial time algorithms for prime factorization and discrete
logarithms on a quantum computer SIAM Journal on Computing
Also LANL preprint quant ph
For information on the relationship b etween quantum complexity classes and
classical complexity classes see
L Adleman J Demarrais M D A Huang Quantum computability
SIAM Journal on Computing
C Bennett E Bernstein G Brassard U Vazirani Strengths and weak
nesses of quantum computing SIAM Journal on Computing
S Fenner F Green S Homer R Pruim Quantum NP is hard for PH
to app ear in Proceedings of the th Italian Conference on Theoretical
Computer Science
L Fortnow J Rogers Complexity limitations on quantum computation
Proceedings of the th IEEE Conference on Computational Complexity
IEEE
J Watrous Relationships b etween quantum and classical space b ounded
complexity classes Proceedings of the th IEEE Conference on Compu
tational Complexity IEEE
For a survey of results in quantum computational complexity from a some
what di erent p ersp ective see
A Berthiaume Quantum Computation chapter in Complexity Retro
spective II Springer
Topic
Solutions
For every decidable language A and any arbitrary language B A
T
B
The halting problem H f x y j y W g is m and therefore also
x
T complete for the class of computably enumerable languages For if A
is computably enumerable then there is an i such that A W The map
i
y i y is an m reduction from A to H
Now let A b e a language which is T equivalent to the halting problem
and supp ose there is a language B which is computably enumerable and T
incomparable to A Since B is computably enumerable B H and since
T
H A it follows that B A Contradiction
T T
If there are two Turing incomparable computably enumerable lan
guages then neither one can b e computable exercise But by the claim
one would have to b e computable if the incomparability could b e demon
strated e ectively
It is su cient to show that if A is computably enumerable and h is
computable then h A is computably enumerable
The following pro cedure enumerates h A
FOR all pairs of integers i j DO
IF h i j th element enumerated into A THEN OUTPUT i
END
END
B is many one This can b e shown even more simply but this metho d shows
reducible to A via g f Since A is computably enumerable it follows that B
is computably enumerable
The element with highest priority x L reaches its nal lo cation
A
at the very b eginning
The element with the second highest priority x L can b e removed
B
from of L and slid over at most one time So there are at most two
A
incarnations for this element
The element with the third highest priority x L has at most three
A
p ossible incarnations
Solutions
In general the numb er of incarnations satis es the recursive formula
f n f n f n f n f a where a is or and f is
de ned to b e This results in the Fib onacci sequence
since
f n f n f n
f n f n f n
f n f n f n f n
f n f n f n f n f n
The numb ers a and b must b e relatively prime
The given system of equations has a solution if and only if
k
X
f x x
i n
i
has a solution
This problem can b e reduced to the solution of the equation
k
Y
f x x
i n
i
n
f x x x has a solution if and only if one of the equations
n
f x x x
n
f x x x
n
f x x x
n
f x x x
n
has a solution in N This demonstrates the existence of a Turing reduction
b etween the two problems
With the help of the previous exercise this disjunction can b e converted
into a single equation That gives a many one reduction
Every natural numb er x can b e written as x u v w z with
u v w z Z So f x x has solutions in the natural numb ers if
n
and only if
Solutions
f u v w z u v w z
n n n n
has integer solutions
The register R can b e set to zero by
j
a IF R GOTO d
j
b DEC R
j
c GOTO a
d
The instruction R R can b e simulated by
m j
a R
n
b R
p
c IF R GOTO h
j
d DEC R
j
e INC R
n
f INC R
p
g GOTO c
h IF R GOTO l
p
i DEC R
p
j INC R
j
k GOTO h
l
If the register R has the value that is W has a in the appropriate
j j
place in the sequence numb er co ding where N has a then there will b e a
i
in the corresp onding bit of B T W at the next step So N cannot
j i
have a there but rather must have a Because of the previous equation
B N N N N must have a there so the next instruction to b e
i l i l
executed will b e instruction l as desired
If the register R contains a value then B T W has a in the
j j
bit corresp onding to the next step This explains the factor of to avoid
changing the value of the bit at the current time step This is also why we
needed the condition B S So N must have a and N a at that
i l
p osition Thus the next instruction to b e executed will b e instruction i
as desired
The exercise follows immediately from the hint and the observation
that and
y
x
The hint can b e proven as follows is the co e cient on the term T in
x
x
the expression T Mo d these co e cients can b e computed as follows
n
Y
i
y y y y
i n
T T T
i
y
n n
i
X Y Y
i i
y
i
j y
i
A
T T
j
j i i
Solutions
In the pro duct on the right side there is exactly one choice of terms which
x
when multiplied together give a term T This term is obtained by setting
Q
n
y y
i
j y in each sum Thus we get mo d which is what we
i
i
x x
i
wanted to prove
Note the computation ab ove would b e correct mo dulo any prime numb er
i i
p Use p instead of
Dioph N to the problem in question f Dioph N if and We reduce
only if f Dioph N The p olynomial f has only non negative values The
p olynomial f can b e written as f g h where g has all the terms of
f with p ositive co e cients and h has all the terms with negative co e
cients once more negated The p olynomials g and h then have only p ositive
co e cients Furthermore since f we have g h So we get
f Dioph N f Dioph N
x h x g x
x h x g x
The map f h g yields the desired reduction
We imagine the p olynomial f as a tree structure The leaves of the
tree are lab eled with the variables x x and the co e cients and the
n
internal no des of the tree are lab eled with the op erations and The ro ot
of the tree represents the function f
Every internal no de is now assigned a new variable y y y b eing
k
assigned to the ro ot Then f is equivalent to requiring that y and
that for each branching of the form
op y
i
e
e
e
e
u v
y u op v i e y u op v where op f g and u and v may b e
i i
new variables y y original variables x x or co e cients
k n
None of the Diophantine equations f has a total degree greater than two
i
Therefore the Diophantine equation that is equivalent to the conjunction of
P
all of these equations namely f has total degree at most four
i
Assignment statements have lo op depth If R is a LOOP program of
the form P Q where P and Q have lo op depth p and q resp ectively then R
has lo op depth max p q If R is a program of the form LOOP X DO P END
and P has lo op depth p then R has lo op depth p
Solutions
The following program simulates subtraction
Y Z
LOOP X DO Y Z Z Z END
X Y
The following program simulates IF X THEN P END
Y Y Y
LOOP X DO Y END
LOOP Y DO P END
In the following instructions that app ear in quotation marks indicate
suitable LOOP programs which exist by the previous exercises
Z X k Y
LOOP Z DO Y END
W X m m k
U
LOOP W DO U END
LOOP U DO Y Y END
Y Y
LOOP Y DO P END
Let the input register of A b e S and the output register T
n
A X n
LOOP S DO D END
n
T
LOOP A DO T END
D must b e a LOOP program so that A will b e a LOOP program
n n
The following program simulates w
Y X
LOOP X DO Y END
Solutions
A LOOP program for x MOD k
Z
Z
Z k
k
LOOP X DO
Z Z
k
Z Z
Z Z
Z Z
k k
END
Y X
A LOOP program for x DIV k
Z
Z
Z
k
LOOP X DO
Z Z
k k
Z Z
k k
Z Z
Z Z
k
END
Y X
A LOOP program can only consist of a sequence of instructions of
the form X X Y and X X From this it is clear that only the
given functions can b e computed
w f x w g w x
k k w k x MOD t w k x MOD t
x MOD t
w k x MOD t t
t
For each index p osition i f ng one has M K many di erent
n
equivalence classes so altogether there are M K
Case Let M max M M K and K K K Let
i f ng b e arbitrary It su ces to show that w f x g x and
0 0
w f x g x di er by a constant indep endent of x where x
x x and
n
Solutions
0
x x x x K x x
i i i n
and each x M By the inductive hyp othesis there are constants and
i i
i
such that
0
f x f x K K K
i i
and
0
f x f x K K K
i i
0
If then by the choice of M and b ecause x M f x and f x
i
i
are b oth strictly greater than Thus the value of the function w is at
b oth places So in this case If and f x then we also
i
have In the case that and f x the value of w is given by
i
f so K K K
i i
Case Let M M and K k K Let i f ng b e arbitrary It
0
is su cient to show that f x DIV k and f x DIV k di er by a constant
indep endent of x where x x x and
n
0
x x x x K x x
i i i n
and x M By the inductive hyp othesis there are constants and such
i i
i
0
that f x f x k K K So we have
i i
i
0
f x DIV k f x DIV k K K
i
k
n
By Lemma the value of f x for x N can b e determined in
the following manner First one rep eatedly reduces each comp onent x of x
i
that lie in and x provided x M by K until one gets two p oints x
i
i i
the interval M M K as sketched in the diagram b elow
x x
x
M K
M i i i
K K K K K K
tK K and x x x M x MOD K x Clearly x
i i
i i i i
for an appropriately chosen integer constant t Now let
x x f x x f x x
n n i
i i
Then
x x x f x f x x
n i
i i i
x tK f x x
n i
i
By reducing each comp onent x in this manner we arrive at an f value for
i
a p oint in Q Thus if the two functions are di erent they must di er on a
p oint in Q
Solutions
M K
Let M max M M and K K K The equivalence relation
M K M K
This means that the two given functions f and and re nes b oth
f agree if and only if all multi linear functions on the common equivalence
classes agree As in the previous exercise these functions are completely
characterized by their values on p oints x with x M K
i
Logical NOR can b e used to simulate the b o olean functions AND
OR and NOT
NOT x NOR x x
AND x y NOR NOT x NOT y
NOR NOR x x NOR y y
OR x y NOR NOR x y NOR x y
So it is su cient to give a LOOP program for NOR
Y
LOOP X DO Y END
LOOP X DO Y END
We can simulate n input variables X X with a single input
n
variable X by adding the following preamble to our program
X X MOD
X X MOD
X X MOD
X X MOD p
n n
where p is the nth prime numb er By the Chinese Remainder Theorem we
n
know that there is a suitable value for X for any choice of X X
n
INPUT x
k f jxj
p
FOR y jy j jxj DO
Run M on y
IF M accepts THEN
IF y x THEN REJECT END
p p
END
END
IF p k THEN ACCEPT ELSE REJECT END
Solutions
Note The b oxed p ortion of the algorithm is nondeterministic
If x A then there is a nondeterministic computation of this algorithm
that accepts on input x at each pass through the for lo op if y A a path
must b e chosen that causes M to accept y In this case at the end of the
lo op we will have p k
If on the other hand there is a nondeterministic computation that causes
this algorithm to accept x then since at the end of the lo op p k all of the
strings y such that jy j jxj and y A were discovered and none of them
was x so x A
O n
Time complexity t n t n
f M
Space complexity s n O n s n
f M
It is su cient if the nondeterministic machine computes f in the fol
lowing sense
Every nondeterministic computation is either successful and outputs a
numeric value or is unsuccessful and outputs nothing
At least one output is successful and therefore pro duces an output
All successful computations pro duce the same output value namely f n
One can mo dify the algorithm given in Solution to use g instead of
f to lo op over all y interpreted as expressions and to verify that S y
instead of starting M on y
n
f Assume k jT j g
i
p
FOR y V jy j n DO
f FALSE
m
FOR z V jz j n DO
i
IF S z THEN
G
m m
IF z y OR z y THEN f TRUE END
G
END
END
IF m k THEN REJECT END
IF f THEN p p END
END
n
f Now p jT j g
i
Note the b oxed p ortion of the algorithm is nondeterministic
In the inner lo op the counter m is compared against k to check that the
nondeterministic choices were made correctly The computation only con
tinues to the outer lo op if this is the case In this way we can b e certain
Solutions
that the numb er p is correct at the end of the algorithm in the case that a
nondeterministic computation reaches that p oint without rejecting so oner
L and NL machines are o line Turing machines which means that
their sp ecially designated input tap es can only b e used to read the input
For the purp oses of this exercise let a con guration of such a machine b e
an instantaneous description of the current state the head p osition on the
read only input tap e the head p osition on the work tap e and the contents
of the work tap e Let Z b e the set of all such con gurations and A the
alphab et Then a c log n space b ounded Turing machine has at most
c log n k
a jZ j n c log n jAj O n
many di erent con gurations where k is an appropriate constant So if the
computation runs for more than a steps some con guration must b e re
p eated But then the machine must b e in an in nite non terminating lo op
and therefore do es not accept
To see that NL P consider the following pro cedure for some xed
NL machine On input x systematically generate all con gurations that can
b e reached from the starting con guration until a halting con guration is
reached or all reachable con gurations have b een generated Since the numb er
of con gurations is b ounded as ab ove this can b e done in p olynomial time
Let M and M b e two log reduction machines On input x it is not
p ossible to simply let the machines run one after the other since M may
output a string that is p olynomially long in the length of x and therefore
cannot b e stored in logarithmic space Instead we pro ceed like the pipe con
cept in UNIX We start M as the main pro cess on an empty tap e but
whenever M tries to read an input tap e symb ol we start M on input x as
a sub pro cess to generate this symb ol All other output symb ols pro duced
by M are immediately erased Since M requires only logarithmic space
which can b e reused each time the pro cess is run and p olynomial time
the total space devoted to pro cess M is logarithmic and the time for M is
p olynomial
It is clear that PATH NL Nondeterministically guess a path from
a to b This can b e done by nondeterministically generating a sequence of n
no des in the graph and checking to see if adjacent pairs in the sequence are
adjacent in the graph and if b oth a and b o ccur in the sequence The only
space needed is the space to store a pair of no des and a counter
Now let A b e in NL via a O log n space b ounded machine M By Exer
cise this machine has at most p olynomially many con gurations on an
input of length n The desired reduction of A to PATH outputs for any x the
graph in which each such con guration is a no de and the there is an edge
from c to c if c is a con guration that could follow c in the computation on
i j j i
input x This can b e done for example by pro ducing for each con guration
the nite list of p ossible successor con gurations
Solutions
The start no de a in the PATH problem is the start con guration of the
Turing machine The end no de in the PATH problem is a unique accepting
condition The machine M can b e mo di ed so that there is always one
unique halting con guration Since each con guration can b e written down
in O log n bits this reduction is logarithmically space b ounded
A con guration can in general have more than one predecessor con g
uration If the graph describ ed in the solution to Exercise is interpreted
as an undirected graph then it may b e p ossible on some input x that is not
in the language to nevertheless reach the halting con guration by traveling
some of the edges backwards that is by traveling from a con guration to
a p ossible predecessor rather than to a successor
Consider the following algorithm on an input of length n
Success FALSE
WHILE NOT Success DO
Cho ose one of a and b with equal probability
a Move the read only head on the input tap e one cell to the right
IF the end of the input is reached THEN Success TRUE
b Move the read only head on the input tap e
back to the rst input symb ol
END
The probability of successfully reaching the end of the input on the rst
n
attempt is Each additional attempt costs at least one step so the ex
n
p ected value of the time until the algorithm successfully halts is at least
For the following directed graph the same statements are true as were
made in the preceding exercise
n
So if we cho ose a and b n then the exp ected length of time to get
from a to b is exp onential
If the edge probabilities are not all the same then there must b e edges
with probability greater than e and others with probability less than e
since the sum of all probabilities is Let p e b e the maximal edge
max
probability that o ccurs in the graph There must b e at least one edge u v
with this probability that has an adjacent edge v w with a strictly smaller
probability Since G is connected if this were not the case every edge would
have probability p but then the sum over all edges would b e greater
max
than All other edges adjacent to u v have probability at most p So
max
P p must b e strictly larger than the weighted sum of the adjacent
max
u v
edges probabilities i e
Solutions
X
P P
u v v w
d v
v w G
as was to b e shown
Let Y b e the following valued random variable
if X a
if X a
Then X a Y From this it follows that E X E a Y a E Y
a P r X a
On average we must visit no de u d u times until v o ccurs as the
successor no de So E u v can b e b ounded ab ove by d u E u u d u
e d u e This is in general a very crude approximation since it p ossible
to get from u to v via some other route that do es not use the edge u v
We pro ceed by induction on n the numb er of no des in the graph The
statement is clearly true for graphs with only one no de Let G b e a connected
graph with n no des There must b e a no de that can b e removed without
destroying the connectivity of the graph In the remaining graph by the
inductive hyp othesis there must b e a path of length n The removed no de
was attached to this path by at least one edge By traveling this edge once in
each direction we obtain a path of length at most n that includes this
no de as well
Let X b e the numb er of steps in a random walk from a to b By Markov
inequality letting a E X we get P r X en P r X E X
E X E X
For each of the n no des we must determine a rst second etc up to
dth adjacent no de Each time we have roughly n p ossibilities So for each
d
no de we have no more than n p ossibilities and altogether there are no more
d n dn
than n n p ossible ways to sp ecify such a graph
x x x
Unsatis able
Solutions
Correctness The last clause in a resolution pro of is clearly unsatis
able Now we work inductively from back to front and show that the set of
input clauses to a resolution pro of is unsatis able Let K K b e two resolv
x g that o ccurs in p ortion of able clauses with resolvent K K K fx
i i
the pro of already considered at which time it represented an input clause
Supp ose that the set of input clauses to the resolution pro of now augmented
by clauses K and K is satis able via an assignment Then in particular
must satisfy K and K The variable x on which the resolution o ccurred
i
is true in one of these clauses its negation in the other false or vice versa
Thus in one of the clauses a di erent literal must b e true under This literal
remains in the resolvent K so K is also true under But this contradicts
the inductive hyp othesis
Completeness For an unsatis able set of clauses with one variable there
is an obvious resolution pro of Now assume completeness for sets of clauses
containing up to n variables Consider an unsatis able set of clauses M with
n variables
First put x all o ccurrences of x can now b e stricken and for
n n
each o ccurrence of x the entire clause in which it o ccurs can b e stricken
n
This results in another unsatis able set of clauses M Analogously form M
by putting x By induction there are resolution pro ofs for M and
n
M Now we reconstruct the original clauses from M that is we reintro duce
x in M and also in the corresp onding resolvents in b oth x in M and
n n
resolution pro ofs Either one of the resulting resolution pro ofs still contains
the empty clause now derived from M in which case we are done or we
x g from which we can derive the empty have the resolvents fx g and f
n n
clause in one more resolution step
We construct the path from back to front starting at the empty clause
If we are at a clause K in the path with K then exactly one of the
predecessors of this clause is true under the other false We select the one
that is false under and continue the construction from there
The existence of a SUPER pro of system for refutation is equivalent to
the existence of a nondeterministic p olynomial time b ounded algorithm for
SAT so SAT NP Since SAT is NP complete SAT is coNP complete By
the closure of NP under p olynomial reductions coNP NP so coNP NP
In the other direction If NP coNP then SAT NP so there is a
SAT The nondeterministic p olynomial time b ounded Turing machine for
p ossible next con guration calculus of this machine is then a SUPER pro of
system for refutation
Solutions
Typ e clauses
Typ e clauses
Empty p ositions in the diagram b elow are understo o d to b e s
n
The typ e clause that has the column of lled with
The same clause as in the preceding exercise
A in the column would cause the clause to b e true under
Since is critical the diagram for must have n s each in a di erent
row Since n s o ccur in the column n p ossible rows are ruled out
Since S has already xed n s that rules out in the worst case another
n p ositions So at least n n n n p ossibilities remain
If there were two s in one column then at least one would have to
b e in a p osition where there is a in From this it would follow that the
clause is true under
The alteration from to in the original column has no e ect
S
since that p osition of K do es not contain The alteration from to
S
in one of the n p ositions has no e ect since that p osition of K by
assumption do es not contain
Solutions
n n
t clauses are taken care of Since Supp ose fewer than
n n
the p osition was chosen maximally in every other p osition strictly fewer than
n n
t s o ccur Now we sum over all p ositions in the clauses and
n n
obtain an upp er b ound on the numb er of s in all t clauses together there
are strictly fewer than n n t This is a contradiction since we
have argued that each of the t clauses that is input to the greedy algorithm
has at least n n s
A tautology has a mo del of every size so cho ose something like F
P P where P is ary
We let F express that there are at least di erent elements in the
universe but that in any set of elements two of them are the same
F x y z x y x z y z
u x y z u x u y u z x y x z y z
We describ e a recursive pro cedure that evaluates F using A L represents
a list of variable conditions which is initially empty
PROCEDURE eval F A L BOOLEAN
VAR b BOOLEAN
BEGIN
IF F G H THEN f is a b o olean op eration g
RETURN eval G A L eval H A L END
IF F x G THEN
b FALSE
FOR w TO jAj DO
b b OR eval G A L f x w g
END
RETURN b
END
IF F x G THEN
b TRUE
FOR w TO jAj DO
b b AND eval G A L f x w g
END
RETURN b
END
IF F P x x f where x w L g
m i i
THEN RETURN The value of the relation corresp onding to P
on the tuple w w
m
END
IF F x x THEN RETURN w w END
i j i j END
Solutions
The running time of this algorithm is p olynomial the degree of the p olyno
mial dep ending on the numb er of quanti ers in F Rememb er F is not part
of the input to the algorithm but xed
n
An NP algorithm for a given formula F on input guesses nonde
terministically a structure A appropriate for F This can b e written down
using p olynomially in n many bits the p olynomial dep ends on F Then the
algorithm tests as in the previous exercise whether A is a mo del for F This
shows that the problem is in NP
On input of a binary representation of n everything works just as ab ove
n k k n
but the running time is increased to with resp ect to the new
logarithmically shorter input length n
An NP algorithm for a given formula F on input A M R co ded
as a string nondeterministically guesses relations R on the universe for
i
each P in the formula These can b e written down using p olynomially many
i
bits Then the algorithm tests as in the previous two exercises whether
A M R R R is a mo del for F This shows that the problem is in
m
NP
n
Min y x y
i
i
n
Min x Min x E x P x y
i
z
i
E x P x y
z
n
Min x Min x E x P x y
i
i
E x P x y
n
Min x P x y
i
i
x y P x y P x y
a b
a
b
b a
k
x y Min x Max y P x y
i i
z
e
i
Solutions
F x x x
x y x y y x
x y z x y y z x z
x y x y y x x y
Supp ose NP is closed under complement and let L NEXP Then
the language
n
t L f j bin n Lg
where bin n is the binary representation of n is in NP By our assumption
t L NP From this it follows that L NEXP
In the other direction supp ose NEXP is closed under complement and
let L NP Then the language
n
b L fbin n j Lg
b L is in NEXP From this it follows that L NP is in NEXP By assumption
This result originated with
R V Bo ok Tally languages and complexity classes Information and
Control
Since the Kolmogorov complexity of a string is the length of some
program it is clear that this must b e The length of a program that
has a dummy input instruction and then pro ceeds to output x without
making any use of the input provides an upp er b ound for K x j y i e
K x j y K x For any string x f g OUTPUT x is always a
p ossible program that outputs x This program has length jxj c for some
constant c So K x jxj c
The program INPUT v OUTPUT v where v is a variable has
constant length so K x j x c for all x
There is a program of xed length that by means of a suitable approx
imation algorithm on input n pro duces the rst n digits of Let c b e the
length of such a program Then for all n K j n c
n
n
In the worst case the K values of the strings of length n are dis
tributed as follows K once K twice K four times K n
n
times and nally K n for one string If we add these values and divide
n
by the numb er of strings we get
Solutions
n
X
i
E K x i n
n
i
n
X
n
i n
i
n
i
n
X
n
j
n j
n
j
n
X X
n
j k
n k
n
j
k
k n
X X X
j k n
A
n
j
k l
X X
k
n
l k l
n
The same argument also holds for E K x j y where y is arbitrary
Supp ose x K x is computable via a program M which halts on all
inputs We can use M as a sub routine in the following sequence of programs
P P where P is
m
x
REPEAT
x successor of x
UNTIL M x outputs a value m
OUTPUT x
Notice that the the constant m is a parameter in this program The length of
P is O log m furthermore each program m describ es some string x
m m
namely the lexicographically rst string such that K x m But since P
m
describ es x K x jP j O log m For large m O log m m
m m m
which is a contradiction since K x m Thus the function x K x
m
cannot b e computable
This pro of resembles the Berry Paradox see page of the b o ok by
Machtey and Young
The argument here is similar to that used in the Exercise
n
X X
i i K xjn n
i
fx jxj ng
Solutions
n
X
i n
i
log m k log m k k
There are at most m
programs of length log m k Each one describ es at most one string x with
k k
K x j y log m k So there must b e at least m m m
strings left over
n n
There are only n s and n strings of length n with
n n
p
n
n
s By Stirling s formula n we get n
e
n n
n n n
r
n
n
n
r
n
n
There is an algorithm of length O log i that outputs the ith string of
n s and n s Since log i is b ounded length n with the prop erty that it has
q
n
by log n this length is strictly less than
n
n So such a string cannot b e Kolmogorov random
The approximation ab ove can also b e obtained via the Binomial Theorem
n
n
X
n
i n i
i
i
n
X
n
i n i
i
i
n
n i
X
n
n n
i
i
n
X
n
n n
i
i
P
n
n
n n n
In general From this it follows that
i
i
P
n
n
n H
it can b e shown in the same manner that where
i i
Solutions
and H log log is the entropy function
See
D Welsh Codes and Cryptography Oxford University Press p
Let T x b e the running time of algorithm A on input x Then
A
wc
T n maxfT x jxj ng
A
A
is the worst case complexity of A and
X
av
x T x T n
A
A
fx jxj ng
is the average case complexity under the universal distribution We have
wc
n and x for some constant already seen that T x T
n A n
A
From this it follows that
X
av
x T x T n
A
A
fx jxj ng
x T x
n A n
wc
x T n
n
A
wc
T n
A
as was to b e shown
Since p n log n So jbin n j log log n and the length of the
i i i
enco ding of the nite sequence n n is O k log log n O log log n
k
Supp ose we have shown that p m log m Then we can put n
m
n log n and solve for n
n n
n since n n
log n log n
INPUT hm k i
Determine the mth prime numb er Call it p
Compute n p k and OUTPUT n
We can determine the length of w from the initial p ortion of the co ding
string the end of which is marked with a then we can read w Since we
know how long w is this is self terminating
The length of the co ding is given by
jco de w j jw j log jw j
Solutions
We get the b ound on p by plugging in
m
log n K bin n log m log log m log n log p
m
from which it follows that
log p log m log log m
m
p m log m
m
If we let co de w co de bin jw j w then the length of an enco ding of
a string of length n is n log n log log n This can b e iterated further to
m
give an enco ding of length n log n log log n log log log n log n
By using co de instead of co de our approximation is improved to
n
n
log n log log n
If n is prime then it can b e describ ed by giving its index in the increas
ing sequence of prime numb ers This leads to the following contradiction
K bin n log n log n log log n
n
We get K g c log g n d Since every n place b o olean
n
function can b e computed with O gates log g n O n Plugging in
n n
yields g O n n
A formula corresp onds to a binary tree in which all interior no des
the gates have exactly two children In such a tree it is always the case
that the numb er of interior no des is less than the numb er of leaves So
the numb er of times an input is mentioned is g The p ost x co de for
a formula consists of g inputs each requiring ab out log n bits and g
op erations each requiring a constant numb er of bits So the length of an
enco ding of a formula with g gates using p ost x notation is b ounded ab ove
by O g O g log n O g log n From this it follows as in the previous
n n
exercise that K O g log n so g log n
Every state that app ears in one of the crossing sequences o ccurs at
some p oint in the computation and therefore contributes step to the total
running time On the other hand every state that o ccurs at some p oint in
the computation must app ear once in some crossing sequence
If the crossing sequences at i are identical then b oth computations
b ehave exactly the same in the p ortions of the computation during which the
tap e head is left of i But by our assumption that Turing machines only halt
with the tap e head at the left end of the tap e they must either b oth accept
or b oth reject
See ab ove
Solutions
We can paste together the p ortions of the computations left and right
of p osition i and we see that the computation remains essentially unchanged
Supp ose the crossing sequences are identical Apply the lemma from
jw j
Exercise to show that the crossing sequences at p osition i of w w
jw j
and w w are also identical Now apply the lemma from Exercise with
i jw j jw j i jw j i
x w y w and z w But xy L and xz L This
is a contradiction so the crossing sequences must b e di erent
INPUT hM m i ci
FOR w jw j m DO
jw j
Simulate M on input w w and
note the crossing sequence at p osition i
If this crossing sequence is the same as c then output w
END
n
X X
jCS x i j jCS x i j time x
M M M
i
i n
n
X
n O log n
i n
n O n log n n
The probability is
Using Exercise this numb er is at most
n
n
X
n
H
i
i
where H is the entropy function
ln
For PAC learnability it su ces to cho ose m log jH j log
p n m
Now let jH j and plug in
ln log ln p n ln
m p n m log m
From this we get
p n ln log ln
m
m
Solutions
It su ces to cho ose
log ln p n ln
m
From this we get
p n ln log ln
m
This is p olynomial in n and
There are n p ossibilities for the choice of a literal z So there are
ij
k
at most n p ossible monomials in fact there are fewer A DNF
n k
formulas consists of an arbitrary subset of these monomials conjunctively
k
n
combined There are such subsets
W
l
Let the function to b e learned b e f m where the monomials
i
i
k
m represent a certain choice of the n monomials with at most k
i
literals The hyp othesis function h initially includes all p ossible monomials
with at most k literals In each pass through the lo op the only monomials
m removed from h are the monomials such that m fm m g This
l
guarantees consistence with the negative examples But h is also consistent
with resp ect to the p ositive examples since we always ensure that h f
Every n place b o olean function is equivalent to a logic formula which
can b e expressed in either disjunctive normal form or conjunctive normal
form These two forms corresp ond corresp ond to depth OR AND and AND
OR circuits But the size of the formulas generated in this way is exp onential
in general
OR
AND AND
A A E E
A A E E
A A E E
E E
A A
x x x
x x x x x
The fan in on the rst level at the AND gates is d and the fan in on
d
the second level at the OR gates is c
Replace all AND gates with OR gates and vice versa and all x input
i
gates with x and vice versa This corresp onds to DeMorgan s laws i
Solutions
x y x y
x y x y
This metho d works for any function For parity in particular there is an even
simpler metho d Since
par x x x par x x x
n n
we only need to swap x with x
Since
par x x par x x
n n
n n
and
par x x par x x
n n
n n
these restrictions always result in another parity function or its complement
Every AND gate on level of a depth circuit for par must have
n
n inputs That is every variable or its complement must b e one of the in
puts Supp ose this were not the case i e there is some AND gate that is
i
i i
n
missing WLOG b oth x and x The inputs are x x x with
n n
n
i f g This AND gate outputs and therefore the entire circuit
j
outputs if we put x exactly when i j n Now
j j
we can set x to b e either or without changing the output of the circuit
n
So the circuit do es not correctly compute parity Contradiction
Now if each AND gate in the circuit has n inputs as we have just shown
is necessary then each of these AND gates can only capture one row of the
n
truth table for par Since there are rows of the truth table with the
n
n
value there must b e AND gates
Using XOR gates with two inputs we can compute parity with a
circuit in the form of a balanced tree
XOR
Z
Z
Z
Z
XOR XOR
XOR XOR XOR XOR
x x x x x x x x
Solutions
Each XOR gate can now b e replaced by a small circuit of AND and OR
gates Note that we need to provide this circuit with b oth variables and their
XOR negations and that the circuit likewise outputs b oth XOR and
XOR x y
XOR x y
OR OR
T
T
T
T
T
T
AND AND AND AND
y
x
y
x
As one can see the ab ove solution only requires AND and OR gates of
constant fan in namely The b o olean functions that can b e computed by
k
k
constant fan in O log n depth circuits form the class NC So we have
just shown that PARITY NC In fact al l symmetric b o olean functions are
in the class NC
Let Y b e a random variable that is if the ith trail results in suc
i
P
n
Y cess otherwise E Y p q p So E X E
i i
i
P
n
E Y n p Furthermore V X E X E X E X E X
i
i
P P P
n n n
E Y np E Y Y np n n p np
i i j
i i j
np n p p
P r jX E X j a P r X E X a By Markov s
inequality P r X E X a E X E X a V X a
V X
P r X P r jX E X j
n
X
n
i n i
p p P r X a
i
i a
n
X
n
i
p
i
i a
n
X
n
a
p
i
i a
a n
p
Solutions
In fact we can show an even b etter approximation
n
X
n
i n i
p p P r X a
i
i a
n a
X
n
a i n a i
p p
a i
i
n a
X
n n a
a i n a i
p p
a i
i
n a
X
n a n
i n a i a
p p p
i a
i
n
a
p
a
a a
n p
a
np
Using this approximation the approximations in Exercises and
can also b e improved
Supp ose there is a family of p olynomial size depth t circuits for
PARITY If we arti cially add a new rst level consisting of AND or OR
gates with fan in one connected to each variable used in the gate then
we get a depth t circuit with constant input fan in which contradicts the
claim
Let X b e the random variable that contains the numb er of variables
r
Then that remain in S
n
p
r
P r fewer than n variables remain in S
n
p
V X
p
P r jX E X j n
n
p
O
n
In fact it is p ossible to prove much sharp er b ounds lo ok for Cherno b ounds
cf Topic
We must show that for every n we can nd a restricted circuit with
exactly n inputs Let n b e xed As a starting p oint consider the circuit S
n
r
With probability greater than the circuit S still has at least n inputs
n
Thus there exists a restricted circuit with m inputs where n m n The
k k
size of this circuit is O n O n If we now set any m n of the
inputs to then we get the desired circuit which is p olynomial in size and
has exactly n inputs
Solutions
P r AND gate not
P r all inputs
k ln n
P r an arbitrary xed input is not
k ln n
for n
k ln
n
k
n since ln x x
P r AND gate dep ends on more than a variables
k ln n
X
p p
n
i n i
n n
i
i a
p
a k ln n
n Exercise
a k
n n
k a
n
Solving k a k for a we obtain the constant a k
P r the AND gate is not constant
P r none of the OR gates obtains the value
d ln n
P r an arbitrary xed OR gate do es not get set to
c d ln n
for n
c
d ln
n
c
d
n since ln x x
k
n plugging in d
In this case there are at most d ln n such OR gates and each one
has fan in at most c So jH j c d ln n
If there were an OR gate with a set of variables disjoint from H
then the set of OR gates that de ned H would not b e maximal since this
OR gate with disjoint variables could b e added to the set
By Exercise jH j cd ln n So by Exercise we get the
approximation
p
cd ln n a
P r h a n
cd a
n n
cd a
n
Solving cd a k for a we get a cd k
Solutions
Q
n
x OR x x
i n
i
t t
The error probability is We put and get t log
AND x x p x x This corresp onds to the
n n
DeMorgan law
AND x x NOT OR NOT x NOT x
n n
n
log n
P r jT S j P r jT S j
log n log n
The probability is that case do es not o ccur and in case
the probability is that there is an i with jT S j Therefore we
i
get a probability of at least
d
O log s log s As long as s s n is a p olynomial for
example is a constant and the depth d is constant then this is p olylogarith
mic in n
Supp ose there is no choice of random decisions with the prop erty that
n
for at least values of a p a f a That means that for all random
n
decisions which lead to a p olynomial p there are less than values of
n
a with p a f a But then the exp ected value must b e less than
a contradiction
In one direction the function is x x in the other x x
The numb er of s in a vector is o dd if and only if the numb er if
terms in the corresp onding vector is o dd This corresp onds to
a pro duct
If the p olynomial q contains a term of the form y then we can
i
replace that term by since we are only interested in the p olynomial for
y f g Thus exp onents greater than are unnecessary
i
n
n
p
From this it Using Stirling s formula we can show that
n
n
follows that
p p
n n n n n
X X X
n n n
i i i
i i
i n
p
n
n
n
n
n
p
n
p
n
n
n
Solutions
Since y f g
i
n
Y Y Y
y y q y y y
i i n i
i
i T i T
Y Y
y y
i
i
i T
i T
Y
y
i
i T
t
We have already shown an upp er b ound of O log s log s for the
degree of an approximating p olynomial Now treat t as a yet undetermined
p
n for the degree of function of n By comparison with the lower b ound
an approximating p olynomial for PARITY we see that for certain constants
p
dt log n
d and e it must b e that e log n n Solving for t yields t
log log n
Let G AC via the constant depth b ound t and the p olynomial
size b ound p If we replace the ctitious G gates in the reduction circuit with
the AC realization of G we get an AC circuit for F of depth t t and
size p n p p n
The ma jority function is a sp ecial case of threshold function
T x x where T x x exactly if at least k of the x s
k n k n i
have the value So maj x x T x x
n n
dn e
In the other direction the ma jority function can also simulate arbitrary
threshold functions Supp ose for example that k n Then
T x x maj x x
k n n
n k
z
n k
If k n then
T x x maj x x
k n k n
z
k n
We can also construct the Exact k functions from the functions T E is
k k
if and only if exactly k of the x s have the value
i
E x x T x x T x x
k n k n n k n
T x x T x x
k n k n
Finally we get the parity function via the following circuit of constant
depth In the diagram b elow assume that n is o dd and that
x x x x x
n n
Solutions
OR
HY
H
I
H
H
H
H
H
E
E E E
n
n
x x x x
Analogous arguments can b e used to show that al l symmetric functions
can b e reduced to ma jority by AC reductions
If a is a b o olean function in the variables S fx x g jS j n
i n
then a can b e written in unabbreviated disjunctive normal form Then
i
a if and only if exactly one clause of the disjunction is true Since at
i
most one clause is true we can get rid of the disjunction and feed the outputs
of the ANDs weighted with the w s directly into the threshold gate
i
In p olynomial p the monomials with i variables have co e cients
n
i
X
So every such monomial in p olynomial q has co e cient n
i
i
i
Let x x b e the common variables of F and G We can give a truth
n
table for H by rst giving simpli ed truth tables for F and G that only
consider the common variables x x We place a value of or in the
n
truth table whenever the values assigned to x x are already su cient
n
to determine the truth value of F or G We place a when the value dep ends
on the assignment to the remaining variables
For formulas F and G for which F G not every combination of
is p ossible For example the combination F and G is not p ossible
since this would mean that there exists an assignment of all variables that
makes F true but G false contradicting F G
The following table shows all p ossible combinations of values for F and
G along with the correct choice for the formula H This demonstrates that a
formula H exists
F H G
or
Solutions
See also
G S Bo olos R C Je rey Computability and Logic Cambridge Univer
sity Press nd edition
In order to b e able to distinguish the enco dings of o ccurrences of dis
tinct variables we need a certain numb er of bits and this numb er increases
with the numb er of variables With bit we can distinguish at most vari
ables with bits at most etc In order to write down m di erent variables
we need at least m log m bits So in a formula co ded with n bits only
m O n log n variables can o ccur
Supp ose an arbitrary assignment of all variables x y and z variables
in F and G is given If this assignment makes F true x A since F is
n n n n
A NP a Co ok formula for A NP Since G is a Co ok formula for
n
G x z and G x z So F G
n n
We must show that x A H x If x A then F is
n n
satis able So there is an assignment to the x y variables with F x y
n
Since F H H x
n n n
On the other hand if x A then there is an assignment to the x z
variables with G x z Since H G equivalently G H
n n n n n
H x i e H x In fact we have F G
n n n n
A Cho ose A A and A
Let F x y b e the Co ok formula for A NP and let G x z the
n n
Co ok formula for A NP Since A and A are disjoint F G If
n n
an interp olant of F and G has p olynomial size circuits i e int F G
n n n n
is p olynomial in n then A and A are PC separable
Supp ose that the last two statements are false i e NP coNP and
interp olants can b e computed in p olynomial time We will show that P NP
Since NP coNP b oth SAT and SAT are NP complete For SAT and SAT
there are corresp onding Co ok formulas F x y and G x z and F
n n n
G In p olynomial time in n we can compute their interp olant H x
n n
for which x S AT H x From this it follows that S AT P so
n
P NP
Consider the language
A fhH x y ai j H is a b o olean formula and a an assign
ment to the x variables such that there
is an assignment to the y variables that
makes H trueg
This language A is in NP and has by hyp othesis p olynomial size circuits
This means there is a sequence of circuits c c of p olynomial size such
that
Solutions
c hH ai hH ai A b H a b
jhH aij
Now let F and G b e two formulas of length n with F G F F x y
and G G x z i e x is the set of common variables Let c b e a circuit
F
family for A as ab ove but with the rst parameter xed equal to F H F
Then c a b F a b We claim that c is a circuit for an
F F
interp olant of F and G It is clear that every assignment that satis es F also
makes c Now let a b e an assignment such that c a Then there
F F
is an assignment b such that F a b Since F G it must b e the case
that for every assignment d G a d Thus c G
F
Every branching in a branching program of the form
x
i
n
H
H
Hj
x
i
can b e lo cally replaced with the following sub circuit
x
i
x
i
Finally all connections that lead into an accepting no de are joined together
in one big OR gate
BP INEQ NP The following describ es a nondeterministic Turing
machine M for BP INEQ
INPUT hB B i
GUESS x x f g
n
IF B x x B x x THEN ACCEPT
n n
ELSE REJECT
END
BP INEQ is NP hard The goal of this pro of is to construct from a predicate
logic formula two branching programs that are inequivalent if and only if
the formula is satis able Consider an example Let F x x x
x x x This formula is satis able From this we must construct
a branching formula B with B F B is made up of subgraphs that
F F F
represent the individual clauses of F
Solutions
x
x x
x x
x
x
x x
x
x
x
The second branching program will b e a program B that computes the
U
function
If F SAT then there is an assignment on which B computes the
F
value Since B computes on every input these branching programs are
U
inequivalent i e B B BP INEQ On the other hand if B and B
F U F U
are di erent then there is an input on which B computes the value Since
F
B F this is also a satisfying assignment for F Altogether we have
F
F SAT B B BP INEQ
F U
It is clear that from any formula in conjunctive normal form one can
construct a branching program B If F consists of k clauses then B has at
F F
most k no des The size of B is a constant for all inputs For example
U
B could b e a branching program that consists of a single rejecting terminal
U
no de So the construction of B and B from the formula F can b e carried
F U
out in p olynomial time
Note that the branching program B is not in general one time only and
F
that this is signi cant for the pro of
The p olynomials in the individual no des are
p
p x
p x x x x x x
p x x
p x x x x x x x x
p x x x x x
x x x x x x
p p
B
Solutions
Truth table
x x x p p p p p p B
As one can see p B Now we prove the claim
B
Let x x f g and let V b e the set of all no des in B that
n m
are reachable from v in exactly m steps Then for all v V
star t m
if v is reachable on the mth step
of B x x
n
p x x
v n
otherwise
p
Base case m p
v
star t
Inductive step Assume the claim is valid for all no des that can b e reached
in m steps
Let v b e a no de in B that is reached in step m step of the computation
and let the no des v v b e the predecessors of v Every assignment to
l
x x determines exactly how the branching program is traversed v is
n
reached in step m then there is exactly one no de v i l which was
i
visited in the m th step By the inductive hyp othesis on input x x
n
p p p p
v v v i v
i
z z
z
p p p
n v i i v i v
n i i
z z
z
So the value of p is completely determined by There are two p ossibilities
v i
for the lab el of the edge v v
i
x in which case p x
j v i j
x in which case p x
j v i j
Since every computation of B halts in some terminal no de v after nitely
e
Finally p is the sum of all p olynomials of accepting many steps p
B v
e
terminal no des so
p x x B x x
B n n
Solutions
Consider the following counterexample Here are two branching pro
grams B and B for the function f x x x x
x x x x x B p
B
x x
x x
B p x x x x x
B
x x
x
x x x
and p are not identical p are equivalent but p B and B
B B
B
p
B
The pro of is by induction on the numb er of variables n
If n then p and q are lines If they are not identical then they can
only intersect in one p oint so they must b e di erent in at least jS j p oints
Now supp ose n Then
X X
i
i
n
x a x p x x
i i n
n
n
i i
n
X X
i
i
n
x a x x
i i
n n
z
i i
n
z
p x x
n
X X
i
i
n
x a x x
i i
n
n
i i
n
z
p x x
n
p x x x p x x
n n
Analogously q x x q x x x q x x
n n n
Since p q either p q or p q First we handle the case when
n
p q By the inductive hyp othesis p and q di er on at least jS j
Solutions
p oints We will show that for every x x there is at most one choice
n
for x such that p and q are equal on the input x x x Solving the
n
equation
p x x x p x x q x x x q x x
n n n n
for x we get
p x x q x x
n n
x
q x x p x x
n n
which is only a solution if the value is in S There are at least jS j choices
for x which lead to di erent values of p and q So altogether there are at
n n
least jS j jS j jS j choices for x x that lead to
n
di erent values of p and q
If p q but p q then the value of x do esn t matter for the equality
n
of p and q So in this case there are actually at least jS j jS j values
for x x that make p and q di erent
n
Let S f g in Theorem
from one time only branching programs The p olynomials p and p
B B
n
are multi linear By Exercise they must agree on all values in f g
n
S By the previous exercise it follows that p p
B B
for all x x f ng B and B are equivalent so p p
n B B
Thus P r M B B rejects
Let p b e the probability that the algorithm accepts By Theorem
n n
n jS j
n
p
n n
jS j n n
q
n
More precisely it can b e shown that lim
n
n e
We will describ e a reduction that maps each b o olean formula F in
conjunctive normal form to two one time only branching programs B and
B such that F is satis able if and only if there is an argument tuple y with
y Expressed di erently F is unsatis able if and y and f f
B B
y for all y y f only if f
B B
Let F b e a CNF formula with k clauses and in which the n variables
x x o ccur The variable set y for each of the branching programs is
n
y y
k
y y
k
y y
n nk
The intended connection b etween a satisfying assignment x x x
n
for F and an assignment y y y for which B y and B y
nk
is the following
Solutions
y if x o ccurs in clause j
ij i
x
i
y otherwise
ij
y if x o ccurs in clause j
ij i
x
i
y otherwise
ij
The basic construction of B is indicated in the following sketch
h h h h h h
R R R
h h fj h
R R R
h h h h h h
i i n
Undrawn edges are to b e understo o d as leading to a rejecting terminal no de
For every i n there are two separate paths b etween the connecting
no des The k edges on the upp er path are lab eled with y j k if
ij
y The lab els on the lower path are x o ccurs in clause j otherwise with
ij i
determined analogously but with x playing the role that x played in the
i i
upp er path
For example the formula
F x x x x x
maps to the following branching program B
y
y
h h h h
y
y
y y
h h gj
R R
y y
y
y
R R
h h h h
y y
n
Clearly there are paths from the start no de to the accepting terminal
n
no de And since the branching program is one time only there are assign
n
ments to y for which B y These assignments can b e assigned to
original assignments for x as describ ed ab ove
Every satisfying assignment for F corresp onds to an assignment to y with
the prop erty that in the arrangement of the y s illustrated ab ove every
ij
column contains a The following branching program B will describ e all
y assignments except those that corresp ond to satisfying assignments to F
in this way that is all assignments for which there is some column with no
s in it
Solutions
y i n
i
i i i in
Z Z Z
Z Z Z
Z Z Z
y i n
i
i i i in
Z Z Z
Z Z Z
Z Z Z
y i n
ik
i i i in
Notice that B only dep ends on k and n and not on the inner structure of
F itself
For example the formula F from ab ove for which k and n
results in the following branching program B
y y
j
Z
Z
y
y
Z
y y
Z
Z
j
Z
Z
y
y
Z
Z
Z
y y
j
Now F is satis able if and only if there is an assignment y with B y
B y so the inclusion problem for one time only branching programs is
coNP complete
This is b ecause f is critical that is for ever x with f x and
0 0
every x that di ers from x in any bit f x So if some path in the
branching program accepts x but do es not query every bit then that path
0 0
cannot distinguish b etween x and x and will incorrectly accept x This
contradicts the assumption that B correctly computes the function f
Solutions
0
Supp ose k x k x Since the p olynomials asso ciated with x and
0
x are di erent by the Interp olation Theorem this di erence must show
itself in any n queries that are answered yes since each such determines
0
a p oint on the p olynomial So the x path and the x path must b e di erent
0
b oth ab ove and b elow k x k x
h
x
0
x
h
0
x
x
m mh
0
By following the x path ab ove k x and the x path b elow we get a new
accepting path corresp onding to an assignment x for which f x This in
turn corresp onds to another p olynomial in POL There can b e no con icting
de nitions for x since B is one time only But this is imp ossible since the
p olynomials corresp onding to x and x cannot b e equal in n n places
and di erent in others
If P NP then all languages except for and in NP are also
NP complete In particular all nite sets are NP complete But an in nite
language such as SAT can not b e isomorphic and therefore certainly not
P isomorphic to a nite language
Let x b e a b o olean formula and let y y y y b e an arbitrary
k
string WLOG over f g Now let
y y
k
u p x y x z z u
SAT
where z u u are new variables that do not o ccur in x y means y and
k
y means y It is clear that p is p olynomial time computable and that this
A
formula is satis able if and only if x is satis able Furthermore the formula
is longer than jxj jy j Injectivity in argument y is also clear and from a
formula p x y as ab ove it is easy to reconstruct y This is the reason for
A
the variable z it allows us to detect where the y part of the formula b egins
So d also exists as required
A
P P
Let A B via f and B A via g Now put
m m
f x p f x x and g x p g x x
B A
By the prop erties of p and p f and g are correct p olynomial time com
A B
putable reductions from A to B and from B to A resp ectively Furthermore
Solutions
jf x j jf x j jxj jxj and jg x j jg x j jxj jxj The inverse
functions for f and g are d and d and are therefore p olynomial time
B A
computable
P P
Let A B via the injective function f and B A via the injective
m m
function g Furthermore supp ose that jf x j jxj jg x j jxj and that the
functions f and g are p olynomial time computable We need to de ne
a bijective function h that is a reduction from A to B For the de nition of
h x there are essentially two choices available f x or g x Of course
the latter choice only exists if x is in the range of g
Sketch
g x f x
t t
S
S
S
S
Sw
t t
x
f g x
By applying the functions g and f we can pro ceed in a zig zag manner
backwards from x until we arrive at a string that is not in the range of g or
f resp ectively In the example ab ove f g x is no longer in the range
of g so the zig zag chain ends on the lower line Now de ne
f x if the chain starting with x ends on the lower line
h x
g x if the chain starting with x ends on the upp er line
Since the functions f and g are length increasing no zig zag chain can b e
longer than jxj so h can b e computed in p olynomial time
Next we must show that h is a bijection For the pro of of injectivity let
x and y b e distinct strings such that h x h y Since f is injective this
can only happ en if h x f x and h y g y or vice versa But then
x and y are in the same zig zag chain and the de nition of h would either
have used f b oth times or g b oth times This is a contradiction
For the pro of of surjectivity let z b e an arbitrary string We must show
that z is in the range of h Consider a zig zag chain starting at z in the
upp er line If this chain ends in the upp er line including the case that z is
not in the range of f then for x g z h x g x z If the chain
ends in the lower line then x f z must exist and h x z
The inverse function for h is also bijective and can b e de ned using a
similar case distinction so it is also p olynomial time computable
Solutions
Let y b e the enco ding of some satis able formula and let l jy j
y
For any x the formula
x x x
y
is also satis able Let f x b e the enco ding of x as a string Then under a
reasonable enco ding scheme jf x j jy j jxj k jxj l k jxj C for
m
some constants k and C Thus there are at least strings of length m C
n C C n
in SAT i e for large enough n there are at least
strings of length at most n in SAT
n
By the previous exercise SAT contains at least strings of length n
for some constants and If SAT were P isomorphic to a language
S where S contained only p n strings of length at most n then there
would b e a bijective function f mapping SAT to S Since f is p olynomial
time computable there must b e a p olynomial q such that jf x j q jxj
n
From this we get the following contradiction f maps strings injectively
into p q n p ossible range elements
P
Lef tSAT via the reduction F F SAT
m
P
SAT via the reduction F a a F x x Lef tSAT
i n
m
a a x x where the formula a a x x is de ned as
i n i n
follows Let E fj f ig j a g and N fj f ig j a g
j j
Then
A
a a x x x x
i n j l
j E l N l j
P
SAT is simply to observe that Another way to show that Lef tSAT
m
Lef tSAT is in NP and that SAT is NP complete
Let b b b e an initial segment of b which we assume by induction
i
b elongs to T By the construction of U b b and b b are added to
i i
U so the correct initial section of b of length i let s call it b is in U
If F b has the same g value as some other string in U then the smaller
of the two is stricken but this cannot b e b since it is the largest element
of U for which F b Lef tSAT and g is a correct reduction So after the
rst reduction of U each string in U has a distinct g value and b is among
them If U contains more than m p q jF j strings then U is restricted to
the rst m of these But this must include b since at most m p ossible values
of g are available for elements of Lef tSAT including F b
Solutions
E
T
T
E
T
E
T
E
T
E
T
E
T
E
T
E
T
s s s s s s s
E
T
E
T
E
E
T
n n
b
The sketch ab ove shows a p ossible distribution of the strings in U following
the rst reduction step
P P P P
Then there is a language L Let L Supp ose
i i i i
with
L fx j y hx y i L g
P
Let By assumption L is also in
i
L fhx y i j z z Qz hx z z i Ag
i i
where A is a language in P Then we have
L fx j y hx y i L g
fx j y z z Qz hx y z z i Ag
i i
fx j u z Qz u hy z i hx y z z i A g
i i
P
The expression in square brackets is a predicate in P so L in
i
P P
We show by induction on k that for all Now supp ose that
i i
P P
k From this it follows that PH The base case of
i k
i i
P
the induction is clear For the induction step let L b e a language in
i k
P
Then for some language L
i k
g L fx j y hx y i L
P P
L so L From this it follows that By the inductive hyp othesis
i i
P P P P
L By assumption so L
i i i i
P P P
Finally supp ose PH Since PH it is immediate that
i i i
P P P
so
i i i
Cho ose a language over the one element alphab et f g that is not in P
This language has p olynomial size circuits for each n design a circuit that
n n
on input outputs either or according to whether or not L Such
Solutions
a language L that is not in P in fact not even computable can b e easily
de ned by diagonalization for example
n
L f j the n th p olynomial time machine on input
n
do es not acceptg
If L has p olynomial size circuits then the corresp onding circuits
c c c can b e co ded in a sparse set
n
S fh y i j y is an initial segment of c g
n
With this language as oracle we can use p olynomially many oracle queries
to reconstruct the circuit c and then simulate the circuit on x
jxj
S
On the other hand if L P for some sparse set S then the con gura
tion transition of the Turing machine can b e suitably co ded into a circuit
demonstrating the L has p olynomial size circuits Details of this can b e found
in
U Sch oning Complexity and Structure Springer
I Wegener The Complexity of Boolean Functions Teubner Wiley
J K obler U Sch oning J Tor an The Graph Isomorphism Problem Its
Structural Complexity Birkh auser
Self reducibility of SAT means that can nd a witness a satisfying
assignment for a formula F SAT by recursively testing other formulas
for memb ership in SAT in the following manner To determine the value
of a variable x in a satisfying assignment for F temp orarily assign it the
value This pro duces a new formula with fewer variables If this formula is
satis able then there is a satisfying assignment for F in which x else we
try x By this same recursive pro cedure we get a satisfying assignment
for the mo di ed formula with x or x This combined with x is
a satisfying assignment for F
Now we want to design a circuit that realizes this metho d Supp ose we
are given p olynomial size circuits for SAT We will use the following notation
for these circuits
H
H
H
H
F SAT
F
The thick lines indicate a bus of wires The input to this circuit is the formula
F co ded as binary string and the single output is if and only if F SAT
We also need another simply constructed p olynomial size circuit which
we will denote by
Solutions
a a
i
F
F
x a x a
i i
This circuit has n i inputs and WLOG n outputs The inputs to this
circuit are the enco ding of F and i additional bits a a The output is
i
an enco ding of the formula that results from substituting the values a a
i
into the formula F for the variables x x The details of this circuit
i
dep end on the enco ding used for formulas
The desired witness circuit is built as follows
F
H
H
H
y
H
H
H
y
H
H
H
y
Let x L Then for every y the formula f hx y i is satis able
Cho ose for c the appropriate p olynomial size witness circuit for SAT which
exists by the assumption and the previous exercise Then c will pro duce a
satisfying assignment for every input F f hx y i SAT
If y c f hx y i is a satisfying assignment for F f hx y i then
F is satis able regardless of how y came to b e
It is clear that all languages in BH i are contained in
i
BH For the reverse direction let L b e a language in BH Then there is a
nite expression that represents the application of the intersection union
and complement op erators used to build L We will show by induction on the
structure of this expression that L is in BH for some k By DeMorgan s
k
laws we can bring all complement op erations to the inside of the expression
so that we only need to consider intersection and union over NP and coNP
languages All NP and coNP languages are in BH Furthermore observe that
all languages in BH with k even can b e expressed in the form
k
A A A A A A A
k k
Solutions
where each A is a language in NP It only remains to show that the intersec
i
tion of any two such languages A and A can again b e expressed in this form
Now we can apply the distributive law to multiply out the expression and
use the equivalence
A A A A A A A A
i i i i
j j j j
and the fact that NP is closed under intersection and union to see that the
result has the desired form This result is due to Hausdor
Let M b e a probabilistic algorithm for L RP and let M b e a
L RP Then the following algorithm has the probabilistic algorithm for
desired b ehavior
INPUT x
Simulate M on x let the result b e y f g
Simulate M on x let the result b e y f g
IF y AND y THEN ACCEPT
ELSE IF y AND y THEN REJECT
ELSE OUTPUT don t know
In the other direction from an algorithm of the typ e given ab ove with
three p ossible outcomes accept reject or don t know we can get an RP
L by making the following mo di cations to the algo algorithm for L or for
rithm
In the rst case to show L RP don t know b ecomes reject
In the second case to show L RP don t know and accept b ecome
reject and reject b ecomes accept
Let L ZPP Then L can b e computed by a p olynomial time b ounded
probabilistic machine M as in the previous exercise We use this machine as
a sub program in the following machine M
INPUT x
REPEAT
y result of simulation of M on x
UNTIL y don t know
IF y accept THEN ACCEPT
ELSE REJECT END
Let b e the least probability that the machine M gives a de nite answer
accept or reject The exp ected numb er of passes through the rep eat until
lo op of M is then
X
i
i
i
Solutions
So the exp ected running time is p olynomial In fact exp ected running time
would still b e p olynomial even if were dep endent on n in such a way that
n
k
n
In the other direction let M b e the algorithm with exp ected p olynomial
running time p n
By Markov s inequality the probability that the running time is actually
more than p n twice the exp ected running time is at most otherwise
the slow running times would already force the exp ected running time to
b e more than p n So if we simulate M for p n steps we get an algorithm
of the typ e in the preceding exercise with a probability constant of
i e our new algorithm answers don t know whenever M has not answered
with accept or reject within p n steps
The probability that we fail to get acceptance t times provided the
t t n
input is in L is at most If we want we see that
n n
t is su cient
log
If we rep eat a Bernoulli trial t times indep endently where each of the
trials has a probability of success and let X b e a random variable that
counts the numb er of successes then the following approximations are valid
for r
r t r t
P r X t r e e
and
r t r t
P r t X r e e
since Derivations of these or similar approximations can b e
found in many places for example in
N Alon J H Sp encer The Probabilistic Method Wiley
H Cherno A measure of the asymptotic e ciency for tests of a hyp oth
esis based on the sum of observations Annals of Mathematical Statistics
Cormen Leiserson Rivest Introduction to Algorithms MIT Press
T Hagerup C Rub A guided tour of Cherno b ounds Information
Processing Letters
E Kranakis Primality and Cryptography Wiley Teubner
C Papadimitriou Computational Complexity Addison Wesley
E M Palmer Graphical Evolution Wiley
P E Pfei er Concepts of Probability Theory Dover
Solutions
Applying these inequalities in our case yields
t
X
t
i t i t t
P r X t t e
i
i
n
To get an error probability of we must cho ose t c n where c is
a suitable constant
The approximations in the case x L are similar
t
X
t
i t i t t
P r t X t e
i
i t
where
Deterministic p olynomial time computations with a xed input
length can b e transformed into p olynomial size circuits Details can b e
found in the b o ok by K obler Sch oning and Tor an If we x a fortunate
choice of the random variable z then our probabilistic algorithm b ecomes
a deterministic algorithm and we can apply the same principle to construct
p olynomial size circuits
For an explanation of self reducibility see Topic the b o ok by
K obler Sch oning and Tor an
Let M b e a BPP algorithm for SAT which we are assuming exists After
n
probability ampli cation assume the error rate is at most Now to show
that SAT RP we use the following probabilistic algorithm the only prob
abilistic part is the use of M as a sub routine The algorithm makes use of
an array a a of bit values
n
INPUT F x x F is a formula in the variables x x
n n
FOR i TO n DO
IF M F a a x x THEN a
i i n i
ELSE a END
i
END
IF F a a THEN ACCEPT
n
ELSE REJECT END
If F SAT then this algorithm will always reject If F SAT then with
n n
probability at least a satisfying assignment a a for F
n
is constructed by the algorithm which causes it to accept So this algorithm
demonstrates that SAT RP
r
Let x and y b e arbitrary distinct element from f g Then
P r h x h y
X
P r h x z h y z
s
z f g
Solutions
X X
P r ax b mo d p u ay b mo d p u
u
u p
s
p dp e
p
s
p p
s
p
s r
s
Here the last sum runs over all u p with u u mo d
Let h h H and x x X b e chosen uniformly at random We
need an upp er b ound for P r h h x h h x i e for P r h
h h x h x This is exactly jH j multiplied by P r h x h x
where h H and x x X are chosen at random If x x then this
s r
probability by the de nition of almost universal is at most
l l
Since jX j the probability for x x is at most So we can give
an upp er b ound of
s r l e
s
jH j jH j
Let X b e the set of sequences of length k with not more than k
s Let D b e a distribution that is similar to the p p distribution of
length k and let E b e the correct p p distribution We approximate
as follows
P r X P r X
D E
k
X
k
i k i
p p
i
i
k
k
So it must b e that
Let X b e an arbitrary set of strings Then
jP r t F X P r H X j
jP r t F X P r t G X j jP r t G X P r H X j
jP r t F X P r t G X j
jP r F t X P r G t X j
If for every language A C gilt the language fhx y i j x A y
g C then C BP C
Solutions
Let q b e an arbitrary p olynomial and let A b e a language in BP C So
for some language B C and some constants and
x A P r hx y i B
x A P r hx y i B
q n
In order to amplify the probability to reach an error rate of at most we
use the well known technique of replacing B with the language B describ ed
by the following algorithm
INPUT hx y y i
t
s
FOR i TO t do
IF hx y i B THEN s s END
i
END
IF s t THEN ACCEPT
ELSE REJECT END
Here t is a linear function in q jxj
What we need is that this algorithm is of typ e C so that B C Since
this program uses B as a sub program it is clear that B P B but that
characterization is to coarse since it would not for example apply with the
class NP in the role of C unless NP coNP If we lo ok more closely we see
that this reduction is actually positive or monotone This means that from
B B it follows that L M B L M B We will use the notation Pos
for this typ e of reduction so B Pos B P NP and many other classes are
closed under Pos
Let A BP BP C Then there are constants and and a language
B BP C such that
x A P r hx y i B
x A P r hx y i B
For B BP C we can apply probability ampli cation so there is a language
C C with
hx y i B P r hx y z i C
hx y i B P r hx y z i C
Putting this together we get
x A P r hx y i C
x A P r hx y i C
This shows that A BP C with the constants and
Solutions
Let L b e a language in Op BP C By the statement of the exercise
there is a language A BP C a p olynomial p and a predicate Q such that
x L Q x A
p n
where n jxj For the language A there is a language B C such that for
all y with jy j p n
P r y A hy z i B
where the probability is over z chosen uniformly at random Because of prob
p n
ability ampli cation we can cho ose Now it follows that
p n p n
P r y jy j p n hy z iB y A
From this it follows that
P r x L Q x B y
p n
where we are using B y to denote the set fx j hx y i B g So L BP Op C
n m
X f hG i j G is isomorphic to G or to G and Aut G g
Supp ose rst that the strings y and y di er in exactly one bit say in
the rst bit y y For every bit p osition j b in the result string
a
M
h y h y h y
ij i j j
i
and
a
M
h y h y h y
ij j j
i
i
L L
a a
where h y h y WLOG we may assume that
ij i ij
i
i i
y and y Then h y and h y m So h y or
j j j j
with probability for each Since the bits m are chosen indep endently
j
b
P r h y z h y z
This argument can easily b e generalized to the case where y and y di er
in more than one bit
Cho ose u u randomly We approximate the probability that
p n
then the rest of statement do es not hold
P r v u v E u v E
p n
X
P r u v E u v E
p n v
Solutions
p n
X Y
P r u v E
i
v
i
p n
X Y
n
v
i
X
n p n
v
p n n p n
n p n
Supp ose there is a choice of u u such that for every v there
p n
p n
is an i p n with u v F Partition the set of v s i e f g
i
according to the i s
p n
f g V V
p n
p n
F g For at least one j jV j p n From this where V fv j u v
j j j
p n p n
it follows that j F j p n and therefore that jF j p n
which is a contradiction to our assumption for large enough n
asm Lm Th
P
coNP BP NP BP NP BP NP
P
P is contained in P since a P machine is also a P machine
P is also closed under complementation since every P machine can b e
extended with one dummy accepting path transforming an o dd numb er of
accepting paths into an even numb er of accepting paths and vice versa
Let L b e a language in FewP as witnessed by a machine M that has at
most p n accepting computation paths A computation has length at most
q n so it can b e represented by a string of length q n Consider now the
following new non deterministic machine M
INPUT x
GUESS m f p jxj g
GUESS y y jy j q jxj with y y
m i m
IF all y represent accepting computations THEN ACCEPT
i
ELSE REJECT
END
If x L then machine M has an even numb er accepting paths on input
x On the other hand if x L then M has m accepting paths and M
P
m
m
m
has an o dd numb er accepting paths
i i
Solutions
AND function f hF F i F F Here F is a version
n
n
i
of the formula F with the variables renamed in such a way that no variable
i
o ccurs in more than one formula F
i
OR function f hF F i F F
n n
A NOT function is nothing other than a p olynomial time many one reduc
tion to the complement of the language In the case of SAT such a function
exists if and only if NP coNP
P is closed under P since BP It su ces to show that SAT BP
p olynomial time many one reductions Let M b e a probabilistic algorithm
that transforms input formulas F into formulas F with
F SAT P r F SAT
p jF j
F SAT F SAT
Now it su ces to show that for any there is a probabilistic algorithm
M that transforms F into F with
F SAT P r F SAT
F SAT F SAT
For this all that is needed is to let F consist of t indep endent formulas of
the form F i e F F combined using the OR function h for SAT
t
F h F F We must still determine how large t must b e chosen
t
dep ending on p and On input F S AT the probability of having no
successes F SAT in t trials is at most
i
t t
t p jF j
p jF j p jF j
d
p jF j p jF j
where d e This probability is supp osed to b e less than From this we
see that t log p jF j su ces In fact from this we see that can
q n
even b e a function of the form where q is an arbitrary p olynomial
The following language
A f F X Y j F is a b o olean formula and X and Y sets of
variables o ccurring in F such that for at least
one assignment of the X variables there is an
o dd numb er of assignments of the Y variables
with F X Y g
P It is su cient to show that A is probabilisti is clearly complete for
cally reducible to SAT This is achieved by the given probabilistic algorithm
which transforms F into F if we consider the halving pro cess of the algo
rithm to act only on the X variables and not the Y variables From this we get
Solutions
F X Y A with high probability there are an o dd numb er
of X assignments for which there are an o dd
numb er of Y assignments that make F
with high probability there is an o dd numb er
of X Y assignments with F
F X Y A there is no X assignment for which there is
an o dd numb er of Y assignments that make
F
there are an even numb er of X Y assignments
with F
If P PH then SAT PH Thus there exists a k such that
P P
SAT Since SAT is complete for P and is closed under many
k k
P
P From this we get PH BP one reductions it follows that P
k
P P P P
BP
k k k k
P P the following language is complete for BP P Since BP
f F X Y Z j F is a b o olean formula and X Y and Z are sets
of variables o ccurring in F such that for every X
assignment there is at least one Y assignment for
which there is an o dd numb er of Z assignments with
F X Y Z g
p p p p
a mo d b x N bx a x N b x a a
p
mo d b
On the other hand a mo d b x N bx a x
P
p
p
p p p i p
N bx a x N a bx x N a b
i
i
P
p
p
i i p
b x a mo d b
i
i
Every non deterministic computation tree can b e augmented with one
additional dummy accepting path The class P is therefore closed under
in fact under addition
By attaching two non deterministic computation trees one after the
other in the sense that the second computation is only started in the case
that the rst one ended in an accepting state we get a non deterministic
computation tree for which the numb er of accepting paths is precisely the
pro duct of the numb ers of accepting paths on the original trees This shows
that P is also closed under multiplication It is imp ortant to note that the
non deterministic running times merely add This means that to compute
p jxj
acc x on input x we can attach p jxj copies of M computations one
M
after another This results in a running time that increases by a factor of
p jxj which is still p olynomial provided M ran in p olynomial time
Solutions
If in some calculus there is p olynomially long pro of for the statement
x A then A NP On input x one can guess a p otential p olynomially
long pro of and verify that it is correct For this we need that pro ofs in
our calculus can b e veri ed in p olynomial time This will b e the case if for
example the individual steps in the pro of are easily checked for syntactic
correctness
If A is in NP then one can de ne the con guration transition calculus
corresp onding to the NP machine A t step pro of of x A consists of a
sequence of legal transitions that result in an accepting con guration
Let A b e computable by an interactive pro of system in the given
sense Then A NP since on input x one can simulate the computation of
the veri er and nondeterministically guess the communication of the prover
x A if and only if such a nondeterministic algorithm accepts x
Just as in the case of a nondeterministic or alternating Turing machine
the con guration transitions of the prover and veri er can b e viewed as a tree
structure with the start con guration as the ro ot Prover branchings are to b e
understo o d and evaluated as existential branchings and veri er branchings
which are used to generate random numb ers as randomized or probabilistic
branchings
Example
l
l
Veri er
l
l
Prover
9 9 9 9
Such a computation tree is evaluated as follows The accepting and re
jecting leaves receive the value or resp ectively The value of an existential
no de is the maximum of values of its children the value of a probabilistic
no de is the mean of the values of its children By the de nition of an interac
tive pro of system for the language A the ro ot must receive a value
if x A and if x A
The evaluation of such computation tree can b e done by a p olynomial
space b ounded machine in exp onential time by evaluating the tree using a
depth rst back tracking algorithm At no time do es this require more space
than that required to store one path through the tree Since the tree has
p olynomial depth and each no de can b e stored in p olynomial space this is a
PSPACE simulation
Solutions
Supp ose that A IP via a veri er V and a prover P who is resp onsible
for x A The prover can b e regarded as a function P x y y z
k
where x is the input y is the communication b etween the prover and the
i
veri er that o ccurred during a previous round i and z is the the communi
cation from the prover in the next round We want to show that A is provable
by an oracle
In place of the prover we use the oracle language
B fhx y y z i j P x y y z and z is an ini
k k
tial segment of z g
In place of of the veri er we have to describ e oracle Turing machine M M
will b ehave exactly like the veri er except that instead of writing a string
y on the communication tap e M systematically writes strings of the form
hx y y y z i where the y s are the previous communications on the
l i
oracle tap e and uses the oracle B to obtain the information z If x A then
B
it is clear that M exhibits the same probabilistic b ehavior as P V and
therefore accepts with probability Now supp ose x A and let B b e an
B
arbitrary oracle This corresp onds to some prover P and since M b ehaves
B
just like P V it will accept with probability at most
B
Now we will try to prove the reverse direction Supp ose A is provable by an
oracle A via the machine M and the oracle language B We want to show that
A IP In place of M we use a veri er V that b ehaves exactly like M except
that instead of querying the oracle ab out a string w w is communicated to
the prover The prover asso ciated with the oracle B is de ned by the function
P x y y w f g where P x y y w if and only if w
k k
B That is the prover ignores the input x and all previous communication
and simply tells the veri er whether or not the last string written on the
communication tap e was a string in B a single bit
If x A then it is clear that P V exhibits exactly the same proba
B
bilistic b ehavior as M and so accepts with probability If x A
however there is a problem Let P b e an arbitrary prover that is a func
tion P x y y z It is not clear how the prover can b e co ded up
k
into an oracle The reason is that an oracle unlike a prover has no mem
ory and therefore cannot make its answers dep endent up on the previous
communication It has in fact b een shown in
L Fortnow J Romp el M Sipser On the p ower of multi prover inter
active proto cols Proceedings of the th Structure in Complexity Theory
Conference IEEE
that a language A is provable by an oracle if and only if A NEXPTIME
This is p otentially a bigger class than IP PSPACE cf Topic
The probability is exactly if the prover always cho oses j f g
Otherwise the probability is p where p is the probability that the prover
cho oses j f g
Solutions
Change the line Accept if i j to Randomly select k f g
Accept if i j and k This has the e ect of translating the probabilities
to and
Instead of computing one random copy of one of the graphs H the
veri er generates k indep endent random copies H H and communi
k
cates these to the prover The veri er exp ects to receive k correct answers
j j back from the prover and only accepts if this is the case If the
k
graphs are isomorphic then the prover can evoke acceptance with probabil
k
ity at most
If the input graphs G and G are isomorphic then the graph H gen
erated by the prover is also isomorphic to b oth graphs and the prover using
his unlimited computational p ower is able to compute the isomorphism
So in this case and with this prover the probability of acceptance is
But if the graphs G and G are not isomorphic then no prover can
pro duce the required isomorphism more than of the time since in half
the cases no such isomorphism will exist By means of the usual techniques
this probability of can b e further reduced to ful ll the de nition of IP
The desired machine M works as follows
INPUT G G
GUESS RANDOMLY j f g
GUESS RANDOMLY S
n
S is the set of p ermutations of f ng
n
OUTPUT G j
j
The distribution of triples generated by this program is the same as the
distribution that can b e observed in the given proto col on an input G G
GI The rst comp onent of this triple is a graph H isomorphic to G or G
chosen under the uniform distribution the second comp onent is a numb er j
f g that is uniformly distributed and indep endent of the rst comp onent
and the third comp onent is an isomorphism b etween G and H
j
The proto col can b e followed word for word except that at the p oint
where the prover is required to determine so that G H instead the
j
prover uses the extra information namely with G G to compute
in p olynomial time as follows
if i j
if i j
if i j
if i j
By DeMorgan s laws one can systematically push all of the negation
symb ols to the inside until they are all in front of variables
Solutions
F G F G
F G F G
x F x F
x F x F
For the variables this is true since only the values and are sub
stituted The use of x for negation turns a into a and vice versa so
i
this is also correct This is why it is imp ortant that negation only o ccurs at
variables Now consider two formulas F and G for which the claim is true
Then one can easily check that multiplication and addition b ehave correctly
for F G and F G Similarly if we have a formula of the form Qx F
Q f g and substituting a or into F for x pro duces a correct arith
metic value for the corresp onding formulas with the variable x replaced by
TRUE or FALSE then multiplication will correctly evaluate and addition
will correctly evaluate
Since the variables x do not o ccur in the in the subformula y z y z
i
this subformula can b e evaluated directly its value is Each application of
m
a universal quanti er squares the value so in the end the value is
We use induction on the length of formulas The largest value that
can b e assigned to a variable or negated variable is For a formula of
length n of the form F G f g the formulas F and G will by
l r
the inductive hyp othesis b e assigned values and with r l n
l r l r n
The value for F G is then at most In the case of a
m
formula Qx F Q f g if the value of F is at most with m n then
m m m m n
value for F is at most
p
n n
n
prime In the interval in question there are at least
n
numb ers Let these b e p p p k If for all i k a mo d p
k i
Q
k
p Since then by the Chinese Remainder Theorem a mo d
i
i
Q Q
n n
k k
n
p and since a it follows that a Con
i
i i
tradiction
Consider the set of numb ers from to n Half of them are not prime
since they are divisible by Of the rest remain if we strike the third that
are divisible by After the next step of the remaining prime numb er
candidates are stricken b ecause they are divisible by If we continue this
p
pro cess until we reach the largest prime numb er less than n then only
the prime numb ers b etween and n will remain The numb er of candidates
remaining is thus exactly n If we express this in formulas letting p run
over prime numb ers we get
p
nc b
Y Y
p p
i p
nc n n n b n n
p i
p
i
n p
Solutions
By the argument ab ove in Exercise it follows that there are at
n n
most prime numb ers p with p for which a mo d p This means
n
that in order to arrive at the desired error probability of we need an
n n n n
interval starting at that contains at least primes If we use the
sharp er approximation n n ln n we see that it su ces to cho ose the
n n
same interval as b efore namely
The p olynomial p x p x also has degree at most d Since the
p olynomial is over a eld it can have at most d zero es The probability of
randomly selecting one of these at most d zero es from the set z f k g
n
is therefore at most d k d
Note that we can extend the de nition of p to include the case where
G
G has more than one free variable in which case the p olynomial p will also
G
have additional variables We are only interested however in the degree of
one variable x OR and op erations corresp ond to addition so they don t
increase the degree of x If F H K then p p p so the degree of
F H K
x in p is at most the sum of the degrees of x in p and p If F y H
F H K
then the degree of x in p is twice the degree of x in p but this is nonzero
F H
only once for a given variable Thus for subformulas H of G the degree of x
in p is b ounded by jH j if H do es not contain the universal quanti er that
H
doubles the degree and and by jH j if it do es
Every subformula in F of the form Qx y H x where Q
f g is replaced by the following equivalent formula
Qx x x x y H x
That is a new variable x is intro duced its equivalence to x is con rmed
and then it is used in place of x The equivalence symb ol can of course b e
expressed using AND OR and NOT
P r error P r no error
n
Y
P r no error in round i
i
n n
n
A
Let A b e PSPACE complete Then PSPACE P PSPACE and by
A
a Savitch s Theorem PSPACE NP NPSPACE PSPACE
n
A xed blo ck contains only strings in A with probability so the
n
n
probability is that none of the blo cks consists solely of strings in
n
n
A Thus the probability of at least one such blo ck is which
approaches e as n gets large
Solutions
On input x jxj n nondeterministically guess a numb er i in
n
f g and verify that all strings in the ith x blo ck are in the oracle
The pro of follows the line of argumentation used in the case of P
versus NP Let N N b e an enumeration of all NP oracle machines then
A A
L A NP P r NP coNP P r
A
P r i L A L N
i
X
A
P r j x L A L N
j
i
i
X Y
A
P r x L A L N j C
j
i
i j
So it su ces to show that for some
A
C P r x L A L N j
j
i
or equivalently that
Now consider the cases and If then we are
done If then from we can conclude that
and since it follows that
A sup erconcentrator is given by the following graph
m m
An n sup erconcentrator can b e made by putting together copies of an
n sup erconcentrator S
h h
Z Z
Z Z
S
Z Z
Z Z
h h
Z Z
Z Z
Z Z
Z Z
Z Z
h h
Z Z
Z Z
Z Z
S
Z Z
h h
Z Z
The size g n of this graph satis es g n g n O n so g n
O n log n
This graph is a sup erconcentrator in fact it is a permutation network
which means that for any arbitrary p ermutation of f ng a no de
disjoint connection can b e found that realizes this p ermutation b etween
input and output no des Let f ng f ng b e a p ermutation
We can assign to a bipartite graph G with n no des partitioned into
two pieces of size n each such that for every i j with i j an
Solutions
undirected edge is drawn b etween the i mo d n th left no de and the
j mo d n th right no de In this way every no de has degree multi edges
are p ossible This bipartite graph consists of one or more disjoint cycles of
even length These can b e colored red and blue in such a way that adjacent
edges always receive di erent colors We assign all red edges to the upp er
n p ermutation network and all blue edges to the lower These are each
p ermutations of n which by the induction hyp othesis can b e realized by
the network S
For the size numb er of edges g n of an n sup erconcentrator we
obtain the recursion g n g n dn where d is a constant and g
From this we get g n O n
Let k n input no des S and k output no des T b e xed Let S b e
those no des in S that corresp ond to p ositions that also o ccur in T so that
they can b e connected directly Let T b e the set of corresp onding p ositions
in T The inputs in S S are routed through G Since k jS S j n
these k inputs to G can b e connected to some k outputs of G inputs
to S The size of T T is also k so for this set of outputs of G there
is some set of k inputs that can b e connected by no de disjoint paths The
sup erconcentrator S connects these two sets of no des This shows that the
sets S and T can b e connected by no de disjoint paths
If for some subset S S jS j jN S j then at least one no de of S
cannot b e connected
In the other direction supp ose that jS j jN S j and let M b e a match
ing that do es not match the no de u S Then jN fug j and the no des
in N fug are already blo cked by other no des Supp ose v N fug and
u v M for some no de u u Then jN fu u g j So there is at
least one other no de v N fu u g so that the matching M can p erhaps b e
altered so that the edges u v and u v are in M instead This will only
fail if v is already matched with some u in M But now jN fu u u g j
so we can pro ceed analogously obtaining a new no de v and hoping to re
arrange the edges of M so it includes u v u v and u v This will
only fail if M already matches some u with v
But we cannot fail forever At the latest when we have used all the no des in
S we must must nd an unmatched no de that allows for the rearrangement
m
For k there are edges to b e connected on one side but
only p ossibilities on the other side so such a matching can only
happ en if k
More generally we can show that if n n n k k k n k
n n n n
Note that is the numb er of ways to and n k then
k k k k
cho ose k of n p eople to o ccupy k available chairs We are only interested
in who gets a chair and who do esn t not who gets which chair Similarly
Solutions
n n
is the numb er of ways to seat k of n p eople with the added restriction
k k
that the rst k chairs are o ccupied by a subset of the rst n p eople and
the remaining k chairs are o ccupied by a subset of the remaining n p eople
Clearly this is a smaller numb er
We compute L L canceling redundant terms and grouping as
k k
follows
k
k
k m
L L
k k
k
k
k m
k k
k k m k m k
k k k
k k
k k k
m k m k
L
k
is monotone in Each of these factors is monotone increasing in k so
L
k
so L is convex creasing in k from which it follows that L L L
k k k
k
If we let k then
m m
m
m
m
If we let k m then
m m
m m
m m
m m
m m
m
m m
m
m
m m
m m
m m
m
m
m
m m
m m
m
m m
m m
m m
m
m
m
A pyramid can b e p ebbled level by level For the lowest level we need
k p ebbles Then we need one more to b egin the second level From that p oint
on we can reuse p ebbles to move up the graph
Solutions
We prove the following by induction on k
Every p ebbling strategy starting at any initial con guration of p eb
bles on P that leaves at least one path from the input no des to the
k
output no de p ebble free requires at least k p ebbles
The case k is clear Now let k and let an arbitrary initial con guration
with at least one p ebble free path b e given Without loss of generality we
may assume that there is a p ebble free path that passes through the no de v
v
e
So
S
f f
v v
J
J
P
k
J
J
J
J
J
P
J
k
J
J
J
J
Now supp ose there were a p ebbling strategy for the output no de v that
used k or fewer p ebbles This strategy would include a strategy for p ebbling
v which by induction would require the use of all k p ebbles At the p oint in
the game where all k p ebbles are on graph P the path down the right edge
k
of P is p ebble free In particular to p ebble v in the copy of P at the
k k
right edge of P will again require all k p ebbles which means that we must
k
remove the p ebble from v Thus with k p ebbles it will not b e p ossible to
cover v and v simultaneously so we will need at least k p ebbles to cover
v Only k p ebbles would b e necessary if we allowed sliding of p ebbles
which would corresp ond to using only two registers to compute things like
x x y
For every no de u with in degree d replace the subtree consisting
of that no de and its predecessors with a binary tree with d leaves assigned to
the original predecessors of u The numb er of p ebbles required to p ebble the
graph remains the same it just takes more time since once the d original
predecessors of u have b een p ebbled it takes only one additional p ebble to
cover u in the tree just as in the original graph see the previous exercises
Clearly the in degree of each no de will now b e and the numb er of edges
will have at most doubled
We start with G and G G and successively move from G
to G no des along with their in coming edges that already have all of their
predecessors in G In this way there cannot b e any back edges in A This
pro cess is continued until at least n edges are in G Since at each step
at most new edges enter G the numb er of edges in G will b e at most
n
Solutions
Case is trivial since it only covers nitely many cases
Case P n P n n log n n log n
For case we have
n
P n P n
log n
n n
c
log n log n
n n
c
log n log n
n n
c
log n log n
n
c
log n
when c and n is large
In case apply n n log n n to get
P n P n n log n P n
n n log n n
c c
log n n log n log n
cn log n cn c cn
log n log n log n log n
cn cn cn
log n log n log n log n
a
to get Now apply the equation
x a x x x a
cn cn cn
P n
log n log n log n log n log n
cn cn
log n log n log n
cn cn cn
log n log n log n log n
Under the assumption that all context sensitive languages can b e
recognized in linear time we get
NSPACE n DTIME n
DTIME n log n Time Hierarchy Theorem
DSPACE n
NSPACE n
Solutions
which is a contradiction
We build up a set of n j inputs with the desired prop erty succes
sively as follows Let E b e an arbitrary j element subset of the inputs
E is connected to A via j no de disjoint paths by the de nition of su
p erconcentrator of which at least one must b e p ebble free Let e b e the
corresp onding input no de Build E E fe g feg for any input e that
has not yet b een considered Once again there is at least one p ebble free path
with corresp onding input e E Rep eated application of this pro cedure
leads to a set of n j inputs with the desired prop erty
n n
jC n j so jG j jG j This recursion relation
n n
n
has the solution n
G includes among other things direct connections b etween the
n
inputs and outputs If we just consider these edges it is clear that the com
bination of G and C n is at least as go o d as C n alone so it must also b e
n
n
a sup erconcentrator
Consider a strategy that takes a total of t steps on G and starting
from a con guration in which at most no des are p ebbled p ebbles
outputs By Lemma of these outputs are connected with at least
outputs via p ebble free paths So there must b e one of these outputs that is
connected to at least of the inputs via p ebble free paths Let a b e this
output and let t b e the last time at which all of the inputs were still
unp ebbled Then the statement of the theorem is satis ed for the interval
t t since in this time interval inputs need to b e p ebbled and
at least one p ebble remains on the graph otherwise t would not have b een
the last time the inputs were all unp ebbled
n
We have a graph family fG g with jG j n edges In order
n n
to p ebble a certain subset of the outputs in any order we need at least
n n
c p ebbles for some c So there must b e an output that requires c
p ebbles else one could p ebble all the outputs with fewer p ebbles by p ebbling
one after the other each time removing all of the p ebbles use
So P m the required numb er of p ebbles satis es
n
P m c c m n c m O log m m log m
where m is the size numb er of edges of G
n
Consider the following P computable distribution de ned on tuples
hx a bi where x is a CNF formula a an assignment to the variables in x
and b a bit
if b is the truth value for x under assignment a
jaj
jxj
hx a bi
otherwise
Solutions
Clearly this is P computable Furthermore x x if and
only if there is a satisfying assignment for x Thus if is P computable
then SAT NP so P NP
m m
so
m m m m m m
N
X
as N
m m N
m
More generally
N
N
X
k k
k
m m
m
as N
k
k k
k N
Let f b e de ned by
n n
if x
f x
otherwise
Then
n n
X
f x
n n
jxj n
but
n n n
X
f x
n
n n n
jxj n
Closure under maximum is demonstrated by
X X X
max f x g x f x g x
x x x
jxj jxj jxj
jxj f x g x f x g x
X X
f x g x
x x
jxj jxj
x x
Clearly we have closure under Closure under sum is then obtained by
noticing that f g max f g Closure under exp onentiation is obtained by
cho osing an appropriately adjusted value for Closure under multiplication
then follows b ecause f g f g
Supp ose the exp ected value of f is p olynomially b ounded with resp ect
to some distribution Then there are constants c and k such that
P
k
f x x cn Let k then for all n
n
jxj n
Solutions
k
X X
f x f x
x x
n n
jxj jxj
jxj n jxj n
X
f x
x
n
k
n
jxj n
X
f x x
n
k
n
jxj n
c
So
X X X X
f x f x
n
x x c c
jxj jxj
n n
jxj n jxj
Consider the following algorithm
Input a graph G
Search G for a copy of K by checking each set of four vertices to see
whether all six edges are present in the graph If a copy of K is found
reject
Search for a coloring by assigning each p ossible coloring to the no des
and checking if two adjacent no des have the same color If a coloring
is found accept else reject
The running time of this algorithm is p olynomially b ounded on instances
n
that contain a copy of K since there are only n p otential copies of
K and each can b e checked in p olynomial time Let the p olynomial b ound
in this case b e p n
If there are no copies of K in G then the algorithm must p erform step
n n
There are p otential colorings so this step takes at most q n time
for some p olynomial q
Now we need a b ound on the probability of nding a copy of K in G
Given any set of four no des the probability that they form a copy of K is
Thus we can give the following rough approximation for the
probability that no K is found
n
P r G contains no K
k
Let c and k b e integer constants such that cn max p n q n
k
Let A b e the set of graphs that contain no copy and
of K By the closure prop erties of p olynomial on average it is su cient
k
to show that g x f x cjxj is p olynomial on average The following
sequence of inequalities shows this to b e the case
Solutions
k
jxj
k
X X X
g x
x x x
jxj jxj
x
x A
x A
X
n k n
A
n
X
n k n
n
X
n
k
n
X
n
n
P P
Supp ose f A B and g B C Clearly
m m
P
h g f A C so it su ces to show that Let p b e a p olynomial
h
m
and let and b e distributions such that
jf x j p jxj
jg y j p jy j
x p jxj x
y p jy j y
P
x and y
f x y
P
x z
g y z
P
as guaranteed by the de nition of For ease of notation we are assuming
m
range f and range g so no scaling is needed The argument
can b e easily mo di ed if this is not the case It follows that
X
y z
g y z
X
p jy j y
g y z
X X
p p jxj x
g y z f x y
X
p jxj p p jxj x
h x z
P
x and x so we can de ne a distribution such that z
h x z
p p jxj p jx x thus
h
P
Supp ose f A B and B AP As in the previous
m
exercise for ease of notation assume that range f The obvious
algorithm for determining whether x A pro ceeds by rst computing f x
and then checking whether f x B Since f is computable in p olynomial
Solutions
k
time there are constants c and k such that jf x j cjxj Since
B AP there is an algorithm for B that runs in time t where t is
p olynomial on average witnessed by some exp onent The running of
k k
our algorithm for A is at most cjxj t cjxj By the closure prop erties of
k
p olynomial on average it su ces to show that h t cjxj is p olynomial on
average
k
First we consider h
k k
X X X
t y h x
x x
k
jxj
jy j
y x
f x y
k
X
t y
y
k
jy j
y
X
t y
y
jy j
y
X
t y
y
jy j
y
k
so h is p olynomial on average This implies that h is p olynomial on
average which implies that h p is p olynomial on average and thus
h h p p is p olynomial on average So A AP
n
If x then the rst di erence in the binary representations
of x and x must come at or b efore the nth digit So let z b e the
x
longest common pre x of the binary representations of x and x
and de ne co de x by
jxj
x if x
co de x
z otherwise
x
Now we verify the three prop erties co de is required to have
E ciency Since is p olynomial time computable so is co de
Uniqueness If x then
z x z x z
x x x
So x is uniquely determined by either x or z
x
jxj
Compression If x then j xj jxj log On the
x
jxj
other hand if x then jz j jxj and
x
jz j
x
z x z
x x
so jz j min jxj log
x
x
Solutions
By the de nition of D HALT
c
q jxj
M co de x
A
jco de x j
q jxj jco de x j
where c is a constant dep ending only on A and co de By Lemma
jco de x j log x so jco de x j log x which means that
x
jco de x j
Thus
c x
q jxj
M co de x
A
q jxj jco de x j
c
jco de x j
q jxj
j M co de x j
A
from which it follows that
f
Supp ose that a graph G with n no des is enco ded using an adjacency
matrix Then the size of the enco ding x for G is n The weight of such a
n jxj
graph is given by n
The following is a sketch of a solution but it can b e easily formalized
k
The enco ding of hM x i takes ab out jM j jxj k bits So if we x M and
x and let k vary the enco ding takes k bits and the weight is k But
k
for every k when k is large enough
p jxj
x B can b e determined nondeterministically in time jx j
N N N
N
N N
m N
For ease of notation we will assume that each no de in the tree has only
two children the general case follows by a similar argument For any no de
v in the tree let p b e the probability lab eling the edge to the rst child
v
p is then the probability lab eling the edge to the other child
v
We prove that the sum of the probabilities at any level k is by induction
on k When k there is only one no de and it has probability since the
empty pro duct is Now supp ose that the sum of the probabilities at level
k is Let q b e the probability of a no de v on level k Then the sum of the
v
probabilities of the no des on level k is
X X X
q p q p q p p q
v v v v v v v v
v v v
where the sums are over all no des on level k
Solutions
t
If the entries in M are real M M The fact that the rows columns
t t
are orthonormal is just another way of expressing that MM M M I
t
since the values in MM are just the dot pro duct of two columns and the
t
values of M M are the dot pro duct of two rows This implies the stated
equivalence
Note that in the following example the probability is for every state
after two steps even though the probability of going from any parent no de
to either of its children is
p p
A A
A A
p p p p
A A
A A
A A
A A
jai jbi jai jbi
With an appropriate lab eling of the internal no des of this tree this corre
p
sp onds to a matrix which is not unitary
p
will work M
P
n
p
jc i W jc i
i
i
P
n i j
p
W jc i jc i where i j is the dot pro duct of the
i i
j
strings i and j This is b ecause a negative sign is intro duced exactly when a
bit is b oth b efore and after the transformation
Since D W FW the e ect of D on jc i is
i
X
W
n i a
p
jc i jc i
i i
a
X
F
n i a
p
jc i jc i
a
a
X X X
W
n n i a a j
jc i jc i
j j
j j
a
Solutions
so
X
i a a i
D
ii
N N
a
X
A
N
a
N N
N N
and
X
i a a j
D
ij
N N
a
X
i i i a a j
N
a
N N
t
Note that D D so it su ces to show that D I Clearly I I
P
N
N
So N N and P P since P
ij
i
N N
D I P I P P I P P I
t
For any vector x x x x P x is a row vector each
N
P
n
comp onent of which is equal to x N so each comp onent of P x is the
i
i
average of the values in x Thus D x I P x x P x P x x P x
is the inversion ab out the average
p
p
N
p
The base case is trivial using sin N and cos
N
Now supp ose that we have shown that the equation is correct for k and l
j j
It is p erhaps most natural to verify the values of k and l are as stated
j j
p
N
and by working backwards Let j Then using sin
N
N
and a tirgonometric identity for sin we see that cos
N
sin j sin
sin cos cos sin
p
p
N N
k N l
j j
N N
N N
l k
j j
N N
The argument for k is similar
j
Solutions
If a jT j N and the algorithm randomly picks an element to query
rep eatedly until a target is found then the exp ected numb er of queries is
i
a a a a a i a a
X
i
i a a
i
i
X X
i
a a
i j
X X
i
a a
j i j
j
X
a
a
a
j
X
j
a
j
a
So if jT j N the exp ected numb er of queries is Even if the algorithm
is mo di ed so that no query is made more than once the exp ected numb er
of queries is nearly when N is large
Assuming the algorithm makes a new query each time it could take
n
N jT j queries
Bibliography
M Aigner Combinatorial Search Wiley Teubner
N Alon J H Sp encer The Probabilistic Method Wiley
K Amb os Spies S Homer U Sch oning ed Complexity Theory current re
search Cambridge University Press
M Anthony N Biggs Computational Learning Theory Cambridge Univer
sity Press
J L Balc azar J Diaz J Gabarr o Structural Complexity I nd edition
Springer
J L Balc azar J Diaz J Gabarr o Structural Complexity II Springer
L Banachowski A Kreczmar W Rytter Analysis of Algorithms and Data
Structures Addison Wesley
J P Barth elemy G Cohen A Lobstein Complexit e Algorithmique et prob
l emes de communications Masson Paris
R Beigel Class Notes on Interactive Pro of Systems Yale University Tech
nical Rep ort YALEU DCS TR
R Bo ok ed Studies in Complexity Theory Pitman
D P Bovet P Crescenzi Introduction to the Theory of Complexity Prentice
Hall
W S Brainerd L H Landweb er Theory of Computation Wiley
G Brassard P Bratley Algorithmics Theory Practice Prentice Hall
T H Cormen C E Leiserson R L Rivest Introduction to Algorithms MIT
Press McGraw Hill
F R Drake S S Wainer ed Recursion Theory its Generalizations and Ap
plications Cambridge University Press
P Dunne The Complexity of Boolean Networks Academic Press
P Dunne Computability Theory concepts and applications Ellis Horwo o d
R W Floyd R Beigel The Language of Machines An Introduction to Com
putability and Formal Languages Computer Science Press
Bibliography
M R Garey D S Johnson Computers and Intractability A Guide to the
Theory of NP Completeness Freeman
A Gibb ons W Rytter E cient Paral lel Algorithms Cambridge University
Press
A Gibb ons P Spirakis ed Lectures on Paral lel Computation Cambridge
University Press
E M Gurari An Introduction to the Theory of Computation Computer Sci
ence Press
J Hartmanis Feasible Computations and Provable Complexity Properties
So ciety for Industrial and Appl Math
J Hartmanis ed Computational Complexity Theory American Math So ci
ety
L Hemaspaandra A Selman ed Complexity Theory Retrospective II
Springer
R Herken ed The Universal Turing Machine A Half Century Survey Kam
merer Unverzagt und Oxford University Press
J Hertz A Krogh R G Palmer Introduction to the Theory of Neural Com
putation Addison Wesley
J Kilian Uses of Randomness in Algorithms and Protocols MIT Press
D E Knuth The Art of Computer Programming Vol Semi Numerical Al
gorithms Addison Wesley
R L Graham D E Knuth O Patashnik Concrete Mathematics A Foun
dation for Computer Science nd edition Addison Wesley
J K obler U Sch oning J Tor an The Graph Isomorphism Problem Its
Structural Complexity Birkh auser
D Kozen The Design and Analysis of Algorithms Springer
E Kranakis Primality and Cryptography Wiley Teubner
L Ku cera Combinatorial Algorithms Adam Hilger
J van Leeuwen ed Handbook of Theoretical Computer Science Volumes A
B Elsevier MIT Press
H R Lewis C H Papadimitriou Elements of the Theory of Computation
Prentice Hall
M Li P Vit anyi An Introduction to Kolmogorov Complexity and its Appli
cations nd edition Springer
C Lund The Power of Interaction MIT Press
M Machtey P Young An Introduction to the General Theory of Algo
rithms North Holland
C Meinel Modi ed Branching Programs and Their Computational Power
Springer Lecture Notes in Computer Science
Bibliography
S Micali ed Advances in Computing Research Vol Randomness and
Computation JAI Press
M L Minsky S A Pap ert Perceptrons MIT Press
B K Natara jan Machine Learning A Theoretical Approach Morgan Kauf
mann
P Odifreddi Classical Recursion Theory North Holland
C Papadimitriou Computational Complexity Addison Wesley
J Parb erry Paral lel Complexity Theory Pitman Wiley
M S Paterson ed Boolean Function Complexity Cambridge University
Press
H Rogers Theory of Recursive Functions and E ective Computability
McGraw Hill Reprint by MIT Press
G Rozenb erg A Salomaa ed Current Trends in Theoretical Computer Sci
ence World Scienti c
G Rozenb erg A Salomaa Cornerstones of Undecidability Prentice Hall
A Salomaa Computation and Automata Cambridge University Press
A Salomaa Public Key Cryptography Springer Verlag
C P Schnorr Rekursive Funktionen und ihre Komplexitt Teubner
U Sch oning Complexity and Structure Lecture Notes in Computer Science
Springer
A L Selman ed Complexity Theory Retrospective Springer
J R Sho en eld Mathematical Logic Addison Wesley
A Sinclair Algorithms for Random Generation and Counting Birkh auser
M Sipser Lecture Notes Computation by Automata MIT
R I Soare Recursively Enumerable Sets and Degrees Springer
R Sommerhalder S C van Westrhenen The Theory of Computability Pro
grams Machines E ectiveness and Feasibility Addison Wesley
H Straubing Finite Automata Formal Logic and Circuit Complexity Birk
h auser
K Wagner G Wechsung Computational Complexity VEB Deutscher Ver
lag der Wissenschaften und Reidel Verlag
O Watanab e ed Kolmogorov Complexity and Computational Complexity
Springer
I Wegener The Complexity of Boolean Functions Wiley Teubner
D Welsh Codes and Cryptography Oxford University Press
D Welsh Complexity Knots Colourings and Counting Cambridge Univer
sity Press
A Wigderson Lecture Notes Barbados Workshop on Complexity
Bibliography
Index
P
see Turing reduction arithmetic formula
T
P
arithmetic hierarchy
see many one reduction
m
arithmetization of a formula
see Turing reduction
T
Arthur
see Turing reduction
T
Arthur Merlin game
Arvind
Aspnes
Asser
f
0
atomic formula
automaton
average case complexity
n see Prime Numb er Theorem
b
rising p ower a
Babai
b
a falling p ower
back tracking
A
P
Baker
A
NP
Balc azar
P
Banachowski
i
P
Barth elemy
i
N
Barwise
R
basis
Z
BDD see binary decision tree
reducibility
Beigel
Bell
abacus
Benio
AC
Bennett
AC reducible
Berman
acc
Berman Hartmanis Conjecture
M
acyclic graph
Adleman
Bernoulli exp eriment
advice
Bernoulli trial
Aho
Bernstein
Aigner
Berry Paradox
Aleliunas
Berthiaume
Allender
Bhatt
Alon
Biggs
alphab et
Biham
Amb os Spies
binary decision tree
amplitude ampli cation
binomial co e cients
AND function
binomial distribution
Angluin
Binomial Theorem
Anthony bipartite graph
AP Biron
Index
Blum collision probability
Blumer communication tap e
Board comparable
Bo ok
complement
b o olean circuit
closure under
b o olean formula
complete graph K
n
b o olean function
complete language
b o olean hierarchy
P for BP
Bo olos
for coNP
Boppana
for DistNP
b ound variable
for P
Bovet
for NL
Boyer
for NP
BP op erator see counting classes
BQP
for P
Brainerd
for PSPACE
branching program
completeness
Brassard
complex clause
Bratley
compressible
Bshouty
computably isomorphic
Buntro ck vi
computation no de
bus
computation tree
concept
conditional Kolmogorov complexity
Cai
Celoni
conditional probability
Chaitin
con guration
Chandra
conjunctive normal form
Chang
Chebyshev s inequality
connected graph
Cherno
coNP see counting classes
Chinese Remainder Theorem
consistent
constant depth circuits
chip
context sensitive language
Chor
Christen
context free language
Chung
convex function
circuit
Co ok
b o olean
Co ok Hyp othesis
p olynomial size see p olynomial size
Co ok s Theorem
circuits
Cormen
size of
counting argument
circuit complexity
counting classes
circuit depth
BP NP
Claus vi
BP op erator
clause
BPP
Cleve
C P
CNF see conjunctive normal form
FewP
Cohen
MOD classes
Coles viii
NP
collapse
of BH co NP
of PH P
collapsing degree P
Index
PP Drake
Dunne
ZPP
Dykstra Pruim viii
C P see counting classes
Craig
e mail
Craig Interp olation Theorem
easy formula
Crescenzi
easy hard argument
critical
Ehrenfeucht
critical assignment
Ekert
crossing sequence
Emde Boas van
cryptology
Enderton
cut and paste
entropy
cycle
equivalence problem
for branching programs
D COL
for circuits
Dahlhaus
for context free languages
Davis
for LOOP programs
decidable
for tournaments
degree
equivalence relation
of a language
evaluation vector
of a no de in a graph
example set
of a no de in graph
exp ected value
of a p olynomial
exp onential running time
expression tree
total
Demarrais
factoring
DeMorgan s laws
Fagin
failure
density function
falling p ower
depth
fan in
of a circuit
Feigenbaum
depth rst
Fenner
descriptive complexity
FewP see counting classes
Deutsch
Feynman
D HALT
Fiat
D HAM
Fib onacci sequence
diagonalization
nite eld
Diaz
nite injury
di usion transformation
nitely presented group
dimension
rst order
of a vector space
at distribution
Diophantine equation
Floyd
Diophantus
formal language
directed graph
formula
disjunctive normal form
arithmetic
b o olean
DistNP
predicate logic
distribution function
quanti ed b o olean
distributional problem
satis able
divide and conquer
size
DNF see disjunctive normal form
Fortnow
DNF
Fortune
n k
dot pro duct
Fourier representation
downward separation Fourier transformation
Index
free variable Hertrampf vi
Hertz
Friedb erg
Hilb ert
Furst
Tenth Problem
Ho mann
G odel
Homer viii
Gabarr o
Hop croft
GAP
Horn formula
Garey
Huang
generalized sp ectrum
Hunt
Gergov
hyp othesis
GF n
hyp othesis space
GI see graph isomorphism
H astad
Gibb ons
Gill
Immerman
Goldreich
Impagliazzo
in degree
Goldsmith viii
inclusion problem
Goldwasser
inclusion exclusion principle
GOTO program
Incompleteness Theorem see G odel
Gr adel
indep endent events
Graham
index of an equivalence relation
grammar
inductive counting
graph
initial con guration
acyclic
input no de
bipartite
interactive pro of systems
connected
graph accessibility see GAP
interp olant
graph coloring
interp olant complexity
graph isomorphism
Interp olation Theorem
Grassl
interpreter
greedy algorithm
inversion ab out the average
Green viii
IP
Grover viii
isomorphic
Gundermann
computably
Gurari
Isomorphism Conjecture
Gurevich
isomorphism degree
H yer
Israeli
Hack
Hagerup
Je rey
Haken
Johnson
halting problem
Jones
Jozsa
Hamiltonian circuit
hard formula
K obler vi
Hartmanis
hash function
Kadin
Hausdor
Kannan
Haussler
Karg vi
Hemachandra
Karp
Hemaspaandra
Kautz viii
Herken Keisler
Index
Kilian via Kolmogorov complexity
Lund
Knuth
Ko
Kolmogorov Machtey
Mahaney
Kolmogorov complexity
ma jority
ma jority vote
Kolmogorov random
Makowsky
Kozen
Manders
Kranakis
many one degree
Krause
many one reduction
Kreczmar
P
Krogh
for distributional problems
m
Ku cera
P
p olynomial time
m
Kuro da
Marja
Kurtz
Markov chain
Markov s inequality
L
Lagrange
Marriage Theorem
Landweb er
matching
language
Matijasevi c
Lautemann
McCullo ch Pitts neuron
law of large numb ers
Meinel vi
LBA see linear b ounded automoton
memory
LBA problem
Merlin
rst
Meyer
second
Micali
learning
learning algorithm
Milterson
Leeuwen van
Minsky
Leftover Hash Lemma
MOD classes see counting classes
Leiserson
mo del
Levin
monomial
Lewis
monotone reduction
lexicographical order
move
Li
Muchnik
Lidar
multi linear p olynomial
linear b ounded automaton
Mundici
linearly indep endent
museum
Lipton
literal
Natara jan
k
Lobstein
NC NC
log reducible
log neural net
Lolli
neuron
Longo
Nisan
Longpr e viii
NL
lo op depth
nondeterminism
LOOP program
NOR
Lorenzen
NOR function
Lov asz
NOT function
lower b ounds
NP see counting classes
for branching programs
NP P computable see DistNP
for circuits
NP search P samplable
for resolution pro ofs NQP
Index
observation Pitt
POL
Occam algorithm
POL
Occam s Razor
Polish notation
Odifreddi
Pollett viii
Ogiwara
p olynomial
one one reduction
p olynomial on average
p olynomial time
p olynomial size circuits
one time only
one way function
p olynomial time hierarchy see PH
cryptographic
Pos see p ositive reduction
OR function
p ositive reduction
oracle results
Post corresp ondence problem
oracle Turing Machine
Post s Problem
oracle Turing machine
Pratt
orthonormal
predicate logic
output no de
predicate symb ol
prime factorization
P
prime numb er
P see counting classes
prime numb er problem
P computable distribution
prime numb er sieve
P isomorphic
Prime Numb er Theorem
P samplable distribution
primitive recursive
PAC probabilistically approximately
priority metho d
correct
probabilistic metho d
PAC learnable
padding
probability ampli cation
padding function
Palmer E M
probability amplitude
Palmer R G
probability distribution
Papadimitriou
probability gap
pro of
Pap ert
pro of calculus
Parb erry
prop erty P
parity function
provable by an oracle
P see counting classes
prover
partial assignment
Pruim
Patashnik
pseudo random numb er
Paterson
PSPACE
PATH
public coins
path
Putnam
Paul
pyramid
p ebble
p ebbling strategy
QBF
p erceptron
quanti ed b o olean formula
p ermutation network
Petri net
quanti er
Pfei er
quantum mechanics
PH
quantum state
PHP
n
quantum Turing machine
pigeonhole principle
QuickSort
pip e
Pipp enger
R odding
Pitassi Rub
Index
Racko second order existential
random bits
second order predicate logic
random oracle
Seiferas
Random Oracle Hyp othesis
self reducible
random p olynomial
self terminating co de
random restriction
Selman
random sequence
sentences
random variable
set of examples
random walk
Sewelson
random Kolmogorov see Kolmogorov
Shamir
random
Sho en eld
Razb orov
Shor viii
Reckhow
sieve of Eratosthenes
reducibility
similar probability distributions
refutation
Simmons
register machine
Simon
regular graph
simple formula
relativization principle
Sinclair
resolution
sink
resolvent
Sipser
reversible computation
Richter
size
Ritchie
of a circuit
Rivest
of a formula
RL
Smolensky
Robbin
SO
Robinson
Soare
Rogers H
Solomono
Rogers J
Solovay
Romp el
Sommerhalder
round
soundness
Rozenb erg
source
Rudich
sparse
Rytter
Sp ecker
sp ectral problem
Salomaa
Sp encer
SAT
Spirakis
SAT
start no de
Savitch
statistical test
Savitch s Theorem
Stirling s formula
Saxe
Sto ckmeyer
Sch oning viii
Strassen
strategy
Straubing
Schaefer viii
Strauss viii
Schmidt
string
Schnorr
structure
scholastics
subspace
Schuler vi
success
Schulz
SUPER pro of system
Schwichtenb erg
sup er dup er concentrator
search algorithm
sup erconcentrator
Second Berman Hartmanis Conjecture
sup erconcentrators
Index
universe sup erp osition
UNIX Swapping Lemma
symmetric b o olean function
Valiant
Szelep cs enyi
valid formula
Tamon
variance
Tapp
Vazirani
Tarjan
vector space
Tarui
vector space homomorphism
tautology
veri er
teacher
Vit anyi
terminal no de
Thierauf vi
Waack
threshold function
Wagner
threshold gate
Wainer
threshold value
Walsh Hadamard
Thue system
Wang
tiling
Warmuth
Time Hierarchy Theorem
Watanab e
time interval
Watrous
time space trade o
Wechsung
To da
Wegener
top ological ordering
Wegman
Tor an vi
Welsh
Westrhenen van
total degree
Wigderson
total ordering
witness
truth table
witness circuit
Tsichritzis
WLOG
Turing machine
word problem
Turing reduction
worst case complexity
Wrathall
Ullman
unb ounded fan in
XOR
undecidable
uniform distribution
Young
unitary
universal hash functions
Zachos
Z ak
universal probability distribution
zero knowledge
ZPP see counting classes
universal traversal sequence
universal Turing machine Zuckerman