Efficiently Decodable Non-adaptive Group Testing
Piotr Indyk∗ Hung Q. Ngo† Atri Rudra‡
Abstract few “tests” as possible. A test is a subset of items, which We consider the following “efficiently decodable” non- returns positive if there is a positive in the subset. The adaptive group testing problem. There is an unknown string semantics of “positives,” “items,” and “tests” depend on x ∈ {0, 1}n with at most d ones in it. We are allowed to test any subset S ⊆ [n] of the indices. The answer to the test the application. For example, the topic of group testing tells whether xi = 0 for all i ∈ S or not. The objective started in 1943 when Dorfman studied the problem of is to design as few tests as possible (say, t tests) such that testing for syphilis in WWII draftees’ blood samples [8]. x can be identified as fast as possible (say, poly(t)-time). Efficiently decodable non-adaptive group testing has appli- In this case, items are blood samples, which are positive cations in many areas, including data stream algorithms and if they are infected, and a test is a pool of samples. Since data forensics. then, group testing has found numerous applications A non-adaptive group testing strategy can be repre- sented by a t × n matrix, which is the stacking of all the (see, e.g., the book [9]). Many applications require the characteristic vectors of the tests. It is well-known that if non-adaptive variant of group testing, in which all tests this matrix is d-disjunct, then any test outcome corresponds are to be performed at once: the outcome of one test uniquely to an unknown input string. Furthermore, we know how to construct d-disjunct matrices with t = O(d2 log n) cannot be used to adaptively design another test. Non- efficiently. However, these matrices so far only allow for a adaptive group testing (NAGT) has found applications “decoding” time of O(nt), which can be exponentially larger in DNA library screening [22], multiple access control than poly(t) for relatively small values of d. This paper presents a randomness efficient construction protocols [3, 31], data pattern mining [21], data forensics of d-disjunct matrices with t = O(d2 log n) that can be de- [15] and data streams [7], among others. coded in time poly(d) · t log2 t + O(t2). To the best of our The main research focus thus far has been on knowledge, this is the first result that achieves an efficient de- designing NAGT strategies minimizing the number of 2 coding time and matches the best known O(d log n) bound tests. In some applications, however, the speed of the on the number of tests. We also derandomize the construc- tion, which results in a polynomial time deterministic con- “decoding” procedure to identify the positive items is struction of such matrices when d = O(log n/ log log n). just as important, as we will elaborate later. In this A crucial building block in our construction is the paper, we consider the following “efficiently decodable” notion of (d, `)-list disjunct matrices, which represent the more general “list group testing” problem whose goal is to NAGT problem. Given integers n > d ≥ 1, and an output less than d + ` positions in x, including all the (at unknown string x ∈ {0, 1}n with at most d ones in it most d) positions that have a one in them. List disjunct (x is the characteristic vector of the positives), we are matrices turn out to be interesting objects in their own right and were also considered independently by [Cheraghchi, allowed to test any subset S ⊆ [n] of the indices. The FCT 2009]. We present connections between list disjunct answer to a test S tells whether xi = 0 for all i ∈ S. matrices, expanders, dispersers and disjunct matrices. List The objective is to design as few tests as possible (say disjunct matrices have applications in constructing (d, `)- sparsity separator structures [Ganguly, ISAAC 2008] and in t tests) such that we can identify the input string x as constructing tolerant testers for Reed-Solomon codes in the efficiently as possible (say, in poly(t)-time). data stream model. A NAGT algorithm can be represented as a t × n matrix M, where each row is the characteristic vector 1 Introduction of the subset of [n] to be tested. (The answers to the The basic group testing problem is to identify the set of tests can be thought of as M being multiplied by x, “positives” from a large population of “items” using as where the addition is logical OR and multiplication is the logical AND.) A well known sufficient condition for ∗CSAIL, MIT. Email: [email protected]. Supported in part such matrices to represent uniquely decodable NAGT by David and Lucille Packard Fellowship, MADALGO (Center algorithms is one of disjunctiveness. In particular, a for Massive Data Algorithmics, funded by the Danish National matrix M is said to be d-disjunct if and only if the union Research Association) and NSF grant CCF-0728645. †Department of Computer Science and Engineering, University of at most d columns cannot contain another column. at Buffalo, SUNY. Email: [email protected]. Supported in Here, each column is viewed as a characteristic vector part by NSF grant CCF-0347565. on the set of rows. It is known that t × n d-disjunct ‡Department of Computer Science and Engineering, University matrices can be constructed for t = O(d2 log n) [25, 2, 9]. at Buffalo, SUNY. Email: [email protected]. Supported by NSF d2 CAREER Award CCF-0844796. A lower bound of t = Ω( log d log n) has also been known for a long time [10, 11, 13]. non-hot items occur at most m/(d + 1) times. In terms of decoding disjunct matrices, not much is The algorithm works as follows: let M be a d- known beyond the “naive” decoding algorithm: keep disjunct t × n matrix. For each test i ∈ [t], maintain removing items belonging to the tests with negative a counter ci. When an item j ∈ [n] arrives (leaves outcomes. The recent survey by Chen and Hwang [5] respectively), increment (decrement respectively) all the lists various naive decoding algorithms under different counters ci such that Mij = 1 (i.e. all counters ci group testing models (with errors, with inhibitors, and for which test i contains item j). The algorithm also variations). The main reason for the lack of “smart” maintains the total number of items m seen so far. At decoding algorithms is that current decoding algorithms any point in time, the hot items can be computed as are designed for generic disjunct matrices. Without follows. Think of the test i corresponding to counter imposing some structure into the disjunct matrices, fast ci as having a positive outcome if and only if ci > decoding seems hopeless. m/(d + 1). Due to the small tail property, a test’s The naive decoding algorithm can easily be imple- outcome is positive if and only if it contains a hot item. mented in time O(nt). For most traditional applica- Thus, computing the hot items reduces to decoding the tions of group testing, this decoding time is fine. How- result vector. ever, in other applications, this running time is pro- When Cormode and Muthukrishnan published their hibitive. This raises a natural question (see e.g., [7, 16]) result, the only decoder known for d-disjunct matrices of whether one can perform the decoding in time that is was the O(nt)-time naive decoder mentioned earlier. sub-linear (ideally, polylogarithmic) in n, while keeping This meant that their algorithm above could not be the number of tests at the best known O(d2 log n). efficiently implemented. The authors then provided Our Main Result: In this paper we show that alternate algorithms, inspired by the group testing such a decoding algorithm indeed exists. Specifically, idea above and left the task of designing an efficiently we present a randomness efficient construction of d- decodable group testing matrix as an open problem. disjunct matrices with t = O(d2 log n) tests that can Our main result answers this open question. This be decoded in time poly(d) · t log2 t + O(t2). In par- application also requires that the matrix M be strongly ticular, we only need R = O(log t · max(log n, d log t)) explicit, i.e. any entry in M can be computed in time many random bits to construct such matrices. Further, (and hence, space) poly(t). Our result satisfies this given these R bits, any entry in the matrix can be con- requirement as well. structed in time poly(t) and space O(log n + log t). We We would like to point out that the solution to the also derandomize the construction, which results in a hot items problem using our result is not as good as poly(n) time and poly(t) space deterministic construc- the best known results for that problem. For example, tion of such matrices when d = O(log n/ log log n). the paper [7] gives a solution which has a lower space To the best of our knowledge, this is the first result complexity than what one can achieve with efficiently that achieves an efficient decoding time and matches the decodable NAGT. Nevertheless, the above application best known O(d2 log n) bound on the number of tests. to finding heavy hitters is illustrative of many other An earlier result due to Guruswami and Indyk gives applications of NAGT to data stream algorithms, and efficient decoding time but with O(d4 log n) tests [16]. A we expect further results along these lines. similar result but with O(d2 log2 n) tests is also implicit Digital forensics. Data forensics refers to the fol- in that paper. lowing problem in data structures and security. As- sume that one needs to store a data structure on a 1.1 Applications In this section we outline two sce- semi-trusted place. An adversary can change up to d narios where efficiently decodable disjunct matrices play values (out of a total of n values) in the data struc- an important role, and how our results improve on the ture (but cannot change the “layout” of the data struc- earlier works. ture). The goal is to “store” extra information into the Computing Heavy Hitters. Cormode and layout of the data structure so that if up to d values Muthukrishnan [7] consider the following problem of in the data structure are changed then an auditor can determining “hot items” or ”heavy-hitters.” Given a quickly pinpoint the locations where data was changed. sequence of m items from [n], an item is considered Goodrich, Atallah and Tamassia [15] introduced the “hot” if it occurs > m/(d + 1) times. Note that problem and used group testing to solve this problem. there can be at most d hot items. Cormode and In particular, using a randomness efficient construc- Muthukrishnan proposed an algorithm based on NAGT tion, they present data forensics schemes on balanced that computes all the hot items as long as the input binary trees, skip lists and arrays (and linked lists) satisfies the following “small tail” property: all of the q q that can handle O 3 n/ log2 n , O 3 n/ log2 n and n1, ci ∈ Li. In the Kautz-Singleton construction, it can be checked that the construction of the Li O p4 n/ log n many changes respectively. Using our can be done in poly(t) time. Thus, we would be randomness efficient construction, we can improve these in business if we can solve the following problem in √ poly(n1) time: Given Li ⊆ [q] for every 1 ≤ i ≤ n1 bounds to O pn/ log n , O pn/ log n and O ( 3 n) with |Li| ≤ d, output all codewords (c1, . . . , cn1 ) ∈ many changes respectively. These latter bounds are the Cout such that ci ∈ Li for every i ∈ [n1]. This is best possible with the techniques of [15]. (For more precisely the (“error-free”) list recovery problem. It is details see Appendix A.) known that for Reed-Solomon codes, this problem can be solved in poly(n1) time (cf. [26, Chap. 6]). This 1.2 Techniques observation was made by Guruswami-Indyk [16]. They Connections to coding theory. Our construc- also presented an efficiently decodable (in fact O(t) time tion involves concatenated codes. A concatenated bi- decoding) d disjunct matrix with t = O(d4 log n) using k1 n1 nary code has an outer code Cout :[q] → [q] over an expander based code as Cout. However, neither of the a large alphabet q = 2k2 and a binary inner code constructions in [16] matches the best known bound of k2 n2 Cin : {0, 1} → {0, 1} . The encoding of a message in t = O(d2 log n). k2 k1 ({0, 1} ) is natural. First, it is encoded with Cout and Overview of our construction. Our randomized then Cin is applied to each of the outer codeword sym- construction is based on an ensemble of codes that bols. The concatenated code is denoted by Cout ◦ Cin. was considered by Thommesen to construct random- (In fact, one can— and we will— use different inner ized binary codes that lie on the Gilbert-Varshamov i codes Cin for every 1 ≤ i ≤ n1.) bound [30]. Cout is the Reed-Solomon code but the in- Concatenated codes have been used to construct ner codes are chosen to be independent random binary d-disjunct matrices at least since the seminal work of codes. Unlike [30], where the inner codes are linear, in Kautz and Singleton [20]. In particular, they picked our case the (non-linear) inner codes are picked as fol- Cout to be a maximum distance separable code of rate lows: every codeword is chosen to be a random vector in 1 n d+1 (e.g., Reed-Solomon code with n1 = q) and the {0, 1} 2 , where each entry is chosen to be 1 with prob- inner code to be the identity code Iq that maps an el- ability Θ(1/d). We show that this concatenated code ement i ∈ [q] to the ith unit vector in {0, 1}q. The with high probability gives rise to a d-disjunct matrix. disjunct matrix M corresponding to Cout ◦ Cin is ob- The use of random, independently generated codes tained by simply putting all the n = qk1 codewords as is crucial for our purpose. This is because the correct- columns of the matrix. The resulting matrix M is d- ness of the aforementioned decoding algorithms cru- 2 2 disjunct and has t = O(d log n) many rows. Recently, cially uses the fact that Cin is d-disjunct. (In fact, Porat and Rothschild presented a deterministic poly- for the identity code, it is n-disjunct.) However, this nomial (in fact, O(nt)) time construction of d-disjunct idea hits a barrier. In particular, as mentioned ear- 2 matrices with O(d log n) rows [25]. In their construc- d2 lier, d-disjunct matrices need to have Ω log d log n tion, Cout is a code over an alphabet size Θ(d) that lies on the Gilbert-Varshamov bound. Their inner code many rows. This implies that if we are using a d- is also the identity code as used by Kautz and Single- disjunct matrix as an inner code and a O(1/d)-rate outer ton. Note that since all these constructions result in Reed-Solomon code then the resulting matrix will have d3 d-disjunct matrices, they can be decoded in O(nt) time. Ω log d log n rows. Thus, this approach seems to fall 2 Next, we outline how if Cout is efficiently list short of obtaining the best known bound of O(d log n). recoverable (and Cin is the identity code), then M can However, consider the following idea that is moti- be decoded in poly(t) time. In particular, consider the vated by list decoding algorithms for binary concate- following natural decoding procedure for an outcome nated codes (cf. [17]). Let MCin be the n2 × q matrix t vector r ∈ {0, 1} . First, think of r = (r1, . . . , rn1 ) as a obtained by arranging the codewords in Cin as columns. vector in ({0, 1}n2 )n1 , where each symbol in {0, 1}n2 Think of MCin as representing a NAGT strategy for lies naturally in one of the n positions in an outer 1 at most d unknown positives. Suppose MCin has the codeword. Now consider the following algorithm. For following weaker property than d-disjunctiveness. Let q each i ∈ [n1], let Li ⊆ [q] be the set of positions y ∈ {0, 1} be a vector with at most d ones in it, in- n where ri has a one. Since there are at most d positives, dicating the positives, and let r ∈ {0, 1} 2 be the test |L | ≤ d, for every i ∈ [n ]. Furthermore, if the jth item i 1 outcomes when applying MCin to y. Suppose it is the q is positive then the jth codeword (c1, . . . , cn1 ) ∈ Cout case that any z ∈ {0, 1} which results in the same test satisfies the following property: for every 1 ≤ i ≤ outcome r as y has to satisfy (i) yi = 1 implies zi = 1 and (ii) |z| ≤ 2d, where |z| denote the number of 1’s list disjunct matrices. First, as pointed out by Cher- in a binary vector z. (Note that in the d-disjunct case, aghchi [6], (d, d)-list disjunct matrices can be used to we need the stronger property that z = y.) Then, if design optimal two-stage group testing algorithms. In we use such an inner code instead of a d-disjunct ma- the two-stage group testing problem, one is allowed to trix, in the two step algorithm mentioned a couple of make tests into two stages (the tests in the second stage paragraphs above we will have |Li| ≤ 2d (instead of can depend on the answers obtained in the first stage). |Li| ≤ d). It turns out that, with this weaker inner The algorithm works as follows: in the first round, one codes, one can still perform list recovery on such inputs uses the (d, d)-list disjunct matrix to obtain a subset for Reed-Solomon codes. The only catch is that the two- S ⊆ [q] of size at most 2d that contains all of the de- step algorithm can return up to O(d2) items including fective positions. In the second stage, one can probe all the (at most d) positives. We can thus run the naive each of the columns in S, to compute the defective lo- decoding algorithm (which will now take O(d2t) time) cations. Note that this algorithm takes O(d log n) many to recover x exactly. Further, this latter step can be tests, which is optimal as there are (n/d)O(d) many pos- done without making another pass over the input. sible answers. Second, we show that (d, `)-list disjunct Perhaps most importantly, the relaxed notion of matrices immediately imply (d, d+`)-sparsity separator disjunct matrices mentioned above can be shown to structures. The latter were used by Ganguly to design exist with O(d log n) rows, which paves the way for the deterministic data stream algorithm for the d-sparsity final matrix to have O(d2 log n) rows as desired. It turns problem [14]. In fact, Ganguly’s construction of (d, 2d)- out that these new objects are interesting in their own sparsity separator uses lossless expander in pretty much right and next, we discuss them in more detail. the same way we do. (See Section 6 for more details.) List Disjunct Matrices. We define (d, `)-list dis- Finally, we point out that the recent work of Rudra and junct matrices as follows. A t × n matrix M is called Uurtamo [27] uses (d, d)-list disjunct matrices to design (d, `)-list disjunct if the following holds: for any disjoint data stream algorithms for the following “tolerant test- subsets S ⊆ [n] and T ⊆ [n] such that |S| ≤ d and ing” of Reed-Solomon codes. Given the received word, |T | ≥ `, there exists a row in which there is at least one the algorithm needs to decide if it is within Hamming 1 among the columns in T while all the columns in S distance d of some codeword or is at distance at least 2d have a 0. It can be shown that given an outcome vector from every codeword. Their algorithm also works with t n r ∈ {0, 1} obtained by using M on an input x ∈ {0, 1} d-disjunct matrices but then one only gets√ a sub-linear with |x| ≤ d, running the naive decoder returns a vector space data stream algorithm for d = o( n), whereas y ∈ {0, 1}n such that it contains x and |y| ≤ |x| + ` − 1. using list disjunct matrices allows them to design a sub- (Hence, the name list disjunct matrix.) In the proof of linear space data stream algorithm for d = o(n). (See our main result, we use the fact that the random inner Section 6 for more details.) codes with high probability are (d, d)-list disjunct. These objects were independently considered by 1.3 Paper Organization We begin with some pre- Cheraghchi [6]. (In fact, he considered more general liminaries in Section 2. In Section 3, we define list versions that can handle errors in the test results.) disjunct matrices and present upper and lower bounds. We study these objects in some detail and obtain the Section 4 formally presents the connection between list following results. First, we show that (d, `)-list disjunct recoverable codes and efficiently decodable disjunct ma- matrices contain d-disjunct column sub-matrices, which trices. We present our main construction of efficiently allows us to prove a lower bound of Ω(d/ log d log(n/`)) decodable disjunct matrices equaling the best known on the number of rows for such objects as long as number of tests in Section 5. Finally we present connec- d ≤ O(log n/ log log n). (This gives a better lower bound tions between list disjunct matrices and pseudo-random for large ` than the bound of d log(n/d)−d−` in [6].) We objects and their applications in Section 6. also show that lossless expanders and dispersers with appropriate parameters imply list disjunct matrices. 2 Preliminaries Using known constructions of expanders and dispersers For an integer ` ≥ 1, let [`] to denote the set {1, . . . , `}. we construct (d, d)-list disjunct matrices with near For any t × N matrix M, we will use Mj to refer to its optimal (d log n)1+o(1) number of rows and (d, `)-list j’th column and Mij to refer to the i’th entry in Mj. disjunct matrices with O(d log n) rows when ` and d Let Fq denote the finite field of q elements. are polynomially large in n. The basic problem of non-adaptive combinatorial We believe that list disjunct matrices are natural group testing can be described as follows. Given a objects and merit study on their own right. To sub- population of N “items” which contains at most d stantiate our belief, we point out three application of “positive” items, we want to identify the positives as 1 n1 quickly as possible using t simultaneous “tests.” Each ent inner codes Cin,...,Cin , one per position. Denote 1 n1 test is a subset of items, which returns “positive” if this general concatenated code as Cout ◦ (Cin,...,Cin ). there’s at least one positive item in the subset. We It is defined in the natural way. Given any message k want to “decode” uniquely the set of positives given the k2 1 m ∈ {0, 1} , let Cout(m) = (x1, . . . , xn1 ). Then, results of the t simultaneous tests. 1 n1 1 n1 It is natural to model the tests as a t × N binary Cout ◦ (Cin,...,Cin )(m) = Cin(x1),...,Cin (xn1 ) . matrix M where each row represents a test and each Let `, L ≥ 1 be integers and let 0 ≤ α ≤ 1. A column is an item. Set Mij = 1 if and only if item j belongs to test i. The test outcome vector is a q-ary code C of block length n is called (α, `, L)-list t-dimensional binary vector r, where a 1 indicates a recoverable if, for every sequence of subsets S1,...,Sn positive test outcome for the corresponding test. such that |Si| ≤ `, ∀i ∈ [n], there are at most L For the decoding to be unique, it is sufficient for codewords c = (c1, . . . , cn) for which ci ∈ Si for at the test matrix M to satisfy a property called d- least αn positions i. A (1, `, L)-list recoverable code disjunctiveness. We say a t × N binary matrix M is d- will be henceforth referred to as (`, L)-zero error list recoverable. We will need the following powerful result disjunct if, for every column Mj in M and every subset due to Parvaresh and Vardy1: of d other columns Mj1 ,...,Mjd (i.e. j 6∈ {j1, . . . , jd}), M 6⊆ M ∪· · ·∪M . Here, we interpret the columns as j j1 jd Theorem 2.1. ([24]) For any given integer s ≥ 1, (characteristic vectors of) subsets of [t]. The following prime power r, power q of r, and every pair of integers lemma gives a sufficient condition for disjunct matrices, 1 < k ≤ n ≤ q, there exists an explicit Fr-linear map and can be shown by a simple counting argument [9]. k n E : Fq → Fqs satisfying the following: Lemma 2.1. Let M be a binary matrix such that w is n 1. The image C ⊆ Fqs of E is a code of minimum the minimum Hamming weight of columns in M and distance at least n − k + 1. let λ be the size of the largest intersection between any two columns in M. Then M is d-disjunct as long as 2. Provided λd + 1 ≤ w. In particular, M is w−1 -disjunct. λ (2.1) α > (s + 1)(k/n)s/(s+1)`1/(s+1), When M is d-disjunct, it is well-known that the s following naive decoder works (cf. [9]). Let the outcome C is an (α, `, O((rs) n`/k))-list recoverable code. vector be r. For every i ∈ [t] and j ∈ [N] such that Further, a list recovery algorithm exists that runs in poly((rs)s, q, `)-time. For s = 1, the (s + 1) factor ri = 0 and Mij = 1, “eliminate” item j. Once all such indices from [N] are eliminated, we will be left in the right-hand-side of (2.1) can be removed. precisely with the set of positives. The naive decoder In the above, the s’th “order” Parvaresh-Vardy code takes O(tN)-time. will be referred to as the PVs code. PV1 is the well- t A(t, k, d)q-code C is a subset C ⊆ [q] of size known Reed-Solomon codes and will be referred to as k q such that any two codewords differ in at least d the RS code. positions. The parameters t, k, d and q are known as The simplest possible binary code is probably the the block length, dimension, distance and alphabet size q “identity code” Iq :[q] → {0, 1} defined as follows. For of C. Sometimes we will drop the distance parameter q any i ∈ [q], Iq(i) is the vector in {0, 1} that has a 1 in and refer to C as a (t, k)q code. the i’th position and zero everywhere else. It is easy to k Given a (t, k, d)q code C, let MC denote the t × q see that matrix whose columns are the codewords in C. We shall construct disjunct matrices by specifying some Lemma 2.2. MIq is q-disjunct. binary code C and show that M is disjunct. The C The following result was shown in [25] by concate- codes C that we construct will mostly be obtained nating a q-ary code on the GV bound (with q = Θ(d) by “concatenating” an outer code – based on Reed- and relative distance 1 − 1/d) with the identity code I . Solomon codes – with a suitably chosen inner code. q
Given an (n1, k1)2k2 code Cout and an (n2, k2)2 code Theorem 2.2. ([25]) Given n and d ≥ 1. There exists 2 Cin, their concatenation, denoted by Cout ◦Cin is defined a t×n d-disjunct matrix MPR such that t = O(d log n). k as follows. Consider a message m ∈ {0, 1}k2 1 . Further, the matrix can be constructed in time (and
Let Cout(m) = (x1, . . . , xn1 ). Then, Cout ◦ Cin(m) = space) O(tn), and all the columns of MPR have the same
(Cin(x1),...,Cin(xn1 )) .Cout ◦ Cin is an (n2n1, k1k2)2- Hamming weight. code. 1 In general, we can concatenate Cout with n1 differ- This statement of the theorem appears in [18]. 3 List Disjunct Matrices and Basic Bounds Proof. Consider the following hypergraph H: each of We now define a generalization of group testing that is the n columns in M forms a vertex and there is a inspired by list decoding of codes and will be useful in (d + 1)-hyperedge among d + 1 vertices in S ⊂ [n] if our final construction. Let `, d ≥ 1 be integers. We call there exists a j ∈ S such that the jth column of M is a t×n boolean matrix M (d, `)-list disjunct if for any two contained in the union of the columns in S \{j}. Note disjoint subsets S and T of [n] where |S| ≤ d, |T | ≥ `, that an independent set in H of size m corresponds S S to a column sub-matrix of M that is d-disjunct. For we have j∈T Mj 6⊆ j∈S Mj. Note that a d-disjunct matrix is a (d, 1)-list disjunct matrix and vice versa. notational convenience define deg(i) to be the degree of Without loss of generality, we assume that ` ≤ n − 1. i in H. When d + ` > n, we can replace d by d = n − `. If Thus, we need to show that there exists a large d+` ≤ n, then it is sufficient to replace the condition in enough independent set in H, for which we use a the definition by |S| = d, |T | = `. Thus, we will assume result of Caro and Tuza [4]. In particular, H has an d + ` ≤ n henceforth, and only consider |S| = d, |T | = ` independent set of size at least (cf. [28, Pg. 2]): as the condition in the definition of (d, `)-list disjunct X 1 1 matrices. (3.2) , where f(i) = Θ (1 + deg(i)) d . f(i) We next prove upper and lower bounds for the i∈[n] optimal “rate” of list disjunct matrices. To prove an upper bound for the optimal number of rows of a We claim that for at least n/2 vertices i, (d, `)-list disjunct matrix, and to devise a nO(d) time construction for it, the idea is to cast the problem of n constructing a (d, `)-list disjunct matrix as a special case (3.3) deg(i) ≤ 2(d + 1) (` − 1). of the so-called k-restriction problem defined and solved d − 1 in Alon, Moshkovitz, and Safra [2]. (The proof appears n d−1 in Appendix B.) Since d−1 ≤ n , (3.3) and (3.2) imply that H has an independent set of size at least Ω((n/(d`))1/d), as Lemma 3.1. Given positive integers n ≥ d + 1, let 0 = desired. d/` ` d . Then, a t×n (d, `)-list disjunct matrix Next, we prove (3.3). First, we claim that the d+` d+` number of edges in H is upper bounded by n n (`−1). d+` d−1 2(d+`) ln n+ln ( ) can be constructed so that t = d in This implies that the average degree of H is at most 0 n (d+1) d−1 (`−1). An averaging argument proves (3.3). d+` d+` d+` 1 time poly d , n , 2 , . In particular, if ` = Now we upper bound the number of edges in H. To this Θ(d), then a t × n (d, `)-list disjunct matrix can be end, “assign” a vertex i to all its incident edges S ⊆ [n] constructed in time nO(d) where t = O(d log n). such that there exists a j 6= i where the jth column is contained in the union of the columns in S \{j}. For Later on, we will see constructions of (d, Θ(d))-list every T ⊆ [n] with |T | = d with i ∈ T , there exists disjunct matrices that can be constructed more time- at most ` − 1 many j such that T ∪ {j} is an edge. efficiently than Lemma 3.1. However, those construc- (This follows from the fact that M is (d, `)-list disjunct.) n tions need more tests (up to logarithmic factors). Thus, any vertex i is assigned to at most d−1 (` − 1) Next we show that any (d, `)-list disjunct matrix edges. Since every edge is assigned at least one vertex, n M contains a large enough d-disjunct sub-matrix. The the total number of edges is bounded by n d−1 (` − 1), proof follows by building a hypergraph from M and then as desired. observing that an independent set in the hypergraph The lower bound on the number of rows in a (d, `)- corresponds to a d-disjunct matrix. Then a result due list disjunct matrix then follows from the lower bound to Caro and Tuza [4] completes the proof. on the number of rows in a d-disjunct matrix. The bound on d ≤ O(log n/ log log n) is needed to make sure Lemma 3.2. Let d, ` ≥ 1 and t, n ≥ 1 be integers. Let that the lower bound on list disjunct matrix holds. (The M be a t × n (d, `)-list disjunct matrix. Then there latter for√ a d-disjunct matrix with N columns needs ˜ is a t × m sub-matrix of M that is d-disjunct with d ≤ O( N)). 1 n d m ≥ Ω d` . In particular, any t × n matrix that is (d, `)-list disjunct (with d ≤ O(log n/ log log n) and Cheraghchi proves a lower bound of d log(n/d) − ` 1 ≤ ` ≤ n1−γ for some constant γ > 0) needs to satisfy for (d, `)-list disjunct matrices [6]. This is better than the bound in Lemma 3.2 for moderately large ` (e.g. t = Ω d · log n . log d when ` = Θ(d)). 4 From List Recoverable Codes to Efficiently E = {m1,..., md} ⊆ ({0, 1}n2 )n1 . Denote mj = j j Decodable Group Tests (m , . . . , mj ) where m ∈ {0, 1}n2 . By definition of 1 n1 i 1 d r, we have ri = m ∪ · · · ∪ m . Consequently, if M i As was mentioned in the introduction section, concate- i i Cin 1 d nating an efficiently list recoverable outer code with is (d, `)-list disjunct, then {mi , . . . , mi } ⊆ Si. As at list disjunct inner codes gives rise to efficiently decod- least αn1 of the M i are (d, `)-list disjunct, E will be Cin able disjunct matrices. The decoding algorithm mimics included in the set T of at most L codewords returned the standard list decoding algorithm for concatenated by the list recovery algorithm for Cout, which is an codes. (α, d + ` − 1,L)-list recoverable code. Finally, we argue the space complexity. Given k2 k1 Theorem 4.1. Let d, `, L ≥ 1 be integers and 0 < i = (i1, i2) ∈ [n1] × [n2] and j ∈ ({0, 1} ) , we can
α ≤ 1 be a real number. Let Cout be an (n1, k1)2k2 compute Mij in the following natural way. First com- code which can be (α, d + ` − 1,L)-list recovered in pute (Cout(j))i1 in space S1(n1, d, `, L, k1, k2) and then 1 n1 i1 time T1(n1, d, `, L, k1, k2). Let C ,...,C be (n2, k2)2 compute C ((Cout(j))i ) in space S2(n2, d, `, k2). in in in 1 i2 codes such that for at least αn1 values of 1 ≤ i ≤ n1, We might need the extra O(log t + log N) space to store M i is (d, `)-list disjunct and can be (“list”) decoded Cin the indices i and j. def in time T2(n2, d, `, k2). Suppose the matrix M = By instantiating the outer and inner codes in The- M 1 n1 is d-disjunct. Note that M is a t × N Cout◦(Cin,...,Cin ) orem 4.1 with different specific codes, we obtain sev- k1k2 matrix where t = n1n2 and N = 2 . Further, suppose eral constructions balancing some tradeoffs: there ex- that any arbitrary position in any codeword in Cout and ists disjunct matrices that can be decoded in poly(t) i 2 2 3 Cin can be computed in space S1(n1, d, `, L, k1, k2) and time with O(d log N), O(d log N(log d + log log N)) 3 S2(n2, d, `, k2), respectively. Then, and O(d log N) many tests. Further, any entry in these matrices can be constructed in space poly(log t, log N), (a) given any outcome vector produced by at most d poly(log t, log N) and poly(t, log N) respectively. Last positives, the positive positions can be recovered but not least, we can also design efficiently decodable in time n1T2(n2, d, `, k2) + T1(n1, d, `, L, k1, k2) + list group testing by using the PVs code with s = 1/ O(Lt); and as the outer code. All the proofs are in Appendix C. (b) any entry in M can be com- Lemma 4.1. Let N > d, s ≥ 1 be integers. Then puted in O (log t + log N) + there exists a constant c ≥ 1 and a t × N matrix M cs 1+1/s 1+s cs 1+1/s O (max{S1(n1, d, `, L, k1, k2),S2(n2, d, `, k2)}) with t = O s d log N that is d, s d - space. list disjunct and can be list decoded in poly(t) time. Further, any entry of the matrix can be computed in Proof. The decoding algorithm for M is the natural poly(log t, log N, s) space. one. Let E ⊆ [N] with |E| ≤ d be the set of positive n The result above achieves better bounds than the items. Further let r = (r , . . . , r ) ∈ ({0, 1}n2 ) 1 be 1 n1 efficient decodable constructions of list disjunct matrices the outcome vector. in [6] (though the results in [6] can also handle errors). In the first step of the algorithm, for each i ∈ [n1], run the list decoding algorithm for MCi on outcome in 5 On Constructing Efficiently Decodable (sub-) vector ri. For each i where MCi is (d, `)- in Disjunct Matrices n2 list disjunct we will obtain a set Si ⊆ {0, 1} with This section contains the main result of the paper. We |Si| ≤ d + ` − 1. There are at least αn1 such i. For the first show probabilistically that, given d and N, there indices i where MCi is not (d, `)-list disjunct, let Si be in exists a t × N efficiently decodable d-disjunct matrix d + ` − 1 arbitrary members of {0, 1}n2 . This step takes with t = O(d2 log N), which matches the best known time n T (n , d, `, k ). 1 2 2 2 constructions of disjunct matrices (whether probabilis- In the second step, we run the list recovery algo- tic or deterministic). We then show that our probabilis- rithm for C on S ,...,S . This step will output a out 1 n1 tic construction can be derandomized with low space or subset T ⊆ [N] such that |T | ≤ L. This step takes time low time complexity. T1(n1, d, `, L, k1, k2). Now if we can ensure that E ⊆ T , then we can run the naive algorithm on M restricted to 5.1 Probabilistic Existence of Efficiently De- the columns in T to recover E in time O(Lt). Thus, the codable Disjunct Matrices total running time of the algorithm is as claimed. To complete the argument above, we need to show Theorem 5.1. Let d, k1, k2 be any given positive in- k2 that E ⊆ T . Without loss of generality, assume tegers such that 10dk1 ≤ 2 . Define n1 = 10dk1, k1k2 n2 = 480dk2, t = n1n2, and N = 2 . Note that Thus, by Chernoff bound, k2 2 n1 ≤ 2 and t = O(d log N). def t n1 1 Pr |A1 ∩ B1| ≥ 2 Let Cout be any n1, k1 = 10d , n1 1 − 10d - 2 2k2 100d 1 n1 code. Then, there exist inner codes Cin,...,Cin , each of 2 1 ≤ Pr |A ∩ B | ≥ t(1 − ) which is an (n2, k2)2 code such that the following hold: 1 1 100d2 10d 1 1 2 ∗ 1 n1 − 2 t(1− 10d ) −t/(600d ) ∗ ≤ e 300d ≤ e . (a) Let C = Cout ◦ (Cin,...,Cin ), then MC is t × N matrix that is d-disjunct. Next, we turn to |A2 ∩ B2|. Note that |A2 ∩ B2| ≤ |A2| t and that A2 is a random vector in {0, 1} 10d where each (b) For every i ∈ [n1], MCi is a (d, d)-list disjunct 1 in bit is 1 independently with probability 10d . Again, by matrix. 2t Chernoff bound the probability that |A2| ≥ 100d2 is at 2 2 most e−t/(300d ) ≤ e−t/(600d ). Proof. We will show the existence of the required inner Thus, by the union bound the probability that codes by the probabilistic method. In particular, we will 4t t i |A1 ∩ B1| + |A2 ∩ B2| < 100d2 < 20d2 is at least refer to each inner code C by its corresponding matrix 2 in 1−2e−t/(600d ) = 1−2e−8k1k2 ≥ 1−2N −11. Taking the MCi . For every 1 ≤ i ≤ n1, pick MCi to be a random N in in union bound over all choices of i and j, we conclude k2 2 n2 × 2 binary matrix where each entry is chosen that (ii) is violated with probability at most N −9. independently at random to be 1 with probability 1 . 10d Overall, condition (a) does not hold with probability We stress that the random choices for each of the inner at most 1/N 39d + 1/N 9 ≤ 2/N 9. codes are independent of each other. We now bound the probability that condition (b) We first bound the probability that condition (a) holds. To this end we will show that for any 1 ≤ i ≤ n1, holds, i.e. MC∗ is d-disjunct. For notational conve- Ci is not (d, d)-list disjunct with probability at most k2 ∗ in nience define q = 2 and M = MC∗ . −58d ∗ q . Then by the union bound and the fact that By Lemma 2.1, M is d-disjunct if the following two k2 n1 ≤ 2 = q, condition (b) does not hold with events hold: probability at most q−57d. For the rest of the proof, t fix an 1 ≤ i ≤ n1. For notational convenience, we will (i) Every column has Hamming weight at least 20d +1. i denote the matrix M i as M . We note that the n2 × q Cin i t matrix M is not (d, d)-list disjunct if there exists two (ii) Every two distinct columns agree in at most 20d2 disjoint subsets S1,S2 ⊆ [q] of size d such that for every positions. i i j ∈ S1, Mj ⊆ ∪`∈S2 M` . We will upper bound the ∗ probability of such an event happening. Note that all Consider event (i). Any arbitrary column Mj of i M ∗ is simply a random vector in {0, 1}t where every the entries in M are independent random variables. 1 This implies (along with the union bound) that the bit is 1 independently with probability 10d . Thus, by i ∗ probability that M is not (d, d)-list disjunct is at most Chernoff bound the probability that Mj has Hamming −t/(120d) ! !n2 weight ≤ t/(20d) is at most e . Taking a union qq − d 1 d 1 d bound over the N choices of j, we conclude that event 1 − 1 − 1 − 1 − − t d d 10d 10d (i) does not hold with probability at most Ne 120d ≤ n N −39d, where the inequality follows from the fact that √ 1 2 ≤ q2d 1 − (1 − 1/ 10 e) · √ 2 10 t = 4800d log2 N. e We now turn to event (ii). Pick two distinct ≤ q2d2−n2/8 ≤ q−58d, columns i 6= j ∈ [N]. By the fact that Cout has 1 relative distance at least 1 − 10d , the ith and jth where the last inequality follows from our choice of codeword in Cout disagree in some positions S ⊆ [n1] n2 = 480dk2. 1 t(1− 1 ) 2 1 10d Thus, with probability at least 1 − − both with |S| = (1 − 10d )n1. Let A1 ∈ {0, 1} and N 9 257dk2 t ∗ of the required properties are satisfied. A2 ∈ {0, 1} 10d denote Mi projected down to S and [N]\S. Let B and B denote the analogous projections 1 2 We use the theorem above to prove the existence for M ∗. Note that we need to bound the random j of an efficiently decodable disjunct matrix matching the variable |A ∩ B | + |A ∩ B |. 1 1 2 2 best upper bound known to date. We start with |A1 ∩B1|. Note that by the definition t(1− 1 ) of A1, B1 and the inner codes, A1 ∩ B1 ∈ {0, 1} 10d , Corollary 5.1. Let N > d ≥ 1 be any given in- 1 where each bit is 1 independently with probability 100d2 . tegers. There exists a t × N d-disjunct matrix with t = O(d2 log N) that can be decoded in time poly(d) · O(t) inner code matrix and another O(log t) space t log2 t + O(t2). to maintain the counter.
Proof. We ignore the issue of integrality for the sake Thus, overall we will need O(log N) space. The next 2 of clarity. First, suppose N ≥ 100d . Set k2 = part of checking condition (a) will need to us compute log2 N N log (10d log N) ≥ 1 and k1 = ≥ 1. It follows the Hamming distance between columns. Using the 2 2 k2 2 k2 accounting above, we conclude that this step can also that 10dk1 ≤ 2 . Theorems 5.1, 4.1 and 2.1 complete the proof. The outer code in Theorem 5.1 is RS with be computed with O(log N) space. 1 We now account for the amount of space needed to rate 10d , which is known to have relative distance 1 check condition (b). To do this, we need to check that all 1 − 10d . The decoding of the outer RS code is done via the algorithm in [1] that runs in poly(d) · t log2 t of the n1 inner codes are (d, d)-list disjunct. For each of time. The naive decoding algorithm is used to decode the inner codes, we need to consider all pairs of disjoint k2 the inner codes, which takes O(t2) time. subsets of [2 ] of size d. (One can keep track of all Next, when d < N < 100d2 we can remove such subsets in space O(dk2) = O(n2) = O(d log t). For k2 arbitrarily 100d2−N columns from a t×100d2 efficiently the latter equality we will need to choose n1 = Θ(2 ).) decodable d-disjunct matrix to obtain a t×N efficiently Then for each pairs of such subsets of columns, we need decodable d-disjunct matrix which still satisfies t = to check if for some row, all the columns in one subset 2 have 0 while at least one column in the other subset has O(d log N). a 1. Since there are n2 rows in the inner code matrix, 5.2 Derandomizing the Proof of Corollary 5.1 this check be done in space O(log n2+log d) = O(log n2). Using Nisan’s PRG for space bounded computation [23], Thus, overall to check condition (b), we need O(d log t) we can reduce the amount of randomness in Corol- space. lary 5.1. Thus, overall we are dealing with a S = O(max(log N, d log t)) space computation on R = O(t2) Corollary 5.2. Let N > d ≥ 1 be any given integer. inputs. However, there is a catch in that Nisan’s PRG Then using O(log t · max(log N, d log t)) random bits, works with R unbiased random bits while in our proof with probability 1 − o(1), one can construct a t × N d- 1 we are dealing with R 10d -biased bits. However, it is disjunct matrix with t = O(d2 log N) that can be decoded easy to convert R0 = O(R log d) unbiased random bits to in time poly(d) · t log2 t + O(t2). 1 R 10d -biased bits (e.g., by grouping O(log d) chunks of unbiased bit and declaring a 1 if and only if all the bits in Proof. The idea is to apply Nisan’s PRG G for space the chunk are 1.) Further, this conversion only needs an bounded computations. In particular, Nisan’s PRG additional O(log log d+log R) space. Given this conver- can fool computations on R input bits with space S ≥ sion, we can assume that our proof needs unbiased bits. Ω(log R) by using only O(S log R) pure random bits. Thus, we can apply G with S0 = S+O(log log d+log t) = (This PRG has low space requirements and is strongly O(S) and R0 = O(t2 log d) = O(t3). Thus, this implies explicit, which useful for data stream algorithms.) The that we can get away with O (log t · max(log N, d log t)) idea is to use G on the computation that checks condi- unbiased random bits. tions (a) and (b) in proof of Theorem 5.1. Let’s start with the checks needed for condition (a). This has two Corollary 5.2 implies that we can get a determinis- parts. First, we need to check that all the N columns in tic construction algorithm with time complexity M ∗ has Hamming weight at least t +1. To do this we 2 20d max N O(log d+log log N),N O(d log d/ log N) , by cycling need O(log N) bits to keep track of the column index. through all possible random bits. Next, we use the Then for each column we have to method of conditional expectation to get a faster de- • Compute the element in the corresponding RS code terministic construction time. for each of the n1 outer positions (we will need O(log n1) = O(k2) space to keep an index for these Corollary 5.3. Let N > d ≥ 1 be any given integer. There exists a deterministic O(t5N 2)+N O(d log d/ log N)- positions). Each such symbol from F2k2 can be computed in space O(k2) (by the fact that the time and poly(t)-space algorithm to construct a t×N d- 2 generator matrix of RS codes is strongly explicit); disjunct matrix with t = O(d log N) that can be decoded in time poly(d) · t log2 t + O(t2). • For each symbol in RS codeword, we need to count the number of ones in the encoding in the Proof. Let us define three sets of random variables. For corresponding inner codes. For this we will need every i ∈ [N], let Xi be the indicator variable of the O(log t) space to access an element of the O(t) × event that the ith column of M ∗ has Hamming weight t at most 20d . Further, for every pair i 6= j ∈ [N], define Note that the construction time is polynomial in N as Yi,j to be the indicator variable of the event that the long as d is O(log N/ log log N). ith and the jth columns of M ∗ agree in strictly more We conclude this proof by showing how to compute 2 than t/(20d ) positions. Finally, for any i ∈ [n1] and the three kinds of probability. We begin with the X(·) disjoint subsets S, T ⊆ [q] with |S| = |T | = d, define indicator variables. Fix j ∈ [N]. First we need to Zi,S,T denote the indicator variable for the event that compute the codeword Cout(j). If Cout is a strongly for the ith inner code (whose matrix is denoted by M i), explicit linear code, this can be accomplished with O(t2) i i P P k2 ∪j∈SMj ⊆ ∪j∈T Mj . Define V = Xj + Yi,j + multiplications and additions over F2k2 . Since 2 ≤ t, P Zi,S,T . Note that in the proof of Theorem 5.1, we each of these operations can definitely be performed in have shown that over the random choices for the inner time O(t). Thus, this entire step takes O(t3) time. Now codes, to compute Pr[Xj|A], we need to substitute the ith (1 ≤ i ≤ n ) symbol in C (j) (over k ) with the X X 1 out F2 2 [V ] = [Xj] + [Yi,j] corresponding column in M i . Note that the resulting E E E Cin j∈[N] i6=j∈[N] vector will have some bits fixed according to A while the n1 rest are independent random bits that take a value of 1 X X X 1 + E[Zj,S,T ] with probability 10d . Thus, we are left with computing j=1 S⊆[q],|S|=d T ⊆[q], the “tail” of a Binomial distribution on O(t) trials each |T |=d,T ∩S=∅ 1 with a success probability of 10d . This can be trivially 2 1 2 < + . computed in time O(t ). Computing Pr[Yi,j = 1|A] is N 9 257dk2 similar to the computation above (the only difference being we have to deal with a Binomial distribution And, since k2 was chosen to be log2(10d log2 N) in 1 2 where the success probability is 100d2 ) and thus, also the proof of Corollary 5.1, we have E[V ] < N 9 + 3 1 takes O(t ) time overall. 57d . (10d log2 N) We finally turn to the Z indicator variables. To derandomize the proof, we will use the standard (·,·,·) Fix i ∈ [n1] and disjoint subsets S, T ⊆ [q] with method of conditional expectation by fixing each of |S| = |T | = d. Note that we want to compute Q p , j∈[n2] j the random bits one at a time (in an arbitrary order) where p denote the probability that if the jth row of and assigning the bit 0 or 1, whichever minimizes the j M i has a one in the columns from S then it also Cin conditional expectation of V given the corresponding has a one in the columns spanned by T . Note that assignment. Recall that the random bits define n1 with the partial assignment A, pj is either 0, 1 or is inner n2 × q matrices, and that each bit is chosen to be 1 a 1 b 1 1 − 1 − 1 − 10d 1 − 10d (where a and b are the one with probability 10d . Note that we can apply the method of conditional expectation if we can compute number of columns in S and T where the values in the the following conditional probabilities (where A denotes jth row are not yet fixed by A). It is easy to check that 2 an assignment to an arbitrary subset of bits in the inner each such pj can be computed in time O(d ). Thus, 2 codes): E[Zi,S,T |A] can definitely be computed in time O(t ). We conclude by noting that the algorithm above 1. For every j ∈ [N], Pr[Xj = 1|A]; can be implemented in poly(t) space.
2. For every i 6= j ∈ [N], Pr[Yi,j = 1|A]; 6 More on List Disjunct Matrices and their Applications 3. For every i ∈ [n1] and disjoint subsets S, T ⊆ [q] We begin with connections between list disjunct matri- with |S| = |T | = d, Pr[Zi,S,T = 1|A]. ces and dispersers and expanders. (We also present a We claim that each of the probabilities above can connection between expanders and disjunct matrices in 3 be computed in time O(t ) (assuming Cout is a strongly Appendix D.) Then we present two applications of list explicit linear code, e.g. RS code).2 This implies that disjunct matrices. 2 each of the n1n2q ≤ t conditional expectation values of V can be computed in time O(Nt3) + O(N 2t3) + tO(d). 6.1 Connection to Dispersers We now state a Thus, the overall running time of the algorithm is simple connection between list disjunct matrices and dispersers. A D-left regular bipartite graph [N]×[D] → 5 2 O(d(log d+log log N)) 5 2 O d log d O(t N ) + 2 = O(t N ) + N ( log N ). [T ] is an (N, L, T, D, )-disperser if every subset S ⊂ [N] of size at least L has a neighborhood of size at least 2We do not try to optimize exact polynomial dependence on t |T |. Given such a bipartite expander G, consider the here. T ×N incidence matrix MG of G. We make the following simple observation. (N, W, T, D, W/2 + 1)-expanders with T = O(D log N) (and W = O(log N)). This by Proposition 6.2 implies Proposition 6.1. Let G be an (n, `, t, t/(d + 1), )- that disperser. Then MG is a (d, `)-list disjunct matrix. Corollary 6.1. There exist t × N matrices that are Proof. Consider two disjoint subsets of columns S1 and (d, d)-list disjunct with t = O(d log N). S2 with |S1| ≥ ` and |S2| = d. To prove the claim, we Later on in this section, we show that there exists need to show that ∪ M 6⊆ ∪ M . We prove the i∈S1 i i∈S2 i an explicit (N, W, T, D, W/2 + 1)-expander with T = later inequality by a simple counting argument. By the (D log N)1+o(1). (We thank Chris Umans for the proof.) value of the degree of G, we get that This with Proposition 6.2 implies the following: dt Corollary 6.2. There is an explicit t×N matrix that | ∪i∈S Mi| ≤ < t. 2 d + 1 is (d, d)-list disjunct with t = (d log N)1+o(1).
On the other hand, as G is a disperser, | ∪i∈S1 Mi| ≥ t, Cheraghchi achieves a slightly better construction which along with the above inequality implies that with d1+o(1) log n tests.
∪i∈S1 6⊆ ∪i∈S2 , as desired. 6.2.1 An Explicit Expander Construction To The best known explicit constructions of dis- construct an explicit expander for our purposes, we will persers is due to Zuckerman [32], who presents ex- use the following two constructions. plicit (N,N δ,N (1−α)δ,O(log N/ log γ−1), γ)-dispersers, which by Proposition 6.1 implies the following: Theorem 6.2. ([19]) Let > 0. There exists an explicit (N1,W1,T1,D1,W1(1 − )) expander with T1 ≤ log W1 Theorem 6.1. Let γ, > 0 be any constants. Then (4D1) and W1 ≤ 2 log N1 log D1/. there exists an explicit t × N (d, `)-list disjunct matrix with t = O(d log N), d = N γ and ` = d1+. Theorem 6.3. ([29]) Let > 0 be a constant. Then there exists an explicit (N2,W2,T2,D2,W2(1 − 6.2 Connection to Expanders We now state a ))-expander with T2 = O(D2W2) and W2 = O(log log N +(log log D )3) simple connection between list disjunct matrices and 2 2 2 . expanders. A W -left regular bipartite graph [N] × We will combine the above two expanders using the [W ] → [T ] is an (N, W, T, D, (1−)W ) expander if every following well known technique. subset S ⊂ [N] of size at most D has a neighborhood (denoted by Γ(S)) of size at least (1 − )|S|W . Given Proposition 6.3. Let G1 be an (N,W1,T1,D,W1(1 − such a bipartite expander G, consider the T × N ))-expander and G2 be an (T1,W2,T2,DW1,W2(1 − incidence matrix MG of G. We have the following simple ))-expander. Then there exists an observation. (N,W1W2,T2,D,W1W2(1 − 2))-expander G. Further, if G1 and G2 are explicit then so is G. Proposition 6.2. Let G be a (n, w, t, 2d, w/2 + 1)- Proof. The graph G is constructed by “concatenating” expander. Then MG is a (d, d)-list disjunct matrix. G1 and G2. In particular, construct the following Proof. Recall that by definition a matrix M is (d, d)- intermediate tripartite graph G0 (on the vertex sets [N], list disjunct if the following is true: for every two [T1] and [T2] respectively), where one identifies [T1] once disjoint S1 and S2 subsets of columns of size exactly as the right vertex set for G1 and once as the left vertex d, both ∪i∈S1 Mi 6⊆ ∪j∈S2 Mj and ∪j∈S2 Mj 6⊆ ∪i∈S1 Mi set of G2. The final graph G is bipartite graph on hold. Note that this property for MG translates to the ([N], [T2]) where there is an edge if and only if there is a 0 following for G:ΓG(S1) 6⊆ ΓG(S2) and vice-versa. We corresponding path of length 2 in G . It is easy to check now argue that the latter is true if G has an expansion that G is an (N,W1W2,T2,D,W1W2(1−2))-expander. of w/2 + 1. Indeed this follows from the facts that |Γ (S ∪ S )| ≥ wd + 2d and |Γ (S )|, |Γ (S )| ≤ wd. G 1 2 G 1 G 2 Next, we prove the following result by combining all the ingredients above. We remark that in our application, we are not Theorem 6.4. There exists an explicit really concerned about the value of W and are (N, W, T, D, W/2 + 1)-expander with T = just interested in minimizing T . (Generally in ex- O(D log N · f(D,N)), where panders one is interested in minimizing both simultane- 3 3 ously.) The probabilistic method shows that there exist f(D,N) = 2O((log log D) +(log log log N) ). o(1) Note that f(D,N) = (D log N) . such that xi = 1 if and only if fi > 0. Since all the frequencies are non-negative, it is easy to see that the By Theorem 6.2, there exists an explicit result vector r computed above is exactly the same as (N,W1,T1,D, 3W1/4 + 1)-expander, where the one that would result from M acting on x. Now consider the two cases (a) x has Hamming weight at W ≤ 16 log N log D, 1 most d. In this case as M is (d, ` − d)-list disjunct, the and decoder for M will output a vector with at most ` − 1 4+log log N+log log D ones in it. (b) Now let us consider the case that x has T1 ≤ (4D) . Hamming weight at least `. In this case, note that each By Theorem 6.3, there exists an explicit xi = 1 will contribute a one to all the rj such that i is (T1,W2,T2,D2, 3W2/4 + 1)-expander, where contained in test j. Thus, the decoder for M will output a vector with at least ` ones in it. This completes the D2 = DW1 ≤ 16D log N log D, proof of correctness of the algorithm above.
3 O(log log T1+(log log D2) ) We would like to point out in the above we assumed W2 is 2 , which in turn is at most 3 3 the following property of the decoder for M: when 2O((log log D) +(log log log N) ), and T is 2 provided with an input x with Hamming weight more 3 3 than d, the decoder will output a vector with Hamming W D ≤ 16D log N log D · 2O((log log D) +(log log log N) ) 2 2 weight larger than that of x. It is easy to check that the ≤ D log N · f(D,N), naive decoding algorithm has this property. However, the decoding time is not longer sub-linear. as desired. We also remark that the the way Ganguly uses Next, we move onto applications of list disjunct lossless expanders to construct (d, 2d)-sparsity separator matrices. structures in exactly the same way we use them to construct (d, d)-list disjunct matrices in Proposition 6.2. 6.3 Sparsity Separator Structure Ganguly in [14] However, [14] uses properties of expanders to come up presents a deterministic streaming algorithm for d- with a more efficient “decoder” than what we have for sparsity testing, which is defined as follows. Given a (d, d)-list disjunct matrices. stream of m items from the domain [n], let f be the vector of frequencies, i.e., for every i ∈ [n], f denotes i 6.4 Tolerant Testing of Reed-Solomon Codes the number of occurrences of i in the stream. The d Rudra and Uurtamo in [27] consider the classical code- sparsity problem is to determine if f has at most d non- word testing problem in the data stream domain. In zero entries. [14] presents an algorithm for the special particular, they consider one pass data stream algo- case when f ≥ 0 (otherwise the problem is known to i rithms for tolerant testing and error detection of Reed- require linear space for deterministic algorithms). Solomon (RS) codes. Informally, in the former problem, A crucial building block of Ganguly’s algorithm is given a received word, the tester has to decide whether what he calls a (d, `)-sparsity separator structure, which it is close to some RS codeword or is far from every is a data structure that can determine if the frequency RS codeword. In the latter problem, one has to decide vector corresponding to a stream has at most d non-zero whether the received word is a codeword or not. entries or it has at least ` non-zero entries. We now Using a slight modification of the well known finger- show how any (d, `−d)-list disjunct t×n matrix M can printing technique, they give a poly-log space one pass be used to build a (d, `)-sparsity separator structure. data stream algorithm to solve the error detection The idea is almost the same as the use of disjunct problem for RS codes. Then they reduce the tolerant matrices in determining hot items [7]: we maintain t testing problem to error detection via (list) disjunct counters, one for each test (i.e. {c } ). Whenever an j j∈[t] matrices. In particular, assume that the tolerant tester item i arrives/leaves, increment/decrement all counters wants to distinguish between the case when the received c such that the jth test contains i. At the end j word is at Hamming distance at most d from some RS convert the counters to a result vector r ∈ {0, 1}t in codeword and the case that it is at a Hamming distance a straightforward manner: r = 1 if and only if c > 0. j j at least ` from every RS codeword. The reduction is not We decode the result vector r according to M and if trivial, so we just sketch the main idea in the reduction the number of ones in the output is at most ` − 1 then here. (They crucially use the fact that any RS code declare the sparsity to be at most d otherwise declare projected onto a (large enough) subset of positions is the sparsity to be at least `. also an RS code.) Here is a simple idea that does not We now briefly argue the correctness of the algo- quite work. Fix a (d, `−d)-list disjunct matrix M. Then rithm above. First define a binary vector x ∈ {0, 1}n for each test, check if the corresponding projected down [12] G. Even, O. Goldreich, M. Luby, N. Nisan, and received word belongs to the corresponding RS code B. Veli˘ckovi´c. Efficient approximation of product (using the error detection algorithm). Then given the distributions. Random Structures Algorithms, 13(1):1– outcome vector use the decoding algorithm to determine 16, 1998. whether at most d or at least ` errors have occurred [13] Z. F¨uredi.On r-cover-free families. J. Comb. Theory, as in the previous application. The catch is that the Ser. A, 73(1):172–173, 1996. [14] S. Ganguly. Data stream algorithms via expander correspondence is not necessarily one to one in the sense graphs. In 19th International Symposium on Algo- that some of the test answers might have false negatives rithms and Computation (ISAAC), pages 52–63, 2008. (as a projected down error pattern might be a codeword [15] M. T. Goodrich, M. J. Atallah, and R. Tamassia. in the corresponding projected down RS code). In [27], Indexing information for data forensics. In Third this issue is resolved by appealing to the list decoding International Conference on Applied Cryptography and properties of RS codes. Network Security (ANCS), pages 206–221, 2005. [16] V. Guruswami and P. Indyk. Linear-time list decoding Acknowledgments We thank Venkat Guruswami and in error-free settings: (extended abstract). In Proceed- Chris Umans for helpful discussions. Thanks again to ings of of the 31st International Colloquium on Au- Chris Umans for kindly allowing us to use his proof of tomata, Languages and Programming (ICALP), pages Theorem 6.4. 695–707, 2004. [17] V. Guruswami and A. Rudra. Concatenated codes can achieve list-decoding capacity. In Proceedings References of the Nineteenth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 258–267, 2008. [1] M. Alekhnovich. Linear diophantine equations [18] V. Guruswami and A. Rudra. Soft decoding, dual over polynomials and soft decoding of reed-solomon BCH codes, and better list-decodable -biased codes. codes. IEEE Transactions on Information Theory, In Proceedings of the 23rd Annual IEEE Conference 51(7):2257–2265, 2005. on Computational Complexity (CCC), pages 163–174, [2] N. Alon, D. Moshkovitz, and S. Safra. Algorithmic 2008. construction of sets for k-restrictions. ACM Trans. [19] V. Guruswami, C. Umans, and S. P. Vadhan. Un- Algorithms, 2(2):153–177, 2006. balanced expanders and randomness extractors from [3] T. Berger, N. Mehravari, D. Towsley, and J. Wolf. parvaresh-vardy codes. In Proceedings of the 22nd An- Random multiple-access communications and group nual IEEE Conference on Computational Complexity, testing. IEEE Trans. Commun., 32(7):769–779, 1984. pages 96–108, 2007. [4] Y. Caro and Z. Tuza. Improved lower bounds on k- [20] W. H. Kautz and R. C. Singleton. Nonrandom binary independence. J. Graph Theory, 15(1):99–107, 1991. superimposed codes. IEEE Transactions on Informa- [5] H.-B. Chen and F. K. Hwang. A survey on nonadaptive tion Theory, 10(4):363–377, 1964. group testing algorithms through the angle of decoding. [21] A. J. Macula and L. J. Popyack. A group testing J. Comb. Optim., 15(1):49–59, 2008. method for finding patterns in data. Discrete Appl. [6] M. Cheraghchi. Noise-resilient group testing: Limita- Math., 144(1-2):149–157, 2004. tions and constructions. In Proceedings of 17th Inter- [22] H. Q. Ngo and D.-Z. Du. A survey on combinato- national Symposium on Fundamentals of Computation rial group testing algorithms with applications to DNA Theory (FCT), 2009. To Appear. library screening. In Discrete mathematical problems [7] G. Cormode and S. Muthukrishnan. What’s hot and with medical applications (New Brunswick, NJ, 1999), what’s not: tracking most frequent items dynamically. volume 55 of DIMACS Ser. Discrete Math. Theoret. ACM Trans. Database Syst., 30(1):249–278, 2005. Comput. Sci., pages 171–182. Amer. Math. Soc., Prov- [8] R. Dorfman. The detection of defective members idence, RI, 2000. of large populations. The Annals of Mathematical [23] N. Nisan. Pseudorandom generators for space-bounded Statistics, 14(4):436–440, 1943. computation. Combinatorica, 12(4):449–461, 1992. [9] D.-Z. Du and F. K. Hwang. Combinatorial group [24] F. Parvaresh and A. Vardy. Correcting errors be- testing and its applications, volume 12 of Series on yond the Guruswami-Sudan radius in polynomial time. Applied Mathematics. World Scientific Publishing Co. In Proceedings of the 46th Annual IEEE Symposium Inc., River Edge, NJ, second edition, 2000. on Foundations of Computer Science, pages 285–294, [10] A. G. D´yachkov and V. V. Rykov. Bounds on the 2005. length of disjunctive codes. Problemy Peredachi Infor- [25] E. Porat and A. Rothschild. Explicit non-adaptive matsii, 18(3):7–13, 1982. combinatorial group testing schemes. In Proceedings [11] A. G. D´yachkov, V. V. Rykov, and A. M. Rashad. Su- of the 35th International Colloquium on Automata, perimposed distance codes. Problems Control Inform. Languages and Programming (ICALP), pages 748–759, Theory/Problemy Upravlen. Teor. Inform., 18(4):237– 2008. 250, 1989. [26] A. Rudra. List Decoding and Property Testing of Error Correcting Codes. PhD thesis, University of authors present a randomness efficient construction of Washington, 2007. a d-disjunct matrix with t = O(d2 log n) that needs [27] A. Rudra and S. Uurtamo. Data stream algorithms for O(d3 log2 n) random bits. (That is, any entry of the codeword testing, July 2009. Manuscript. matrix can be computed given these bits and with [28] H. Shachnai and A. Srinivasan. Finding large indepen- high probability over the random choices, the resulting dent sets in graphs and hypergraphs. SIAM Journal matrix is d-disjunct.) Thus, the overall number of bits on Discrete Math, 18:488–500, 2004. to be encoded into the layout of the data structure is [29] A. Ta-Shma, C. Umans, and D. Zuckerman. Loss- 3 2 less condensers, unbalanced expanders, and extractors. O(d log n). Given a possibly altered data structure, Combinatorica, 27(2):213–240, 2007. the auditor uses the information stored in the layout of [30] C. Thommesen. The existence of binary linear con- the data structure to recompute the hashes according catenated codes with Reed-Solomon outer codes which to the current values for each test in the matrix. If asymptotically meet the Gilbert-Varshamov bound. the computed hash value differs from the one stored, IEEE Transactions on Information Theory, 29(6):850– it amounts to the test being positive. Computing the 853, November 1983. locations where values were changed now just amount to [31] J. K. Wolf. Born again group testing: multiaccess decoding of the disjunct matrix. Using this framework, communications. IEEE Transaction on Information the authors prove that for balanced binary trees, skip Theory, IT-31:185–191, 1985. lists and arrays (as well as linked lists) that store n [32] D. Zuckerman. Linear degree extractors and the in- items, the scheme above allows the auditor to pin-point approximability of max clique and chromatic number. q q Theory of Computing, 3(1):103–128, 2007. up to O( 3 n/ log2 n), O( 3 n/ log2 n) and O(p4 n/ log n) many changes respectively. (The authors show that A Data Forensics and Group Testing for the layout of the data structures can store Θ(n), Goodrich, Atallah and Tamassia [15] present the follow- Θ(n) and Θ(n log n) bits respectively. The bounds ing application of group testing to data forensics. Data on d are obtained by equating the space requirements forensics refers to the following problem in data struc- for the algorithms to the space available in the data tures and security. Assume that one needs to store a structures.3) data structure on a semi-trusted place. An adversary We remark that in [15], the representation of can change up to d values (out of a total of n values) in the matrix takes up more space than the space re- the data structure (but cannot change the “layout” of quired to store the hash values. Our random- the data structure). The goal is to “store” extra infor- ness efficient construction requires O(max(log n(log d + mation into the data structure so that if up to d values log n log n), d(log d + log log n)2)) = O(d log2 n) many in the data structure are changed then an auditor can random bits, which is better than the bound in [15]. quickly pinpoint the locations where data was changed. Using our result in the framework of Goodrich et al., The authors point out that an auditor generally has we can improve their results to show that for balanced very few resources and thus one needs to store all the binary trees, skip lists and arrays (as well as linked book-keeping information in the data structure itself. lists) that store n items, the auditor can√ pin-point up to p2 p2 The authors consider certain data structures and show O( n/ log n), O( n/ log n) and O( 3 n) many changes that extra information can be encoded into the “layout” respectively. Note that these bounds are the best possi- of the data structure. For example, given n numbers, ble with the techniques of [15]. In addition, given that there are n! possible layouts and thus one can encode our disjunct matrices are efficiently decodable, the audi- O(n log n) bits of information into the layout. (They tor can pin-point the changes faster than the algorithm also show how to do this information “hiding” in other in [15]. We remark that the result of Porat and Roth- data structures such as balanced binary search trees and schild [25] will also provide similar improvements in the skip lists.) number of changes that can be tolerated, though the The question then is what information should be decoding algorithms will be slower than ours. stored in the layout of the data structure such that the auditor can detect the changes. The authors propose B Omitted proofs from Section 3 the use of a d-disjunct matrix (where each of the B.1 Proof of Lemma 3.1 We will only need the columns correspond to the n items) as follows. For binary version of the k-restriction problem, which is each test in the matrix, concatenate all the values defined as follows. The input to the problem is an of the items present in the test and store its hash value. This requires O(t) bits of storage. Further, the 3For the array/linked list data structures, due to technical matrix itself has to be stored. Towards this end, the reasons, the space requirement of the algorithms has be be increased by a factor of d + 1. alphabet Σ = {0, 1}, a length n, and a set of m Theorem B.2. ([12]) Any product distribution on Σn k possible demands fi :Σ → {0, 1}, 1 ≤ i ≤ m. is k-wise efficiently approximable. For every demand fi, there is at least one vector k We are now ready to prove Lemma 3.1. a = (a1, . . . , ak) ∈ Σ such that fi(a) = 1 (in First, we show that constructing (d, `)-list disjunct words, fi “demands” vector a). Thus, every demand k matrices is a special case of the k-restriction problem. fi “demands” a non-empty subset of vectors from Σ . A feasible solution to the problem is a subset S ⊆ Σn Consider an arbitrary t × n (d, `)-list disjunct matrix M. The matrix satisfies the property that, for any set such that: for any choice of k indices 1 ≤ j1 < j2 < of d columns C ,...,C , and any disjoint set C0 ,...,C0 ··· < jk ≤ n, and any demand fi, there is some 1 d 1 ` of ` other columns, there exists a row i of M in which vector v = (v1, . . . , vn) ∈ S such that the projection C0 (i) = ··· = C0 (i) = 0 and C (i) + ··· + C (i) > 0. of v onto those k indices satisfies demand fi; namely, 1 d 1 ` Let k = d + `. Define m = d+` “demands” fi(vj1 , vj2 , . . . , vjk ) = 1. The objective is to find a d k [k] feasible solution S with minimum cardinality. Alon, fI :Σ → {0, 1} as follows. For every d-subset I ∈ , P d Moshkovitz, and Safra gave a couple of algorithmic fI (a) = 1 if and only if aj = 0, ∀j ∈ I and j∈ /I aj > 0. solutions to the k-restriction problems. In order to We just set up an instance of the k-restriction problem. describe their results, we need a few more concepts. A solution S to this instance forms the rows of our t×n Given a distribution D :Σn → [0, 1], the density of matrix. a k-restriction problem with respect to D is Next, let D be the product distribution on {0, 1}n defined by setting each coordinate to be 1 with proba- := min {Pr [f (v , v , . . . , v ) = 1]} v←D i j1 j2 jk bility p to be determined. Then, D is k-wise efficiently 1≤j1<···
Similarly, by using the (q − 1, k)q RS code as the C.1 Proof of Lemma 4.1 Pick q and k such that d is outer code and the code from Corollary C.1 as the q s 1 the largest integer such smaller than k · (s+1)s+1 . Let inner code, we obtain efficient Group Testing with s Cout be the PV code with block length q and dimension O d3 log N(log d + log log N) tests. k and pick the inner code to be Cqs . The required matrix corresponds to the concatenated code C ◦C . Corollary C.2. Given any integers N > d ≥ 1, out in Note that N = qk and t = qs+1. This along with the there exists a t × N d-disjunct matrix M with t = choice of d, implies the claimed bound on t. O(d3 log N(log d+log log N)) that can be decoded in time The list decoding of M s proceeds along ob- poly(t). Furthermore, each entry of M can be computed PV ◦Cqs vious lines: think of the outcome vector as r = in space poly(log t, log N). qs q (r1, . . . , rq) ∈ ({0, 1} ) and decode each ri for Cqs s Proof. Fix d ≥ 1. Pick q, k and Cout as in the proof resulting in a set Si ⊆ [q ] of size |Si| ≤ d (the lat- of Corollary C.1. Then pick Cin to be the concatenated ter is because Cqs is d-disjunct). We will use The- code C∗ corresponding to Corollary C.1 with N = Θ(q). orem 2.1 with α = 1, ` = d, r = 2 and n = q. Note that every codeword in C∗ has the same weight By the choice of d, (2.1) is satisfied. Thus, Theo- ∗ s (say w) and for every two codeword in C , there are rem 2.1 implies that MPV ◦Cqs is (d, L)-list disjunct for at most w/d positions where both of them have a L = O(sO(s)qd/k) = O sO(s)d1+1/s , where the latter value 1. In other words, every column of MRS◦C∗ has equality follows from the choice of parameters. Decod- Hamming weight wq and every two columns agree in at ing the inner codes takes time O(qs+1) while decoding most 2wq/d positions, which by Lemma 2.1 implies that the outer code takes time poly(sO(s), q, d). Thus, by MRS◦C∗ is also d/2-disjunct. Further, by Corollary C.1, the choice of the parameters, the total decoding time is C∗ is (d, d)-list disjunct and can be “list decoded” in poly(t) as required. time poly(d, log q). Note that for MRS◦C∗ , we have Finally, the claim on the low space constructibility 2 3 k s t = O(d k log q) and N = q . of MPV ◦Cqs follows from the fact that any position in s an arbitrary codeword in PV and Cqs can be computed The following corollary presents efficient Group in space poly(s, log q, k) and O(s log q) respectively. Testing with O d3 log N tests and is better than Corollary C.2 in terms of the number of tests, but worse D Disjunct Matrices and Expanders in construction space complexity for non-constant d. To Here is a connection between expanders and disjunct prove the corollary, use the RS code as the outer code matrices. and the code from Theorem 2.2 as the inner code. 1 Corollary C.3. Let d ≥ 1. There exists t × N d- Proposition D.1. Let G be a (n, w, t, 2, w(1 − 3d ))- disjunct matrix M with t = O(d3 log N) that can be expander. Then MG is a d-disjunct matrix. decoded in time poly(t). Further, each entry of M can be computed in space poly(t, log N). Proof. First we show that for any two columns in MG, 2w there are at most 3d positions where both the columns Proof. The idea here is to use RS code as the outer code have a 1. This condition is equivalent to saying that and the code from Theorem 2.2 as the inner code. for any i 6= j ∈ [n], |ΓG({i}) ∩ ΓG({j})| ≤ 2w, where 1 = 3d . This condition in turn implied by the facts that |ΓG({i, j})| ≥ 2w(1 − ) and |ΓG({j})| = w. Finally, note that every column in MG has weight w. Lemma 2.1 completes the proof. The probabilistic method gives the following:
Theorem D.1. There exist (n, w, t, K, w(1 − )) ex- pander with t = O(K log N/2). Theorem D.1 implies that there exists a d-disjunct matrix with O(d2 log n) rows. It is known that any d-disjunct matrix needs to have d2 Ω( log d log n) rows [10, 11, 13]. Thus, Proposition D.1 implies the following lower bound:
Corollary D.1. Let G be a (n, w, t, 2, w(1 − )) ex- log n pander. Then t = Ω 2 log(1/) .