Randomized Methods in Computation – Lecture Notes

Oded Goldreich

Department of Computer Science and Applied Mathematics Weizmann Institute of Science, Israel.

Email: oded@wisdom.weizmann.ac.il

Spring 2001

\begin{matrix}c \\ ◯ \end{matrix}Copyright 2001 by Oded Goldreich. Permission to make copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that new copies bear this notice and the full citation on the first page. Abstracting with credit is permitted.

II

Preface

A variety of randomized methods are being employed in the study of computation. The aim of the current course is to make the students familiar with some of these methods. We wish to stress three aspects regarding this course:

1. This course focuses on methods (i.e., tools and techniques) and not on concepts.

2. These methods are randomized and so the result is often not “(fully) explicit”.

3. The focus is on applications to the study of computation.

Specific topics included:

• Elements of Probability Theory: Linearity of expectation, Markov Inequality, Chebyshev’s Inequality, and Laws of Large Numbers (for pairwise-independent, k-wise independent, and fully independent samples).

• The Probabilistic Method: Applications to Max3SAT and the existence of (3-regular) expander graphs.

• Pairwise-independence, hashing, and related topics: Constructions (via matrices and lowdegree polynomials), and applications (to Max3SAT, Approximate Counting, and Uniform Generation).

• Small bias spaces, the XOR Lemma, application to hardness of MaxQE(2).

• Expander graphs and random walks on them: Eigenvalues versus expansion, analysis of random walks (Hitter), and the Median-of-Average Sampler.

• Some randomized algorithms: Randomized Rounding (applied to Max3SAT), and Primality Testing (via SQRT extraction).

The lecture notes were taken by students attending the course, which was given in the Spring 2001 semester at the Weizmann Institute of Science. The background and level of the students attending the class was quite mixed, and this forced a slower pace than originally planned.

Original Plan

1. Elements of Probability Theory and the Probabilistic Method (2-3 lectures) :

• Linearity of expectation. Example: for every 3CNF there exists a truth assignment that satisfies 7/8 of the clauses.

• Markov Inequality and its underlying reasoning. Application: finding a truth assignment as above in probabilistic polynomial-time.

• Chebyshev’s Inequality and its application to the sum/average of pairwise-independent random variables. Extension to k-wise independent sampling (with a proof).

• Chernoff/Hoeffding Bound (with a proof sketch).

• Using the Probabilistic Method to prove the existence of 3-regular expanders.

• Using the Probabilistic Method to prove the existence of (non-trivial) pseudorandom distributions.

• Lovasz Local Lemma.

2. Pairwise-independence, hashing, and applications (2-3 lectures):

• Constructions: Using linear transformations (by arbitrary or Toeplitz matrices), and using low-degree polynomials (to obtain k-wise independence). Discussion of the technical difficulties regarding the finite field construction.

• Application to derandomizing the above Max3SAT algorithm.

• The (“Leftover”) Hashing Lemma.

• Approximate Counting and Uniform Generation of NP-witnesses with an NP-oracle.

• Hitters and Samplers (first encounter).

3. Small bias spaces, codes, and applications (1-2 lectures):

• The distance from uniform distribution over n-bit strings versus the bias of the XOR of some of the bits.

• A construction of small bias sample spaces.

• Application: NP-hardness of MaxSAT for Quadratic Equations over GF(2).

• Relation to linear error correcting codes.

4. Expander graphs and random walks on them (2-3 lectures):

• Eigenvalues versus expansion.

• Hitter: Analysis of a random walk on an expander.

• The Median-of-Averages Sampler.

5. Randomness-Extractors (1-2 lectures): constructions and applications.

6. Some randomized algorithms (2-3 lectures):

• Finding perfect matchings.

• Randomized Rounding (applied to MaxSAT).

• Primality Testing via Modular Square Root extraction.

State and usage of these notes

These notes are neither complete nor fully proofread, let alone being far from uniformly well-written (although the notes of some lectures are quite good). Furthermore, due to the mixed level of the class, the pace was slower than I originally planned, and so these notes include less material than originally intended. Still, I do believe that these notes suggest a good outline for a course on the subject.

Bibliographic Notes

There are several books and lecture notes that cover parts of the material. These include:

• N. Alon and J.H. Spencer: The Probabilistic Method, John Wiley & Sons, Inc., 1992.

• O. Goldreich. Modern Cryptography, Probabilistic Proofs and Pseudorandomness. Algorithms and Combinatorics series (Vol. 17), Springer, 1998.

Appendix B has been revised and appears as a separated text called “A Taste of Randomized

     Computations”, available from http://www.wisdom.weizmann.ac.il/∼oded/rnd.html.

• O. Goldreich: Introduction to Complexity Theory – Lecture Notes, 1999. Available from

     http://www.wisdom.weizmann.ac.il/∼oded/cc.html.

• R. Motwani and P. Raghavan: Randomized Algorithms, Cambridge University Press, 1995.

However, the presentation of the material in the current lecture notes does not necessarily follow these sources.

In addition to the above general sources, we refer the reader to bibliographic notes presented at the end of each lecture.

VI

Acknowledgments

I am most grateful to the students who have attended the course and partipiated in the project of preparing the lecture notes. So thanks to Omer Angel, Boaz Barak, Uri Bernholz, Eden Chlamtac, Ariel Elbaz, Lev Faivishevsky, Maksim Frenkel, Safro Ilya, Amos Gilboa, Ya’ara Goldschmidt, Olga Grinchtein, Dani Halevi, Yehuda Hassin, Eran Keydar, Sergey Khristo, Shimon Kogan, Amos Korman, Oded Lachish, David Lehmann, Yehuda Lindell, Itsik Mantin, Yaniv Meoded, Shai Mor, Hani Neuvirth, Lior Noy, Eran Ofek, Yoav Rodeh, Alon Rosen, Yaki Setty, Ido Shaked, Denis Simakov, Benny Stein, Eran Tromer, Erez Waisbard, and Udi Wieder.

I am grateful to Shafi Goldwasser who gave a guest lecture (on Primality Testing via SQRT extraction) during the course.

VIII

Lecture Summaries

Lecture 1: Probability and the Probabilistic Method. After a general introduction to the course, we recall some of the basic notions of probability theory, such as the expectation and the variance of a random variable, Chebyshev’s inequality and the first law of large numbers. We also introduce the Probabilistic Method and demonstrate one application of it (specifically, to the Max3SAT problem). Finally, we develop a simple algorithm that given a 3CNF formula, finds an assignment that satisfies many of its clauses.

Notes taken by Maksim Frenkel and Safro Ilya.

Lecture 2: Laws of Large Numbers and Existence of Expanders. We focus on various laws of large numbers. These laws bound the probability that a random variable deviates far from its expectation. Motivated by the problem of approximating the average of a function, we present stronger bounds than Chebyshev’s inequality: moving from pairwise independent sampling to 2k-wise independent sampling and then to totally independent sampling makes the the error probability exponentially vanishing (in the amount of independence). We also define families of graphs called expanders, and prove the existence of a family of 3-regular expanders (using the probabilistic method).

Notes taken by Shai Mor and Erez Waisbard.

Lecture 3: Small Pairwise-Independent Sample Spaces. Often in applications that use randomness (i.e., coin tosses) we do not need all these tosses to be independent, instead it may suffice that they are k-independent for some k. In the case of totally independent random variables the probability space is at least of the size |S|^{m}, where m is the number of variables in the sequence and S is the sample space for an individual variable. In contrast, we present a construction for pairwise independent sample space of the size max(|S|, m)^{2} and its extension to the k-wise independent case. This will allow us to derandomize the algorithm (presented on the first lecture) that finds assignments for 3CNF that satisfies at least 7/8 clauses. The resulting algorithm will conduct an exhaustive search of the relatively small 3-wise independent sample space (of boolean assignments).

Notes taken by Lev Faivishevsky and Sergey Khristo.

Lecture 4: Small Pairwise-Independent Sample Spaces (cont.) and Hash Functions. A good construction of k-wise independent sample space should be small and convenient to use in applications. The construction presented in the previous lecture satisfies the first requirement (i.e., is small), but is rather complex, imposing a number of technical problems in application. In this lecture a second, much more convenient construction is presented, based on affine transformations. This construction yields small sample spaces, but only for pairwise independence (k = 2). We also start discussing hash functions, showing their relation to small pairwise-independent sample spaces, and presenting a Hashing Lemma. The latter asserts that pairwise-independent hash functions map sets in a “smooth” manner (i.e., each image obtains about the same number of preimages).

Notes taken by Olga Grinchtein and Denis Simakov.

Lecture 5: Approximate Counting. We discuss quantitative problems related to NP-relations. Specifically, given an element in an NP-language, we ask how many witnesses this element has. We use Hash functions in order to present an efficient randomized procedure that given an an instance in an NP-language and oracle access to NP, approximates the number of witnesses for that instance.

Notes taken by Oded Lachish, Eran Ofek and Udi Wieder.

Lecture 6: Uniform Generation. This lecture continues the discussion of approximate counting and uniform generation using an NP-oracle begun in the previous lecture, where we saw an approximate counting algorithm. Here we will see the equivalence (up to Turing reductions) of approximate counting to uniform generation of NP-witnesses. Together with the approximate counting algorithm from last lecture, this yields a uniform generation algorithm using an NP-oracle. We conclude with an alternative (direct) uniform generation algorithm, which uses n-wise independent hash functions (as opposed to the pairwise independent hash functions used for approximate counting).

Notes taken by Eden Chlamtac and Shimon Kogan.

Lecture 7: Small Bias Sample Spaces (Part 1). In this lecture we introduce the notion of ϵ-bias sample spaces. Informally, a random variable over {0, 1}^{n} is ϵ-bias if for every subset of bits, the difference between the probability that the parity of the bits is zero and the probability it is one is at most ϵ. We show a connection between ϵ-bias and the statistical distance to the uniform distribution. Next, we present an application of ϵ-bias sample spaces for generating “approximately” k-wise independent distributions (i.e., distributions over {0, 1}^{n} in which every k bits look almost uniformly distributed over {0, 1}^{n}).

Notes taken by Yehuda Lindell and Alon Rosen.

Lecture 8: Small Bias Sample Spaces (Part 2) In this lecture we present a construction of small bias sample spaces; that is, we present an algorithm that given n and ϵ outputs an explicit representation of an ϵ-bias sample space over {0, 1}^{n} and does so within time poly(n/ϵ). We also present an application of such a construction to proving a tight result regarding the hardness of approximating MaxQE (where one is given a sequence of quadratic equations over GF (2) and is asked to find an assignment that satisfies as many as possible of these equations).

Notes taken by Boaz Barak and Itsik Mantin.

Lecture 9: Expansion and Eigenvalues. This lecture is about expander graphs, which are very useful in many probabilistic applications. First, we would describe some of the combinatorial definitions and properties of an expander graph. Second, we would see one application which uses expander graphs, and finally, we would discuss some of the algebraic properties of such graphs.

Notes taken by Ya’ara Goldschmidt, Eran Keydar and Yaki Setty.

Lecture 10: Random Walks on Expanders. In this lecture we show that taking an l-step random walk in an expander graph is in a way similar to choosing l vertices at random. The advantage of using a random walk over taking an independent sample is that a random walk on a d-regular graph requires log_{2} d random bits per each step, whereas selecting each random vertex of an N-vertex graph requires log_{2} N random bits. As a warm up, we show that satring at any vertex in an N-vertex expander and taking O(log N) random steps, one reaches approximately the uniform distribution (on the vertices). We also recall the relation between two definitions of expanders and describe some known constructions of expanders.

Notes taken by Omer Angel, Dani Halevi and Amos Gilboa.

Lecture 11: Primality Testing (via SQRT extraction). In this lecture we present two randomized algorithms in number theory. The first algorithm finds in expected polynomial-time a square root of a number in Z_{p}^{∗} for any prime number p. We then use this algorithm to construct a polynomial time algorithm with two-sided errors for testing primality.

Notes taken by Yehuda Hassin, Lior Noy and Benny Stein.

Lecture 12: Hitters and Samplers. A sampler is an oracle Turing machine that estimates the average of any function f : {0, 1}^{n} → [0, 1] with bounded deviation and bounded probability of failure. A hitter is an oracle Turing machine that, given a boolean function f : {0, 1}^{n} → {0, 1}, finds x s.t. f(x) = 1 with a bounded probability of failure if f has value 1 on at least a constant fraction of {0, 1}^{n}. These are fundamental and useful problems. We consider the randomness and query complexities of hitters and samplers whose running time is polynomially bounded, and recall lower bounds for each. We show several simple constructions for both, as well as improved constructions based on pairwise-independent sample spaces and on random walks on expander graphs. We then show composite constructions whose complexities match the lower bounds.

Notes taken by Amos Korman, Yoav Rodeh and Eran Tromer.

Lecture 13: Randomized Rounding. Some approximation algorithms use linear programming on a variant of the original problem. The solutions of a linear programming problem are non-integral, in the general case, while legitimate solutions to the original problems are integers only. One idea which can be used to turn real numbers into integer solutions, is to use the linear programming (non-integer) solutions as probability parameters for selecting (integer) solutions to the original problem. In this lecture we present this idea and demonstrate it in two cases.

Notes taken by Ariel Elbaz, David Lehmann and Ido Shaked.

XII

Contents

Preface III

Acknowledgments VII

Lecture Summaries IX

1 Probability and the Probabilistic Method 1 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Basic Notions from Probability Theory . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2.1 Outcomes, Events and Probability . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2.2 Random variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2.3 Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2.4 The Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.5 Markov’s and Chebyshev’s Inequalities and a Law of Large Numbers . . . . . 7

1.3 The Probabilistic Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.3.2 An application of the probabilistic method: the 3CNF-SAT problem . . . . . 10 1.3.3 Obtaining a randomized algorithm . . . . . . . . . . . . . . . . . . . . . . . . 11

Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2 Laws of large numbers and Existence of Expanders 13

2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.1.1 A motivating application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.1.2 Independence of random variables . . . . . . . . . . . . . . . . . . . . . . . . 14

2.2 Stronger laws of large numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.2.1 Moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.2.2 Law for 2k-independent trials . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.2.3 The Chernoff/Hoefding bound . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.2.4 Multiplicative Chernoff Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.3 Application: Estimating the mean of a function . . . . . . . . . . . . . . . . . . . . . 18 2.4 Existence of Expanders via the Probabilistic Method . . . . . . . . . . . . . . . . . . 20

2.4.1 Expander Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.4.2 On suitable Sets of graph sizes . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.4.3 On the existence of 3-regular expanders . . . . . . . . . . . . . . . . . . . . . 22 2.4.4 Non-existence of 2-regular expanders . . . . . . . . . . . . . . . . . . . . . . . 25

Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3 Small Pairwise-Independent Sample Spaces 27

3.1 Construction of k-wise independent sample space . . . . . . . . . . . . . . . . . . . . 27 3.2 Annoying technicalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.3 Application to 3CNF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

4 Pairwise-Independent Spaces (cont.) and Hash Functions 37

4.1 A second construction of small pairwise independent sample space . . . . . . . . . . 37 4.1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4.1.2 Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 4.1.3 Analysis: pairwise independence . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.2 Improvement of initial construction (Toeplitz matrices) . . . . . . . . . . . . . . . . . 41 4.2.1 Improved construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 4.2.2 Analysis: pairwise independence . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.3 Hash Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4.3.1 Small pairwise independent sample spaces viewed as hash functions . . . . . 44 4.3.2 Good hash functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4.3.3 Applications of hash functions . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.3.4 Hashing Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5 Approximate Counting 49

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 5.1.1 Formal Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 5.1.2 How close are the quantitative questions to the qualitative ones? . . . . . . . 50

5.2 Approximating the size of of the witness set with oracle access to NP . . . . . . . . . 50 5.2.1 The algorithmic scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 5.2.2 Implementation of the algorithmic scheme . . . . . . . . . . . . . . . . . . . . 51 5.2.3 Analysis of the algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

5.3 Amplifying the success probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

6 Uniform Generation 57 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 6.2 Approximate Counting Reduces to Uniform Generation . . . . . . . . . . . . . . . . 57

6.2.1 Intuition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 6.2.2 Formal Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 6.2.3 Algorithm Approx_{R′} (x, α) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 6.2.4 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 6.2.5 An alternative reduction of Approximate Counting to Uniform Generation . . 60

6.3 Uniform Generation Reduces to Approximate Counting . . . . . . . . . . . . . . . . 61 6.3.1 Intuition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 6.3.2 Formal Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 6.3.3 Algorithm ShApprox_{R′} (x, α) . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 6.3.4 Algorithm UniGen_{R′}(x, α) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 6.3.5 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

6.4 An alternate algorithm for uniform generation . . . . . . . . . . . . . . . . . . . . . . 66 6.4.1 Intuition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 6.4.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 6.4.3 Algorithm UniGen_{R}(x) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 6.4.4 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

7 Small Bias Sample Spaces (Part 1) 69 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 7.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

7.2.1 Statistical Difference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 7.2.2 ϵ-Bias Sample Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

7.3 Statistical Difference versus Maxbias . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 7.3.1 Proof of Theorem 7.5 (Main Result) . . . . . . . . . . . . . . . . . . . . . . . 73 7.3.2 Proof of Lemma 7.3.6 (Technical Lemma from Linear Algebra) . . . . . . . . 78

7.4 (ϵ, k)-Approximations of the Uniform Distribution . . . . . . . . . . . . . . . . . . . 80 7.4.1 (ϵ, k)-approximation of the uniform distribution . . . . . . . . . . . . . . . . . 80 7.4.2 Achieving (ϵ, k)-approximations of uniform . . . . . . . . . . . . . . . . . . . 81 7.4.3 An application of (ϵ, k)-approximations of uniform . . . . . . . . . . . . . . . 81

Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

8 Small Bias Sample Spaces (Part 2) 85 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 8.2 Construction of a Small Bias Sample Space . . . . . . . . . . . . . . . . . . . . . . . 86

8.2.1 On the Existence of poly(\frac{n}{ϵ})-size ϵ-bias sample spaces. . . . . . . . . . . . . . 86 8.2.2 The Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 8.2.3 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

8.3 Using Small Size Sample Spaces for proving Hardness of MaxQE . . . . . . . . . . . 90 8.3.1 The problem of Quadratic Equations . . . . . . . . . . . . . . . . . . . . . . . 91 8.3.2 The Optimization Problem MaxQE . . . . . . . . . . . . . . . . . . . . . . . 92 8.3.3 MaxQE is hard to (\frac{1}{2} + ϵ)-approximate . . . . . . . . . . . . . . . . . . . . . . 93

Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

9 Expanders and Eigenvalues 97

9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 9.1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 9.1.2 Family of Expander Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 9.1.3 Definitions of Neighborhood Sets . . . . . . . . . . . . . . . . . . . . . . . . . 98 9.1.4 Definitions of Expander Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . 98 9.1.5 Constructibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

9.2 On the Diameter of Expander Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . 99 9.3 Amplification of Expander Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

9.3.1 The Amplified Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 9.3.2 Constructibility of G_{N}^{k} . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

9.4 An Application of Expander Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 9.4.1 The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 9.4.2 The Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 9.4.3 Analysis of the Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 9.5 Algebraic Definition of Expanders . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

9.5.1 The Normalized Adjacency Matrix . . . . . . . . . . . . . . . . . . . . . . . . 103 9.5.2 Eigenvectors and Eigenvalues of the Matrix . . . . . . . . . . . . . . . . . . . 104

Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

10 Random Walks on Expanders 107 10.1 Two Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 10.2 Known Constructions of Expanders . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 10.3 Random Walks on Expanders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

10.3.1 Mixing Time of Random Walks . . . . . . . . . . . . . . . . . . . . . . . . . . 109 10.3.2 A random walk yields a sequence of “good samples” . . . . . . . . . . . . . . 111

Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

The Expander Mixing Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 The Expander Smoothing Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

11 Square roots in Z_{p}^{∗} and Primality Testing 117 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 11.2 Definitions and Simple Facts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

11.2.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 11.2.2 Some facts about Z_{p}^{∗} . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

11.3 Finding a Square Root . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 11.3.1 Description of the Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 11.3.2 An Algorithm For Finding the Square-Root (mod p) . . . . . . . . . . . . . . 121 11.3.3 A Monte-Carlo version of the SQRT algorithm . . . . . . . . . . . . . . . . . 124

11.4 Randomized Algorithm for Primality Testing . . . . . . . . . . . . . . . . . . . . . . 124 11.5 Short proof for prime numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

12 Hitters and Samplers 129

12.1 Definitions and Lower Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 12.1.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 12.1.2 Complexity Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 12.1.3 Lower Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

12.2 Simple Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 12.2.1 Constructing a Hitter from a Boolean Sampler . . . . . . . . . . . . . . . . . 134 12.2.2 A Naive Hitter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 12.2.3 A Naive Sampler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

12.3 Better Constructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 12.3.1 A Pairwise-Independent Hitter . . . . . . . . . . . . . . . . . . . . . . . . . . 136 12.3.2 A Pairwise-Independent Sampler . . . . . . . . . . . . . . . . . . . . . . . . . 137 12.3.3 An Expander-Walk Hitter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

12.4 Composed Constructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 12.4.1 A Composed Hitter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 12.4.2 A Composed Sampler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 13 Randomized Rounding 145

13.1 The General Idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 13.2 Application to Max-SAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

13.2.1 The Basic Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 13.2.2 Improving the Max-SAT approximation algorithm . . . . . . . . . . . . . . . 149

13.3 Application to the Minimal-Set-Cover problem . . . . . . . . . . . . . . . . . . . . . 150 13.3.1 The approximation algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 13.3.2 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

Bibliography 153

Lecture 1

Probability and the Probabilistic Method

Notes taken by Maksim Frenkel and Safro Ilya

Summary: This lecture is a general introduction to the course “Randomized methods in Computation”. We review some of the basic notions of probability theory, such as the expectation and the variance of a random variable, Chebyshev’s inequality and the first law of large numbers. We also introduce the Probabilistic Method and demonstrate one application of it (specifically, to the Max3SAT problem). Finally, we will develop a simple algorithm that given a 3CNF formula, finds an assignment that satisfies many of its clauses.

1.1 Motivation

Oded’s Note: I tried to stress three aspects regarding this course as hinted by its name: First, this course focuses on methods (i.e., tools and techniques) and not on concepts. Second, these methods are randomized and so the result is often not “(fully) explicit”. This can be seen both in the form of randomized algorithms (and procedures) that do not always return the same answer (but rather have an output distribution that satisfies some desired property with high probability) as well as in arguments (e.g., in the “probabilistic method”) that do not explicitly present a desired object (but rather only specifies a distribution of objects so that with high probability an object in this distribution possesses the desired property). Lastly, the third aspect of this course is that it focuses on applications to the study of computation. This is reflected in the emphasis put on the complexity of generating certain distributions, where complexity may mean a variety of complexity measures (such as time, space, and randomness).

Consider the following problem: given a set of integer numbers, sort them in an ascending order. There exist a lot of different algorithms that solve this problem. These algorithms have different worst and average running times. Some of them are kno wn to perform badly on certain problem instances (for example the QuickSort algorithm) while on the average their running times are good. A possible solution is to randomly permute the given integer sequence such that it will resemble the average case inp ut. In this example our algorithm will use the sequence of outcomes of coin tosses which explicitly determine the permutation. Such an algorithm is called randomized.

A randomized algorithm gets as its input a sequence of n outcomes of coin toss in addition to its regular input. The algorithm makes its decisions based upon this sequence. Standard (deterministic) algorithm has one answer given one input instance. Randomized algorithm has a set of answers (or a set of ways to reach an answer) given the problem input instance. Some of these answers (or ways to reach an answer) are good whenever other are bad. The advantage of a good randomized algorithm is its abil ity to reach the good answer (or to find a good way to this answer) fast.

Oded’s Note: The above motivation (which is not at all what I said in class) addresses just one aspect of the course. See above note for what I did say.

1.2 Basic Notions from Probability Theory

Oded’s Note: I assume that most students know most of the contents of the current section. My goals in covering this basic material were two-fold. First, to notify all student of what they are assumed to know, and teach the few that don’t know some of this material the little they don’t know. Second, in a confusing business such as probabilistic analysis it is good to get reminded of the basic underlying definitions: in case of difficulty one should remember that the complex notions can and should (when in doubt) be traced back to the underlying notions. That is, when faced with a complex analysis it is a good practice to ask basic questions like “what is the underlying probability space” (which is typically only implicit in complex discussions).

1.2.1 Outcomes, Events and Probability

We assume that the reader is familiar with the basic notions of Probability Theory. Therefore the purpose of this section will be to review those concepts, which will be useful for us in upcoming lectures.

Let us consider an experiment that results in several outcomes. Denote these outcomes by A_{1}, A_{2}, ..., A_{m}. If we will repeat this experiment n times under various conditions, A_{1} will occur k_{1} times, while other outcomes will occur n − k_{1} times. We refer to a certain subset of the outcomes as an event and to the set of all possible outcomes as a probability space Ω. When we talk about the probability of the event X occurring over its probability space Ω we consider the ratio of the sizes of the two sets: \frac{|X|}{|Ω|}. The meaning of this ratio is the answer to the question: how to measure the possibility that the event X will occur as a result of our experiment. This ratio we denote by P rob(X). It is easy to see that

0 ≤ P rob(X) ≤ 1

Example 1.1 Suppose we are rolling a die and we are interested in obtaining an even outcome. Here

Ω = {1, 2, 3, 4, 5, 6}

The event X will be the following subset of Ω:

X = {2, 4, 6}

Then

P rob(X) = \frac{|X|}{|Ω|} = \frac{3}{6}

Remark 1.2.1 In this course we will consider the probability space to be a f̄inite set.

Sometimes the outcome of one event may influence the outcome of another one. To measure the amount of influence we introduce the conditional probability of event B occurring given that the event A has occurred:

P rob(B|A) = \frac{P rob(A ∩ B)}{P rob(A)}

Recall that events can be either independent or dependent. The events are independent when the trials of our experiment are carried out independently, which implies that the result of each trial does not influence the other results. If we are interested in the probability of the events A and B occurring when they are independent then

P rob(A ∩ B) = P rob(A) · P rob(B)

In other words, A and B are independent if the conditional probability of B given A is just equal to the probability that B occurs without paying attention to A at all, i.e.

P rob(B|A) = P rob(B)

Oded’s Note: Indeed, probabilities are merely ratios of sizes of sets, and so probabilistic analysis is merely combinatorics. Yet, as is often the case in mathematics and science, using certain definitions and notations (rather than others) may simplify the analysis tremendously. Specifically, in many cases, the analysis is much simpler to carry out in terms of probabilities than in terms of set sizes.

1.2.2 Random variables

A random variable is a function from a probability space to the reals (or complexes); that is, a random variable X is

X : Ω ⟶ R (possibly C)

As we remarked earlier, Ω in our case is a finite set and hence the set of values that X is mapped onto is also finite. Intuitively a random variable is a function that can be calculated as a result of an experiment. The adjective “random” reminds us that we are dealing with a sample space and are trying to describe “random events”. Random variables are often defined without explicitly specifying a probability space (but such a space is always implicit in the definition). For example, the number of hits out of three shots, the number of tails in five coin tosses, etc. Once we have defined what our random variable measures we can define the probability that it will take on a certain value α:

P rob[X = α] = \frac{|\{w ∈ Ω : X(w) = α\}|}{|Ω|}

Let us remind also that if X and Y are random variables then

X + Y, X − Y, XY, \frac{X}{Y} , αX + βY

(where α and β are real numbers) are also random variables.

Discrete random variables X and Y are independent if

P rob(X = x and Y = y) = P rob(X = x) · P rob(Y = y)

for all values x in the range of X and y in the range of Y.

By definition of a random variable one can not define it without specifying the corresponding probability space. However, often, when the associated probability space is obvious we will drop this convention. We will usually describe the set of values that can be associated with the random variable with their corresponding weights, instead of explicitly defining the function with its domain.

Oded’s Note: Indeed the last paragraph refers to my main motivation in recalling all these trivialities. We will often talk of random variables without specifying the underlying probability space, and will often be able to carry out the analysis that way. However, when in doubt, one should specify explicitly the probability space that underliees the analized events or random variables.

1.2.3 Expectation

For a random variable X defined on a probability space Ω, its expectation is the number E(X) defined by the formula

E(X) = ∑_{ω∈Ω} P rob({ω})X(ω)

If we think of P rob({ω}) as the weight attached to ω then E(X) can be understood as the weighted average of the function X. The intuition of the value E(X) might be obtained from its analogue in mechanics. Consider a line with n physical points distributed on it. Every point has a weight p_{i}. Let X_{i} be the location of the i-th point. Taking the sum of all the weights to be 1, the equilibrium point of the system has the coordinate:

X = \frac{∑_{i=1}^{n} p_{i}X_{i}}{∑_{i=1}^{n} p_{i}} = \frac{∑_{i=1}^{n} p_{i}X_{i}}{1} = ∑_{i=1}^{n} p_{i}X_{i}

This is exactly the expectation of the random variable X. The following theorem introduces some properties of the expectation. These properties will be very useful in the future discussion.

Proposition 1.2.2 For two random variables X and Y,

1. E(aX) = aE(X), for any real a.

2. E(X+Y) = E(X) + E(Y), even if X and Y are dependent.

3. If X and Y are independent then E(XY)=E(X)E(Y).

Proof: Let X be the random variable that takes on discrete values and let a be a number. Let aX be defined on the set Ω, then

E(aX) = ∑_{v} v · P rob(aX = v)
= a · ∑_{v} \frac{v}{a} · P rob(X = \frac{v}{a})
= a · E(X)

In order to prove Part 2 let us assume that Y takes on discrete values.

E(X + Y) = ∑_{(x,y)}[(x + y) · P rob(X = x and Y = y)]
= ∑_{x} ∑_{y} x · P rob(X = x and Y = y) + ∑_{y} ∑_{x} y · P rob(X = x and Y = y)
= ∑_{x} x · P rob(X = x) + ∑_{y} y · P rob(Y = y)
= E(X) + E(Y)

And finally, for independent X and Y, we have

E(XY) = ∑_{x} ∑_{y} xy · P rob(X = x and Y = y)
= ∑_{x} ∑_{y} xy · P rob(X = x) · P rob(Y = y)

= (∑_{x} x · P rob(X = x)) · (∑_{y} y · P rob(Y = y))
= E(X) · E(Y)

In general, if X_{1}, ...X_{n} are random variables and X = α_{1}X_{1} + ... + α_{n}X_{n}, then

E(X) = α_{1}E(X_{1}) + ... + α_{n}E(X_{n})

The linearity of expectation is a very useful tool in randomized methods. Its power comes from the idea that it may be possible to decompose a complex instance X as the sum of many simple instances X_{i}, and the linearity will hold regardless of the dependence of X_{i}’s. The expectation of each of the X_{i}-s can be calculated relatively easy compared to that of X. We will see an application of this principle in Theorem 1.6. Another useful fact regarding the expectation is that it cannot be always higher than the random variable. That is:

Proposition 1.2.3 For any random variable X

P rob[X ≥ E(X)] > 0

Proof Sketch: We need to show that there exists x, such that P rob[X = x] > 0 and x ≥ E(X). Let x_{max} be the maximal value that X may be equal to such that P rob[X = x_{max}] > 0 and P rob[X > x_{max}] = 0.

E(X) = ∑_{x} P rob[X = x] · x ≤ ∑_{x≤x_{max}} P rob[X = x] · x_{max} = x_{max}

Therefore E(X) ≤ x_{max} and since we know that P rob[X = x_{max}] > 0 we have shown that P rob[X ≥ E(X)] > 0.

1.2.4 The Variance

Oded’s Note: The following paragraph is not something I said. I would have said that the variance is a measure of the “spread” or “variability” of a random variable, and that its specific definition is adequate for various Laws of Large Numbers as well as for Chebyshev’s Inequality that underlies some of them.

The variance of a random variable X measures the unpredictability of a random variable or in other words this is a measure of its variation from the mean E(X). Suppose that two random variables X and Y have the same mean μ, but the values of X are distributed more closely to E(X) than are the values of Y to E(Y). That is, the probability is higher that X will be closer to μ than Y will be. We would like to define some measure that will reflect that fact. Let us shift both X and Y to 0 and consider the values X − μ and Y − μ. Since Y was spread farther around μ than X was, Y − μ will be spread farther around 0 than X − μ. Let’s square the above differences to make sure that they are positive. The expectation of (X − μ)^{2} will be closer to 0 than (Y − μ)^{2}. Therefore it makes sense to define the variance of a random variable X as follows:

V AR(X) = E((X − μ)^{2}), where μ = E(X)

Using the linearity of expectation, we see that

V AR(X) = E((X − μ)^{2})
= E(X^{2} − 2μX + μ^{2})
= E(X^{2}) − 2μE(X) + E(μ^{2})
= E(X^{2}) − 2μμ + μ^{2}
= E(X^{2}) − E(X)^{2}

The above formula provides a computationally easier method of evaluating the variance.

The variance is not always linear. An exceptional case when the variance is linear is when applied to a sum of paiwise-independent random variables. Following is a special case (which can be easily generalized to a sequence of pairwise independent random variables):

Proposition 1.2.4 If X and Y are independent then

V AR(X + Y) = V AR(X) + V AR(Y)

Proof: Since X and Y are independent, E(XY) = E(X) · E(Y) = μ_{X}μ_{Y}, where μ_{X} and μ_{Y} equal to E(X) and E(Y) respectively. Then

V AR(X + Y) = E((X + Y)^{2}) − (μ_{X} + μ_{Y})^{2}
= E(X^{2} + 2XY + Y^{2}) − (μ_{X}^{2} + 2μ_{X}μ_{Y} + μ_{X}^{2})
= E(X^{2}) − μ_{X}^{2} + E(Y^{2}) − μ_{Y}^{2} + 2E(XY) − 2μ_{X}μ_{Y}
= V AR(X) + V AR(Y)

since the last two terms cancel.

On the other hand, in case X and Y are dependent V AR(X +Y) may be different from V AR(X)+ V AR(Y). In particular, if X = Y (i.e., P rob(X = Y) = 1) then V AR(X + Y) = 2V AR(X) + 2V AR(Y), whereas if X = −Y than V AR(X + Y) = 0. Both claims follows from the next proposition:

Proposition 1.2.5 Let a and b be constants. Then

V AR(aX + b) = a^{2} · V AR(X)

Proof: Let μ = E(X) and observe that

E(aX + b) = aE(X) + b = aμ + b

Now

V AR(aX + b) = E((aX + b)^{2}) − (E(aX + b))^{2}
= E(a^{2}X^{2} + 2abX + b^{2}) − (aμ + b)^{2}
= a^{2} · E(X^{2}) + 2ab · E(X) + b^{2} − (a^{2}μ^{2} + 2abμ + b^{2})
= a^{2} · (E(X^{2}) − μ^{2})
= a^{2} · V AR(X)

Note that the above result is intuitive, since the values of the random variable aX + b vary from its expectation E(aX + b) as do the values of aX from its expectation E(aX). On the other hand, simply shifting the distribution by b does not change its spread. Another intuitive property of the variance follows:

Proposition 1.2.6 The variance of a random variable X is zero if and only if X is a constant.

Proof: The backward direction (⇐) is trivial and follows from Proposition 1.2.5, when a = 0. The forward direction follows from the definition of the variance. Specifically, we have

V AR(X) = E((X − μ)^{2}) = 0

where μ = E(X). Since (X −μ)^{2} is a nonnegative value, the fact that its expectation is zero implies that X = μ.

1.2.5 Markov’s and Chebyshev’s Inequalities and a Law of Large Numbers

In analyzing the performance of a randomized algorithm we will often need to show that the algorithm behaves well almost all the time. While the concept of the expectation of a random variable provides information regarding the average case scenario (for instance, the algorithm’s average running time), lots of times we will interested in the “typical” (rather than average) behavior; that is, we want to know an interval of values that is guaranteed to contain the value of the random variable, almost always. In doing so we will need to analyze the probability that a random variable deviates from its expectation by a given amount.

The first inequality we introduce provides a bound on the probability that a random variable deviates from its expectation.

Theorem 1.2 (Markov’s Inequality): If X is a random variable that assumes only non-negative values, then for all real and positive v

P rob[X ≥ v] ≤ \frac{E(X)}{v}

Proof:

E(X) = ∑_{x} P rob[X = x] · x
= ∑_{x≥v} P rob[X = x] · x + ∑_{x<v} P rob[X = x] · x
≥ ∑_{x≥v} P rob[X = x] · v + ∑_{x<v} P rob[X = x] · 0
= P rob[X ≥ v] · v

Thus

P rob[X ≥ v] ≤ \frac{E(X)}{v}

An equivalent formulation of Markov inequality is :

P rob[X ≥ α · E(X)] ≤ \frac{1}{α}

As we can see the upper bound of the probability that a random variable deviates from its expectation decreases as α increases. As noted before the probability space of X is not directly mentioned in the above theorem but it is obviously implied. Usually we use the Markov Inequality when our knowledge about the distribution is limited to its expectation and at least one bound on the distribution of a random variable. If the variance of a given random variable X is known, the Markov inequality can be used to achieve a better bound on the probability that X will deviate from its expectation, as seen in the next theorem.

Theorem 1.3 (Chebyshev’s Inequality): For any random variable Y and any δ > 0,

P rob[|Y − E(Y)| ≥ δ] ≤ \frac{V AR(Y)}{δ^{2}}

Proof: Let X = (Y − E(Y))^{2} and note that X is non-negative and that

E(X) = E((Y − E(Y))^{2}) = V AR(Y)

Next, observe that

P rob[|Y − E(Y)| ≥ δ] = P rob[X ≥ δ^{2}]

Finally, invoking Markov’s inequality on X and δ^{2} yields

P rob[X ≥ δ^{2}] ≤ \frac{E(X)}{δ^{2}} = \frac{V AR(Y)}{δ^{2}}

which completes the proof. Indeed, as expected for any distribution, the probability that the values of X are more than a certain fixed distance away from its mean gets smaller as V AR(X) gets smaller. A special case where this is useful is for the sum of pairwise independent random variables.

Definition 1.4 The random variables X_{1}, ..., X_{n} are called pairwise independent if for every i ≠ j and all a and b,

P rob[X_{i} = a ∧ X_{j} = b] = P rob[X_{i} = a] · P rob[X_{j} = b]

Corollary 1.2.7 Let X_{1}, ..., X_{n} be pairwise independent random variables with common mean μ and variance σ^{2}. Denote S_{n} = ∑_{i=1}^{n} X_{i}, and let \overline{X} = S_{n}/n. Then

P rob[|\overline{X} − μ| ≥ ϵ] ≤ \frac{σ^{2}}{nϵ^{2}}

Proof: First, observe that V AR(S_{n}) = nσ^{2}, by (a generalization of) Proposition 1.2.4.

Oded’s Note: Specifically, we use V AR(∑_{i=1}^{n} X_{i}) = ∑_{i=1}^{n} V AR(X_{i}), which is proven as Proposition 1.2.4:

V AR(∑_{i=1}^{n} X_{i}) = E ((∑_{i=1}^{n} X_{i})^{2}) − E (∑_{i=1}^{n} X_{i})^{2}

= E ((∑_{i=1}^{n} X_{i}^{2}) + (∑_{i≠j} X_{i}X_{j})) − (∑_{i=1}^{n} E(X_{i}))^{2}

= ∑_{i=1}^{n} E(X_{i}^{2}) + ∑_{i≠j} E(X_{i}X_{j}) − ∑_{i=1}^{n} E(X_{i})^{2} − ∑_{i≠j} E(X_{i})E(X_{j})

= ∑_{i=1}^{n} E(X_{i}^{2}) − ∑_{i=1}^{n} E(X_{i})^{2}

= ∑_{i=1}^{n} V AR(X_{i})

where the key fact being used is E(X_{i}X_{j}) = E(X_{i})E(X_{j}), for i ≠ j, which follows becuase X_{i} and X_{j} are independent.

Now, by Proposition 1.2.5:

V AR(\overline{X}) = V AR(S_{n}/n) = \frac{V AR(S_{n})}{n^{2}} = \frac{σ^{2}}{n}

Finally, invoking Chebyshev’s inequality on \overline{X}, we get

P rob[|\overline{X} − μ| ≥ ϵ] ≤ \frac{V AR(\overline{X})}{ϵ^{2}} = \frac{σ^{2}}{nϵ^{2}}  (1.1)

Remark 1.2.8 The above corollary generalizes to the case where the X_{i}’s are pairwise independent, but not necessarily with identical mean. In such a case, we let μ = \frac{1}{n} ∑_{i=1}^{n} E(X_{i}) and have:

P rob[|\overline{X} − μ| ≥ ϵ] ≤ \frac{max_{i} V AR(X_{i})}{nϵ^{2}}

1.3 The Probabilistic Method

1.3.1 Introduction

The probabilistic method is a way to prove the existence of objects having certain properties, by proving their existence “in abundence” (with respect to some probability spcae). That is, suppose we want to prove the existence of some combinatorial object with property T. The construction of such an object may be too hard. The probabilistic method suggests to define some probability space S, and to show that the probability that an object selected uniformly in S satisfies property T with non-zero probability. This would imply the existence of such an object (i.e., satisfying property T).

1.3.2 An application of the probabilistic method: the 3CNF-SAT problem

Definition 1.5 A 3CNF boolean formula is a boolean formula φ written in conjunctive normal form where each clause contains exactly three different literals:

φ = ⋀_{i=1}^{m} C_{i}, where C_{i} = ⋁_{j=1}^{3} l_{ij}

each literal l_{ij} can be either a boolean variable or negation of a boolean variable.

Theorem 1.6 For every 3CNF boolean formula φ there exists a truth assignment that satisfies at least \frac{7}{8} of clauses in φ.

Note that the theorem is not true in case we consider arbitrary CNF formulae (or CNF formulae in which each clause may have any three literals that are not necessarily different). Consider for example, the formula

(x_{1} ∨ x_{2}) ∧ (¬x_{1} ∨ x_{2}) ∧ (x_{1} ∨ ¬x_{2}) ∧ (¬x_{1} ∨ ¬x_{2})

Clearly, each possible truth assignment satisfies only 3 out of the 4 possible clauses. A similar example shows that the Theorem 1.6 is indeed optimal: there exists 3CNF boolean formulae for which any truth assignment satisfies only 7/8 fraction of the clauses.

Proof: For sake of simplicity, we assume that no clause of φ may contain two literals corresponding to the same variable (i.e., a boolean variable and its negation). Note that any clause that conatins a boolean variable and its negation is satisfied under any truth assignment. Thus, we may omitting such clauses, observing that any assignment that satisfies 7/8 of the remaining clauses also satisfies all the omitted clauses.

The idea of the proof is to select a random assignment to the variables of φ uniformly (over all possible choices) and to evaluate the expected number of clauses satisfied by it. That is, we take as a probability space the set of all possible assignments to the variables of φ. Choose a random assignment and let X be the number of clauses (of φ = ⋀_{i=1}^{m} C_{i}) satisfied by it. Let X_{i} be the indicator random variable such that

X_{i} = \{ \begin{matrix}1 & if the assignment satisfies C_{i} \\ 0 & otherwise \end{matrix}  (1.2)

For φ = ⋀_{i=1}^{m} C_{i}, we have X = X_{1} + ... + X_{m}. These X_{i}-s are not necessarily independent, because some clauses may contain the same literals (or just the same variables). Nevertheless, this fact does not prevent us from considering the expectation of X and using linearity of expectation:

E(X) = E(∑_{i=1}^{m} X_{i}) = ∑_{i=1}^{m} E(X_{i})

Thus we have broken down the relatively complicated problem of calculating the value E(X) into a number of simpler problems of calculating each of the E(X_{i})-s. Suppose that the clause C_{j} contains the three literals l_{j1}, l_{j2}, l_{j3}. Since these three literals are of different variables, there are 8 possible assignments to these three literals and 7 of them satisfy C_{j}. Since each of these 8 possibilities is equaly likely, when an assignment to all variables (of φ) is selected uniformly, we get:

E(X_{i}) = P rob[X_{i} = 1] = \frac{7}{8}

So E(X) = \frac{7}{8}m. Now as we already observed (see Proposition 1.2.3) P rob[X ≥ E(X)] > 0, so we can say that in our case P rob[X ≥ \frac{7}{8}m] > 0. This implies the existence of a truth assignment for φ that satisfies at least \frac{7}{8} of its clauses.

In the above proof we have defined the probability space Ω of all possible truth assignments to a given 3CNF formula, and then shown that the probability of choosing an assignment from Ω that satisfies the required property is non-zero. We then concluded that an assignment as required does exist.

1.3.3 Obtaining a randomized algorithm

In the previous section we have shown that there exists an assignment that satisfies at least \frac{7}{8} of the clauses in a given 3CNF formula. Here we will present a randomized algorithm that finds such an assignment with high probability in polynomial time. The idea behind this algorithm will be to repeteadly select random assignments to the variables of the given formula, untill it finds an assignment that satisfies at least \frac{7}{8} of the clauses. The expected running time of the algorithm depends on the probability that the chosen random assignment satisfies at least \frac{7}{8} of the clauses. The following claim will show that this probability is quite noticeable (i.e, bigger than a polynomial fraction):

Claim 1.3.1 Let φ be a 3CNF formula with m clauses. Let X be the random variable that counts the number of clauses satisfied by a uniformly random assignment to the variables of φ. Then

P rob(X ≥ \frac{7}{8}m) ≥ \frac{1}{m + 1}

Proof: As in the proof of Theorem 1.6, we assume for simplicity (and without loss of generality) that each clause in φ consists of three literals corresponding to three different variables.

Recall that E(X) = ∑_{x} x · P rob[X = x]. We can decompose this sum into two components. The first component G will represent the values of X that are greater or equal to E(X). The second component R will represent the rest (i.e. the other possible x-s). Thus

E(X) = ∑_{x∈G} xP rob[X = x] + ∑_{x∈R} xP rob[X = x]

The possible values that X can get are integers. Let T be the largest integer that is smaller than E(X). Using the definition of T and the fact that X is an integer, we have that X < E(X) implies X ≤ T. Thus:

E(X) ≤ m · P rob(X ≥ E(X)) + T · P rob(X < E(X))  (1.3)

On the other hand

P rob(X ≥ E(X)) + P rob(X < E(X)) = 1

Denote by α the quantity P rob(X ≥ E(X)). Then P rob(X < E(X)) = 1 − α. Hence, plugging the appropriate values into Equation 1.3 above, we obtain

E(X) ≤ mα + T · (1 − α)  (1.4)

Recalling the definition of T, we have

T = ⌈E(X) − 1⌉ ≤ E(X) − \frac{1}{8}

where the inequality holds because E(X) = \frac{7}{8}m, which equals a + \frac{b}{8} for some non-negative integers a and b so that b ≤ 7. (Extra help: if b = 0 then ⌈a + \frac{b}{8} − 1⌉ = a − 1 < (a + \frac{b}{8}) − \frac{1}{8}, whereas if b > 0 then ⌈a + \frac{b}{8} − 1⌉ = a ≤ (a + \frac{b}{8}) − \frac{1}{8}.) Thus, Equation 1.4 yields:

E(X) ≤ mα + (E(X) − \frac{1}{8}) · (1 − α)  (1.5)

By a sequence of manipulations, we get the following sequence of equivalent conditions:

αE(X) ≤ mα − \frac{1 − α}{8}  (1.6)
\frac{1}{8} ≤ (m − E(X) + \frac{1}{8}) · α  (1.7)
α ≥ \frac{1}{8m − 8E(X) + 1} = \frac{1}{8m − 7m + 1}  (1.8)

We conclude that α ≥ \frac{1}{m+1}, and the claim follows.

Having proved the above claim, we can now present a randomized algorithm which will get a formula φ as its input and output an assignment to the variables of φ that will satisfy \frac{7}{8} of its clauses with exponentially high probability. The algorithm is as hinted above: it will selected at random truth assignments to the variables of φ until finding one as desired (which happens with probability at least 1/|φ| at each attempt). In fact, we prefer to present a version in which the number of attempts is bounded although the algorithm fails with small probability. The algorithm is as follows:

Repeat Steps 1–3 for |φ|^{2} times 1. select a random assignment for the formula φ

2. check how many clauses of φ the chosen assignment satisfies

3. if the assignment satisfies at least \frac{7}{8} of the clauses then output the assignment

Each iteration of the algorithm takes polynomial time, and so does the entire algorithm. Using the Claim 1.3.1 the probability that the algorithm will succeed in each single iteration is at least \frac{1}{m+1}, and this is at least \frac{1}{|φ|} (since |φ| > m). Thus, the probability that the algorithm will fail in one iteration is less or equal to 1 − \frac{1}{|φ|}. It follows that the probability that we fail in all |φ|^{2} iterations is less or equal to (1 − \frac{1}{|φ|})^{|φ|^{2}} which is approximately e^{−|φ|}. We conclude that the probability that algorithm succeeds is at least 1 − e^{−|φ|}.

Bibliographic Notes

The Probabilistic Method is the topic of an excellent book by Alon and Spencer [10], and the reader is strongly encourage to learn more about it from that source.

Laws of large numbers and Existence of Expanders

Notes taken by Shai Mor and Erez Waisbard

Summary: The focus of the lecture is various laws of large numbers. These laws bound the probability that a random variable deviates far from its expectation. Motivated by the problem of approximating the average of a function, we present stronger bounds than we had in the previous lecture. Moving from pairwise independent sampling to 2k-wise independent sampling and then to totally independent sampling makes the the error probability exponentially vanishing (instead of linearly vanishing). We also define families of graphs called expanders, and prove the existence of a family of 3-regular expanders (using the probabilistic method).

2.1 Introduction

2.1.1 A motivating application

Loosely speaking the law of large numbers tells us that we can give a good estimation for the average of a bounded function by sampling enough points. Lets say we have a bounded function f : {0, 1}^{n} → {0, 1}. It is infeasible for us to compute

f̄ =^{def} \frac{1}{2^{n}} ∑_{x∈\{0,1\}^{n}} f(x)

in time which is polynomial in n, however if we sample enough points we can give a good estimation that will be correct with high probability.

With respect to such sampling, we can associate a random variable X_{i} withe the value of f obtained at each sample point. Clearly each X_{i} has expected value f̄, and if we take enough samples then the average of the corresponding X_{i}’s is very likely to be close to f̄. Our focus here is on quantitative statements of the above intuition.

In the previous lecture we gave the following bound for a set of pairwise independent random variables {X_{1} … X_{m}}:

P rob [|(\frac{1}{m} ∑_{i=1}^{m} X_{i}) − (\frac{1}{m} ∑_{i=1}^{m} E(X_{i}))| ≥ δ] ≤ \frac{max_{i}\{V ar(X_{i})\}}{mδ^{2}}  (2.1)

We see that the more X_{i}’s we have, the better our estimation gets. This is really what the law of large numbers is all about. It says that if you give me more random variables of the same distribution then I can give a better estimation for the average of their expectation. A crucial point is that they have to be at least pairwise independent. This property enables us to get a good error bound. We make two observations looking at the above inequality:

1. There is a tradeoff between the deviation we want to bound and the error probability. Naturally we would like to be able to improve the tradeoff by both reducing the deviation and dropping the error probability.

2. The above bound gives us an error probability that is linearly dropping in the number of samples (i.e., m).

By imposing a stronger than pairwise independence restriction on the random variables, we can get a stronger dependence of the error-bound on the number of samples. Loosely speaking, the error probability is exponentially vanishing with the level of independence (between the random variables).

2.1.2 Independence of random variables

We now define independence between random variables. In the previous lecture we defined pairwise independence. We now define total independence between random variables.

Definition 2.1 Let X_{1} … X_{m} be random variables. We say that they are totally independent if for every m values v_{1} … v_{m}, the following holds

P rob[X_{1} = v_{1} ∧ … ∧ X_{m} = v_{m}] = ∏_{i=1}^{m} P rob[X_{j} = v_{j}]

Total independence and pairwise independence are two types of notions of independence of random variables. The notion of independence can be extended to express independence of subsets of certain size of random variables.

Definition 2.2 Let X_{1} … X_{m} be random variables. We say that they are t-wise independent if for every subset of t of them, X_{i_{1}} … X_{i_{t}}, and for every values v_{1} … v_{t}, the following holds

P rob[X_{i_{1}} = v_{1} ∧ … ∧ X_{i_{t}} = v_{t}] = ∏_{i=1}^{t} P rob[X_{i_{j}} = v_{j}]

In other words, X_{1} … X_{m} are t-wise independent, if every t of them are totally independent. Another (equivalent) way to define independence between random variables is by saying that the probability for a random variable to have some value is not changed even if we know the values of the other random variables.

Definition 2.3 Let X_{1} … X_{m} be random variables. We say that they are t-wise independent if for every subset of t of them, X_{i_{1}} … X_{i_{t}}, and for every values v_{1} … v_{t}, the following holds

P rob[X_{i_{1}} = v_{1}|X_{i_{2}} = v_{2} ∧ … ∧ X_{i_{t}} = v_{t}] = P rob[X_{i_{1}} = v_{1}]

We now see that pairwise independence is simply t−wise independence for t = 2 and total independence is t−wise independence for t = m. Clearly it holds that (i + 1)-independent random variables are also i-independent random variables. The question one can ask at this point is whether it is really a stronger notion? The following example shows that it is.

Example 2.4 Let X and Y be two independent random variables and let Z = X ⊕ Y be a third random variable. Clearly every two of these random variables are independent, however the three of them are dependent - knowing the value of any two of them immediately implies value of the third.

2.2 Stronger laws of large numbers

In the previous lecture we initiated the study of techniques for bounding the probability that a random variable deviates far from its expectation. In this lecture we focus on a technique for obtaining considerably sharper bound on such probability. We now come to define the moments of a random variable as a measurement for its concentration. Taking advantage on the fact that the deviation of totaly independent random variables grows much slower than the deviation of dependent random variables we present the Chernoff bound who’s error probability is exponentially vanishing. In addition, we present probability bound for the sum of k-wise independent random variables, where this bound vanish exponentially with k.

2.2.1 Moments

Loosely speaking the moments of a random are a measurement of its concentration. One type of moment is the variance, which was already defined in the last lecture. Other (higher) moments gives us a better measure for its concentration. We will be interested mostly with the moments of a normalized random variable.

Definition 2.5 The t-centralized moment of a random variable X is

M_{t}(X) = E((X − E(X))^{t})

We can immediately see that V ar(X) = M_{2}(X). We now come to use the bound on the moments to give a better error probability on the deviation of random variables.

2.2.2 Law for 2k-independent trials

Theorem 2.6 For 2k-wise independent random variables X_{1} … X_{m}

P rob [|(\frac{1}{m} ∑_{i=1}^{m} X_{i}) − (\frac{1}{m} ∑_{i=1}^{m} E(X_{i}))| ≥ δ] ≤ max_{i} (max_{t≥2}(M_{t}(X_{i}))^{2k/t})_{(\frac{mδ^{2}}{k^{2}} )k}

Note that for the case that k = 1 this is exactly what we had in Eq. (2.1). Proof: In order to prove the above theorem we first normalize the random variables. That is, we define

X̄_{i} = X_{i} − E(X_{i})

This technicality will play a crucial part in the proof because the expectation of a normalized random variable is 0.

We rewrite the probability in the claim in terms of the normalized random variable, using the fact that we raise it to an even power to get rid of the absolute value:

P rob [|(\frac{1}{m} ∑_{i=1}^{m} X̄_{i})| ≥ δ] = P rob [(\frac{1}{m} ∑_{i=1}^{m} X̄_{i})^{2k} ≥ δ^{2k}]

Applying the Markov inequality we get

P rob [(\frac{1}{m} ∑_{i=1}^{m} X̄_{i})^{2k} ≥ δ^{2k}] ≤ \frac{E ((\frac{1}{m} ∑_{i=1}^{m} X̄_{i})^{2k})}{δ^{2k}}

= \frac{E((∑_{i=1}^{m} X̄_{i})^{2k})}{m^{2k}δ^{2k}}

We now focus on the numerator. Using the linearity of expectations we get

E ((∑_{i=1}^{m} X̄_{i})^{2k}) = ∑_{(i_{1}...i_{2k})∈[m]} E(X̄_{i_{1}} ⋯ X̄_{i_{2k}})

where [m] = {1, …, m}. We now split the right sum to the sum over two disjoint subsets A and B, where A is a set of 2k−long sequences in which some integer appears uniquely, and B is a set of 2k−long sequences in which each integer appears with multiplicity at least 2. That is (i_{1} … i_{2k}) ∈ A if there exist i such that a unique j satisfies i_{j} = i and (i_{1} … i_{2k}) ∈ B if for every j there exist j′ ≠ j such that i_{j} = i_{j′}.

(Note that if we think of m as being much greater than k, we can expect most of the terms to be from the subset A. As we will see next, these terms will have no contribution to the total sum.)

∑_{(i_{1}...i_{2k})∈[m]} E(X̄_{i_{1}} ⋯ X̄_{i_{2k}}) = ∑_{(i_{1}...i_{2k})∈A} E(X̄_{i_{1}} ⋯ X̄_{i_{2k}}) + ∑_{(i_{1}...i_{2k})∈B} E(X̄_{i_{1}} ⋯ X̄_{i_{2k}})
= 0 + ∑_{(i_{1}...i_{2k})∈B} E(X̄_{i_{1}} ⋯ X̄_{i_{2k}})

The reason that the sum over A equals zero follows from the independence of the X̄_{i}’s and using E(X̄_{i_{j}}) = 0 for i_{j} which appears with multiplicity 1. That is, for (i_{1} … i_{2k}) ∈ A so that i_{j} ≠ i_{j′} for every j ≠ j′, we have

E(X̄_{i_{j}} ∏_{j≠j′} X̄_{i_{j′}}) = E(X̄_{i_{j}})E( ∏_{j≠j′} X̄_{i_{j′}}) = 0 · E( ∏_{j≠j′} X̄_{i_{j′}}) = 0

where the first equality uses the fact that X̄_{i_{j}} is independent of ∏_{j≠j′} X̄_{i_{j′}} (since all i_{j′} are different from i_{j}). We now need to bound the sum over B. We do it by counting the number of sequences of length 2k over [m] in which each occurrence is with multiplicity at least 2 (which clearly upper bounds |B|). Let j denote the number of integers occurring in the sequence. Since each such integer appears at least twice, j can not be larger than k, thus an upper bound on the number of such sequences is:

∑_{j=1}^{k} \binom{m}{j}j^{2k} ≪ m^{k}k^{2k}

Each of these sequences contributes

E(X̄_{i_{1}}^{e_{1}} ⋯ X̄_{i_{j}}^{e_{j}}) = E(X̄_{i_{1}}^{e_{1}}) ⋯ E(X̄_{i_{j}}^{e_{j}}) = M_{e_{1}}(X̄_{i_{1}}) ⋯ M_{e_{j}}(X̄_{i_{j}})

where the e_{j}’s are the number of occurrences of each i_{j} (and so ∑_{j} e_{j} = 2k). So we get

E((∑_{i=1}^{m} X̄_{i})^{2k}) ≤ m^{k}k^{2k} · max_{i}(max_{e≥2}(M_{e}(X̄_{i})^{\frac{2k}{e}})

which in turn implies

\frac{E((∑_{i=1}^{m} X̄_{i})^{2k})}{m^{2k}δ^{2k}} ≤ \frac{max_{i}(max_{e≥2}(M_{e}(X_{i}))^{\frac{2k}{e}})}{(\frac{mδ^{2}}{k^{2}} )k}

and the theorem follows.

This is clearly a much stronger bound than what we had before. Intuitively, if we consider k as a constant and consider X_{i} as a Bernoulli random variable, we get a very similar bound to the one we had before only now we get the k exponent in the denominator.

2.2.3 The Chernoff/Hoefding bound

Up until now the only restriction on the random variables was for them to be fully or partially independent. We now restrict ourselves to bounded random variables and come up with a tighter bound. The following theorem, which is called the Chernoff bound or the Hoefding bound, is presented here without proof.

Theorem 2.7 For totaly independent random variables X_{1} … X_{m}, where X_{i} ∈ [a, b]

P rob [|(\frac{1}{m} ∑_{i=1}^{m} X_{i}) − (\frac{1}{m} ∑_{i=1}^{m} E(X_{i}))| ≥ δ] ≤ 2e^{− \frac{2δ^{2}}{(b−a)^{2}} ·m}

In the special case where X_{i} ∈ [0, 1], we get

P rob [|(\frac{1}{m} ∑_{i=1}^{m} X_{i}) − (\frac{1}{m} ∑_{i=1}^{m} E(X_{i}))| ≥ δ] ≤ 2e^{−2δ^{2}·m}  (2.2)

Clearly, Theorem 2.7 follows from Eq. (2.2); e.g., for X_{i}’s as in Theorem 2.7, define Y_{i} = (X_{i} − a)/(b − a) and apply Eq. (2.2).

Oded’s Note: Following are some remarks regarding the proof of Eq. (2.2). Firstly, we argue that it suffices to consider 0-1 random variables (intuitively, because they have the largest variance among random variables with range [0, 1]). For 0-1 X_{i}’s, denote p_{i} = Pr[X_{i} = 1] and X̄_{i} = X_{i} − p_{i}. Without loss of generality we bound the probability that \frac{1}{m} ∑_{i=1}^{m} X̄_{i} is greater from δ. For any λ > 0, we have

Pr [\frac{1}{m} ∑_{i=1}^{m} X̄_{i} ≥ δ] = Pr [e^{λ ∑_{i=1}^{m} X̄_{i}} ≥ e^{λδm}]

< \frac{E(e^{λ ∑_{i=1}^{m} X̄_{i}})}{e^{λδm}}

Using the total independence of the X_{i}’s we have E(e^{λ ∑_{i=1}^{m} X̄_{i}}) = ∏_{i=1}^{m} E(e^{λX̄_{i}}). On the other hand, E(e^{λX̄_{i}}) = p_{i}e^{λ·(1−p_{i})} + (1 − p_{i})e^{λ·(0−p_{i})} ≤ e^{λ^{2}/8}, where the last inequality holds for any p_{i} ∈ [0, 1] (and has elementary proofs unrelated to probability theory). Combining all the above, we get

Pr [\frac{1}{m} ∑_{i=1}^{m} X̄_{i} ≥ δ] < \frac{(e^{λ^{2}/8})^{m}}{e^{λδm}}

= e^{(\frac{λ^{2}}{8} − λδ)·m}

Setting λ = 4δ, we get Pr [\frac{1}{m} ∑_{i=1}^{m} X̄_{i} ≥ δ] < e^{2δ^{2}·m}, and the claim follows.

2.2.4 Multiplicative Chernoff Bounds

Oded’s Note: Unfortunately, I did not manage to say the following in class.

In some cases it is more useful to use “Multiplicative Chernoff Bounds” of the following form. For independent and indentically distributed X_{i}’s in [0, 1] with p = Pr[X_{i} = 1], and for every γ > 0, it holds that

Pr [\frac{1}{m} ∑_{i=1}^{m} X_{i} >, (1 + γ) · p] < e^{γ^{2}p·m/3}  (2.3)

Pr [\frac{1}{m} ∑_{i=1}^{m} X_{i} <, (1 − γ) · p] < e^{γ^{2}p·m/2}  (2.4)

Note that by applying the standard (i.e., additive) Chernoff bound we get a probability bound of e^{2(γp)^{2}·m} in both cases. The Multiplicative forms are thus preferable for small values of p (e.g., for p < 0.1).

2.3 Application: Estimating the mean of a function

In this section we discuss probabilistic polynomial time computations that will do approximations to problems that require by their inherent nature exponential-time deterministic computations. Note that we regard computations as feasible only if their time of execution is at most of polynomial order of magnitude and infeasible if their time is of exponential order.

These approximations will rely heavily on the ”Laws of Large Numbers” presented before. We will use samples from the problem’s (i.e., function’s) domain.

Given a boolean function f : {0, 1}^{n} → [0, 1], we denote f’s mean value by f̄. That is

f̄ =^{def} \frac{1}{2^{n}} ∑_{x∈\{0,1\}^{n}} f(x).

In order to exactly compute f̄ one is required to take 2^{n} evaluations of f (i.e. sample f’s domain in its entirety). Thus, a feasible estimation of f̄ can not precisely follow its strict definition because the domain of function f is of exponential size in n. Fortunately there are ”shortcuts” but inevitably these come with a ”price tag”. Here we shall introduce the notion of estimating (or approximating) f̄ in (probabilistic) polynomial time.

The problem we face is how to give a satisfactory probabilistic approximation of f̄ by means of a polynomial computation.

Definition 2.8 An (ϵ, δ)-sampler, S, is a probabilistic polynomial-time oracle machine that when given oracle access to the function f satisfies

P rob[|S^{f}(1^{n}, ϵ, δ) − f̄| > δ] < ϵ.

In the above, S^{f}(1^{n}, ϵ, δ) denotes the output distribution of the oracle machine S on input 1^{n}, ϵ and δ, and oracle access to the function f. The above definition means that the probability that the output of the sampler deviates from the actual value, f̄, by more than δ is no more than ϵ.

The question we now face is ”How many oracle access are needed in order to give a (ϵ, δ) approximation to f̄”?. That is, what is the query complexity of (ϵ, δ) samplers.

Theorem 2.9 (a constructive upper bound) The query complexity of (ϵ, δ) samplers is at most O(\frac{1}{δ^{2}} log(\frac{1}{ϵ})).

Proof: In order to assert the above bound on the number of queries we suggest the following

• Take m =^{def} O(\frac{1}{δ^{2}} log(\frac{1}{ϵ})) independent uniform samples on f’s domain, denoted by s_{1}, s_{2}, ..., s_{m} ∈ {0, 1}^{n}.

• Output \frac{1}{m} ∑_{i=1}^{m} f(s_{i}) which will be implement as follows:

For i=1 to m do: Make a oracle call to f with input s_{i}, obtain v_{i} = f(s_{i}), compute \frac{1}{m} ∑_{i=1}^{m} v_{i} and output it.

We will prove that the estimation we gave on the number of samples is indeed tight from bellow (i.e., m samples are necessary for an (ϵ, δ) approximation).

Oded’s Note: The probability space consists of m-long sequences of n-bit strings; that is, of all possible choices of \overline{s} = (s_{1}, ..., s_{m}) ∈ (\{0, 1\}^{n})^{m}. We define m random variables X_{1}, X_{2}, ..., X_{m} so that X_{i}(\overline{s}) = f(s_{i}). Thus, X_{i} actually depends only on s_{i}, and so these random varibales are clearly independent.

Let X_{1}, X_{2}, ..., X_{m} be m random variables, such that X_{i} = f(s_{i}). These random variables are clearly independent of one another, and each ranges in [0, 1] and has expectation that equals f̄. Applying Chernoff’s Bound, we get

P rob [|S^{f}(1^{n}, ϵ, δ) − f̄| > δ] = P rob [|\frac{1}{m} ∑_{i=1}^{m} X_{i} − f̄| > δ] < 2e^{−2δ^{2}·m}

Thus, the bound is exponentially vanishing in δ^{2}m. We compare this bound to ϵ and derive an bound on the number of samples, m, which are required by our (ϵ, δ) approximation algorithm; that is, 2 exp[−2δ^{2} · m] = ϵ which implies

m = \frac{1}{2δ^{2}} · log(\frac{2}{ϵ}).

as required.

Notice that the bound on the number of sufficient samples does NOT dependent on n, the function f’s domain size. This means that the bound does not distinguish between functions of large domain and functions of relatively smaller domain.

The upper bound (on the number of samples that are sufficient for an (ϵ, δ) approximation) given in Theorem 2 is tight up to a multiplicative factor. That is:

Theorem 2.10 (a lower bound) The query complexity of (ϵ, δ) samplers is at least Ω(\frac{1}{δ^{2}} log(\frac{1}{ϵ})).

2.4 Existence of Expanders via the Probabilistic Method

In this section we will show the existence of expanders via the ”The Probabilistic Method”, which was introduced in previous lecture. This is a combinatorial tool for demonstrating the existence of combinatorial objects. The example we will focus on is a probabilistic proof of existence for an expander (i.e., a family of graphs with certain properties). By a probabilistic proof we mean that we do not know how to explicitly find the expander we are looking for and the proof of its existence is implicit. We merely argue that a random object that we construct is not an expander is strictly smaller than 1. This is enough to obtain the theorem.

2.4.1 Expander Graphs

In order to start speaking about expanders, we will need some definitions from graph-theory to assist us in what follows.

Definition 2.11 In a graph G(V, E) a neighborhood, Γ, of a vertex v ∈ V is the set of all adjacent vertices to v (vertices that are connected to v by an edge). A neighborhood is described by the following notation :

Γ(v) =^{def} {u : (v, u) ∈ E}.

Note also that one can also extend the above definition for an arbitrary subset V′ ⊆ V (referring to the neighborhoods of a subset of vertices) by

Γ(V′) =^{def} {v ∈ V | ∃(v′, v) ∈ E for some v′ ∈ V′}.

Thus,

Γ(V′) = ⋃_{v′∈V′} Γ(v′).

Continuing with the spirit of the above definition we shall now proceed with defining the outer section of a vertices set

Definition 2.12 For a set S, the boundary of the set, Γ′(S), is the set of vertices that are adjacent to some vertex in S and are not in S themselves. That is:

Γ′(S) =^{def} Γ(S) \ S.

Oded’s Note: The following two motivational paragraphs are due to the scribes themselves. I did not motivate the definition of expander graph beyond saying that they will play an important role in some future lectures.

A very import class of sparse graphs are expander graphs. Among other things, they can be considered as a model for a network with certain desirable properties. Basically an expander has the property that every subset of its vertices has a large set of neighbors. That implies that any pair of vertices are connected by a short path. Furthermore, removing random edges in the graph (simulating local connection failures) does not reduce the property by much, so a network which is an expander can be considered as a fault-tolerant. We focus on expander graphs that are regular.

Definition 2.13 A graph is d-regular if and only if every vertex has degree d.

Now let us proceed to the definition of expander Definition 2.14 A d-regular graph G(V, E) is a (d, c)-expander or has a c-expansion (for some c > 0) if and only if for every subset V′ ⊆ V of size |V′| ≤ |V |/2 it holds that

|Γ′(V′)| ≥ c|V′|.

To see that an expander makes a good network, suppose that you want to route a message from a vertex a to a vertex b in G. Every vertex has d neighbors. By the properties of an expander, there are at least (1 + d)(1 + c) vertices at distance no more than 2 from a. Going further away, there will be at least (1+d)(1+c)^{k} vertices at distance no more than k +1 from a. We can continue expanding from a until the reachable set of vertices V_{a} has more than |V |/2 vertices. The vertex b may not be among them, but if we expand from b in the same way, we eventually obtain a set V_{b} of more than |V |/2 vertices reachable from b. The sets V_{a} and V_{b} both contain more than |V |/2 so they must overlap. The overlap contains vertices on the path from a to b. In this way, we have shown that for any pair of vertices a to b, there is a path of length no more than 2(k + 1) from a to b, where k = log_{1+c} |V |/2. As one can observe from the expression for k, the larger the value of c the shorter the expected path between any two vertices.

The notion of an expander will usually be employed in the context of taking c as a small constant and of V as a huge vertex set (because we are interested in cases where we face very large graphs). We will generalize the above definition to a collection of graphs, referred to as a ”family”, each of which have an expansion.

The above definition of a (d,c)-expander extends naturally a family, F, of graphs F = {G_{N}}_{N∈S}, where G_{N} is an N-vertex graph and S is an infinite subset of N. That is, for each N in S there exists a single graph in the family that has N vertices and all the graphs in the family are (d, c)-expanders.

2.4.2 On suitable Sets of graph sizes

Let us define a function next_{S} so that for every integer m:

next_{S}(m) =^{def} min_{n∈S}{n ≥ m}.

That is, next_{S} returns the smallest integer that is at least as big as m. The growth rate of next_{S} determines the suitability of S to be used as graph sizes in families of graphs. The criterion that we will adopt in order to examine the suitability of a given set to be used in this context is that the output of the next_{S} function is bound by a polynomial on its input. Our reason for adopting such a criterion is because upon given an m, our purpose is to construct a graph, G, with n ≥ m vertices. Typically, we will need to embed m objects as values of G. The difficulty is that we don’t want the graph, G, to be too big because our measure of computational complexity for any given task/algorithm on a graph is based on the size of that graph.

Some typical examples of sets S that are practical and useful in various applications are the following:

• S = {2n : n ∈ N}. Here next_{S}(m) ≤ m + 1.

• S = {P rimes}. Here next_{S}(m) is quite complex but certainly at most 2m (because each interval [m, 2m] must contain a prime).

• S = {n^{2} : n ∈ N}. Here

next_{S}(m) = ⌈\sqrt{m}⌉^{2}
< m + 2\sqrt{m} + 1
< m + o(m).

Even for S = {2^{n} : n ∈ N}, We have

next_{S}(m) = 2^{⌈log_{2}(m)⌉}
≤ 2m.

On the other hand, bad examples correspond to very-sparse sets such as S = {2^{2^{2^{n}}} : n ∈ N}. Here, for some m,

next_{S}(m) = 2^{2^{2^{log log log(m)+1}}}
= 2^{2^{2 log log m}}
= 2^{(log m)^{2}}
= m^{log(m)}

and that is more than any polynomial in m.

2.4.3 On the existence of 3-regular expanders

Now, we proceed to providing a probabilistic proof of the existence of expanders. We will use the following graph-algorithmic definition :

Definition 2.15 Given a graph G(V, E) a matching of G is a subset of the edges of G such that every vertex appears in at most one edge of the subset. A perfect matching is a matching where all vertices in G appear.

Now we get to the main theorem of our discussion about expanders.

Theorem 2.16 There exists a constant c > 0 for which, for all sufficiently large even N, there exists an N-vertex, 3-regular graph with expansion c.

Proof: We use the probability method. We will show that for sufficiently small c > 0 and all sufficiently large (even) N, with high probability, a random 3-regular N-vertex graph has expansion c.

We will generate a random 3-regular graph G by selecting three random perfect matchings, denoted G_{1}, G_{2} and G_{3}. To generate a random undirected graph G_{1} with N = 2K vertices and K edges, we will select edges by selecting uniformly at random a pair of distinct vertices, and removing them from the pile of available vertices (so our selection of a pair is a probabilistic event which is dependable on previous selections!). We will continue this selection of pairs of vertices until we exhaust the entire set of N vertices. When we are done, we actually have a perfect matching on G (i.e., K edges where created, every vertex is covered once). We will independently repeat this process for three times, thus getting three graphs {G_{1}, G_{2}, G_{3}} and three very-possibly disjoint sets of edges, matchings, (no edge will repeat itself unless the same pair was selected more than once, which is highly unlikely as N becomes very large). Then we combine our three graphs into one possibly multi-graph G which is 3-regular (every vertex has no more then three neighbors, if there are less than three then there exists a neighbor to which it has a multi edge (two or three parallel edges)). We want to prove that the probability that graph G has expansion c is bigger than zero (and moreover it’s close to 1). We will prove this by showing that the probability of the complement event (that there is no such expansion c of G) is strictly less than 1 (and actually close to 0). That is, the probability that there exists a subset of vertices that has too small boundary and so does not permit the existence of an expansion c of G is strictly less than 1. We shall bound the probability of the union of all these ”bad events” by means of the Union Bound argument:

Proposition 2.4.1 (Union Bound): For every series of sets {A_{i}} and every random variable X:

P rob[X ∈ ⋃_{i=1}^{n} A_{i}] ≤ ∑_{i=1}^{n} P rob[X ∈ A_{i}].

We now get to the details of the actual proof. Let N = 2K be the number of vertices of graph G_{N}. We will evaluate the probability of the complement event to the event that we are interested in; that is, the probability that G_{N} does not have expansion c. That is, we look at the probability of the following (complementary) event:

there exists a subset V′ s.t. |V′| ≤ K and |Γ′(V′)| < c|V′|.  (2.5)

By the union bound, the probability that the event in Eq. (2.5) occurs is at most

∑_{V′:|V′|≤K} P rob[|Γ′(V′)| < c|V′|]  (2.6)

Again, by the union bound, Eq. (2.6) is bounded from above by

∑_{V′:|V′|≤K} ∑_{V′′:|V′′|=c|V′|} P rob[Γ′(V′) ⊂ V′′]  (2.7)

We give an upper bound on Eq. (2.7) by considering all possible V′ of size i and all V′′ of size ci, where i goes from 1 to K (i.e., we are enumerating over all possible sizes of the two sets). Clearly, it suffices to consider in the second sum only the sets V′′ that are disjoint of V′. Thus, we get the bound

∑_{i=1}^{K} \binom{N}{i} · \binom{N − i}{ci} · P rob[Γ′(V′_{0}) ⊂ V′′_{0}]

where V′_{0} and V′′_{0} are arbitrary fixed disjoint sets of sizes i and ci, respectively. Equivalently, we may bound

∑_{i=1}^{K} \binom{N}{i} · \binom{N − i}{ci} · P rob[Γ(V′_{0}) ⊂ V′_{0} ∪ V′′_{0}]

which, using V′_{0} = [i] =^{def} {1, ..., i} and V′′_{0} = {i + 1, ..., i + ci}, is upper bounded by

∑_{i=1}^{K} \binom{N}{i} · \binom{N}{ci} · P rob [Γ([i]) ⊆ [i + ci]]  (2.8)

We now focus on bounding the probability that Γ([i]) ⊆ [i + ci]. That is, we bound the probability that in all three perfect matchings, the vertices in [i] are matched to vertices in [i + ci]. Focusing on each of these three perfect matchings, the probability that in a single perfect matching all the vertices in [i] are matched to vertices in [i + ci] is at most

\frac{i + ci − 1}{N − 1} · \frac{i + ci − 3}{N − 3} ⋯ \frac{i + ci − i + 1}{N − i + 1} = ∏_{j=1}^{i/2} \frac{i + ci − 2j + 1}{N − 2j + 1}

where the bound is obtained by considering a process of iteratively matching i/2 vertices in [i], where in each step we match the next unmmatched vertex and consider the probability that it is matched to one of the remaining unmatched vertices in [i + ci]. By algebraic manipulations we get:

∏_{j=1}^{i/2} \frac{i + ci − 2j + 1}{N − 2j + 1}

< ∏_{j=0}^{(i/2)−1} \frac{i + ci − 2j}{N − 2j}

= \frac{2^{i/2} · ∏_{j=0}^{(i/2)−1} \frac{i+ci}{2} − j}{2^{i/2} · ∏_{j=0}^{(i/2)−1} \frac{N}{2} − j}

= \begin{matrix}( & {(i+ci)/2 \atop i/2} & ) \\ ( & {N/2 \atop i/2} & ) \end{matrix}

Recall that the above line provides an upper bound on the probability that a single random perfect matching matches all vertices in [i] to vertices in [i + ci]. Thus, the probability that is happens in all three perfect matchings used to construct the graph is the cube power of the above, and we get:

P rob [Γ([i]) ⊆ [i + ci]] < (\begin{matrix}( & {(i+ci)/2 \atop i/2} & ) \\ ( & {N/2 \atop i/2} & ) \end{matrix})^{3}

Plugging this into Eq. (2.8), we obatin the bound

∑_{i=1}^{K} \frac{\binom{N}{i} · \binom{N}{ci} · \binom{(i+ci)/2}{i/2}^{3}}{\binom{N/2}{i/2}3}  (2.9)

Using the approximatiom \binom{n}{αn} = 2^{H_{2}(α)·n}, where H_{2}(α) = α log_{2}(1/α) + (1 − α) log_{2}(1/(1 − α)) is the binary entropy function, we get

log_{2} (\frac{\binom{N}{i} · \binom{N}{ci} · \binom{(i+ci)/2}{i/2}^{3}}{\binom{N/2}{i/2}3} )

≈ H_{2}(i/N) · N + H_{2}(ci/N) · N + 3H_{2}(i/(i + ci)) · \frac{i + ci}{2} − 3H_{2}(i/N) · \frac{N}{2}
= −(H_{2}(i/N) − 2H_{2}(ci/N)) · \frac{N}{2} + 3H_{2}(1/(1 + c)) · \frac{i + ci}{2}

For a sufficiently small c > 0 (and i ≤ N/2), we have H_{2}(i/N) < c′ · H_{2}(ci/N) for c′ ≈ c log(1/c), and H_{2}((1/(1 + c)) ≈ H_{2}(c) ≈ c log_{2}(1/c). Plugging all this into Eq. (2.9), we obatin the bound

∑_{i=1}^{K} 2^{−(1−c′)H_{2}(i/2K)·K+2c′·i}

which approaches zero when c does. The theorem follows.

2.4.4 Non-existence of 2-regular expanders

Contrary to the previous theorem, the following theorem asserts exactly the opposite for 2-regular graphs

Theorem 2.17 For all c > 0 and for all sufficiently large N, there exists NO (2, c)-expander with N vertices.

Proof: Fix some arbitrary c > 0. Consider an N-vertex graph which is 2-regular. Without loss of generality consider a connected graph [if the graph is build out of components then we can choose all vertices in the smallest component (which contains at most N/2 vertices) and get that their boundary is null!]. Such a connected graph must be a closed cycle. Let us consider a connected subset of N/2 vertices. The boundary of such a subset is of size 2. In order to contradict the requirement for the existence of an expander, choose the number of vertices in the graph, N, such that c · N/2 > 2 (i.e. N > 4/c will do).

Bibliographic Notes

For more details on the proof of Chernoff Bound, the reader is referred to [10, Apdx. A]. The Multiplicative Chernoff Bounds are presented, proved and discussed in [34, Chap. 4]. Historical notes regarding the various forms of Chernoff Bound are also provided in [34, Chap. 4]. A proof of Theorem 2.10 (i.e., a lower bound on the sample complexity of (ϵ, δ)-samplers) can be found in [14].

26 LECTURE 2. LAWS OF LARGE NUMBERS AND EXISTENCE OF EXPANDERS

Small Pairwise-Independent Sample Spaces

Notes taken by Lev Faivishevsky and Sergey Khristo

We ...solemnly Publish and Declare that...

these are ...free and independent...

(Declaration of Independence)

Action of Second Continental Congress, July 4, 1776

Summary: Often in applications that use randomness (for example coin tosses) we do not need all these tosses to be independent, instead it may suffice that they are k-independent for some k. In the case of independent variables the probability space is at least of the size |S|^{m}, where m is the number of variables in the sequence and S is the sample space for an individual variable. In this lecture we present a construction for pairwise independent sample space of the size max(|S|, m)^{2} and its extension to k-wise independent case. This will allow us to derandomize the algorithm that finds assignment for 3CNF that satisfies at least 7/8 clauses, presented on the first lecture, just by means of exhaustive search.

3.1 Construction of k-wise independent sample space

Suppose we have sample space of the form S^{m} = S ×S ×…×S, where S is a finite set, with uniform distribution on it. Without loss of generality we can think of S as a set of integers {1, …, |S|}.

Any element of S^{m} is a sequence of m elements from S, denoted (s_{1}, …, s_{m}). Clearly, every two components, taken alone, also have uniform distribution. Note, that if the pair (x_{1}, x_{2}) ∈ X × X is uniformly distributed on X × X then x_{1} and x_{2} are independent and uniformly distributed on X.

We are seeking for the subspace Ω that is substantially smaller than S^{m}, but the uniform distribution on Ω induces uniform distribution on any two coordinates. In the construction for Ω, presented below, we achieve |Ω| ≃ max(|S|, m)^{2}. This allow us, in some sense, to save randomness.

The construction that is described in detail below is a nice example but there are some technicalities that should be taken into account when working with it. They are discussed at the end of this lecture.

Suppose, for the beginning, that m ≤ |S|. This restriction will be waived later. We will try to associate some algebraic structure to S; for example, by treating S as a finite field F. Finite fields exist only of sizes which are prime powers. The case when the size of S is not a prime power will be discussed later. Let us now define such a sample space Ω:

Ω =^{def} {The set of all linear polynomials over F} = {ax + b|(a, b) ∈ F^{2}}
≡ {(a · 1 + b, a · 2 + b, …, a · m + b)|(a, b) ∈ F^{2}}.

The first line above gives a succinct representation of a set of m-long sequences over F, whereas the second line gives an explicit representation of the same sequences.

Note that given a field F, the space Ω can be easily sampled: you just pick two random elements from the field and get an element of Ω.

Let us denote by Pr_{x∈_{R}X} the probability over the set X with uniform measure on it. Write p_{a,b}(i) = a · i + b (the linear polynomial in i with coefficients a and b).

Claim 3.1.1 For every i ≠ j from [m] ⊆ F and for every α and β from F

Pr_{(a,b)∈_{R}F^{2}}[p_{a,b}(i) = α ∧ p_{a,b}(j) = β] = \frac{1}{|F|^{2}} .

In other words, uniform measure on Ω results in pairwise independent uniform sample space.

Proof: Let us write formally

Pr_{(a,b)∈_{R}F^{2}}[p_{a,b}(i) = α ∧ p_{a,b}(j) = β] = \frac{|\{(a, b)| p_{a,b}(i) = α ∧ p_{a,b}(j) = β\}|}{|F|^{2}}.

We have to show that the nominator equals one. Indeed, it is equal to the number of pairs (a, b) such that the following system of two equations is satisfied.

\begin{matrix}ai + b = α \\ aj + b = β \end{matrix}

Thinking of a and b as variables and of i, j, α and β as constants, we conclude that if i is not equal j then this system has unique solution because the matrix

( \begin{matrix}i & 1 \\ j & 1 \end{matrix} )

has full rank.

Observe that the whole construction is independent of the field we use and one can use any field that he finds convenient. For example, if the size of S is prime then modular arithmetic is a good choice.

Note that the shown construction is not 3-wise independent. First, we do not have uniform (and independent) distribution on any three coordinates, since the there can be no solution for the system of the equations as in the proof of claim.

More generally, it is not possible to have a k-wise independent sample space that is smaller then |S|^{k}, because any fixed k coordinates are uniformly distributed and every value of (x_{1}, …, x_{k}) has some nonzero probability hence it is impossible to have less then |S|^{k} elements in this set.

3.1. CONSTRUCTION OF K-WISE INDEPENDENT SAMPLE SPACE 29

After a little meditating it becomes clear how to make our construction produce k-wise independent variables. For this it is enough to consider in the above definition of Ω polynomials of degree less than k. Specifically, we define

Ω′ =^{def} {The set of polynomials of degree less then k over F}
= \{\overline{c} = (c_{0}, …, c_{k−1}) : p\overline{c} = ∑_{i=0}^{k−1} c_{i}x^{i}\}
≡ \{(p\overline{c}(1), …, p\overline{c}(m)) : \overline{c} ∈ F^{k}\}.

Again, the first lines give a succinct representation of a set of m-long sequences over F, whereas the last line gives an explicit representation of the same sequences.

Recall that sample space is called k-wise independent if on any fixed k locations we have uniform distribution. Analogously to Claim 3.1.1, we have the following:

Claim 3.1.2 For every different i_{1}, …, i_{k} from [m] ⊂ F and for every α_{1}, …, α_{m} from F

Pr_{\overline{c}∈_{R}F^{k}}[∀j = 1 … k, p\overline{c}(i_{j}) = α_{i_{j}}] = (\frac{1}{|F|})^{k} .

Proof: Similar to what we had in the previous claim,

Pr_{\overline{c}∈_{R}F^{k}}[∀j = 1 … k, p\overline{c}(i_{j}) = α_{j}] = \frac{|\{\overline{c}|∀j = 1 … k, p\overline{c}(i_{j}) = α_{j}\}|}{|F|^{k}}

The nominator is equal the number of solutions to the following system:

{ c_{0} + c_{1}i_{1} + … + c_{k−1}i_{1}^{k−1} = α_{1} ; ⋮ ; c_{0} + c_{1}i_{k} + … + c_{k−1}i_{k}^{k−1} = α_{k}

Again one should think of c_{i}’s as variables in the linear system.

To see that in fact there is exactly one solution let us write the system in the matrix form:

( \begin{matrix}1 & i_{1} & i_{1}^{2} & … & i_{1}^{k−1} \\ 1 & i_{2} & i_{2}^{2} & … & i_{2}^{k−1} \\ ⋮ & ⋮ & ⋮ & ... & ⋮ \\ 1 & i_{k} & i_{1}^{k} & … & i_{k}^{k−1} \end{matrix} ) ( c_{0} ; c_{1} ; ⋮ ; c_{k−1} ) = ( α_{0} ; α_{1} ; ⋮ ; α_{k−1} )

The matrix in the above system is a Van der Monde matrix and it is full rank (since i_{l} ≠ i_{l′} for every l ≠ l′).

This construction is optimal in the size of Ω, but remember we are under the conditions that m ≤ |S| and that the size of S is must prime power (it allows us to treat S as a field and make all that claims about uniqueness of the solution of a linear system). The next section addresses these issues and shows how to deal with fields from the computational point of view.

3.2 Annoying technicalities

In this section we will cope with several technical pitfalls, that can potentially spoil our construction. These potential difficulties are:

1. The size of S may be not a prime power. It means that direct representation S as a finite field is impossible in this case.

2. What should we do if m > |S|?

3. How do we deal with fields, i.e. how can we define the sample subspace and how can we operate in it?

First of all, let’s address the first problem. A general recipe is to consider substantially larger set S′ with a ”nice” embedding of S into S′. The meaning of nice is that each element of S appears in S′ about the same number of times. So if we make a construction for S′, it will remain valid for S also because there will be some good mapping from S′ to S. More formally, we should have some mapping μ : S′ → S with the following properties:

∀a, b ∈ S |μ^{−1}(a)| ≈ |μ^{−1}(b)|

The formula says that each image should have nearly the same number of preimages (it means that the mapping μ has to be ”regular”). Actually, it is better that each element of S will have the same number of representatives in S′. If this condition doesn’t hold then (exact) uniform distribution in S′ wouldn’t induce uniform distribution in S. Vice versa, if each element in S has exactly the same number of representatives in S′ then uniform distribution in S′ will induce uniform distribution in S via this mapping μ. It should be noticed that small discrepancy in this approximate equality like:

∀a, b ∈ S |μ^{−1}(a)| = |μ^{−1}(b)| ± 1

doesn’t disturb the validity of our constructions by much (especially if |S′|/|S| is large; indeed, the bigger S′ the better the approximation but the cost comes in terms of complexity which grows with S′). Hence whenever the initial space S doesn’t satisfy us, we may switch to a larger S′. The same trick will help us further, in coping with the second problem.

Now we will give a similar solution to the second problem. In order to be more clear, we’ll do it through an example. Suppose, we need to build sample space of 3-wise independent elements over S={0,1}. More specifically, we want to construct the sample space of 3-wise independent sequences of length m with elements of S. We’ll show that size of this sample has to be m^{3} only. By Claim 3.1.2 (with k = 3) we would be able to do it if |S| were bigger than m. Thus, we ought to take

S′ = {0, 1}^{l}, where l = ⌈log_{2} m⌉

Note, that this condition guarantees that |S′| ≥ m, because

|S′| = 2^{l} = 2^{⌈log_{2} m⌉} ≥ m

Finally one should mention, that |S′| is a prime power. The size of the whole construction will be

|S′|^{3} ≈ (2l)^{3} ≈ m^{3}

The price that we have paid for the trick is that our constructed sample subspace has rather big size ≈ m^{3}. The last step that should be done is to define the mapping μ : S′ → S that in our case implies

μ : {0, 1}^{l} → {0, 1}

The simple mapping may be just taking the first bit of an element in {0, 1}^{l} as the element’s image in {0, 1}. All needed properties of μ are satisfied. Generally, defining such a mapping is a relatively easy task, because we impose only the following restrictions on μ: it should be easy to compute and the mapping should be regular (remember, this means that each image should have nearly the same number of preimages).

The last technical problem to be treated is the handling of finite fields. We have assigned algebraic structure to constructed sample subspace. But how can we handle it? This issue clearly consists of several significant parts:

1. How do we build the sample space?

2. How do we explicitly write the elements of it?

3. How do we perform operations with the elements (like addition and multiplication)?

The answer to these questions amounts to providing an algorithm for constructing sample spaces. We will claim that this algorithm exists ”modulo the field”. In case of a field of prime cardinality, the meaning of the sentence ”modulo the field” is that we are given the prime p. The general case will be considered later.

Claim 3.2.1 Given prime p and number m ≥ p we may construct in poly(m)-time the sample space Ω:

Ω = {The set of all linear polynomials over field F} ≡ {ax + b|(a, b) ∈ F^{2}}

≡ {(a · 1 + b, a · 2 + b, …, a · m + b)|(a, b) ∈ F^{2}}.

Proof: In order to prove the claim we will explicitly give the algorithm that outputs p^{2} such sequence of length m each. The algorithm will run in time polynomial in p ∗ m.

The algorithm consists of two nested loops.

   for a,b ∈ F
       for i = 1 to m

output (a · i + b) mod p).

We can see that running time of the algorithm is O(p^{2}). Also if we want to generate a specific element and we know which one and a specific coordinate in it, we are also able to do it very efficiently.

Oded’s Note: What I tried to explain is that in general, we may be given an arbitrary set S and an integer m and need to construct a pairwise (or k-wise) independent space over S. In case |S| is a prime and |S| ≥ m, the above claim provides the answer, but in general |S| may not be a prime (or even a prime power) or m > |S|. In such a case, we shall resort to the technique presented above; that is, construct a pairwise (or k-wise) independent space over some sufficiently large S′, and map S′ to S. But in such a case, we have to find S′ (which has to be of prime (or prime power) cardinality) by ourselves. We may choose to find S′ of prime cardinality or find S′ of cardinality p^{e} where p is a small prime and e is an integer. In the first case, we face the problem of finding a large prime, and if the desired prime is huge then we are in real trouble (becuase we don’t know of an efficient deterministic algorithm; see details below). In such a case, we turn to the second alternative, and use S′ of cardinality p^{e}, when p is samll (e.g., p = 2). Such a prime poer is easy to find, but operating with a field of that cardinality is more involved.

About the complexity of finding primes. Certainly, given an integer N, we can find a prime in the interval [N, 2N] by using the trivial algorithm that scans all candiadtes and tries to factor each of them. This algorithm is deterministic and run in time polynomial in N, which is fine when N is not too big. A randomized algorithm can find such a prime in time polynomial in the length of the binary representation of N (i.e., poly log N), but randomized algorithms are often not admissible in the context of constructing probability spaces.

We have shown that the sample space is explicitly constructible if we are given a prime p (i.e, as in a case we are given a set S and and integer m < |S|, and are asked to construct a sample space Ω for S^{m}, where |S| is a prime). But what happens if we are not given p (i.e., as in case we are asked to construct such space when either |S| is not a prime or m > |S|, and we proceed by constructing a sample space over a bigger S′ of prime cardinality, which we have to find by ourselves)? In this case we face the problem of finding big primes (when the desired set is large). This is considered to be a difficult task (for deterministic algorithms). Typically, in such a case, it will be better to seek a field of cardinality p^{e} for some small prime p (e.g., 2) and big integer e. We’ll consider this case in a while.

We should be able to represent explicitly elements of the field, that is trivial task in the case of a field of prime cardinality, and to make field operations, like addition and multiplication. Performing these operations is relatively easy when we are talking about a field of a prime cardinality (the operations are simply addition and multiplication modulo the prime).

Now we proceed to the case the cardinality of field is a prime power. We denote such a field as GF (p^{e}), where p is a prime, and e is an integer bigger than 1. We again should give an answer on the following list of questions:

1. How do we represent elements of this field?

2. How do we operations in it, i.e. how can addition and multiplication be performed in such a field?

The set of elements of that kind of the field is typically represented by polynomials of degree smaller or equal to e − 1 over GF (p). This is the canonical representation of this kind of the field. Here we assume that we already know how to operate with GF (p). In particular, in the case of GF (2) it’s very easy. We are talking about numbers 0 and 1, and operations are addition and multiplication modulo 2.

Having defined the set of the elements, now we should equipped the field with operations. In this representation the field always comes with an irreducible polynomial of degree exactly e over GF (p). A polynomial is called irreducible, if it has no non-trivial polynomial division. In other words, Q(x) is irreducible over F (p) if for every two polynomials Q_{1}(x) and Q_{2}(x) over GF (p) if

Q = Q_{1} · Q_{2}

then one of these polynomials is actually a constant.

Now we are ready to define the operations. Addition is easily implemented as an addition of polynomials. Remember, that elements of our field are polynomials, and therefore we may add elements in the field directly as polynomials. We obtain the description of the field operation addition:

addition: simple addition of polynomials We would like to introduce multiplication in the field also as a multiplication of the polynomials. Unfortunately, we cannot do it so easily, because product of two polynomials of degree less or equal to e − 1 may be the polynomial of degree bigger than e − 1, which is not a valid representation of a field element. The right way is to define this operation is as multiplication modulo irreducible polynomial, denoted Q. It means that we do multiplication in the following way: firstly, we calculate usual product of two polynomials and then we reduce the result modulo the irreducible polynomial Q(x). To clarify the notion of reduction modulo polynomial, let’s write explicitly Q(x). Without loss of generality, we may assume that in Q(x) coefficient of x^{e} is 1:

Q(x) = x^{e} + ∑_{i=0}^{e−1} q_{i} · x^{i} (in GF (p))

Reduction modulo Q(x) implies that the polynomial Q(x) is considered to be a zero of the field GF (p^{e}):

Q(x) = x^{e} + ∑_{i=0}^{e−1} q_{i} · x^{i} = 0

Thus in GF (p^{e}) we have

x^{e} = − ∑_{i=0}^{e−1} q_{i} · x^{i}

Now whenever we encounter term x^{e} in result of product of polynomials, we may replace it by

− ∑_{i=0}^{e−1} q_{i} · x^{i}. Furthermore, we may replace x^{e+1} firstly by x · x^{e}, and then substitute the above

representation of x^{e}.

x^{e+1} = x · x^{e} = x · (− ∑_{i=0}^{e−1} q_{i} · x^{i}) = − ∑_{i=0}^{e−1} q_{i} · x^{i+1}

= − ∑_{i=0}^{e−2} q_{i} · x^{i+1} − q_{e−1} · x^{e} = − ∑_{i=0}^{e−2} q_{i} · x^{i+1} + q_{e−1} · ∑_{i=0}^{e−1} q_{i} · x^{i}

So we have recursively represented x^{e+1} as a polynomial of degree less or equal to e − 1. Obviously, we may represent any x^{e+k},(where k = 1...e − 2) as a polynomial of degree less or equal to e − 1 by the same recursive procedure.

Finally, we are able to represent any possible product of 2 polynomials degree less or equal to e − 1 (it can be obviously a polynomial of degree up to 2e − 2 ) as a polynomial of degree less or equal to e − 1. So now we complete our description of the field by introducing the operation of multiplication:

multiplication: multiplication of polynomials modulo

irreducible polynomial Q(x).

Unfortunately, there exists an number of difficulties. The problems are connected with the field representation of sample space. They don’t appear when the set of field elements is somewhat small, because small primes can be found very easily. But if |S| (or rather |S′|) has to be very big, then there are two alternative for field representation: either through a field of prime cardinality or through a field, whose size is a prime power. The former leads us to problem of finding big primes via a deterministic algorithm. Up to now it’s not clear how we can solve it effectively. There exists probabilistic algorithms for big prime finding, but by definition it only gives a prime with high probability, while we would like to have a deterministic algorithm that produces a big prime.

The other option for the field representation; that is, as a field of prime power cardinality, seems to be less problematic. If p is small, then we just need to find an irreducible polynomial. There are solutions to this issue also, but they are beyond the scope of this lecture. We only give one special case for GF (2^{l}). If l is of the form l = 2 ∗ 3^{i},where i is an integer, then the following polynomial is irreducible:

x^{l} + x^{l/2} + 1

Now, having resolved the technical difficulties of the construction, let’s proceed to a nice application of this technique.

3.3 Application to 3CNF

Recall the algorithm for finding assignment for 3CNF formula that satisfies at least 7/8 clauses (presented on the first lecture). The algorithm works as follows: choose random assignment (uniformly) and test it. If it satisfies more than 7/8 of clauses then output the assignment, else choose another assignment. This algorithm works in expected polynomial time and requires a source of random bits. Here we present a modification that works in deterministic polynomial time.

Denote by m the number of variables in the input formula φ and by n the number of clauses. Let Ω ⊂ {0, 1}^{m} be three-wise independent sample space. Using the technique described above, the size of Ω is of the order m^{3}. We claim that the expected number of clauses satisfied by a uniformly selected assignment in Ω equals \frac{7}{8} · n (exactly as is the case when selecting a totally random assignment (i.e., a uniformly selected assignment in {0, 1}^{m})):

Claim 3.3.1

E_{τ∈_{R}Ω}(number of clauses satisfied by τ) = \frac{7}{8} · number of clauses.

Proof: Denote by I_{i} an indicator of the event that i’th clause is satisfied. Then,

E_{τ∈_{R}Ω}[number of clauses satisfied by τ] = E(∑_{i=1}^{n} I_{i}) = n · E(I_{1}),

where equality is due to linearity of expectation and to the fact that the I_{j}’s are identically distributed. In order to compute E(I_{1}) recall that the distribution of individual letters in the assignment is three-wise independent and the fact that in each clause we have exactly three variables hence they are uniformly distributed over the set {0, 1}^{3}. Thus,

E(I_{1}) = Pr[clause is satisfied ] · 1 + Pr[clause is not satisfied] · 0 = 7/8.

The claim follows.

Recall that Pr[X ≥ E(X)] > 0 for every random variable X. Letting X represent the number of clauses satisfied by an assignment uniformly selected in Ω, we conclude that Ω must contain an assignment that satisfies at least the expected number of satisfied clauses, (i.e., a 7/8th fraction of the clauses). Therefore, it is enough to scan the whole sample space Ω in search for such an assignment, and since the size of the space is only O(m^{3}) we can do it efficiently.

3.4 Conclusion

The construction we have presented in this lecture is rather elegant. It enables dealing with k-wise independent elements in substantially small sample space. For the sake of simplicity, let’s suppose that S is a field, and m is smaller than the size of S Thus we obtain k-wise independent sample space Ω and the size of the construction |Ω| is equal to |S|^{k}. This sample space can be explicitly constructed, and we have given an algorithm for it. One can write down all the elements of it in time:

O(m · |S|^{k}) · (time of field operations).

The size of our construction is (in some sense) optimal in terms |S|, because as we have discussed before, the size of construction of uniform over S variables with k-wise independency property cannot be less than |S|^{k}.

In case we are dealing with S = {0, 1} and we follow the recipe which was described in Section 3.2, then the size of the sample space will be m^{k}. Let’s consider the optimality of this size. More exactly, whether we can have the construction with the same property, but of substantially smaller size than m^{k}. The answer is no. There is a lower bound for k-wise independent sample space of sequences of length m over S = {0, 1} that asserts that that the size of such a space must be at a least m^{⌊k/2⌋}. That is, the basis is m and the exponent is k.

Bibliographic Notes

The constructions of pairwise and k-wise independent sample spaces (presented above) were rediscovered in computer science by Chor and Goldreich [16] and Alon, Babai and Itai [6], respectively. (They were originally discovered and appeared in the probability theory literature [29].)

36 LECTURE 3. SMALL PAIRWISE-INDEPENDENT SAMPLE SPACES

Small Pairwise-Independent Sample Spaces (cont.) and Hash Functions

Notes taken by Olga Grinchtein and Denis Simakov

Pluralitas non est ponenda sine neccesitate (Entities should not be multiplied unnecessarily)

Ockham’s Razor (William of Ockham)

Summary: A good construction of k-wise independent sample space should be small and convenient to use in applications. The construction presented in the previous lecture satisfies the first requirement (i.e., were small), but is rather complex, imposing a number of technical problems in application. In this lecture a second, much more convenient construction is presented, based on affine transformations. This construction yields small enough sample spaces, but only for pairwise independence (k = 2).

We also start discussing hash functions, show their relation to small pairwise independent sample spaces, and present a Hashing Lemma, connecting k-wise independency of hash functions to their “uniformness”.

4.1 A second construction of small pairwise independent sample space

4.1.1 Motivation

Let’s recall the problem we address. We have a finite set S and an integer m. We want to generate random strings of length m over S. So conceivably our sample space is S^{m}. But the size of this sample space is |S|^{m} that is often much more than we can afford.

So naturally the task is to find set Ω ⊂ S^{m}, which would be rather small and would possess the property of k-wise independency (k is one more integer parameter). This would enable us to use this small sample space Ω instead of huge S^{m} in problems, where k-wise independence suffices.

More specifically, k-wise independence means that if we watch strings through arbitrary k “windows”, and these strings are uniformly chosen from Ω, then we will see in these windows uniform distribution, that is, for any distinct i_{1}, …, i_{n} and any α_{1}, …, α_{k} ∈ S

Pr_{e∈_{R}Ω} [e_{i_{1}} = α_{1} ∧ ⋯ ∧ e_{i_{k}} = α_{k}] = \frac{1}{|S|^{k}} .

38 LECTURE 4. PAIRWISE-INDEPENDENT SPACES (CONT.) AND HASH FUNCTIONS

Concerning the size of Ω, typically we want it to be polynomial in max(|S|, m)^{k}.

A first construction, solving this problem, we saw the last time. It is based on associating S with a finite field F. Then k-wise independent Ω is constructed as the set of polynomials of degree less than k over the field F. In explicit representation (in terms of elements of S^{m}) Ω is the set of values of such polynomials on some m distinct elements of the field:

Ω = {polynomials of degree < k over field F}
= {(P (1), …, P (m)) : P – such polynomials} .

This construction is quite satisfactory regarding independence (it works for any k) and size of Ω (|Ω| = |S|^{k}, that is the smallest possible for k-wise independent space). But it is rather inconvenient for practical use due to a number of annoying technical difficulties that we have to overcome. We have to impose on S algebraic structure which can be complex (for example, finite fields), we have constraints on the size of S (it should be a prime power), and there is also inherent condition that m ≤ |S|. Although we have demonstrated how to manage all these technicalities, we still would like more convenient methods.

In this lecture we will present the second construction of small pairwise independent space. It will have the following major differences, compared to the first construction:

+ less problematic;

− only for k = 2 (pairwise independence), and we don’t know how to generalize it for arbitrary k.

So the second construction we present below is nice, and can be recommended for applications that require only pairwise independence. But whenever we need higher degree of independence, we still have our first construction at hand.

4.1.2 Construction

We’ll assume that S = {0, 1}^{n} (if it’s not the case, we can embed our sample space into set of binary strings). Unlike the first construction, we do not assume anything about the relation of m vs |S|.

Instead of polynomials over some field, in the second construction we will use affine transformations of boolean vectors. Affine transformation, given by pair (A, b), where A is n × l boolean matrix, b is n-dimensional boolean vector, maps all vectors v ∈ {0, 1}^{l} into vectors Av + b ∈ {0, 1}^{n} (all operations are done modulo 2).

So we define our new small sample space as follows:

Ω =^{def} \{ \begin{matrix}set of all affine transformations \\ from [m] = \{0, 1\}^{l} to S = \{0, 1\}^{n} \end{matrix} \}
≡ \{(A, b) : \begin{matrix}A is n × l boolean matrix, \\ b is n-dimentional boolean vector \end{matrix} \}
= \{(Av_{1} + b, …, Av_{m} + b) : \begin{matrix}∀(A, b), v_{1}, …, v_{m} are fixed \\ distinct l-dimensional vectors \end{matrix} \} .

4.1. A SECOND CONSTRUCTION OF SMALL PAIRWISE INDEPENDENT SAMPLE SPACE39

Remark. In this definition (and likewise in the 1st construction) Ω is given in two representations:

1. Implicit representation (as the set of affine transformations; or polynomials as before). This is how we define Ω, but in this representation Ω is not a subset of the initial sample space S^{m}. Implicit representation is a compact way of writing elements of Ω, and it allows us, for example, to generate all elements in Ω.

2. Explicit representation (as affine transformations, evaluated at m points; or evaluated polynomials). It is our implicit representation, now embedded into S^{m}. For Ω, written in this way, we can speak about pairwise independence.

Both representations are schematically presented in the picture:

[figure: S^{m}; \underline{Ω (explicit)}; \{(Av_{1}+b,...,Av_{m}+b)\}; values of affine; transformations; \underline{Ω (implicit)}; \{(A,b)\}; affine; transformations]

Let’s return to our new construction. Matrix A has n rows and l columns, vector b has length n. The sequence of m distinct vectors {v_{i}}_{i=1}^{m} (each of length l) can be chosen, for example, to include binary numbers in canonical order (i.e. v_{i} is binary representation of number i).

So the sequence (Av_{1} + b, …, Av_{m} + b) consists of m vectors of length n, and lies in S^{m} = ({0, 1}^{n})^{m}.

The parameter l is typically chosen to be ⌈lg_{2} m⌉. We do not assume here any relation between m and n, so both cases l ≥ n and l < n can happen.

Let’s calculate the size of Ω. The number of binary n×l matrices A is 2^{nl}, the number of binary n-dimensional vectors b is 2^{n}, so

|Ω| = number of pairs (A, b) = 2^{nl+n} ≅ 2^{n(1+lg m)},
|S^{m}| = (2^{n})^{m} = 2^{nm} ≫ |Ω|.

This shows that we have achieved our goal of finding small sample space. Yet in the first construction we had size of k-wise independent sample space |Ω| = 2^{kn} = 2^{2n} for k = 2, which is considerably better. We’ll address the issue of improving |Ω| in Sect. 4.2, but first let’s check whether we indeed obtained a pairwise independent space.

4.1.3 Analysis: pairwise independence

Let’s formulate the condition of pairwise independence as applied to the constructed Ω (from here on A is assumed to be l × n boolean matrix and b a boolean vector of length n):

∀i ≠ j ∀α, β ∈ S Pr_{(A,b)}[Av_{i} + b = α ∧ Av_{j} + b = β] = \frac{1}{|S|^{2}} .

It’s convenient for us to formulate and proof another claim (clearly a more general one).

Claim 4.1.1 (pairwise independence of the construction)

\begin{matrix}∀u ≠ v ∈ \{0, 1\}^{l} ∀α, β ∈ S = \{0, 1\}^{n} \\ Pr_{(A,b)}[Av + b = α ∧ Au + b = β] = \frac{1}{|S|^{2}} . \end{matrix}

Proof:

Step 1. We present a simplified expression for Pr_{(A,b)}[Av + b = α ∧ Au + b = β].

Let’s rewrite our probability in the following form:

Pr_{(A,b)}[Au + b = α ∧ Av + b = β]
= Pr_{(A,b)}[Av + b = α] · Pr_{(A,b)}[Au + b = β|Av + b = α]  (4.1)

Consider the multipliers in the expression (4.1) one by one.

1. b is chosen uniformly, A and b are independent, whereas v and α are fixed, so Av + b is uniformly distributed as well. So Pr_{(A,b)}[Av + b = α] = \frac{1}{|S|}. Let’s show this more formally in two steps. First, for any fixed matrix A we have Pr_{b}[b = α − Av] = \frac{1}{|S|} (because b is uniformly distributed on S). Second, A and b are independent, so from the fact that Pr_{b}[Av + b = α] = \frac{1}{|S|} for any fixed A it follows, that this probability is also \frac{1}{|S|} when A is random. This follows immediately by expanding as follows:

Pr_{(A,b)}[Av + b = α] = ∑_{A} Pr_{(A,b)} [b = α − Av | A = A] · Pr_{A}[A = A]

= ∑_{A} Pr_{b}[b = α − Av] · Pr_{A}[A = A]

= \frac{1}{|S|} ∑_{A} Pr_{A}[A = A]

= \frac{1}{|S|}

2. In Pr_{(A,b)}[Au + b = β|Av + b = α] the event Au + b = β is considered only under the condition Av + b = α, so we can eliminate b by substitution b = α − Av:

Pr_{(A,b)}[Au + b = β|Av + b = α]
= Pr_{(A,b)}[A(u − v) = β − α|b = α − Av]
= Pr_{(A,b)}[Aw = γ|b = α − Av]  (4.2)

where w =^{def} u − v ≠ 0 and γ =^{def} β − α are fixed vectors. Now we see that event b = α − Av in (4.2) is totally irrelevant to Aw = γ. Hence Pr_{(A,b)}[Aw = γ|b = α − Av] = Pr_{A}[Aw = γ] (this can be shown by using Bayesian Rule^{1}).

1Pr [Aw = γ | b = α − Av] = \frac{Pr[b=α−Av | Aw=γ]·Pr[Aw=γ]}{Pr[b=α−Av]} = Pr[Aw = γ], because for every A Pr_{b} [b = α − Av | Aw = γ] = Pr_{b}[b = α − Av] = 2^{−n}

Thus, we obtained the following expression:

Pr_{(A,b)}[Au + b = α ∧ Av + b = β] = \frac{1}{|S|}Pr_{A}[Aw = γ]  (4.3)

where w and γ are completely arbitrary, except for the fact that w ≠ 0 (this is of course crucial). Notice, that this result was obtained without using any assumptions about the internal structure of A (e.g., the distribution of its elements and so on).

Now in order to prove our claim, we need only to show that Pr_{A}[Aw = γ] = \frac{1}{|S|}.

Step 2. Prove that Pr_{A}[Aw = γ] = \frac{1}{|S|}.

Since w ≠ 0 there is a non-zero entry somewhere in w. Let the i’th element of w be not zero (i.e. be equal to 1).

Look at the i’th column of A. A is totally random, and its i’th column is also random and independent of all other columns of A. This column will add to Aw, consequently Aw is also fully random (uniformly distributed).

More formally we can write it in the following way. Let A = (C_{1}, …, C_{l}), where C_{1}, … C_{l} are columns of A. They are selected independently and uniformly of one another.

Pr_{A}[Aw = γ] = Pr_{C_{1},...,C_{l}}[ ∑_{j} C_{j}w_{j} = γ],

∀C_{1}, …, C_{i−1}, C_{i+1}, …, C_{l} Pr_{C_{i}}[C_{i} = γ − ∑_{j≠i} C_{j}w_{j}] = \frac{1}{|S|}.

Now the same argument as in Step 1 works: the probability for every fixed tuple of values of non-i’th columns is \frac{1}{|S|}, hence it is not changed when \{C_{j}\}_{j≠i} are random.

Step 1 together with Step 2 complete the proof of the claim. We also noticed that conclusions of Step 1 were made without exploiting information about internal structure of matrix A, while Step 2 relies on the fact that the columns of A are independent.

So now we know that Ω is pairwise independent. Next we will consider improvement of the construction, that enables us to pass from |Ω| ∼ 2^{nl} to |Ω| ∼ 2^{n+l}.

4.2 Improvement of initial construction (Toeplitz matrices)

4.2.1 Improved construction

To achieve smaller size of Ω we want to constrain the matrix space. For this purpose let’s consider Toeplitz matrices, i.e. matrices with equal elements on all diagonals. The construction will be the same, but we will use a small subspace of matrices.

Definition 4.1 n × l matrix T = {t_{i,j}} is called Toeplitz if for every i, j (1 ≤ i ≤ n − 1, 1 ≤ j ≤ l − 1) it holds that t_{i,j} = t_{i+1,j+1}.

The following picture illustrates the notion of Toeplitz matrix.

\underbrace{l}_{n \{ \begin{matrix}& & & 1 \\ 1 & & & & 1 \\ & 1 \\ & & 1 \\ 0 & & & 1 \\ & 0 & & & 1 \\ & & 0 \end{matrix}}

A Toeplitz matrix is fully specified by n + l − 1 bits (for example, by values in the 1st row and the 1st column). So there are 2^{n+l−1} such matrices, which gives the size of our sample space |Ω| = 2^{l+2n+1} — comparable with 2^{2n} that we had in the first construction, and much less than 2^{nl+n} in the case of arbitrary affine transformations.

4.2.2 Analysis: pairwise independence

The proof of pairwise independence is very similar to the analogous proof for the case of arbitrary matrices (Claim 4.1.1). Part of it will be literally the same, and the rest will require some extra elaboration.

Here and below we will denote by T n × l boolean Toeplitz matrix and by b boolean vector of length n.

Claim 4.2.1 (pairwise independence of the Toeplitz matrices construction)

\begin{matrix}∀u ≠ v ∈ \{0, 1\}^{l} ∀α, β ∈ S = \{0, 1\}^{n} \\ Pr_{(T,b)}[T v + b = α ∧ T u + b = β] = \frac{1}{|S|^{2}} . \end{matrix}

Proof:

Step 1. As we notices in the proof of Claim 4.1.1, Step 1 doesn’t rely on the kind of matrices we deal with. Thus the results obtained there remain true if arbitrary matrix A is replaced by Toeplitz matrix T. So, we have “for free” the following conclusion (Eq. (4.3) of Claim 4.1.1):

Pr_{(T,b)}[T u + b = α ∧ T v + b = β] = \frac{1}{|S|}Pr_{T} [T w = γ]

where w and γ are completely arbitrary, but w ≠ 0.

Step 2. We show that Pr_{T} [T w = γ] = \frac{1}{|S|}, where w ≠ 0 and γ are some fixed vectors (if w = 0 then T w = 0 independently of random choice of T, so condition that w ≠ 0 is crucial).

We cannot simply repeat the argument we had for the case of arbitrary random matrices A, because the columns of T are no longer independent.

Now instead of using an arbitrary non-zero element in w we select a particular one. Let it be the first one, let its index be i.

According to this given i (recall that w and γ are fixed, and only T is random) we choose specific representation of T: the last i − 1 elements of the 1st column, all elements of the i’th column, and the last l −i elements of the 1st row, as described in the picture (arrows describe how values “spread” over Toeplitz matrix):

T ; \begin{aligned}↖ & & ⋄ & ⋆ & ⋆ & ⋆ \\ & ↖ & ⋄ & & ↘ & ↘ \\ & & ⋄ & & & ↘ \\ & & ⋄ & ↘ \\ & & ⋄ & & ↘ \\ & & ⋄ & & & ↘ \\ ⋆ & & ⋄ \\ ⋆ & ↘ & ⋄ \end{aligned} w ; 0 0 1 ? = γ

So we can describe T by ⋄ and ⋆ parameters: T ≡ (⋄, ⋆), where ⋄ is the i’th column in T, ⋆ – the other part of the representation. These two parts of representation are independent from each other (for uniformly chosen T). We will show below a procedure, which for a fixed ⋆ determines ⋄ uniquely (from the equation T (⋄, ⋆)w = γ). This implies that for any value of ⋆, the conditional probability Pr[T w = γ| ⋆] equals \frac{1}{|S|} (because T w = γ for given ⋆ determines a unique column ⋄ out of the 2^{l} = |S| possible ones in the residual probability spcae). This fact combined with the independence of ⋆ and ⋄ will give us what we need: Pr[T w = γ] = \frac{1}{|S|} (by the same argument as in the Claim 4.1.1).

To complete the proof we only need to describe the procedure of solving equation T (⋄, ⋆)w = γ with respect to ⋄.

We will solve this equation row-by-row from top to bottom. First i − 1 columns of T don’t influence T w (because they correspond to elements of w that are zero), so we can simply ignore them. Also taking into account that w_{i} = 1, we can write equation T w = γ in the following detailed form:

\{ \begin{matrix}t_{1,i} + t_{1,i+1}w_{i+1} + … + t_{1,l}w_{l} = γ_{1} \\ t_{2,i} + t_{2,i+1}w_{i+1} + … + t_{2,l}w_{l} = γ_{2} \\ … \\ t_{j,i} + t_{j,i+1}w_{i+1} + … + t_{j,l}w_{l} = γ_{j} \\ … \\ t_{n,i} + t_{n,i+1}w_{i+1} + … + t_{n,l}w_{l} = γ_{n} \end{matrix}  (4.4)

The leftmost terms of these equations form precisely our unknown ⋄ = {t_{1,i}, t_{2,i}, …, t_{n,i}}. Let’s now look at the first row in (4.4). All values t_{1,i+1}, …, t_{1,l} are known (they are from ⋆, fixed in advance), hence we determine t_{1,i} (the first element of ⋄) in the obvious way: t_{1,i} = γ_{1} − t_{1,i+1}w_{i+1} − … − t_{1,l}w_{l}.

But now, since we know the value of t_{1,i}, we can “propagate” it down the diagonal (i.e., it determines t_{2,i+1}). Then in the second row equation in (4.4) will remain only one unknow t_{2,i}.

In general, suppose we have found a unique specification for t_{1,i}, …, t_{j,i}, j < n. Since we work with a Toeplitz matrix, this gives us values of all elements of T in the (j + 1)’s row except for t_{j+1,i} (black circles • in the picture below represent already known values of ⋄):

1 → • ⋆ ⋆ ⋆ ⋆
• ^{↘} ^{↘}
• ^{↘} ^{↘} ^{↘}
j + 1 → ⋄ ^{↘} ^{↘} ^{↘} ↘
⋄
⋄

So we find a unique specification for t_{j+1,i} from (j + 1)’s equation in (4.4):

t_{j+1,i} = γ_{j+1} − t_{j+1,i+1}w_{i+1} − … − t_{j+1,l}w_{l}

(we showed that all values on the right-hand side are known from the previous steps).

Proceeding in this manner to the n’s row, we determine the whole ⋄ element by element.

Remark: The construction of small pairwise independent sample spaces, presented here, is certainly not limited to binary strings (S = {0, 1}^{n}). We can be easily extended to sequences over any field F (for example, over field F = {0, …, p − 1} with usual modular multiplication and addition, where p is a prime). We can simply replace 2-element field F_{2} = {0, 1} by an arbitrary field F (then S = F^{n}), and all our arguments will remain valid (because they only depend on the distinction between zero and non-zero elements). The same remark refers to the first construction.

4.3 Hash Functions

4.3.1 Small pairwise independent sample spaces viewed as hash functions

Consider sequences of m elements over S as functions from [m] to S. Then we can write our small sample space Ω as space of hash functions:

Ω = {h : [m] → S} = H

or, in our case (i.e. the second construction), Ω = {h : {0, 1}^{l} → {0, 1}^{n}}. We obtain hash functions (i.e. shrinking of input) when l > n.

Even in the first construction of small pairwise independent sample space, using procedure of expansion to the case m > |S| (recall that the basic case was m ≤ |S|), we get hash functions. If we have an embedding μ of large space S′ (identified with a finite field), |S′| ≥ m into our small initial space S, and polynomial P over field of size m, we get hash function h = μ ∘ P from S′ to S.

This gives us the way of obtaining hash functions when we need k-wise independence, k ≠ 2 (in this case the less problematic second construction doesn’t work).

4.3.2 Good hash functions

Hash functions map long strings into short ones. We expect from hash functions a sort of “justice”: each short string is hit by approximately the same number of long strings.

Definition 4.2 Hash function h is called k-wise independent, if for any distinct i_{1}, …, i_{k} and any α_{1}, …, α_{k}

Pr_{h∈_{R}H} [h(i_{1}) = α_{1} ∧ ⋯ ∧ h(i_{k}) = α_{k}] = \frac{1}{|S|^{k}}

In the first construction, the representation of k-wise independent hash function is of size k · l. When considering k = 2 this representation is compatible to l + 2n + 1, obtained in our second construction.

Hash functions can be computed rather efficiently, in polynomial time or even better.

4.3.3 Applications of hash functions

Typically in applications we want to map large space {0, 1}^{l} into small space {0, 1}^{n} with hash functions, so that this mapping looks like random.

To define an arbitrary (boolean) hash function we need 2^{l} bits – to specify value at every point. To define a k-wise independent hash function we need only kl bits.

But usually we want even more: we have a subset I ⊂ {0, 1}^{l}, and wish our mapping to be uniform also with respect to this subset I. If |I| < 2^{n} we want that there will be no collisions, and even when I is bigger than the target set, we still want the mapping to distribute I evenly.

This case (I is a bit larger than {0, 1}^{n}) often appears in complexity theory.

It turns out, that even pairwise independence gives us a considerable degree of uniformness in distribution, as states the following Lemma 4.3.1.

4.3.4 Hashing Lemma

In the following lemma, one may think of m > 2^{n}, although the Lemma holds also fo m ≤ 2^{n}. Note that μ =^{def} \frac{|I|}{2^{n}} is the expected number of elements in I that are mapped by a random function (or uniformly chosen hash function) to any fixed point. Our focus is on |I| > 2^{n} (i.e., μ > 1), but the lemma holds also for |I| ≤ 2^{n} (i.e., μ ≤ 1).

Lemma 4.3.1 Let H = {h : [m] → {0, 1}^{n}} be a family of pairwise independent hash functions. Let I ⊆ [m] and μ =^{def} \frac{|I|}{2^{n}}. Then for every α ∈ \{0, 1\}^{n}

Pr_{h∈H} [| |\{i ∈ I : h(i) = α\}| − μ| > ϵμ] < \frac{1}{ϵ^{2}μ}

Remarks: Recall that μ is the expected number of hits; and so the bound is meaningful when ϵ > \frac{1}{\sqrt{μ}} (in which case it is less than 1). Setting ϵ = μ^{− \frac{1}{3}} , we obtain

Pr_{h∈H} [| |\{i ∈ I : h(i) = α\}| − μ| > μ^{\frac{2}{3}} ] < μ^{− \frac{1}{3}}

which is meaningful for μ > 1.

Proof: The main thing in this proof is to define an appropriate random variable and makes some simple observations. After that the conclusion comes out naturally.

Fixing α ∈ {0, 1}^{n}, define for each i ∈ I a random variable X_{i}:

X_{i} =^{def} \begin{cases}1 & if h(i) = α \\ 0 & otherwise \end{cases}

(the sample space here is H, and so X_{i} is actually X_{i}(h)).

Notice the following facts:

1. Pr[X_{i} = 1] = 2^{−n}, because for fixed i and random h ∈ H the value h(i) is uniformly distributed in {0, 1}^{n}. Therefore E(X_{i}) = 1 · Pr[X_{i} = 1] + 0 · Pr[X_{i} = 0] = 2^{−n}.

2. The X_{i}’s are pairwise independent (they are functions of pairwise independent values, i.e. the values h(i)’s).

3. var(X_{i}) = E(X_{i}^{2}) − E(X_{i})^{2} = E(X_{i}) · (1 − E(X_{i})) ≤ E(X_{i}) (valid for any {0, 1} -valued random variables).

4. For a random choice of h the number of elements from I that hit a particular element from

{0, 1}^{n} equals |{i ∈ I : h(i) = α}| = ∑_{i∈I} X_{i}.

5. E( ∑_{i∈I} X_{i}) = ∑_{i∈I} E(X_{i}) = \frac{|I|}{2^{n}} = μ.

6. var( ∑_{i∈I} X_{i}) = ∑_{i∈I} var(X_{i}) ≤ ∑_{i∈I} E(X_{i}) ≤ μ.

These facts together with Chebishev’s inequality (applied to ∑_{i∈I} X_{i}) imply:

Pr [|∑_{i∈S} X_{i} − E( ∑_{i∈S} X_{i})| > ϵμ] < \frac{1}{(ϵμ)^{2}} · var( ∑_{i∈I} X_{i}) ≤ \frac{μ}{ϵ^{2}μ^{2}} = \frac{1}{ϵ^{2}μ}

and the lemma follows.

Oded’s Note: Using a k-wise independent family of hash functions allows to prove a stronger bound than the one provided by Lemma 4.3.1. Specifically, for such a family H and notations as in Lemma 4.3.1 the following holds: for every α ∈ {0, 1}^{n}

Pr_{h∈H} [| |\{i ∈ I : h(i) = α\}| − μ| > ϵμ] < (\frac{k^{2}}{ϵ^{2}μ})^{k/2}

The proof follows by using the stronger bounds provided in Lecture 2 for the sum of k-wise independent random variables. In particular, using the union bound and k = n, we get

Pr_{h∈H} [∃α ∈ \{0, 1\}^{n} s.t. | |\{i ∈ I : h(i) = α\}| − μ| > ϵμ] < (\frac{4k^{2}}{ϵ^{2}μ} )^{k/2}

which will be used in Lecture 6.

Oded’s Note: An alternative way of stating Lemma 4.3.1 follows. Let H, I ⊆ [m] and μ = |I|/2^{n} be as in the lemma, and let X be uniformly distributed over I. Then, for every α ∈ {0, 1}^{n}, for all but an ϵ^{−2}μ^{−1} fraction of the h’s in H it holds that

|Pr [h(X) = α] − μ · 2^{−n}| ≤ ϵμ · 2^{−n}

where the probability is taken over the choice of X (i.e., uniformly over I). A different generalization of Lemma 4.3.1 is obtained by considering arbitrary random variables X over [m] satisfying Pr[X =x] ≤ μ^{−1}2^{−n}, for every x.

Bibliographic Notes

The constructions of pairwise independent sample spaces and hash functions (presented above) are due to Carter and Wegman [15]. Versions of the hashing lemma were discovered independently in [26] and [13]. The lemma can be viewed as an extension (or quantification) of the ideas underlying Sipser’s work [40].

48 LECTURE 4. PAIRWISE-INDEPENDENT SPACES (CONT.) AND HASH FUNCTIONS

Approximate Counting

Notes taken by Oded Lachish, Eran Ofek, Udi Wieder

Summary: This lecture discusses quantitative problems related to NP relations. Namely, given an element in an NP language, we ask how many witnesses this element has. We use Hash functions in order to prove that given an oracle to NP and an instance in an NP language, it is possible to approximate the number of witnesses for that instance.

5.1 Introduction

Given a set S, a few natural questions one can ask about S can be divided into two categories.

1. Qualitative Questions:

• Decision Question - is S empty or not? (I.e., S =^{?} ϕ)

• Search Query - produce an element of S. Find y such that y ∈ S.

2. Quantitative Questions:

• Counting - What is the cardinality of S. (I.e., |S| =?)

• Uniform Generation - Generate an element of S with uniform probability among all possibilities.

Typically quantitative questions are harder to answer than qualitative ones. If one can count the number of elements in S then one can surely tell whether S is empty or not. We investigate what happens the other way around; namely, when being able to answer qualitative questions, what can be said about the ability to answer the quantitative ones?

5.1.1 Formal Setting

We are given a family of sets {S_{x}}_{x∈{0,1}^{∗}} where:

• S_{x} ⊆ {0, 1}^{∗}. Furthermore there exists a polynomial p(·) such that the length of elements in S_{x} is at most p(|x|).

• Given x and y it is easy (=done in polynomial time) to check whether y ∈ S_{x}.

In other words, the family {S_{x}}_{x∈{0,1}^{∗}} defines an NP relation. Recall:

Definition 5.1 (NP relation) : An NP relation is a relation R ⊆ {0, 1}^{∗} × {0, 1}^{∗} such that:

• R is polynomial time decidable.

• There exists a polynomial p(·) such that for every (x, y) ∈ R, it holds that |y| ≤ p(|x|).

We let S_{x} be the witness set of x (or set of solutions for x); i.e., S_{x} =^{△} {y : (x, y) ∈ R}. The witness set of x (with respect to relation R) is also denoted by R(x).

Using this notation, we rephrase the four problems at hand:

1. The Decision Question is the usual NP decision problem, namely given x check if ∃y such that (x, y) ∈ R. In other words, check if R(x) =^{?} ϕ.

2. The Search Problem is: given x find y such that y ∈ R(x).

3. The Counting Problem: given x compute |R(x)|.

4. The Generation Problem: given x, generate y uniformly in R(x).

5.1.2 How close are the quantitative questions to the qualitative ones?

First we investigate the connection between the two qualitative questions. How close are the Decision Problem and the Search Problem? Surly, given an oracle to the Search Problem, it is easy to solve the Decision Problem. A well known result shows that for NP-complete relations the other way around holds as well. In other words:

Fact 5.1.1 (Self Reducibility of NP): For any NP-relation R, given an element x and an oracle access to NP decision questions, it is possible in polynomial time to find y such that y ∈ R(x).

Note that the oracle can answer any query which is an NP decision problem, and not necessarily queries regarding the specific relation R. In case R is NP-complete, we can use an oracle to L(R) =^{△} {x : R(x) ≠ ϕ} instead of an oracle to NP.

Fact 5.1.1 demonstrates a tight connection between the two qualitative questions. In the rest of the lecture we show that in some sense the quantitative questions are not much harder than the qualitative ones. Namely, given an oracle to the Decision Question, it is possible to approximate the number of witnesses of an element. In the next lecture we will present a result in the spirit of Fact 5.1.1 that demonstrates a tight connection between counting and uniform generation.

5.2 Approximating the size of of the witness set with oracle access to NP

Goal: For a fixed NP-relation R, provide an procedure that, given an element x, an approximation paramater ϵ > 0, and oracle access to NP, runs in time poly (\frac{|x|}{ϵ}) and satisfies:

Pr [Alg^{NP} (x) ∉ (1 ± ϵ)|R(x)| ≤ \frac{1}{3}] .

As shown in the next section, the error probability can be reduced at a cost logarithmic in the desired error bound.

5.2. APPROXIMATING THE SIZE OF OF THE WITNESS SET WITH ORACLE ACCESS TO NP51

Tools: The tools we use are a family of pairwise independent Hash functions H_{n,m}, that hashes {0, 1}^{n} → {0, 1}^{m} (m < n), and an oracle for any NP-complete language.

The strategy for solving the problem is first to show how to find a solution to the case in which |R(x)| is small. (This is done using an oracle to NP). Then show how to reduce the general case (a large witness set), to a case in which the witness set is small. (This is done using the hash functions). Using a family of pairwise-independent hash functions has a price: there is a constant probability that the hash function chosen does not behave well, we denote this probability by δ. When the chosen hash function behaves well, the answer of the algorithm will be in the interval (1 ± ϵ) · |R(x)|. Where ϵ is called the deviation parameter, and δ is called the error probability. Thus, for an NP relation R and an input x, the algorithm we describe efficiently produces a result that behaves as follows:

P rob [Alg^{N P}(x) ∉ (1 ± ϵ)|R(x)|] < δ

where the probability is over the coin tosses of the algorithm.

5.2.1 The algorithmic scheme

Assume that |x| = n and that R(x) ⊆ {0, 1}^{n} (this assumption is with out loss of generality, since padding can bring the element x and the witnesses to the same size). Let l = O(lg n). Following is a sketch of the algorithm (which is actually an algorithmic scheme):

Algorithm A:

1. If |R(x)| < 2^{l} output |R(x)|.

2. If |R(x)| ≥ 2^{l} then

For i = 1 to n − l
• Select h ∈_{R} H_{n,i}.
• If |R(x) ∩ h^{−1}(0^{i})| < 2^{l} then
output 2^{i} · |R(x) ∩ h^{−1}(0^{i})| and Halt.

In Stage 1 we handle the case in which |R(x)| is small. In the iterations of Stage 2 we hash the witness set into an increasingly larger set, until the number of witnesses that are mapped to 0^{i} is small. Specifically, if the hash function maps \{0, 1\}^{n} to \{0, 1\}^{i}, we expect it to map \frac{|R(x)|}{2^{i}} witnesses to each element in {0, 1}^{i}. Once 2^{i} is the ’right’ number, we have that a small number of elements are mapped into 0^{i}. (0^{i} was chosen arbitrarily). This case is similar to the case of a small a witness set, so is identifiable, and can be counted exactly. As a result the size of the set of witnesses mapped to 0 is used to approximate the entire number of witnesses.

5.2.2 Implementation of the algorithmic scheme

How to perform Stage (1): We check if |R(x)| < 2^{l} by defining the following NP relation:

R′ = {x, (y_{1}, …, y_{2^{l}}) : y_{1} ≠ … ≠ y_{2^{l}} and (x, y_{1}), … (x, y_{2^{l}}) ∈ R}.

That is, L(R′) = {x : ∃y′(x, y′) ∈ R′} contains x iff x has at least 2^{l} distinct witnesses (i.e. |R(x)| ≥ 2^{l}). Since R is an NP relation we know that the length of each y_{i} is polynomially bounded by the length of x. Since 2^{l} is polynomial in |x| we have that the length of (y_{1}, y_{2}, …, y_{2^{l}}) is also polynomially bounded by |x|. We conclude that R′ is an NP relation.

Thus, to check whether |R(x)| ≥ 2^{l} we query the oracle whether R′(x) =^{?} ϕ. If the oracle answers ’NO’ then |R(x)| < 2^{l}. To compute the exact size of |R(x)| define the following NP relation:

R′′ = {(x, 1^{t}), (y_{1}, …, y_{t}) : y_{1} ≠ … ≠ y_{t} and (x, y_{1}), … (x, y_{t}) ∈ R}.

That is, the language L(R′′) contains (x, 1^{t}) iff x has at least t distinct witnesses (i.e. |R(x)| ≥ t). In order to ensure that R′′ is indeed an NP relation, we have to ensure that the length of the witnesses is polynomially bounded by the length of the instances. Note that t is represented in unary, therefore the length of the instance (x, 1^{t}) is |x| + t. The witnesses are composed of t y_{i}’s, whose lengths are polynomially bounded by |x|. Therefore, for every t, the length of (y_{1}, y_{2}, …, y_{t}) is polynomially bounded by |(x, 1^{t})| and we conclude that R′′ is an NP relation. Thus, to find the exact size of R(x), we iteratively query the oracle on the decision problem of R′′(x, 1^{t}) for t = 1 … 2^{l} and find the largest t for which (x, 1^{t}) ∈ L(R′′). Note that 2^{l} is polynomial in n, thus the whole operation runs in polynomial time.

How to perform stage (2): For each iteration i define the following NP relation:

R′′′ = {(x, h), (y_{1}, …, y_{2^{l}}) : y_{1} ≠ … ≠ y_{t} and h(y_{1}) = h(y_{2}) = ⋯ = h(y_{2^{l}}) = 0^{i}
and (x, y_{1}), …, (x, y_{2^{l}}) ∈ R}

That is, L(R′′′) contains (x, h) iff x is in L(R), and it has at least 2^{l} distinct witnesses mapped to 0^{i} by h. Note that the hash function can be evaluated in time that is polynomial in |x|. Therefore it is possible to verify in polynomial time whether ((x, h), (y_{1}, y_{2}, …, y_{2^{l}})) ∈ R′′′. Thus L(R′′′) ∈ N P. Furthermore, since h can be represented in space which is polynomial in |x|, the length of (x, h) is polynomial in |x|, and it is possible to produce an NP query for R′′′ in polynomial time (in |x|). Querying the oracle about R′′′(x) enables us to identify when less than 2^{l} distinct witnesses were mapped to 0^{i} by h. In this case, finding the exact number of such witnesses, is done in a similar manner to the first case when |R(x)| was small.

5.2.3 Analysis of the algorithm

The parameter l is chosen to be big enough so that ϵ ≥ 10 · 2^{−l/2} and also 2^{l} ≥ n^{2}. Note that since the running time is polynomial in 2^{l}, it follows that the running time is polynomial in max\{n, \frac{1}{ϵ}\}. This means that ϵ can be at best small as \frac{1}{poly(n)}, otherwise the running time would not be polynomial in n. Let t be such that 2^{t} = |R(x)| (for sake of simplicity and without loss of generality assume t is an integer).

General strategy of the proof: We will show that the algorithm is not likely to terminate before iteration t − l; that is, with high probability it will reach iteration t − l. Then, we show that given that A reached iteration t − l + 1 it is very likely that A will terminate in this iteration. This implies that the algorithm is likely to terminate in one of the iterations {t − l, t − l + 1}. The last step is to show that in both iterations (i = t − l or i = t − l + 1) the value |R(x) ∩ h^{−1}(0^{i})| · 2^{i} is an ϵ approximation of |R(x)| with high probability. Thus if A terminates in these iterations (as expected) it will output an ϵ approximation of |R(x)| with high probability.

Recall the hashing lemma from the previous lecture:

Lemma 5.2.1 (Hashing lemma) Let H = {h : {0, 1}^{n} → {0, 1}^{i}} be a family of pairwise independent hash functions. Let I ⊆ \{0, 1\}^{n} and α ∈ \{0, 1\}^{i} ,and let ρ = \frac{|I|}{2^{i}}. Then

Pr_{h} [||I ∩ h^{−1}(α)| − ρ| ≥ ϵ · ρ] ≤ \frac{1}{ϵ^{2} · ρ}

where h is uniformly selected in H.

We will use this lemma to prove that algorithm A will return a good approximation of |R(x)|, with high probability. We do this by examining the value |R(x) ∩ h^{−1}(0^{i})| in various iterations of A. Let us fix some i (iteration number in the running of A).

Recall that since h ∈ H is pairwise independent, it follows that

E_{h} [|R(x) ∩ h^{−1}(0^{i})|] = \frac{|R(x)|}{2^{i}}  (5.1)

We denote ρ =^{def} \frac{|R(x)|}{2^{i}}. For every y ∈ R(x) denote by X_{y} the following random variable:

X_{y} = X_{y}(h) = \{ \begin{matrix}1 & h(y) = 0^{i}, \\ 0 & otherwise. \end{matrix}

where the probability space is of all choices of h ∈ H. Recall that by the properties of H, in iteration i we have Pr_{h}[X_{y} = 1] = \frac{1}{2^{i}} . Notice that

|R(x) ∩ h^{−1}(0^{i})| = ∑_{y∈R(x)} X_{y}

Thus, by linearity of the expectation:

E_{h} [|R(x) ∩ h^{−1}(0^{i})|] = ∑_{y∈R(x)} E_{h}(X_{y})

= ∑_{y∈R(x)} Pr[X_{y} = 1] = \frac{|R(x)|}{2^{i}} = ρ

which indeed agrees with equation (5.1).

First we show that the algorithm A is not likely to terminate in any of the iterations i ∈ {1, 2, .., t − l − 1} .

Claim 5.2.2 For every i ∈ {1, 2, .., t − l − 1} the probability that A terminates at iteration i is smaller than \frac{1}{10n} .

Proof:

ρ = \frac{|R(x)|}{2^{i}} ≥ \frac{2^{t}}{2^{t−l−1}} ≥ 2^{l+1}

We wish to bound the probability that algorithm A terminates at iteration i, so we need to bound the probability that |R(x) ∩ h^{−1}(0^{i})| < 2^{l}. Notice that if |R(x) ∩ h^{−1}(0^{i})| < 2^{l} then ||R(x) ∩ h^{−1}(0^{i})| − ρ| > \frac{1}{2} · ρ (this is true since ρ ≥ 2^{l+1}). By using the hashing lemma we can bound the probability that ||R(x) ∩ h^{−1}(0^{i})| − ρ| > \frac{1}{2} · ρ:

(The set R(x) ⊆ {0, 1}^{n} plays the roll of the set I from the lemma).

Pr_{h} [||R(x) ∩ h^{−1}(0^{i})| − ρ| > \frac{1}{2} · ρ] ≤ \frac{1}{(\frac{1}{2})^{2} · ρ}

≤ \frac{1}{(\frac{1}{2})^{2} · 2^{l+1}} = 2^{−l+1} ≤ 2 · \frac{1}{n^{2}} ≤ \frac{1}{10 · n}

(for big enough n’s).

Now we will show that A is not likely to continue beyond iteration t − l + 1.

Claim 5.2.3 Given that A reached iteration t − l + 1, the probability that A terminates at this iteration is at least 0.9.

Proof: Here, we have

ρ = \frac{|R(x)|}{2^{t−l+1}} = \frac{2^{t}}{2^{t−l+1}} = 2^{l−1}

We wish to bound the probability that algorithm A does not terminate in this iteration. This happens only if |R(x) ∩ h^{−1}(0^{i})| ≥ 2^{l} , but this event is contained in the event that ||R(x) ∩ h^{−1}(0^{i})| − ρ| ≥ ρ (since ρ = 2^{l−1}). We use the hashing lemma to bound this probability:

Pr_{h} [|R(x) ∩ h^{−1}(0^{i})| − ρ|] ≥ 1 · ρ] ≤ \frac{1}{ρ}

= \frac{1}{2^{l−1}} = 2^{−l+1} = 2 · \frac{1}{n^{2}} ≤ 0.1

(for large enough n’s)

It remains to show that given that A terminates at iteration t − l or t − l + 1 it is likely that A outputs an ϵ approximation of R(x).

Claim 5.2.4 Assume that A terminates at iteration t − l or t − l + 1, then

Pr_{h} [A(x) ∉ (1 ± ϵ) · |R(x)|] ≤ \frac{1}{10}

Proof: Here, we have

ρ = \frac{|R(x)|}{2^{i}} ≥ \frac{2^{t}}{2^{t−l+1}} = 2^{l−1}

Since \frac{A(x)}{2^{i}} = |R(x) ∩ h^{−1}(0^{i})|, it follows that the following two conditions are equivalent:

|A(x) − |R(x)|| ≥ ϵ · |R(x)|  (5.2)
||R(x) ∩ h^{−1}(0^{i})| − \frac{|R(x)|}{2^{i}} | ≥ ϵ · \frac{|R(x)|}{2^{i}}  (5.3)

Equation (5.3) can be written as:

||R(x) ∩ h^{−1}(0^{i})| − ρ| ≥ ϵ · ρ

Using the hashing lemma, we have

Pr_{h} [||R(x) ∩ h^{−1}(0^{i})| − ρ| ≥ ϵ · ρ] ≤ \frac{1}{ϵ^{2} · ρ}
≤ \frac{1}{ϵ^{2} · 2^{l−1}}
≤ \frac{1}{(10 · 2^{l/2})^{2} · 2^{l−1}} ≤ \frac{1}{10}

Using the equivalence of equations (5.2) and (5.3), the claim follows.

Corollary 5.2.5 The probability that A outputs an ϵ approximation of |R(x)| is at least \frac{2}{3}.

Proof: The probability that A terminates before iteration t − l is at most \frac{1}{10} (using the union bound with Claim 1). It follows that the probability for algorithm A to terminate at iteration t − l or at iteration t − l + 1 is at least 0.8, because

Pr[A terminates at iteration t-l or t-l+1] =

Pr [ \begin{matrix}A reaches, \\ iteration t-l \end{matrix} ] · Pr [ \begin{matrix}A terminates at, \\ iteration t-l or t-l+1 \end{matrix} | \begin{matrix}A reaches, \\ iteration t-l \end{matrix} ]

≥ 0.9 · 0.9 > 0.8

Using again the union bound with Claim 3 we conclude that the probability that A terminates at iterations t − l or t − l + 1 and outputs an ϵ approximation is at least 0.8 − 0.1 > \frac{2}{3}.

5.3 Amplifying the success probability

In the previous section we saw that we can approximate the size of the witness set with deviation ϵ, and in running time that is polynomial in \frac{1}{ϵ}. The success probability of the approximation was shown to be larger than \frac{2}{3}. Like in many other probabilistic algorithms, we wish to amplify the success probability. We will show that in polynomial time, it is possible to reduce the error probability to be exponentially small. We start tackling this problem in a standard way: assume we wish the error probability to be smaller than δ. We set t =^{△} O(log(\frac{1}{δ})). The algorithm A is invoked t times, each invocation with fresh random coins, and we receive t (possibly different) answers. The question is what to do with these t answers? At first thought one might think of taking the average of these results. On second thought this idea should be excluded, since the wrong solutions might be very wrong. If it happens that the algorithm miserably fails, it might give an answer that is so far from the true one, that it would skew the average. Consider the following illustrative example: Assume |R(x)| = 2^{\frac{n}{2}} , and Pr[A(x) = 2^{\frac{n}{2}} ] = \frac{2}{3}, yet Pr[A(x) = 2^{n}] = \frac{1}{3}. That is, even though A is correct most of time, if it fails it fails miserably. Now the average is E (A(x)) = \frac{2}{3} · 2^{\frac{n}{2}} + \frac{1}{3} · 2^{n} = Θ(2^{n}) , which is very far from the correct answer.

A much better idea is to use the median of the t results. Let A′ be the following algorithm:

Algorithm A’:

1. Set t =^{△} O(log(\frac{1}{δ}))

2. Invoke the algorithm A, t times, each invocation with fresh random coins, and receive t answers.

3. Output the median of the t answers.

Claim 5.3.1 P rob [A′(x) ∉ (1 ± ϵ) · |R(x)|] ≤ δ

Proof: We use the following fact:

Fact 5.3.2 If the median of t numbers is outside the range (a, b) then at least \frac{t}{2} of the numbers are outside the range (a, b).

Define random variables

X_{i} = \{ \begin{matrix}1 & the i’th invocation of A gives a value in (1 ± ϵ) · |R(x)|, \\ 0 & otherwise. \end{matrix}

Note that Pr[X_{i} = 1] ≥ \frac{2}{3}, therefore E(∑_{i=1}^{t} X_{i}) ≥ \frac{2t}{3}. According to Fact 5.3.2, if A′ fails then at least half of the invocations of A failed. This means that (in case of failure) ∑_{i=1}^{t} X_{i} ≤ \frac{t}{2} and thus the sum is very far from it’s expectation. We use the Chernoff bound to show that this is very unlikely. According to Fact 5.3.2 we have:

Pr[A′(x) ∉ (1 ± ϵ) · |R(x)|] ≤ Pr [∑_{i=1}^{t} X_{i} ≤ \frac{t}{2}]

≤ Pr [|∑_{i=1}^{t} X_{i} − E(∑_{i=1}^{t} X_{i})| ≥ \frac{t}{6}]

≤ exp (− (\frac{1}{6})^{2} · t)

where the last inequality is due to Chernoff Bound. Thus, setting t = 36 ln (\frac{1}{δ}) we get Pr[A′(x) ∉ (1 ± ϵ) · |R(x)|] ≤ δ , and the claim follows.

Bibliographic Notes

The approximate counting (oracle) procedure presented above is a variant of a procedure originally suggested by Stockmeyer [42].

Uniform Generation

Notes taken by Eden Chlamtac and Shimon Kogan

Summary: This lecture continues the discussion of approximate counting and uniform generation using an NP-oracle begun in the previous lecture, where we saw an approximate counting algorithm. Here we will see the equivalence (up to Turing reductions) of approximate counting to uniform generation of NP-witnesses. Together with the approximate counting algorithm from last lecture, this yields a uniform generation algorithm using an NP-oracle. We conclude with an alternative (direct) uniform generation algorithm, which uses n-wise independent hash functions (as opposed to the pairwise independent hash functions used for approximate counting).

6.1 Introduction

Recall from the previous lecture that we were concerned with answering quantitative questions about sets for which we have an oracle. Specifically, given some NP-relation R ⊆ {0, 1}^{∗} × {0, 1}^{∗}, and some input string x ∈ {0, 1}^{n}, we would like to answer quantitative questions regarding the set

S = R(x) =^{def} {y : (x, y) ∈ R} ⊆ {0, 1}^{n}

(Note that we assume if y ∈ R(x) then |y| = n = |x|. This is without loss of generality— it can be achieved with suitable padding.)

The previous lecture was concerned with approximate counting—approximating the size of S. In this lecture we are concerned with the closely related problem of uniform generation, or sampling with uniform distribution an element in S. In fact, we will show that the two computational problems are equivalent. That is, we show that Approximate Counting and Uniform Generation are computationally equivalent.

6.2 Approximate Counting Reduces to Uniform Generation

At our current state of knowledge (i.e., having seen n approximate counting algorithm in the previous lecture but not a uniform generation one), this is the less interesting direction of the equivalence. Still, it is simpler to explain and is needed to establish the computational equivalence of the two tasks.

6.2.1 Intuition

Assume we have a generic uniform generation algorithm UniGen, which for any NP-relation R and input x outputs UniGen_{R}(x)—a string chosen with uniform distribution from the set R(x).

Using UniGen we will generate a sufficiently large sample from |S| to estimate what fraction of the strings in |S| start with a particular bit σ. Denoting this subset of S by S_{σ}, we estimate

p_{σ} =^{def} \frac{|S_{σ}|}{|S|}

We then recursively approximate the quantity |S_{σ}|, and then produce an approximation for |S| using the formula

|S| = \frac{1}{p_{σ}} · |S_{σ}|

The recursion will proceed to S_{σ} such that p_{σ} > \frac{1}{2} (the larger of the two values p_{σ} and pσ̄), since we need a lower bound on p_{σ} in order to get a good bound on an estimate of the form \frac{1}{p_{σ}±ϵ}.

Our goal is to define an algorithm Approx, which, given some arbitrarily small parameters ϵ > 0, δ > 0, satisfies:

Pr[Approx_{R}(x) ∉ (1 ± ϵ) · |R(x)|] < δ

6.2.2 Formal Definition

Recall that our uniform generation algorithm relates to the NP-relation R. In order to apply it to subsets of S we need to define new NP-relations. Formally, we define

R′ =^{def} {((x, α), β) : α, β ∈ {0, 1}^{∗}, (x, α ∘ β) ∈ R}

(Here, α ∘ β denotes the concatenation of the two strings α and β.) Clearly, if R is an NP-relation, then R′ is also an NP-relation. Moreover, for any α_{0} ∈ {0, 1}^{∗}, R′ satisfies

R′(x, α_{0}) = {β : (x, α_{0} ∘ β) ∈ R} ≅ {α ∘ β ∈ R(x) : α = α_{0}}

Note that the set on the right is of the sort we want to quantify in the recursion.

Note that since the error and deviation bounds will increase with successive approximations at various levels of recursion, we will have to start out with stricter bounds. Specifically, we take

ϵ̃ = \frac{ϵ}{3n} and δ̃ = \frac{δ}{n}

(Here ϵ and δ are the deviation and error parameters in the performance specification noted above).

Since |R(x)| = |R′(x, λ)| (where λ is the empty string), for the sake of uniformity of notation, we present an algorithm for approximating |R′(x, α)| where the string α ∈ {0, 1}^{∗} is part of the input (along with x ∈ {0, 1}^{n} of course), and initially α = λ.

6.2.3 Algorithm Approx_{R′} (x, α)

1. We let n =^{def} |x| as before. If |α| = n return \{ \begin{matrix}1 & if α ∈ R(x) \\ 0 & otherwise \end{matrix} 2. Otherwise, for i = 1, …, m uniformly and independently generate sample points in R′(x, α):

s_{i} ← UniGen_{R′}(x, α)

where

m = O (\frac{1}{ϵ̃^{2}} log \frac{1}{δ̃})

3. For σ = 0, 1 set

p̃_{σ} =^{def} \frac{|\{i : s_{i} ∈ R′(x, α ∘ σ)\}|}{m}

(an approximation for p_{σ} =^{def} \frac{|R′(x,α∘σ)|}{|R′(x,α)|} )

4. For σ s.t. p̃_{σ} ≥ p̃σ̄ return

\frac{1}{p̃_{σ}} · Approx_{R′} (x, α ∘ σ)

6.2.4 Analysis

We would like to see how the error and deviation bounds propagate through the recursion. First note that in step (2), at every level of the recursion, by choice of m and applying Chernoff bound, we get:

Pr[|p_{σ} − p̃_{σ}| > ϵ̃] < δ̃

At the ith level of recursion (where |α| = i) we will denote the error and deviation bounds by δ_{i} and ϵ_{i}, so that:

Pr[Approx_{R′} (x, α) ∉ (1 ± ϵ_{i}) · |R′(x, α)|] < δ_{i}

Given these two probability bounds, let us show how the error and deviation propagate from level i + 1 of the recursion up to level i. Recall that:

Approx_{R′} (x, α) =^{def} \frac{Approx_{R′} (x, α ∘ σ)}{p̃_{σ}}

We know that with probability > 1 − (δ̃ + δ_{i+1}) we have both |p_{σ} − p̃_{σ}| ≤ ϵ̃ and

Approx_{R′} (x, α ∘ σ) ∈ (1 ± ϵ_{i+1}) · |R′(x, α)|

In this case it holds that:

\frac{Approx_{R′} (x, α ∘ σ)}{p̃_{σ}} = \frac{1 ± ϵ_{i+1}}{p̃_{σ}} · |R′(x, α ∘ σ)|
= \frac{1 ± ϵ_{i+1}}{p̃_{σ}} · p_{σ} · |R′(x, α)|
= \frac{(p̃_{σ} ± ϵ̃) · (1 ± ϵ_{i+1})}{p̃_{σ}} · |R′(x, α)|
= ((1 ± ϵ_{i+1}) ± \frac{ϵ̃}{p̃_{σ}} (1 ± ϵ_{i+1})) · |R′(x, α)|
= (1 ± (ϵ_{i+1} + \frac{ϵ̃}{p̃_{σ}} (1 + ϵ_{i+1}))) · |R′(x, α)|

where the second equality follows from from the equality |R′(x, α ∘ σ)| = p_{σ}|R′(x, α)|, which follows immediately from the definition of p_{σ}. Now it remains to bound

ϵ_{i} =^{def} ϵ_{i+1} + \frac{ϵ̃}{p̃_{σ}} (1 + ϵ_{i+1})

Recall that by our choice of p̃_{σ}, we know

p̃_{σ} ≥ 1/2

Also, as will be apparent from the following analysis, if we assume w.l.o.g. that ϵ < 1/2 then it is easy to show that

ϵ_{i} < 1/2

for all i. Keeping these two inequalities in mind, we have

ϵ_{i} = ϵ_{i+1} + \frac{ϵ̃}{p̃_{σ}} (1 + ϵ_{i+1})
< ϵ_{i+1} + \frac{ϵ̃}{1/2}(1 + (1/2))
= ϵ_{i+1} + 3ϵ̃

So clearly the bounds we chose—

ϵ̃ = \frac{ϵ}{3n} and δ̃ = \frac{δ}{n}

are sufficient.

6.2.5 An alternative reduction of Approximate Counting to Uniform Generation

Here is another way of reducing approximate counting to uniform generation, suggested by Udi Wieder. Rather than recursively taking smaller and smaller subsets of S, we may take some predefined set K (with known cardinality and easily decidable) of comparable size to S, such that S ⋂ K = ∅ , and then get a good approximation of S by using repeatedly generating random samples in S ∪ K, to approximate

β =^{def} \frac{|S|}{|S ⋃ K|}

Since |K| is known, this will yield an approximation of |S| derived from the following equation:

|S| = |K| · \frac{β}{1 − β}

Here we will want to be able to bound β both in terms of upper- and lower-bounds. The situation is similar to that which we had with p_{σ} in the last algorithm. If we had β ≈ 0 or β ≈ 1 then knowing we had a good approximation of the form β ± ϵ would not help, since even a small deviation will no longer yield a good relative approximation for β or \frac{1}{1−β}. Hence, it is essential that we find a set K of comparable size to S (where S ⊆ {0, 1}^{n}).

Specifically, for all i = 1, …, n we define

K_{i} =^{def} {1^{n+1} ∘ y : y ∈ {0, 1}^{i}} ⊆ {0, 1}^{n+i+1}

S_{i} =^{def} {0^{i+1} ∘ s : s ∈ S} ⊆ {0, 1}^{n+i+1}

Note that these sets satisfy the following properties for all i:

6.3. UNIFORM GENERATION REDUCES TO APPROXIMATE COUNTING 61

• |K_{i}| = 2^{i}

• |S_{i}| = |S|

• K_{i} ∩ S_{i} = ∅ (since z ∈ K_{i} start with 1, and z ∈ S_{i} start with 0) Intuitively, S_{i}∪K_{i} is a witness set for the input x∘01^{i}. This motivates the definition of the following NP-relation:

R′′ =^{def} \{(x ∘ 01^{i}, z) : \begin{matrix}1 ≤ i ≤ |x| ∧ x ∈ \{0, 1\}^{∗} \\ ∧ [(z = 0^{i+1} ∘ y ∧ (x, y) ∈ R) ∨ ((z = 1^{n+1} ∘ y) ∧ y ∈ \{0, 1\}^{i})] \end{matrix} \}

Clearly, for some 1 ≤ i ≤ n we have \frac{1}{2}|K_{i}| ≤ |S| ≤ |K_{i}| (where S = R(x) and x ∈ \{0, 1\}^{n} as above). If β is defined as above for K = K_{i} (i.e. β = \frac{|S|}{|S|+|K_{i}|}), then we get \frac{1}{3} ≤ β ≤ \frac{1}{2}. Hence if we can find such an i, knowing β is bounded as above, we can guarantee a good approximation of |S|. Here, since the deviation does not need to propagate through any recursion (there is no recursion), we can take ϵ̃ =^{def} \frac{ϵ}{9}. The error remains as in the last algorithm (we still apply union bound on the probabilities), i.e. δ̃ =^{def} \frac{δ}{n}. Here’s the formal algorithm:

1. For i = 1 to n do:

• For j = 1, …, m uniformly and independently generate sample points in S ⋃ K_{i}:

s_{j} ← UniGen_{R′′}(x ∘ 01^{i})

where again

m = O (\frac{1}{ϵ̃^{2}} log \frac{1}{δ̃})

• Set β_{i} =^{def} \frac{|\{j:s_{j}∈S_{i}\}|}{m}

2. For i_{0} =^{def} min \{i : β_{i} ≥ \frac{1}{4}\} return

2^{i_{0}} · \frac{β_{i_{0}}}{1 − β_{i_{0}}}

6.3 Uniform Generation Reduces to Approximate Counting

Given our current state of knowledge, this is indeed the more interesting reduction.

6.3.1 Intuition

Suppose we had perfect counting. In particular, suppose we could calculate the exact quantity |S|, and for σ = 0, 1 we could calculate the size of the set of all elements in S which start with σ (denoted S_{σ}), as in the previous section. In particular, we could compute the exact values

p_{σ} =^{def} \frac{|S_{σ}|}{|S|}

We want to output y ∈ S with uniform distribution. So

• with probability p_{0} output (recursively) y ∈ S_{0} (with uniform distribution)

• with probability p_{1} output (recursively) y ∈ S_{1} (with uniform distribution)

6.3.2 Formal Definitions

Now assume we have an approximate counting algorithm Approx which for any NP-relation R and inputs x, ϵ, δ returns Approx_{R}(x, ϵ, δ) where

Pr[Approx_{R} (x, δ, ϵ) ∉ (1 ± ϵ)|R(x)|] < δ

and

Approx_{R}(x, δ, ϵ) = |R(x)| if |R(x)| = 0

Now using Approx we’ll generate a random element of R(x) bit by bit in a recursive manner. This we do once more using the modified form of R which we defined in the previous section:

R′ =^{def} {((x, α), β) : α, β ∈ {0, 1}^{∗}, (x, α ∘ β) ∈ R}

where we assume that Pr[Approx_{R′} (x, α) ∉ (1 ± ϵ)|R′(x, α)|] < δ and Approx_{R′} (x, α) = |R′(x, α)| if |R′(x, α)| = 0. First we’ll define an auxiliary algorithm ShApprox which returns slightly shifted approximation values. Then using this algorithm we’ll define the uniform generation algorithm UniGen. As always we assume that in R the length of all witnesses is equal to the length of the input x.

6.3.3 Algorithm ShApprox_{R′} (x, α)

1. Let n = |x|, i = |α|

2. Return \{ \begin{matrix}\frac{4n^{2}+4n−4i}{4n^{2}+1} · Approx_{R′} (x, α) if & i < n \\ Round(Approx_{R′} (x, α)) if & i = n \end{matrix}

In all the calls made to Approx by this algorithm we fix ϵ and δ to be ϵ = \frac{1}{4n^{2}} and δ = \frac{1}{3n^{2}} . Notice that by the above definition we have that for i = n ShApprox returns 1 if α ∈ R(x) and 0 otherwise. The purpose of this shifted approximation (as we’ll show later in the analysis) is to ensure that with high probability we have:

ShApprox_{R′} (x, α ∘ 0) + ShApprox_{R′} (x, α ∘ 1) ≤ ShApprox_{R′} (x, α)

6.3.4 Algorithm UniGen_{R′}(x, α)

1. Let n = |x|, i = |α|

2. If i = n return λ (where λ is the empty string)

3. For σ = 0, 1 set

p̃_{σ} =^{def} ShApprox_{R′} (x, α ∘ σ) ; ShApprox_{R′} (x, α)

In this calculation, reuse value returned by calculation of ShApprox_{R′} (x, α) in the previous level of recursion (unless this is the first level of recursion, in which case we apply ShApprox_{R′} (x, λ) once and use the same value in calculating p_{0} and p_{1}).

4. If p̃_{0} + p̃_{1} > 1 return(⊥)

5. With probability p̃_{0} return (0 ∘ UniGen_{R′}(x, α ∘ 0)) With probability p̃_{1} return (1 ∘ UniGen_{R′}(x, α ∘ 1)) Otherwise return(⊥)

We invoke the algorithm by UniGen_{R′}(x, λ) , i.e. UniGen_{R}(x) = UniGen_{R′}(x, λ).

6.3.5 Analysis

Let’s begin with a definition

Definition 6.1 We say that a call to Approx_{R′} (x, α) is successful if

Approx_{R′} (x, α) ∈ (1 ± ϵ) |R′(x, α)|

First of all notice that

Pr [all calls to Approx are successful during the running of UniGen] ≥ 1 − \frac{1}{n}

since Pr [Approx unsuccessful] < δ = \frac{1}{3n^{2}} and at each stage of the recursion we invoke Approx (via ShApprox) at most 3 times. Now we’ll begin establishing the main results.

Claim 6.3.1 For all 0 ≤ i < n if the call to Approx_{R′} (x, α) in ShApprox_{R′} (x, α) is successful then

(1 + \frac{n − i − 1}{n^{2}}) · |R′(x, α)| ≤ ShApprox_{R′} (x, α) ≤ (1 + \frac{n − i}{n^{2}}) · |R′(x, α)|

Proof: For 0 ≤ i < n we have

ShApprox_{R′} (x, α) = \frac{4n^{2} + 4n − 4i}{4n^{2} + 1} · Approx_{R′} (x, α)

so if Approx_{R′} (x, α) ∈ (1 ± ϵ) |R′(x, α)| (for ϵ = \frac{1}{4n^{2}} ) then

\frac{4n^{2} + 4n − 4i}{4n^{2} + 1} · (1 − \frac{1}{4n^{2}} ) · |R′(x, α)| ≤ ShApprox_{R′} (x, α) ≤ \frac{4n^{2} + 4n − 4i}{4n^{2} + 1} · (1 + \frac{1}{4n^{2}} ) · |R′(x, α)|

Now

\frac{4n^{2} + 4n − 4i}{4n^{2} + 1} · (1 + \frac{1}{4n^{2}} ) = \frac{(4n^{2} + 4n − 4i) · (4n^{2} + 1)}{(4n^{2} + 1) · 4n^{2}} = 1 + \frac{n − i}{n^{2}}

and thus the upper bound follows. For the lower bound we begin by noticing that

\frac{4n^{2} + 4n − 4i}{4n^{2} + 1} · (1 − \frac{1}{4n^{2}} ) = \frac{(n^{2} + n − i) · (4n^{2} − 1)}{(4n^{2} + 1) · n^{2}}

Now since (n^{2} + n − i)(4n^{2} − 1) > (n^{2} + n − i − 1)(4n^{2} + 1), we get that

\frac{(n^{2} + n − i)(4n^{2} − 1)}{(4n^{2} + 1)n^{2}} > \frac{n^{2} + n − i − 1}{n^{2}} = 1 + \frac{n − i − 1}{n^{2}}

and the lower bound follows.

Corollary 6.3.2 If all calls to Approx during the running of UniGen are successful then in any stage of the recursion p̃_{0} + p̃_{1} ≤ 1 Proof: Assume all calls to Approx are successful. Let i =^{def} |α|. Now if 0 ≤ i ≤ n − 2 then by the lemma above

ShApprox_{R′} (x, α ∘ 0) + ShApprox_{R′} (x, α ∘ 1)
≤ (1 + \frac{n − i − 1}{n^{2}}) · |R′ (x, α ∘ 0) | + (1 + \frac{n − i − 1}{n^{2}}) · |R′ (x, α ∘ 1) |
= (1 + \frac{n − i − 1}{n^{2}}) · |R′(x, α)|
≤ ShApprox_{R′} (x, α)

On the other hand, for i = n − 1 we have for every σ ∈ {0, 1}

|R′(x, α ∘ σ)| = Round(Approx_{R′}(x, α ∘ σ)) = ShApprox_{R′}(x, α ∘ σ)

where the first equality is due to the fact that |R′(x, α ∘ σ)| ≤ 1 and so rounding the value of the good approximation must yield exactly the correct value, which is preserved by ShApprox. Again, by Claim 6.3.1: |R′(x, α)| ≤ ShApprox_{R′} (x, α), and so it follows that

ShApprox_{R′} (x, α ∘ 0) + ShApprox_{R′} (x, α ∘ 1) = |R′(x, α)| ≤ ShApprox_{R′} (x, α)

We conclude that in both cases

p̃_{0} + p̃_{1} = ShApprox_{R′} (x, α ∘ 0) ; ShApprox_{R′} (x, α) + ShApprox_{R′} (x, α ∘ 1) ; ShApprox_{R′} (x, α) ≤ 1

Lemma 6.3.3 Assume that all calls to Approx are successful then for all x, α, β

Pr [UniGen_{R′} (x, α) = β] = \{ \begin{matrix}\frac{1}{ShApprox_{R′}(x,α)} & if α ∘ β ∈ R(x) \\ 0 & otherwise \end{matrix}

furthermore if α ∘ β ∉ R(x) then Pr[UniGen_{R′} (x, α) = β] = 0 regardless of the success of the Approx calls.

Proof: First of all, we show that if α ∘ β ∉ R(x) then

Pr [UniGen_{R′} (x, α) = β] = 0

Observe that if α ∘ β ∉ R(x) then R′(x, α ∘ β) = 0. Now let β = γ ∘ σ, where |γ| = |β| − 1 and look at the last recursive step of UniGen_{R′}(x, α) (i.e. the call to UniGen_{R′}(x, α ∘ γ)). If we have not arrived to this stage then the output was ⊥, which is certainly not β. In case we did arrive at this step, the probability to output σ and thus finish outputting β as is

p̃_{σ} = ShApprox_{R′} (x, α ∘ γ ∘ σ) ; ShApprox_{R′} (x, α ∘ γ)

Now

ShApprox_{R′} (x, α ∘ γ ∘ σ) = Approx_{R′} (x, α ∘ γ ∘ σ) = 0

because by our definition of Approx we have that Approx_{R′} (x, α) = 0 if R′(x, α) = 0. Thus we get p̃_{σ} = 0 and so σ will not be output and the claim follows. Also notice that we have not used in any part of the above proof the assumption that all calls to Approx were successful.

We now turn to the more interesting case; that is, when α ∘ β ∈ R(x). We will be proven the claim for this case by downward induction on |α|. The claim is trivially true for |α| = n as in that case β = λ and

Pr [UniGen_{R′} (x, α) = λ] = 1 = \frac{1}{ShApprox_{R′} (x, α)}

Now we’ll assume that the claim holds for |α| > k and prove it for |α| = k. Let β = σ ∘ β′ then

Pr [UniGen_{R′} (x, α) = β] = p̃_{σ} · Pr [UniGen_{R′} (x, α ∘ σ) = β′]

Now by the induction hypothesis

Pr [UniGen_{R′} (x, α ∘ σ) = β′] = \frac{1}{ShApprox_{R′} (x, α ∘ σ)}

So we get that

Pr [UniGen_{R′} (x, α) = β] = p̃_{σ} · \frac{1}{ShApprox_{R′} (x, α ∘ σ)}
= \frac{ShApprox_{R′} (x, α ∘ σ)}{ShApprox_{R′_{α}}(x)} · \frac{1}{ShApprox_{R′} (x, α ∘ σ)}
= \frac{1}{ShApprox_{R′} (x, α)}

Corollary 6.3.4 Assuming all calls to Approx are successful we have for all β ∈ R(x)

Pr [UniGen_{R} (x) = β] = \frac{1}{ShApprox_{R′} (x, λ)}

Proof: Take α = λ in Lemma 6.3.3.

Corollary 6.3.5 If all calls to Approx are successful then for any β ∈ R(x) we have

Pr [UniGen_{R}(x) = β | UniGen_{R}(x) ≠ ⊥] = \frac{1}{|R(x)|}

Proof: By Corollary 6.3.4 we have

Pr [UniGen_{R} (x) = β] = \frac{1}{ShApprox_{R′} (x, λ)}

and this quantity is independent of β so all valid witnesses will be generated with the same probability.

Now we’ll try to bound the probability that UniGen_{R′} (x, α) = ⊥.

Lemma 6.3.6 Under the assumption that all calls to Approx are successful we have that

Pr [UniGen_{R} (x) = ⊥] ≤ \frac{2}{n}

Proof: By Claim 6.3.1 we know that

(1 + \frac{n − i − 1}{n^{2}}) · |R′(x, α)| ≤ ShApprox_{R′} (x, α) ≤ (1 + \frac{n − i}{n^{2}}) · |R′(x, α)|

Thus we have that in each recursive step of the algorithm the following inequality holds:

p̃_{0} + p̃_{1} = ShApprox_{R′} (x, α ∘ 0) ; ShApprox_{R′} (x, α) + ShApprox_{R′} (x, α ∘ 1) ; ShApprox_{R′} (x, α)

≥ \frac{(1 + \frac{n−i−2}{n^{2}}) · (|R′(x, α ∘ 0)| + |R′(x, α ∘ 1)|)}{(1 + \frac{n−i}{n^{2}}) · |R′(x, α)|}

= \frac{n^{2} + n − i − 2}{n^{2} + n − i} ≥ 1 − \frac{2}{n^{2}}

Since there are n recursion levels and by the above the probability of outputting ⊥ at Stage 5 of the UniGen algorithm is smaller then \frac{2}{n^{2}} at each recursion level, we conclude that

Pr [UniGen_{R} (x) = ⊥] ≤ n · \frac{2}{n^{2}} = \frac{2}{n}

Corollary 6.3.7

Pr [UniGen_{R} (x) = ⊥] ≤ \frac{3}{n}

Proof: Let AS denote the event All calls to Approx are Successful. Using Lemma 6.3.6 and the fact that Pr [AS] ≥ 1 − \frac{1}{n} we have that

Pr [UniGen_{R} (x) = ⊥] = Pr [(UniGen_{R} (x) = ⊥) ∧ AS] + Pr [(UniGen_{R} (x) = ⊥) ∧ \overline{AS}]
≤ Pr [UniGen_{R} (x) = ⊥ | AS] + Pr [\overline{AS}]
≤ \frac{2}{n} + \frac{1}{n}

Combining Corollaries 6.3.5 and 6.3.7, we conclude that the procedure UniGen_{R} behaves as desired.

6.4 An alternate algorithm for uniform generation

6.4.1 Intuition

Now we shall sketch an alternative algorithm for uniform generation. In addition to using approximate counting this algorithm will use hash functions in a direct way. The hash functions will be of high independence (and thus have stronger ”random properties”) than pairwise independence used before.

6.4.2 Definitions

Let H(n, m, t) denote a collection of t-wise independent hash functions of n bits to m bits. This means that for any y_{1}, …, y_{t} ∈ {0, 1}^{m} and any distinct x_{1}, …, x_{t} ∈ {0, 1}^{n} we have for a hash function h uniformly chosen from H(n, m, t) that

Pr [h(x_{1}) = y_{1} ∧ … ∧ h(x_{t}) = y_{t}] = 2^{−mt}

Let h ∈ H(n, m, t) then for α ∈ {0, 1}^{m} we let h^{−1}(α) = {y ∈ {0, 1}^{n} | h(y) = α}. Recall that S = R(x) and n = |x|.

6.4.3 Algorithm UniGen_{R}(x)

1. If |S| ≤ 2n^{5} compute a listing y_{1}, …, y_{s} of the members of the set |S|, select j at random from 1, …, s, output y_{j} and halt.

2. Otherwise, let N ≈ |S| (we obtain a good approximation of |S| using approximate counting), and let m = log_{2} N − 5 log n

3. Select h uniformly in H(n, m, n)

4. If ∃α ∈ {0, 1}^{m} s.t. |S ∩ h^{−1}(α)| ≥ 2n^{5} then output ⊥

5. Select α uniformly in {0, 1}^{m}

6. Compute S ∩ h^{−1}(α) and for every y ∈ S ∩ h^{−1}(α) output it with probability \frac{1}{2n^{5}} , otherwise output ⊥

The implementation of Steps 1, 4, 6 involves the use of an NP-oracle. In each case we build an appropriate NP-relations and use the oracle to check membership in those relations. For example in Step 4 we would use the following NP-relation and corresponding NP-set:

S′′ = {(x, h) : ∃α, y_{1}, …, y_{k} such that ((x, h), (α, y_{1}, …, y_{k})) ∈ R′′}

R′′ = \{((x, h), (α, y_{1}, …, y_{k})) : \begin{matrix}k = 2|x|^{5} and y_{1}, …, y_{k} are distinct \\ and h ∈ H(|x|, m, |x|) \\ and R(x, y_{i}) = 1 (for every i ∈ [k]) \\ and h(y_{i}) = α (for every i ∈ [k]) \end{matrix} \}

6.4.4 Analysis

To show that the algorithm indeed is a uniform generator we note that if |S| ≤ 2n^{5} then the algorithm halts in Step 1 outputting a uniformly chosen element of S. If |S| > 2n^{5} then considering any possible (fixed) choice of h in Step 3 and assuming a non-⊥ result in Step 4 we have that the randomization is only over the choice of α in step 5 and the choices in step 6. And so we have that the probability that any fixed y ∈ S will be output is:

Pr[y is output] = Pr [h(y) = α : α ⟵^{R} \{0, 1\}^{m}] · \frac{1}{2n^{2}}
= 2^{−m} · \frac{1}{2n^{2}}

and this quantity is independent of y.

Another important claim is that the probability of a non-⊥ output is a positive constant. First notice that if h is drawn at random from H(n, m, n) then the expected size of |S ∩ h^{−1}(α)| is \frac{|S|}{2^{m}} = n^{5}, and from this fact it’s possible to show via standard t-wise independence techniques that the probability that |S ∩ h^{−1}(α)| < 2n^{5} for all α is a positive constant. We conclude that the probability of not halting in Step 4 of the algorithm is a positive constant.

Conditioned on not halting in Step 4, the probability that any fixed y ∈ S is output in Step 6 is 2^{−m} · \frac{1}{2n^{5}} , so it follows that the probability of outputting ⊥ in Step 6 equals

1 − |S| · \frac{2^{−m}}{2n^{5}} = 1 − |S| · \frac{2^{5 log_{2} n−log_{2}N}}{2n^{5}} = 1 − \frac{|S|}{2N}

Now since N is a good approximation of |S|, we have \frac{|S|}{2N} > \frac{1}{3}. Thus the probability of outputting an element of S in Step 6 is at least 1/3. Finally, as we have shown that the probability of halting in Step 4 is a positive constant, and the probability of a non-⊥ result in Step 6 is a positive constant, we conclude that the probability of the algorithm to output an element of S is a positive constant.

Bibliographic Notes

The relation between approximate counting and uniform generation was first explicitly studied by Jerrum, Valiant and Vazirani [28]. The direct presentation of a uniform generation procedure is due to Bellare, Goldreich and Petrank [12].

Small Bias Sample Spaces (Part 1)

Notes taken by Yehuda Lindell and Alon Rosen

Summary: In this lecture we introduce the notion of ϵ-bias sample spaces. Informally, a random variable over {0, 1}^{n} is ϵ-bias if for every subset of bits, the difference between the probability that the parity of the bits is zero and the probability it is one is at most ϵ. We show a connection between ϵ-bias and the statistical distance to the uniform distribution. Next, we present an application of ϵ-bias sample spaces for generating “approximately” k-wise independent distributions (i.e., distributions over {0, 1}^{n} in which every k bits look almost uniformly distributed over {0, 1}^{k}).

7.1 Introduction

In previous lectures, the notion of k-wise independent distributions was considered. Such distributions are a relaxation of the uniform distribution in that any subset of size k is uniformly distributed (and yet nothing can be said of subsets of size larger than k). The motivation for considering such relaxations is the desire to obtain (efficiently constructible) “small” sample spaces (where by small, we mean of size polynomial in n where, for example, the space is distributed over {0, 1}^{n}).

In this lecture, we introduce a different relaxation of uniform distributions, called ϵ-bias. Loosely speaking, we say that a random variable (over {0, 1}^{n}) is ϵ-bias if for every subset of bits, the difference between the probability that the parity of the bits is 0 and 1 is at most ϵ. This is a natural relaxation of the uniform distribution, for which the probability that any subset of bits has parity 0 equals exactly 1/2 (likewise for parity 1). In other words, the uniform distribution has zero bias. As we shall see in this lecture, the converse is also true and thus a random variable is uniformly distributed if and only if it has zero bias. Therefore, the notion of bias can be used to fully characterize the uniform distribution.

As we have mentioned, a zero-bias sample space is uniform. A natural question to ask is how to characterize the distribution of an ϵ-bias sample space, for ϵ that is strictly greater than zero. Indeed, the bulk of this lecture focuses on understanding the “distance” of an ϵ-bias sample space from the uniform distribution (where the notion of distance used is the classical one of statistical difference).

We also present an application of ϵ-bias sample spaces (another more “direct” application is presented in the next lecture). Loosely speaking, we show that “almost” k-wise independent spaces can be derived from ϵ-bias sample spaces, where almost k-wise independent spaces are such that when considering any subset of size k, the resulting distribution is “approximately” uniform (rather than being truly uniform as in the case of k-wise independence). We note that such spaces can be made much smaller than is possible for (fully) k-wise independent spaces.

7.2 Definitions

7.2.1 Statistical Difference

We begin by defining a natural measure of the distance between distributions. This measure, called the statistical difference (or variation distance) is the traditional way of measuring the distance between distributions. We note that, as discussed in Lecture 1, we abuse notation and view a random variable as a distribution (or alternatively, a sample space), rather than as a function mapping n-bit strings to [0, 1].

Definition 7.1 (statistical difference): Let X, Y ∈ {0, 1}^{n} be random variables. The statistical difference between X and Y is defined by the function:

Δ(X, Y) =^{def} \frac{1}{2} · ∑_{α∈\{0,1\}^{n}} |Pr [X = α] − Pr [Y = α]|

Observe that removing the absolute value from the definition would render it meaningless (as it would always sum to zero). A seemingly strange part of this definition is the constant factor of 1/2. Firstly, this factor provides a normalization of the statistical difference in that for every X and Y, it holds that 0 ≤ Δ(X, Y) ≤ 1. Secondly, including this factor results in the above definition being equivalent to the following one:

Definition 7.2 (statistical difference – alternative definition): Let X, Y ∈ {0, 1}^{n} be random variables. The statistical difference between X and Y is defined by,

Δ(X, Y) =^{def} max_{S⊆{0,1}^{n}} {Pr [X ∈ S] − Pr [Y ∈ S]}

Proposition 7.2.1 Definitions 7.1 and 7.2 are equivalent. That is, for all random variables X and Y, Δ_{1}(X, Y) = Δ_{2}(X, Y), where Δ_{i} denotes the statistical difference by Definition i.

Proof: We begin by observing that the set S for which the value (Pr[X ∈ S] − Pr[Y ∈ S]) is maximum has the following property:

1. For every α ∈ S, it holds that Pr[X = α] ≥ Pr[Y = α], and

2. For every α ∉ S, it holds that Pr[X = α] ≤ Pr[Y = α].

Intuitively, the above property holds because otherwise an α contradicting it could be moved from S to \overline{S} (or the reverse) and a “larger than maximum” value would be obtained. That is, if there exists an α ∈ S for which Pr[X = α] < Pr[Y = α], then it is easy to verify that

Pr[X ∈ S \ {α}] − Pr[Y ∈ S \ {α}] > Pr[X ∈ S] − Pr[Y ∈ S]

in contradiction to the maximality of the value (Pr[X ∈ S]−Pr[Y ∈ S]) induced by S. An analogous argument holds for item (2) of the above property.

We now claim that for every set S for which the above property holds, we have that

Pr[X ∈ S] − Pr[Y ∈ S] = \frac{1}{2} · ∑_{α∈\{0,1\}^{n}} |Pr [X = α] − Pr [Y = α]| = Δ_{1}(X, Y)  (7.1)

Eq. (7.1) follows from the following simple calculation:

Pr[X ∈ S] − Pr[Y ∈ S] = \frac{1}{2} · (Pr[X ∈ S] − Pr[Y ∈ S] − Pr[Y ∈ S] + Pr[X ∈ S])
= \frac{1}{2} · (Pr[X ∈ S] − Pr[Y ∈ S] + Pr[Y ∉ S] − Pr[X ∉ S])
= \frac{1}{2} · (∑_{α∈S} (Pr[X = α] − Pr[Y = α]) + ∑_{α∉S} (Pr[Y = α] − Pr[X = α]))
= \frac{1}{2} · ∑_{α∈\{0,1\}^{n}} |Pr[X = α] − Pr[Y = α]|

where the last equality is due to the fact that the above property holds with respect to S. This concludes the proof of the proposition because we have that for S for which Pr[X ∈ S] − Pr[Y ∈ S] is maximum, Eq. (7.1) holds. This implies that Δ_{1}(X, Y) = Δ_{2}(X, Y).

Notation: We denote by U_{n} the uniform distribution over n-bit strings. In this lecture, we are interested in distributions X such that Δ(X, U_{n}) is small (clearly, Δ(X, U_{n}) = 0 if and only if X = U_{n}).

An advanced comment: The sets S in Definition 7.2 can be viewed as (not necessarily efficiently computable) tests for the statistical difference between X and Y. That is, for each set S, associate a distinguisher D_{S} who upon input α outputs 1 if and only if α ∈ S. Then, it follows that

Pr[D_{S}(X) = 1] − Pr[D_{S}(Y) = 1] = Pr[X ∈ S] − Pr[Y ∈ S]

However, notice that such a distinguisher may not be efficient (i.e., polynomial-time). This is because there may not necessarily be efficient procedures for recognizing whether or not α ∈ S.

The notion of computational indistinguishability, in a specific incarnation that states that no polynomial-size circuit can distinguish between two distribution ensembles {X_{n}}_{n∈N} and {Y_{n}}_{n∈N} (where for every n, X_{n} and Y_{n} range over strings of length polynomial in n), is a relaxed notion of statistical closeness. (We say that {X_{n}} and {Y_{n}} are statistically close if Δ(X_{n}, Y_{n}) is a negligible function in n.) To see this, first notice that for every circuit C_{n} and every n, it holds that C_{n} can distinguish X_{n} from Y_{n} with probability at most Δ(X_{n}, Y_{n}). This is because we can define S_{n} = {α | C_{n}(α) = 1} and thus it follows that for every n,

|Pr[C_{n}(X_{n}) = 1] − Pr[C_{n}(Y_{n}) = 1]| = |Pr[X_{n} ∈ S_{n}] − Pr[Y_{n} ∈ S_{n}]| ≤ Δ(X_{n}, Y_{n})

From this, we can conclude that if {X_{n}} and {Y_{n}} are statistically close, then they are also computationally indistinguishable. (Notice that the notion of computational indistinguishability, as described above, refers to polynomial-size circuits. Nevertheless, this implies computational indistinguishability by probabilistic polynomial-time algorithms, since one may hardwire into a circuit the algorithm’s “best” coin-tosses. Thus, we are guaranteed that the distinguishing gap of the circuit is at least as good as that of the algorithm.)

7.2.2 ϵ-Bias Sample Spaces

We introduce the notion of ϵ-bias sample spaces or random variables. (We will move freely between the notions of sample spaces and random variables, with the intention being that the random variable is defined according to the distribution of the corresponding sample space.) Loosely speaking, the bias of a random variable, with respect to a subset of indices I, equals the difference between the probability that the parity of the bits indexed by I equals zero and the probability that it equals one. An ϵ-bias random variable is one for which the bias (over all subsets of indices) is at most ϵ. In some sense this has the flavor of uniformity in that the parity of any subset of bits of a uniformly distributed string is unbiased to 0 or 1. In fact, the following two conditions are equivalent:

1. The bits of the random variable X = X_{1}, …, X_{n} are independent and uniformly distributed.

2. For every subset of bits of X, it is equally likely that the parity of the subset is either zero or one.

(The fact that the above two properties are equivalent is proven in the next section, see Corollary 7.6.) Therefore, ϵ-bias can be viewed as some measure of distance from the uniform distribution. We stress that although there is some connection between ϵ-bias and statistical distance from uniform, the notions are not at all equivalent.

Definition 7.3 (bias): Let X = X_{1}, …, X_{n} ∈ {0, 1}^{n} be a random variable, and let I ⊆ [n] (where [n] denotes the set {1, …, n}). The bias of the random variable X, relative to the subset I, is defined by,

bias_{I}(X) =^{def} Pr [⨁_{i∈I} X_{i} = 0] − Pr [⨁_{i∈I} X_{i} = 1]

where by convention for any (x_{1}, …, x_{n}), it holds that ⨁_{i∈∅} x_{i} ≡ 0.

An equivalent definition of bias is given by:

bias_{I}(X) =^{def} 2 · Pr [⨁_{i∈I} X_{i} = 0] − 1  (7.2)

The equivalence is derived from the trivial fact that Pr [⨁_{i∈I} X_{i} = 1] = 1−Pr [⨁_{i∈I} X_{i} = 0]. Notice that −1 ≤ bias_{I}(X) ≤ 1 always holds.

The above definition of bias refers to a specific subset of bits of X. We now present the definition of max-bias, which extends this to all subsets of X.

Definition 7.4 (max-bias and ϵ-bias sample spaces): Let X ∈ {0, 1}^{n} be a random variable. The max-bias of X is defined by,

maxbias(X) =^{def} max_{{I⊆[n] \atop I≠∅}} \{|bias_{I}(X)|\}

We say that X is an ϵ-bias sample space if maxbias(X) ≤ ϵ.

Clearly, the max-bias of a uniform space equals 0 (i.e., maxbias(U_{n}) = 0). As we shall see, the converse is also true. However, as we have mentioned above, it is not true that an ϵ-bias sample space X satisfies Δ(X, U_{n}) ≤ ϵ. Despite this, there is a connection between these notions (as stated in Theorem 7.5 below).

7.3 Statistical Difference versus Maxbias

The following theorem demonstrates the connection between ϵ-bias and statistical difference. In particular, it implies that if X is ϵ-bias, then X is at most \frac{ϵ·\sqrt{2^{n}}}{2} -far from uniform.

Theorem 7.5 Let X ∈ {0, 1}^{n} be a random variable. Then,

maxbias(X) ≤ 2 · Δ(X, U_{n}) ≤ \sqrt{2^{n}} · maxbias(X)

We remark that an immediate corollary of Theorem 7.5 is that a random variable X is uniformly distributed (i.e., Δ(X, U_{n}) = 0) if and only if maxbias(X) = 0. Thus, the notion of bias offers an alternative characterization of the uniform distribution.

Corollary 7.6 A random variable X is uniform if and only if maxbias(X) = 0.

7.3.1 Proof of Theorem 7.5 (Main Result)

We first show that

maxbias(X) ≤ 2 · Δ(X, U_{n})  (7.3)

Let ϵ = maxbias(X). Then, there exists a non-empty subset I ⊆ [n] such that bias_{I}(X) = ϵ. Now, define S_{I} = {α | ⨁_{i∈I} α_{i} = 0}. Intuitively, this subset of strings (for which X is biased towards zero) constitutes a “test” for the statistical difference by Definition 7.2. Specifically,

Pr [X ∈ S_{I}] = Pr [⨁_{i∈I} α_{i} = 0] = \frac{1}{2} + \frac{ϵ}{2}

where the second equality is from the alternative definition of ϵ-bias as shown in Eq. (7.2). On the other hand, clearly Pr[U_{n} ∈ S_{I}] = \frac{1}{2}. Therefore, we have found a set S for which

Pr[X ∈ S] − Pr[U_{n} ∈ S] = \frac{ϵ}{2}

Eq. (7.3) now follows. (We note that it is not necessarily true that maxbias(X) = 2 · Δ(X, U_{n}) because there may be other sets S, for which the difference Pr[X ∈ S] − Pr[U_{n} ∈ S] is even greater than ϵ/2.)

We now show that

2 · Δ(X, U_{n}) ≤ \sqrt{2^{n}} · maxbias(X)  (7.4)

In general, probability distributions over {0, 1}^{n} are viewed as functions f mapping n-bit strings to [0, 1] (and satisfy ∑_{α} f(α) = 1). An alternative view that is instructive for this proof, is to represent such functions as vectors in IR^{N}, where N = 2^{n} (and IR denotes the set of real numbers). According to this view, the length-N vector representation of a function f is such that the coordinate indexed by x ∈ {0, 1}^{n} contains the value f(x). Specifically, the set f : {0, 1}^{n} → IR of real functions on {0, 1}^{n} forms an N-dimensional real vector space where vectors in this space are of the form:

v_{f} =^{def} ⟨f(x)⟩_{x∈{0,1}^{n}}

We call such a vector a probability vector if all its entries are non-negative and they sum-up to 1. From now on, we associate functions f with vectors v_{f} of the above form. The inner-product of two vectors (functions) f and g is defined by:

⟨f, g⟩ = ∑_{x∈{0,1}^{n}} f(x) · g(x)

The above way of viewing probability distributions turns out to be useful when analyzing sample spaces. As we will see, there is a correspondence between different bases of such vector spaces and the notions of max-bias and statistical difference. Then, using properties of linear algebra, we derive Eq. (7.4). We consider two different N-dimensional bases for the above vector space:

1. Kronecker Basis: The Kronecker Basis, denoted K = {k_{α}}_{α∈{0,1}^{n}} is defined as follows:

k_{α}(x) =^{def} \{ \begin{matrix}1 & if x = α \\ 0 & if x ≠ α \end{matrix}

This is the standard basis for vector spaces (in vector notation this is the basis in which the i^{th} element of the basis is the vector with a single 1 in the i^{th} place and zeros elsewhere).

2. Fourier Basis: The Fourier Basis, denoted F = {f_{I}}_{I⊆[n]} is defined as follows:

f_{I}(x) =^{def} \frac{1}{\sqrt{N}} · (−1)^{∑_{i∈I} x_{i}}

In the proof we switch freely between viewing a random variable X as a distribution on {0, 1}^{n} (that is, a sample space) and viewing it as a probability function π : {0, 1}^{n} → [0, 1] (representing the probability that a specific string in {0, 1}^{n} is chosen in the sample space).

We now show that both the Kronecker and Fourier bases are orthonormal (this technicality is needed later in the proof).

Fact 7.3.1 The Kronecker basis is orthonormal. That is, for every α ≠ β ∈ {0, 1}^{n}:

⟨k_{α}, k_{α}⟩ = 1 and ⟨k_{α}, k_{β}⟩ = 0

Proof: Note that for every α, β ∈ {0, 1}^{n}:

⟨k_{α}, k_{β}⟩ = ∑_{x∈{0,1}^{n}} k_{α}(x) · k_{β}(x)
= k_{α}(β)

where the last equality is due to the fact that k_{β}(β) = 1, whereas for every x ≠ β, k_{β}(x) = 0. This completes the proof, because k_{α}(α) = 1 and k_{α}(β) = 0 for every β ≠ α.

Fact 7.3.2 The Fourier basis is orthonormal. That is, for every I ≠ J ⊆ [n]:

⟨f_{I}, f_{I}⟩ = 1 and ⟨f_{I}, f_{J}⟩ = 0

Proof: Let I ⊆ [n]. Then:

⟨f_{I}, f_{I}⟩ = ∑_{x∈{0,1}^{n}} f_{I}(x) · f_{I}(x)
= \frac{1}{N} · ∑_{x∈\{0,1\}^{n}} (−1)^{∑_{i∈I} x_{i}} · (−1)^{∑_{i∈I} x_{j}}
= \frac{1}{N} · ∑_{x∈\{0,1\}^{n}} (−1)^{2 ∑_{i∈I} x_{i}}
= \frac{1}{N} · ∑_{x∈\{0,1\}^{n}} 1
= \frac{1}{N} · N

as required. On the other hand, for I ≠ J ⊆ [n], we have:

⟨f_{I}, f_{J}⟩ = ∑_{x∈{0,1}^{n}} f_{I}(x) · f_{J}(x)
= \frac{1}{N} · ∑_{x∈\{0,1\}^{n}} (−1)^{∑_{i∈I} x_{i}} · (−1)^{∑_{j∈J} x_{j}}
= \frac{1}{N} · ∑_{x∈\{0,1\}^{n}} (−1)^{∑_{i∈I∇J} x_{i}}  (7.5)

where I∇J denotes the symmetric difference between sets I and J (that is, I∇J =^{def} (I \J)∪(J \I)). Notice that I∇J ≠ ∅ (since I ≠ J), and that the sum in Eq. (7.5) ranges over all x ∈ {0, 1}^{n}. For, b ∈ {0, 1}, consider the sets S_{I∇J}^{b} = {x | ⊕_{i∈I∇J} x_{i} = b}. Using this notation, we have:

\frac{1}{N} · ∑_{x∈\{0,1\}^{n}} (−1)^{∑_{i∈I∇J} x_{i}} = \frac{1}{N} · ( ∑_{x∈S_{I∇J}^{0}} (−1)^{∑_{i∈I∇J} x_{i}} + ∑_{x∈S_{I∇J}^{1}} (−1)^{∑_{i∈I∇J} x_{i}} )

= \frac{1}{N} · (|S_{I∇J}^{0} | · 1 − |S_{I∇J}^{1} | · 1)

The key observation is that if I∇J ≠ ∅, then the size of S_{I∇J}^{0} equals the size of S_{I∇J}^{1}. This can be seen by showing a bijection between the two sets. A simple such bijection is a function that flips the i^{th} bit of x ∈ S_{I∇J}^{0}, where i ∈ I∇J. That is, for an arbitrary i ∈ I∇J define,

F_{i}(x_{1} ⋯ x_{n}) =^{def} x_{1} ⋯ x_{i−1} · \overline{x}_{i} · x_{i+1} ⋯ x_{n}

Clearly, for x ∈ S_{I∇J}^{0}, we have F_{i}(x) ∈ S_{I∇J}^{1} (the fact that F_{i} is a bijection is trivial). We conclude that,

⟨f_{I}, f_{J}⟩ = \frac{1}{N} · (|S_{I∇J}^{0} | − |S_{I∇J}^{1} |)
= \frac{1}{N} · 0

as required. In the following fact, we show that the bias of a probability function π, relative to the subset I, can be stated in terms of the inner-product of π with f_{I}.

Fact 7.3.3 For every I ⊆ [n] and for every probability function π : {0, 1}^{n} → [0, 1],

bias_{I}(π) = \sqrt{N} · ⟨π, f_{I}⟩

Proof: The key observation for proving the fact is that for b ∈ {0, 1}, the probability that the parity of x equals b can be computed by summing π(x) for all strings x for which the parity of x is indeed b. Denote the set of strings x, for which the parity of the bits of x indexed by I equals b, by S_{I}^{b}. That is, S_{I}^{b} =^{def} {x | ⨁_{i∈I} x_{i} = b}. Then, for b ∈ {0, 1} and random variable X = (X_{1}, …, X_{n}) with distribution π:

Pr [⨁_{i∈I} X_{i} = b] = ∑_{x∈S_{I}^{b}} π(x)

We thus have,

bias_{I}(π) = ( ∑_{x∈S_{I}^{0}} π(x)) − ( ∑_{x∈S_{I}^{1}} π(x))

= ∑_{x∈{0,1}^{n}} π(x) · (−1)^{∑_{i∈I} x_{i}}

where the second equality is due to the fact that for x ∈ S_{I}^{b}, it holds that (−1)^{∑_{i∈I} x_{i}} = (−1)^{b}, and thus π(x)(−1)^{∑_{i∈I} x_{i}} = π(x)(−1)^{b}. Now, by the definition of the Fourier basis (and the innerproduct), it follows that,

bias_{I}(π) = ∑_{x∈\{0,1\}^{n}} π(x) · \sqrt{N} · f_{I}(x)
= \sqrt{N} · ⟨π, f_{I}⟩

as required.

Given Fact 7.3.3, we are ready to show a connection between maxbias(π) and the L_{∞}^{F} norm (i.e., the L_{∞} norm with respect to the Fourier basis F). Following this, we show a connection between Δ(π, U_{n}) and the L_{1}^{K} norm (i.e., the L_{1} norm with respect to the Kronecker basis K). We then conclude by presenting a fact from linear algebra connecting between the L_{1} and L_{∞} norms. We first recall the definition of L_{1}^{B} and L_{∞}^{B}, for a basis B:

∥ v ∥_{1}^{B} =^{def} ∑_{b∈B} |⟨v, b⟩|
∥ v ∥_{∞}^{B} =^{def} max_{b∈B}{|⟨v, b⟩|}

In general for any integer p (e.g., p = 2) the L_{p}^{B} norm is defined as:

∥ v ∥_{p}^{B} =^{def} (∑_{b∈B} |⟨v, b⟩|^{p})^{1/p}

Now, define the constant function μ : \{0, 1\}^{n} → IR such that for every x ∈ \{0, 1\}^{n}, μ(x) = \frac{1}{N}. The function μ is the probability function associated with the uniform distribution U_{n}. The following claim establishes the connection between the maxbias of a probability function π and the L_{∞}^{F} norm of π (or, actually of π − μ).

Claim 7.3.4 For every probability function π : {0, 1}^{n} → IR, maxbias(π) = \sqrt{N}· ∥ π − μ ∥_{∞}^{F}

Proof: By Fact 7.3.3 we have that

maxbias(π) =^{def} max_{{I⊆[n] \atop I≠∅}} \{|bias_{I}(π)|\} = \sqrt{N} · max_{{I⊆[n] \atop I≠∅}} \{|⟨π, f_{I}⟩|\}  (7.6)

We now make the following two observations regarding the Fourier basis:

1. For every non-empty subset I ⊆ [n], ⟨μ, f_{I}⟩ = 0.

This is due to the fact that ⟨μ, f_{I}⟩ = \frac{1}{N\sqrt{N}} ∑_{x}(−1)^{∑_{i∈I} x_{i}}. As we have previously seen (in the proof of Fact 7.3.2), for any non-empty subset I, the above sum equals zero.

2. For every probability vector v, ⟨v, f_{∅}⟩ = \frac{1}{\sqrt{N}} (by the definition of the Fourier basis and the

fact that the sum of the elements of v equals 1). This implies that for every I ⊆ [n],

⟨π − μ, f_{I}⟩ = ⟨π, f_{I}⟩ − ⟨μ, f_{I}⟩ = \{ \begin{matrix}⟨π, f_{I}⟩ & if I ≠ ∅ \\ 0 & if I = ∅ \end{matrix}  (7.7)

Then, combining Eq. (7.6) and( 7.7) we obtain that

maxbias(π) = \sqrt{N} · max_{{I⊆[n] \atop I≠∅}} \{|⟨π − μ, f_{I}⟩|\} = \sqrt{N}· ∥ π − μ ∥_{∞}^{F}

as required. We now turn to establish the connection between the statistical difference of π from U_{n} and the L_{1}^{K} norm of π − μ. Claim 7.3.5 For every probability function π : {0, 1}^{n} → IR:

2 · Δ(π, U_{n}) = ∥ π − μ ∥_{1}^{K}

Proof: Recall that for every vector v, ⟨v, k_{α}⟩ = v(α). Thus:

2 · Δ(π, U_{n}) = ∑_{x∈{0,1}^{n}} |π(x) − μ(x)|
= ∑_{x∈{0,1}^{n}} |⟨π, k_{x}⟩ − ⟨μ, k_{x}⟩|
= ∑_{x∈{0,1}^{n}} |⟨π − μ, k_{x}⟩|
= ∥ π − μ ∥_{1}^{K}

as required. We conclude the proof by combining Claims 7.3.4 and 7.3.5 with the following technical lemma from linear algebra (proven below).

Lemma 7.3.6 For every two orthonormal bases A and B, and for every vector v:

∥ v ∥_{1}^{A}≤ N· ∥ v ∥_{∞}^{B}

Using Lemma 7.3.6 we have:

2 · Δ(π, U_{n}) = ∥ π − μ ∥_{1}^{K}
≤ N· ∥ π − μ ∥_{∞}^{F}
= \sqrt{N} · maxbias(π)

Eq. (7.4) follows. Thus, all that remains to complete the proof of Theorem 7.5 is to prove Lemma 7.3.6.

7.3.2 Proof of Lemma 7.3.6 (Technical Lemma from Linear Algebra)

In order to prove Lemma 7.3.6 we will make use of the following three facts:

Fact 7.3.7 For every basis A:

∥ v ∥_{1}^{A} ≤ \sqrt{N}· ∥ v ∥_{2}^{A}

Proof: We start by recalling the Cauchy-Schwartz inequality. Let {α_{i}}_{i=1}^{N} and {β_{i}}_{i=1}^{N} be two sequences of real numbers. Then:

(∑_{i=1}^{N} α_{i} · β_{i})^{2} ≤ (∑_{i=1}^{N} α_{i}^{2}) · (∑_{i=1}^{N} β_{i}^{2})

Now, let A = {a_{i}}_{i=1}^{N} be a basis. Then, by the definition of the L_{1}^{A} and L_{2}^{A} norms and by the Cauchy-Schwartz inequality:

(∥ v ∥_{1}^{A})^{2} = (∑_{i=1}^{N} |⟨v, a_{i}⟩|)^{2}

≤ (∑_{i=1}^{N} 1^{2}) · (∑_{i=1}^{N} |⟨v, a_{i}⟩|^{2})
= N · (∥ v ∥_{2}^{A})^{2}

as required.

Fact 7.3.8 For every two orthonormal bases A and B:

∥ v ∥_{2}^{A} = ∥ v ∥_{2}^{B}

Proof: Intuitively, the proof follows from the fact that for any orthonormal basis, A, the L_{2}^{A} norm of the vector v is equal to the square root of the inner product of v with itself (regardless of the choice of the orthonormal basis A). This is proved formally as follows. Let A = {a_{i}}_{i=1}^{N} be an orthonormal basis and let v be a vector. Then there exist a sequence of real numbers {α_{i}}_{i=1}^{N} (specifically, α_{i} = ⟨v, a_{i}⟩) so that v can be written as ∑_{i=1}^{N} α_{i} · a_{i}. By the definition of the L_{2}^{A} norm and by the orthonormality of A we have:

⟨v, v⟩ = ⟨∑_{i=1}^{N} α_{i} · a_{i}, ∑_{j=1}^{N} α_{j} · a_{j}⟩

= ∑_{i=1}^{N} ∑_{j=1}^{N} α_{i} · α_{j} · ⟨a_{i}, a_{j}⟩

= ∑_{i=1}^{N} α_{i}^{2}  (7.8)

= ∑_{i=1}^{N}⟨v, a_{i}⟩^{2}
= (∥ v ∥_{2}^{A})^{2}

where Eq. (7.8) follows from the fact that for every i ≠ j, ⟨a_{j}, a_{i}⟩ = 0 and ⟨a_{i}, a_{i}⟩ = 1. It follows that for every two orthonormal bases A and B:

(∥ v ∥_{2}^{A})^{2} = ⟨v, v⟩ = (∥ v ∥_{2}^{B})^{2}

as required.

Fact 7.3.9 For every basis B:

∥ v ∥_{2}^{B} ≤ \sqrt{N}· ∥ v ∥_{∞}^{B}

Proof: Let B = {b_{i}}_{i=1}^{N} be a basis. By definition of the L_{2}^{B} and L_{∞}^{B} norms:

(∥ v ∥_{2}^{B})^{2} = ∑_{i=1}^{N} |⟨v, b_{i}⟩|^{2}
≤ N · max_{i≤i≤N}{|⟨v, b_{i}⟩|^{2}}
= N · ( max_{i≤i≤N}{|⟨v, b⟩|})^{2}
= N · (∥ v ∥_{∞}^{B})^{2}

as required.

By combining Facts 7.3.7, 7.3.8 and 7.3.9 we obtain:

∥ v ∥_{1}^{A} ≤ \sqrt{N}· ∥ v ∥_{2}^{A}
= \sqrt{N}· ∥ v ∥_{2}^{B}
≤ N· ∥ v ∥_{∞}^{B}

which completes the proof of Lemma 7.3.6.

7.4 (ϵ, k)-Approximations of the Uniform Distribution

7.4.1 (ϵ, k)-approximation of the uniform distribution

In this section, we introduce the notion of “approximately” uniform distributions and show how they can be derived from ϵ-bias sample spaces. Intuitively, we say that a distribution X is an ϵ-approximation of U_{n} if Δ(X, U_{n}) ≤ ϵ. We combine this ϵ-approximation with the k-wise independent relaxation that we have seen in previous lectures. That is, we say that X is an (ϵ, k)- approximation of U_{n} if, when projecting X on any subset of size k of it coordinates, the resulting distribution is an ϵ-approximation of U_{k}. Formally,

Definition 7.7 ((ϵ, k)-approximation of the uniform distribution): Let X ∈ {0, 1}^{n} be a random variable, let ϵ > 0 and let k ∈ [n] be an integer. We say that X = (X_{1}, …, X_{n}) is an (ϵ, k)- approximation of the uniform distribution, if for every subset I = {i_{1}, …, i_{k}} ⊆ [n] (of size k) the distribution X_{I} = {X_{i_{1}}, …, X_{i_{k}}} is ϵ-far from uniform (that is, Δ(X_{I}, U_{k}) ≤ ϵ).

Clearly, an (0, k)-approximation of the uniform distribution is a k-wise independent sample space. Furthermore, an (ϵ, n)-approximation of the uniform distribution is what we informally called an ϵ-approximation above. It is easily verified (see Proposition 7.4.1 below) that for every ϵ > 0 and every k > 1, an (ϵ, k)-approximation of the uniform distribution is also (ϵ, k−1)-approximation of the uniform distribution. Essentially, an (ϵ, k)-approximation of uniform is a relaxation of k-wise independence. As we shall see, this further approximation enables us to construct sample spaces that are far smaller than k-wise independent spaces.

Proposition 7.4.1 For every ϵ > 0 and every k > 1, an (ϵ, k)-approximation of the uniform distribution is also an (ϵ, k − 1)-approximation of the uniform distribution.

Proof: Let X ∈ {0, 1}^{n} be a random variable, let ϵ > 0, let k > 1 be an integer and suppose that X is an (ϵ, k)-approximation of the uniform distribution. If X is not an (ϵ, k−1)-approximation of the uniform distribution, then there exists a set I ⊆ [n] of size k−1 (for simplicity, assume that I = {1, …, k −1}) so that the distribution X_{I} = {X_{1}, …, X_{k−1}} is not ϵ-far from U_{k−1} (that is, Δ(X_{I}, U_{k−1}) > ϵ). By the alternative definition of statistical difference (Definition 7.2) this means that there exists a set S ⊆ {0, 1}^{k−1} so that:

Pr [X_{I} ∈ S] − Pr [U_{k−1} ∈ S] > ϵ  (7.9)

For b ∈ {0, 1} consider now the set S_{b} ⊆ {0, 1}^{k} that is obtained by concatenating each element α ∈ S with the bit b (that is, S_{b} = {α ∘ b | α ∈ S}). Notice that Pr [U_{k} ∈ S_{0} ∪ S_{1}] = Pr [U_{k−1} ∈ S]. Similarly, Pr [X_{I∪{k}} ∈ S_{0} ∪ S_{1}] = Pr [X_{I} ∈ S] (where X_{I∪{k}} = {X_{1}, …, X_{k}} is the distribution that is induced by the k-sized subset I ∪ {k}). Thus, for the set I ∪ {k} ⊆ [n] of size k, we have:

Δ(X_{I∪{k}}, U_{k}) = max_{S⊆{0,1}^{k}} {Pr [X_{I∪{k}} ∈ S] − Pr [U_{k} ∈ S]}
≥ Pr [X_{I∪{k}} ∈ S_{0} ∪ S_{1}] − Pr [U_{k} ∈ S_{0} ∪ S_{1}]
= Pr [X_{I} ∈ S] − Pr [U_{k−1} ∈ S]

which by Eq. (7.9) is greater than ϵ. This is in contradiction to the assumption that X is an (ϵ, k)-approximation of the uniform distribution.

7.4. (ϵ, K)-APPROXIMATIONS OF THE UNIFORM DISTRIBUTION 81

7.4.2 Achieving (ϵ, k)-approximations of uniform

The following corollary of Theorem 7.5 establishes the relation between the max-bias of X and the quality of the (ϵ, k)-approximation of the uniform distribution achieved by X.

Corollary 7.8 Let X ∈ {0, 1}^{n} be a random variable such that maxbias(X) ≤ ϵ′. Then for every k ∈ [n], X is an (ϵ, k)-approximation of the uniform distribution, where ϵ =^{def} ^{\frac{\sqrt{2^{k}}}{2}} · ϵ′.

Proof: Let I ⊆ [n] be any subset of indices of size k; denote I = {i_{1}, …, i_{k}}. Then, as with k-wise independence, consider the distribution X_{I} = {X_{i_{1}}, …, X_{i_{k}}} (as induced by I). Note that, by definition, it holds that maxbias(X_{I}) ≤ maxbias(X), implying that maxbias(X_{I}) ≤ ϵ′. Then, by applying Theorem 7.5 we have that

2 · Δ(X_{I}, U_{k}) ≤ \sqrt{2^{k}} · maxbias(X_{I}) ≤ \sqrt{2^{k}} · ϵ′

Therefore, for every I ⊆ [n] of size k, the distribution X_{I} is ^{\frac{\sqrt{2^{k}}}{2}} · ϵ′-far from uniform.

Corollary 7.8 implies that in order to obtain (ϵ, k)-approximations of uniform, it is enough to construct an ϵ′-bias sample space (for the appropriate ϵ′). In the next lecture, such a construction will be presented. That is, the following theorem will be proven:

Theorem 7.9 There exist ϵ-bias sample spaces of size poly(n/ϵ). Moreover, such sample spaces can be constructed in poly(n/ϵ)-time.

Combining Corollary 7.8 and Theorem 7.9, we obtain:

Corollary 7.10 For every ϵ > 0 and k ≤ n, one can construct in poly(n2^{k}/ϵ)-time an (ϵ, k)- approximation of the uniform distribution over {0, 1}^{n}.

Recall that k-wise independent spaces must be of size at least Ω(n^{⌊k/2⌋}). Therefore, efficiently constructible spaces of size poly(n) can only be k-wise independent for constant values of k. On the other hand, Corollary 7.10 yields efficiently constructible (ϵ, k)-approximations of uniform, with size just poly(n · 2^{k}/ϵ). An immediation ramification of this is that, unlike for k-wise independence, efficiently constructible polynomial-size (\frac{1}{poly(n)}, k)-approximations of uniform exist for non-constant values of k (i.e., for k = O(log n)).

We comment that one can improve over Corollary 7.10 by combining Theorem 7.9 with some known k-wise independent constructions, taking advantage on the linearity of the latter. Details are given in the Appendix.

7.4.3 An application of (ϵ, k)-approximations of uniform

We conclude by briefly showing an application of (ϵ, k)-approximations of the uniform distribution to the problem of kCNF. Suppose that we would like to construct a deterministic algorithm, that given a kCNF formula φ as input finds an assignment that satisfies at least 1 − 2^{−k} of the clauses of φ. As we have already seen in Lecture 3, one can use a 3-wise independent sample space in order to solve this problem for k = 3 (i.e., in the case of 3CNF).

Specifically, we saw a deterministic algorithm that receives a 3CNF formula φ as input and finds an assignment satisfying at least 7/8 of the clauses of φ. The algorithm described worked by first constructing a 3-wise independent sample space (of size polynomial in the length of φ). The key observation was that for any clause, the probability that it is satisfied by a random element of this sample space equals the probability that it is satisfied by a uniformly chosen assignment. Thus, the probability that a given clause is satisfied is exactly 7/8. (The derandomization step consisted of simply traversing the entire 3-wise independent sample space and checking the number of clauses satisfied by each assignment from the space.)

By using a k-wise independent sample space (rather than a 3-wise independent space) one can generalize this approach to construct an algorithm that, given a kCNF formula φ, finds an assignment that satisfies at least 1 − 2^{−k} of the clauses of φ. The problem is that the size of any k-wise independent space must be at least Ω(n^{⌊k/2⌋}). Therefore, the resulting algorithm is polynomial-time only for constant values of k and also in that case its running time is Ω(n^{⌊k/2⌋}) (which is more than the alternative presented below). However, by replacing the k-wise independent space with an appropriately chosen (ϵ, k)-approximation of the uniform distribution one can do significantly better. In particular, one can construct a polynomial-time algorithm that generates the required assignment even if k is non-constant (specifically, it will work as long as k = O(log n)). Details follow.

When using an (ϵ, k)-approximation of the uniform distribution (instead of a k-wise independent sample space), the probability that any given clause is satisfied is at least (1 − 2^{−k}) − ϵ, and the expected number of clauses that are satisfied is at least (1 − 2^{−k}) · n − ϵ · n (where n equals the number of clauses in φ). Now, if we set ϵ such that ϵ · n < 2^{−k} (e.g., ϵ = 1/(10 · 2^{k} · n)), then we have that there exists a satisfying assignment in the sample space satisfying 1 − 2^{−k} of the clauses. By Corollary 7.8, we have that an (ϵ, k)-approximation of the uniform distribution can be obtained from a (2 · ϵ/\sqrt{2^{k}})-bias space. Thus, for ϵ = 1/(10 · 2^{k} · n) we need to construct (and scan) an ϵ′-bias sample space with ϵ′ = 2/(10 · 2^{3·k/2} · n). By Theorem 7.9 this can be done in time poly(n/ϵ′) = poly(n, 2^{k}), giving us the desired result. We note that O((n · 2^{k})^{c}) is preferrable to O(n^{⌊k/2⌋}) for every k > 2 · c and sufficiently large n.

Bibliographic Notes

Small-bias sample spaces were introduced by Naor and Naor [35]. The relation between statistical difference and the maximum bias of XORs of bits was first discovered by Vazirani [43], but the proof presented here follows the one given in the appendix of [8].

Appendix

Oded’s Note: The following material was not presented in class (although this was the original plan). This, as well as other misfortunes, is due to the slower than expected pace I was forced to use.

We present an improvement over Corollary 7.10. Recall that Corollary 7.10 was proven by combining the generic relation between small-bias and statistical-distance (as applied to “windows of size k” of the distribution) with a construction of small-bias sample spaces (i.e., Theorem 7.9). Here we combining Theorem 7.9 with some known k-wise independent constructions, taking advantage of their linearity. We start with the transformation.

Lemma (Naor and Naor [35]): Let S_{n} ⊆ {0, 1}^{n} be an ϵ-bias sample space. Let k ≤ N and L_{k,N}^{n} ⊆ {0, 1}^{N} be a k-wise independent sample space of cardinality 2^{n}. Suppose L_{k,N}^{n} is defined by the linear mapping T : {0, 1}^{n} → {0, 1}^{N} (i.e, T (x ⊕ y) = T (x) ⊕ T (y)). Then, the sample space R_{k,N} constructed by applying the linear map T to each sample point in S_{n}, has bias at most ϵ relative to any non-empty subset of size at most k. Hence, R_{k,N} is a (2^{k/2} · ϵ, k)-approximation of the uniform distribution over {0, 1}^{N}. Indeed, |R_{k,N}| = |S_{n}|, where n is such that a k-wise independent sample space (over {0, 1}^{N}) as required exists. We will see that such spaces exist for n = k log_{2} N, and so we will get an (2^{k/2} · ϵ, k)-approximation (of the uniform distribution) over {0, 1}^{N} using a sample space of size poly(n/ϵ) = poly((k log N)/ϵ), improving over Corollary 7.10 (which yields size poly(N/ϵ) for such spaces).

Proof: Let X = X_{1} ⋯ X_{N} denote a random variable that is uniformly distributed in R_{k,N} ⊆ {0, 1}^{N}. By the construction, X = T (Y), where Y = Y_{1} ⋯ Y_{n} denote a random variable that is uniformly distributed in S_{n} ⊆ {0, 1}^{n}. Our hypothesis is that Y is ϵ-bias, and our first goal is to bound the bias of X relative to any non-empty subset I of size at most k. Let M = (a_{i,j}) denote the N-by-n Boolean matrix effecting the linear map T (i.e., T (Y) = MY). We have

Pr [⨁_{i∈I} X_{i} = 0] = Pr [⨁_{i∈I} T (Y)_{i} = 0]

= Pr [⨁_{i∈I} (⨁_{j=1}^{n} a_{i,j}Y_{j}) = 0]

= Pr [⨁_{j=1}^{n} ⨁_{i∈I} a_{i,j}Y_{j} = 0]

= Pr [⨁_{j∈J} Y_{j} = 0]

where j ∈ J if and only if ⨁_{i∈I} a_{i,j} = 1. Thus, bias_{I}(X) = bias_{J}(Y), which is bounded by ϵ, provided that J ≠ ∅ (because Y is ϵ-bias). But J ≠ ∅ must hold, because otherwise bias_{I}(T (U_{n})) = bias_{∅}(U_{n}) = 1 (where U_{n} is the uniform distribution over {0, 1}^{n}) in contradiction to the hypothesis that T (U_{n}) is k-wise independent (which implies that bias_{I}(T (U_{n})) = 0 for every non-empty subset of size at most k, such as I). The establishes the main claim. The second claim follows by applying Corollary 7.8.

Constructing k-wise independent sample spaces via linear maps: It is left to show that we can construct k-wise independent sample spaces (of the uniform distribution) over {0, 1}^{N} by using linear maps from {0, 1}^{n}, where n = k log_{2} N. Recall that the sample spaces presented in Lecture 3 have the desired size (i.e., 2^{n} = N^{k}), but are they linear? These sample spaces were described in terms of polynomial of degree k − 1 over F =^{def} GF (2^{ℓ}), where ℓ = ⌈log_{2} N⌉, and are naturally presented as a mapping from F^{k} to F^{N}. Specifically, a degree k − 1 polynomial represented by the coefficient sequence (c_{0}, ..., c_{k−1}) ∈ F^{k} is mapped to the sequence of values (v_{1}, ...., v_{N}) ∈ F^{N}, where v_{j} = ∑_{i=0}^{k−1} c_{i}α_{j}^{i} and α_{j} is the jth element of the field F. Noting that here the α_{j}’s are fixed, we get linear maps over F:

(c_{0}, ..., c_{k−1}) ↦ ∑_{i=0}^{k−1} α_{j}^{i}c_{j}

Recalling that F = GF (2^{ℓ}), these linear maps over F can be written as linear maps over GF (2): elements of F = GF (2^{ℓ}) are viewed as polynomials of degree ℓ − 1, and multiplication by fixed elements (i.e., the powers of the α_{j}’s) can be expressed as linear combinations of the GF (2)- coefficients of these polynomials. (For further details, consult any standard Coding Theory book; for example, [32, Chap. 4].)

Small Bias Sample Spaces (Part 2)

Notes taken by Boaz Barak and Itsik Mantin

Summary: In the previous lecture we discussed the definition of small bias sample spaces and some properties of such spaces. In this lecture we prove that such sample spaces can be constructed in polynomial time, and demonstrate an application of this construction in proving a tight hardness of approximation result for an N P-complete problem.

8.1 Introduction

In the previous lecture, we have seen the definition of ϵ-bias sample spaces. Let us recall this definition:

Definition 8.1 Let n ∈ N and ϵ > 0. A (multi) set Ω ⊆ {0, 1}^{n} is called an ϵ-bias sample space over {0, 1}^{n} if for any non-empty I ⊆ {1, …, n}

| Pr_{s∈Ω}[⨁_{i∈I} s_{i} = 0] − Pr_{s∈Ω}[⨁_{i∈I} s_{i} = 1]| < ϵ

where s = s_{1} … s_{n}.

An equivalent formulation is that for any non-empty I ⊆ {1, …, n} it holds that

Pr_{s∈Ω}[⨁_{i∈I} s_{i} = 0] ∈ \frac{1}{2} ± \frac{ϵ}{2}

(where by a ± c we mean the interval (a − c, a + c))

Last lecture we noted that the uniform distribution has bias zero, and proved a relation between the bias of a distribution and its statistical difference from the uniform distribution.

This lecture we will see (in Section 8.2) that ϵ-bias sample spaces can be constructed in time polynomial in \frac{n}{ϵ}. In particular this ϵ-bias sample spaces (over \{0, 1\}^{n}) has size poly(\frac{n}{ϵ}) (as opposed to the uniform sample space over {0, 1}^{n} that is of size 2^{n}).

In Section 8.3 we will use this construction to prove that a specific N P-complete problem (Quadratic Equations) that can be approximated with ratio \frac{1}{2} cannot be approximated with any better constant ratio, unless P = N P. This is one of the few tight hardness of approximation results that can be obtained without using the PCP Theorem.

86 LECTURE 8. SMALL BIAS SAMPLE SPACES (PART 2)

8.2 Construction of a Small Bias Sample Space

In this section we will show that it is possible to construct an ϵ-bias sample space of size poly(\frac{n}{ϵ}). That is, we prove the following Theorem:

Theorem 8.2 There exists an algorithm that, on input n and ϵ outputs a list of n-bit strings that constitutes an ϵ-bias space and does so within time poly(\frac{n}{ϵ}).

8.2.1 On the Existence of poly(\frac{n}{ϵ})-size ϵ-bias sample spaces.

Before proving Theorem 8.2 we note that the mere existence of an ϵ-bias sample space of poly(\frac{n}{ϵ}) size can be easily proved using the probabilistic method. That is, we have the following:

Proposition 8.2.1 For any n ∈ N, ϵ > 0 , there exists an ϵ-bias sample space over {0, 1}^{n} of size O(\frac{n}{ϵ^{2}} ).

Proof: Let Ω be a (multi) set containing strings in {0, 1}^{n}. For any non-empty I ⊆ {1, …, n} we define f_{I}(Ω) to be the number of strings s = s_{1} … s_{n} ∈ Ω such that ⨁_{i∈I} s_{i} = 0. By definition, it holds that Ω is an ϵ-bias sample space if (and only if) for any such I, it holds that f_{I}(Ω) ∈ (\frac{1}{2} ± \frac{ϵ}{2}) · |Ω|.

Consider a random (multi) set Ω of size \frac{n}{ϵ^{2}} . That is Ω = \{s^{(1)}, …, s^{(n/ϵ^{2})}\} where for each j, s^{(j)} is chosen uniformly and independently in {0, 1}^{n}. Let us fix I to be a non-empty subset of {1, …, n}, and let X_{I} be the random variable f_{I}(Ω). Another way to define X_{I} is as ∑_{j=1}^{|Ω|} Y_{j}

where Y_{j} is a random variable that is defined to be 1 if ⊕_{i∈I}s_{i}^{(j)} = 0 and 0 otherwise. The fact that the uniform distribution has bias zero implies that Pr[Y_{j} = 1] = \frac{1}{2}, and so μ =^{def} Ex[X] = \frac{|Ω|}{2} = \frac{n}{2ϵ^{2}} .

The random variables Y_{1}, …, Y_{k/ϵ^{2}} are independent. Therefore, we can apply the Chernoff bound and get that (for every I ≠ ∅)

Pr_{Ω}[f_{I}(Ω) ∉ (\frac{1}{2} ± \frac{ϵ}{2}) · |Ω|] = Pr_{Ω}[|X_{I} − μ| > ϵμ]

= Pr_{Ω}[| ∑_{j=1}^{|Ω|} Y_{j} − μ| > ϵμ]
< 2e^{−2ϵ^{2}μ}
= 2e^{−2ϵ^{2}·(n/2ϵ^{2})}
= 2e^{−n}

For sufficiently large n, this probability is strictly smaller than 2^{−n}. As there are at most 2^{n} subsets of {1, …, n}, we get that by the union bound there is a positive probability that for a random Ω of size \frac{n}{ϵ^{2}} , for any non-empty set I ⊆ \{1, …, n\} the number f_{I}(Ω) is in (\frac{1}{2} ± \frac{1}{2}ϵ)·|Ω|. In particular there must exist at least one (multi) set Ω of size \frac{n}{ϵ^{2}} that satisfies the above condition for any non-empty I ⊆ {1, …, n}. Such a set Ω will be an ϵ-bias sample space.

8.2.2 The Construction

We will prove Theorem 8.2 by showing a polynomial-time algorithm that outputs an ϵ-bias sample space Ω of size O((\frac{n}{ϵ})^{2}). This sample space will be larger than the sample space of Proposition 8.2.1 (which is O(\frac{n}{ϵ^{2}} )) but it will be explicitly constructible. We will start by showing the construction and then provide its analysis.

LFSR sequences

The primary tool we will use is a Linear Feedback Shift Register (LFSR). An LFSR is a sequence of bits defined by two parameters: a starting sequence and a feedback rule. The name linear feedback shift register comes from a popular hardware implementation of this sequence.

Notation. In this section it will be convenient for us to treat strings as indexed from 0 to l − 1 (where l is the string’s length) instead of being indexed from 1 to l.

Definition 8.3 Let l ∈ N and let s = s_{0} ⋯ s_{l−1} and f = f_{0} ⋯ f_{l−1} be two l-bit strings. The LFSR sequence defined by s and f is the sequence (r_{0}, r_{1}, r_{2}, …) defined as follows:

r_{i} =^{def} \{ \begin{matrix}s_{i} & 0 ≤ i ≤ l − 1 \\ ∑_{j=0}^{l−1} f_{j} · r_{(i−l)+j} & i ≥ l \end{matrix}

We call s the starting sequence and f the feedback rule.

Note that the definition of an LFSR sequence is a recursive definition, with each bit r_{i} (for i ≥ l) depending only on the previous l bits. Therefore, this sequence will repeat itself after at most 2^{l} steps, or in other words, it is a periodic sequence with period at most 2^{l}.

Let us fix l, n ∈ N with l < n (actually we will use l ≪ n). We define the function seq_{n} : {0, 1}^{l} × {0, 1}^{l} → {0, 1}^{n} as follows: for s, f ∈ {0, 1}^{l}, seq_{n}(s, f) consists of the first n bits of the LFSR sequence defined by s and f. That is seq_{n}(s, f) = r_{0} ⋯ r_{n−1}, where r_{0}, …, r_{n−1} are defined as in Definition 8.3.

Polynomials over GF (2). To describe the rest of the construction, we will need to use polynomials with coefficients in GF (2). Recall that one can define multiplication and division for such polynomials. We say that a polynomial is irreducible if it has no non-trivial factors (i.e., the only polynomials that divide it are itself and the polynomial f(t) ≡ 1). That is, irreducible polynomials in the polynomial ring correspond to prime numbers in the domain of natural numbers.

It will be useful for us to identify the feedback rule f = (f_{0}, …, f_{l−1}) with the following l-degree polynomial with coefficients in GF (2):

f(t) = −t^{l} + ∑_{j=0}^{l−1} f_{j} · t^{j}

(Note that −1 = +1 over GF (2). The reason that we prefer to write the definition of the polynomial f(t) in the above way, will become clear later on)

We say that f is a valid feedback rule if the corresponding polynomial is irreducible over GF (2). We let IP_{l} ⊆ {0, 1}^{l} denote the set of all valid feedback rules (or equivalently, the set of all irreducible polynomials of degree exactly l over GF (2)). The following fact is well-known:

Fact 8.2.2 |IP_{l}| = Θ(\frac{2^{l}}{l}).

The actual sample space

We define the following sample space

Ω_{n,l} =^{def} {seq_{n}(f, s) : f ∈ IP_{l}, s ∈ {0, 1}^{l}}

We have that |Ω_{n,l}| = Θ(\frac{2^{2l}}{l}). Moreover, it is possible to enumerate the elements of Ω_{n,l} in time polynomial in n and 2^{l}. Indeed, given s, f ∈ {0, 1}^{l} the function seq_{n}(s, f) can be evaluated in poly(n) time, and enumerating all possible choices for s, f can be done in 2^{O(l)} time (note that checking whether a polynomial of degree l over GF (2) is irreducible can be done trivially in time 2^{O(l)} by checking all possible factors

8.2.3 Analysis

We prove Theorem 8.2 by proving the following proposition:

Proposition 8.2.3 For some constant k and for every l, n ∈ N with l < n, the set Ω_{n,l} is a k · \frac{n}{2^{l}} -bias sample space over \{0, 1\}^{n}.

Proposition 8.2.3 immediately implies Theorem 8.2. This is because if we let l = log(k · \frac{n}{ϵ}), then by Proposition 8.2.3, Ω_{n,l} is an ϵ-bias sample space over \{0, 1\}^{n} of size O(\frac{2^{2l}}{l}) = O((\frac{n}{ϵ})^{2}) that is constructible in poly(\frac{n}{ϵ})-time.

We will start by some preliminary observations on the connection between the LFSR sequence and the polynomial that corresponds to the feedback rule. Then, in Section 8.2.3, we shall use these observations to prove Proposition 8.2.3.

Preliminary Observations

Let us fix l < n to be two natural numbers as in the statement of Proposition 8.2.3. Let s and f be a starting sequence and a feedback rule in {0, 1}^{l} and let (r_{0}, …, r_{n−1}) = seq_{n}(s, f). We know that for any i ≥ l

r_{i} = ∑_{j=0}^{l−1} f_{j} · r_{(i−l)+j}  (8.1)

It is clear that any value r_{i} is a linear combination of the value of the starting sequence s_{0}, …, s_{l−1}. That is, there exists an n × l matrix C = (c_{i,j}) over GF (2) such that for any i

r_{i} = ∑_{j=0}^{l−1} c_{i,j} · s_{j}  (8.2)

In matrix notation, we can write Cs = r, where r is the vector seq_{n}(s, f) = (r_{0}, …, r_{n−1}). The entries of the matrix C can be obtained by repeated applications of Equation 8.1, and hence are determined by the value of the feedback rule f. We will sometimes denote this matrix C by C(f).

Consider the polynomial f(t) of degree l that corresponds to f (that is, f(t) = −t^{l} +∑_{j=0}^{l−1} f_{j} ·t^{j}). We can reduce any polynomial (of degree at least l) modulo this polynomial f(t). In particular we have that

t^{l} = ∑_{j=0}^{l−1} f_{j} · t^{j} − f(t) ≡ ∑_{j=0}^{l−1} f_{j} · t^{j} (mod f(t))

For i ≥ l we have that

t^{i} = t^{i−l}t^{l} ≡ t^{i−l}(∑_{j=0}^{l−1} f_{j}t^{j}) ≡ ∑_{j=0}^{l−1} f_{j}t^{(i−l)+j} (mod f(t))  (8.3)

Note the analogy between Equations 8.1 and 8.3. The polynomial t^{i} is equivalent (modulo f(t)) to a polynomial of degree l − 1, or in other words to a linear combination of t^{0}, …, t^{l−1}. The coefficients of this linear combination can be obtained by repeated application of Equation 8.3. The analogy between Equations 8.1 and 8.3 implies that these will be the same coefficients as in Equation 8.2. That is, the following holds:

t^{i} ≡ ∑_{j=0}^{l−1} c_{i,j} · t^{j} (mod f(t))  (8.4)

where C = (c_{i,j}) is the very same n × l matrix C(f) defined above. In matrix notation we have that

C ( 1 ; t ; ⋮ ; t^{l−1} ) ≡ ( 1 ; t ; ⋮ ; ⋮ ; t^{n−1} ) (mod f(t))  (8.5)

By applying the transpose operation to Equation 8.5, we obtain the following equation that will be useful for us later on:

(1 ⋯ t^{l−1})C^{T} ≡ (1 ⋯ t^{n−1}) (mod f(t))  (8.6)

Proving Proposition 8.2.3

In order to prove that Ω_{n,l} is a O(\frac{n}{2^{l}})-bias sample space we need to show that the following holds for any non-empty I ⊆ {0, …, n − 1}:

Pr_{r∈Ω_{n,l}} [⨁_{i∈I} r_{i} = 0] ∈ \frac{1}{2} ± O(\frac{n}{2^{l}})

Equivalently, we need to prove that for any non-zero vector v ∈ {0, 1}^{n}

Pr_{r∈Ω_{n,l}} [⟨v , r⟩ = 0] ∈ \frac{1}{2} ± O(\frac{n}{2^{l}})  (8.7)

where ⟨· , ·⟩ denotes the standard dot-product (i.e., ⟨v , r⟩ =^{def} v^{T} r = ∑_{i=0}^{n−1} u_{i} · r_{i} (mod 2)). In this notation, the fact that the uniform distribution has bias 0 can be phrased as follows:

Fact 8.2.4 Let v ≠ 0 be a vector in {0, 1}^{n}, then

Pr_{r∈\{0,1\}^{n}}[⟨v , r⟩ = 0] = \frac{1}{2}

Let us fix a non-zero vector v ∈ {0, 1}^{n} and let p denote the probability defined in Equation 8.7. By the definition of the set Ω_{n,l} we know that

p = Pr_{s∈{0,1}^{l},f∈IP_{l}}[⟨v , seq_{n}(s, f)⟩ = 0]
= Pr_{s∈{0,1}^{l},f∈IP_{l}}[⟨v , C(f)s⟩ = 0]
= Pr_{s∈{0,1}^{l},f∈IP_{l}}[⟨C(f)^{T} v , s⟩ = 0]

where the last equality holds by a well-known property of the dot product. Yet, by Fact 8.2.4 above, we know that, conditioned on C(f)^{T} v being a non-zero vector, the above probability is exactly half. We also know that if C(f)^{T} v is the zero vector, then we will have that ⟨C(f)^{T} v , s⟩ = 0 with probability 1. It follows that if we let A denote the event that C(f)^{T} v is the zero vector, and let ρ denote the probability (over f ∈ IP_{l}) that A happens, then

p = (1 − ρ) · \frac{1}{2} + ρ · 1 = \frac{1}{2} + \frac{1}{2} · ρ

Therefore, to prove Proposition 8.2.3, all we need to do is to prove that ρ = O(\frac{n}{2^{l}}). Consider the case that the event A happens. In this case we have that

C(f)^{T} v = 0

and so (for C = C(f)) it holds that for any l-bit vector x

xC^{T} v = 0

(note that in the first equation 0 denotes the zero l-bit vector, while in the second equation we use the scalar 0).

In particular if we plug x = (1 ⋯ t^{l−1}) we will get that the polynomial (1 ⋯ t^{l−1})C^{T} v is the zero polynomial (modulo f(t)). Combining this with Equation 8.6 we get that

(1 ⋯ t^{n−1})v ≡ 0 (mod f(t))

which means that the polynomial f(t) divides the non-zero polynomial gv(t) =^{def} (1 ⋯ t^{n−1})v = ∑_{i=0}^{n−1} u_{i} · t^{i}. Note that the polynomial gv(t) depends only on v and has degree at most n. We have that ρ ≤ Pr_{f∈IP_{l}}[f(t) divides gv(t)]. Yet any polynomial of degree at most n has at most \frac{n}{l} irreducible^{1} factors of degree l, and so the probability that f(t) happens to be one of these factors is at most \frac{n/l}{|IP_{l}|}, which by Fact 8.2.2 is equal to \frac{n/l}{Θ(2^{l}/l)} = O(\frac{n}{2^{l}}).

8.3 Using Small Size Sample Spaces for proving Hardness of MaxQE

8.3. USING SMALL SIZE SAMPLE SPACES FOR PROVING HARDNESS OF MAXQE 91

In this section we present the problem of Quadratic Equations satisfiablity (QE). We state that the QE problem is N P-Complete (and sketch the proof) and discuss the approximability of the optimization variant, denoted as MaxQE. We state that this problem can be approximated with a ratio of one half, and we use a construction of small bias sample spaces (e.g., from Theorem 8.2), to prove that this bound is tight and no polynomial algorithm can approximate MxaQE with a better constant ratio (unless P = N P).

1Note that because different reducible polynomials may have shared factors, it may be the case that a polynomial of degree n has much more than n/l factors of degree l that are not irreducible. Therefore it is crucial that we limit ourselves only to valid feedback rules (i.e., feedback rules that correspond to irreducible polynomials).

8.3.1 The problem of Quadratic Equations

A quadratic expression is a multi-variate polynomial with maximal degree of 2. A quadratic equation is an equation whose left side is a quadratic expression, and whose right side is a scalar. Given a set of quadratic equations sharing the same variables, a question that arise is whether there exists an assignment for this variables, which satisfies all equations, or formally:

Definition 8.4 The set Quadratic Equations denoted QE, consists of all mutually satisfiable sequences of quadratic equations over GF (2).

Since in GF (2) it holds that x = x^{2} for any number x, we can assume without loss of generality that all the summands are either of degree 2 or are constants. The formulation of QE as a decisional problem is given by:

Input A sequence of n quadratic equations with t variables over GF (2)

Question Is there an assignment to the variables that satisfies all the equations.

Theorem 8.5 QE ∈ N PC

Proof Sketch: Clearly QE is in N P as a satisfying assignment is an easily verifiable witness. To see that it is N P-hard we will see a reduction from 3SAT.

Consider an instance of 3SAT, denoted φ = ⋀_{i=1}^{m} C_{i}, where each clause C_{i} is the disjunction

of three literals l_{i1}, l_{i2} and l_{i3}, each of them being either one of the variables p_{1}, …, p_{n} or its negation.

The reduction consists of two stages, the first one translating the formula into a sequence of m cubic equations, while the second translates this sequence into a sequence of quadratic equations.

The first translation is natural, as 3CNF formulas have immediate correspondence to cubic equations. For example, a Boolean equation of the form p_{1}∨p_{2}∨¬p_{3} =^{?} T RUE can be translated by De-Morgan’s rules into ¬p_{1} ∧ ¬p_{2} ∧ p_{3} =^{?} F ALSE, which corresponds to the cubic equation

(1 − x_{1})(1 − x_{2})x_{3} =^{?} 0.

It is clear that the 3SAT input φ is satisfiable if and only if the sequence of cubic equations is satisfiable. Next, we describe the translation of these equations into quadratic equations. For each pair of variables x_{i} and x_{j} in the cubic system, we introduce a new variable z_{ij} and a new equation z_{ij} = x_{i}x_{j}. Then, we go through all the equations and whenever we find a term of degree 3 (e.g., x_{i}x_{j}x_{k}) we replace it by a term of degree 2 (e.g., z_{ij}x_{k}) by replacing the multiplication of two variables x_{i}x_{j} with the new variable z_{ij}. Our new equation system is composed of two parts, the first consists of the new added equations {z_{ij} = x_{i}x_{j}}_{i,j=1}^{n} (and thus is quadratic in the size of the original input), and the second part is the transformed sequence of equations all of degree 2 (linear in the size of the input).

Consider for example, the 3SAT input

(x_{1} ∨ x_{2} ∨ ¬x_{3}) ∧ (¬x_{2} ∨ x_{3} ∨ x_{4}) ∧ (x_{1} ∨ x_{4} ∨ x_{5}) =^{?} T RUE

92 LECTURE 8. SMALL BIAS SAMPLE SPACES (PART 2)

By applying De-Morgan’s rules we get an equivalent formula

(¬x_{1} ∧ ¬x_{2} ∧ x_{3}) ∨ (x_{2} ∧ ¬x_{3} ∧ ¬x_{4}) ∨ (¬x_{1} ∧ ¬x_{4} ∧ ¬x_{5}) =^{?} F ALSE

which is (notice that for the formula to be satisfied, each of the clauses must be FALSE)

( \begin{matrix}¬x_{1} ∧ ¬x_{2} ∧ x_{3} \\ x_{2} ∧ ¬x_{3} ∧ ¬x_{4} \\ ¬x_{1} ∧ ¬x_{4} ∧ ¬x_{5} \end{matrix} ) =^{?} ( \begin{matrix}F ALSE \\ F ALSE \\ F ALSE \end{matrix} )

This triplet of Boolean equations is translated into the cubic equations

( \begin{matrix}(1 − x_{1})(1 − x_{2})x_{3} \\ x_{2}(1 − x_{3})(1 − x_{4}) \\ (1 − x_{1})(1 − x_{4})(1 − x_{5}) \end{matrix} ) =^{?} ( \begin{matrix}0 \\ 0 \\ 0 \end{matrix} )

which is

( \begin{matrix}x_{1}x_{2}x_{3} − x_{1}x_{3} − x_{2}x_{3} + x_{3} \\ x_{2}x_{3}x_{4} − x_{2}x_{3} − x_{2}x_{4} + x_{2} \\ −x_{1}x_{4}x_{5} + x_{4}x_{5} + x_{1}x_{4} + x_{1}x_{5} − x_{4} − x_{5} − x_{1} + 1 \end{matrix} ) =^{?} ( \begin{matrix}0 \\ 0 \\ 0 \end{matrix} )

Notice that all the cubic terms include either the term x_{2}x_{3} or the term x_{1}x_{4}, and thus it suffices to replace only these quadratic products by new variables, i.e. z_{23} and z_{24} to reduce the maximal degree into 2

( \begin{matrix}x_{1}x_{4} − z_{14} \\ x_{2}x_{3} − z_{23} \\ x_{3} + x_{1}z_{23} − x_{1}x_{3} − z_{23}) \\ x_{2} + z_{23}x_{4} − z_{23} − x_{2}x_{4} \\ −z_{14}x_{5} + x_{4}x_{5} + x_{1}x_{4} + x_{1}x_{5} − x_{4} − x_{5} − x_{1} + 1 \end{matrix} ) =^{?} ( \begin{matrix}0 \\ 0 \\ 0 \\ 0 \\ 0 \end{matrix} )

The last technical step is to go over the equation sequence and apply the following three technical transforms: a) Replace subtractions by additions (its the same in GF (2)); b) Replace every linear term by its square; c) Move the scalar ’1’ to the right side. Finally, we get an input for the QE problem

( \begin{matrix}x_{2}x_{3} + z_{23}^{2} \\ x_{1}x_{4} + z_{14}^{2} \\ x_{3}^{2} + x_{1}z_{23} + x_{1}x_{3} + z_{23}^{2} \\ x_{2}^{2} + z_{23}x_{4} + z_{23}^{2} + x_{2}x_{4} \\ z_{14}x_{5} + x_{4}x_{5} + x_{1}x_{4} + x_{1}x_{5} + x_{4}^{2} + x_{5}^{2} + x_{1}^{2} \end{matrix} ) =^{?} ( \begin{matrix}0 \\ 0 \\ 0 \\ 0 \\ 1 \end{matrix} )

which is satisfiable if and only if the original 3SAT input φ was satisfiable. Since the entire procedure of reducing a 3CNF to a sequence of quadratic equations can be done in polynomial time we have reduced 3SAT to QE.

8.3.2 The Optimization Problem MaxQE

Next, we define the optimization variant of QE, denoted MaxQE:

Input A sequence of n quadratic equations with t variables over GF (2)

Problem Find an assignment that satisfies as many of the equations as possible.

We define the function MaxQE(I) ∈ [0, 1] to be the maximal fraction of equations in I, which can be mutually satisfied by a single assignment.

Example: The following is an instance for both QE and MaxQE, with the parameters n = 5 and t = 6.

\{ \begin{matrix}x_{2}x_{3} + z_{23}^{2} = 0 \\ x_{1}x_{4} + z_{14}^{2} = 0 \\ x_{3}^{2} + x_{1}z_{23} + x_{1}x_{3} + z_{23}^{2} = 0 \\ x_{2}^{2} + z_{23}x_{4} + z_{23}^{2} + x_{2}x_{4} = 0 \\ z_{14}x_{5} + x_{4}x_{5} + x_{1}x_{4} + x_{1}x_{5} + x_{4}^{2} + x_{5}^{2} + x_{1}^{2} = 1 \end{matrix}

Notice that the assignment

( \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \\ x_{4} \\ x_{5} \\ z_{23} \\ z_{14} \end{matrix} ) = ( \begin{matrix}1 \\ 1 \\ 1 \\ 0 \\ ∗ \\ 1 \\ 0 \end{matrix} )

satisfies all the equations (there are several satisfying assignments for this QE input), and consequently the the answer to the decisional problem is “YES”, whereas the answer for the MaxQE problem is one of the satisfying assignments (and MaxQE(I) = 1).

The best known poly-time approximation algorithm for MaxQE gives an approximation ratio of \frac{1}{2}. The algorithm that achieves this ratio (and its analysis) are beyond the scope of this course and are omitted.

8.3.3 MaxQE is hard to (\frac{1}{2} + ϵ)-approximate

Theorem 8.6 For any ϵ > 0 MaxQE is N P-Hard to approximate to within factor \frac{1}{2} + ϵ.

Before getting to the proof itself, we present new terminology which we use in the proof. We extend the decisional QE problem into a family of problems, denoted GapQE_{α,β}, parameterized by α, β ∈ [0, 1]. For this purpose let us define a promise problem. Loosely speaking, a promise problem is a problem for which the input-output relation is only partially defined, i.e. for some inputs the output is strictly defined (as in typical problems), while for other inputs the output is insignificant.

Next we define two classes of QE inputs

∀α Y esQE_{α}:

A sequence of n quadratic equations with t variables over GF (2) such that there is an assignment to the variables that satisfies at least a fraction α of the equations.

∀β NoQE_{β}:

A sequence of n quadratic equations with t variables over GF (2) such that all the assignments to the variables satisfy strictly less than a fraction β of the equations.

Example: Y esQE_{1} is the set of all satisfiable QE inputs, while NoQE_{1} is the set of all other QE inputs (those which are not satisfiable).

The problem GapQE_{α,β} (for α ≥ β ∈ [0, 1]), is the promise problem of distinguishing Y esQE_{α} inputs from NoQE_{β}. However, the fact that NoQE_{β} includes inputs whose best assignments satisfy strictly less than a fraction of β, implies that GapQE_{α,α} is defined. Notice that in this case the output is defined for inputs which are α-satisfiable (and thus in Y esQE_{α}) and for inputs which are not α-satisfiable (and thus in NoQE_{α}), i.e. for all inputs. Consequently GapQE_{α,α} is actually a standard decisional problem. Example: GapQE_{1,1} ≡ QE

Lemma 8.3.1 Suppose that there exists a polynomial time β-approximation algorithm for QE. Then GapQE_{1,β} is decidable in polynomial time.

Proof: For QE inputs from Y esQE_{1}, a β-approximation algorithm is promised to provide an assignment satisfying at least β of the equations. For inputs of NoQE_{β} no algorithm can find such an assignment, since no assignment satisfies β of the equations. Thus, applying such an algorithm to some QE input, one can compare the number of satisfied equations to β, and decide about the output for GapQE_{1,β} accordingly.

The proof of Theorem 8.6 is constructed in the following way: we prove that for every ϵ > 0, GapQE_{1, \frac{1}{2} +ϵ} is N P-Hard, and use Lemma 8.3.1 to conclude that MaxQE is N P-Hard to \frac{1}{2} + ϵapproximate.

Lemma 8.3.2 For every ϵ > 0, there exists a polynomial reduction from QE to GapQE_{1, \frac{1}{2} +ϵ}

Proof: Let I be an input for QE. We show a polynomial time algorithm that constructs from I an input I′ for Gap_{1, \frac{1}{2} +ϵ}, where I′ ∈ Y esQE_{1} ∪ NoQE_{\frac{1}{2} +ϵ} (i.e. the output I′ satisfies the promise of GapQE), and furthermore I′ ∈ Y esQE_{1} if and only if I ∈ QE. (It follows that I′ ∈ NoQE_{\frac{1}{2} +ϵ} if and only if I ∉ QE.)

Let us first introduce some notations we use in the proof. Whenever we deal with a sequence I of n quadratic equations with the Boolean variables x_{1}, … x_{t}, we denote every equation as e_{k} (1 ≤ k ≤ n), the coefficients of the quadratic terms on e_{k} as c_{i,j}^{(k)} and the free term of e_{k} as b^{(k)}. That is, such a sequence is denoted by

{e_{k} : ∑_{i=1}^{t} ∑_{j=1}^{t} c_{i,j}^{(k)}x_{i}x_{j} = b^{(k)}}_{k=1}^{n}

Whenever we deal with a sample space Ω ⊆ {0, 1}^{n}, we denote its size by m (and use m = poly(n)), its l^{th} element as s^{(l)} and the k^{th} bit of s^{(l)} as s_{k}^{(l)}. Thus, Ω = {s^{(1)}, …, s^{(m)}}) and s^{(l)} = (s_{1}^{(l)}, s_{2}^{(l)}, …, s_{n}^{(l)}).

Let I be an input for QE. The transformation algorithm is the following:

1. Construct an ϵ-bias sample space Ω over {0, 1}^{n} of size m = poly(n).

2. Construct I′, a QE input with the same variables x_{1}, …, x_{t} as I, but with linear combinations of e_{1}, …, e_{n} as the quadratic equations. The linear combinations are those corresponding to the elements s_{1}, …, s_{m} of Ω. That is, for every l for which 1 ≤ l ≤ m, we define f_{l} to be the quadratic equation

f_{l} : ∑_{k=1}^{n} s_{k}^{(l)}e_{i}

which is

∑_{k=1}^{n} s_{k}^{(l)} \underbrace{∑_{i=1}^{t} ∑_{j=1}^{t} c_{i,j}^{(k)}x_{i}x_{j}}_{LEF T (e_{i})} = ∑_{k=1}^{n} s_{k}^{(l)} · \underbrace{b^{(k)}}_{RIGHT (e_{i})}

in other words the new l^{th} equation is

∑_{i=1}^{t} ∑_{j=1}^{t} \underbrace{∑_{k=1}^{n} s_{k}^{(l)} · c_{i,j}^{(k)}}_{coefficient of f_{l}} ·x_{i}x_{j} = \underbrace{∑_{k=1}^{n} s_{k}^{(l)} · b^{(k)}}_{free term of f_{l}}

We now prove that the this transformation is indeed a polynomial reduction.

Claim 8.3.3 The transformation can be applied in polynomial time

Proof: Constructing a polynomial sized ϵ-bias sample space (see Section 8.2) can be done in polynomial time (i.e., polynomial in n and \frac{1}{ϵ}, where here ϵ is a fixed constant). The time required for the second stage (generating the new equations) is linear in the size of the space, i.e. polynomial.

Claim 8.3.4 I is satisfiable ⟹ I′ ∈ Y esQE_{1}

Proof: Suppose that there is an assignment satisfying all the equations of I, and let τ = (τ_{1}, …, τ_{t}) be this assignment. Notice that an assignment that satisfies some equations, satisfies also every linear combination of them. Consequently τ satisfies all the equations of I′, i.e. I′ ∈ Y esQE_{1}.

Claim 8.3.5 I is not satisfiable ⟹ I′ ∈ NoQE_{\frac{1}{2} +ϵ}

Proof: Let I be an unsatisfiable sequence of quadratic equations. We need to prove that no assignment of x_{1}, …, x_{t}, satisfies a fraction of \frac{1}{2} +ϵ of the equations of I′. Fixing any assignment τ to the variables of I, we show that τ satisfies less than \frac{1}{2} + ϵ of the equations in I′. Our argument relies on the following fact:

Fact 8.3.6 Let τ be an assignment to x_{1}, …, x_{t}, and let K ⊆ {1, …, n} be the set of indices of equations in I that the assignment τ does not satisfy. Let f be a linear combination of the quadratic equations in I, that is f = ∑_{k=1}^{n} s_{k}e_{k} where s_{1}, …, s_{n} ∈ {0, 1}. Then τ satisfies f if and only if

⨁_{k∈K} s_{k} = 0

that is, the number of k’s such that k ∈ K and s_{k} = 1 is even.

Proof: Every s_{k} that is assigned 1, represents an equation that is included within f. Each of these k’s that is in K, represents an equation that is not satisfied by τ, and contributes 1 to the difference between LET F (f) and RIGHT (f).

Recall that f is an equation (mod2), and thus a pair of contributions of 1 cancel each other. Thus, whenever the number of these contributions is even, their total difference is summed up to 0, and otherwise a difference of 1 remains, which means that in this case the equation f is not satisfied by the assignment τ.

Motivating Discussion: Before explicitly proving the upper bound on the number of satisfied equations in I′, which has a somewhat technical proof, we first give some intuition for the main ideas in constructing it, via proving such an upper bound for slightly different construction of I′. Consider the construction of I′, which instead of sampling m elements of {0, 1}^{n}, takes the whole {0, 1}^{n} as Ω (note that such an instance I′ has exponential size and cannot be constructed in polynomial time from I), which is a perfect sample space. Let τ be an assignment, and recall that I is not satisfiable, and thus the set K of equations that are not satisfied by τ is non-empty.

Recall that the uniform distribution is an 0-bias sample space (as described in Section 8.2), which means that for every non-empty K, the fraction of n-bit vectors, whose ⨁ over K equals 0, is exactly one half. Thus, we can apply Fact 8.3.6 and conclude that the fraction of s’s that induce equations unsatisfied by τ, is exactly one half.

We return now to the original construction of I′. Although our sample space has only polynomial size (m = poly(n)), recall that the main property of an ϵ-bias sample space is that the parity bit of every non-empty subset, has an almost uniform distribution in Ω (it has at most a bias of \frac{ϵ}{2} towards each of the values in {0, 1}). Consequently, the fraction of linear combinations where the number of unsatisfied e_{k}’s is even, is bounded from above by \frac{1}{2} + \frac{ϵ}{2} (and from below by \frac{1}{2} − \frac{ϵ}{2}) (details follow).

Let τ be an assignment, and suppose that τ does not satisfy some of the equations e_{1}, …, e_{n} in I (as must be the case since I ∉ QE). Let K ⊆ {1, …, n} be the set of indices of equations in I that are not satisfied by τ. We know that K is not empty as I is not satisfiable. An equation f_{l} in I′ is the linear combination of the equations {e_{k}}_{k=1}^{n}, with coefficients corresponding to the element s^{(l)} in Ω. Again we use the observation of Fact 8.3.6 that τ satisfies f_{l} if and only if ⨁_{k∈K} s_{k}^{(l)} = 0. However, the fraction of these s^{(l)}’s in Ω is bounded from above by \frac{1}{2} + \frac{ϵ}{2}. This fact stems from the definition of ϵ-bias sample spaces.

Thus, the fraction of f_{l}’s that are satisfied by τ is at most \frac{1}{2} + \frac{ϵ}{2}, which is strictly smaller than \frac{1}{2} + ϵ.

Thus, the reduction is polynomial and valid and the lemma follows.

Bibliographic Notes

Small-bias sample spaces were introduced and first constructed by Naor and Naor [35]. The construction presented in this lecture is one of three alternative constructions presented by Alon, Goldreich, Håstad and Peralta [8]. The application to hardness of MaxQE is due to Håstad, Phillips and Safra [25], who also demonstrate the tightness of the non-approximability factor (by presenting a suitable polynomial-time approximation algorithm). For an introduction to probabilistic checkable proofs (i.e., PCP), see [21, Sec. 2.4].

Expanders and Eigenvalues

Notes taken by Ya’ara Goldschmidt, Eran Keydar, Yaki Setty

Summary: This lecture is about expander graphs, which are very useful in many probabilistic applications. First, we would describe some of the combinatorial definitions and properties of an expander graph. Second, we would see one application which uses expander graphs, and finally, we would discuss some of the algebraic properties of such graphs.

9.1 Introduction

9.1.1 Motivation

We have already seen (in Lecture 2) a proof, using the probabilistic method, for the existence of 3-regular expander graphs. In the following lectures we study expanders, their applications and constructions, at greater depth. As was presented in Lecture 2, expander graphs are sparse graphs with the property that every subset of vertices (which is not too large), has “many” adjacent neighbors. This implies the existence of a “short” path between any two vertices. These properties are very useful in many probabilistic applications. For example they ensure that when performing a random walk on an expander graph, the distribution of reaching any of the vertices will converge to uniformity, relatively fast.

9.1.2 Family of Expander Graphs

Usually, when speaking on an expander graph G_{N}, we actually mean a family of graphs {G_{N}}_{N∈S}, where S is some infinite subset of the natural numbers. The members of the family are d-regular graphs (d is fixed for the entire family) and have the same expansion coefficient c > 0. Three remarks for the definition of the set of graphs:

1. The graphs are not necessarily simple; that is, self-loops and parallel edges are allowed. That means that the adjacency matrix representing such a graph can have entries in the range 0 to d instead of only boolean entries.

2. In fact, we assume that all vertices have self-loops; that is, the adjacency matrix has 1’s on the diagonal.

3. For our purpose, it is convenient to assume that N is equal to 2^{n} for some n, and that the vertices of the graph are the 2^{n} strings of length n (i.e., {0, 1}^{n}).

9.1.3 Definitions of Neighborhood Sets

Before formally defining what an expander graph is, we define some notations about neighborhood sets. For a set of vertices U in graph G(V, E), we define the following three neighborhood sets:

Definition 9.1 Γ(U) =^{△} {v : ∃u ∈ U , (u, v) ∈ E}; the set of all vertices reachable from any vertex of U in exactly one step (may include vertices in U).

Definition 9.2 Γ^{∗}(U) =^{△} Γ(U)\U; the set of all vertices reachable in one step from vertices in U, excluding U. This is called the boundary of U.

Definition 9.3 Γ^{+}(U) =^{△} Γ(U) ∪ U; the set of all vertices reachable in at most one step from U.

Notice: under the assumption that all vertices have a self-loop, Γ^{+}(U) = Γ(U).

9.1.4 Definitions of Expander Graphs

We can now formally define an expander graph.

Definition 9.4 [First Definition] N-vertex graph has expansion coefficient c > 0, if ∀ set U of vertices with size ≤ \frac{N}{2} : |Γ^{∗}(U)| ≥ c · |U|

Sometimes one would find it more convenient to use the following definition for an expander graph:

Definition 9.5 [Second Definition] N-vertex graph has expansion c > 0, if ∀ set U of vertices: |Γ^{+}(U)| ≥ min \{(1 + c)|U|, \frac{N}{2} + 1\}

Notice: any graph G satisfying Definition 9.4 also satisfies Definition 9.5, but the converse is not necessarily true. Definition 9.5 implies that every expander graph is connected, as stated in the following proposition:

Proposition 9.1.1 Any expander graph is a connected graph.

Proof: Let G_{N}(V, E) be an expander graph with expansion coefficient c > 0. Suppose in contradiction that G_{N} is not connected. In that case, V is composed of at least 2 disjoint subsets of vertices, with no edges connecting them. Hence for any such subset U, Γ^{+}(U) = U. Let U_{min} be the subset of smallest size, clearly |U_{min}| ≤ \frac{N}{2} , hence |Γ^{+}(U_{min})| < min \{(1 + c)|U_{min}|, \frac{N}{2} + 1\}. This contradicts the second definition of expander graph (Definition 9.5).

9.1.5 Constructibility

When speaking on expander graphs, one should consider the time required to construct the graph. This is the requirement of constructibility. There are two natural versions for this requirement: a weak version and a strong one.

Definition 9.6 [Weak Version] An expander graph G_{N} is (weakly) constructible, if an explicit representation of it can be given in time poly(N).

An explicit representation means an adjacency matrix or a neighbors list. This definition is weak, since the construction is polynomial in the number of vertices. As mentioned before, in some applications we would like to relate to vertices as strings over {0, 1}^{n}, and then the output would be exponential in n. This leads to the stronger definition of constructibility:

Definition 9.7 [Strong Version] An expander graph G_{N} is (strongly) constructible, if when given a string of length n bits, representing vertex v in G_{N} (N = 2^{n}), we can construct a list of all its neighbors in time poly(n).

Clearly, Definition 9.7 implies Definition 9.6.

9.2 On the Diameter of Expander Graphs

In this section we prove that there is a relatively short path between any two vertices in an expander graph.

Definition 9.8 The diameter of a connected graph G, denoted Diam(G), is the maximal distance between any two vertices in the graph.

Theorem 9.9 [Logarithmic Diameter] Let G_{N} be an expander graph with an expansion coefficient c, then:

Diam(G_{N}) ≤ \frac{2 log(N)}{log(1 + c)}

Proof: In order to prove the theorem, we would prove that for every two vertices u and v, there exists a path connecting u and v of length at most \frac{2 log((N/2)+1)}{log(1+c)}. Define l = log_{1+c}(\frac{N}{2} + 1), and define the following sets inductively:

U_{0} = {u}
U_{i} = Γ^{+}(U_{i−1})

Then, U_{i} is the set of the vertices reachable from u in at most i steps. Using the second definition of expansion (Definition 9.5) we conclude:

|U_{i}| = |Γ^{+}(U_{i−1})|
≥ min \{(1 + c)|U_{i−1}|, \frac{N}{2} + 1\}
≥ min \{(1 + c)^{i}, \frac{N}{2} + 1\}

Using i = l, we get |U_{l}| > min \{(1 + c)^{l}, \frac{N}{2} + 1\} , which is at least \frac{N}{2} + 1 provided (1 + c)^{l} > \frac{N}{2} . The latter is correct by the definition of l. Applying the same argument for v we get |V_{l}| > \frac{N}{2} . Remember that U_{l} and V_{l} are the sets of vertices reachable in at most l steps from u and v, respectively. Both U_{l} and V_{l} are subsets of the set of N vertices, therefore these two sets must have non-empty intersection. Therefore there must be a path between u and v of length at most:

2l = 2 · \frac{log(\frac{N}{2} + 1)}{log(1 + c)} ≤ \frac{2 log(N)}{log(1 + c)}

9.3 Amplification of Expander Graphs

For some applications we would need an expansion coefficient that is larger than the one associated with the {G_{N}}_{N∈S} which we are given or have been constructed. To achieve this we amplify the graph G_{N}, to get a graph with larger expansion coefficient, but also with more edges.

9.3.1 The Amplified Graph

Given the graph G_{N} with vertex set [N], we can construct the graph G_{N}^{k} in the following way:

G_{N}^{k} =^{△} ([N], \{(u, v) : \begin{matrix}∃ path of length exactly k \\ between u and v in G_{N} \end{matrix} \})

Given a representation of G_{N}^{k} as an adjacency matrix - M_{N}, the adjacency matrix M_{N}^{k} of G_{N}^{k} can be calculated by raising M_{N} to the k-th power: M_{N}^{k} = (M_{N})^{k}. We can now present the following proposition about G_{N}^{k} constructed from an expander graph G_{N} which is d-regular and has expansion c.

Proposition 9.3.1 Let G_{N} be a d-regular expander graph with expansion coefficient c, and let k be an integer. Then G_{N}^{k} has the following properties:

1. G_{N}^{k} is d^{k}-regular.

2. The expansion coefficient of G_{N}^{k} is (1 + c)^{k} − 1.

Proof:

1. By the construction of G_{N}^{k}, the degree of each vertex in G_{N}^{k} is the number of paths (not necessarily simple) of length k in G_{N}. Since G_{N} is d-regular, the number of paths is exactly d^{k}. Hence G_{N}^{k} is d^{k}-regular.

2. For any set U, we denote by U_{i} the set of vertices reachable from U in at most i steps in the graph G_{N}. Then U_{0} = U, U_{1} = Γ^{+}(U), and in general U_{i} = Γ^{+}(U_{i−1}).

|U_{0}| = |U|
|U_{i}| ≥ min \{(1 + c)|U_{i−1}|, \frac{N}{2} + 1\}
≥ min \{(1 + c)^{i}|U|, \frac{N}{2} + 1\}

In G_{N}^{k} the set Γ_{G_{N}^{k}} (U) is the set of all vertices reachable from U in exactly one step, that is, it is equal to the set of vertices reachable in exactly k steps from U in G_{N}. But since every vertex in G_{N} has a self-loop, Γ_{G_{N}^{k}} (U) also equals to the set of vertices reachable from U in at most k steps. This also implies that U is a subset of Γ_{G_{N}^{k}} (U), hence Γ_{G_{N}^{k}}^{+} (U) = Γ_{G_{N}^{k}} (U) = U_{k}.

Therefore |Γ_{G_{N}^{k}}^{+} (U)| ≥ min \{(1 + c)^{k}|U|, \frac{N}{2} + 1\}. Using the second definition of expansion,

the expansion coefficient of G_{N}^{k} is (1 + c)^{k} − 1.

9.3.2 Constructibility of G_{N}^{k}

We now present two claims about the constructibility of G_{N}^{k}:

Claim 9.3.2 Let G_{N} be a d-regular expander graph which is weakly constructibility, and let k be an integer, then G_{N}^{k} is weakly constructible as well.

Proof: We notice that if we take k to be as large as the diameter of the graph, then G_{N}^{k} is a complete graph, therefore we can assume that k is not larger than the diameter of the graph. Based on Theorem 9.9, k = O(log(N)). If G_{N} is weakly constructible then we can construct its adjacency matrix in time poly(N). Raising it to the k-th power, requires at most k times poly(N)- time operations. Since we assumed k = O(log(N)) the whole construction takes poly(N) time.

Claim 9.3.3 Let G_{N} be a d-regular expander graph which is strongly constructible, and let k be an integer. Then the generation of the list of neighbors of each vertex in G_{N}^{k} takes poly(n · |output|); that is, poly(n · d^{k}).

Notice that since k may not be constant, the number of neighbors of a vertex in G_{N}^{k} can be exponential in n, therefore G_{N}^{k} is not necessarily strongly constructible. Proof: If G_{N} is strongly constructible, then the generation of the neighbors list of v in G_{N}^{k} can be done by using a BFS search on G_{N} for depth k. The number of vertices in depth till k is upper bounded by ∑_{i=1}^{k} d^{i}, that is O(d^{k}). Since finding the neighbors of a vertex in G_{N} takes poly(n) time, the complexity of the BFS is poly(n · d^{k}). And this is the most expensive operation of the construction. Notice that if d^{k} is polynomial in n, then G_{N}^{k} is strongly constructible.

9.4 An Application of Expander Graphs

The following section presents one application of expander graphs.

9.4.1 The Problem

Consider the following problem: Suppose W is a set of witnesses, W ⊆ \{0, 1\}^{n} such that |W | ≥ \frac{1}{2} ·2^{n}. Our aim is to present a randomized algorithm that when given δ < \frac{1}{2} (e.g. δ = 0.01) satisfies:

Pr[Output of the Alg ∈ W] > 1 − δ

An obvious solution would be to pick log (\frac{1}{δ}) strings independently (each giving a probability of at least \frac{1}{2} to hit W). But we seek for a solution which would satisfy the following restrictions:

1. Running time should be poly(\frac{n}{δ})

2. At most n bits of randomness are allowed to be used.

The second restriction prevents us from using the above (obvious) solution, and even does not allow to construct a pairwise independent sample space.

9.4.2 The Algorithm

We start with a d-regular expander graph G_{N} with expansion c, we then construct the graph G_{N}^{l} (as explained in Section 9.3), which is d^{l}-regular and has expansion coefficient (1 + c)^{l} − 1. By choosing l to be \frac{log(1/δ)}{log(1+c)}, we ensure that the expansion coefficient of G_{N}^{l} is approximately \frac{1}{δ}. Using this construction, consider the following algorithm:

1. Select at random a vertex v in G_{N}^{l}.

2. Scan the neighbors of v, and output a neighbor in W if such exists, else fail.

Three remarks:

1. Each vertex is represented as a string of length n bits, thus only n bits of randomness are required for Stage 1.

2. We actually do not explicitly construct G_{N}^{k}. Instead, after randomly selecting the string v (in Stage 1), we use the fact that G_{N} is strongly constructible to generate the neighbors of v in G_{N}^{k} in time poly(\frac{n}{δ}).

3. Since the graph has self-loops, in Stage 2 of the algorithm v is scanned along with its neighbors.

9.4.3 Analysis of the Algorithm

Time Complexity

Using Claim 9.3.3, the time complexity of Stage 2 of the algorithm is poly(n · d^{l}), where:

d^{l} = d^{O(log(\frac{1}{δ}))} = exp (log (\frac{1}{δ})) = poly (\frac{1}{δ})

provided that d is constant. That is, the complexity of Stage 2 is poly(\frac{n}{δ}). This is the most expensive operation of the algorithm, hence this determines its time complexity.

Correctness

We now analyze the probability of the algorithm to fail, which happens when none of the neighbors of the chosen v are in W. We define a vertex as bad if all its neighbors (and itself) are in {0, 1}^{n}\W. Denote B = {v : v is bad}.

Claim 9.4.1 |B| < δ · 2^{n}

Proof: Suppose in contradiction that |B| ≥ δ·2^{n}(= δ·N). Using the second definition of expansion on Γ^{+}(B) we get:

|Γ^{+}(B)| = min \{\underbrace{((1 + c)^{l} − 1)}_{> \frac{1}{δ}} ·δN, \frac{N}{2} + 1\} = \frac{N}{2} + 1

Hence |Γ^{+}(B)| ≥ \frac{N}{2} , but also Γ^{+}(B)∩W = ϕ. This contradicts the hypothesis that |W | > \frac{N}{2} .

9.5 Algebraic Definition of Expanders

Until now we have been discussing the combinatorial properties of expanders. In the following section we will talk about some algebraic properties of expander graphs. This will lead us, in the next lecture, to the algebraic definition of an expander graph and to the relations between the combinatorial and algebraic definitions. We will study the algebraic properties by exploring the connection they have with a random walk on the graph. A random walk is a sequence of random steps, at which we start from an initial vertex and uniformly select one of its outgoing edges. We follow that edge and repeat the random selection for the vertex we have reached, and so on for any desired number of steps. Since the selection of an outgoing edge in each step is uniform, the probability to reach each vertex in that step is proportional to the number of edges connecting the current vertex to it.

9.5.1 The Normalized Adjacency Matrix

Consider the symmetric adjacency matrix of an undirected d-regular graph with vertex set [N]. The entries in the matrix are between 0 and d (since we allow self-loops and parallel edges). We define the normalized adjacency matrix to be the above matrix divided by d. This devision ensures that all the entries are in [0, 1], and the sum of each column and each row is 1. The normalized adjacency matrix is a stochastic matrix, and as such it can be used to describe a random walk on the graph. Denote the normalized adjacency matrix by A, then entry a_{ij} is the probability to reach vertex i in one random step, given that we are now in vertex j. We denote this probability by Pr[ i | j]. We can now look at the following operation:

\overbrace{[ \begin{matrix}a_{11} & … & a_{1N} \\ ⋮ & ... & ⋮ \\ a_{N1} & … & a_{NN} \end{matrix} ]}^{A:N×N} \underbrace{[ p_{1} ; ⋮ ; p_{N} ]}_{P:N×1} = \underbrace{[ p′_{1} ; ⋮ ; p′_{N} ]}_{P′:N×1}

We claim that the above operation describes one step of a random walk on the graph: That is, if P is the probability distribution of being in any vertex at the current stage, then P′ is the distribution after doing one random step. To explain this claim we look at the i-th entry of P′ which is:

\begin{aligned}p′_{i} & = & a_{i1} · p_{1} & + & a_{i2} · p_{2} & + & … & + & a_{iN} · p_{N} \\ & = & Pr[ i | 1 ] · p_{1} & + & Pr[ i | 2 ] · p_{2} & + & … & + & Pr[ i | N ] · p_{N} \end{aligned}

That is, p′_{i} describes the probability to reach vertex i after that step, so P′ is the probability distribution after that step. Basically, if we start with initial distribution vector P^{(0)}, then the distribution after one random step is P^{(1)} = AP^{(0)}. After another random step the distribution is P^{(2)} = AP^{(1)} = A^{2}P^{(0)}. In general, the distribution after t random steps would be P^{(t)} = A^{t}P^{(0)}.

We call a distribution P a stationary distribution if AP = P. Clearly, if for some t > 0 it holds that A^{t}P^{(0)} = P^{(t)} is the stationary distribution, then for any t′ ≥ t, the distribution P^{(t′)} remains the stationary distribution. We say that a random walk converges if for any initial distribution P^{(0)}, and for every ϵ > 0 there exists t > 0 such that A^{t}P^{(0)} = P^{(t)} is ϵ-close (in variation distnace) to the stationary distribution. As a corollary to the facts proven in the next section, one may conclude that for any connected bipartite (regular) graph a random walk on its vertices converges to the uniform distribution. In the next lecture, we shall see that the rate of convergence (i.e., the dependence of t on ϵ) depends on the eigenvalues of the adjacency matrix.

9.5.2 Eigenvectors and Eigenvalues of the Matrix

First, we would explore the eigenvalues and eigenvectors of the (normalized) adjacency matrix:

Fact 9.5.1 All eigenvalues are real.

This follows from the fact that the adjacency matrix is symmetric.

Fact 9.5.2 The eigenvectors form a basis.

This again follows from the fact that the adjacency matrix is symmetric and therefore diagonalizable.

Fact 9.5.3 The value 1 is an eigenvalue corresponding to the uniform eigenvector ē_{1} = (\frac{1}{N} , …, \frac{1}{N}).

That is: A · ē_{1} = ē_{1}. Proof: Indeed examine p̄ = A · ē_{1}. For any entry 1 ≤ i ≤ N in p̄:

p_{i} = ∑_{j=1}^{N} a_{ij} \frac{1}{N} = \frac{1}{N} ∑_{j=1}^{N} a_{ij} = \frac{1}{N} · 1 = \frac{1}{N}

Remark : ē_{1} is the stationary distribution.

Fact 9.5.4 Eigenvectors corresponding to distinct eigenvalues are orthogonal. That is, if Av̄_{1} = λ_{1}v̄_{1}, Av̄_{2} = λ_{2}v̄_{2} and λ_{1} ≠ λ_{2}, then v̄_{1}^{⊤}v̄_{2} = 0 (i.e. v̄_{1} ⊥ v̄_{2}).

Proof: Examine v̄_{1}^{⊤}Av̄_{2} On the one hand:

v̄_{1}^{⊤}Av̄_{2} = (A^{⊤}v̄_{1})^{⊤}v̄_{2} =^{(∗)} (Av̄_{1})^{⊤}v̄_{2} = (λ_{1}v̄_{1})^{⊤}v̄_{2} = λ_{1}v̄_{1}^{⊤}v̄_{2}

where (∗) holds since A^{⊤} = A. On the other hand:

v̄_{1}^{⊤}Av̄_{2} = v̄_{1}^{⊤}λ_{2}v̄_{2} = λ_{2}v̄_{1}^{⊤}v̄_{2}

Hence λ_{1}v̄_{1}^{⊤}v̄_{2} = λ_{2}v̄_{1}^{⊤}v̄_{2}. Thus λ_{1} ≠ λ_{2} implies v̄_{1}^{⊤}v̄_{2} = 0.

Fact 9.5.5 Any eigenvector v̄ corresponding to an eigenvalue ≠ 1 has sum 0.

Proof: This is deduced from Fact 9.5.4. Since v̄ is orthogonal to the uniform vector v̄^{⊤}· [ \frac{1}{N} ; ⋮ ; \frac{1}{N} ]

= 0. This implies that ∑_{i=1}^{N} v_{i} \frac{1}{N} = 0, therefore \frac{1}{N} ∑_{i=1}^{N} v_{i} = 0, and so ∑_{i=1}^{N} v_{i} = 0.

Fact 9.5.6 For any eigenvalue λ, |λ| ≤ 1.

Proof: Let \left\{{ \atop ē_{i}}\right\} be the eigenvectors, where ē_{1} = (\frac{1}{N} , …, \frac{1}{N}). Any probability vector p̄ can be represented as a linear combination of the basis {e_{i}} (Fact 9.5.2): p̄ = ∑_{i=1}^{N} c_{i}ē_{i}. Let t ≥ 1 be an integer.

A^{t}p̄ = A^{t} ∑_{i=1}^{N} c_{i}ē_{i} = ∑_{i=1}^{N} c_{i}A^{t}ē_{i} = ∑_{i=1}^{N} c_{i}λ_{i}^{t}ē_{i}

If λ_{i} > 1 and c_{i} ≠ 0 then the above sum diverges, which is not possible since A^{t}p̄ is a probability vector. However if c_{i} = 0, then we cannot conclude that |λ_{i}| ≤ 1. Thus, for any i ≥ 2, consider p̄ = ē_{1} +ϵē_{i}, where ϵ is small enough such that p̄ is a probability vector. That is:

ϵ ≤ \frac{1}{|m|N} where m = min_{1≤j≤N}\{e_{i_{j}}\}

Taking such ϵ ensures that p̄ is a probability vector:

• All entries of p̄ are non-negative: Since m is the minimal entry of ē_{i}, all entries in ϵē_{i} are greater or equal to −\frac{1}{N}. Moreover, all entries in ē_{1} are \frac{1}{N}, therefore all the entries in p̄ are non-negative.

• The sum of the entries of p̄ is 1: The sum of entries in ē_{1} is 1, and the sum of entries in ē_{i} is 0 (Fact 9.5.5), hence the sum of entries in ē_{1} + ϵē_{i} is 1.

Now consider:

p̄′ = A^{t}p̄ = A^{t}(ē_{1} + ϵē_{i}) = A^{t}ē_{1} + A^{t}ϵē_{i} = ē_{1} + ϵλ_{i}^{t}ē_{i}

p̄′ should also be a probability vector, but when |λ_{i}| > 1 and for t large enough, the above term diverges. Hence, we conclude that |λ_{i}| ≤ 1

Fact 9.5.7 If G is connected, then the eigenvalue 1 has a unique eigenvector.

Proof: Suppose in contradiction, that ē_{1} and ē_{2} are two orthogonal eigenvectors corresponding to the eigenvalue 1. Let p̄ = ē_{1} + ϵē_{2}, where ϵ = \frac{1}{|m|N} and m is the minimal entry of ē_{2}. Such ϵ ensures that for any entry with value m in e_{2}, the corresponding entry in p̄ is 0 (see the proof of Fact 9.5.6). That means that p̄ has some zero entries and the rest are positive entries. Since λ_{1} = λ_{2} = 1, the following holds:

Ap̄ = A(ē_{1} + ϵē_{2}) = Aē_{1} + ϵAē_{2} = 1 · ē_{1} + ϵ · 1 · ē_{2} = ē_{1} + ϵē_{2} = p̄

Therefore for any positive t, A^{t}p̄ = p̄. Let i be an index of a positive entry, and let j be an index of a zero entry. Thus, the i-th entry in p̄ is positive, whereas the j-th entry in A^{t}p̄ is zero. It follows that, for any integer t > 0, the probability to reach from vertex i to vertex j in t steps is 0 (since (A^{t}p̄)_{j} is the sum of the probability of reaching j in t steps, when starting in vertex k with probability p_{k}). This implies that for any t > 0, there is no path of length t between i and j which implies that the graph is not connected, thus we get contradiction.

Fact 9.5.8 For a connected aperiodic graph G, −1 is not an eigenvalue.

Definition 9.10 A graph G is said to be aperiodic if the GCD of its cycle (not necessarily simple cycles) lengths is 1.

We first notice that the GCD of all cycle lengths is at most 2. If the GCD is 2, then the graph must be bipartite with no self loops. In this case all cycles would be of even length. However, since the graph is aperiodic its GCD is 1, hence there must exist an odd cycle on the graph.

Proof: Suppose in contradiction that ē_{2} is an eigenvector with a corresponding eigenvalue −1. Let p̄ = ē_{1} + ϵē_{2}, where ϵ = \frac{1}{|m|N} (as in the proof of Fact 9.5.7). Let u be an index of a zero entry in p̄, hence negative in ē_{2}. First we claim that since there exists an odd cycle in the graph, then there is an odd cycle going through u. Such cycle can be constructed by combining the path from u to the odd cycle, the cycle itself, and the path back to u. (The path from u to the odd cycle exists because the graph is connected.) Let l be the length of such an odd cycle. Consider A^{l+1}p̄: On the one hand:

A^{l+1}p̄ = A^{l+1}(ē_{1} + ϵē_{2}) = A^{l+1}ē_{1} + A^{l+1}ϵē_{2} = λ_{1}^{l+1}ē_{1} + ϵλ_{2}^{l+1}ē_{2} = ē_{1} + (−1)^{l+1}ϵē_{2} = p̄.

In particular, the u-th entry of A^{l+1}p̄ is 0 (just as this entry is in p̄). On the other hand:

A^{l+1}p̄ = A^{l}(Ap̄) = A^{l}(λ_{1}ē_{1} + ϵλ_{2}ē_{2}) = A^{l}(ē_{1} − ϵē_{2}).

Since the u-th entry of ē_{2} is negative, the u-th entry of (ē_{1} − ϵē_{2}) is positive. Since u is a vertex on an odd cycle with length l, there must be a positive probability to reach u in l steps when starting from u. Hence the u-th entry of A^{l}(ē_{1} − ϵē_{2}) = A^{l+1}p̄ must be positive. This contradicts the previous conclusion that the u-th entry equals 0.

In particular, this fact holds for expanders, since the existence of self-loops implies they are aperiodic.

Bibliographic Notes

The elementary facts regarding the eigenvalues of a symmetric matrix can be found in many standard textbooks on linear algebra. The properties of eigenvalues and eivenvectors of matrices associated with graphs were investigated in the combinatorics literature; see [9] and the references therein.

The relation of the algebraic definition (i.e., the second eigenvalue of the normalized adjacency matrix of a graph) and the combinatorial definition of expansion is further discussed in the next lecture (and the most relevant references are [9, 5]).

Random Walks on Expanders

Notes taken by Omer Angel, Dani Halevi, Amos Gilboa

Summary: The purpose of this lecture is to show that taking an l step random walk in an expander graph is in a way similar to choosing l vertices at random. The advantage of using a random walk on an expander over uniform independent sampling is that it requires only log_{2} d random bits for each step, whereas when you choose l different vertices at random each vertex requires log_{2} N random bits. As a warm up, we’ll show that in O(log N) random steps on an N-vertex expander beginning at any vertex we’ll reach a roughly uniform distribution on the vertices.

We also recall the relation between two definitions of expanders and describe some known constructions of expanders.

10.1 Two Definitions

Recall the combinatorial and algebraic definitions of expansion:

Definition 10.1 A sequence of graphs {G_{N}} has combinatorial expansion C if any of vertices S with |S| < |V |/2 has |Γ^{−}(S)| > C|S| where Γ^{−}(S) denotes the neighbours of a set S that are not in S.

Definition 10.2 A sequence of graphs {G_{N}} has algebraic expansion with gap ϵ > 0 if for all n the second eigenvalue of G_{n} satisfies |λ_{2}| ≤ 1 − ϵ.

remark. We assume that the eigenvalues are ordered by non-increasing order, i.e.

1 = λ_{1} > |λ_{2}| ≥ … |λ_{n}|

(Recall that we proved that for the normalized adjacency matrix of any connected regular and non-bipartite graph λ_{1} = 1 and |λ_{i}| < 1 for all i > 1.)

The following theorem implies that the combinatorial definition of expansion based on boundary sizes and the algebraic definition based on eigenvalue separation are equivalent. In fact it gives a quantitative translation between the two notions. (This Theorem is not proved here).

Theorem 10.3 Algebraic expansion ⟺ Combinatorial expansion. In particular:

• If {G_{N}} are expanders w.r.t. the algebraic definition with constant ϵ then they are expanders w.r.t. the combinatorial definition with constant C ≥ \frac{2ϵ}{1+2ϵ}.

• If {G_{N}} are d-regular expanders w.r.t. the combinatorially definition with constant C then they are expanders w.r.t the algebraic definition with ϵ ≥ \frac{C^{2}}{(4+2C^{2})d}.

Note. These inequalities are not tight. One can see it by beginning with expansion parameter C, substituting in the second inequality to derive ϵ, and substituting this ϵ in the first inequality to derive

C′ ≥ \frac{C^{2}}{C^{2} + (2 + C^{2})d}

Since this way we got C′ < C, it follows that at least one of the inequalities is not always tight.

Using the second inequality, we have a practical and easy way to conclude that a graph does not have an expansion C, just calculate ϵ and check whether ϵ ≥ \frac{C^{2}}{(4+2C^{2})d} , if this is not the case - obviously the graph does not have expansion C. On the other hand if ϵ ≤ \frac{C}{2−2C} then the graph certainly has expansion C. However, as we just observed the inequalities are not tight, so if we find that ϵ is in the interval [\frac{C^{2}}{(4+2C^{2})d}, \frac{C}{2−2C}] then we can not determine from the theorem whether or not the graph has expansion C. Thus this theorem is more important as a qualitative statement.

Just as there is a relation between the combinatorial expansion of a graph G and the expansion of its power G^{k} (i.e. given by 1 + C_{G^{k}} > (1 + C_{G})^{k}) there is also a similar relation between the algebraic expansions of G and G^{k}. This is given by:

Proposition 10.1.1 λ_{2}(G^{k}) = λ_{2}(G)^{k}

Proof: The normalized adjacency matrix of G^{k} is given by A′ = A^{k} (i.e. A raised to power k). If v is an eigenvector of A with eigenvalue λ then

A′v = A^{k}v = λ^{k}v

So the eigenvalues of G′ are the k’th powers of the eigenvalues of G. Since A has a basis of eigenvectors there are no addition eigenvalues of A′.

10.2 Known Constructions of Expanders

Following are two known ways to build big expander graphs. These graphs are strongly constructible (i.e. given a vertex, we can find it’s neighbours in time poly(n), where n is the number of bits needed to represent a vertex).

• For any N such that N = m^{2} for some integer m, we construct an expander in the following way: the set of vertices will be the set of all pairs (x, y) with x, y ∈ {0, …, m − 1}. The neighbours of a vertex (x, y) are the pairs

{(x, y ± (2x + σ)) : σ ∈ {0, 1}} ∪ {(x ± (2y + σ), y) : σ ∈ {0, 1}}

where operations are modulo m. Each vertex can be represented in 2 log m bits its neighbours can be found in time poly(log m).

This graph is 8-regular and has λ ≈ 0.9.

• For primes p, q primes such that p, q ≡ 1 (mod 4) and (\frac{p}{q}) = 1 and for any integer k there

is an N-vertex expander with N = \frac{q^{k}(q^{2k}−1)}{2}. The vertices are 2 × 2 matrices over the field with q^{k} elements (GF (q^{k})). This graph is (p + 1)-regular and has λ ≈ \frac{2}{\sqrt{d}}.

These graphs are less convenient to work with, because of the constraints on p, q (and so on N). However, it is still easy to find the neighbours of any given vertex. They also have the advantage of having λ close to optimal, as seen from:

Theorem 10.4 For any d-regular graph λ_{2} ≥ \frac{2}{\sqrt{d}}

10.3 Random Walks on Expanders

Throughout this section λ will denote an upper bound on |λ_{2}|.

10.3.1 Mixing Time of Random Walks

Let X_{0}, X_{1}, …, X_{l} be a random walk on an expander, starting at X_{0} (i.e. X_{0} is given either deterministically or by some distribution, and X_{i+1} is chosen uniformly at random from the neighbours of X_{i}). We show that for l = O(log N) the distribution of X_{l} tends to the uniform distribution on V, regardless of the starting point (or its distribution). In a way this number of steps is optimal. This is due to the fact that the diameter of an expander graph is logarithmic in N (because the graph has constant degree). Thus we can’t expect to get from one vertex to some other vertices in less than O(log n) steps.

Theorem 10.5 Let X_{0}, X_{1}, …, X_{l} be a random walk on an expander graph starting at X_{0}. For every δ there is an l = O(log \frac{1}{δ}) such that for every vertex v

|Pr(X_{l} = v) − \frac{1}{N} | < δ

Note. The important fact here is not the dependence on δ which is the same for every graph (not just expanders). The difference between expanders and other graphs is that the constant implied by the O() notation does not depend on the size of the graph. As the proof will show, the dependence of l on the second eigenvalue λ_{2} is l = \frac{log 1/δ}{log 1/λ_{2}} . If the bound on the eigenvalues λ = 1 − ϵ is close to 1 then we get

l ≈ \frac{log 1/δ}{log 1/(1 − ϵ)} ≈ \frac{log 1/δ}{ϵ}

For a fixed δ, the same l will work for all graphs with a given bound λ. Typically we will want to use δ ≤ 1/2N. For that value of δ we have

\frac{1}{2N} < Pr(X_{l} = v) < \frac{3}{2N}

So that the probability of being at any vertex is within a constant factor of the uniform distribution. For δ = 1/2N, we need to have l = O(log N), so we get:

Corollary 10.6 If {G_{N}} are expander graphs then after l = O(log N) random steps we are within constant factor of the uniform distribution.

Proof: [Proof of Theorem 10.5] Let X be a random vertex in G with probability vector v (i.e. v_{i} is the probability that X = i). Let Y be a uniformly chosen neighbour of X. We claim that the probability vector of Y is given by Av where A is the normalized adjacency matrix of G. To see this write the Bayesian equation:

Pr(Y = i) = ∑_{j} Pr(Y = i|X = j)Pr(X = j)
= ∑_{j} A_{ij}v_{j}
= (Av)_{i}

Applying this recursively we see that if the distribution of X_{0} is given by a probability vector v then the distribution of X_{l} is given by A^{l}v.

Next we write v as the sum of the uniform probability vector π = (N^{−1}, …, N^{−1})^{T} and another vector v_{⊥} = v − π. Since the sum of the coordinates in v is 1 and the same is true of π we see that the sum of the coordinates of v_{⊥} is 0. This shows that v_{⊥} is perpendicular to π justifying the notation v_{⊥}.

A is a real symmetric matrix and as such has perpendicular eigenvectors. Thus v_{⊥} is spanned by the remaining eigenvectors of A (other then π). We can now estimate the difference in probabilities:

∥A^{l}v − π∥_{∞} ≤ ∥A^{l}v − π∥_{2}
= ∥A^{l}(v_{⊥} + π) − π∥_{2}
= ∥A^{l}v_{⊥} + A^{l}π − π∥_{2}
= ∥A^{l}v_{⊥}∥_{2}

Here we used the fact that π is an eigenvector of A with eigenvalue 1. Since all eigenvalues of A except 1 are bounded by λ the norm of A as an operator on the space perpendicular to π is |λ_{2}|. Since v_{⊥} is perpendicular to π we have:

∥A^{l}v_{⊥}∥_{2} ≤ λ^{l}∥v_{⊥}∥_{2}
≤ λ^{l}∥v∥_{2}
≤ λ^{l}∥v∥_{1}
= λ^{l}

since v_{⊥} is the projection of v on the space perpendicular to π it has smaller ∥ · ∥_{2} norm then v and v is a probability vector.

If λ^{l} < δ then we conclude that for every vertex |Pr(X_{l} = v)−N^{−1}| < δ. This happens whenever

l > \frac{log 1/δ}{log 1/λ} = O(log \frac{1}{δ})

It is interesting to note the combinatorial approach to estimating the mixing time of a random walk. The combinatorial definition of an expander says that there are no small cuts in the graph. Specifically it gives a relation between the size of a set and the number of edges leaving it. (The requirement that |S| < |V |/2 is not important since if S contains half the vertices we can remove one from S and find that S still has many neighbours.) There are estimates of the mixing time of the random walk on a graph in terms of the sizes of cuts in the graph and vice versa, but they will not be shown here. We will note, however, that absence of small cuts is a necessary requirement for fast mixing. If a graph is composed of two large vertex sets with a small number of edges (e.g., a single edge) between them then the random walk on the graph started in one set will take a long time to reach the other set.

10.3.2 A random walk yields a sequence of “good samples”

Consider the following problem: In a large set V there is a large subset G of good elements (satisfying some condition) and we wish to find one of them. By large we will mean that the size of G is a constant fraction of the size of V. Call the set of bad elements B and denote its density by ρ(B) = |B|/|V |. The set G (and B) are given by an oracle allowing us to check whether some x is in G or in B.

Deterministic algorithms have the drawback that they may take very long for some particularly bad sets B (It is possible to see most of V before finding a point in G). Some randomness must be used to avoid the possibility of terrible sets (at the cost of introducing a probability of error). The trivial random algorithm is to randomly choose elements in V and checking if they are in G. If we make l samples independently then the probability of not finding one in G is ρ(B)^{l}.

This is close to the optimal success probability. The optimal success probability is achieved by also ensuring that the samples are distinct, but since l ≪ |V | the improvement is negligible. The drawback here is that this requires the use of many (log |V |) random bits for each sample used.

Expanders can be used to improve this. The algorithm we consider is as follows: Using an expander G = (V, E), Generate l samples in V by taking X_{1} ∈_{R} V uniformly at random and X_{2}, …, X_{l} are a random walk on G started at X_{1}. Use the X_{i}’s as samples, while checking if X_{i} ∈ B. The algorithm is successful if not all X_{i} fall in B.

Since at each step we only need to choose randomly one of d neighbours (if G is d-regular) then the number of random bits needed is log |V | + (l − 1) log d. On the other hand we will see that the probability of success is comparable to that of the trivial random algorithm (with the same number of vertex checked).

Theorem 10.7 Let G = (V, E) is d-regular with |λ_{2}| = λ < 1 and B ⊂ V has density ρ = \frac{|B|}{|V |}. Choose X_{1} ∈_{R} V uniformly at random and let X_{2}, …, X_{l} be a random walk on G starting at X_{1}.

Pr(∀i X_{i} ∈ B) ≤ \sqrt{ρ} · (\sqrt{ρ} + λ)^{l−1}

Note this estimate is not quite the same as for independent samples. First of all, even if λ is very small the bound is only ρ(B)^{l/2} compared to ρ(B)^{l} for independent samples. While this is indeed a difference, the most harm it can do is force us to use twice as many random steps as independent samples, hence the damage is at most multiplication by a constant of complexity and random bits needed. In addition, it is possible to prove the stronger estimate

Pr(∀i X_{i} ∈ B) ≤ ρ(B) · (ρ(B) + λ(1 − ρ(B))^{l−1}

which when λ is close to 0 is close to ρ(B)^{l}.

If λ is not very small then the bound is even worse. This drawback can be alleviated by reducing λ by using instead of G the power G^{k}. By taking k large enough we can make λ smaller. However, we don’t gain much by doing this since each random step on G^{k} takes as many random bits as k steps on G. Using G^{k} is equivalent to doing k times as many steps and only checking if X_{i} ∈ Good on every k’th step. Since we are taking these steps anyway, usually there is no loss in checking the intermediate vertices as well, i.e. using the random walk on G and not G^{k}.

One situation where using G^{k} rather then G may be useful is when the cost of checking whether or not some X is good or bad is not negligible. Then it may be better to take a few more random steps between checks to increase the independence of the positions checked.

Proof: Let P = P_{B} be a projection on the space of vectors supported in B, i.e. P is a matrix with P_{ij} = 1 if i = j ∈ B and 0 otherwise. If v is a distribution vector then P v is the residual distribution vector of the distribution v conditioned on being in the set B.

We claim that the probability we wish to estimate is given by: ∥(P A)^{l−1}P π∥_{1} where as before, π = (1/N, …, 1/N)^{T} is the uniform distribution. To see this, we need to understand the actions of A and P on probability vectors. The action of A is known: for a probability vector v, Av is the probability vector after a random step has been taken.

The action of P is to nullify all the coordinates outside the set B. This transforms a probability vector into the residual probability vector of the same distribution, but conditioned on being in B. Thus P π is the residual probability of the uniform distribution conditioned to be in B. AP π is the residual distribution after a random step has been takes. P AP π is the residual probability conditioned on the random step remaining in B. Repeating this we see that (P A)^{l−1}P π is the residual probability vector of the random initial point and all l − 1 steps being in B. Since we don’t care where in B we end up we need to sum the coordinates of this vector, hence the required probability is indeed given by ∥(P A)^{l−1}P π∥_{1}.

To estimate this we will use the following lemma:

Lemma 10.3.1 For any non negative vector v:

∥P Av∥_{2} ≤ (\sqrt{ρ} + λ) · ∥v∥_{2}

Given this lemma we proceed as follows: using the bound on the ratio between ∥v∥_{1} and ∥v∥_{2} and applying the lemma l times:

∥(P A)^{l−1}P π∥_{1} ≤ \sqrt{N} · ∥(P A)^{l−1}P π∥_{2}
≤ \sqrt{N} · (\sqrt{ρ} + λ)^{l−1}∥P π∥_{2}
= \sqrt{N} · (\sqrt{ρ} + λ)^{l−1}\sqrt{\frac{ρ}{N}}
= \sqrt{ρ} · (\sqrt{ρ} + λ)^{l−1}

Since ∥P π∥_{2} = \sqrt{|B|/N^{2}} = \sqrt{ρ/N}.

Proof: [Proof of Lemma 10.3.1] P. The idea of the proof is that A shrinks all components of a vector except the uniform distribution component, whereas P will shrinks the uniform component without increasing anything else. Together they reduce all parts of a vector.

To prove the above formally, break up v as before: v = v_{1} + v_{⊥} where v_{1} is a constant vector, i.e. a multiple of the uniform distribution vector, and v_{⊥} is orthogonal to the uniform distribution vector. In the rest of the proof all norms are Euclidean.

∥P Av∥ = ∥P Av_{1} + P Av_{⊥}∥ ≤ ∥P Av_{1}∥ + ∥P Av_{⊥}∥

Next we note how P A effects each part of v. Assume v_{1} = (a, a, …, a)^{T}. Then we have ∥v_{1}∥ = \sqrt{N}a. Since v_{1} is an eigenvector of A with eigenvalue 1:

∥P Av_{1}∥ = ∥P v_{1}∥ = \sqrt{|B| · a^{2}} = \sqrt{ρN} · a = \sqrt{ρ} · ∥v_{1}∥

Next we look at the effect on v_{⊥}. P ’s effect is to multiply some coordinates by 0 without changing the others, so P can only shrink a vector. Since v_{⊥} is perpendicular to the constant vector, it is spanned by the remaining eigenvectors of A, all with eigenvalues at most λ. This implies that

∥P Av_{⊥}∥ ≤ ∥Av_{⊥}∥ ≤ λ∥v_{⊥}∥

Combining the two inequalities together with

||v_{1}||, ||v_{⊥}|| ≤ ||v||

We see that

∥P Av∥ ≤ ∥P Av_{1}∥ + ∥P Av_{⊥}∥ ≤ \sqrt{ρ}∥v_{1}∥ + λ∥v_{⊥}∥ ≤ (\sqrt{ρ} + λ)∥v∥

as required.

Bibliographic Notes

Theorem 10.3 (relating the second eigenvalue and the combinatorial definition of expansion) is taken from Alon’s work (see [5, Thm. 2.5]): The lower bound on expansion in terms of the second eigenvalue is stated in [5, Cor. 2.3], which in turn relies on [9, Thm. 2.5]. The converse direction is stated in [5, Lem. 2.4].

The two explicite construction of expanders are due to Gaber and Galil [18] (following Margulis [33]) and Lubotzky, Phillips and Sarnak [31], respectively. Actually, the latter construction was originally presented (in [31]) only for primes (i.e., k = 1), and the extension to prime powers in due to Alon et. al. [7].

The hitting property of random walks on expanders was discovered by Ajtai, Komlos, and Szemerédi [3]. Various formulations of this discovery were given in [3, 17, 27, 22] culminating in Kahale’s optimal analysis [30, Sec. 6]. (A generalization to the case of hitting different sets is presented in [11].)

The fact that a random walk on an arbitrary regular graph reaches almost unifom distribution after polynomially many steps was established in [4].

Appendix

Oded’s Note: Adapted from an early version of [23].

We complement the material in this lecture by presenting two lemmas related to the properties of one random step taken on a d-regular expander.

Recall that when we talk of expanders we refer to families of d-regular graphs where a fixed upper bound holds for the absolute value of all eigenvalues (except the biggest one) of the corresponding adjacency matrix. Unlike the presentation in the lecture, here it is more convinient to consider the adjacency matrix itself (rather than a normalized (by division with d) form of it). Thus, the upper bound on the second eigen-value, denoted λ, is smaller than d (rather than being smaller than 1). (Similarly, the first eigenvalue is d (rather than being 1).)

The Expander Mixing Lemma

The following lemma is folklore and has appeared in many papers (cf. Corollary 2.5 in [10, Chap. 9]). Loosely speaking, the lemma asserts that expander graphs (for which d ≫ λ) have the property that the fraction of edges between two large sets of vertices approximately equals the product of the densities of these sets. This property is called mixing.

Expander Mixing Lemma: Let G = (V, E) be an expander graph of degree d and λ be an upper bound on the absolute value of all eigenvalues, except the biggest one, of the adjacency matrix of the graph. Then for every two subsets, A, B ⊆ V, it holds

|\frac{|(A × B) ∩ E|}{|E|} − \frac{|A|}{|V |} · \frac{|B|}{|V |}| ≤ \frac{λ\sqrt{|A| · |B|}}{d · |V |} < \frac{λ}{d}

Proof: Let A, B ⊆ V be two sets and denote N =^{def} |V |, ρ(A) =^{def} |A|/N and ρ(B) =^{def} |B|/N. Denote by M the adjacency matrix of the graph G, and let us denote it eigenvalues by λ_{1}, ..., λ_{N}, where |λ_{i}| ≥ |λ_{i+1}|. Note that λ_{1} = d, whereas, by the statement of the lemma, λ ≥ |λ_{2}|. Hence, the claim of the lemma is restated as

|\frac{|(A × B) ∩ E|}{d · N} − ρ(A) · ρ(B)| ≤ \frac{λ\sqrt{ρ(A) · ρ(B)}}{d}

We proceed by bounding the value of |(A × B) ∩ E| (from both directions). To this end we let ā denote the N-dimensional Boolean vector having 1 in the i^{th} component iff i ∈ A. The vector b̄ is defined similarly. Clearly, |(A × B) ∩ E| equals āMb̄^{⊤}. We consider the orthogonal eigenvector basis, ē_{1}, ..., ē_{N}, where ē_{i}ē_{i}^{⊤} = N for each i, and write each vector as a linear combination of the vectors in the basis, denoting by a_{i} the coefficient of ā in the direction of ē_{i} (i.e., ā = ∑_{i} a_{i}ē_{i}). Recall that ē_{1} is the constant vector (i.e., with 1’s in all entries), and all other ē_{i}’s are zero-sum vectors. One can easily verify the following two facts:

1. a_{1} = ρ(A).

This follows because the entries in ā sum-up to |A|, and so this must be the sum of the entries in a_{1}ē_{1} (since all other ē_{i}’s are zero-sum vectors).

2. ∑_{i=1}^{N} a_{i}^{2} = ρ(A).

This follows because, on one hand āā^{⊤} = ρ(A)N, and on the other hand āā^{⊤} = ∑_{i,j} a_{i}a_{j}ē_{i}ē_{j}^{⊤} = N · ∑_{i=1}^{N} a_{i}^{2}.

Similarly for b̄ (i.e., b̄ = ∑_{i} b_{i}e_{i}). It now follows that

|(A × B) ∩ E| = āMb̄^{⊤}
= āM(b_{1}ē_{1}^{⊤} + ∑_{i=2}^{N} b_{i}ē_{i}^{⊤})

= ρ(B)āMē_{1}^{⊤} + ∑_{i=2}^{N} b_{i}āMē_{i}^{⊤}

= ρ(B)dāē_{1}^{⊤} + ∑_{i=2}^{N} b_{i}λ_{i}ē_{i}^{⊤}

= d · ρ(B) · |A| + (∑_{j=1}^{N} a_{j}ē_{j}) (∑_{i=2}^{N} λ_{i}b_{i}e_{i}^{⊤})

= ρ(B)ρ(A) · dN + ∑_{i=2}^{N} λ_{i}a_{i}b_{i}N

∈ [ρ(B)ρ(A) · dN ± λ · N ∑_{i=2}^{N} a_{i}b_{i}]

Using ∑_{i=1}^{N} a_{i}^{2} = ρ(A) and ∑_{i=1}^{N} b_{i}^{2} = ρ(B), and applying Cauchy-Schwartz Inequality, we bound ∑_{i=2}^{N} a_{i}b_{i} by \sqrt{ρ(A)ρ(B)}. The lemma follows.

The Expander Smoothing Lemma

The following lemma follows easily by the standard techniques of dealing with random walks on expander graphs (cf. [3] and [10, Chap. 9]). Loosely speaking, the lemma refers to a process that starts with a distribution on the expander vertices and consists of taking one random step on the expander. The lemma bounds the variation distance of the end distribution from the uniform one, in terms of the initial distribution and the quality of the expander. Typically, the end distribution is much closer to the uniform distribution than the initial distribution. That is, a random step on the expander “smoothens” the distribution.

Expander Smoothing Lemma: Let G = (V, E), d and λ be as in the previous lemma. Let X be a random variable, distributed over V, so that Pr(X = v) ≤ \frac{K}{|V |}, for every v ∈ V, and Y denote the vertex reached from X by following a uniformly chosen edge. Then

∑_{v∈V} |Pr(Y =v) − \frac{1}{|V |}| < \frac{λ}{d} · \sqrt{K − 1}

Proof: Let N =^{def} |V |, and x denote the N-dimensional probability vector defined by X (i.e., x_{i} =^{def} Pr(X = i)). Let A denote the Markov process defined by traversing a uniformly selected edge in G; namely, the matrix A is the adjacency matrix of the graph G, normalized by division by d. Denote the eigenvalues of A by λ_{1}, ..., λ_{N}, and note that λ_{1} = 1 and |λ_{i}| ≤ \frac{λ}{d}, for every i > 1. We consider the orthogonal eigenvector basis, e_{1}, ..., e_{N}, where e_{i}e_{i}^{⊤} = \frac{1}{N} for each i, and e_{1} = (\frac{1}{N} , ..., \frac{1}{N}). Again, we write each vector as a linear combination of the vectors in this basis. Denote by c_{i} the coefficient of x in the direction of e_{i} (whereas the x_{i}’s are the coefficients in the Kroniker basis). We start by bounding ∑_{i} c_{i}^{2} as follows

∑_{i=1}^{N} c_{i}^{2} \frac{1}{N} = (∑_{i=1}^{N} c_{i}e_{i}^{⊤}) · (∑_{i=1}^{N} c_{i}e_{i}^{⊤})^{⊤}
= x · x^{⊤}
= ∑_{i=1}^{N} x_{i}^{2}

≤ max ; \begin{matrix}v_{1}, ..., v_{N} s.t. \\ ∀i v_{i} ∈ [0, (K/N)] \\ ∑_{i} v_{i} = 1 \end{matrix} \{∑_{i} v_{i}^{2}\}

= \frac{N}{K} · (\frac{K}{N} )^{2}

getting ∑_{i=1}^{N} c_{i}^{2} ≤ K. It is also easy to see that c_{1} = 1 (since e_{1} is the only non-zero-sum vector and the sum of its entries equals 1, just as ∑_{i} x_{i}). Thus, we have

∑_{i=2}^{N} c_{i}^{2} ≤ K − 1  (10.1)

We now consider the differences vector, denoted z, representing the deviation of the random variable Y from the uniform distribution.

z^{⊤} =^{def} Ax^{⊤} − e_{1}^{⊤}
= A(∑_{i} c_{i}e_{i})^{⊤} − e_{1}^{⊤}
= ∑_{i>1} λ_{i}c_{i}e_{i}

Recall that the lemma claims an upper bound on the norm-1 of z. Instead, we start by providing a bound on its norm-2:

∑_{i} z_{i}^{2} = z · z^{⊤}
= ∑_{i>1} λ_{i}^{2}c_{i}^{2} · e_{i}e_{i}^{⊤}

≤ (\frac{λ}{d})^{2} ∑_{i>1} c_{i}^{2} · \frac{1}{N}

≤ (\frac{λ}{d})^{2} \frac{K − 1}{N}

where the last inequality is due to Eq. (10.1). Maximizing the sum of the |z_{i}|’s, subject to the above bound, the lemma follows. (In other words, ∑_{i=1}^{N} |z_{i}| ≤ \sqrt{N} · \sqrt{∑_{i=1}^{N} z_{i}^{2}}, which in this case is bounded by \frac{λ}{d} · \sqrt{K − 1}).

Square roots in Z_{p}^{∗} and Primality Testing

Lecture given by Shafi Goldwasser

Notes taken by Yehuda Hassin, Lior Noy and Ben Stein

Summary: In this lecture we introduce two randomized algorithms in number theory. The first algorithm finds in expected polynomial time the square root of a number in Z_{p}^{∗} for any prime number p. We then use this algorithm to construct a polynomial time algorithm with two-sided errors for primality testing.

11.1 Introduction

Primality testing is known as one of the first areas for the use of randomized algorithms. Moreover, from theoretic point of view this example is still very important. Many of the problems that have randomized algorithms have also a deterministic versions. Hence, the use of randomized algorithms is usually done just for simplicity reasons or as a building stone for deterministic ones. However, there is still no deterministic polynomial time algorithm for primality testing, although we know of some randomized polynomial time algorithms for this problem for twenty five years.

First we introduce some simple definitions and basic facts in group theory.

As a first step towards primality testing we introduce an algorithm SQRT for finding square roots in Z_{p}^{∗}. The algorithm SQRT is a Las Vegas algorithm, i.e., an expected polynomial time algorithm with no error probability. We then convert SQRT into a Monte Carlo algorithm, i.e., an algorithm that runs in polynomial time but has an error probability. We use randomization in this algorithm in order to find a non square number in Z_{p}^{∗}. Although half of the numbers in Z_{p}^{∗} have no square roots, no deterministic algorithm that finds such a number is known.

We then construct an algorithm that tests weather a given number n is a prime number. Let n = ∏_{i=1}^{k} q^{α_{i}}. We rely on the fact that the number of square roots for a square in Z_{n}^{∗} is 2^{k}. We then invoke algorithm SQRT to find a square root. Choosing a random solution to this equation and using the above fact (that for composite, not a prime power, there are many solutions), we introduce a two-sided error randomized algorithm for primality testing.

Although the above algorithm does not give us an absolute proof for prime numbers, in the last part of the lecture we will prove that there is a short proof, and hence the language of prime numbers is in N P.

118 LECTURE 11. SQUARE ROOTS IN Z_{P}^{∗} AND PRIMALITY TESTING

11.2 Definitions and Simple Facts

11.2.1 Definitions

We start with a definition of a group. A group G is a set of elements with a binary operation that satisfies the following properties: The binary operation is associative, has an identity element e, (i.e., for every a ∈ G, a∘e = a = e∘a), and for every a ∈ G there exists an inverse element, denoted a^{−1}, such that a ∘ a^{−1} = e.

Example: The additive group Z_{p} = {0, 1, 2, ..., p − 1}, when p is an integer, with the binary operation: a + b =^{def} (a + b) mod p. By definition, Z_{p} is a group because

1. [(a + b) mod p + c] mod p = [a + (b + c) mod p)] mod p (association).

2. (a + 0) mod p = a = (0 + a) mod p (the identity element is 0).

3. [a + (p − a)] mod p = 0 = [(p − a) + a] mod p (the inverse element of a is p − a).

Definition 11.1 Group order: The order of a group G, denoted by |G|, is the number of elements in G.

Denote by h^{m} the m-wise multiplication h ∘ ⋯ ∘ h.

Definition 11.2 Element order: The order of an element h in a group G is the smallest integer number m such that h^{m} = 1.

Definition 11.3 Cyclic group: A Cyclic group is a group that contains an element g such that G = {g, g^{2}, …, g^{m−1}, g^{m}}, m = |G|, and g^{m} is the identity of the group. The element g is called the generator of the group.

Example: The group Z_{p} is a cyclic group because g = 1 is a generator of the group:

(1) mod p = 1
(1 + 1) mod p = 2
(1 + 1 + 1) mod p = 3
·
·
·

\underbrace{(1 + 1 + ... + 1)}_{p−1} modp = p − 1

\underbrace{(1 + 1 + ... + 1 + 1)}_{p} modp = 0

Another example of a group that we use frequently is the group Z_{p}^{∗}.

Let Z_{p}^{∗} = {1 ≤ x ≤ p | gcd(x, p) = 1} with the binary operation being multiplication mod p,

i.e., a ∗ b =^{def} (a ∗ b) mod p.

Definition 11.4 The order of Z_{p}^{∗} (the number of elements in Z_{p}^{∗}) is known as the Euler Totient Function: |Z_{p}^{∗}| = ϕ(p).

Proposition 11.2.1 If p is a prime number then |Z_{p}^{∗}| = p − 1.

Proof: Since for a prime p, for every 1 ≤ x ≤ p − 1, we know that gcd(x, p) = 1. It follows simply from the definition that |Z_{p}^{∗}| = p − 1.

11.2.2 Some facts about Z_{p}^{∗}

1. From a computational point of view, for every a ∈ Z_{p}^{∗} there is a polynomial time algorithm that finds b ∈ Z_{p}^{∗} such that (a ∗ b) mod p = 1. This can be done, for example, by looking at the output of Extended-gcd(a, p): for a ∈ Z_{p}^{∗} the output is two integers α and β such that αp + βa = gcd(a, p) = 1. Reducing this equation mod p we get that βa ≡ 1 mod p, and so β is the multiplicative inverse of a.

2. Euler’s theorem: for every a ∈ Z_{p}^{∗} : a^{ϕ(p)} ≡ 1 mod p.

3. Fermat’s theorem: when p is prime, for every a ∈ Z_{p}^{∗} : a^{(p−1)} ≡ 1 mod p.

Remark: if the converse of the Fermat’s sentence was true, then one could suggest that when we want to test if p is prime, we should check weather, for every a ∈ Z_{p}^{∗}, it holds that a^{(p−1)} ≡ 1 mod p. But, the converse of Fermat’s theorem is not true and there are numbers, known as the Carmichael numbers, such that a^{(p−1)} ≡ 1 mod p for every a ∈ Z_{p}^{∗}, but p is not prime. Therefore, we can not use Fermat’s theorem as a primality test.

4. When p is a prime, Z_{p}^{∗} is a cyclic group of order p−1, meaning, there exist an element g ∈ Z_{p}^{∗} such that Z_{p}^{∗} = {g^{1}, g^{2}, …, g^{p−1}} = {g^{i} : i = 1, 2, …, p − 1}.

Example: Z_{7}^{∗} = {1, 2, 3, 4, 5, 6} = {3, 2, 6, 4, 5, 1} = {3^{1} mod 7, 3^{2} mod 7, 3^{3} mod 7, 3^{4} mod p, 3^{5} mod 7, 3^{6} mod 7}, so 3 is a generator of Z_{7}^{∗}.

Remark: We can see that for every element a ∈ Z_{p}^{∗} and a generator g of it, there exists a unique index i such that: a ≡ g^{i} mod p . This i is called the index of a with respect to g.

11.3 Finding a Square Root

11.3.1 Description of the Problem

Let a ∈ Z_{p}^{∗}. Given the equation

a ≡ x^{2} mod p

is there a solution to this equation?

If a solution x exists such that the above equation is true, a is called quadratic residue mod p or a square mod p. If no such solution exists then a is called quadratic non-residue mod p or a non-square mod p.

What do we know about the number of solutions for such an equation? This depends whether p is a prime or not. If p is prime the following fact is known:

Fact 11.3.1 Given a prime p, the equation a ≡ x^{2} mod p has either no solutions, or exactly 2 solutions: x and −x.

Proof: If the equation has no solutions we are done. Assume there is at least one solution x, then clearly −x is also a solution. We’ll show that x and −x are the only possible solutions. Let y be another solution, i.e., a ≡ y^{2} mod p iff y^{2} ≡ a mod p. Since x is a solution, we also have a ≡ x^{2} mod p iff x^{2} ≡ a mod p, so we can write

x^{2} ≡ y^{2} mod p
x^{2} − y^{2} ≡ 0 mod p
(x − y)(x + y) ≡ 0 mod p.

Because of the primality of p, either y = x or y = p − x = −x.

We will later use this fact to discriminate between primes and non-primes. So, our next task should be to find a way to know if a given a is a square mod p. The following fact will be helpful:

Fact 11.3.2 Given a prime p and a generator g for Z_{p}^{∗}, then a ∈ Z_{p}^{∗} is square mod p iff the index of a w.r.t. p and g is even.

Proof: (⇐ even-index imply square:) Let a be an element with an even index, i.e., for some j it holds that a ≡ g^{2j} mod p ≡ (g^{j})^{2} mod p. This implies that a is a square mod p, with a square root g^{j}. (⇒ square imply even-index:) a is a square-mod-p, so exist y ∈ Z_{p}^{∗} s.t. a ≡ y^{2} mod p. Since g is the generator, g generates every member of Z_{p}^{∗} and specifically y, so for some t, y ≡ g^{t} mod p. Plugging the last equation into a ≡ y^{2} mod p we get: a ≡ ((g^{t} mod p)^{2}) mod p ≡ g^{2t} mod p. Since the index of a w.r.t g is unique we get that: i ≡ 2t mod (p − 1), and the index of a is even (because p is prime and hence odd).

Example: we saw before that 3 is a generator of Z_{7}^{∗}, and we note that elements with even index are indeed square-mod-p. For example, notice first that the element 4 has an even index: the index is 4, since 4 ≡ 3^{4} mod 7. 4 is a square, with two roots: the obvious 2 and 5, since 4 ≡ 5^{2} mod 7. Note that 5 ≡ −2 mod 7 so we see that Fact 11.3.1 also holds.

Due to the fact that half of the elements are generated with an even index, we can conclude the next corollary:

Corollary 11.5 Let p be a prime then half of the elements of Z_{p}^{∗} are squares-mod-p.

Remark: keeping in mind our ultimate goal for this class, which is to test whether a number is prime or not, and that our current task is to find if an element of Z_{p}^{∗} is a square-mod-p, one can suggest the following method to find if a is a square: find the index i of a such that a = g^{i}. If i is even, conclude that a is square-mod-p, else conclude that it is not. This simple idea crushes on the shores of reality due to the fact that finding the index of an element, known as the discrete log problem is considered a computationally hard task: that is, no efficient algorithm is known for this task. The next claim will present another way to find out if a is square-mod-p.

Definition 11.6 Given a prime p and a ∈ Z_{p}^{∗}, the Legendre symbol (\frac{a}{p}) is defined as

(\frac{a}{p}) = \{ \begin{matrix}1 & a is a square-mod-p \\ −1 & a is not a square-mod-p \end{matrix}

Theorem 11.7 Given a prime p and a ∈ Z_{p}^{∗},

a^{\frac{p−1}{2}} = (\frac{a}{p}) .

This is called the Euler Criteria.

To prove the claim, first observe that the following fact is correct:

Fact 11.3.3 For a prime p, and a generator g of Z_{p}^{∗}, it holds that g^{\frac{p−1}{2}} = −1.

Proof: Look at (g^{\frac{p−1}{2}} )^{2} = g^{p−1} = 1 (the last equality holds for every a ∈ Z_{p}^{∗}). So by Fact 11.3.1

g^{\frac{p−1}{2}} is either 1 or -1. We’ll show that 1 can not be a solution. If g^{\frac{p−1}{2}} = 1, we have found an index j smaller then p − 1 s.t. g^{j} = g. If such index exist g can not generate all the elements in Z_{p}^{∗}, but since we know that g is a generator, the assumption that g^{\frac{p−1}{2}} = 1 can not be true, and we get that g^{\frac{p−1}{2}} = −1.

Example: we saw that 3 is the generator of Z_{7}^{∗}. Computing 3^{\frac{7−1}{2}} mod 7 = 3^{3} mod 7 = 27 mod 7 ≡ −1 we see that the fact holds.

We can now prove the Euler Criteria:

Proof (of Theorem 11.7): If a is a square-mod-p, then by Fact 11.3.2 a has an even index or in other words a = g^{2j} for some j. Hence, a^{\frac{p−1}{2}} = g^{2j} ^{\frac{p−1}{2}} = g^{j(p−1)} = (g^{p−1})^{j} = 1^{j} = 1 = (\frac{a}{p}).

If a is not a square-mod-p, then by Fact 11.3.2 a has an odd index or in other words a = g^{2j+1} for some j. So,

a^{\frac{p−1}{2}} = (g^{2j+1})^{\frac{p−1}{2}} = g^{[j(p−1)+ \frac{p−1}{2} ]} = g^{j(p−1)}g^{\frac{p−1}{2}}
= (g^{(p−1)}^{j})(−1) = 1^{j}(−1) = −1 = (\frac{a}{p})

11.3.2 An Algorithm For Finding the Square-Root (mod p)

Now that we have an easy criterion to find out if a is a square-mod-p, we can develop an algorithm for finding the square-root, i.e., an x which satisfy a ≡ x^{2} mod p. This algorithm will give us an efficient way to compute the value of the square-root if it exists.

The input to the Square-Root Algorithm will be prime p and a quadratic-residue a ∈ Z_{p}^{∗}. The output will be a number x such that a ≡ x^{2} mod p. The algorithm operates differently in two cases, depending on p, one case is easy and the other is more complex.

The two cases For any number p, the value of p mod 4 is in {0, 1, 2, 3}. For p that is prime, the possible values are only 1 or 3, since 0 and 2 imply that p is even and hence not a prime. This constitute our two cases:

(A) p mod 4 = 3
(B) p mod 4 = 1

(A) p mod 4 ≡ 3. We can write p as 4t + 3 for some t. We assume that a is a square-mod-p so starting from Theorem 11.7, we develop the following sequence of equivalent conditions:

(\frac{a}{p}) ≡ 1 mod p
a^{\frac{p−1}{2}} ≡ 1 mod p
a^{\frac{4t+3−1}{2}} ≡ 1 mod p
a^{2t+1} ≡ 1 mod p

a^{2t+2} ≡ a mod p
a^{2(t+1)} ≡ a mod p
(a^{t+1})^{2} ≡ a mod p.

So we conclude that x = a^{(t+1)} is the square-root of a.

(B) p mod 4 ≡ 1. Trying to follow the same way as in (A), let us write p = 4t + 1 ⇒ a^{\frac{p−1}{2}} = a^{\frac{4t}{2}} = a^{2t} = 1 mod p. But here we hit a wall: multiplying both sides by a as before leaves us with an odd power on the left side, and not an even one to get the square-root. Our goal is therefor to have an odd power before the multiplication.

A promising direction to achieve this goal is to write the power of a as 2t = 2^{i}(2j + 1). We will first show an “idea” to use this representation to get an odd power in the left side. This “idea” has a problem, but soon we will see how to tackle this problem to get a full solution.

Basic Idea

a^{\frac{p−1}{2}}  ≡  1 mod p
a^{2t}  ≡  1 mod p
a^{2^{l}(2j+1)}  ≡  1 mod p (take square-root of both sides)
a^{2^{(l−1)}(2j+1)}  ≡  1 mod p (continue with the square-root until...)
…
a^{2j+1}  ≡  1 mod p (multiple both side with a)
a^{2j+2}  ≡  a mod p
(a^{j+1})^{2}  ≡  a mod p
⇒ x = a^{j+1} is the square-root of a.

We have implicitly assumed that taking square roots of a^{2} = 1 we get a = 1. But it may be that a = −1. Indeed, what if in one of the square-root operations, the result for the right side is −1 and not 1? In this case we will need an extra “rescue equation”. This equation uses a new element b, which is a non-square-mod-p, and hence according to the Euler criteria, b^{2t} = b^{\frac{p−1}{2}} ≡ −1 mod p. Using this we will manipulate the equation to get an odd power in the left side just before the final step.

The Full Solution We will follow the steps of the basic idea. As long as taking a square-root on both sides results with a 1 in the right side we continue. If the operation results with a −1 (i.e., we reach a^{2^{(l−1)}(2j+1)} = −1), we guess a b, and verify that it is indeed non-square with the Euler Criteria. Since b^{2t} = −1 we will multiple both side of the ’bad’ equation with b^{2t} and get b^{2t} · a^{2^{(l−1)}(2j+1)} ≡ 1 mod p, and then continue with the sequence of square-root operations. After i times of taking square-root we will get something like:

b^{\frac{2t}{2^{i_{1}}}} b^{\frac{2t}{2^{i_{2}}}} ⋯ b^{\frac{2t}{2^{i_{k}}}} a^{2j+1} ≡ 1 mod p [for some i_{s}’s]
b^{(2^{l_{1}}+2^{l_{2}}+···+2^{l_{k}})(2j+1)}a^{(2j+1)} ≡ 1 mod p [ for l_{s} = l − i_{s}’s, using 2t = 2^{l}(2j + 1) ].

Observe that the power of b is even. This is true since for each 1 ≤ s ≤ k, l_{s} is positive (at least in the first time of taking square root we take square root from a^{2t} and not from the b term), the power of b is just a sum of the 2^{l_{s}}’s (multiplied by 2j + 1) and hence is an even number. So writing the power of b as 2D, for some D, we finally have:

b^{2D}a^{2j+1}  ≡  1 mod p (multiple both side with a)
b^{2D}a^{2(j+1)}  ≡  a mod p
(b^{D}a^{j+1})^{2}  ≡  a mod p
⇒ (the square-root of a is:) b^{D}a^{j+1}

Algorithm SQRT(a,p)

Input: an odd prime p, and a square-mod-p element a Output: The element x such that a ≡ x^{2} mod p

A) If p mod 4 = 3, find t such that p = 4t + 3. OUTPUT a^{t+1} mod p.

B) If p mod 4 = 1, find t such that p = 4t + 1.

B.1) Repeatedly choose uniformly b ∈ Z_{p}^{∗}, until b^{\frac{p−1}{2}} mod p ≡ −1

B.2) initialize i = 2t (the power of a), k = 0 (the power of b)

B.3) REPEAT until i is odd

B.3.1) i ← \frac{i}{2} , k ← \frac{k}{2}

B.3.2) IF a^{i}b^{k} ≡ −1 mod p THEN k ← k + 2t

B.4) OUTPUT a^{\frac{i+1}{2}} b^{\frac{k}{2}}

Figure 11.1: Algorithm SQRT(a,p)

Comment: There is no known efficient deterministic algorithm to perform step B.1 (above), i.e., to find a non-square element in Z_{p}^{∗}. However, the Extended Riemann Hypothesis (ERH) implies that there is a non-square element in the first log^{2} p numbers of Z_{p}^{∗}. Hence, this implies a deterministic algorithm, since we can check all the numbers less than log^{2} p and find a non-square.

Correctness of SQRT The correctness of the algorithm follows from the above discussion. Specifically, Stage B.3.1 performs the square root operation by dividing the current power of a (which is i) and b (which is k) by 2. As we explained if the square root is −1 we multiply both sides of the equation by −1; this is done in Stage B.3.2 by increasing the power of b by 2t (which is the same as multiplying by b^{2t} ≡ −1).

Analysis of the running time of SQRT It is easy to see that all the deterministic steps run in time which is polynomial with the description of p. What about the random step B.1? Since we saw before that half of the elements of Z_{p}^{∗} are non-square-mod-p, we have a probability of \frac{1}{2} to choose a ’good’ ( i.e. non-square ) element. The expected running time is therefore 2 tries for this step, and polynomial-time for all the algorithm.

Comment: the SQRT algorithm described above never fails, but can take in the worst case infinite time, due to a sequence of bad choices in step B.1. It is therefore a Las-Vegas type algorithm. The next section describes how it can be turned into a Monte-Carlo type algorithm.

11.3.3 A Monte-Carlo version of the SQRT algorithm

We next provide a monte carlo algorithm MSQRT (a, p, s) for finding square root. The running time is no longer expected polynomial time but rather a polynomial time algorithm. Although, we introduce a failure probability 2^{−s}. The algorithm is different from the last one only in stage B.1. Instead of choosing repeatedly until finding a non-square, we stop after s failures, making sure that the algorithm will always stop, and succeeds with high probability.

The Monte-Carlo version will always run in polynomial-time. For every a and p the algorithm output “FAIL” only when step B.1 did not manage to find a ’good’ b in s tries. ¿From Corollary 11.5 the probability for a failure in one try is \frac{1}{2}, the probability to FAIL is (\frac{1}{2})^{s} = 2^{−s}.

11.4 Randomized Algorithm for Primality Testing

The first randomized algorithms for primality testing were given in 1976-77 by Solovay and Strassen and by Miller and Rabin. Both algorithms have only one-side error. We will describe an algorithm given by Silvio Micali. This algorithm is two-sided error but has a simple analyses.

In order to introduce an algorithm that will separate between prime numbers and composite numbers, we need first to find a mathematical property that separates them. This property is needed to be “easy” for being inspected. The property that our algorithm is going to use is the number of solutions to the equation x^{2} ≡ a mod n. By Fact 11.3.1 we know that if n is a prime number then the number of solutions to the above equation is (0 or) 2. Let n = ∏_{i=1}^{k} p_{i}^{α_{i}}. We next show that if n is odd and not a prime power (i.e., k > 1), then the number of solutions to the above equation is 0 or 2^{k}.

In order to prove this fact, recall the Chinese remainder theorem (CRT).

Theorem 11.8 Let n = n_{1}n_{2} ⋯ n_{k}, where the n_{i} are pairwise relatively prime. For any sequence of residues r_{1} ∈ Z_{n_{1}}, …, r_{k} ∈ Z_{n_{k}}, there is a unique r ∈ Z_{n} such that

r ≡ r_{i} mod n_{i} (for 1 ≤ i ≤ k).

Moreover r can be computed in polynomial time.

We next prove the theorem regarding the number of solutions to the equation x^{2} ≡ a mod n.

Theorem 11.9 Let n = ∏_{i=1}^{k} q_{i}^{α_{i}} be the factorization of an odd number that is not a prime or a prime power. The number of solutions to the equation x^{2} ≡ a mod n is either 0 or 2^{k}.

Proof: Let n_{i} = q_{i}^{α_{i}} and a_{i} ≡ a mod n_{i}. Then by the CRT we know that being equal to a mod n is exactly as being equal simultaneously to a_{i} mod n_{i} for all i’s. But Fact 11.3.1 can be generalized to a prime power such that the equation x^{2} ≡ a_{i} mod n_{i} has either 0 solutions or 2 solutions. If there exists some i with 0 solutions then there is 0 solutions to the equation x^{2} ≡ a mod n (since if there is any solution to this equation it solves also the equation x^{2} ≡ a_{i} mod n_{i}). On the other hand, if every equation x^{2} ≡ a_{i} mod n_{i} has 2 solutions {y_{i}^{1}, y_{i}^{2}} then by using again the CRT there is a unique and different y ∈ Z_{n} that corresponds to any one of the 2^{k} choices of simultaneous solutions for the equations (choosing an array of length k each cell with two choices).

We next present the algorithm Primality-Test. The algorithm first chooses uniformly at random solution to the equation x^{2} ≡ a mod p. This is done by choosing x at random and then squaring it. Then we ask algorithm MSQRT (a, p, s) to find a sqrt to a. If p is prime then the only solutions are {x, −x} and, with probability 1 − 2^{−s}, algorithm MSQRT (a, p, s) finds this solution. But if p is not a prime then even if algorithm MSQRT (a, p, s) returns a square root with good probability, it may happen to be not in {x, −x}, but rather be another solution (one of the other 2^{k} −2 possible solutions). The algorithm Primality-Test is presented formally in Figure 11.2.

Algorithm Primality-Test.

Input: a number p and a safety parameter s.

Output: “Prime” or “Composite”.

1. If p is either even or a prime power return “Prime”.

2. Choose x ∈ Z_{p}^{∗} uniformly at random.

3. If gcd(x, p) ≠ 1 return “Composite”.

4. Compute a = x^{2} mod p.

5. val ← MSQRT (a, p, s).

6. If val equals “FAIL” or val ∉ {x, −x} or val^{2} ≠ a mod p then return “Composite”,

7. Else (i.e., val ∈ {x, −x}) return “Prime”.

Figure 11.2: Algorithm Primality-Test

Theorem 11.10 Algorithm Primality-Test(p, s) is a monte carlo algorithm that runs in polynomial time and distinguish between a prime number and a composite number. If p is prime then the algorithm outputs “prime” with probability at least 1 − 2^{−s} and if p is composite then the algorithm outputs “composite” with probability at least \frac{1}{2}.

Proof: We first analyze the running time of the algorithm. The first step is implemented in polynomial time, since one can check if a number p is a power just by trying all the possible powers k ≤ log p and finding a possible solution on the integers by a simple binary search (each time dropping half of the possible integer solutions). Then we use Euclid algorithm and algorithm MSQRT (a, p, s), which are both polynomial time algorithms.

We next prove two claims regarding the two sided errors.

Claim 11.4.1 If p is a prime number then the probability that algorithm Primality-Test(p, s) output “Composite” is less than 2^{−s}.

Proof: If p is a prime number than algorithm MSQRT (a, p, s) returns the unique solutions {x, −x} with probability at least 1 − 2^{−s}.

Claim 11.4.2 If p is composite then the probability that algorithm Primality-Test(p, s) outputs “Prime” is at most \frac{1}{2}.

Proof: If p is composite then we can guarantee nothing on the output of MSQRT (a, s). But even if this algorithm outputs a square root of a then by Theorem 11.9 it is one of 2^{k} possible solutions, where k is the number of primes that divides p. Since k ≥ 2, the probability that the output of MSQRT is in \{x, −x\} is at most \frac{2}{2^{k}} ≤ \frac{1}{2}. It follows that if p is composite then the algorithm outputs “prime” with probability at most \frac{1}{2}.

Combining the two claims, the theorem follows.

11.5 Short proof for prime numbers

Let PRIME be the set of all prime numbers and let COMPOSITE denote the complement set. It is easy to see that COMPOSITE∈ N P, since the witness for a number n is simply two integers p and q such that their product is n (and none is 1). Computing a product of two numbers is polynomial in the length of the numbers. We emphasize that checking if the N P witness is good must be polynomial in the length of n which is log n. Our goal in this section is to show that there exists short proofs also for PRIME.

In first glance this result is somehow surprising since PRIME is a set that is a coN P kind (i.e., for p in the set one must prove that for every x < p, x does not divide p). However, Pratt discovered a short proof for PRIME. Since COMPOSITE is also in N P this means that PRIME ∈ N P ∩ coN P hence it is believed that PRIME is not N P-COMPLETE.

Again we are looking for a mathematical property that separates the prime numbers from the composite. By Corollary 11.2.1 we know that Z_{p}^{∗} has p−1 elements if p is a prime number, otherwise it has less elements. The first idea to use this characterization (of the primes) is to use the fact that Z_{p}^{∗} has a generator g. Given such a g we just have to check that the order of g is p − 1. It seems that we did nothing because this can be checked by looking at the set {g^{1}, g^{2}, …, g^{p−1}} and verifying that the only power that equals 1 is p − 1. But the size of this set is exponential in the length of p. In order to solve this problem we use Fact 11.5.1. From this fact we conclude that it suffices to check a set of size at most O(log p) (the number of distinct primes that divides p) in order to guarantee that g is a generator of Z_{p}^{∗} and that the order of g is p − 1.

Fact 11.5.1 Let p be a prime then g is a generator of Z_{p}^{∗} iff for every prime q that divides p − 1

it holds that g^{\frac{p−1}{q}} ≠ 1.

Proof: If g is a generator and p is prime clearly its order is p − 1 hence any power of g less than p − 1 does not equals 1. Conversely, if g is not a generator then the set {g^{1}, g^{2}, …, g^{ord(g)}} is a subgroup of Z_{p}^{∗} and thus divides p − 1. But if ord(g) < p − 1 and divides p − 1 it must also divides

\frac{p−1}{q} for some prime q. Hence we conclude that g^{\frac{p−1}{q}} ≡ 1 mod p.

But given the prime factorization of p − 1 = ∏_{i=1}^{k} q_{i}^{α_{i}} ; how do we know that every q_{i} is itself a prime? this is done recursively.

Theorem 11.11 Let p be a prime number and p − 1 = ∏_{i=1}^{k} q_{i}^{α_{i}}. The following recursive witness for p,

W (p) = g ∘ (q_{1}, α_{1}) ∘ (q_{2}, α_{2}) ∘ ⋯ ∘ (q_{k}, α_{k}) ∘ W (q_{1}) ∘ W (q_{2}) ⋯ W (q_{k}) ,

is of polynomial size in the length of p and can be verified in polynomial time, hence P RIME ∈ N P.

Proof: In order to check the witness W (p), for the integer p, we first verify that indeed p − 1 =

∏_{i=1}^{k} q_{i}^{α_{i}}. Then we check that g^{\frac{p−1}{q_{i}}} ≠ 1 mod p, for every 1 ≤ i ≤ k. Finally, we continue by checking recursively that indeed each q_{i} is a prime by checking all the W (q_{i})’s. We have already explained why W (p) is a witness for p.

We next prove that it is of length polynomial in the length of p. Open the recursion in a tree fashion and look at the i′th level of the tree. Notice (can be checked recursively) that the product of all the q′s in the i′th level is always less than p. Hence their total length is less than twice the length of p. If we add the generators of the i′th level and the α’s we get no more than 4 log p. The number of levels is less than log p because each time there is at least a division by 2. Multiplying the number of levels with the size of each level completes the claim about the size of the witness.

The time it takes to verify a witness includes mainly checking that g^{\frac{p−1}{q_{i}}} ≠ 1 for every 1 ≤ i ≤ k. Since taking powers is logarithmic in the exponent, we can conclude that each “piece” of the witness needs O(poly(log p)) time to verify, all together polynomial time in O(log p).

Bibliographic Notes

The square root extraction algorithm presented in this lecture is due to Adleman, Manders and Miller [2].

The primality testing algorithm is due to Silvio Micali. Recall that Micali’s algorithm uses any algorithm that extracts square root modulo a prime in order to detect that a given number is not a prime (since the algorithm is likely to fail in such a case). This algorithm is analogous to Rabin’s reduction [37] of factoring to extracting sqaure roots modulo a composite. The difference is that Micali’s algorithm is a randomized reduction of the problem of testing primality to the problem of extracting square root modulo a prime, and since the latter is known to have a probabilistic polynomial-time algorithm the same follows for the former. In contrast, Rabin’s reduction is from a problem believed to be hard (i.e., factorization) to a problem that is not known to have a probabilistic polynomial-time algorithm, such that assuming the intractability of the former it follows that so is the latter.

We comment that the primality tester presented above is less efficient than the standard probabilistic polynomial-time algorithms used for this problem; that is, the primality testers of Solovay and Strassen [41] and of Rabin [38]. We mention that the latter algorithms have one-sided error (i.e., they never declare a prime to be composite), whereas Micali’s algorithm may err both ways (but all errors occur with small probability). In contrast, errorless algorithms (which with small probability output no verdict) do exist [1] (building upon [24]) but are far more complex.

The construction of NP-certificates for primality (i.e., the fact that the set of primes is in N P) is due to Pratt [36].

128 LECTURE 11. SQUARE ROOTS IN Z_{P}^{∗} AND PRIMALITY TESTING

Hitters and Samplers

Notes taken by Amos Korman, Yoav Rodeh, Eran Tromer

Summary: A sampler is an oracle Turing machine that estimates the average of any function f : {0, 1}^{n} → [0, 1] with bounded deviation and bounded probability of failure. A hitter is an oracle Turing machine that, given a boolean function f : {0, 1}^{n} → {0, 1}, finds x s.t. f(x) = 1 with a bounded probability of failure if f has value 1 on at least a constant fraction of {0, 1}^{n}. These are fundamental and useful problems. We consider the randomness and query complexities of hitters and samplers whose running time is polynomially bounded, and recall lower bounds for each. We show several simple constructions for both, as well as improved constructions based on pairwiseindependent sample spaces and on random walks on expander graphs. We then show composite constructions whose complexities match the lower bounds.

12.1 Definitions and Lower Bounds

12.1.1 Definitions

Suppose we are given black-box access to a function f : {0, 1}^{n} → [0, 1] or f : {0, 1}^{n} → {0, 1}. A problem of interest is to estimate its average value:

f̄ =^{def} ∑_{x∈{0,1}^{n}} f(x)

An exact solution would require evaluating f at all points, which takes time exponential in n. However, an approximate solution could be quite useful: for instance, given a BPP algorithm that has error probability bounded by \frac{1}{3}, estimating f̄ within error smaller than \frac{1}{6} would suffice to obtain a decision with certainty.

By relaxing the requirement further to allow some probability of failure, we obtain the notion of a sampler. Given oracle access to f : {0, 1}^{n} → [0, 1] the sampler outputs a value that is within ε of f̄ with probability at least 1 − δ. A boolean sampler is similar but the domain of f is limited to {0, 1}. Obviously any sampler is a boolean sampler, but the converse is not true.

Throughout the discussion we assume that the values of f are given with at least log_{2}(1/ε) bits of precision (otherwise samplers don’t exist), but no more than poly(log(1/ε), log(1/δ), n) bits (otherwise merely reading the oracle tape would exceed the polynomial time bound discussed below).

Definition 12.1 (Sampler) A probabilistic oracle Turing machine M is a sampler if for any n ∈ N, ε ∈ (0, 1] and δ ∈ (0, 1] and for every function f : {0, 1}^{n} → [0, 1]:

Pr [|M^{f}(ε, δ, n) − f̄| > ε] < δ

Definition 12.2 (Boolean Sampler) A probabilistic oracle Turing machine M is a boolean sampler if for any n ∈ N, ε ∈ (0, 1] and δ ∈ (0, 1] and for every function f : {0, 1}^{n} → {0, 1}:

Pr [|M^{f}(ε, δ, n) − f̄| > ε] < δ

Another scenario is this: we are given black-box access to f : {0, 1}^{n} → {0, 1} and are assured that it has value 1 on at least an ε fraction of its domain. The problem is to find one of these values, i.e., to find a representative of the “good set” f^{−1}(1). As before, in the worst case we could check an exponential number of values without succeeding. By allowing some probability of error we obtain the notion of a hitter.

For f : {0, 1}^{n} → {0, 1} and a ∈ {0, 1}, denote:

f^{−1}(a) =^{def} {x ∈ {0, 1}^{n} | f(x) = a}

For ε ∈ [0, 1], denote:

F (n, ε) =^{def} {f | f : {0, 1}^{n} → {0, 1}, |f^{−1}(1)| ≥ ε · 2^{n}}

That is, F (n, ε) the set of functions that have value 1 on at least an ε fraction of their domain.

Definition 12.3 (Hitter) A probabilistic oracle Turing machine M is a hitter if for any n ∈ N, ε ∈ (0, 1] and δ ∈ (0, 1] and for any function f ∈ F (n, ε):

Pr [M^{f}(ε, δ, n) ∈ f^{−1}(1)] > 1 − δ

For both samplers and hitter, the input δ specifies the allowed error probability. The input ε differs: for samplers it specifies the precision required of the answer, and for hitters it specifies a promise made on the input function.

In Subsection 12.2.1, we show that any sampler yields a boolean hitter with related complexity.

12.1.2 Complexity Measures

As suggested above, the motivation for the relaxed definition is the ability to reduce complexity. For any probabilistic oracle Turing machine M (specifically, for samplers, boolean samplers and hitters), we are interested in the following:

Time complexity T_{M}(ε, δ, n): We require T_{M}(ε, δ, n) = poly(n, \frac{1}{ε}, log(\frac{1}{δ})). That is, M should

halt within time polynomial in n, \frac{1}{ε} and log(\frac{1}{δ}).

Query complexity Q_{M}(ε, δ, n): The maximum number of queries made by M^{f}(ε, δ, n), taken over all possible coin tosses and all possible oracles f ∈ F (n, ε). We wish to minimize this, because evaluation may be expensive or the context may bound the number of permissible evaluations.

Randomness complexity R_{M}(ε, δ, n): The maximum number of probabilistic bits used by M^{f}(ε, δ, n), i.e., the number of binary coin flips it may use, taken over all possible oracles f ∈ F (n, ε). We wish to minimize this parameter as well: for small δ we may need to try many different random choices, so it is useful to reduce the space of this choices (we’ll see an example of this in Section 12.4).

12.1.3 Lower Bounds

The following lower bounds on complexity hold:

Q_{M}(ε, δ, n) ≥ R_{M}(ε, δ, n) ≥
Hitters  Ω (\frac{1}{ε} log(\frac{1}{δ})) n + log_{2}(\frac{1}{δ})− log_{2}(Q_{M}(ε, δ, n)) − O(1)
Samplers Ω (\frac{1}{ε^{2}} log(\frac{1}{δ})) n + log_{2}(\frac{1}{δ})− log_{2}(Q_{M}(ε, δ, n)) − O(1)

We proceed to prove the lower bounds for hitters^{1}. For proofs of the lower bounds on the complexity of samplers, see [14] or the weaker results of Subsection 12.2.1.

The lower bounds which we derive on the query and randomness complexities of hitters apply only when ε, as a function of n and δ, is upper bounded by some constant c < 1. Some limitation must indeed be imposed on ε, since when ε approaches 1 very quickly it’s possible that few queries or little randomness are required. For instance, consider the case ε = 1 − \frac{1}{2^{n}}, i.e., |f^{−1}(0)| = 1. In this case, for any fixed δ and sufficiently large n, a single random query will hit f^{−1}(1) with probability greater than δ. Similarly, when |f^{−1}(0)| = 1 evaluating any two fixed elements guarantees success regardless of δ, using no randomness.

For the bounds to hold we also need to restrict the values of δ, as the following shows: for given n and ε, the minimum query complexities is no more than (1 − ε) · 2^{n}, regardless of δ. Similarly, an algorithm that is allowed to make 2^{n} queries needs no randomness, regardless of δ.

We will need the following technical lemma:

Lemma 12.1.1 For any constant c ∈ (0, 1), there exists a constant c′ > 0 such that for any N, s ∈ N and ε ∈ (0, c) such that (1 − ε) · N > s:

\begin{matrix}( & {N−s \atop ε·N} & ) \\ ( & {N \atop ε·N} & ) \end{matrix} > e^{−c′·s·ε}

Moreoever, when ε ≪ 1, s ≪ N, ε · N ≫ 1 and (1 − ε) · N − s ≫ 1: c′ ≈ 2.

Proof: According to Stirling’s formula: k! ≈ \sqrt{2πk} (\frac{k}{e} )^{k} for k ≫ 1. This yields the following approximation for the binomial coefficient \binom{a}{b} when b and (a − b) are large:

\binom{a}{b} = \frac{a!}{b!(a − b)!} ≈ \frac{\sqrt{2πa} (\frac{a}{e})^{a}}{\sqrt{2πb} (\frac{b}{e})^{b} · \sqrt{2π(a − b)} (\frac{a−b}{e})^{a−b}} = \sqrt{\frac{a}{2π · b · (a − b)}} · (\frac{a}{b})^{b} · (\frac{a}{a − b})^{a−b}

Denoting σ = \frac{s}{N} and cancelling out N factors, we get:

\begin{matrix}( & {N−s \atop ε·N} & ) \\ ( & {N \atop ε·N} & ) \end{matrix} ≈ \frac{\sqrt{\frac{1−σ}{N·2π·ε·(1−σ−ε)}} · (\frac{1−σ}{ε} )^{ε·N} (\frac{1−σ}{1−σ−ε})^{N−s−ε·N}}{\sqrt{\frac{1}{N·2π·ε·(1−ε)}} · (\frac{1}{ε})^{ε·N} (\frac{1}{1−ε})^{N−ε·N}} = ^{\begin{matrix}√ & \overline{\frac{1−σ}{(1−σ−ε)}} \\ √ & \overline{\frac{1}{(1−ε)}} \end{matrix}} · \begin{matrix}(\frac{1−σ}{ε} ) & ε·N \\ (\frac{1}{ε}) & ε·N \end{matrix} · \begin{matrix}(\frac{1−σ}{1−σ−ε}) & N−s−ε·N \\ (\frac{1}{1−ε}) & N−ε·N \end{matrix}

1The proofs in this section are provided by the scribes and did not appear in the lecture.

Note that \frac{1−σ}{1−σ−ε} > \frac{1}{1−ε}, hence for some constants c_{1}, c_{2} < 1 and c′, we get the lower bound:

\begin{matrix}(\frac{1−σ}{ε} ) & ε·N \\ (\frac{1}{ε}) & ε·N \end{matrix} · \begin{matrix}(\frac{1}{1−ε}) & N−s−ε·N \\ (\frac{1}{1−ε}) & N−ε·N \end{matrix} = (1 − σ)^{ε·N} ·(1−ε)^{s} = (1−σ)^{\frac{1}{σ} ·s·ε} ·(1−ε)^{\frac{1}{ε} ·s·ε} > (c_{1})^{s·ε} ·(c_{2})^{s·ε} = e^{−c′·s·ε}

where the last inequality relies on ε and σ = \frac{s}{N} being upper bounded by constants smaller than 1. The approximations (Stirling’s formula) can be accounted for by a small increase of c′ (rigorous analysis is possible using strict bounds instead of approximations). The main part of the lemma follows.

Moreover, the additional conditions listed in the lemma imply that the approximations are very accurate, and furthermore ε, σ ≪ 1 implies c_{1}, c_{2} ≈ \frac{1}{e}, hence c′ ≈ 2.

Theorem 12.4 For any constant c ∈ (0, 1): for any polynomial time hitter H and any n ∈ N, ε ∈ (0, c) and δ ∈ (0, e^{−ε·(1−ε)·2^{n}}),

Q_{H}(ε, δ, n) ≥ Ω (\frac{1}{ε}log(\frac{1}{δ}))

Proof: Using Yao’s Lemma [44], to prove the lower bound on probabilistic algorithms it suffices to show a probability distribution Φ on F (n, ε) such that every deterministic oracle Turing machine D needs Ω (\frac{1}{ε}log(\frac{1}{δ})) queries to hit a good element with probability at least δ, where the probability is taken over the distribution Φ.

We define Φ to be the uniform distribution over all function f ∈ F (n) s.t. |f^{−1}(1)| = ⌈ε · 2^{n}⌉. Let D be a a deterministic oracle Turing machine D that succeeds with probability at least

δ. Define Samples_{D}(ε, δ, n) to be the set of all queries made by D^{f}(ε, δ, n) when D receives only “0” answers to its oracle queries. This set depends only on the inputs n, ε and δ, and not on the function f. Let s = |Samples_{D}(ε, δ, n)|.

If s ≥ \frac{1}{ε}log(\frac{1}{δ})then we’re done. Otherwise: the limited domain of δ implies \frac{1}{ε}log(\frac{1}{δ})< (1−ε)·2^{n}, so we have s < (1 − ε) · 2^{n}. The probability that D does not succeed is

Pr [f^{−1}(1) ∩ Samples_{D}(ε, δ, n) = ∅]

= \frac{# ways to choose ε · 2^{n} elements outside Samples_{D}(ε, δ, n)}{# ways to choose ε · 2^{n} elements}

= \begin{matrix}( & {2^{n}−s \atop ε·2^{n}} & ) \\ ( & {2^{n} \atop ε·2^{n}} & ) \end{matrix} > e^{−c′·ε·s}

for some constant c′ < 1, where the inequality follows from Lemma 12.1.1 using N = 2^{n}.

The probability e^{−c′·ε·s} must be smaller than δ, and s ≤ Q_{D}(ε, d, n) by the definition of query complexity, so we obtain:

Q_{D}(ε, d, n) ≥ s ≥ Ω (\frac{1}{ε} log(\frac{1}{δ}))

Note that when the conditions of Lemma 12.1.1 hold (and in particular ε is small), the constant hidden in the Ω notation is approximately 2.

Theorem 12.5 For any polynomial time hitter H and constant ε ∈ (0, 1) s.t. Q_{H}(ε, δ, n) < (1 − ε) · 2^{n}:

R_{H}(ε, δ, n) ≥ n + log_{2}(\frac{1}{δ})− log_{2}(Q_{H}(ε, δ, n)) − O(1)

Proof: Fix ε, δ and n. Denote by r the maximum number of random bits used by H^{f}(ε, δ, n), taken over all possible f ∈ F (n, ε).

First we show that δ ≥ \frac{1}{2^{r}}. If this does not hold then H answers correctly for any random seed, so by fixing its random choices arbitrarily we obtain a deterministic hitter H′ with query complexity Q_{H′}(ε, δ, n) < (1 − ε) · 2^{n}. This is impossible since H′ fails when given the following oracle f_{H′}(x) ∈ F (n, ε):

f_{H′}(x) = \{ \begin{matrix}0 & if H′ queries x \\ 1 & otherwise \end{matrix}

We now turn to the heart of the analysis (of H). For w ∈ {0, 1}^{r}, let Samples(w) ⊆ {0, 1}^{n} denote the set of all queries made by H with w as its random coin tosses when H receives only “0” anwers to its oracle queries. For U ⊆ {0, 1}^{r}, define Samples(U) = ⋃_{w∈U} Samples(w), and define the function f_{U} : {0, 1}^{n} → {0, 1} such that

f_{U}(x) = \{ \begin{matrix}0 & if x ∈ Samples(U) \\ 1 & otherwise \end{matrix}

If the random coin toss of H is w ∈ U then given f_{U} as an oracle, H receives only “0” answers and thus fails. Since the coin toss of H is in U with probability at least \frac{1}{2^{r}} |U|, this yields a lower bound on the probability of failure ^{2}:

Pr [H^{f_{U}} ∉ f_{U}^{−1}(1)] > \frac{1}{2^{r}} |U|  (12.1)

Denote by q the maximum number of samples made by H^{f} taken over all f ∈ F (n, ε) and random coin tosses. Then, for every w, |Samples(w)| ≤ q. Therefore, Samples(U) ≤ q · |U|, and so:

|f_{U}^{−1}(1)| = 2^{n} − |Samples(U)| ≥ 2^{n} − q · |U|

Setting |U| = ⌊\frac{1}{q} · (1 − ε) · 2^{n}⌋, we get

|f_{U}^{−1}(1)| ≥ 2^{n} − q · |U| = 2^{n} − q · ⌊\frac{1}{q} · (1 − ε) · 2^{n}⌋ ≥ 2^{n} − q · \frac{1}{q} · (1 − ε) · 2^{n} = ε · 2^{n}

Thus, f_{U} ∈ F (n, ε). H is a hitter, so according to (12.1) we have \frac{1}{2^{r}} |U| < δ, i.e.,

\frac{1}{2^{r}} ⌊\frac{1}{q} · (1 − ε) · 2^{n}⌋ ≤ δ

Hence:

\begin{matrix}\frac{1}{2^{r}} (\frac{1}{q} · (1 − ε) · 2^{n} − 1) ≤ δ \\ \frac{1}{q} · (1 − ε) · 2^{n} ≤ 2^{r} · δ + 1 ≤ 2^{r+1} · δ \end{matrix}

where the last inequality follows from δ ≥ \frac{1}{2^{r}}. After taking logarithms and rearranging:

r + 1 ≥ n + log_{2}(1 − ε) + log_{2}(\frac{1}{δ})− log_{2}(q)

ε is constant, so the claim follows.

2We assume w.l.o.g. that H always output a value which it has queried.

Corollary 12.6 A hitter H with optimal query complexity has R_{H}(ε, δ, n) ≥ n+Ω(log(\frac{1}{δ}))−log_{2}(\frac{1}{ε})

Proof: If we assume the H is optimal with respect to the number of queries it makes, then Q_{H}(ε, δ, n) = O(\frac{1}{ε} log(\frac{1}{δ})). This gives us:

R(H) ≥ n + log_{2}(\frac{1}{δ})− log_{2}(O(\frac{1}{ε} log(\frac{1}{δ})))

And therefore

R(H) ≥ n + log_{2}(\frac{1}{δ})− log_{2}(\frac{1}{ε})− log_{2} log(\frac{1}{δ})− O(1)
= n + Ω(log(\frac{1}{δ})) − log_{2}(\frac{1}{ε})

12.2 Simple Algorithms

In the rest of this lecture, we present several constructions that fulfill the time bound and analyze their query and randomness complexities. This chapter shows three simple constructions.

12.2.1 Constructing a Hitter from a Boolean Sampler

First, we show that any boolean sampler yields a hitter with related complexity. Using a boolean sampler S, we construct the hitter Monitor-Hitter_{S} thus:

Algorithm: Monitor-Hitter_{S}^{f}(ε, δ, n)

Invoke S^{f} with parameters (\frac{ε}{2}, \frac{δ}{2}, n). Monitor the operation of S^{f}: if some query x is answered by f(x) = 1 then output x and halt. If S halts before the above happens, output ⊥.

Proposition 12.2.1 Monitor-Hitter_{S} is a hitter.

Proof: Let S be a boolean sampler, let f ∈ F (n, ε). Let Monitor-Hitter_{S}^{f}(ε, δ, n) be a random variable taken from output probability distribution of Monitor-Hitter_{S}^{f} on input (ε, δ, n). We need to prove that

Pr [Monitor-Hitter_{S}^{f}(ε, δ, n) ∉ f^{−1}(1)] < δ

This is exactly the probability that the invoked sampler S^{f} (\frac{ε}{2}, \frac{δ}{2}, n) does not receive any “1”

answers on its queries. Define the random variable s = S^{f} (\frac{ε}{2}, \frac{δ}{2}, n) to represent the output of S on

input (\frac{ε}{2}, \frac{δ}{2}, n) and access to the oracle f. Then:

Pr [S^{f} receives only 0’s] =

Pr [S^{f} receives only 0’s and s < \frac{ε}{2}] + Pr [S^{f} receives only 0’s and s ≥ \frac{ε}{2}]

(12.2)

Note that f̄ > ε (because f ∈ F (n, ε)), and since S is a sampler we have:

Pr [s < \frac{ε}{2}] < \frac{δ}{2}  (12.3)

On the other hand: in all cases where the coin tosses of S caused S to receive only 0’s, they will cause exactly the same scenario if f were replaced by the constant zero function f_{0} (∀x : f_{0}(x) = 0), for which f̄_{0} = 0. Thus:

Pr [s ≥ \frac{ε}{2} and S receives only 0’s] ≤ Pr [S^{f_{0}}(\frac{ε}{2}, \frac{δ}{2}, n) ≥ \frac{ε}{2}] < \frac{δ}{2}  (12.4)

Combining equations (12.2), (12.3) and (12.4) we get:

Pr [Monitor-Hitter_{S}^{f}(ε, δ, n) ∉ f^{−1}(1)] = Pr [S^{f} receives only 0’s] < δ

Note that using the lower bound Q_{H} ≥ \frac{1}{ε} log (\frac{1}{δ}) on the query complexity of hitters, this

reduction immediately yields a (non-tight) lower bound of Q_{S} ≥ \frac{1}{2ε} log (\frac{1}{2δ} ) on the query complexity

of samplers. Similarly, the R_{H} ≥ n + log_{2}(\frac{1}{δ}) − log_{2}(Q_{H}(ε, δ, n)) − O(1) lower bound on the

randomness complexity of hitters implies a lower bound R_{S} ≥ n + log_{2}(\frac{1}{2δ} )− log_{2}(Q_{S}(2ε, 2δ, n)) − O(1) on the randomness complexity of samplers.

12.2.2 A Naive Hitter

The simplest method to obtain a hitter is to sample random points uniformly, and independently, hoping to hit a good one:

Algorithm: Naive-Hitter^{f}(ε, δ, n)

Set q = \frac{1}{ε} log(\frac{1}{δ}). Randomly choose x_{1}, …, x_{q} ∈ {0, 1}^{n}, uniformly and independently. Query f on x_{1}, …, x_{q}. If f(x_{i}) = 1 for some i then output the first such x_{i}. Otherwise output 0.

Proposition 12.2.2 Naive-Hitter is a hitter.

Proof: For any f ∈ F (n, ε),

Pr [Naive-Hitter^{f}(n, ε, δ) ∈ f^{−1}(1)] = 1 − Pr [∀i, f(x_{i}) = 0] ≥ 1 − (1 − ε)^{q} = 1 − δ

The complexities of Naive-Hitter are:

• QNaive-Hitter(δ, ε, n) = q = \frac{1}{ε} log(\frac{1}{δ}) — this matches the lower bound

• RNaive-Hitter(δ, ε, n) = q · n = \frac{n}{ε} log(\frac{1}{δ})

We see that by simply choosing an appropriate number of samples at random, we attain optimal (up to a constant multiplicative factor) query complexity. However, the randomness complexity is much higher than the lower bound.

12.2.3 A Naive Sampler

A similar approach yields a sampler (and hence a boolean sampler):

Algorithm: Naive-Sampler^{f}(ε, δ, n)

Set q = \frac{1}{ε^{2}} log(\frac{2}{δ}). Randomly choose x_{1}, …, x_{q} ∈ {0, 1}^{n}, uniformly and independently. Query f on x_{1}, …, x_{q}. Output \frac{1}{q} ∑_{i} f(x_{i}).

Proposition 12.2.3 Naive-Sampler is a sampler.

Proof: Observe that for each x_{i}, the expected value of f(x_{i}) is f̄. Using Chernoff’s bound:

Pr [|Naive-Sampler^{f}(n, ε, δ) − f̄| > ε] =

Pr [|\frac{1}{q} ∑_{i} f(x_{i}) − \frac{1}{q} ∑_{i} E(f(x_{i}))| > ε] < 2e^{−ε^{2}q} = δ

The complexities of Naive-Sampler are:

• QNaive-Sampler(ε, δ, n) = q = \frac{1}{ε^{2}} log(\frac{2}{δ}) — this matches the lower bound

• RNaive-Sampler(ε, δ, n) = n · q = \frac{n}{ε^{2}} log(\frac{2}{δ})

Similarly to Naive-Hitter, the query complexity is optimal (up to a constant multiplicative factor) but the randomness complexity is bad.

12.3 Better Constructions

We have seen that the naive approach requires too much randomness. In previous lectures we have seen approaches for generating objects that are “close to random” in various senses while using few random bits. Using these approaches we can build samplers and hitters with reduced complexity.

12.3.1 A Pairwise-Independent Hitter

First we consider small pairwise independent samples spaces (Lectures 3 and 4). For concreteness, consider the construction based on finite fields. We wish to produce pairwise-independent elements from {0, 1}^{n}, so we can use the field GF (2^{n}) ≡ Z_{2}[x]/f(x) where f(x) is an irreducible polynomial in Z_{2}[x]. Assuming we need q ≤ 2^{n} elements, the construction yields a sample space of size merely 2n. That is, only 2n bits of randomness are required to get q pairwise-independent n-bit elements.

Algorithm: Pairwise-Hitter^{f}(ε, δ, n)

Set q = \frac{1}{ε·δ}. If q > 2^{n} then evaluate f(x) for all x ∈ {0, 1}^{n} and output the first x ∈ f^{−1}(1). Otherwise: Generate a sequence of q pairwise independent elements x_{1}, …, x_{q} ∈ {0, 1}^{n}. Query f on x_{1}, …, x_{q}. If for some i, f(x_{i}) = 1, output x_{i}. Otherwise output ⊥.

12.3. BETTER CONSTRUCTIONS 137 Proposition 12.3.1 Pairwise-Hitter is a hitter. Proof: For q > 2^{n}, correctness is obvious. Otherwise, assume f ∈ F (n, ε) and define q random variables ζ_{1}, …, ζ_{q}, where

ζ_{i} = \{ \begin{matrix}1 & if f(x_{i}) = 1 \\ 0 & otherwise \end{matrix}

We have that E(ζ_{i}) = f̄ and Var(ζ_{i}) = E(ζ_{i}^{2}) − E(ζ_{i})^{2} = E(ζ_{i}) − E(ζ_{i})^{2} = f̄ · (1 − f̄). Using Chebychev’s inequality:

Pr [Pairwise-Hitter ∉ f^{−1}(1)] = Pr [\frac{1}{q} ∑_{i} ζ_{i} = 0]
≤ Pr [|\frac{1}{q} ∑_{i} ζ_{i} − f̄| > f̄]
< \frac{Var(ζ_{i})}{q · f̄^{2}} = \frac{f̄· (1 − f̄)}{q · f̄^{2}}
< \frac{f̄}{q · f̄^{2}} = \frac{1}{q·f̄} ≤ \frac{1}{q·ε} = δ

The complexities of Pairwise-Hitter are:

• QPairwise-Hitter(ε, δ, n) ≤ q = \frac{1}{ε·δ}

• RPairwise-Hitter(ε, δ, n) = 2n Note that this randomness complexity is optimal (up to a constant factor) for any value of ε. The query complexity is optimal when δ is constant, but otherwise it is not optimal.

12.3.2 A Pairwise-Independent Sampler

Similarly, we obtain a sampler based on pairwise-independent elements of {0, 1}^{n}.

Algorithm: Pairwise-Sampler^{f}(ε, δ, n)

Set q = \frac{1}{ε^{2}·δ}. If q > 2^{n} then evaluate f(x) for all x ∈ {0, 1}^{n} and output the average. Otherwise: Generate a sequence of q pairwise independent elements x_{1}, …, x_{q} ∈ {0, 1}^{n} using the finite-field construction. Query f on x_{1}, …, x_{q}. Output \frac{1}{q} ∑_{i} f(x_{i}).

Proposition 12.3.2 Pairwise-Sampler is a sampler. Proof: For q > 2^{n} correctness is obvious. Otherwise, define q random variables ζ_{1}, …, ζ_{q}, where

ζ_{i} = \{ \begin{matrix}1 & if f(x_{i}) = 1 \\ 0 & otherwise \end{matrix}

As before, E(ζ_{i}) = f̄ and Var(ζ_{i}) = f̄(1 − f̄). Using Chebychev’s inequality:

Pr [|S^{f}(n, ε, δ) − f̄| > ε] = Pr [|\frac{1}{q} ∑_{i} ζ_{i} − f̄| > ε]
< \frac{Var(ζ_{i})}{q · ε^{2}} = \frac{f̄· (1 − f̄)}{q · ε^{2}}
< \frac{1}{q·ε^{2}} = δ

The complexities of Pairwise-Sampler are:

• QPairwise-Sampler(ε, δ, n) = q = \frac{1}{ε^{2}·δ}

• RPairwise-Sampler = 2n.

As before, this randomness complexity is optimal (up to a constant factor), whereas the query complexity is optimal only when δ is constant.

12.3.3 An Expander-Walk Hitter

Another approach to producing “random-looking” elements using little randomness is random walks on expander graphs (Lectures 9 and 10). For any given ε, we will need a strongly-constructible family of expander graphs \{G_{n}\} with algebraic expansion coefficient λ ≤ \frac{ε}{10} and degree d = poly(\frac{1}{λ}).

This can be obtained directly using the second construction mentioned in Lecture 10 . Alternatively, we can start with any construction that has d = poly(\frac{1}{λ}) and use amplification to achieve sufficiently low λ (i.e., connect each vertex in G_{n} to all nodes in its k-neighbourhood, where a suitable value of k is chosen using Theorem 3 in Lecture 10 and Proposition 3.1 in Lecture 9). The relation d = poly(\frac{1}{λ}) is maintained by the amplification.

Algorithm: Walk-Hitter^{f}(ε, δ, n)

Choose a strongly-constructible expander graph G_{2^{n}} as specified above (λ ≤ \frac{ε}{10}, d = poly(\frac{1}{λ})).

Set q = O (\frac{1}{ε} log(\frac{1}{δ})) (the constant is determined by the proof, see below). Generate a sequence of q elements x_{1}, … x_{q} ∈ {0, 1}^{n} by conducting a random walk of length q on the expander from a random initial point. Query f on x_{1}, …, x_{q}. If for some i, f(x_{i}) = 1, output the first such x_{i}. Otherwise output 0.

Proposition 12.3.3 Walk-Hitter is a hitter.

Proof: Assume f ∈ F (n, ε). Denote B = f^{−1}(0).

Pr [Walk-Hitter ∉ f^{−1}(1)]

= Pr [a walk of length q stays in B] (using Theorem 6 in Lecture 10)

< \sqrt{ρ(B)} · (\sqrt{ρ(B)} + λ)^{q−1}

≤ \sqrt{1 − ε} · (\sqrt{1 − ε} + λ)^{q−1}

< (1 − \frac{1}{3}ε) · (1 − \frac{1}{3}ε + \frac{1}{10}ε)^{q−1}

≤ (1 − \frac{ε}{5})^{q} = δ The complexities of Walk-Hitter are:

• QWalk-Hitter(ε, δ, n) = q = O (\frac{1}{ε} log(\frac{1}{δ}))

• RWalk-Hitter(ε, δ, n) = n + (q − 1) log d = n + O (\frac{1}{ε} log(\frac{1}{δ})log(\frac{1}{ε}))

Note that the query complexity is optimal (up to constant factor), but the randomness complexity is optimal only when ε is constant. This is the reverse of the case for the pairwise-independent constructions above.

12.4 Composed Constructions

We have seen:

Q_{M}  R_{M}

Pairwise-Hitter optimal for constant δ

optimal

Pairwise-Sampler Walk-Hitter optimal

optimal for constant ε

In this section we combine the two constructions, using the “well-handled” parameter (ε or δ) of one to fix the “badly-handled” parameter of the other. We thus obtain composite constructions that are optimal in all parameters.

12.4.1 A Composed Hitter

Pairwise-Hitter uses 2n bits of randomness. Consider a deterministic version d-Pairwise-Hitter of it, which accepts these 2n bits as an additional input. That is,

Pairwise-Hitter^{f}(ε, δ, n) ≡ [r ∈_{R} {0, 1}^{2n} ; return d-Pairwise-Hitter^{f}(ε, δ, n, r)]

We fix the input δ of d-Pairwise-Hitter as \frac{1}{2}, since it is not optimal in this parameter. This gives a natural definition of good values of r:

Definition 12.7 (Good Seed) r ∈ {0, 1}^{2n} is called a good seed (w.r.t. f, ε, n) if d-Pairwise-Hitter^{f}(ε, \frac{1}{2}, n, r) ∈ f^{−1}(1). Denote G =^{def} \{r | r is a good seed\}.

If we come up with a good seed r ∈ G then we can find some x ∈ f^{−1}(1) just by running d-Pairwise-Hitter(ε, \frac{1}{2}, n, r) — this costs \frac{2}{ε} queries and no random bits. Thus we have a new problem: finding elements of G inside \{0, 1\}^{2n}. For f ∈ F (n, ε), Pairwise-Hitter(ε, \frac{1}{2}, n, r) should succeed with probability at least \frac{1}{2} (since we fixed its δ at \frac{1}{2}), so at least half the choices of r are good seeds: |G| ≥ \frac{1}{2}2^{2n}. Hence the problem of finding good seeds is just the job for a hitter with ε = \frac{1}{2}. Walk-Hitter is an attractive choice, since it is optimal for such fixed ε. This yields the following algorithm.

Algorithm: Composed-Hitter^{f}(ε, δ, n)

Define the following deterministic subroutine (using the oracle f and the inputs ε and n):

Check-Seed(r) ≡ [return f (d-Pairwise-Hitter^{f}(ε, \frac{1}{2}, n, r))] where r ∈ \{0, 1\}^{2n}

Compute r ← Walk-HitterCheck-Seed(2n, \frac{1}{2}, δ); that is, execute Walk-Hitter with input (2n, \frac{1}{2}, δ) and answer its oracle queries using Check-Seed. Output d-Pairwise-Hitter(ε, \frac{1}{2}, n, r).

Proposition 12.4.1 Composed-Hitter is a hitter.

Proof: Note that by definition r is a good seed iff Check-Seed(r) = 1. Since f ∈ F (n, ε) and Pairwise-Hitter is a sampler, at least half of the r values are good, i.e., Check-Seed ∈ F (2n, \frac{1}{2}). Thus by Walk-Hitter being a hitter:

Pr [Walk-HitterCheck-Seed(\frac{1}{2}, δ, 2n) ∈ G] > 1 − δ

Hence with probability at least 1 − δ, Walk-Hitter outputs a good seed r, and in this case d-Pairwise-Hitter(ε, \frac{1}{2}, n, r) outputs x such that f(x) = 1.

The complexities of Composed-Hitter are:

• QComposed-Hitter(ε, δ, n) ≤ QWalk-Hitter(\frac{1}{2}, δ, 2n) · QPairwise-Hitter(ε, \frac{1}{2}, n) = O (2 · log(\frac{1}{δ})) · (\frac{2}{ε}) = O(\frac{1}{ε})· log(\frac{1}{δ})

• RComposed-Hitter(ε, δ, n) = RWalk-Hitter(\frac{1}{2}, δ, 2n) = 2n + O (log(\frac{1}{δ}))

Both complexities are optimal, since they match the corresponding lower bounds (up to a constant factor).

12.4.2 A Composed Sampler

We have obtained an optimal hitter by comining two suboptimal hitters. Can we do the same for samplers? There are some difficulties: first, we haven’t seen any sampler with the characteristics of Walk-Hitter (namely, optimal query complexity for constant ε). Furthermore, in the two-level construction we have seen, the first level needs to somehow combine the results of the second level. For hitters this was trivial, but here it’s not clear whether using a sampler in the first level to estimate the average of the second one would work. By appropriate adaptation of the two-level scheme we indeed obtain an optimal sampler.

As before, we use a construction based on pairwise-independent sample spaces for the second level. Pairwise-Sampler uses 2n bits of randomness, so we consider a deterministic version d-Pairwise-Sampler of it which accepts these 2n bits as an additional input. That is,

Pairwise-Sampler^{f}(ε, δ, n) ≡ [r ∈_{R} {0, 1}^{2n} ; return d-Pairwise-Sampler^{f}(ε, δ, n, r)]

Definition 12.8 (Good Seed) r ∈ {0, 1}^{2n} is called a good seed (w.r.t. f, ε, n) if |d-Pairwise-Sampler^{f}(ε, \frac{1}{100}, n, r) − \overline{f}| < ε. Denote G =^{def} \{r | r is a good seed\}.

Unlike the case for hitters, we can’t easily know whether a seed is good. However, we know that most seeds are good, and we can use random walk on an expander graph to obtain a set of seeds in which, with high probability, most seeds are good. This intuition yields the following algorithm.

Algorithm: Composed-Sampler^{f}(ε, δ, n)

Choose a strongly-constructible expander graph G_{2^{2n}} with an algebraic expansion coefficient λ ≤ \frac{1}{100} and degree d = poly(\frac{1}{λ})= \frac{1}{O(1)} (the constants are fixed in the proof below).

Set q = O (log(\frac{1}{δ})) Conduct a random walk of length q on G_{2^{2n}} to generate a sequence of q seeds r_{1}, …, r_{q} ∈ {0, 1}^{2n}. For each i ∈ \{1, …, q\}, compute s_{i} = d-Pairwise-Sampler(ε, \frac{1}{100}, n, r_{i}). Output the median of {s_{1}, …, s_{q}}.

To prove correctness we need the following generalization of Theorem 6 in Lecture 10:

Theorem 12.9 Let G = (V, E) be an expander with algebraic expansion coefficient λ. Let B_{1}, … B_{l} ⊆ V. Let the random variables v_{1}, …, v_{l} denote the vertices encountered in a random walk of length l on G with a random starting point. For any B ⊆ V denote ρ(B) = \frac{|B|}{|V |}. Then:

Pr [∀i : v_{i} ∈ B_{i}] ≤ \sqrt{ρ (B_{1})} ∏_{i=2}^{l} (\sqrt{ρ (B_{i})} + λ)

Proof: The proof of this theorem is very similar to the proof of Theorem 6 in Lecture 10. For a given set B ⊆ V, define the projection:

P_{B} (x) = \{ \begin{matrix}1 & if x ∈ B \\ 0 & otherwise \end{matrix}

We define π to be the vector in R^{V} for which ∀v ∈ V : π_{v} = \frac{1}{n} (i.e., π represents the uniform distribution on V). The following holds (similarly to the proof of Theorem 6 in Lecture 10):

Pr [∀i : v_{i} ∈ B_{i}] = ∥P_{B_{l}}AP_{B_{l−1}}A … P_{B_{1}}π∥_{2}

Recall the following lemma:

Lemma 12.4.2 (Lemma 3.1 in Lecture 10) For every set B ⊆ V and vector α ∈ R^{V}:

∥P_{B}Aα∥_{2} ≤ (\sqrt{ρ (B)} + λ) · ∥α∥_{2}

Denote N = |V |. By l successive applications of Lemma 12.4.2, we obtain:

∥P_{B_{l}}AP_{B_{l−1}}A … P_{B_{1}}π∥_{1} ≤ \sqrt{N} · ∥P_{B_{l}}AP_{B_{l−1}}A … P_{B_{1}}π∥_{2}
≤ \sqrt{N} · ∏_{i=2}^{l} (\sqrt{ρ(B_{l})} + λ) ∥P_{B_{1}}π∥_{2}
= \sqrt{N} · ∏_{i=2}^{l} (\sqrt{ρ(B_{l})} + λ) \sqrt{\frac{ρ(B_{1})}{N}}
= \sqrt{ρ (B_{1})} ∏_{i=2}^{l} (\sqrt{ρ (B_{i})} + λ)

Using Theorem 12.9, we prove the following:

142 LECTURE 12. HITTERS AND SAMPLERS Lemma 12.4.3 Let G = (V, E) be an expander with algebraic expansion coefficient λ. Let B ⊆ V and denote ρ(B) = \frac{|B|}{|V |}. Let the random variables v_{1}, …, v_{l} denote the vertices encountered in a random walk of length l on G with a random starting point. Then:

Pr [|\{i : v_{i} ∈ B\}| ≥ \frac{1}{2}l] ≤ (8\sqrt{ρ(B)} + 8λ)^{\frac{1}{2} l}

Proof: For a given I ⊆ {1, …, l}, we set:

B_{i}^{I} = \{ \begin{matrix}B & if i ∈ I \\ V & otherwise \end{matrix}

Using Theorem 12.9:

Pr [|\{i : v_{i} ∈ B\}| ≥ \frac{1}{2}l] ≤ ∑_{I⊆\{1,...,l\}, |I|≥ \frac{1}{2} l} Pr [∀i ∈ V : v_{i} ∈ B_{i}^{I}]
< ∑_{I⊆\{1,...,l\}, |I|≥ \frac{1}{2} l} \sqrt{ρ(B_{1}^{I})} ∏_{i=2}^{l} (\sqrt{ρ(B_{i}^{I})} + λ)
< ∑_{I⊆\{1,...,l\}, |I|≥ \frac{1}{2} l} ∏_{i=1}^{l} (\sqrt{ρ(B_{i}^{I})} + λ)
< ∑_{I⊆\{1,...,l\}, |I|≥ \frac{1}{2} l} (∏_{i∈I} (\sqrt{ρ(B_{i}^{I})} + λ) · ∏_{i∉I} (\sqrt{ρ(B_{i}^{I})} + λ))
= ∑_{I⊆\{1,...,l\}, |I|≥ \frac{1}{2} l} (∏_{i∈I} (\sqrt{ρ(B)} + λ) · ∏_{i∉I} (1 + λ))
= ∑_{I⊆\{1,...,l\}, |I|≥ \frac{1}{2} l} ((\sqrt{ρ(B)} + λ)^{|I|} · (1 + λ)^{l−|I|})
≤ ∑_{I⊆\{1,...,l\}, |I|≥ \frac{1}{2} l} ((\sqrt{ρ(B)} + λ)^{\frac{1}{2} l} · (1 + λ)^{\frac{1}{2} l})
≤ 2^{l} · (1 + λ)^{\frac{1}{2} l} · (\sqrt{ρ(B)} + λ)^{\frac{1}{2} l}

Since λ < 1, we obtain:

Pr [|\{i : v_{i} ∈ B\}| ≤ \frac{1}{2}l] < 2^{\frac{3}{2} l} (\sqrt{ρ(B)} + λ)^{\frac{1}{2} l}
= (8\sqrt{ρ(B)} + 8λ)^{\frac{1}{2} l}

To prove correctness of our sampler we also need the following technical fact:

Fact 12.4.4 Let A = {a_{1}, …, a_{n}} be a set of real numbers and x < y be also real numbers. If there exists a subset I ⊆ A s.t. |I| > \frac{1}{2}n and all elements in I fall in the interval (a, b) then the median of A also falls in this interval. In other words, if |\{a ∈ A : x < a < y\}| > \frac{1}{2}n, then x < median(A) < y.

Proof: Define:

G = {a ∈ A | a ≥ median(A)}
L = {a ∈ A | a ≤ median(A)}

By the definition of a median: |G| ≥ ⌈\frac{1}{2}n⌉ and |L| ≥ ⌈\frac{1}{2}n⌉. Since |I| > \frac{1}{2}n, there exist some g ∈ G ∩ I and l ∈ L ∩ I. Because g ∈ I we have g < y, so using g ∈ G we get median(A) ≤ g < y. Similarly, because l ∈ I we have x < l, so using l ∈ L we get median(A) ≥ l > x. The claim follows.

We can now prove correctness of the sampler.

Proposition 12.4.5 Composed-Sampler is a sampler.

Proof: Consider the execution of Composed-Sampler^{f}(ε, δ, n) with some f : {0, 1}^{n} → {0, 1}.

Recall that the algorithm produces q seeds {r_{1}, …, 1_{q}} and runs d-Pairwise-Sampler on each of them to obtain corresonding values {s_{1}, …, r_{q}}. For each seed r_{i}, if r_{i} is a good seed then by definition: |s_{i} − f̄| < ε. If strictly more than one half of {r_{1}, …, r_{q}} are good seeds then by Fact 12.4.4, the output s = median{s_{1}, …, s_{q}} also satisfies |s − f̄| < ε. Thus, it suffices to show that with probability at least δ, strictly more than one half of {r_{1}, …, r_{q}} are good seeds.

For every ε, n and f ∈ F (n), at least \frac{99}{100} of the seeds are good (by Pairwise-Sampler(ε, \frac{1}{100}, n) being a sampler), i.e., |G| ≥ \frac{99}{100}|\{0, 1\}^{2n}|. Letting B = \{0, 1\}^{2n} \ G be the set of “bad” seeds, we obtain ρ(B) ≤ \frac{1}{100}. Thus, by invoking Lemma 12.4.3 on G_{2n} we obtain:

Pr [more than one half of \{r_{1}, …, r_{q}\} are good seeds] = 1 − Pr [|\{i : r_{i} ∈ B\}| > \frac{1}{2}q]
≤ (8\sqrt{ρ(B)} + 8λ)^{\frac{1}{2} q}

≤ (8 · \sqrt{\frac{1}{100}} + 8 · \frac{1}{100})^{\frac{1}{2} q}
≤ 0.88^{\frac{1}{2} q} = δ

q is chosen as to satisfy the last inequality.

The complexities of Composed-Sampler are as follow (where q = O (log(\frac{1}{δ})) is the number of tested seeds):

• QComposed-Sampler(ε, δ, n) = q · QPairwise-Sampler(n, ε, \frac{1}{100})) = O (log(\frac{1}{δ})) · \frac{1}{ε^{2}·100} =

O (log(\frac{1}{δ})· \frac{1}{ε^{2}} )

• RComposed-Sampler(ε, δ, n) = 2n + O (log(\frac{1}{δ})· log(d)) = 2n + O (log(\frac{1}{δ}))

This result is optimal, since it matches the lower bounds (up to a constant factor).

144 LECTURE 12. HITTERS AND SAMPLERS

Bibliographic Notes

Our discussion of hitters and samplers is based on [20]. The median-of-averages sampler is due to Bellare, Goldreich and Goldwasser [11].

Randomized Rounding

Notes taken by Ido Shaked, Ariel Elbaz and David Lehmann

Summary: Some approximation algorithms use linear programming on a variant of the original problem. The solutions of a linear programming problem are non-integral, in the general case, while legitimate solutions to the original problems are integers only. One idea which can be used to turn real numbers into integer solutions, is to use the linear programming (non-integer) solutions as probability parameters for selecting (integer) solutions to the original problem. In this lecture we present this idea and demonstrate it in two cases.

13.1 The General Idea

Given an instance of a boolean optimization problem (e.g. Max-SAT which is NP-Hard), we may reduce that problem to an analogue problem of Integer Linear Programming (ILP). Doing so, we arrive at an equivalent problem for which all the feasible solutions are integers in the range [0, 1] which means only 0,1, and such that every optimal solution for the ILP problem expresses an optimal solution for the original boolean problem. We then treat that problem as an instance of a simple Linear Programming, by dropping the integrality requirement, (note that this problem is known to be in P) and solve that problem, getting a solution expressed as Real numbers in the range [0,1]. We then use Random Rounding, a method we describe here, to convert the solution to a solution in integers 0,1 (which translates to a solution in T,F for the boolean problem), and which will have a good chance of becoming a ”good” solution (one that is relatively ”close” to the optimal solution).

13.2 Application to Max-SAT

Definition 13.1 Max-SAT:

Input: A boolean formula φ over the variables \overrightarrow{X} = (x_{1}, ..., x_{n}) with the following structure

φ = ⋀_{j=1}^{m} C_{j} = ⋀_{j=1}^{m}( ⋁_{i∈P_{j}} x_{i}) ∨ ( ⋁_{i∈N_{j}} ¬x_{i})

where N_{j} and P_{j} are the sets of negated and unnagated variables in clause C_{j} respectively.

Desired Output: A truth assignment for \overrightarrow{X} that satisfies the maximal number of clauses C_{j}.

13.2.1 The Basic Method

We now translate an instance of Max-SAT to an analogue instance of Integer Linear Programming (ILP), as follows.

max{∑_{j=1}^{m} y_{j}}

st (i) ∀j ∈ [m] : 0 ≤ y_{j} ≤ 1, y_{j} ∈ Z
(ii) ∀i ∈ [n] : 0 ≤ x_{i} ≤ 1, x_{i} ∈ Z
(iii) ∀j ∈ [m] : y_{j} ≤ ∑_{i∈P_{j}} x_{i} + ∑_{i∈N_{j}}(1 − x_{i}).

Any optimal solution to the ILP problem is an optimal solution to the Max-SAT problem, and vice versa.

Oded’s Note: Looking at the ILP formulation, Condition (ii) forces the x-variables to be Boolean (or have values in {0, 1}), whereas the other conditions force a correspondance between the y_{j}’s and indicators representing whether the corresponding clauses are satisfied by the assignment to the x_{i}’s.

We now remove the integrality requirement and arrive at the following Linear Programming (LP) problem:

max{∑_{j=1}^{m} y_{j}}

st (i) ∀j ∈ [m] : 0 ≤ y_{j} ≤ 1
(ii) ∀i ∈ [n] : 0 ≤ x_{i} ≤ 1
(iii) ∀j ∈ [m] : y_{j} ≤ ∑_{i∈P_{j}} x_{i} + ∑_{i∈N_{j}}(1 − x_{i}).

It is known that LP is solvable in polynomial time using, say, the Ellipsoid method. Thus:

Theorem 13.2 LP is in P.

Denote the optimal solution to the LP problem X̂ = (x̂_{1}, ..., x̂_{n}), Ŷ = (ŷ_{1}, ..., ŷ_{m}). Note that x̂_{i}’s are in the range [0,1]. We will use those values as probabilities when rounding to

integers 0 and 1. Suppose \overrightarrow{Y^{∗}} = (y_{1}^{∗}, …, y_{n}^{∗}) is an optimal solution to the ILP. Clearly ∑_{j=1}^{n} ŷ_{j} ≥ ∑_{j=1}^{n} y_{j}^{∗}.

Randomized Rounding: We obtain a new integer solution by rounding: every x_{i} is determined

by x_{i} = \{ \begin{matrix}1 & with probability x̂_{i} \\ 0 & otherwise \end{matrix} .

This technique is called Randomized Rounding (R.R.).

13.2. APPLICATION TO MAX-SAT 147 Definition 13.3 Size of Clause: c_{j} =^{def} |P_{j}| + |N_{j}|. Lemma 13.2.1 Every clause C_{j} is satisfied, with probability at least (1 − \frac{1}{e}) · ŷ_{j}, by the assignment obtained from the Randomized Rounding. That is:

Pr[C_{j} is satisfied by R.R.(X̂)] ≥ (1 − (1 − \frac{1}{c_{j}})^{c_{j}}) · ŷ_{j}
≥ (1 − \frac{1}{e}) · ŷ_{j}

As a direct corollary to Lemma 13.2.1, we get: Corollary 13.4 The expected fraction of clauses that are satisfied by Randomized Rounding on an optimal LP solution is greater than 0.6. Proof: The expected number of clauses that are satisfied by Randomized Rounding is

∑_{j=1}^{m} Pr[C_{j} is satisfied by R.R.(X̂)]

≥ ∑_{j=1}^{m}(1 − \frac{1}{e}) · ŷ_{j}
= (1 − \frac{1}{e}) · ∑_{j=1}^{m} ŷ_{j}
≥ (1 − \frac{1}{e}) · ∑_{j=1}^{m} y_{j}^{∗} ≈ 0.6 · ∑_{j=1}^{m} y_{j}^{∗}

Proof of Lemma 13.2.1: Fixing any j ∈ [m], we have.

Pr[C_{j} is not satisfied by R.R.(X̂)]

= ∏_{i∈P_{j}} Pr[x_{i} = 0] · ∏_{i∈N_{j}} Pr[x_{i} = 1]
= ∏_{i∈P_{j}}(1 − x̂_{i}) · ∏_{i∈N_{j}} x̂_{i}

Define, for every i in P_{j}∪N_{j}, ẑ_{i} = \{ \begin{matrix}1 − x̂_{i} & i ∈ P_{j} \\ x̂_{i} & i ∈ N_{j} \end{matrix} . By Condition (iii), we have

ŷ_{j} ≤ ∑_{i∈P_{j}} x̂_{i} + ∑_{i∈N_{j}}(1 − x̂_{i}) = ∑_{i∈P_{j}∪N_{j}}(1 − ẑ_{i})

and hence

∑_{i∈P_{j}∪N_{j}} ẑ_{i} ≤ c_{j} − ŷ_{j}.

Clearly

∏_{i∈P_{j}}(1 − x̂_{i}) · ∏_{i∈N_{j}} x̂_{i}

= ∏_{i∈P_{j}∪N_{j}} ẑ_{i}
≤ max ; \begin{matrix}\overrightarrow{W} = (w_{1}, …, w_{n}) \\ ∑_{i∈P_{j}∪N_{j}} w_{i} ≤ c_{j} − ŷ_{j} \end{matrix} \{ ∏_{i∈P_{j}∪N_{j}} w_{i}\}

Note that the product of m non-negative variables, when their sum is bounded, is maximized when they are all equal. Thus, since we have ∑_{i∈P_{j}∪N_{j}} w_{i} ≤ c_{j} − ŷ_{j}, we derive the maximum \overrightarrow{W^{∗}} to be when all w_{i}^{∗} are equal; that is, w_{i}^{∗} = (\frac{c_{j}−ŷ_{j}}{c_{j}}) for every i ∈ P_{j}∪N_{j}. It follows that

∏_{i∈P_{j}∪N_{j}} w_{i}^{∗} = (\frac{c_{j} − ŷ_{j}}{c_{j}})^{c_{j}} = (1 − \frac{ŷ_{j}}{c_{j}})^{c_{j}}

So far we have proven that for every clause C_{j},

Pr[C_{j} is satisfied by R.R.(X̂)] ≥ 1 − (1 − \frac{ŷ_{j}}{c_{j}})^{c_{j}}

To complete our proof we need to show that

1 − (1 − \frac{ŷ_{j}}{c_{j}})^{c_{j}} ≥ (1 − (1 − \frac{1}{c_{j}})^{c_{j}}) · ŷ_{j} ≥ (1 − \frac{1}{e}) · ŷ_{j}  (13.1)

We will use the equality : 1^{c} − b^{c} = (1 − b) · ∑_{i=1}^{c−1} b^{i}. (*) Specifically, applying (*) for b = (1 − \frac{ŷ_{j}}{c_{j}}) and c = c_{j}, we get:

1 − (1 − \frac{ŷ_{j}}{c_{j}})^{c_{j}} = (1 − (1 − \frac{ŷ_{j}}{c_{j}})) · [ ∑_{i=1}^{c_{j}−1}(1 − \frac{ŷ_{j}}{c_{j}})^{i}]
= \frac{ŷ_{j}}{c_{j}} · [ ∑_{i=1}^{c_{j}−1}(1 − \frac{ŷ_{j}}{c_{j}})^{i}]
≥ \frac{ŷ_{j}}{c_{j}} · [ ∑_{i=1}^{c_{j}−1}(1 − \frac{1}{c_{j}})^{i}]
= \frac{1}{c_{j}} · [ ∑_{i=1}^{c_{j}−1}(1 − \frac{1}{c_{j}})^{i}] · ŷ_{j}
= (1 − (1 − \frac{1}{c_{j}})^{c_{j}}) · ŷ_{j}

where the last equality is due to applying (*) for b = (1 − \frac{1}{c_{j}}) and c = c_{j}.

This proves the first inequality in Eq. (13.1). The second inequality (i.e., (1 − (1 − \frac{1}{c_{j}})^{c_{j}}) · ŷ_{j} ≥ (1 − \frac{1}{e}) · ŷ_{j}) is derived from the fact that, for every x ≥ 1, it holds that (1 − \frac{1}{x})^{x} < \frac{1}{e}.

13.2.2 Improving the Max-SAT approximation algorithm

The above approximation algorithm achieves an approximation ratio of (1 − \frac{1}{e}) of the optimal solution. Yet there is a known approximation algorithm with an approximation ratio of \frac{3}{4}. We wish to modify the algorithm we have just seen, to achieve an approximation ratio of \frac{3}{4}.

Consider a random assignment for all the variables, i.e. \overrightarrow{X} ∈_{R} \{T, F \}^{n}. Each clause C_{j} is not satisfied if and only if every literal in it has a false value. If we assign each variable X_{i}, independently, a random value from {T, F }, the probability that C_{j} is not satisfied is 2^{−c_{j}} (c_{j}, as before, is the number of literals in C_{j}), hence the probability that C_{j} is satisfied is (1 − 2^{−c_{j}}).

We note that for any j, when using a random assignment, the probability of C_{j} being satisfied increases with the size of the clause, c_{j}. However, using the assignment we got from solving the linear programming problem, the probability of C_{j} being satisfied decreases with c_{j}. Noting this difference calls for some combination of the two methods, to get a method that gives higher probability for each clause being satisfied regardless of it’s size.

A \frac{3}{4}-approximation algorithm: Given a boolean formula φ = ⋀_{j=1}^{m} C_{j} as above, we first run the previous algorithm, using Linear Programming and Randomized Rounding, and get a solution

\overrightarrow{X^{1}}. We also select a random assignment \overrightarrow{X^{2}} ∈_{R} \{T, F \}^{n}. We output \overrightarrow{X^{1}} with probability \frac{1}{2}, \overrightarrow{X^{2}} with probability \frac{1}{2}.

Analysis We claim that the expected number of satisfied clauses is at least \frac{3}{4} of the optimum. We define a new random variable v_{j}^{1}, which, for each clause C_{j}, will be 1 if the clause is

satisfied using the assignment \overrightarrow{X^{1}}, or 0 otherwise. We use v_{j}^{1}’s to write an expression for the expected number of satisfied clauses, which is simply the sum of these variables E(∑_{j=1}^{m} v_{j}^{1}), which

is equal to ∑_{j=1}^{m} E(v_{j}^{1}). The expected number of satisfied clauses, when using \overrightarrow{X^{1}}, is at least ∑_{j=1}^{m}(1 − (1 − \frac{1}{c_{j}})^{c_{j}}) · ŷ_{j}, as shown above.

We also define the variables v_{j}^{2}’s, for use with the assignment \overrightarrow{X^{2}}. Using \overrightarrow{X^{2}}, we’ve seen that for every j E(v_{j}^{2}) = (1 − 2^{−c_{j}}).

Taking each of the two assignments with probability \frac{1}{2}, the expected number of satisfied clauses is

E(∑_{j=1}^{m} v_{j}) = \frac{1}{2} · E (∑_{j=1}^{m} v_{j}^{1}) + \frac{1}{2} · E (∑_{j=1}^{m} v_{j}^{2})

≥ \frac{1}{2} · ∑_{j=1}^{m}(1 − (1 − \frac{1}{c_{j}})^{c_{j}}) · ŷ_{j} + \frac{1}{2} · ∑_{j=1}^{m}(1 − 2^{−c_{j}})

≥ ∑_{j=1}^{m} (\frac{1 − (1 − \frac{1}{c_{j}})^{c_{j}}}{2} + \frac{1 − 2^{−c_{j}}}{2} ) · ŷ_{j}

≥ ∑_{j=1}^{m} \frac{3}{4} · ŷ_{j}

= \frac{3}{4} · ∑_{j=1}^{m} ŷ_{j}

where the last inequality uses the following claim:

Claim 13.2.2 For any positive integer x it holds that \frac{1}{2}(1 − (1 − \frac{1}{x})^{x}) + \frac{1}{2}(1 − 2^{−x}) ≥ \frac{3}{4}.

Proof: It is sufficient to prove that for any positive integer x it holds that (1 − \frac{1}{x})^{x} + 2^{−x} ≤ \frac{1}{2}. The claim can easily be checked for x = 1 and 2.

Oded’s Note: For x = 1 we have (1 − \frac{1}{x})^{x} + 2^{−x} = 0 + 0.5 = 0.5, whereas for x = 2 we have (1 − \frac{1}{x})^{x} + 2^{−x} = 0.5^{2} + 2^{−2} = 0.5.

For any x ≥ 3, we have

(1 − \frac{1}{x})^{x} + 2^{−x} < e^{−1} + 2^{−3} < \frac{1}{2}

13.3 Application to the Minimal-Set-Cover problem

Definition 13.5 Minimal Set Cover: Given n and {S_{j}}_{j=1}^{m}, a collection of sets which cover [n], a minimal set cover is a minimal sub-collection, J, which still covers [n]; that is, a minimal J for which ⋃_{J} S_{j} = [n].

13.3.1 The approximation algorithm

We will go by the same path as we did for the MaxSAT approximation algorithm: first we translate the problem to an ILP problem, then solve the related LP problem, and finally use Randomized Rounding to get an Integer solution from the (non Integer) solution of the LP problem.

For every i, we name C_{i} to be the collection of sets that contain i. Now we rephrase the minset-cover problem: to cover every i, we must take into J at least one of the subsets S_{j} that contain i, and these are all in C_{i}. We also should find a minimal such J. We can write the rephrased problem as an Integer Linear Programming problem:

min{ ∑_{j∈[m]} y_{j}}

st (i) ∀j ∈ [m] : 0 ≤ y_{j} ≤ 1 , y_{j} ∈ Z
(ii) ∀i ∈ [n] : ∑_{j∈C_{i}} y_{j} ≥ 1

From a solution to this system corresponds to a cover J = {S_{j} | y_{j} = 1}. Requirement (ii) ensures that J indeed covers every i, and the minimal sum ensures the minimality of J.

We now remove the integrality requirement from the y_{j}’s, and get a LP problem. We denote by Opt the optimal solution of the LP problem. That is, denoting the optimal LP solution by ŷ_{j}’s,

we have Opt =^{def} ∑_{j∈[m]} ŷ_{j}.

Applying Randomized Rounding. The ŷ_{j}’s can be seen as a measure of the importance of S_{j} in the cover. However, should we, as previously in the Max-SAT problem, choose J simply by taking each S_{j} with probability ŷ_{j}, we will end up with J that has a small expected size (i.e., Opt which is at most as big as the optimum of the ILP), but will probably leave some of the i’s uncovered. For example, with C_{i} = {1, 2} and a possible solution ŷ_{1} = ŷ_{2} = 0.5, with probabilty 1/4 none of these sets is taken (and the element i remains uncovered).

To ensure that no element remains uncover, we will take a larger random cover. Specifically, we will include S_{j} in J with probability min{1, K · ŷ_{j}} (rather than with probability ŷ_{j}), where

K =^{def} 2 ln(2n).

13.3.2 Analysis

Claim 13.3.1 The expected size of this cover is, at most, K · Opt.

Proof: E(|J|) = ∑_{j=1}^{m} Pr(j ∈ J) ≤ ∑_{j=1}^{m} K · ŷ_{j} = K · Opt.

Claim 13.3.2 For every i, Pr(i isn’t covered) ≤ \frac{1}{2n}.

Proof: If, for any j in C_{i}, ŷ_{j} is at least \frac{1}{K}, then S_{j} is always in J (because K · ŷ_{j} ≥ 1) and i is always covered. We therefore focus on the case where ŷ_{j} < \frac{1}{K}.

Let us define, for every j in C_{i}, a new random variable:

η_{j} = \{ \begin{matrix}1 & with probability K · ŷ_{j} \\ 0 & otherwise \end{matrix}

The above is well-define because K · ŷ_{j} < 1. Furthermore, since K · ŷ_{j} = min(K · ŷ_{j}, 1), the random variable η_{j} indicates whether or not j was taken to J. Thus, for every i in [n]:

Pr(i isn’t covered) = Pr ( ∑_{j∈C_{i}} η_{j} = 0) .  (13.2)

We will use the following Chernoff bound

A Chernoff bound: For independent 0-1 random variables x_{1}, ..., x_{n}, with λ = E(∑_{i=1}^{n} x_{i}) and any ϵ ∈ [0, 1], it holds that

Pr [∑_{i=1}^{n} x_{i} ≤ (1 − ϵ)λ] ≤ e^{\frac{−ϵ^{2}λ}{2}}

In our case, we have λ =^{def} E(∑_{j∈C_{i}} η_{j}) = K · ∑_{j∈C_{i}} ŷ_{j}, which by Condition (ii) is at least K. Applying the above bound (with ϵ = 1), we get

Pr[ ∑_{j∈C_{i}} η_{j} = 0] = Pr[ ∑_{j∈C_{i}} η_{j} ≤ 0]
≤ e^{−λ/2} ≤ e^{−K/2}

Recalling that K = 2 ln(2n), we have e^{−K/2} = 1/2n and the claim follows.

Conclusion: We have shown that the probability that any specific i is not covered is at most \frac{1}{2n}. Using the union bound, it follows that the probability that there is an uncovered i ∈ [n] is at most \frac{1}{2}. Thus, J is a collection of subsets S_{j}, with expected size no more than 2 ln(2n) · Opt, which, with probability at least \frac{1}{2}, covers [n]. Furthermore, with constant probability, J has size O(log n) · Opt and covers all [n].

152 LECTURE 13. RANDOMIZED ROUNDING

Bibliographic Notes

The Randomized Rounding technique was introduced by Raghavan and Thompson [39]. The application to MaxSAT is due Goemans and Williamson [19].

Bibliography

[1] L.M. Adleman and M. Huang. Primality Testing and Abelian Varieties Over Finite Fields. Springer-Verlag Lecture Notes in Computer Science (Vol. 1512), 1992. Preliminary version in 19th ACM Symposium on the Theory of Computing, 1987.

[2] L.M. Adleman, K. Manders and G. Miller. On Taking Roots in Finite Fields. In 18th IEEE Symposium on Foundations of Computer Science, pages 175–177, 1977

[3] M. Ajtai, J. Komlos, E. Szemerédi. Deterministic Simulation in LogSpace. In 19th ACM Symposium on the Theory of Computing, pages 132–140, 1987.

[4] R. Aleliunas, R.M. Karp, R.J. Lipton, L. Lovász and C. Rackoff. Random walks, universal traversal sequences, and the complexity of maze problems. In 20th IEEE Symposium on Foundations of Computer Science, pages 218–223, 1979.

[5] N. Alon. Eigenvalues and expanders. Combinatorica, Vol. 6, pages 83–96, 1986.

[6] N. Alon, L. Babai and A. Itai. A fast and Simple Randomized Algorithm for the Maximal Independent Set Problem. J. of Algorithms, Vol. 7, pages 567–583, 1986.

[7] N. Alon, J. Bruck, J. Naor, M. Naor and R. Roth. Construction of Asymptotically Good, Low-Rate Error-Correcting Codes through Pseudo-Random Graphs. IEEE Transactions on Information Theory, Vol. 38, pages 509–516, 1992.

[8] N. Alon, O. Goldreich, J. Håstad, R. Peralta. Simple Constructions of Almost k-wise Independent Random Variables. Journal of Random structures and Algorithms, Vol. 3, No. 3, (1992), pages 289–304.

[9] N. Alon and V.D. Milman. λ_{1}, Isoperimetric Inequalities for Graphs and Superconcentrators, J. Combinatorial Theory, Ser. B, Vol. 38, pages 73–88, 1985.

[10] N. Alon and J.H. Spencer. The Probabilistic Method, John Wiley & Sons, Inc., 1992.

[11] M. Bellare, O. Goldreich, and S. Goldwasser. Randomness in Interactive Proofs. Computational Complexity, Vol. 4, No. 4, pages 319–354, 1993.

[12] M. Bellare, O. Goldreich, and E. Petrank. Uniform Generation of NP-witnesses using an NP-oracle. Inform. and Comp., Vol. 163, pages 510–526, 2000.

[13] C.H. Bennett, G. Brassard and J.M. Robert. Privacy Amplification by Public Discussion. SIAM Journal on Computing, Vol. 17, pages 210–229, 1988.

[14] R. Canetti, G. Even and O. Goldreich. Lower Bounds for Sampling Algorithms for Estimating the Average. Information Processing Letters, Vol. 53, pages 17–25, 1995.

[15] L. Carter and M. Wegman. Universal Hash Functions. Journal of Computer and System Science, Vol. 18, 1979, pages 143–154.

[16] B. Chor and O. Goldreich. On the Power of Two–Point Based Sampling. Jour. of Complexity, Vol 5, 1989, pages 96–106. Preliminary version dates 1985.

[17] A. Cohen and A. Wigderson. Dispensers, Deterministic Amplification, and Weak Random Sources. In 30th IEEE Symposium on Foundations of Computer Science, 1989, pages 14–19.

[18] O. Gaber and Z. Galil. Explicit Constructions of Linear Size Superconcentrators. Journal of Computer and System Science, Vol. 22, pages 407–420, 1981.

[19] M. Goemans and D. Williamson. New 3/4-approximation algorithms for the maximum satisfiablity problem. SIAM Journal on Discrete Mathematics, Vol. 7, No. 4, pages 656–666, 1994.

[20] O. Goldreich. A Sample of Samplers – A Computational Perspective on Sampling. ECCC, TR97-020, May 1997.

[21] O. Goldreich. Modern Cryptography, Probabilistic Proofs and Pseudorandomness. Algorithms and Combinatorics series (Vol. 17), Springer, 1999.

[22] O. Goldreich, R. Impagliazzo, L.A. Levin, R. Venkatesan, and D. Zuckerman. Security Preserving Amplification of Hardness. In 31st IEEE Symposium on Foundations of Computer Science, pages 318–326, 1990.

[23] O. Goldreich and A. Wigderson. Tiny Families of Functions with Random Properties: A Quality–Size Trade–off for Hashing. Journal of Random structures and Algorithms, Vol. 11, Nr. 4, December 1997, pages 315–343.

[24] S. Goldwasser and J. Kilian. Primality Testing Using Elliptic Curves. Journal of the ACM, Vol. 46, pages 450–472, 1999. Preliminary version in 18th ACM Symposium on the Theory of Computing, 1986.

[25] J. Håstad, S. Phillips and S. Safra. A Well Characterized Approximation Problem. Information Processing Letters, Vol. 47:6, pages 301–305. 1993.

[26] R. Impagliazzo, L.A. Levin and M. Luby. Pseudorandom Generation from One-Way Functions. In 21st ACM Symposium on the Theory of Computing, pages 12–24, 1989.

[27] R. Impagliazzo and D. Zuckerman. How to Recycle Random Bits. In 30th IEEE Symposium on Foundations of Computer Science, 1989, pages 248–253.

[28] M. Jerrum, L. Valiant and V. Vazirani. Random Generation of Combinatorial Structures from a Uniform Distribution. Theoretical Computer Science, Vol. 43, pp. 169–188, 1986.

[29] A. Joffe. On a set of almost deterministic k-independent random variables. Annals of Probability, Vol. 2 (1), pages 161–162, 1974.

[30] N. Kahale, Eigenvalues and Expansion of Regular Graphs. Journal of the ACM, 42(5):1091– 1106, September 1995.

[31] A. Lubotzky, R. Phillips, P. Sarnak, Ramanujan Graphs. Combinatorica, Vol. 8, pages 261–277, 1988.

[32] F.J. MacWilliams and N.J.A. Sloane. The Theory of Error-Correcting Codes. North-Holland, Amsterdam, The Netherlands, 1977.

[33] G.A. Margulis. Explicit Construction of Concentrators. Prob. Per. Infor. 9 (4) (1973), 71–80. (In Russian, English translation in Problems of Infor. Trans. (1975), 325–332.)

[34] R. Motwani and P. Raghavan. Randomized Algorithms, Cambridge University Press, 1995.

[35] J. Naor and M. Naor. Small-bias Probability Spaces: Efficient Constructions and Applications. SIAM J. on Computing, Vol 22, 1993, pages 838–856.

[36] V. Pratt. Every Prime has a Succinct Certificate. SIAM Journal on Computing, Vol. 4, pages 214–220, 1975.

[37] M.O. Rabin. Digitalized Signatures and Public Key Functions as Intractable as Factoring. MIT/LCS/TR-212, 1979.

[38] M.O. Rabin. Probabilistic Algorithm for Testing Primality. Journal of Number Theory, Vol. 12, pages 128–138, 1980.

[39] P. Raghavan and C.D. Thompson. Randomized Rounding. Combinatorica, Vol. 7, pages 365–374, 1987.

[40] M. Sipser. A Complexity Theoretic Approach to Randomness. In 15th ACM Symposium on the Theory of Computing, pages 330–335, 1983.

[41] R. Solovay and V. Strassen. A Fast Monte-Carlo Test for Primality. SIAM Journal on Computing, Vol. 6, pages 84–85, 1977. Addendum in SIAM Journal on Computing, Vol. 7, page 118, 1978.

[42] L. Stockmeyer. On Approximation Algorithms for #P. SIAM Journal on Computing, Vol. 14 (4), pages 849–861, 1985. Preliminary version in 15th ACM Symposium on the Theory of Computing, pages 118–126, 1983.

[43] U.V. Vazirani. Randomness, Adversaries and Computation. Ph.D. Thesis, EECS, UC Berkeley, 1986.

[44] A. C. Yao. Probabilistic computations: Towards a Unified Measure of Complexity. In 17th FOCS, pages 222–227, 1977.