Lecture #8: Balls and Bins
1. Balls and Bins
The setting is simple: balls, bins. When you consider a ball, you pick a bin independently and uniformly at random, and add the ball to that bin. In HW #2 you proved:
One could use a Chernoff bound to prove this, but here is a more direct calculation of this theorem: the chance that bin has at least balls is at most
which is (say) for . To see this, note that
So union bounding over all the bins, the chance of some bin having more than balls is . (I’ve been sloppy with constants, you can do better constants by using Stirling’s approximation.)
Here is a semantically identical way of looking at this calculation: let be the indicator r.v. for bin having or more balls. Then . And hence if , then . So by Markov, . In other words, we again have
This idea of bounding the expectation of some variable , and using that to upper bound some quantity (in this case the max-load) is said to use the first moment method.
1.1. Tightness of the Bound
In fact, is indeed the right answer for the max-load with balls and bins.
Here is one way to show this, via the second moment method. To begin, let us now lower bound the probability that bin has at least balls:
which for is at least , since . And so we expect bins to have at least balls.
Let us define some random variables: if is the indicator for bin having at least balls, and is the expected number of bins with at least balls, we get that
Alas, in general, just knowing that will not imply . Indeed, consider a random variable that is w.p. , and otherwise—while its expectation is , is more and more likely to be zero as increases. So we need some more information about to prove our claim. And that comes from the second moment.
Let’s appeal to Chebyshev’s inequality:
You have probably seen covariance before: . But since the bins are negatively correlated (some bin having more balls makes it less likely for another bin to do so), the covariance . Moreover, since , ; by the above calculations, . So summarizing, we get
In other words, there is a chance that some bin contains more than balls:
(Later, you will see how to use martingale arguments and Azuma-Hoeffding bounds to give guarantees on the max-load of bins. You can also use the “Poisson approximation” to show such a result, that’s yet another cool technique.)
1.2. So, in Summary
If you want to show that some non-negative random variable is zero with high probability, show that it’s expectation is tends to zero, and use Markov—the first moment method. If you want to show that it is non-zero with high probability, show that the variance divided by the squared mean tends to zero, and use Chebyshev—the second moment method.
1.3. Taking it to the Threshold
Such calculations often arise when you have a random process, and a random variable defined in terms of a parameter . Often you want to show that is zero whp when lies much below some “threshold” , and that is non-zero whp when is far above . The first things you should try are to see if the first and second moment methods give you rough answers. E.g., take vertices and add each of the edges independently with probability (also called the Erdös-Rényi graph ), and define to be the expected number of cliques on vertices. Show that is such a threshold for .
2. The Power of Two Choices
The setting now is: balls, bins. However, when you consider a ball, you pick two bins (or in general, bins) independently and uniformly at random, and put the ball in the less loaded of the two bins. The main theorem is:
The intuition behind the proof is the following picture:
The actual proof is not far from this intuition. The following lemma says that if at most fraction of the bins have at least balls, then the fraction of bins having balls can indeed be upper bounded by , where is the Binomial random variable.
Proof: For the proof, let us consider the “heights” of balls: this is the position of the ball when it comes in, if it is the first ball in its bin then its height is , etc. Observe that if there are bins with load , then there must be at least balls with height . I.e., if is the number of balls with height at least , then , and so we’ll now upper bound .
Consider the following experiment: just before a ball comes in, an adversary is allowed to “mark” at most bins. Call a ball marked if both its random bins are marked. Note that when we condition on , we know that the final number of bins with load at least is at most . In this case, we can imagine the adversary marking the bins with load at least (and maybe some more). Now the chance that a ball is marked is at least the chance that it has height and there are at most bins with height at least . Hence, if is the number of marked balls, we get
The second equality follows from the fact that .
If you’d like to be more precise about proving (*) above, see the details in the notes from the Mitzenmacher-Upfal. (CMU/Pitt access only.)
Now we can use Chernoff to prove tail bounds on the Binomial distribution.
Proof: We’re interested in where each w.p. , and otherwise. The expectation . And the chance that this number exceeds is at most
which proves the first part. For the second part, , and the probability that exceeds is at most
So, now let us define to be the fraction of bins we’re aiming to show have load at least . Define , and . (The reason it is instead of , which is the expectation, is for some breathing room to apply Chernoff: in particular, the factor comes from the first part of Lemma 5.)
For each , let be the good event “”; recall that is the number of bins with load at least . We want to lower bound the probability that this good event does not happen.
Proof: We prove this by induction. The base case is when , when at most bins can have load at least (by Markov). So .
For the induction,
Consider . By the Lemma 6, . Hence, with probability , we have the number of bins with more than balls in them is at most .
We’re almost done, but there’s one more step to do. If this number were small, say , then we could have done a union bound, but this number may still be about . So apply Lemma 4 and the second part of Lemma 5 once more to get:
Finally, since is so small whp, use Lemma 4 and a union bound to say that
Finally, the calculations in Section 2 show that , which completes the proof.
2.1. A Calculation
Since , and , we calculate
So, for , it suffices to set
3. A Random Graphs Proof
Another way to show that the maximum load is —note that the constant is worse—is to use an first-priciples analysis based on properties of random graphs. We build a random graph as follows: the vertices of correspond to the bins, and the edges correspond to balls—each time we probe two bins we connect them with an edge in . For technical reasons, we’ll just consider what happens if we throw fewer balls (only balls) into bins—also, let’s imagine that each ball chooses two distinct bins each time.
Theorem 7 If we throw balls into bins using the best-of-two-bins method, the maximum load of any bin is whp.
Hence for balls and bins, the maximum load should be at most times as much, whp. (It’s as though after every balls, we forget about the current loads and zero out our counters—not zeroing out these counters can only give us a more evenly balanced allocation; I’ll try to put in a formal proof later.)
To prove the theorem, we need two results about the random graph obtained by throwing in random edges into vertices. Both the proofs are simple but surprisingly effective counting arguments, they appear at the end.
Given the graph , suppose we repeatedly perform the following operation in rounds:
In each round, remove all vertices of degree in the current graph.
We stop when there are no more vertices of small degree.
Proof: Condition on the events in the two previous lemmas. Any component of size at least in the current graph has average degree at most ; by Markov at least half the vertices have degree at most and will be removed. So as long as we have at least nodes in a component, we halve its size. But the size of each component was to begin, so this takes rounds before each component has size at most .
Lemma 11 If a node/bin survives rounds before it is deleted, its load due to edges that have already been deleted is at most . If a node/bin is never deleted, its load is at most , where is the total number of rounds.
Proof: Consider the nodes removed in round : their degree was at most , so even if all those balls went to such nodes, their final load would be at most . Now, consider any node that survived this round. While many edges incident to it might have been removed in this round, we claim that at most of those would have contributed to ‘s load. Indeed, the each of the other endpoints of those edges went to bins with final load at most . So at most of them would choose as their less loaded bin before it is better for them to go elsewhere.
Now, suppose is deleted in round : then again its load can be at most : ten because it survived the previous round, and 10 from its own degree in this round. OTOH, if survives, then consider all the edges incident to that were deleted in previous rounds. Each of them went to nodes that were deleted in rounds or , and hence had maximum load at most . Thus at most of these edges could contribute to ‘s load before it was better for them to go to the other endpoint. The same inductive argument holds for any round .
Finally, the process ends when each component has size at most , so the degree of any node is at most . Even if all these edges contribute to the load of a bin, it is only .
3.1. Missing Proofs of Lemmas
Proof: We have a graph with vertices and edges where we connect vertices at random.
Since and . Now the probability that any such set exists can bounded above by the union bound
which proves the claim.
By a union bound over all sets, the probability that such a set exists is
where . Now, we can break this sum into two: for small values of , the term would be very small, else the term would be small. Indeed, for , we know that , so
Now for the rest:
for , say.
Bibliographic Notes: The layered induction appears in Balanced Allocations Azar, Broder, Karlin, and Upfal. The random graph analysis is in the paper Efficient PRAM Simulation on a Distributed Memory Machine by Karp, Luby, and Meyer auf der Heide; I learned it from Satish Rao. The Always-go-left algorithm and analysis is due to How Asymmetry Helps Load Balancing by Berthold Vöcking.
Update: Here’s a survey on the various proof techniques by Mitzenmacher, Sitaraman and Richa.