Sum of Independent Random Variables

Finding the probability generating function of \(Z=X+Y\)

There is a very important theorem that covers this case called the Convolution Theorem.

Convolution Theorem: Suppose two independent discrete random variables \(X\) and \(Y\) have probability generating functions \(G_X(t)\) and \(G_Y(t)\). The probability generating function of \(Z=X+Y\) is

\[G_Z(t)=G_X(t)G_Y(t).\]

Let's look at an application.

The main benefit of the Convolution Theorem is that it gives you a way to find the probability generating function without constructing a table, which leads to fewer opportunities for errors.

Finding the probability generating function of \(Z=aX+b\)

Let's take a quick look at how you could build the probability generating function of \(Z=nX\) where \(n\) is a natural number from the probability generating function of \(X\). Starting with \(n=2\),

\[ Z = 2X = X+X\]

so you can use the Convolution Theorem to get that

\[G_Z(t)=G_X(t)G_X(t) = (G_X(t))^2.\]

Then you could use proof by induction to show that for any natural number \(n\), if \(Z=nX\) then

\[G_Z(t)=\underbrace{G_X(t)G_X(t)\cdots G_X(t)}_{n \text{ times}} = (G_X(t))^n.\]

As you know, there are more than just the natural numbers out there, and you still need to account for that '\(+b\)' too. So it helps to look at the alternate definition of the probability generating function:

\[G_Z(t) = \text{E}(t^Z).\]

Then you can use properties of the expected value function to get the following:

\[\begin{align} G_Z(t) &= \text{E}(t^Z) \\ &= \text{E}(t^{aX+b}) \\ &= \text{E}(t^{aX}t^b) \\ &= t^b\text{E}(t^{aX}) \\ &= t^b\text{E}\left((t^a)^X\right) \\ &= t^bG_X(t^a) . \end{align}\]

While this property doesn't have a fancy name, it is worth stating separately.

If \(X\) is a discrete random variable and has a probability generating function of \(G_X(t)\), the probability generating function of \(Z\) where \(Z=aX+b\) is:

\[G_Z(t)=t^bG_X(t^a).\]

Let's take a look at a quick example.

Expectation of the sum of independent random variables

Just like all random variables, sums of independent random variables also have an expectation or mean. You can use the Convolution Theorem and the alternate definition of the probability generating function to find the expectation of the sum of independent random variables, along with the formulas:

• \(G'_X(1) = E(X)\);

• \(\text{E}(aX+b) = a\text{E}(X) + b\); and

• \(\text{E}(X+Y) = \text{E}(X) + \text{E}(Y)\).

Let's look at an example.

Variance of the sum of independent random variables

Just as you have found the mean above, you can also find the variance of sums of independent random variables. To do this you will need the formulas:

• \(\text{Var}(aX+b) = a^2\text{Var}(X)\); and
• \(\text{Var}(Z)= G''_Z(1)+G'_Z(1)-(G'_Z(1))^2\).

Let's look at an example.

Sum of independent random variables examples

You have already seen some examples of funding the sum of independent random variables, along with their mean and variance. However for specific kinds of distributions, like the binomial distribution and the uniform distribution, looking at them, in particular, can be illuminating. So keep going for the specifics!

Sum of independent binomial random variables

Suppose you have two independent random variables that follow binomial distributions. In other words, \(X \sim \text{Bin}(n_X, p_X)\) and \(Y \sim \text{Bin}(n_Y, p_Y)\). You already know that

\[G_X(t) = (1-p_X+p_Xt)^{n_X}\]

and

\[G_Y(t) = (1-p_Y+p_Yt)^{n_Y}.\]

So using the Convolution Theorem, if \(Z = X+Y\) then

\[ \begin{align} G_Z(t) &= G_X(t) G_Y(t) \\ &= (1-p_X+p_Xt)^{n_X} (1-p_Y+p_Yt)^{n_Y}. \end{align}\]

Let's take an example.

Sum of independent uniform random variables

Remember that a discrete uniform random variable takes on equal probabilities for each possible outcome. So if distribution \(X\) has events occurring with probability \(\dfrac{1}{n}\), then

\[ G_X(t) = \frac{t(1-t^n)}{n(1-t)}.\]

So if \(Y\) is a second discrete uniform random distribution that has events occurring with probability \(\dfrac{1}{m}\), and \(Z = X+Y\), then

\[ \begin{align} G_Z(t) &= G_X(t) G_Y(t) \\ &= \left(\frac{t(1-t^n)}{n(1-t)} \right)\left(\frac{t(1-t^m)}{m(1-t)}\right) \\ &= \frac{t^2(1-t^n)(1-t)^m}{nm(1-t)^2} .\end{align}\]

Let's take an example.

It actually isn't necessary to memorise the different formulas for the various kinds of discrete probability distributions as long as you keep in mind the Convolution Theorem!