Central Limit Theorem

Understanding the Central Limit Theorem

Consider the following example.

Now, if instead of 4 balls numbered with 2, 4, 6 and 8, you had 5 balls numbered with 2, 4, 6, 8 and 10, then you’d have 25 possible combinations, which leads to 25 means.

What would the graph bar of this new list of means look like? Yes, it would have a similar form to that of a normal curve.

If you kept increasing the number of numbered balls, the corresponding bar graph would get closer and closer to a normal curve.

"Why is that?" you ask. This leads you to the next section.

Definition of Central Limit Theorem

The Central Limit Theorem is an important theorem in statistics, if not the most important, and is responsible for the effect of approximating the bar graphs for increasing values of the number of numbered balls to the curve of the normal distribution in the above example.

Let's start by looking at its statement, and then recall two important concepts involved in it: a distribution of sample means, and the useful normal distribution.

Central Limit Theorem Statement

The statement of the Central Limit Theorem says:

If you take a sufficiently large number of samples from any random distribution, the distribution of the sample means can be approximated by the normal distribution.

Easy-peasy, right?! “Uhh… No…!!” Ok, ok. Let's understand it by simplifying its statement a bit:

If you take a big number of samples from a distribution, the sample mean of this distribution can be approximated by the normal distribution.

Let's forget for a moment "a sufficiently large number" and "any random distribution", and focus on:

• a sample mean;

• and normal distribution.

Understanding the Distribution of Sample Means

Imagine you have to perform a statistical study for a particular attribute. You identify the population of your study and from it, you’ll draw a random sample. You will then calculate a particular statistic related to that attribute you’re interested in from this sample, and it’ll be the mean.

Now imagine drawing another sample randomly from the same population, with the same size as the previous one, and calculating the mean of the attribute of this new sample.

Imagine doing this a few more (and more and more) times. What you’ll end up with is a list of means from the samples you’ve drawn. And voilà! That list of means you end up with constitutes a distribution of sample means.

To deepen your knowledge on this topic, read our article Sample Mean.

Recalling the Normal Distribution

One big usefulness of the normal distribution is associated with the fact that it approximates quite satisfactorily the frequency curves of physical measurements. That is, physical measures such as the height and weight of a sample of elements of the human population can be approximated by this distribution. Now you’re close to seeing another important application of this distribution.

By now you may already know that the normal distribution is a probability distribution with two parameters, a mean \(\mu\) and a standard deviation \(\sigma\), and that has a graphical appearance of a bell-shaped curve – see figure 1.

Fig. 1 – Normal curve of a normal distribution of mean 0 and standard deviation 0.05

The mean is the value at which the distribution is centered, and the standard deviation describes its degree of dispersion.

In the case of figure 1, the normal curve is centered at 0 and its dispersion is somewhat low, 0.05. The lower the dispersion, the closer the curve is to the \(y\)-axis.

To refresh your memory on this topic, read our article Normal Distribution.

How Many is Enough?

What you need to understand here is that the Central Limit Theorem tells us that for a "number” of samples from a distribution, the sample mean will get closer to the normal distribution.

Now including what we took out for a moment, Central Limit Theorem says that no matter what the distribution is - "any random distribution" -, the distribution of its mean approaches normal distribution as the number of samples grows - "a sufficiently large number of samples".

Now the question imposes itself, what is a sufficiently large number of samples? This leads us to the next section.

Conditions for the Central Limit Theorem

There are two main conditions that must be met for you to apply the Central Limit Theorem.

The conditions are the following:

• Randomness – the sample collection must be random, this means every element of the population must have the same chance of being selected.

Coming back to the first example, you had the 4 balls on a bag, and they were indistinguishable to touch. These elements randomize the experiment.

• Sufficiently large sample: as a practical rule, when the number of samples is at least 30 the distribution of the sample means will satisfactorily approach a normal distribution.

This is why the example above serves only the purpose of illustrating with simplicity the idea of the Central Limit Theorem. We got 16 samples from it, and if there were 5 balls, we could only get 25 samples, which again is not enough large number of samples.

Central Limit Theorem Formula

Addressing the Central Limit Theorem formula is equivalent to restating it by introducing all the necessary notation, and giving it further details.

It’s worth repeating the first statement:

If you take a sufficiently large number of samples from any random distribution, the distribution of the sample means can be approximated by the normal distribution.

Now introducing the appropriate notation:

Assume you have an initial distribution, with either an unknown or known probability distribution, and let \(\mu\) be its mean and \(\sigma\) be its standard deviation.

Also, assume you’ll take \(n\) samples from this initial distribution, and \(n\ge30\).

Then, the sample mean, \(\bar{x}\), with mean \(\mu_\bar{x}\) and standard deviation \(\sigma_\bar{x}\), will be normally distributed with mean \(\mu\) and standard variation \(\frac{\sigma}{\sqrt{n}}\).

As a result of this new restatement of the Central Limit Theorem, you can conclude that:

1. The mean of the distribution of the sample mean \(\bar{x}\) will be equal to the mean of the initial distribution, i.e., \[\mu_\bar{x}=\mu;\]
2. The standard deviation of the distribution of the sample mean \(\bar{x}\) will be \(\frac{1}{\sqrt{n}}\) of the standard deviation of the initial distribution, i.e., \[\sigma_\bar{x}=\frac{\sigma}{\sqrt{n}};\]

   This is actually good: notice that for an increasing value of \(n\), \(\frac{\sigma}{\sqrt{n}}\) decreases, the dispersion of \(\bar{x}\) decreases, which means it behaves more and more like a normal distribution.

3. The Central Limit Theorem applies to any distribution with many samples, be it known (like a binomial, a uniform, or a Poisson distribution) or an unknown distribution.

Let's look at an example where you'll see this notation in action.

Calculations Involving the Central Limit Theorem

As you by now know, the Central Limit Theorem allows us to approximate any distribution of means, for a large number of samples, to the normal distribution. This means that some of the calculations where the Central Limit Theorem is applicable will involve calculations with the normal distribution. Here, what you'll be doing is converting a normal distribution to the standard normal distribution.

To recall more of the last concept topic, please read our article Standard Normal Distribution.

The importance of doing this conversion is that then you'll have access to a table of values of the standard normal, also known as z-score, to which you can refer to proceed with your calculations.

Any point \(x\) from a normal distribution can be converted to the standard normal distribution \(z\) by doing the following

\[z=\frac{x-\mu}{\sigma},\]

where \(z\) follows the standard normal distribution (with mean \(\mu=0\) and standard deviation \(\sigma=1\)).

Because \( \bar{x}\) is normally distributed with mean \(\mu\) and standard deviation

\[\frac{\sigma}{\sqrt{n}},\]

the conversion will be more like

\[z=\frac{x-\mu}{\frac{\sigma}{\sqrt{n}}}.\]

You can refresh your memory on this topic by reading our article z-score.

This example serves as a reminder of the conversion to the standard normal distribution.

Examples of the Central Limit Theorem

To consolidate the learnings from this article, let's now turn to application examples. Here, you will see an overview of all the main aspects of the Central Limit Theorem.

To the first example.

Let's do a final word problem.

Importance of the Central Limit Theorem

There are many situations in which the Central Limit Theorem is of importance. Here are some of them:

• In instances where it is difficult to collect data on each element of a population, the Central Limit Theorem is used to approximate the features of the population.

• The Central Limit Theorem is useful in making significant inferences about the population from a sample. It can be used to tell whether two samples were drawn from the same population, and also check if the sample was drawn from a certain population.

• To build robust statistical models in data science, the Central Limit Theorem is applied.

• To assess the performance of a model in machine learning, the Central Limit Theorem is employed.

• You test a hypothesis in statistics using the Central Limit Theorem to determine if a sample belongs to a certain population.