Law of Large Numbers

In statistical terms, the law of large numbers is a theorem that postulates that as the size of the sample of a random variable increases, its average will approach the theoretical average. In layman’s terms, the law of large numbers simply says that over time, the more times you roll a dice, the more likely the average of the rolls will turn out to be 3.5.

If your sample size is one, meaning a single roll, you have a 2 in six chance of getting a 3 or 4, both close to the average. But you also have a 2 in six chance of being as far away from the expected average as possible by rolling a 1 or a 6. Also note that there is no chance of rolling a 3.5, the theoretical average, with a single dice.

Roll twice, and the odds of getting snake eyes (two ones) or boxcars (two sixes) drops dramatically. There is only a 2 in 36 chance of rolling those numbers. There is also a far greater likelihood of getting a combination exactly at 3.5. A six and a one, a 5 and a two, and a 4 and a 3 will all yield a 3.5.

Simply put, the law of large numbers means that you should be wary of inferring characteristics about a population from a small sample.

Lean and the Law of Large Numbers

The law of large numbers tends to work against getting people interested in Lean. In general, people are a lot more likely to comment about a bad experience than about a positive one. So, the stories that people hear about Lean might be biased towards the negative.

A person facing a Lean transition might talk to a small number of people about some of their experiences with Lean and hear a couple of bad stories. They assume that the bad stories are representative of all Lean activity and become resistant.

Or a person new to Lean might have a bad experience with something early on and assume that all future experiences will also be bad. In both cases, that small sample may be representative of what Lean will be like, but it also may include a few outliers. It might not be a representative sample.

Law of Large Numbers and Processes

With process-related outcomes, the law of large numbers also holds true. When a new process is implemented in a kaizen event, the first few units of output may indicate something about its average results, but it is also possible that you are seeing something clustering near one of the tails of the distribution. The truth is that this is unlikely in any single process improvement project. But if you make enough changes, the odds increase that it will eventually happen in one of your events. The results may show a very poor or very good process with a small sample, but over time as more data is collected, the average will drift toward the true process capability.

That is one of the reasons that most kaizen facilitators recommend that you do follow-up audits on new processes. The reviews give you an opportunity to see a larger sample and get a better feel of what the true output of the process looks like.

Sample Size and the Law of Large Numbers

So, what size does a sample have to be to trust that it is representative of the entire population?

The truth is that there is a lot of math and statistics behind this. You end up with a confidence interval that the sample fits within the true range of the actual population. The calculations determine what your sample size needs to be for that confidence interval.

In practice, though, in your Lean travels, you will take samples and try to determine something about your process characteristics, and in nearly all cases, you won’t be using any math.

Perhaps you sample the length of tubing cut by a machine. If you take one sample, how confident are you that the machine is producing correctly? How about 3? 7?

Whenever you take a cycle time of a process, you are sampling it. How many runs through a process make you feel confident you are seeing the real picture?

The point is that the more samples you take, the more you believe that the machine is working properly or that the time is accurate enough to base your Standard Work on. But in the end, if you skip the math, you’ll have to just make a gut check and judge for yourself how many samples to take based on risk and need.

Over the years, I’ve heard rules of thumb that you need between 30 and 40 samples to be really confident. Take that with a grain of salt, as most rules of thumbs are just shortcuts. But in most cases, you will be reasonably safe using those numbers. Just be sure to get an expert to do the math for you if it is something really important.

The law of large numbers works against you more often than it works for you. If a process looks better than it really is, you pass on a chance to fix a problem. If it looks worse than it really is, you waste resources fixing something that isn’t broken.

If an employee sees something bad about Lean early on, you run the risk of them assuming that all Lean things are bad.
Before making any assumptions about small sets of data points, test your theory. Run a few more cycles and see if the average drifts. Have the person with the bad experience try a few more things or talk to a few more people to get a more representative view of the real situation. They key is to increase the sample size and confirm that the results you saw.