Small Samples, Big Decisions

"We don't have enough data"

This statement stops measurement before it even begins. Behind it lies a familiar assumption: to speak with certainty, you need a large sample size. Statistical power requires hundreds of respondents. And so measurement becomes something only large organizations can do.

But what if the question is wrong?

Traditional statistics asks: "Is the result statistically significant?" This question demands a large sample size because it tries to almost entirely rule out the possibility of chance. A p-value of 0.05 means that if the null hypothesis were true, the probability of observing a result this extreme or more extreme would be at most 5 percent. In practice, this is often interpreted as the risk of error – although that interpretation is not literally the same thing.

The Bayesian approach asks differently: "What can we infer from the data we have?"

From certainty to probability

When measurement shifts from asking about certainty to asking about probabilities, small sample sizes cease to be an obstacle. They become a starting point.

From 20 respondents, you can say: "With 75% probability, reliability is at a good level."

That is not certainty. But it is far more than "we cannot say anything".

And when the next measurement comes, understanding updates. 20 + 15 + 25 respondents are not three separate "too small" samples – they are cumulative knowledge. (Assuming, of course, that the same phenomenon is being measured in the same context.)

Why does this work?

Bayesian inference is based on a simple idea: knowledge updates with new evidence. You start from what is known or assumed beforehand (prior), and update that understanding based on data (posterior).

This means that:

Prior knowledge is not wasted. If similar phenomena have been measured in the organization before, that knowledge can be utilized. A new measurement does not start from zero.

Uncertainty is part of the result. Bayesian analysis does not give a single number, but a probability distribution. It tells you where the true value probably lies – and how confident we can be.

Even small data tells you something. 20 respondents are not enough for certainty, but they are enough to see the direction. And direction is often what decision-making needs.

Credible interval vs. confidence interval

Traditional statistics produces confidence intervals. Bayesian analysis produces credible intervals. The difference sounds technical, but it is practically significant.

A confidence interval answers the question: "If we repeated the measurement many times, how often would the interval contain the true value?" This is a counterintuitive interpretation, and it often leads to misunderstandings.

A credible interval directly answers what we want to know: "In what range does the true value probably lie?"

When a report says "95% credible interval is 0.72–0.88", it means literally: "With 95% probability, the true value is within this range." No circumlocutions, no interpretation guides needed.

A practical example

Imagine this situation: An organization has delivered leadership training to 25 people. The question is whether participants' ability to give constructive feedback improved.

Traditional approach: "Sample size too small for statistically significant results. We need at least 50 participants."

Bayesian approach: "Based on the data, there is an 82% probability that feedback skills improved. The average change was 0.6 units (on a scale of 1–5), and with 95% probability the true change is between 0.3 and 0.9."

The latter answer is not certain. But it is usable. It tells you which direction things probably went, by how much, and how confident we can be.

When is a small sample size not enough?

The Bayesian approach is not a magic wand. There are situations where a small sample size truly limits inference:

Complex models. If you want to examine relationships between multiple variables simultaneously, more data is needed.

Rare phenomena. If you are measuring something that occurs only in some respondents, small numbers produce large uncertainties.

Publication requirements. Academic peer review often requires traditional statistical tests. (Although this too is changing.)

But in most practical situations – evaluating training effectiveness, tracking team development, measuring intervention success – a Bayesian approach produces usable answers even with small sample sizes.

Making uncertainty visible

Perhaps the most important change is not technical but a change in mindset. Traditional statistics produces dichotomies: a result is either significant or it is not. This creates an illusion of certainty where none exists – and prevents decision-making where it would be possible.

The Bayesian approach makes uncertainty visible. It says: "This is the most probable estimate, and this is our uncertainty."

This is not a weakness. It is information.

When you know how uncertain you are, you can make better decisions. You can decide whether more data is needed or whether current knowledge is sufficient. You can assess whether to wait or act now.

Making uncertainty visible adds information, it does not reduce it.

evaluoi.ai uses Bayesian statistics as its primary analytical method. This enables reliable decision-making even with small sample sizes – without measurement being abandoned due to "insufficient data".