July 23, 2026
How to Understand Hypothesis Testing
Hypothesis testing explained: the steps, null versus alternative hypotheses, what a p value means, significance levels, and Type I and Type II errors, with an example.
The direct answer: hypothesis testing is a structured way to use sample data to decide whether there is enough evidence to reject a default claim. You state a null hypothesis, collect data, compute a test statistic and a p value, then compare that p value to a chosen significance level. This guide walks through each step, defines the two error types, and shows a concrete example.
Hypothesis Testing at a Glance
| Question | Answer |
|---|---|
| What is the null hypothesis? | The default claim, written H0, often "no effect" or "no difference." |
| What is the alternative? | Ha, the claim you are looking for evidence to support. |
| What is a p value? | The probability of data this extreme if H0 were true. |
| Common significance level | alpha = 0.05, but it depends on the field. |
| What is a Type I error? | Rejecting H0 when it is actually true (false alarm). |
| What is a Type II error? | Failing to reject H0 when it is false (missed signal). |
The Basic Steps
A standard test follows the same skeleton every time.
- State the hypotheses. H0 is the status quo. Ha is what you suspect.
- Choose a significance level alpha, such as 0.05.
- Collect data and check that assumptions hold (random sample, roughly normal, and so on).
- Compute a test statistic, such as a z or t value.
- Find the p value, the probability of a statistic at least this extreme under H0.
- Decide. If p is less than alpha, reject H0. Otherwise, do not reject H0.
Notice the careful wording. You never "accept" the null. You either reject it or fail to reject it, because a sample can only provide limited evidence.
Null and Alternative Hypotheses
The null hypothesis usually contains an equals sign. For a coin you suspect is biased toward heads, you might write:
H0: the true heads probability p = 0.5 Ha: p is greater than 0.5
One sided versus two sided
The alternative can be one sided (greater than, or less than) or two sided (not equal to). A two sided test splits the significance level across both tails, so it is a bit harder to reject. Choose the form before looking at the data to avoid biased choices. Picking the direction after seeing the result is a known way to manufacture false findings, which is why pre registration matters in real research NIST.
What a P Value Really Means
A p value is not the probability that the null hypothesis is true. It is the probability of seeing data at least as extreme as yours, assuming the null hypothesis is true. A small p value means the observed result would be surprising under H0, which is evidence against H0.
If p = 0.03 and alpha = 0.05, the result is statistically significant, and you reject H0. If p = 0.12, you do not have enough evidence at the 0.05 level, so you fail to reject H0. The p value does not measure the size of an effect, only how surprising the data are. The Boston University module stresses this same point with clinical examples BU.
A Worked Example
Suppose a factory claims its bags of rice weigh 500 grams on average. You sample five bags and get weights 498, 501, 502, 499, 500. The sample mean is 500.0 and the sample standard deviation is about 1.58. To test H0: mu = 500 against Ha: mu is not 500 at alpha = 0.05, you use a t statistic:
t = (x bar minus mu) divided by (s divided by sqrt(n)) = (500.0 minus 500) divided by (1.58 divided by sqrt(5)) = 0.
With t = 0, the p value is 1.0, far above 0.05, so you do not reject H0. The sample gives no evidence that the mean differs from 500 grams.
Type I and Type II Errors
Every test can go wrong in two ways. A Type I error rejects a true null (a false alarm). Its probability is alpha, the significance level you chose. A Type II error fails to reject a false null (a missed signal). Its probability is called beta.
The trade off
Lowering alpha reduces false alarms but raises the chance of missed signals. Raising alpha does the opposite. This trade off is why you should pick alpha based on how costly each error is, not just by habit. A medical screen for a serious disease may accept more false alarms to avoid missing real cases, while a quality check on cheap parts may do the reverse.
The Power of a Test
A Type II error has a mirror concept worth knowing: the power of a test, which is 1 minus beta, the probability of correctly rejecting a false null. Power grows when the effect is larger, the sample is bigger, or the significance level is relaxed.
A concrete sense of power
Imagine testing whether a coin is fair with only 10 flips. Even a badly biased coin, say 70 percent heads, could easily show 5 or 6 heads in 10 tosses just by chance, so the test would often fail to reject. Run 1,000 flips and the same bias is almost certain to show up. The sample size is what gives the test the sensitivity to see the effect, and power is the name for that sensitivity. When someone reports "no link found," the first question to ask is how many observations they had.
Why power matters in design
Before collecting data, researchers estimate power to make sure the study can detect an effect of the size they care about. A small sample can make even a real effect invisible, which produces a fail to reject result that looks like no difference but is really just low sensitivity. This is why "no evidence" from a tiny study is weak evidence of nothing. When you read a claim that a test "found no link," check whether the study had enough participants to notice one.
P Hacking and Selective Reporting
A known trap in real research is p hacking: running many cuts of the data, or choosing a one sided test after seeing the result, until p dips below 0.05. Each separate look at the data raises the chance of a false alarm, so a "significant" result found this way may be noise dressed as signal.
How honest practice avoids it
Pre register your hypothesis and analysis plan before touching the data, or at least decide the test form up front. Report effect sizes alongside p values, because a tiny effect can be significant with a huge sample yet mean little in practice. And treat a single study as one piece of evidence, not proof. This habit is what separates a classroom exercise, where the numbers are given, from the messier real world, where the choices you make shape the answer.
Common Misconceptions
- Saying "accept the null" instead of "fail to reject."
- Reading the p value as the probability that H0 is true.
- Picking a one sided test after seeing the data.
- Equating statistical significance with a large or important effect.
- Forgetting that a failed rejection is not proof that H0 is true.
- Believing a smaller p means a bigger effect. It only means stronger evidence against H0, not a larger difference.
- Thinking a confidence interval and a test disagree whenever one is "significant." They ask the same question from two angles, as shown in confidence intervals.
Frequently Asked Questions
What does "fail to reject" actually mean?
It means the sample did not provide strong enough evidence against H0. It is not the same as proving H0 true.
Why is 0.05 used so often?
It is a long standing convention from early statistical practice. Many fields use it, but medicine or physics may demand smaller alpha. The right level depends on the cost of being wrong.
Is a smaller p value always better?
A smaller p value gives stronger evidence against H0, but it says nothing about the size or practical importance of the effect.
What is the difference between a z test and a t test?
Both compare a statistic to a distribution. A z test is used when the population standard deviation is known or the sample is large. A t test is used with unknown population standard deviation and smaller samples.
How are confidence intervals related to tests?
A two sided test at level alpha lines up with a (1 minus alpha) confidence interval. If the null value sits outside the interval, you reject H0.
Can a test prove a hypothesis?
No. It can only supply evidence against a null claim. Absence of evidence is not evidence of absence, which is why "fail to reject" stays careful.
Sources
About the author
Michael R. is a study skills coach with 12 years of experience and a learning specialist. He helps students develop effective study strategies and organizational systems.