Question 1

How do I interpret the p-value?

Accepted Answer

The p-value is the probability of observing a statistic as extreme as your sample, or more, assuming H0 is true. It is not the probability that H0 is true. With p = 0.0794 and α = 0.05, the data are not "unlikely enough" to reject H0; with p = 0.0028, they are. Very small p-values appear in scientific notation: 2.5e-7 means 0.00000025.

Question 2

Why say "fail to reject H0" instead of "accept H0"?

Accepted Answer

Because a test only measures evidence against H0. A non-significant result may come from no effect at all or from a sample too small to detect one. That is why the calculator writes "there is not enough evidence that...", never "H0 is true". To argue that two groups are equivalent you need a different design, such as an equivalence test (TOST).

Question 3

z test or t test for one mean?

Accepted Answer

Use z when the population standard deviation (σ) is known, which is rare outside textbook problems or industrial processes where σ has been measured for a long time. If you only have the sample standard deviation (s), use t, which pays for the extra uncertainty with heavier tails. At 29 degrees of freedom the two-sided 5% critical value is 2.045 for t against 1.960 for z; at 194 it falls to 1.972.

Question 4

Welch or pooled t for two means?

Accepted Answer

Welch does not assume equal variances and loses almost no power when they are equal, which is why it is the default. The pooled t (df = n₁ + n₂ − 2) is only valid if the population variances are equal; when sizes differ and variances differ too, it can get the p-value seriously wrong. The calculator shows the F and Levene tests as a hint, but choosing the method from a preliminary test is worse than always using Welch.

Question 5

One-sided or two-sided test?

Accepted Answer

Use two-sided when any difference matters (H1: ≠). Use one-sided only if, before seeing the data, you already knew that only one direction matters (H1: > or <). Choosing the tail after looking at the result doubles the chance of a false positive. In the two-sided test the p-value is twice the one-tail value and the 5% critical value is ±1.96 (z), against 1.645 for one-sided.

Question 6

Chi-square goodness of fit or independence? And what if expected counts are small?

Accepted Answer

Goodness of fit compares one categorical variable with an expected distribution (a fair die, the 9:3:3:1 Mendel ratio). Independence checks whether two categorical variables are associated, in a contingency table. In both, expected frequencies should be at least 5 in nearly all cells. Below that, merge categories or, for 2×2, use Fisher exact test, which the calculator shows alongside. Always use counts, never percentages.

Question 7

When should I use the paired t instead of the two-sample t?

Accepted Answer

When both measurements come from the same subjects (before and after) or from matched pairs. The paired test checks the mean of the differences and removes between-person variation, which usually gives far more power. In Student sleep data, treating the two columns as independent gives p = 0.079; treating them as pairs gives p = 0.0028. Using the independent test on paired data throws that information away; the reverse is a method error.

Question 8

What are Cohen d, Cramér V and η²?

Accepted Answer

These are effect sizes: they say how large a difference is, not just whether it is detectable. d is the difference of means in standard deviations (0.2 small, 0.5 medium, 0.8 large); Cramér V measures association in a table (0.1, 0.3 and 0.5 for 1 degree of freedom); η² is the fraction of variance explained by the groups in ANOVA (0.01, 0.06 and 0.14). With huge samples tiny effects become significant; with small samples large effects may not.

Question 9

The ANOVA was significant. Now what?

Accepted Answer

A significant F only says that at least one mean differs. To learn which pairs differ, use a post hoc test such as Tukey, or pairwise comparisons with correction for multiple comparisons (Bonferroni, Holm); running several t tests without correction inflates the type I error. If the group variances differ a lot, look at Welch ANOVA, which the calculator shows alongside. In the R PlantGrowth data, F = 4.846 (df 2 and 27) and p = 0.0159; Welch gives F = 5.181 and p = 0.0174.

Question 10

Does my data need to be normal? Can I trust the calculations?

Accepted Answer

The t tests and ANOVA assume roughly normal data, but they are robust with samples of about 30 or more per group; with small, strongly skewed samples consider a nonparametric test (Mann-Whitney, Wilcoxon, Kruskal-Wallis), which this calculator does not do. The calculator warns when skew is strong in small samples. The results match NIST/SEMATECH and R: for example, t = 2611.284 with 194 df in the NIST ZARR13 example. Nothing leaves your browser.

Group	Size (n)	Mean	s
Group 1	10	0.75	1.789
Group 2	10	2.33	2.0022

Group	Size (n)	Mean	s
Group 1	10	0.75	1.789
Group 2	10	2.33	2.0022

Hypothesis Test Calculator (t, z, chi-square)