Applied Statistics Notes for [5 Hypothesis Testing]

Definition

We would first translate our question of interest into a hypothesis about an unknown parameter like mean (\(\mu_{X}\)) or variance \((\sigma_X^2)\), and then test it.

(comment: 与点估计和区间估计不同，这里是先假设参数的值，再根据数据判断我们的假设是否有充分证据被拒绝。)

• Null Hypothesis 零假设

  It is the hypothesis that is assumed to be true and then tested to be rejected or not to be rejected formally. It always contains "=" sign. (i.e., \(=, \leq, \geq\))

• Alternative Hypothesis 备择假设

  It is the hypothesis that contains the values of the parameter we would support if we reject Null. It do not contain "=" sign.

(comment: 人话解释就是，假设现在有一个科研项目，零假设是认为这个项目没有用，备择假设是认为这个项目有用，这两个假设通常是对立面，并且通常我们要试图拒绝零假设。)

Type I and Type II Error

Type I error: 得到的结论是拒绝零假设，但实际的情况是证据不足拒绝零假设。

Type II error: 得到的结论是证据不足拒绝零假设，但实际的情况是拒绝零假设。

\(\alpha=P(\text{Type I error})=P(\text{Reject } H_{0} \text{ if } H_0 \text{ is true})\)

\(\beta=P(\text{Type II error})=P(\text{Not reject } H_{0} \text{ if } H_0 \text{ is false})\)

(comment: \(\alpha\) 也常被称为显著性水平，即英文 level of significance。)

Power of a Test Statement

A power of the test statement is defined as \(1-\beta\).

\[\beta=P(\text{Type II error})=P(\text{Not reject } H_{0} \text{ if } H_0 \text{ is false}) \]

p-value

Roughly speaking, p-value is the probability of getting our data under \(H_0\).

• By statistical belief, if the p-value (the probability of getting data under \(H_0\)) is "small", then we would say that the null hypothesis is wrong.

• Usually, we compare the p-value with the significance level \(\alpha\).

  • If the p-value \(<\alpha\), then we can say that we have enough evidence to reject the null hypothesis \(H_0\) at a level \(\alpha\);
  • while, the p-value \(>\alpha\) only implies that we do NOT have enough evidence to reject \(H_0\) at that level.

(comment: 简单来说，p-value 是判断当前数据是否足够拒绝零假设的判据，这个过程通过将 p-value 与 \(\alpha\) (显著性水平) 进行比较实现。)

Hypothesis Testing (Normal Case)

(comment: 这部分内容对应上个章节参数估计最后的三个模型。实际上就是算一下要求的显著性水平 \(\alpha\) 对应的区间，然后看一下样本提供的数据是否落在置信区间内，如果是则拒绝零假设，否则没有充分证据拒绝零假设。)

(comment: 这里的区间可以类比置信区间，但不同的是需要根据实际情况判断是单尾 (one-tailed) 还是双尾 (two-tailed)，这个可以通过看零假设是否是恰好等于判断，是则为双尾，否则为单尾。)

A/B Testing - Hypothesis Testing for Two Populations

Variance are known OR large sample size

• Purpose: Compare the means of two independent populations when population variances are known (or sample sizes are large enough to invoke the Central Limit Theorem).

  \[H_0: \mu_1=\mu_2 \]
  \[H_{a}: \mu_1\neq \mu_2 \]
  \[Z=\frac{\bar{X_1}-\bar{X_2}-(\mu_1-\mu_2)}{\sqrt{\frac{\sigma_{1}^2}{n_1}+\frac{\sigma_{2}^2}{n_2}}}\sim\mathscr{N}(0, 1) \]

  C.I. : \((\bar{X_1}-\bar{X_2})\pm Z_{\frac{\alpha}{2}}\sqrt{\frac{\sigma_{1}^2}{n_1}+\frac{\sigma_{2}^2}{n_2}}\)

• Purpose: Compare the proportions of two independent populations. (e.g., conversion rates, success rates)

  \(\hat{p_{1}}=\frac{x_1}{n_1}, \hat{p_{2}}=\frac{x_2}{n_2}\), where \(x_1, x_2\) are numbers of successes in each sample.

  \[H_0: p_1=p_2 \]
  \[H_{a}: p_1\neq p_2 \]

  \(\hat{p}_{pool}=\frac{x_1+x_2}{n_1+n_2}\)

  \[Z=\frac{\hat{p_1}-\hat{p_2}-(p_1-p_2)}{\sqrt{\hat{p}_{pool}(1-\hat{p}_{pool})\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}}\sim\mathscr{N}(0, 1) \]

  C.I. : \((\hat{p_1}-\hat{p_2})\pm Z_{\frac{\alpha}{2}}\sqrt{\frac{\hat{p}_1(1-\hat{p}_1)}{n_1}+\frac{\hat{p}_2(1-\hat{p}_2)}{n_2}}\)

• Relation:

  For \(X\sim\mathscr{B}(n, p)\):

  • \(\hat{p}=\frac{X}{n}\)

  • \(E[\hat{p}]=p\)

  • \(\sigma^2=\frac{p(1-p)}{n}\)

  When \(H_0\) is true, we can substitute \(p_1=p_1=p\)

  • \[Z=\frac{\hat{p_1}-\hat{p_2}-(p_1-p_2)}{\sqrt{p(1-p)\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}}\sim\mathscr{N}(0, 1) \]