Nonparametric Statistic

Nonparametric Statistic

1. Introduction

Parametric statistical methods

• These statistical testing methods assume that \(X \sim \mathcal{N}(\mu, \sigma^2)\) or \(\varepsilon \sim \mathcal{N}(0, \sigma^2)\)

Non-parametric statistical methods

• A statistical method is nonparametric if it satisfies at least one of the following criteria:

• The method may be used on data with a nominal scale of measurement

• The method may be used on data with an ordinal scale of measurement

• The method may be used on data with an interval or ratio scale of measurement, where the distribution function of the random variable producing the data is either unspecified or specified except for an infinite number of unknown parameters

Advantage of non-parametric methods

• Data need NOT be quantitative (can be categorical or ranked)

• Test procedures are quick and easy to perform

Disadvantages of non-parametric methods

• Do not make use of full information in the samples (e.g., numerical sampling values).

• Normally requires a larger sample size c.f. parametric methods.

• Less efficient – not as easy to reject \(H_0\)

• As far as possible, use parametric statistical methods

2. Sign Test

Assumptions:

• \(X\) follows a distribution with a median \(\tilde{\mu}\), i.e..,

  • \(\tilde{\mu} : P(X \leq \tilde{\mu}) = P(X \geq \tilde{\mu}) = 0.5\)

• A sample \({x_1,x_2,\cdots,x_n}\)

Hypotheses:

\[\begin{aligned} H_0 : & \ \tilde{\mu} = \tilde{\mu}_0 \\ H_1 : & \ \tilde{\mu} \neq \tilde{\mu}_0 \text{ or } \tilde{\mu} < \tilde{\mu}_0 \text{ or } \tilde{\mu} > \tilde{\mu}_0 \end{aligned} \]

Determine the test statistics

• Step 1: From \(n\) samples \(\{x_1,x_2,\cdots,x_n \}\), compute: \(\{x_i-\tilde{\mu}_0, i=1,\cdots,n\}\)

• Step 2: Count \(r+\) , the number of "+" signs in \(\{x_i-\tilde{\mu}_0, i=1,\cdots,n\}\)

• test statistic: \(R^+\)

Remark: If \(x_i-\tilde{\mu}_0\), set this data point aside, do not include in sign test.

If \(X\) has median value of \(\tilde{\mu}_0\), then we have

\[\begin{aligned} P("+" \text{ sign}) &= P(X>\tilde{\mu}_0) = 0.5 = p \\ P("-" \text{ sign}) &= P(X<\tilde{\mu}_0) = 0.5 = p \end{aligned} \]

Note: we can do this because it is median instead of mean

The distribution of combinations of “+” and “-“ signs follows the binomial distribution (for any given value of \(p\))

\[\binom{n}{r} \, p^r \, (1-p)^{n-r} \]

P-value: use the binomial distribution (with \(p=0.5\)) to construct critical regions and critical values.

2.1 One-sided and two-sided Hypothesis

2.1.1 One-side Alternative Hypothesis: \(H_1:\tilde{\mu} > \tilde{\mu}_0\)

Null hypothesis: \(H_1:\tilde{\mu} > \tilde{\mu}_0\)

P-value:

\[P\text{-value}=\Pr(R^+ \ge r^+) = \sum_{r=r^+}^n \binom{n}{r} \, p^r \, (1-p)^{n-r} \]

where is the area under the curve for \(R^+ \ge r^+\)

Analysis

• If \(H_1:\tilde{\mu}>\tilde{\mu}_0\), more "+" signs, higher \(r^+\)

• Reject \(H_0\) when P-value \(\Pr(R^+ \ge r^+) <\alpha\), that is to say \(r+\) falls into the critical region on the right hand side (RHS) of binomial probability.

2.1.2 One-side Alternative Hypothesis: \(H_1:\tilde{\mu} > \tilde{\mu}_0\)

• Null hypothesis: \(H_1:\tilde{\mu} < \tilde{\mu}_0\)

• P-value:

\[P\text{-value}=\Pr(R^+ \le r^+) = \sum_{r=0}^{r^+} \binom{n}{r} \, p^r \, (1-p)^{n-r} \]

where is the area under the curve for \(R^+ \le r^+\)

• Analysis

  • If \(H_1:\tilde{\mu}<\tilde{\mu}_0\), lesser "+" signs, lower \(r^+\)

  • Reject \(H_0\) when P-value \(\Pr(R^+ \le r^+) <\alpha\), that is to say \(r+\) falls into the critical region on the left hand side (LHS) of binomial probability.

2.1.3 Two-side Alternative Hypothesis: \(H_1:\tilde{\mu} \neq \tilde{\mu}_0\)

P-value:

• Case 1: If \(r^+ < \dfrac{n}{2}\), then

\[P\text{-value}=2 \Pr (R^+\le r^+) = 2 \sum \limits_{r=0}^{r^+} \binom{n}{r} p^{r} (1-p)^{n-r} \]

• Case 2: If \(r^+ > \dfrac{n}{2}\), then

\[P\text{-value}=2 \Pr (R^+ \ge r^+) = 2 \sum \limits_{r=r^+}^{n} \binom{n}{r} p^{r} (1-p)^{n-r} \]

Analysis: Reject \(H_0\):

• when \(\Pr (R^+ \ge r^+) < \alpha /2\) provieded \(r^+ > n /2\)

• or \(\Pr (R^+ \le r^+) < \alpha /2\) provided \(r^+ \leq n /2\).

That is to say that \(r^+\) falls into the critical region on either tail of binomial probability.

2.2 Normal Distribution Approximation for Sign Test

The binomial distribution \(\text{B}(n, p)\) can be approximated by the normal distribution \(\mathcal{N}(np, np(1-p))\) if \(np \geq 5\) and \(n(1-p)\geq 5\)

Test statistic:

\[Z_0 = \frac{R^{+}-n p}{\sqrt{n p(1-p)}} \]

Hypothesis test:

• Case 1: If \(H_1: \tilde{\mu} > \tilde{\mu}_0\), reject \(H_0\) when \(z_0 > z_{\alpha}\)

• Case 2: If \(H_1: \tilde{\mu} < \tilde{\mu}_0\), reject \(H_0\) when \(z_0 < - z_{\alpha}\)

• Case 3: If \(H_1: \tilde{\mu} \ne \tilde{\mu}_0\), reject \(H_0\) when \(|z_0| > z_{\alpha / 2}\)

2.3 Sign Test for Paired Samples

Aims: Give significane level \(\alpha\), test whether two populations have the same median

Assumption: \(n\) pairs of observations have been made from 2 populations each with a continuous distribution, e.g. \((x_{1j},x_{2j}), j=1,2,\cdots,n\)

Hypotheses:

\[\begin{aligned} &H_0: \tilde{\mu}_1=\tilde{\mu}_2 \quad \Rightarrow \quad \tilde{\mu}_D=\tilde{\mu}_1-\tilde{\mu}_2=0 \quad \Rightarrow \quad \Pr(+)=\Pr(-)=0.5 \\ &H_1: \tilde{\mu}_1 \neq \tilde{\mu}_2 \quad \Rightarrow \quad \tilde{\mu}_D=\tilde{\mu}_1-\tilde{\mu}_2 \neq 0 \quad \Rightarrow \quad \Pr(+) \neq \Pr(-) \neq 0.5 \end{aligned} \]

Test statistics:

\[r = \min (r^+, r^-) \]

where:

• \(r^+\) the number of plus signs in the sample, \(\{d_1, d_2, \cdots, d_n\}\)

• \(r^-\) the number of minus signs in the sample, \(\{d_1, d_2, \cdots, d_n\}\)

• \(d_j\) is calculated as:

\[d_j = x_{1j}-x_{2j} \quad \forall j=1,2,\cdots,n \]

Remark: \(r < n/2\)

Critical value for two-side test \(r_{\alpha/2}^*\)

• A maximal integer \(r_{\alpha/2}^*\) such that:

\[\Pr \left( R^{+} \leq r_{\alpha / 2}^* \right) = \sum_{r=0}^{r_{\alpha / 2}} \binom{n}{r} \, 0.5^r \, (1-0.5)^{n-r} \leq \alpha / 2 \]

• Reject \(H_0\) if \(r \leq r_{\alpha/2}^*\)

2.3 Type II Error for the Sign Test

• Type II Error

\[\beta = \text{type II error} = \Pr \left( \text{ do not reject } H_0 | H_0 \text{ is false } \right) \]

If \(X \sim \mathcal{N}(\mu, \sigma^2)\), median equals to mean, the \(t\)-test for the mean (or difference between two means) has a smaller \(\beta\) c.f. sign test.

• It is harder for sign test to reject \(H_0\)

• In general, if \(X\) is non-normal but symmetrical, the \(t\)-test will have smaller \(\beta\) c.f. sign test.

• In principle, \(t\)-test only applies to a normal distribution with unknown variance.

3 Wilcoxon Signed-Rank Test

Drawback of sign test

• Utilizes only the plus (+) and minus (-) signs of the differences between the observations in the one-sample case, or the plus and minus of the differences between the pairs of the observations in the pair-sample case

• Does not take into consideration magnitudes of these differences.

Wilcoxon Signed-Rank Test

• Wilcoxon, F. (1945) Individual comparisons by ranking methods. Biometrics, Vol. 1, 80-83.

Assumptions:

• \(X\) follows some one symmetrical and continuous distribution,

  • thus, the mean and median are equal, i.e., \(\mu=\tilde{\mu}\)

Hypotheses

\[\begin{aligned} & H_0 : \mu = \mu_0 \\ & H_1 : \mu \neq \mu_0 \text {, or } \mu>\mu_0 \text {, or } \mu<\mu_0 \end{aligned} \]

Wilcoxon Signed-Rank Test Procedure

• Step 1: From \(n\) samples \(\{x_1,x_2,\cdots, x_n\}\), compute \(x_i-\mu_0, i=1,2,\cdots,n\)

• Step 2: Discard those points with \(x_i = \mu_0\). Rank \(|x_i-\mu_0|\) in ascending order and then attach "+" or "-" to each rank.

• Remark: If several samples have the same \(|x_i-\mu_0|\) then assign the average value of ranks they would receive.

• Step 3: Let

  • \(w^+ = \text{ sum of all "+" ranks}\)

  • \(w^- = \text{ sum of all "-" ranks}\)

Test Statistic: \(w=\min(w^+, w^-)\)

• \(w^+ + w^- = n(n+1)/2\)

Cases:

• If \(H_1: \mu \neq \mu_0\), reject \(H_0\) when \(\min(w^+, w^-) \leq w_{\alpha/2}^*\)

• If \(H_1: \mu < \mu_0\), reject \(H_0\) when \(w^+ \leq w_{\alpha}^*\)

• If \(H_1: \mu > \mu_0\), reject \(H_0\) when \(w^- \leq w_{\alpha}^*\)

3.1 Normal Distribution Approximation

If \(n>20\), it can be shown that:

\[W^+ \text{ or } W^- \sim \mathcal{N}(\mu_w, \sigma_w^2) \]

where

\[\mu_w = \frac{n(n+1)}{4}, \quad \sigma_w^2 = \frac{n(n+1)(2n+1)}{24} \]

To test \(H_0: \mu=\mu_0\), we can use the \(z\) test

\[z = \frac{w-\mu_w}{\sigma_w} \]

3.2 Wilcoxon Signed-Rank Test for Paired Observations

Assumption:

• Samples \((x_{1j}, x_{2j}), j=1,2,\cdots,n\) from 2 continuous distributions of the same type but with different mean values.

Hypothesis:

\[\begin{aligned} H_0 : \mu_1 = \mu_2 \quad \text{ or } \quad \mu_D = \mu_1-\mu_2=0 \end{aligned} \]

Procedure:

• The absolute differences \(|x_{1j}-x_{2j}|, j=1,\cdots,n\) are ranked in ascending orders, and the ranks are given the "+" or "-" sign.

• The rest of the test is as usual

3.3 Comparison to the \(t\)-test

• When \(X\) is not normally distributed but continuous random variable, Wilcoxon signed-rank test may be superior than the
  \(t\)-test. Because \(t\)-test assumes that \(X\) follows a normal distribution.

• When \(X\) is normal, \(t\)-test is superior, but Wilcoxon signed-rank test may not be much worst.

• Wilcoxon signed-rank test is a good non-paramterical test

4 Wilcoxon Rank-Sum Test (Mann-Whitney Test)

Assumptions:

• \(X_1\) and \(X_2\) are two independent populations with means \(\mu_1\) and \(\mu_2\), respectively.

• \(X_1\) and \(X_2\) have the same shape and spread, and differ possibly in the means, i.e., \(\text{Var}[X_1] = \text{Var}[X_2]=\sigma^2\)

• There are two independent samples: \({x_{1,1},x_{1,2},\cdots,x_{1,n1}}\) from \(X_1\) and \(\{x_{2,1}, x_{2,2}, \cdots, x_{2,n2} \}\) from \(X_2\) with \(n_1 \leq n_2\).

Hypotheses:

\[\begin{aligned} & H_0 : \mu_1 = \mu_2 \\ & H_1 : \mu_1 \neq \mu_2 \text { or } \mu_1 < \mu_2 \text { or } \mu_1 > \mu_2 \end{aligned} \]

Test Procedure:

• Step 1: Arrange all \(n_1+n_2\) samples in ascending order of magnitude and assign ranks

  • Remarks: If two or more observations are tied, use the mean of the ranks that would be assigned if the values differed slightly

• Step 2: Let \(W_1\) be the sum of the ranks in \(n_1\) (smaller) samples, \(W_2\) be the sum of the ranks in \(n_2\) samples, or

\[W_2 = \frac{(n_1+n_2)(n_1+n_2+1)}{2} - W_1 \]

\(X\) gives critical values \(w_\alpha^*\)

• If \(H_1 : \mu_1 \neq \mu_2\), reject \(H_0\) when either \(W_1\) or \(W_2 \leq w^*_{\alpha/2}\)

• If \(H_1 : \mu_1 < \mu_2\), reject \(H_0\) when \(W_1 \leq w^*_{\alpha}\)

• If \(H_1 : \mu_1 > \mu_2\), reject \(H_0\) when \(W_2 \leq w^*_{\alpha}\)

4.1 Large Sample Approximation

When \(n_1 > 8\) and \(n_2 > 8\), \(W_1\) can be approximated by \(\mathcal{N}(\mu_{W_1}, \sigma_{W_1}^2)\)

\[\begin{aligned} \mu_{W_1}=\frac{n_1\left(n_1+n_2+1\right)}{2} \qquad \sigma_{W_1}^2=\frac{n_1 n_2\left(n_1+n_2+1\right)}{12} \end{aligned} \]

We can use the test statistic:

\[z=\frac{W_1-\mu_{W_1}}{\sigma_{W_1}} \]

Note that if there are many ties in the data, the test statistic is modified as:

\[z = \frac{W_1-\mu_{W_1}}{ \sqrt{ \dfrac{n_1 n_2}{ (n_1+n_2)(n_1+n_2-1)} \times \sum \limits_{i=1}^{n_1+n_2} R_i^2 - \dfrac{n_1 n_2 (n_1+n_2+1)^2}{4(n_1+n_2-1)}}} \]

• where \(R_i\) is the rank assigned to data \(i\)

5 Kruskal-Wallis Test

• Introduced in 1952 by W. H. Kruskal and W. A. Wallis.

• An extension of Wilcoxon Rank-Sum method

Assumptions:

• \(X_1, X_2, \cdots, X_a\) are independent populations with means \(\mu_1,\mu_2,\cdots,\mu_a\), respectively.

• \(X_1, X_2, \cdots, X_a\) have the same shape and spread, and differ possibly in the means, i.e., \(\text{Var}[X_1] = \text{Var}[X_2]=\cdots=\text{Var}[X_a]=\sigma^2\)

• There are \(a\) independent samples:

  • \(\{x_{1,1},x_{1,2},\cdots,x_{1,n_1} \}\) from \(X_1\)

  • \(\{x_{2,1}, x_{2,2},\cdots,x_{2,n_2}\}\) from \(X_2\)

  • \(\{x_{a,1}, x_{a,2},\cdots,x_{a,n_a}\}\) from \(X_a\)

Hypotheses:

\[H_0: \mu_1=\mu_2=\cdots=\mu_a \]

Test Procedure:

• Step 1: Rank \(N\) observations in ascending order of their values where \(N=\sum_{i=1}^{n_a}n_i\)

• Step 2: Rank transformation – refer to replacing observed values by their ranks.

Note: In the case of ties (i.e., identical observations), we follow the usual procedure of replacing the observations by the means of the ranks that the observations would have if they are distinguishable.

Analysis:

• If \(H_0\) is true, then the N observations comes from the same distribution, and all possible assignments of the ranks to the \(a\) factor levels are equally likely to occur.

• Let \(R_{ij}\) be the rank of observation \(x_{ij}\), and \(R_{i \cdot}\) the \(\bar{R}_{i \cdot}\) total and average of the \(n_i\) ranks in the \(i\)th treatment.

When \(H_0\) is true

\[\begin{aligned} & \mathbb{E} [R_{i j}] = \frac{N+1}{2} \\ & \mathbb{E} [\bar{R}_{i \bullet}] =\frac{1}{n_i} \sum_{j=1}^{n_i} \mathbb{E} [R_{i j}] = \frac{N+1}{2} \end{aligned} \]

Kruskal-Wallis test statistic \(H\):

\[\begin{aligned} H &=\frac{12}{N(N+1)} \sum_{i=1}^a n_i\left(\bar{R}_{i \cdot}-\frac{N+1}{2}\right)^2 \\ &= \left[\frac{12}{N(N+1)} \sum_{i=1}^a \frac{R_{i \cdot}^2}{n_i}\right]-3(N+1) \end{aligned} \]

• where \(n_i\) can be treated as the weights of treatment's
  sample size.

• \(\bar{R}_{i \cdot}\) is the average rank for each treatment.

When observations \(x_{ij}\) are tied, assign the average rank to each of the tied observations, and replace the test statistic by

\[H=\frac{1}{S^2}\left[\sum_{i=1}^a \frac{R_{i \cdot}^2}{n_i}-\frac{N(N+1)^2}{4}\right] \]

where \(S^2\) is the variance of the ranks

\[S^2=\frac{1}{N-1} \left[ \sum_{i=1}^a \sum_{j=1}^{n_i} R_{i j}^2-\frac{N(N+1)^2}{4} \right] \]

5.1 Large Sample Approximation

Conditions:

• Case 1: \(a=3\) and \(n_i \geq 6\) for \(i=1,2,3\)

• Case 2: \(a>3\) and \(n_i \geq 5\) for \(i=1,2,\cdots,a\)

\(H\) asymptotically follows the Chi-square distribution with \(a-1\) d.o.f.

\(H_0\) is rejected if \(h \geq \chi^2_{\alpha, a-1}\)