Ordinal Logit Regression

Ordinal Logit Regression

1. Definition

• Ordinal variable of target: \(Y\)

  • i.e., unlikely < somewhat likely < very likely

• Number of categories of the target variable \(Y\): \(J\)

• Cumulative probability of \(Y\) less than or equal to a specific category: \(P(Y \leq j), j = 1, \cdots, J-1\).

The odds of being less than or equal a particular category can be defined as

\[\frac{P(Y \leq j)}{P(Y > j)}, \quad j = 1, \cdots, J-1 \]

1.1. Logit Regression

The log odds is also known as the logit, so that

\[\log \left( \frac{P(Y \leq j)}{P(Y > j)} \right) = \text{logit} \left( P(Y \leq j) \right) \]

The ordinal logistic regression model is parameterized as

\[\text{logit} \left( P(Y \leq j) \right) = \theta_{j0} - \boldsymbol{\beta}^{\top} \boldsymbol{x} \]

1.2. Probit Regression

1.3 Log-likelihood function

\[\log {\mathcal {L}}(\boldsymbol{\beta}, \boldsymbol{\theta } \mid \boldsymbol{x}_{i} , y_{i}) = \sum _{k=1}^{K} \mathbb{I}(y_{i}=k) \log \left[ \Phi (\theta_k - \boldsymbol{\beta}^\top \boldsymbol{x}_{i}) - \Phi ( \theta_{k-1} - \boldsymbol{\beta}^\top \boldsymbol{x}_i) \right] \]

where \(\Phi (x)\) indicate the quantile function, i.e.

\[\Phi (x) = P(X \leq x) \]

Reference

Ordinal regression, Wikipedia, site

ORDINAL LOGISTIC REGRESSION | R DATA ANALYSIS EXAMPLES, site

Frank, E., Hall, M. (2001). A Simple Approach to Ordinal Classification, Machine Learning: ECML 2001, vol 2167. Springer, Berlin, Heidelberg. doi.org/10.1007/3-540-44795-4_13