What is a log-bilinear model?

A log-bilinear Language Model (LM) computes the probability of the next word \(w_i\) given the previous words (context) as follows:

\[P(w_{i}=w|w_{i-1},...,w_{1})=\frac{exp(\phi(w)^{T}c)}{\sum_{w^{'} \in V}exp(\phi(w^{'})^{T}c)} \]

Here \(\phi(w)\) is a word-vector and c is the context for \(w_{i}\) computed as

\[c=\sum_{n=1}^{i-1}[\alpha_{n}\phi(w_{n})] \]

Thus, the log-bilinear LM computes a context vector as a linear combination of the previous word vectors. And a distribution of the next word \(w_{i}\) is computed based on similarity between the word embedding \(\phi(w)\) and the context, by taking a softmax over the vocabulary V.

The log-bilinear name comes from the fact that the log of the numerator is a bilinear map: \(f(u, v)=u^{T}v\) for two vectors \(u\) and \(v\).