CCJ PRML Study Note - Chapter 1 Summary : MLE (Maximum-likelihood Estimate) and Bayesian Approach
Chapter 1 Summary : MLE (Maximum-likelihood Estimate) and Bayesian Approach
Chapter 1 Summary : MLE (Maximum-likelihood Estimate) and Bayesian Approach
Christopher M. Bishop, PRML, Chapter 1 Introdcution
1. Notations and Logical Relation
- Training data:
input values 
and their corresponding target values 
. For simplicity, written as 
. - Goal of Making Prediction: to be able to make predictions for the target variable
given some new value of the input variable 
. - Assumption of the predictive distribution over : we shall assume that, given the value of , the corresponding value of has a Gaussian distribution with a mean equal to the value y(x, w) of the polynomial curve given by (1.1). Thus we have
- Likelihood function of i.i.d. training data
: 
- MLE of parameters
and 
:
for linear regression
: 
- ML plugin prediction for new values of : substituting the maximum likelihood parameters into (1.60) to give
- Prior distribution over : For simplicity, let us consider a Gaussian distribution of the form
where
- hyperparameter
is the precision of the distribution, - M +1 is the total number of elements in the vector for an
order polynomial.
- hyperparameter
- Posterior distribution for : using Bayes’ Theorem,
- MAP: a step towards a more Bayesian approach, note MAP is still a point estimate. We find that the maximum of the posterior is given by the minimum of
Although we have included a prior distribution
, we are so far still making a point estimate of
- Fully Bayesian approach:
- Here we shall assume that the parameters
and 
are fixed and known in advance (in later chapters we shall discuss how such parameters can be inferred from data in a Bayesian setting). - A Bayesian treatment simply corresponds to a consistent application of the sum and product rules of probability, which allow the predictive distribution to be written in the form
- Here we shall assume that the parameters
- Result of Integration in (1.68):
- (1.66): this posterior distribution is a Gaussian and can be evaluated analytically.
- (1.68) can also be performed analytically with the result that the predictive distribution is given by a Gaussian of the form
where the mean and variance are given by 
Here the matrix S is given by 
where 
is the unit matrix, and we have defined the vector 
with elements 
for 
.
2. Flowchart
The relation between all of those equations or notions above:
