Decision Tree & Random Forest

1 Decision Tree

1.1 Information content

This measures the surprise or information gained when an event x occurs:

$$I(x)=-log_{2}P(x)$$

• if P(x) is high, then I(x) is small
• if P(x) is low, then  I(x) is large

 

1.2 Information Entropy

Entropy is expected information content. This is a central quantity in information theory, measuring the average uncertainty or "information" of a random variable X.

$$H(x)=E(I(x))=-\sum_{x\in \chi}^{}P(x)log_{2}P(x)$$

 

 

1.3 ID3 Algorithm

Core Idea: Recursively split the dataset based on attributes that provide the most information gain. 

$$IG(S,A) = H(S)-\sum_{v\in Value}^{}\frac{S_{v}}{S}H(S_{v})$$

 

pros:

• simple, intuitive
• fast training, no data normalization needed

cons:

• no pruning
• can not handle continuous / numeric data
• no handle of missing data
• sensitive to small variations in data
• when a variable has many values, the second part of the formula is small, so the gain is high. It affects the final result.

 

1.4 C4.5 Algorithm

C4.5 is the direct successor to ID3, 

$$GainRatio(S,A)= \frac{SplitInformation(S,A)}{InformationGain(S,A)}$$

$$SplitInformation(S,A)=−\sum_{i=1}^{v}\frac{|S_{i}|}{|S|}log_{2}\frac{|S_{i}|}{|S|}$$

• The more values S has, the higher SplitInformation is. Then the final result is affected by SplitInformation

 

1.5 CART Algorithm

$$GiniGain(D,X)=\sum_{i=1}^{n}\frac{|D_i|}{|D|}*Gini(D_i)$$

$$Gini(D)=1-\sum_{i=1}^{n}p_{i}^{2}$$

$$GiniGain(D,X)=\sum_{i=1}^{n}\frac{|Di|}{|D|}*Gini(Di)$$

 

2. Random Forest

Random Forest is an ensemble learning method that combines multiple decision trees to create a more accurate and stable predictive model.

2.1 Bootstramp Sampling (Bagging)

• Creates many different training datasets by randomly sampling with replacement

• Each tree gets a slightly different version of the original data 

2.2 Feature Randomness

• At each split in each tree, only a random subset of features is considered

• Prevents all trees from looking the same

• Reduces correlation between trees

2.3 Advantages:

• High accuracy - often outperforms single decision trees

• Robust to overfitting - unlike single decision trees

• Handles missing values well

• No need for feature scaling (unlike SVM or neural networks)

• Provides feature importance scores

• Works "out of the box" with minimal tuning