深入解析：Deep Learning|02 Handcraft Code of BRF Network

Deep Learning|02 Handcraft Code of BRF Network

Implement a RBF Network in Stock

文章目录

• Deep Learning|02 Handcraft Code of BRF Network
• • Based on RBF general form, mathmatical training process of RBF network!
  • Handcraft code

Genrally, training RBF network is divided into two steps: the first step is to select the centers of neurons; the second step is to optimize the network parameters using the BP algorithm. There are many methods for selecting neuron centers, such as random selection, clustering-based selection, and so on. At the same time, we can also select the centers of RBFs through supervised learning, which is also the most general form of RBF networks.

Based on RBF general form, mathmatical training process of RBF network!

We will derive the training process of RBF network based on Gauss core.

1. Gauss core

The definition of Gauss core is :

$$\begin{array}{r}\Phi (x_i,c_j)=e^{-\frac{\Vert x_i-c_j{\Vert }^2}{2{\sigma }^2}}\end{array}$$

where $c_j$is the center point of the$j$-th neuron;$\sigma$is the width of the Gaussian kernel, and$\Vert x_i-c_j\Vert$is the Euclidean distance from the sample$x_i$to the center point$c_j$.

2. BRF network

BRF network is defined by :

$$f(x)=\sum _{j=1}^q{w}_j\cdot \Phi (x,c_j)$$

where $w_j$is theweightof the$j$-th neuron,$q$is the total number of neuron.

3. Error function

We define the error function as the mean squared error, and the goal is to minimize the error function:

$$\begin{array}{rl}E& =\frac{1}{2m}\sum _{i=1}^m{e}_i^2\\ & =\frac{1}{2m}\sum _{i=1}^m(f(x)-y{)}^2\\ & =\frac{1}{2m}\sum _{i=1}^m{\left(\sum _{j=1}^q{w}_j\cdot \Phi (x,c_j)-y\right)}^2\end{array}$$

We use the BP algorithm to propagate errors backward and the Gradient Descent method to respectively determine the optimization directions for the parameters of the RBF network.

• Linear weights of output layer neurons
  $$\Delta w=\frac{\partial E}{\partial w}=\frac{1}{m}\sum _{i=1}^m(f(x)-y)\cdot \phi (x,c)=\frac{1}{m}\sum _{i=1}^m{e}_i\cdot \phi (x,c)$$

• Weight iteration formula

$$w_{k+1}=w_k-\eta \cdot \Delta w$$

• Neuron center points of hidden layer

$$\begin{array}{rl}\Delta c_j& =\frac{\partial E}{\partial c_j}=\frac{\partial E}{\partial \phi (x,c_j)}\cdot \frac{\partial \phi (x,c_j)}{\partial c_j}\\ & =\frac{1}{m}\sum _{i=1}^m(f(x)-y)w\cdot \frac{\partial \phi (x,c_j)}{\partial c_j}\\ & =\frac{1}{m}\sum _{i=1}^m(f(x)-y)w\cdot \phi (x,c_j)\cdot \frac{x-c_j}{{\sigma }_j^2}\\ & =\frac{1}{m\cdot {\sigma }_j^2}\sum _{i=1}^m(f(x)-y)w\cdot \phi (x,c_j)\cdot (x-c_j)\end{array}$$

Neuron center point iteration formula

$$c_{k+1}=c_k-\eta \cdot \Delta c$$

• Gaussian kernel width of hidden layer

$$\begin{array}{rl}\Delta {\sigma }_j& =\frac{\partial E}{\partial {\sigma }_j}=\frac{\partial E}{\partial \phi (x,c_j)}\cdot \frac{\partial \Phi (x,c_j)}{\partial {\sigma }_j}\\ & =\frac{1}{m}\sum _{i=1}^m(f(x)-y)w\cdot \frac{\partial \Phi (x,c_j)}{\partial {\sigma }_j}\\ & =\frac{1}{m\cdot {\sigma }_j^3}\sum _{i=1}^m(f(x)-y)w\cdot \Phi (x,c_j)\cdot \Vert x_i-c_j{\Vert }^2\end{array}$$

Gaussian kernel width iteration formula
$${\sigma }_{k+1}={\sigma }_k-\eta \cdot \Delta \sigma$$

Handcraft code

import numpy as np
import matplotlib.pyplot as plt
import os
class BRF:
    def __init__(self, hidden_nums, r_w, r_c, r_sigma, tol=1e-5):
        self.hidden_nums = hidden_nums
        self.r = {
            'w': r_w,
            'c': r_c,
            'sigma': r_sigma
        }
        self.tol = tol
        self.errList = []
        self.c = None
        self.w = None
        self.sigma = None
    def train(self, X, y, iters):
        self.X = X
        self.y = y.reshape(-1, 1)
        self.n_samples, self.n_features = X.shape
        sigma, c, w = self.init()
        for i in range(iters):
            hi_output = self.change(sigma, X, c)
            yi_input = self.addIntercept(hi_output)
            yi_output = np.dot(yi_input, w)
            error = self.calSSE(yi_output, self.y)
            if error < self.tol:
                break
            self.errList.append(error)
            delta_w = np.dot(yi_input.T, (yi_output - self.y))
            w -= self.r['w'] * delta_w / self.n_samples
            delta_sigma = np.divide(
                np.multiply(
                    np.dot(np.multiply(hi_output, self.l2(X, c)).T, (yi_output - self.y)),
                    w[:-1]
                ),
                sigma**3
            )
            sigma -= self.r['sigma'] * delta_sigma / self.n_samples
            deltac1 = np.divide(w[:-1], sigma**2)
            deltac2 = np.zeros((1, self.n_features))
            for j in range(self.n_samples):
                deltac2 += (yi_output - self.y)[j] * np.dot(hi_output[j], X[j] - c)
            deltac = np.dot(deltac1, deltac2)
            c -= self.r['c'] * deltac / self.n_samples
        self.c = c
        self.w = w
        self.sigma = sigma
        self.n_iters = i
    def guass(self, sigma, X, ci):
        return np.exp(-np.linalg.norm((X - ci), axis=1)**2 / (2 * sigma**2))
    def change(self, sigma, X, c):
        newX = np.zeros((self.n_samples, len(c)))
        for i in range(len(c)):
            newX[:, i] = self.guass(sigma[i], X, c[i])
        return newX
    def init(self):
        sigma = np.random.random((self.hidden_nums, 1))
        c = np.random.random((self.hidden_nums, self.n_features))
        w = np.random.random((self.hidden_nums + 1, 1))
        return sigma, c, w
    def addIntercept(self, X):
        return np.hstack((X, np.ones((self.n_samples, 1))))
    def calSSE(self, prey, y):
        return 0.5 * (np.linalg.norm(prey - y))**2
    def l2(self, X, c):
        m, n = np.shape(X)
        newX = np.zeros((m, len(c)))
        for i in range(len(c)):
            newX[:, i] = np.linalg.norm((X - c[i]), axis=1)**2
        return newX
    def predict(self, X):
        hi_output = self.change(self.sigma, X, self.c)
        yi_input = self.addIntercept(hi_output)
        yi_output = np.dot(yi_input, self.w)
        return yi_output