# Functional Link Neural Network Computer Science Essay

Published:

This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.

Functional Link Neural Network has emerged as an important tool for solving non-linear classification problem and has been successfully applied in many engineering and scientific problems. FLNN is known to be conveniently used for pattern recognition with faster convergence rate and lesser computational load compared to ordinary feedforward network like the Multilayer Perceptron (MLP). This paper presents the ability of Functional Link Neural Network (FLNN) in overcoming the complexity structure of MLP and proposes a Modified Artificial Bee Colony (mABC) to optimize the FLNN learning in order to achieve better classifications accuracy result. The result of the classification accuracy made by FLNN with mABC as learning scheme (FLNN-mABC) is compared with the original FLNN architecture with standard Backpropagation (BP) learning scheme (FLNN-BP) and standard ABC algorithm learning scheme (FLNN-ABC). Our experimental results showed the network performance by FLNN-mABC outperformed both FLNN-BP and FLNN-ABC in term of classification accuracy percentage.

## Keywords: Functional Link Neural Network, Learning Scheme, Artificial Bee Colony

## INTRODUCTION

Classification is one of the most frequent studies in the area of Artificial Neural Networks (ANNs) [1]. The recent vast research activities in neural classification have established that ANNs are indeed a promising tool and have been widely applied to various real world classification task especially in industry, business and science [1-3]. One of the best known types of Neural Networks is the Multilayer Perceptron (MLP). The MLP structure consists of multiple layers of nodes which give the network the ability to solve problems that are not linearly separable. However, MLP usually requires a fairly large amount of available measures in order to achieve good classification ability. When the number of input to the model and the number of training set becomes large, the MLP architecture will become complex and thus resulting in slower operation. An alternative approach of avoiding this problem is by removing the hidden layers from the architecture which prompted to an alternative network architecture named Functional Link Neural Network (FLNN) [4]. The FLNN is a type of a flat network (without the existences of hidden layers) which reduced the neural architectural complexity while at the same time possesses the ability to provide a nonlinear decision boundary for solving non-linear separable classification tasks.

FLNN network is usually trained by adjusting the weight of connection between neurons. The standard method for tuning the weight in FLNN is by using a Backpropagation (BP) learning algorithm. The BP-learning algorithm developed by Rumelhart [1] is the most well-known and widely used for training a Neural Networks. The idea of BP-learning algorithm is to reduce network error, until the networks learned the training data. However, one of the crucial problems with the standard BP-learning algorithm is that it can easily get trapped in local minima especially for those non-linearly separable classification problems [5]. To recover the drawback of BP-learning, Artificial Bee Colony (ABC) optimization algorithm with the ability of exploration and exploitation in searching optimal weight set in training neural network [6], is used to optimize the FLNN weights. The ABC algorithm was originally proposed by Karaboga [7] for solving numerical optimization problem by simulating the intelligent foraging behaviour of a honey bee. In this study, some modifications to the standard ABC (mABC for short) are introduced for training the FLNN. The modifications are implemented on the employed bees' foraging behaviour to provide with comprehensive local search ability for the FLNN network in searching the optimal weights set. Our Experimental results showed that the FLNN performance trained with mABC learning scheme led to a better accuracy on classifying the out-of-sample or unseen data.

## FUNCTIONAL LINK NEURAL NETWORK

Functional Link Neural Network (FLNN) is a class of Higher Order Neural Networks (HONNs) that utilize higher combination of its inputs [4, 6]. It was created by Klassen and Pao [6] and has been successfully used in many applications such as system identification [7-12], channel equalization [13], classification [14-17], pattern recognition [18, 19] and prediction [20, 21]. FLNN is much more modest than MLP as it has a single-layer of trainable weights compared to the MLP whilst able to handle a non-linear separable classification tasks. The flat architecture of FLNN has also make the learning algorithm in the network less complicated [2]. In order to capture non-linear input-output mapping for a classification task, the input vector of FLNN is extended with a suitable enhanced representation of the input nodes which artificially increase the dimension of input space [4, 6].

Our focused in this work is on FLNN with generic basis architecture. This architecture uses a tensor representation. Pao [6], Patra [8], Namatamee [22] has demonstrated that this architecture is very effective for classification tasks. Fig.1 depicts the functional link neural network structure up to the second order with 3 inputs. The first order consist of the 3 inputs x1, x2 and x3, while the second order of the network is the extended input based on the product unit x1x2, x1x3, and x2x3. The learning part of this architecture on the other hand, consists of a standard Backpropagation as the training algorithm.

Fig. 1. The 2nd order FLNN structure with 3 inputs

## FUNCTIONAL LINK NEURAL NETWORK LEARNING SCHEME

Most previous learning algorithm used in the training of FLNN, is the BP-learning algorithm [2, 5, 15, 21, 23-26]. As shows as in Fig. 1, the weight values between enhanced input nodes and output node are randomly initialized. The output node, of FLNN would correspond to the input pattern x and the number of input patterns, n. For tensor representation with single output node, enhance input can be noted as n+n(n-1)/2. Let the enhanced input node of tensor x be represented as . Let f denotes the output node's activation function which in this work we applied a logistic sigmoid activation function:

(1)

The output value of the FLNN is obtained by:

(2)

where is the output while denotes the output node activation function and is the bias. In Eq. (1), is the aggregate value which is the inner product of and . The square error E, between the target output and the actual output will be minimized as:

(3)

where is the target output and is the actual output of the ith input training pattern, while n is the number of training pattern. During the training phase, the BP-learning algorithm will continue to update w and b until the maximum epoch or the convergent condition is reached.

Although BP-learning is the mostly used algorithm in the training of FLNN, the algorithm however has several limitations; It is tends to easily gets trapped in local minima especially for those non-linearly separable classification problems. The convergence speed of the BP learning also can gets is too slow even if the learning goal, a given termination error, can be achieved. Besides, the convergence behavior of the BP-learning algorithm depends on the choices of initial values of the network connection weights as well as the parameters in the algorithm such as the learning rate and momentum [5].

## STANDARD ARTIFICIAL BEE COLONY OPTIMIZATION

The Artificial Bee Colony (ABC) algorithm is an optimization tool, which simulates the intelligent foraging behavior of a honey bee swarm for solving multidimensional and multimodal optimization problem [27]. In this model, three groups of bees which are employed, onlooker and scout bees determined the objects of problems by sharing information to one another. The employed bee uses random multidirectional search space in the Food Source area (FS). They carry the profitability information (nectar quantity) of the FS and share this information with the onlookers. Onlooker bees evaluate the nectar quantity obtained by the employed and bees and choose FS depending on the probability value base on the fitness. If the nectar amount of FS is higher than that of the previous one in their memory, they memorize the new position and forget the previous one [27]. The employed bee whose food source has been abandoned becomes a scout and starts to search for finding a new food source randomly. The following is the standard ABC pseudo code:

Initialization population of scout bee with random solution xi,j i=1,2…FS

Evaluate fitness of the population

Cycle = 1:MCN

form new population for the employed bees using:

(4)

where k is a random selected solution in the neighbourhood of i, Φ is a random number in the range [-1,1] while j is a random selected dimension vector in i and evaluate them

Apply greedy selection between and

Calculate the probability values pi for the solutions xi using:

(5)

Produce the new solutions υi for the onlookers from the solutions xi selected depending on pi and evaluate them

Apply the greedy selection process for onlookers

Determine the abandoned solution for the scout, if exists, and replace it with a new randomly produced solution xi using

(6)

Memorize the best solution

cycle=cycle+1

Stop when cycle = Maximum cycle number (MCN).

## FUNCTIONAL LINK NEURAL NETWORK WITH MODIFIED ARTIFICIAL BEE COLONY

In this study, a modified Artificial Bee Colony (mABC) is introduced as a learning scheme for training the FLNN. The modification is done on the part of employed bees' foraging phase so that they would exploit all weights and biases in the FLNN architecture in order to improve the network ability on searching the optimal weights set. In standard ABC algorithm, the position of a food source (FS) represents a possible solution to the optimization problem, and the nectar amount of a food source corresponds to the profitability (fitness) of the associated solution. In the case of training the FLNN with ABC, the weight, and bias, of the network are treated as optimization parameters to the optimization problem (finding minimum Error, E) as presented in Eq. (3). The FLNN optimization parameters are represented as D-dimensional vector for the solution xi,j where and and each vector is exploited by only one employed bee. In order to produce a candidate food source from the old one xi,j in memory, the ABC uses Eq. (4) where and both k and j are a randomly chosen indexes. The food source of xi,j can be represented in a form of matrix.

(7)

As can be seen from Eq. (4) and matrix representation from Eq. (7), for each row of FS only one element from D will be chosen randomly and exploited by the employed bee by using:

(8)

However in the case of FLNN mainly for classification tasks which are always deal with large number of optimization parameters (weights + bias), exploiting one element in each solution vector xi will cause longer foraging cycle in finding the optimal solution [28]. Random selection of elements in each vector xi during employed bee phase also leads to a poor ability for FLNN network in finding the optimal weights set which result to a low classification accuracy on unseen data [29]. To overcome this, we eliminate the random employed bee behavior in selecting the elements in vector dimension as in Eq. (8). In the other hand, we direct the employed bee to visit all elements in D to exploit them before evaluating the vector xi. The modified ABC is performed as shown in pseudo code below, where the box indicates the improvement made to the standard ABC:

Cycle = 0

Initialize FLNN optimization parameters, D

Initialize population of scout bee with random solution xi, i=1,2…FS

Evaluate fitness of the population

Cycle = 1:MCN

Form new population ( for employed bees

select solution, k in the neighbourhood of i, randomly

For j = 1:D

Direct employed bee to exploit nectar value of j in population () using Eq. (4)

where is a dimension vector in i

j= j+1;

exit loop when j = D;

evaluate the new population (

Apply greedy selection between and

Calculate the probability values pi for the solutions xi using Eq. (4)

Produce the new solutions υi for the onlookers from the solutions xi selected depending on pi and evaluate them

Apply the greedy selection process for onlookers

Determine the abandoned solution for the scout, if exists, and replace it with a new randomly produced solution xi using Eq. (5)

Memorize the best solution

cycle=cycle+1

Stop when cycle = Maximum cycle number (MCN).

## SIMULATION RESULTS

In order to evaluate the performance of the FLNN model trained with modified ABC (FLNN-mABC) for classification problems, simulation experiments were carried out on a 2.30 GHz Core i5-2410M Intel CPU with 8.0 GB RAM in a 64-bit Operating System. The comparison of mABC algorithm with standard BP training and standard ABC algorithms is discussed based on the simulation results implemented in Matlab 2010b. In this work we considered 7 benchmark of classification problems obtained from UCI Machine Learning Repository [30]; Breast Cancer Wisconsin (CANCER), PIMA Indian Diabetes, Indian Liver Patient Disorder (ILPD), IRIS dataset, GLASS Identification, THYROID Disease and WINE dataset.

During the experiment, simulations were performed on the training of the second order FLNN architecture with Backpropagation algorithm (FLNN-BP), second order FLNN architecture with ABC algorithm (FLNN-ABC) and second order FLNN architecture with modified ABC algorithm (FLNN-mABC). The best training accuracy for every benchmark problems were noted from these simulations. The Learning rate and momentum used for the FLNN-BP were 0.3 and 0.7 with the maximum of 1000 epoch and the minimum error=0.001 as for the stopping criteria. Parameters setup for the both FLNN-ABC and FLNN-mABC however, only involved the setting up of stopping criteria of maximum 1000 cycles and minimum error=0.001. The activation function used for the FLNN network output is Logistic sigmoid function. Table 1 below summarized the parameters considered in this simulation.

Table 1. Parameters considered for FLNN-BP, FLNN-ABC and FLNN-mABC simulation

## parameters

## FLNN-BP

## FLNN-ABC

## FLNN-mABC

Learning rate

0.3

## -

## -

Momentum

0.7

## -

## -

Maximum epoch/cycle

1000

1000

1000

Minimum error

0.001

0.001

0.001

Ten trials were performed on each simulation of the FLNN-BP, FLNN-ABC and FLNN-mABC with the best accuracy result is noted from these 10 trials. In order to generate the training and test sets, each datasets were randomly divided into two equal sets (1st-Fold and 2nd-Fold). Each of these two sets was alternately used either as training set or as a test set. The average values of each datasets result were then used for comparison. Table 2 below, presents the simulation result on FLNN-BP, FLNN-ABC and FLNN-mABC models.

Table 2. Result obtained from FLNN-BP, FLNN-ABC and FLNN-mABC models

## Datasets

## Classification Accuracy (%)

## FLNN-BP

## FLNN-ABC

## FLNN-mABC

## CANCER

Training set

97.63%

98.49%

98.43%

Test set

96.50%

95.40%

96.66%

## BUPA

Training set

65.09%

70.78%

74.97%

Test set

60.97%

63.53%

70.44%

## ILPD

Training set

71.21%

73.20%

74.26%

Test set

66.98%

68.84%

70.82%

## IRIS

Training set

93.07%

93.00%

96.95%

Test set

92.02%

89.34%

94.74%

## GLASS

Training set

87.47%

87.92%

89.08%

Test set

83.77%

83.40%

86.62%

## THYROID

Training set

92.28%

96.06%

96.73%

Test set

91.09%

94.02%

95.70%

## WINE

Training set

97.73%

96.29%

99.45%

Test set

95.42%

91.30%

95.77%

Table 2, shows that incorporating FLNN model with modified ABC (FLNN-mABC) as learning scheme has given a better accuracy result both for training set and test Set for all the datasets. The accuracy result on classifying an unseen data by FLNN trained with modified ABC (FLNN-mABC) model also has increase up to 1.26% as for CANCER, 6.91% as for BUPA, 1.98% as for ILPD, 5.4% as for IRIS, 3.22% as for GLASS, 1.68% as for THYROID and 4.47% as for WINE datasets as compared to the FLNN trained with standard ABC (FLNN-ABC). Hence it can be seen that training scheme by mABC algorithm has facilitate the FLNN learning by providing a good exploitation capabilities in the dimension vector of problem solutions in searching optimal weights set in the FLNN weights space as compared to the standard ABC algorithm.

## CONCLUSION

In this work, we evaluated the FLNN-mABC model for the task of pattern classification problems. The experiment has demonstrated that FLNN-mABC performs the classification task quite well. For the case of CANCER, BUPA, ILPD, IRIS, GLASS, THYROID and WINE, the simulation result shows that the proposed modified ABC (mABC) algorithm can successfully train the FLNN for solving classification problems with better accuracy percentage on unseen data.

## ACKNOWLEDGEMENT

The authors wish to thank the Ministry of Higher Education Malaysia and Universiti Tun Hussein Onn Malaysia for the scholarship given in conducting these research activities.