Ternary Multicomponent Adsorption Modelling Using ANN, LS-SVR, and SVR Approach – Case Study

The aim of this work was to develop three artificial intelligence-based methods to model the ternary adsorption of heavy metal ions {Pb2+, Hg2+, Cd2+, Cu2+, Zn2+, Ni2+, Cr4+} on different adsorbates {activated carbon, chitosan, Danish peat, Heilongjiang peat, carbon sunflower head, and carbon sunflower stem). Results show that support vector regression (SVR) performed slightly better, more accurate, stable, and more rapid than least-square support vector regression (LS-SVR) and artificial neural networks (ANN). The SVR model is highly recommended for estimating the ternary adsorption kinetics of a multicomponent system.


Introduction
Due to increasing concentrations of various toxic and non-biodegradable contaminants like heavy metals in industrial wastewater, which have an adverse effect on human health and the environment, 1 various purification techniques have been proposed for wastewater treatment; adsorption is commonly being employed because of its high removal capacity of heavy metal ions using different adsorbents, low-cost of installation, operation and maintenance, and simple design. 2 Heretofore, numerous studies have reported the simultaneous interactions of multicomponent adsorption phenomenon of heavy metal ions on the adsorbent. [3][4][5] To design the adsorption equipment, it is mandatory to know the adsorption mechanisms. Thus, various empirical and theoretical models have been proposed in the literature to evaluate the equilibrium adsorption of heavy metals, namely Langmuir, Freundlich, Toth, and other models. 6 Since the multicomponent adsorption process is highly complex phenomena explained by the competition and interaction nature (synergism, synergism and non-interaction) 7 between adsorbent and multiple adsorbates, as well as operating conditions (pH, time, temperature, and concentration), it is difficult to model using the theoretical models. 8 Various artificial intelligence methods are presented in the literature to overcome the limitations of the theoretical models. Most of them are established to model the removal of a single heavy metal, but few discuss the application of these models to model the multicomponent heavy met-al adsorption process. 9,10 ANNs are applied successfully to model the non-linear behaviour between dependent and independent variables without knowing any previous details about the physical process in complex systems. 7,[11][12][13][14] However, to the best of our knowledge, very few studies are devoted to the application of LS-SVM or SVM approach to model the competitive adsorption of heavy metals. 15,16 Therefore, the major motivation behind this study was to assess the predictability power of three modelling approaches {ANN, SVM, and LS-SVM} in modelling the nonlinear relationships between the removal capacity from aqueous solution of five ternary heavy metal systems on different adsorbents and the independent parameters. The experimental data set employed in this work to optimise the three model parameters was extracted from previously published literature. The performance of these models will be evaluated using well-known statistical metrics and compared with the experimental data. Artificial neural network (ANN) by its similarity with human brain functionality, can learn the complex relationship between the response and its effecting parameters from previous experimental data set, and can use the obtained knowledge in future predictions. The performance of the ANN depends on some parameters, such as the number of hidden layers and the number of neurons in each hidden layer, the transfer function, and the normalisation func-tion. 17,18 The output of each hidden neuron (j) can be written in the following form in terms of inputs X i , weights and biases, and the transfer function f(x): The final output (k) can also be expressed in the following form: 19 The transfer function, weights and biases can be determined during the training stage.

LS-SVM and SVM model
Support vector machine (SVM) presents a number of superiorities in comparison to ANN, it can map the nonlinear relationship between inputs and output(s) avoiding to be stocked into local minima, it can solve problems using only support vector and it can deal with small data set. 20,21 The performance of SVM model can be determined based on the selected kernel function and its parameters. 16 The predicted output can be expressed via SVM model as follows: 16 (2) where K(x i ,x j ) can be linear, polynomial, Gaussian or radial basis function kernel. α i and b denote Lagrange multiplier and threshold parameter, respectively. In support vector regression, it is mandatory to optimise the following expression: 16 where W denotes the margin and represents the complexity of the SVM model, 1 n i i = ∑ represents the sum of the training errors, and C is a tuning parameter. Compared to the conventional SVM, LS-SVM can convert the inequality constraints into equality constraints. 21 3 ANN, LS-SVR, and SVR modelling

Data set collection and processing
The data set used in this work has been collected from previously published papers in literature and organised in a matrix of {84 points, 11 parameters} including 8 inputs and 3 outputs. Table 1 presents details and sources of the selected data set.  26 The data set was divided into 8 inputs explained as follows: BET surface, structure index of each adsorbent (micropores, mesopores, macropores), molecular weight of each compound (M w1 , M w2 and M w3 ), initial concentration for each compound (c e1 , c e2, c e3 ) and the removal capacity of each heavy metal ion (q e1 , q e2 , and q e3 ) as outputs.
The data set was divided randomly: 74 % for training and 26 % for the test stage (case of SVR and LS-SVR model), and 76 % for training, 12 % for the test, and 12 % for validation (case of ANN model). In order to ensure the rapid convergence of the models, a normalisation stage of inputs was done using the proposed function expressed by expression 4, while the outputs were normalised/post-normalised using the two functions programmed in MATLAB {premnmx(Y)/postmnmx(Y)} as represented in expressions 5 and 6, respectively. The aim of the normalisation was to have the data in the same range and avoid greater errors. 21

Performance analysis of developed models
The performance of these models was assessed using two metrics, namely, the mean squared error (MSE) and the determination coefficient (R 2 ): where N is the number of points, y iexp , y ical , and exp i y are the experimental, calculated, and mean of the experimental target vectors. These metrics explain how well the model can predict the experimental data. 27 Better accuracy of the model can be found when R 2 is close to 1 and MSE is close to 0.

ANN model
In this work, a multi-layer perceptron (MLP) was selected and trained with Levenberg-Marquardt algorithm. Hyperbolic tangent and linear function was adopted in the hidden and the output layers, respectively. Table 2 presents the mathematical expression of these two transfer functions. The flowchart of ANN model development is presented in Fig. 1. This procedure was programmed in MATLAB software.
Since there are no rules for the exact determination of the ANN parameters, a trail-and-error method was adopted and based on the obtained values of the selected metrics. Details of the best ANN model parameters are presented in Table 3.      (11) where w kj and b hj denote weights and biases between inputs-hidden layer. w 1k and b 1k denote weights and biases between hidden-output layer. f o and f h are linear and tangent hyperbolic transfer function. Weight and biases matrix of the proposed optimal ANN-MLP model are presented in Tables 5 and 6 to allow other researchers to reproduce results and make appropriate use of this ANN-MLP model.
An example of the comparison between experimental and predicted values is given in Fig. 3. It shows that the points  Fig. 3 -Predicted by ANN vs experimental removal capacity of the ternary system against concentration predicted by ANN follow exactly and match well the trend of the experimental points, which confirms the capability of the ANN model to model the non-linear behaviour of the multicomponent adsorption system of the selected heavy metal ions.

LS-SVR model
For LS-SVR and SVR, different kernel functions have been tested to model multicomponent adsorption system. Table 7 represents the expression of these functions.  With the selected LS-SVR structure, the maximum MSE of 1.3571 and R 2 above 0.99 were obtained during the generalisation stage. As may be seen in Fig. 4, all predicted points are very close and around the unity line, showing the satisfactory and robust LS-SVR.
The performance of the developed LS-SVR model was further analysed through the calculation of some metrics (Table 8).

SVR model
Compared to the LS-SVR and ANN models, the prediction accuracy of the SVR model was higher. Fig. 5 shows  that R 2 was higher than 0.99, and Table 9 shows that the maximum MSE was 0.8983. This result explains the higher capability of this model in fitting the multicomponent adsorption systems. By comparison, ANN and LS-SVR with SVR, the latter shows good performance in prediction accuracy and computational speed (Fig. 6).

Conclusions
This work aimed to model a ternary adsorption system of different heavy metals on several adsorbents, using ANN, SVR, and LS-SVR models. The optimised static neural network was found with a structure of {8-6-3}, tangent sigmoid activation function for the hidden and the linear for the output layer, Levenberg-Marquardt learning algorithm.