Ordered Code-based Kernel Extreme Learning Machine for Ordinal Regression
LI Peijia①②③ SHI Yong②③④ WANG Huadong②③ NIU Lingfeng②③④
①(School of Computer and Control Engineering, University of Chinese Academy of Sciences, Beijing 101408, China) ②(Key Laboratory of Big Data Mining and Knowledge Management, Chinese Academy of Sciences, Beijing 100190, China) ③(Research Center on Fictitious Economy and Data Science, Chinese Academy of Sciences, Beijing 100190, China) ④(School of Economics and Management, University of Chinese Academy of Sciences, Beijing 100190, China)
Abstract：Ordinal regression is one of the supervised learning issues, which resides between classification and regression in machine learning fields. There exist many real problems in practice, which can be modeled as ordinal regression problems due to the ordering information between labels. Therefore ordinal regression has received increasing interest by many researchers recently. The Extreme Learning Machine (ELM)-based algorithms are easy to train without iterative algorithm and they can avoid the local optimal solution; meanwhile they reduce the training time compared with other learning algorithms. However, the ELM-based algorithms which are applied to ordinal regression have not been exploited much. This paper proposes a new ordered code-based kernel extreme learning ordinal regression machine to fill this gap, which combines the kernel ELM and error correcting output codes effectively. The model overcomes the problems of how to get high quality feature mappings in ordinal regression and how to avoid setting the number of hidden nodes by manual. To validate the effectiveness of this model, numerous experiments are conducted on a lot of datasets. The experimental results show that the model can improve the accuracy by 10.8% on average compared with traditional ELM-based algorithms and achieve the state- of-the-art performance with the least time.
NAKOV P, RITTER A, ROSENTHAL S, et al. SemEval- 2016 task 4: Sentiment analysis in Twitter[C]. International Workshop on Semantic Evaluation, San Diego, USA, 2016: 1-18. doi: 10.18653/v1/S16-1028.
[2]
TIAN Q, CHEN S, and TAN X. Comparative study among three strategies of incorporating spatial structures to ordinal image regression[J]. Neurocomputing, 2014, 136: 152-161. doi: 10.1016/j.neucom.2014.01.017.
[3]
CORRENTE S, DOUMPOS M, GRECO S, et al. Multiple criteria hierarchy process for sorting problems based on ordinal regression with additive value functions[J]. Annals of Operations Research, 2017, 251(1/2): 117-139. doi: 10.1007/ s10479-015-1898-1.
[4]
GUTIÉRREZ P A, PÉREZ-ORTIZ M, SANCHEZ- MONEDERO J, et al. Ordinal regression methods: Survey and experimental study[J]. IEEE Transactions on Knowledge and Data Engineering, 2016, 28(1): 127-146. doi: 10.1109/ TKDE.2015.2457911.
[5]
HUANG G B, ZHU Q Y, and SIEW C K. Extreme learning machine: Theory and applications[J]. Neurocomputing, 2006, 70(1): 489-501. doi: 10.1016/j.neucom.2005.12.126.
[6]
RAJASEKARAN S and PAI G A V. Neural Networks, Fuzzy Systems and Evolutionary Algorithms: Synthesis and Applications[M]. Haryana, India: Rajkamal Electric Press, 2017: 151-168.
[7]
CHU W and KEERTHI S S. Support vector ordinal regression[J]. Neural Computation, 2007, 19(3): 792-815. doi: 10.1162/neco.2007.19.3.792.
[8]
HUANG G B, ZHOU H, DING X, et al. Extreme learning machine for regression and multiclass classification[J]. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 2012, 42(2): 513-529. doi: 10.1109/TSMCB. 2011.2168604.
[9]
UCAR A, DEMIR Y, and GÜZELI C. A new facial expression recognition based on curvelet transform and online sequential extreme learning machine initialized with spherical clustering[J]. Neural Computing and Applications, 2016, 27(1): 131-142. doi: 10.1007/s00521-014-1569-1.
XU Tao, GUO Wei, and LÜ Zonglei. Prediction model of airport noise based on fast extreme learning machine and differential evolution[J]. Journal of Electronics & Information Technology, 2016, 38(6): 1512-1518. doi: 10.11999/JEIT 150986.
[11]
GOODFELLOW I, BENGIO Y, and COURVILLE A. Deep Learning[M]. Massachusetts, USA, MIT Press, 2016: 165-480. doi: 10.1038/nature14539.
[12]
DENG W Y, ZHENG Q H, LIAN S, et al. Ordinal extreme learning machine[J]. Neurocomputing, 2010, 74(1): 447-456. doi: 10.1016/j.neucom.2010.08.022.
[13]
RICCARDI A, FERNÁNDEZ-NAVARRO F, and CARLONI S. Cost-sensitive AdaBoost algorithm for ordinal regression based on extreme learning machine[J]. IEEE Transactions on Cybernetics, 2014, 44(10): 1898-1909. doi: 10.1109/TCYB. 2014.2299291.
[14]
HORNIK K, STINCHCOMBE M, and WHITE H. Multilayer feedforward networks are universal approximators[J]. Neural Networks, 1989, 2(5): 359-366. doi: 10.1016/0893-6080(89) 90020-8.
[15]
HUANG G B and BABRI H A. Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions[J]. IEEE Transactions on Neural Networks, 1998, 9(1): 224-229. doi: 10.1109/72.655045.
[16]
HUANG G B, CHEN L, and SIEW C K. Universal approximation using incremental constructive feedforward networks with random hidden nodes[J]. IEEE Transactions on Neural Networks, 2006, 17(4): 879-892. doi: 10.1109/TNN. 2006.875977.
[17]
HUANG G B. Learning capability and storage capacity of two-hidden-layer feedforward networks[J]. IEEE Transactions on Neural Networks, 2003, 14(2): 274-281. doi: 10.1109/TNN.2003.809401.
[18]
BARTLETT P L. The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network[J]. IEEE Transactions on Information Theory, 1998, 44(2): 525-536. doi: 10.1109/18.661502.
[19]
TANG J, DENG C, and HUANG G B. Extreme learning machine for multilayer perceptron[J]. IEEE Transactions on Neural Networks and Learning Systems, 2016, 27(4): 809-821. doi: 10.1109/TNNLS.2015.2424995.
[20]
HOERL A E and KENNARD R W. Ridge regression: Biased estimation for nonorthogonal problems[J]. Technometrics, 1970, 12(1): 55-67. doi: 10.1080/00401706.1970.10488634.
[21]
ALLWEIN E L, SCHAPIRE R E, and SINGER Y. Reducing multiclass to binary: A unifying approach for margin classifiers[J]. Journal of Machine Learning Research, 2000, 1(12): 113-141. doi: 10.1162/15324430152733133.
LEI Lei, WANG Xiaodan, LUO Xi, et al. An overview of multi-classification based on error-correcting output codes[J]. Acta Electronica Sinica, 2014, 42(9): 1794-1800. doi: 10.3969 /j.issn.0372-2112.2014.09.020.
[23]
HUANG G, HUANG G B, SONG S, et al. Trends in extreme learning machines: A review[J]. Neural Networks, 2015, 61: 32-48. doi: 10.1016/j.neunet.2014.10.001.
[24]
LIU Q, HE Q, and SHI Z. Extreme support vector machine classifier[C]. 12th Pacific-Asia Conference on Knowledge Discovery and Data Mining, Osaka, Japan, 2008: 222-233. doi: 10.1007/978-3-540-68125-0_21.
[25]
FRÉNAY B and VERLEYSEN M. Using SVMs with randomised feature spaces: an extreme learning approach[C]. European Symposium on Artificial Neural Networks (ESANN), Bruges, Belgium, 2010: 315-320.
[26]
HUANG G B, DING X, and ZHOU H. Optimization method based extreme learning machine for classification[J]. Neurocomputing, 2010, 74(1): 155-163. doi: 10.1016/j.neucom. 2010.02.019.
[27]
CHU W and GHAHRAMANI Z. Gaussian processes for ordinal regression[J]. Journal of Machine Learning Research, 2005, 6(7): 1019-1041.
[28]
BACCIANELLA S, ESULI A, and SEBASTIANI F. Evaluation measures for ordinal regression[C]. The Ninth International Conference on Intelligent Systems Design and Applications, Pisa, Italy, 2009: 283-287. doi: 10.1109/ISDA. 2009.230.
