This is a repository copy of A novel logistic-NARX model as a classifier for dynamic binary classification. White Rose Research Online URL for this paper: http://eprints.whiterose.ac.uk/116081/ Version: Accepted Version Article: Ayala Solares, J.R., Wei, H. orcid.org/0000-0002-4704-7346 and Billings, S.A. (2017) A novel logistic-NARX model as a classifier for dynamic binary classification. Neural Computing & Applications. ISSN 0941-0643 https://doi.org/10.1007/s00521-017-2976-x Reuse Items deposited in White Rose Research Online are protected by copyright, with all rights reserved unless indicated otherwise. They may be downloaded and/or printed for private study, or other acts as permitted by national copyright laws. The publisher or other rights holders may allow further reproduction and re-use of the full text version. This is indicated by the licence information on the White Rose Research Online record for the item. Takedown If you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing [email protected] including the URL of the record and the reason for the withdrawal request. [email protected] https://eprints.whiterose.ac.uk/ Noname manuscript No. (will be inserted by the editor) Accepted Manuscript by Neural Computing and Applications Anovellogistic-NARXmodelasaclassifierfordynamicbinaryclassification Jose Roberto Ayala Solares · Hua-Liang Wei · Stephen A. Billings the date of receipt and acceptance should be inserted later Abstract System identification and data driven modeling techniques have seen ubiquitous applications in past decades. In particular, parametric modelling methodologies such as linear and nonlinear autoregressive with ex- ogenous inputs models (ARX and NARX) and other similar and related model types have been preferably applied to handle diverse data driven modeling problems due to their easy-to-compute linear-in-the-parameters structure, which allows the resultant models to be easily interpreted. In recent years, several variations of the NARX method- ology have been proposed that improve the performance of the original algorithm. Nevertheless, in most cases, NARX models are applied to regression problems where all output variables involve continuous or discrete-time sequences sampled from a continuous process, and little attention has been paid to classification problems where the output signal is a binary sequence. Therefore, we developed a novel classification algorithm that combines the NARX methodology with logistic regression and the proposed method is referred to as logistic-NARX model. Such a combination is advantageous since the NARX methodology helps to deal with the multicollinearity prob- lem while the logistic regression produces a model that predicts categorical outcomes. Furthermore, the NARX approach allows for the inclusion of lagged terms and interactions between them in a straight forward manner resulting in interpretable models where users can identify which input variables play an important role individu- ally and/or interactively in the classification process, something that is not achievable using other classification techniques like random forests, support vector machines andk-nearestneighbors.Theefficiencyoftheproposed method is tested with five case studies. Keywords Nonlinear system identification · Dynamic systems · Binary classification · NARX models · Logistic regression 1Introduction System identification focuses on finding models from data and use them to understand or analyze the properties or behaviours of the underlying systems [1]. Linear models have been widely used in many applications [2]. However, its applicability is limited since most of the real world problems may not be well presented using linear models [3]. Research on nonlinear system identification has been carried out and advanced since the 1980s [1]. One of the most popular methodologies is the Nonlinear AutoRegressive Moving Average with eXogenous inputs (NARMAX) methodology, which has proved to be suitable for a wide class of nonlinear systems [1, 4–8]. The NARMAX approach can detect an appropriate model structure and select the most important model terms from a dictionary consisting of a great number of candidate model terms. In recent years, several variants have been proposed that improve the performance of the original algorithm. Such variations include the use of more complex and flexible predefined functions [6, 9–13], novel dependency Jose Roberto Ayala Solares Department of Automatic Control and Systems Engineering Faculty of Engineering The University of Sheffield United Kingdom E-mail: jrayalasolares1@sheffield.ac.uk Hua-Liang Wei ( ) E-mail: w.hualiang@sheffield.ac.uk Stephen A. Billings E-mail: s.billings@sheffield.ac.uk 2 Jose Roberto Ayala Solares et al. metrics [5, 8, 14–21]ordifferentsearchmechanisms[22–26]. Nevertheless, the different versions of the NARX methodology have been designed under the assumption that thevariablesinvolvedarecontinuous. Many real-life systems involve a mixed combination of continuous and discrete variables. In this work, we focus on systems with binary responses that depend on continuous time predictors. Binary responses are commonly studied in many situations such as the presence or absence of adisease,grantingaloan,ordetectingthefailure of a process, system or product [27,28]. However, the use of traditional regression techniques to deal with systems with a dichotomous response variable may not be appropriate given that they are sensitive to outliers and the distribution of the classes [27]. In this work, we propose a novel approach that combines logistic regression with the NARX methodology. The main motivation comes from the fact that logistic regressionmodelsaremoresuitableforbinaryclassification problems given that they provide probabilities of belongingornottoaparticularclass.Oneimportantconsid- eration when constructing a logistic regression model is multicollinearity. In general, it is important to always check for high inter-correlations among the predictor variables. In the ideal scenario, the predictor variables will have a strong relationship to the dependent variable but should not be strongly related to each other [29]. This problem is adequately solved using the NARX approach, since the model terms selected are orthogonal (uncorre- lated) to each other. Furthermore, the NARX approach allows for the inclusion of lagged terms and interactions between them in a straight forward manner resulting in interpretable models, something that is not achievable using other popular classification techniques like random forests [30], support vector machines [31]andk-nearest neighbors [32]. This work is organised as follows. Section 2 includes a brief summary of nonlinear system identification and adiscussionoftheOrthogonalForwardRegressionalgorithm. In section 3 our new methodology is described. Section 4 presents three numerical case studies that show theeffectivenessofournewmethod.Insection5the logistic-NARX model is applied to two real applications. Section 6 discusses advantages and disadvantages of the technique. The work is concluded in section 7. 2NonlinearSystemIdentification System identification, as a data based modelling approach, aims to find a model from available data that can represent as close as possible the system input and output relationship [1, 2]. While conventionally linear models have been applied in many applications, its applicability islimitedasthelinearityassumptionmaybeviolated for many nonlinear system modelling problems [3]. Nonlinear system identification techniques have been ad- vanced since the 1980s [1]. In particular, the Nonlinear AutoRegressive Moving Average with eXogenous inputs (NARMAX) methodology has proved to be a powerful tool for nonlinear system identification [1, 12, 33, 34]. In general, system identification consists of several steps,includingdatacollectionandprocess- ing, selection of mathematical representation, model structure selection, parameter estimation, and model validation [2]. Data processing is an important part given that data preparation plays a key role when training a model. Generally, this consists of dealing with missing values and outliers, data normalization and transformation, dimensionality reduction, and performing feature engineering. In [32, 35], these issues are widely discussed. Regarding the selection of mathematical represen- tation, this work focuses on NARX models. Model structure detection has been tackled using different methods like clustering [36, 37], the Least Absolute Shrinkage and Selection Operator (LASSO) [38, 39], elastic nets [40,41], genetic programming [42,43], the Orthogonal Forward Regression (OFR) using the Error Reduction Ratio (ERR) approach [33], and the bagging methodology [21]. Parameter estimation has been performed using the traditional least squares method, gradient descent and the Metropolis-Hastings algorithm [44,45]. Finally, for model validation, a set of statistical correlation tests have been developed in [46]andcanbeusedtotestand verify the validity of the identified nonlinear input-outputmodels.Insummary,systemidentificationisaprocess that builds a parsimonious model that satisfies a set of accuracy and validity tests [10]. 2.1 Orthogonal Forward Regression Algorithm The NARX model is a nonlinear recursive difference equation with the following general form: y (k) =f y (k − 1) ,...,y(k − ny) , ! u (k − 1) ,...,u(k − nu) + e (k) (1) " where f (·) represents an unknown nonlinear mapping, y (k), u (k) and e (k) are the output,