View metadata, citation and similar papers at core.ac.uk brought to you by CORE
provided by DigitalCommons@University of Nebraska
University of Nebraska - Lincoln DigitalCommons@University of Nebraska - Lincoln Publications, Agencies and Staff of the .SU . U.S. Department of Commerce Department of Commerce
2012 Modeling Microalgal Abundance with Artificial Neural Networks: Demonstration of a Heuristic ‘Grey-Box’ to Deconvolve and Quantify Environmental Influences David F. Millie Palm Island Enviro-Informatics LLC., [email protected]
Gary R. Weckman Ohio University - Main Campus
William A. Young II Ohio University - Main Campus
James E. Ivey Florida Fish and Wildlife Conservation Commission
Hunter J. Carrick Central Michigan University
See next page for additional authors
Follow this and additional works at: http://digitalcommons.unl.edu/usdeptcommercepub
Millie, David F.; Weckman, Gary R.; Young, William A. II; Ivey, James E.; Carrick, Hunter J.; and Fahnenstiel, Gary L., "Modeling Microalgal Abundance with Artificial Neural Networks: Demonstration of a Heuristic ‘Grey-Box’ to Deconvolve and Quantify Environmental Influences" (2012). Publications, Agencies and Staff of ht e U.S. Department of Commerce. 501. http://digitalcommons.unl.edu/usdeptcommercepub/501
This Article is brought to you for free and open access by the U.S. Department of Commerce at DigitalCommons@University of Nebraska - Lincoln. It has been accepted for inclusion in Publications, Agencies and Staff of the .SU . Department of Commerce by an authorized administrator of DigitalCommons@University of Nebraska - Lincoln. Authors David F. Millie, Gary R. Weckman, William A. Young II, James E. Ivey, Hunter J. Carrick, and Gary L. Fahnenstiel
This article is available at DigitalCommons@University of Nebraska - Lincoln: http://digitalcommons.unl.edu/usdeptcommercepub/ 501 Environmental Modelling & Software 38 (2012) 27e39
Contents lists available at SciVerse ScienceDirect
Environmental Modelling & Software
journal homepage: www.elsevier.com/locate/envsoft
Modeling microalgal abundance with artificial neural networks: Demonstration of a heuristic ‘Grey-Box’ to deconvolve and quantify environmental influences
David F. Millie a,b,c,*, Gary R. Weckman d, William A. Young II e, James E. Ivey c, Hunter J. Carrick f, Gary L. Fahnenstiel g a Palm Island Enviro-Informatics LLC., Sarasota, FL 34232, USA b Loyola University New Orleans, Department of Biological Sciences, New Orleans, LA 70118, USA c Florida Fish & Wildlife Conservation Commission, Fish & Wildlife Research Institute, St. Petersburg, FL 33701, USA d Ohio University, Russ College of Engineering and Technology, Department of Industrial and Systems Engineering, Athens, OH 45701, USA e Ohio University, College of Business, Management Systems Department, Athens, OH 45701, USA f Central Michigan University, Institute for Great Lakes Research, Mount Pleasant, MI 48559, USA g National Oceanic and Atmospheric Administration, Great Lakes Environmental Research Laboratory, Lake Michigan Field Station, Muskegon, MI 49441, USA article info abstract
Article history: An artificial neural network (ANN)-based technology e a ‘Grey-Box’, originating the iterative selection, Received 4 February 2012 depiction, and quantitation of environmental relationships for modeling microalgal abundance, as Accepted 10 April 2012 chlorophyll (CHL) a, was developed and evaluated. Due to their robust capability for reproducing the Available online 2 June 2012 complexities underlying chaotic, non-linear systems, ANNs have become popular for the modeling of ecosystem structure and function. However, ANNs exhibit a holistic deficiency in declarative knowledge Keywords: structure (i.e. a ‘black-box’). The architecture of the Grey-Box provided the benefit of the ANN modeling Artificial intelligence structure, while deconvolving the interaction of prediction potentials among environmental variables Ecological modeling fl Environmental informatics upon CHL a. The in uences of (pairs of) predictors upon the variance and magnitude of CHL a were Output response surfaces depicted via pedagogical knowledge extraction (multi-dimensional response surfaces). This afforded Pedagogical knowledge extraction derivation of mathematical equations for iterative predictive outcomes of CHL a and together with an algorithmic expression across iterations, corrected for the lack of declarative knowledge within conventional ANNs. Importantly, the Grey-Box ‘bridged the gap’ between ‘white-box’ parametric models and black-box ANNs in terms of performance and mathematical transparency. Grey-Box formulations are relevant to ecological niche modeling, identification of biotic response(s) to stress/disturbance thresh- olds, and qualitative/quantitative derivation of biota-environmental relationships for incorporation within stand-alone mechanistic models projecting ecological structure. Ó 2012 Elsevier Ltd. All rights reserved.
“Ecologists . should be aware that neural networks are not just 1. Introduction black boxes: they can open the hood, see what is in and try some trick.” Scardi (2001) Artificial Neural Networks (ANNs) have become popular tools for modeling phytoplankton abundances and production/toxicity dynamics as a function of environmental ‘predictors’ across diverse aquatic systems (e.g. Recknagel et al., 1997; Barciela et al., 1999; Scardi and Harding,1999; Olden, 2000; Scardi, 2001; Lee et al., 2003; Millie et al., 2006a, 2006b; Teles et al., 2006; Chan et al., 2007; Jeong fi Abbreviations: AMB, ambient temperature; ANN, arti cial neural network; BP, et al., 2008). Briefly, ANNs are a core form of Artificial Intelligence barometric pressure; CHL, chlorophyll; CURDIR, current direction; CURSPD, current models that discern complex associations among variables through speed; DO, dissolved oxygen; MLR, multiple linear regression; NOX N, nitrite/ nitrateenitrogen; PAR, photosynthetic active radiation; PE, processing elements; iterative and repetitive data presentation. In essence, the correlated pH, water acidity, basicity; PRECIP, precipitation; RH, relative humidity; SAL, nonlinear patterns between ‘predictor’ and ‘response’ variables are salinity; TEMP, water temperature; TURB, turbidity; UR N, ureaenitrogen; identified, with the complex interactions reproduced and mapped. WNDDIR, wind direction; WNDSPD, wind speed. Network computations easily accommodate data of non-normal * Corresponding author. Palm Island Enviro-Informatics LLC., 4645 Stone Ridge fl Trail, Sarasota, FL 34232, USA. Tel.: þ1 941 544 7926; fax: þ1 941 378 5769. probability distributions and/or variables re ecting cyclic variation E-mail addresses: [email protected], [email protected] (D.F. Millie). (Maier et al., 1998), traits typically observed within large ‘noisy’,
1364-8152/$ e see front matter Ó 2012 Elsevier Ltd. All rights reserved. doi:10.1016/j.envsoft.2012.04.009 28 D.F. Millie et al. / Environmental Modelling & Software 38 (2012) 27e39 even chaotic environmental data sets (see Peck et al., 2003; Rohani concerning the fundamental relationships of and/or interactive et al., 2004; Murray and Conner, 2009; Wood, 2010). complexities between/among the biotic (response) and environ- ANNs typically have high-dimensional input space and do not mental (predictor) variables, MLR may result in model estimations exhibit any explicit or declarative knowledge structure. Generally, having little or no interpretive relevance (after Millie et al., 2006a). only the input-output characteristics of ANNs are of interest, with Accurate, reproducible prediction of system-level patterns and the ‘knowledge’ of variable relationships encoded almost incom- processes is a basic tenet of ecological forecasting. In order to prehensibly by synaptic weights embedded within network provide worthwhile bases for natural resource stewardship and/or architecture (Fig. 1). Because of this holistic lack of model trans- proactive mitigation of environmental disturbance/stressors, parency, many researchers consider ANNs to be ‘black-boxes’ (Lek statistical-based modeling efforts require a coupling of reliable and Guegan, 1999; Olden and Jackson, 2002) and entrust a low prediction with a conceptual interpretation of biotic structure and confidence to their utilization as empirical models of ecological function (after Millie et al., 2006a,b, 2011). Clearly, a heuristic processes. From this, ANNs might appear to be of limited value for knowledge-extraction technique that provides exact quantitative scientific theory generation, environmental problem solving and/or formulations pertaining to non-linear variable interaction and natural resource decision-making. prediction influences is highly desirable (c.f. Saito and Nakano, Aquatic scientists traditionally have relied upon multivariate 2002). Such an approach would allow for a mathematically linear regression (MLR) to model microalgal-environmental rela- comprehensive, yet pragmatic understanding of environmental- tionships and functionality (e.g. Cattaneo, 1987; Sarnelle, 1992; biota complexity and interaction, whilst (potentially) eliminating Bachmann et al., 1996, 2001; Dodds et al., 2002, 2006; Heffernan the black-box mentality for ANNs. et al., 2010). Such ‘white-box’ parametric models are far less The chlorophyll (CHL) a concentration of a water column is abstract than ANNs - the derivation of defined coefficients (based a universally accepted measurement of planktonic algal abundance on correspondence between the response and predictor variables) and used to quantify community dynamics and/or growth in affords users with a comfortable degree of model transparency. changing environments (Millie et al., 2010). Here, we present However, underlying assumptions for and/or limitations associated a network-based approach e hereafter, referred to as a ‘Grey-Box’ with MLR (e.g. requirements of a normal probability distribution (Young and Weckman, 2009; see Oussar and Dreyfus, 2001; and homoscedasticity for variables, inappropriate selection of Johannet et al., 2007), originating the iterative selection, depiction, predictors arising from efforts to reduce model error, strong auto- and quantitation of environmental variable relationships in correlation among variables, etc.) restrict its ‘across the board’ modeling water-column CHL a concentrations within a coastal application and may result in models without statistical merit. environment. The Grey-Box formulation: 1) was based upon Moreover, a prerequisite for ecological applications of linear knowledge extracted from a trained and validated ANN; regression is an a priori knowledge of appropriate predictors for 2) provided interpretable, multi-dimensional response surfaces model inclusion or exclusion. In the absence of knowledge depicting modeled environmental-CHL a relationships; and 3)
Fig. 1. Schematic of the artificial neural network (ANN) used to model chlorophyll (CHL) a (after Principe et al., 2000; Hu and Hwang, 2001). A feed-forward, multi-layer perceptron modeled the interaction among input variables (x1.n), synaptic weights (u), processing elements (PEs), and CHL concentrations. The optimal network (see section 2.3) had ten and four PEs in the hidden layers one and two, respectively. Modeled and measured concentrations were compared, from which the error was computed and ‘back-propagated’, with the weights incrementally adjusted via a conjugate gradient learning algorithm. As error minimized with repeated data presentations, weight values stabilized and modeled concentrations increasingly approximated measured concentrations. Inset figure: Schematic depicting computational formulations of a hidden layer PE. Input values (i), whether from x1.n or a net function from a PE, were multiplied by a synaptic weight (u1.n), the products summed, and combined with a bias value (q) to produce (u) that then was transformed (via a sigmoid activation function, f(u) ¼ 1/(1 þ e u) to produce the output (a). Note: the PE in the output layer utilized a linear function, f(u) ¼ au. D.F. Millie et al. / Environmental Modelling & Software 38 (2012) 27e39 29 quantified environmental influences and interactions for CHL modeled concentration was compared to the desired measured concentration, from a through the summation of the response-surface equations. which the mean-square error (MSE) was computed. After presentation of all data vectors (or ‘epoch’), the error was ‘back-propagated’ to the network and the weights incrementally adjusted, through gradient descent with momentum learning, in the 2. Methodology direction of the minimum error among PEs (Principe et al., 2000; Olden, 2000; Lee et al., 2003). With repeated presentation of epochs, the MSE gradually minimized 2.1. Sample site and data collection and modeled concentrations increasingly approximated measured concentrations. For training, 70% of all exemplars were presented repeatedly to the network fi Designated as a water body of national signi cance, Sarasota Bay (southwest (typically 1000 to 2000 times), with ‘learning’/momentum rates and step sizes Florida, USA) is a productive, lagoonal estuary and one of 24 systems nationwide allowed to vary (thereby accelerating network ‘learning’ and ensuring convergence to fi selected to be a member of the National Marine Estuary Program for scienti c research a global minimum; Barciela et al., 1999; Olden, 2000; Principe et al., 2000; Olden and and education. The Bay is an excellent site with which to model phytoplankton Jackson, 2002). Cross-validation data (15% of all exemplars) confirmed an unbiased dynamics in response to alterations in water-quality; exchange of Bay waters with estimation of prediction concurrent with training. If the generated MSE within the ‘ ’ Gulf of Mexico waters is limited (restricted to four narrow passes ) and the Bay has no training or cross-validation data subsets fell below 0.01 or began to increase (an major freshwater tributary. As a consequence, water/nutrient inputs primarily arise indication that the network began to memorize the data; Karul et al., 2000; Gurbuz from precipitation and storm-runoff events within a highly developed (residential/ et al., 2003), training was terminated. The trained network then was applied to commercial) watershed. Such hydrological conditions result in a slightly greater testing data (15% of all exemplars, not used in training and cross-validation). Data e e salinity (SAL) reaching 37 39 (via the Practical Salinity Scale) within Bay waters, subsets for training, cross-validation, and testing were selected randomly. e fi than that of near-shore Gulf waters (ca. 35 36). Following signi cant rain events, SAL A correlation coefficient denoted the agreement between modeled and measured decreases rapidly (but not to levels of Gulf waters) and coincides with increased CHL a concentrations. Linear regression was used to illustrate ‘trend lines’ for mod- concentrations of non-point source terrigeneous nutrients. These conditions, in turn, eled:measured relationships of concentrations. An analysis of covariance tested impact the phytoplankton assemblage, potentially leading to altered abundances, whether slopes of regression estimations differed from the slope of a corresponding1:1 compositional structure and holistic dominance (Ivey, unpubl. data; see Millie et al., modeled:measured relationship (SigmaPlot v12 software; Systat, Inc., Chicago, IL USA). 2004; Gilbert et al., 2006). Because of the effects of non-point source contaminants upon system-level functionality, the Bay remains listed (since 1998) by the U. S. 2.3. Network selection Environmental Protection Agency as a ‘303d impaired’ water body (FDEP, 2006). From May to October in 2009, meteorological and hydrological data were 0 00 0 00 In order to identify the most optimal network for the Grey-Box model, an iter- acquired at a single site (27 21 32 N, 82 36 14 W) in the Bay via an autonomous ative strategy was implemented and within which diverse network architectures e instrument platform and a bottom-mounted Acoustic Doppler Current Profiler inclusive of linear, sigmoid, and hyperbolic transfer functions for PEs within hidden/ (Paerl et al., 2005; Millie et al., 2006a). Data for select variables (Table 1) were output layers, and diverse learning algorithms e inclusive of conjugant gradient, recorded hourly from sub-surface waters (ca. 1 m depth). Diel means of all variables Levenberg-Marquardt, and momentum learning gradients, were evaluated. Initially, were calculated and utilized for model development. networks were developed and trained with the number of PEs in the first hidden layer varying by two, between two and 50, and the PEs in the second hidden layer fi 2.2. Arti cial neural networks arbitrary held to 10. The selected training algorithm was evoked using the lower bound of PEs in the first hidden layer. Twenty-five networks were trained, prior to Concentrations of CHL a were modeled from environmental variables using ANNs incrementally increasing (by two, up to the maximum) the number of PEs in the first fi incorporating supervised training. Speci cally, feed-forward multi-layer perceptrons hidden layer. Resulting networks were evaluated with the network producing the ‘ ’ (see Principe et al., 2000; Hu and Hwang, 2001) utilizing a back-propagation learning least MSE within the cross-validation data set (and having optimal architecture and algorithmwere constructed via NeuroSolutions v6.0 software (NeuroDimension, Inc.; most efficient transfer functions/learning algorithm) chosen for further develop- Gainesville, Florida USA). Predictor and modeled variables were normalized to match ment. Using this network, the process was repeated in toto, with the number of PEs ’ the range of the non-linear transfer functions in the network s hidden and output in the first hidden layer held to the optimal number determined in the initial layers, respectively (Fig. 1; see Goh, 1995; Olden and Jackson, 2002). training phase, the number of PEs in the second hidden layer varied by two, between fl ‘ ’ Brie y, predictor values in each data vector (or exemplar ) were multiplied by two and 50. The final network architecture producing the least MSE and having the scalar weights prior to their summation and processing via transfer functions in most efficient transfer functions/learning algorithm then was selected for the Grey hidden-layer processing elements (PEs). Values generated for hidden-layer PEs Box formulation (Fig. 1). similarly were multiplied by weights and summed, prior to their processing via the transfer function in the output layer to yield a modeled CHL concentration (Fig.1). The 2.4. Regression modeling and comparison to the ANN/Grey-Box
To compare results of the Grey-Box with that of a parametric model, a step-wise Table 1 (forward) MLR model was constructed for the entire data set, as: Meteorological and sub-surface hydrological variables collected within Sarasota Bay. ½ ¼b þ b þ b þ b /b þ ε CHL a 0 1X1 2X2 3X3 iXi (1) Variable Abbreviation (units) Mean SE Range b Ambient AMB ( C) 27.15 0.18 15.94e29.48 where X1.i are the predictor variables, 0.i are regression parameters (intercept/ ε temperature slopes of the regression estimation), and is the error (SigmaPlot v12 software). fi Wind speed WNDSPD (m s 1) 4.15 0.10 2.14e8.56 Modeling veri cation statistics (root mean square error, RMSE; reliability index, fi Wind direction WNDDIR 194.19 5.34 41.90e324.76 RI; average error, AE; average absolute error, AAE; modeling ef ciency, ME) were (compass degrees) computed to afford direct comparison of the MLR model with the holistic ANN, and fl Precipitation PRECIP (mm) 2.6 0.60 0.00e39.60 the Grey-Box model (refer to Reckhow et al., 1990; Stow et al., 2003). Brie y, the Barometric pressure BP (Hg) 101.53 0.02 100.70e102.09 RMSE, AE, and AAE statistics denoted the generalized accuracy of performance (by Relative humidity RH (%) 75.29 0.46 53.14e88.60 quantifying differences between modeled:measured CHL concentrations), with Photosynthetic PAR (mEm 2 s 1). 10.24 0.21 1.89e14.28 exact agreement between modeled:measured concentrations producing a value of fi active zero. The RI statistic signi ed the mean factor by which modeled CHL concentrations radiation ( 103) corresponded to measured concentrations (i.e. when modeled concentrations Current CURSPD (m s 1) 3.30 0.07 2.05e5.25 exactly agreed with measured concentrations, a value of one resulted). The ME speed ( 10 2) statistic symbolized model prediction relative to the mean of measured CHL fi Current direction CURDIR 232.42 5.80 90.44e342.02 concentrations; a value of one signi ed exact similarity between modeled:measured fi (compass degrees) concentrations whereas a value equal to or less than zero signi ed that the Water temperature TEMP ( C) 28.90 0.22 20.94e31.94 measured mean concentration would be no better or a better predictor that modeled Turbidity TURB (NTU) 1.70 0.06 0.35e3.90 value, respectively (from Stow et al., 2003). Salinity SAL (via the 36.52 0.08 35.04e38.77 Practical 3. Grey-Box derivation and results Salinity Scale) þ Water pH ( log [H ]) 8.13 0.01 7.82e8.37 acidity/basicity Derivation of the Grey-Box utilized the most optimal ANN Dissolved oxygen DO (mg L 1) 6.30 0.06 4.91e8.20 (section 2.3) and incorporated a step-wise (bottom-up) approach, Urea UR-N (mM N) 3.064 0.23 0.27e13.85 with extraction of predictor-response information in the form of m 1 e Chlorophyll a CHL a ( gL ) 10.66 0.62 1.49 35.43 iterative, output surfaces and mathematical equations (after Young 30 D.F. Millie et al. / Environmental Modelling & Software 38 (2012) 27e39 and Weckman, 2009). Each predictor variable was permitted an 3.1. Iterative response surfaces opportunity to explain the variation of the response variable and presumed to explain an equal (and quantifiable) fraction of CHL a. 3.1.1. Initial iteration Response surfaces for successive iterations accounted for progres- The initial output surface, f (x1, y1), was postulated to represent sively lesser amounts of model prediction, as: the majority of variation within CHL a. Originally, ANNs incorpo- rating 15 candidate predictor variables were developed, trained, and tested. The slope of the modeled:measured regression for the ½CHL a ¼w $ f ðx ; y Þþr ; r ¼ w $f ðx ; y Þþr ; 1 1 1 1 1 2 2 2 2 (2) ‘most optimal’ network (Fig. 2A) did not differ from a correspond- r ¼ w $f ðx ; y Þþr ; and r ¼ wn$f ðxn; ynÞþrn 2 3 3 3 3 n 1 ing 1:1 relationship. Following this, a sensitivity analysis (S1.n) provided an approximate measure of the relative importance for fi where w1..n, f(x1...n, y1...n), and r1..n were scaling factors, output predictor variables upon CHL a concentrations (Fig. 2B). Speci cally, surfaces, and iterative remainders of CHL a, respectively (section each predictor was altered (plus/minus two standard deviations 3.1). In this manner, model derivation was apportioned into ‘n’ about the mean, 50 steps-per-side), while other predictors iterations, each utilizing two predictor variables to explain remained fixed at their respective means (Principe et al., 2000). In a portion of CHL a (in the form of output response planes). Output this manner, approximately 95% of the data range for each predictor surfaces then were summed as: variable was incorporated into the analysis and reduced the like- lihood that extreme data outliers would bias the outcome (Jeong ½ ¼½ þ½ þ et al., 2003). CHLa Grey Box CHLa 1st iteration CHLa 2nd iteration. (3) Pertinent network information (i.e. input-hidden-output layer ½ þ CHLa nth iteration rn architecture, weights, biases, transfer-threshold functions) was
Fig. 2. A) Modeled CHL a concentrations as a function of measured concentrations for the ‘holistic’ artificial neural network (ANN) (i.e. iteration one; see section 3.1.1). The dashed line represents a ‘desired’ 1:1 relationship. The solid line and corresponding statistical information represents a ‘best fit’ for the modeled:measured relationship, as derived from linear regression. B) Results of a sensitivity analysis across one SD variation performed on the training data. Black-filled bars indicate variables selected for development of the subsequent iterative response surface. C) Response surface (generated via the ANN) for CHL as a function of TEMP and SAL, varied across their data ranges. Values for other predictors held to their respective sample means. D) Response surface (generated via a ‘Power B’ equation) for CHL as a function of TEMP and SAL, varied across their data ranges. Corresponding statistical information denotes similarity to the ANN-derived response plane as a function of x, y variable pairs. D.F. Millie et al. / Environmental Modelling & Software 38 (2012) 27e39 31
Fig. 3. A) Modeled concentrations as a function of measured concentrations for the artificial neural network (ANN) modeling r1 CHL a (i.e. iteration two; see section 3.1.2). B) Results of a sensitivity analysis across one SD variation performed on the training data. Black-filled bars indicate variables selected for development of the subsequent iterative response surface. C) Response surface (generated via the ANN) for r1 CHL as a function of pH and TURB, varied across their data ranges. Values for other predictors held to their respective sample means. D) Response surface generated via a ‘Rational C’ equation for r1 CHL as a function of pH and TURB, varied across their data ranges. Lines and statistical information as in Fig. 2. incorporated into Excel ver. 2010 spreadsheet software (Microsoft ½ ¼ : þ : þ 6:59* 17:01 CHL a1st iteration 1 16 5 0exp 17 TEMP SAL Corp.; Seattle, WA USA). Based upon results of the initial sensitivity analysis, water temperature (TEMP) and SAL were the two predic- (4) tors having the greatest impact upon CHL a and chosen to create f (x1, A scaling factor (wi.n) for the response plane then was calcu- fi y1). Within the ANN, TEMP and SAL were varied between their lated. Speci cally, factors were determined as the relative respective minimum- maximum data ranges (with values for other summation for the two variables having the greatest impact upon predictors held to their sample means e a procedure similar to that the modeled variable (determined via sensitivity scores), as: of a sensitivity analysis) and from which, a three-dimensional ! response surface for CHL a was generated (Fig. 2C). ð Þþ ð Þ nX 1 ¼ P Sm xm PSm ym $ ; Next, a trial-and-error procedure to fit a planar equation to the wm n n 1 wi or ¼ S ðx Þþ ¼ S ðy Þ modeled response surface was undertaken. Fifteen distinct math- i m i i i m i i i ¼ 1 (5) ð Þþ ð Þ ematical expressions, inclusive of power, polynomial, rational, ¼ S1 TEMP S1 SAL $ð Þ¼ : w1 ð Þþ ð Þþ/ ð Þ 1 0 0 41 sigmoid, and Taylor series equations, were examined. The response S1 TEMP S1 SAL Si yi surface generated by a ‘Power B’ equation was equivalent to the modeled response (p > 0.05; Fig. 2D), as determined by a Wilcoxon Importantly, the generalized form for the scaling expression Signed Rank Test assessing CHL concentrations as a function of x,y required that calculated factors sum to one. Because w1 represents variable pairs. (Note: concentrations of all network- and equation- the scaling factor for the initial response surface, there were no derived surfaces were neither normally distributed nor displayed previous factors to consider; as such, its calculation included equal variances, thereby necessitating non-parametric compari- a multiplicand of one (i.e. 1e0). From Eq. (2), the mathematical sons). A Generalized Reduced Gradient nonlinear programming expression for the first iteration became: algorithm (GRG2; Lasdon et al., 1978) optimized the constant (a) and variable factors (b, c, d) for the ‘Power B’ equation, as: ¼ : $ : þ : 6:59$ 17:01 CHL a1st Iteration 0 41 1 16 5 0 exp17 TEMP SAL ½ ¼ þ c * d ; CHL a1st iteration a b x1 y1 or (6) 32 D.F. Millie et al. / Environmental Modelling & Software 38 (2012) 27e39
3.1.2. Subsequent iterations S2ðpHÞþS2ðTURBÞ A second iteration was undertaken to deconvolve (in part) the w2 ¼ $ð1 w1Þ (8) S2ðpHÞþS2ðTURBÞþ /S2ðxnÞþS2ðynÞ remaining ‘unexplained’ variation (r1) between CHL a concentra- tions and the scaled response surface of the initial iteration The mathematical expression for the second iteration then became: (Eq. (2)). It first was necessary to correct the response variable in each data vector for the amount of variation accounted for by the ð : : * þ : * Þ initial response surface (from Eq. (5)), as: ½ ¼ : $ 0 72 0 01 pH 0 115 TURB CHL a2nd Iteration 0 35 ð : * þ * Þ 1 1 08 log10pH e log10TURB ¼½ ð ð ; Þ$ Þ r1 CHL a1/n f x1 y1 w1 (7) (9) The ANN training/testing strategy (Section 2.2) was repeated, Succeeding iterations continued to deconvolve the remaining incorporating 13 predictor variables to model r (Fig. 3A; note: the 1 variation in CHL a left unexplained by the response surfaces of variables, TEMP and SAL, utilized for the response surface of the previous iterations (see Eq. (2)). Prior to modeling, respective first iteration were eliminated from consideration). The slope of the iterative remainders for use in the third and fourth iterations (r modeled:measured regression for the network did not differ from 2 and r , respectively) were corrected for the amount of variation a corresponding 1:1 relationship. A sensitivity analysis for the 3 explained by the response surface(s) within the preceding itera- second trained network (S ) identified water acidity/basicity (pH) 2 tion(s), as: and turbidity (TURB) as the variables having the greatest impact upon modeled r (Fig. 3B). These two variables then were used to 1 r2 ¼ r1 ðf ðx2; y2Þ$w2Þ; and r3 ¼ r2 ðf ðx3; y3Þ$w3Þ (10) create a second modeled response surface, f (x2, y2)(Fig. 3C). The response surface produced from a ‘Rational C’ equation (Fig. 3D) Network training/testing (Fig. 4A and B) for the third and fourth was similar (p > 0.05) to the modeled surface. The GRG2 algorithm iterations used 11 and nine predictor variables, respectively, after determined the appropriate equation constants/factors and the successive removal of pairs of predictors used in previous itera- scaling factor for the response surface (w2) was calculated (taking tions. For ANNs modeling r2 and r3, the slope of the mod- into account the aforementioned requirement that the derived eled:measured regression differed from a corresponding 1:1 scaling factors for iterations sum to one), as: relationship. Sensitivity analyses identified photosynthetic active
Fig. 4. A & B) Modeled concentrations as a function of measured concentrations for the artificial neural network (ANN) modeling (A) r2 CHL a and (B) r3 CHL a (i.e. iterations three and four; see section 3.1.2). Lines and statistical information as in previous figures. C & D) ANN-derived response surfaces for (C) r2 CHL a as a function of PAR and CURSPD, varied across their data ranges and (D) r2 CHL a (generated via the ANN) as a function of WNDSPD and CURDIR, varied across their data ranges. Accompanying sensitivity analyses and equation-derived response planes not shown (for details, see section 3.1.2). D.F. Millie et al. / Environmental Modelling & Software 38 (2012) 27e39 33 radiation (PAR) and current speed (CURSPD), as (x3, y3), and wind plotted cumulatively as a function of measured concentrations. speed (WNDSPD) and current direction (CURDIR), as (x4, y4), to Specifically, the plot (Fig. 5A) for the first iteration was constructed have the greatest impact upon modeled r2 and r3, respectively (data from Eq. (6), incorporating the predictor variables, TEMP and SAL. not shown). These variables then were used to create the response The plot following the second iteration (Fig. 5B) arose from the surfaces, f (x3, y3) and f (x4, y4)(Fig. 4C and D, respectively). cumulative contribution of Eqs. (6) and (9), incorporating the Response surfaces created from ‘Rational B’ equations were iden- variables, pH and TURB. Subsequent plots were derived incorpo- tical (p > 0.05) to modeled surfaces for both the third and fourth rating the cumulative contributions of iteration three with vari- iterations. Scaling factors then were calculated, again taking into ables, CURSPD and PAR, and iteration four with variables, CURDIR account the requirement that derived factors for all iterations sum and WNDSPD (Eq. (11); Fig. 5C and D). to one. Mathematical expressions for the third and fourth iterations The final cumulative plot then was ‘rotated’ and ‘offset’, became: whereby the summation of the scaled estimates was ‘centered’ via
ð70:89 þ 12:69*log CURSPD 12:37*log PARÞ ½CHL a ¼0:12 10 10 ; and 3rd Iteration ð1 13:43*CURSPD þ 0:01*PARÞ ð8:01 1:98*log CURDIR 2:69*log WNDSPDÞ ½Chl a ¼0:06$ 10 10 (11) 4th Iteration ð1 0:01*CURDIR þ 0:03*WNDSPDÞ
After four iterations, cumulative improvements (i.e. decreases) the GRG2 algorithm onto measured CHL concentrations within the in error metrics for networks of training/cross-validation datasets training and cross-validation datasets (Fig. 6): had lessened (Table 2). Importantly, the value of the scaling factor ¼ þ ð½ þ½ / for the fourth iteration was small (0.06), having been sequentially CHL aGrey Box a b CHL aiteration 1 CHL aiteration 2 reduced from 0.41 to 0.35, to 0.12 in the first, second, and third þ½ Þ CHL aiteration n iterations, respectively). Also, determination coefficients for mod- (12) eled:measured comparisons of cumulative iterative networks had maximized (to ca. 0.75). As such, additional iterations were deemed where a and b represented the equation constant (y-intercept) and unnecessary. slope, respectively. Successive iterations of the algorithm sought to minimize the sum of squares of the differences between plotted and measured values. Initially, the algorithm utilized values of zero 3.2. Grey-Box rotation and offset and one for a and b, respectively, and quadratic extrapolation, after which a conjugate search method and a central differencing tech- To depict the amount of CHL accounted for by successive itera- nique calculated the derivative for each iteration. The final, rotated tions, response surface equations for modeled concentrations were and offset Grey-Box became:
0 1 6:592 17:01 B 0:41*1:16 þ 5:0E17* TEMP *SAL C B C B C B ð0:72 0:01*pH þ 0:115*TURBÞ C B þ 0:35* C B ð : * þ * Þ C B 1 1 08 log10pH e log10TURB C ¼ : þ : B C CHL aGrey Box 13 3 2 25B ð : þ : * : * Þ C (13) B 70 89 12 69 log10CURSPD 12 37 log10PAR C B þ 0:12* C B ð1 13:43*CURSPD þ 0:01*PARÞ C B C B C @ ð8:01 1:98*log CURDIR 2:69*log WNDSPDÞ A þ 0:06* 10 10 ð1 0:01*CURDIR þ 0:03*WNDSPDÞ
Table 2 Error metrics (SSE, sum of squares; MSE, Mean square error; MAE, mean absolute error) and determination coefficients (r2), derived via regression of modeled against measured chlorophyll concentrations for the holistic artificial neural network (ANN) and the Grey-Box model. Iterative data represent cumulative contributions of iterative surface equations (using appropriate predictor variables) in successive iterations (see sections 3.1.1 and 3.1.2).
Data set Metric Iteration 1 Iteration 2 Iteration 3 Iteration 4 Grey Box ANN Training and cross validation SSE 13092.62 6807.35 5874.57 5573.30 1736.40 679.37 MSE 108.20 56.26 48.55 46.06 14.35 5.62 MAE 7.86 5.27 4.83 4.67 2.93 1.76 RSQ 0.35 0.74 0.74 0.75 0.75 0.90
Test SSE 1739.17 848.33 712.32 672.62 178.07 98.91 MSE 82.82 40.40 33.92 32.03 8.48 4.71 MAE 6.99 4.53 3.91 3.71 2.25 1.70 RSQ 0.25 0.79 0.79 0.79 0.77 0.88 34 D.F. Millie et al. / Environmental Modelling & Software 38 (2012) 27e39 B 40 A 1:1 1:1 Iteration 1 Iterations 1 & 2 30 ) -1
20 (mg L a 10
0
40 C D 1:1 1:1 Iterations 1, 2 & 3 Iterationsns 1, 2, 3 & 4 30
20
10 Cumulative Iterative Chlorophyll Cumulative Iterative Chlorophyll
0 010203040010203040 Measured Chlorophyll a (mg L-1)
Fig. 5. Cumulative concentrations of CHL a accounted for by planar surface equations (using appropriate predictor variables) in successive iterations as a function of measured concentrations (see sections 3.1.1 and 3.1.2). A) ‘Power B’ equation, using the predictors, SAL/TEMP (Eq. (12)); B) ‘Power B’ and ‘Rational C’ equations using the predictors, SAL/TEMP and pH/TURB, respectively; C) ‘Power B’, ‘Rational C’, and ‘Rational B’ equations using the predictors, SAL/TEMP and pH/TURB, and PAR/CURSPD, respectively; D) ‘Power B’, ‘Rational C’, ‘Rational B’ and ‘Rational B’ equations using the predictors, SAL/TEMP, pH/TURB, PAR/CURSPD, and WNDSPD/CURDIR, respectively. The dashed line represents a ‘desired’ 1:1 relationship. The ‘pie-shaped wedge’ represents the amount of variation in CHL a left ‘unexplained’ after four iterations due to the use of only eight (of the potential 15) predictor variables and that attributable to autocorrelation among all predictors. )
-1 3.3. MLR and comparison to the ANN and Grey-Box 40 1:1 The forward-stepwise MLR model selected a linear combination
(mg L 2 ¼
a of six variables to estimate CHL a concentrations (r 0.65, 30 adj n ¼ 142, p 0.001) as:
½CHL a ¼44:408 þð3:44*TURBÞ ð1:93*SALÞþð1:11*TEMPÞ 20 þð377:86*CurSpdÞ ð1:72*WndSpdÞ ð0:001*PARÞ 10 (14) Interestingly, the six predictor variables selected for the step- 0 wise MLR also were utilized (of eight variables total) within the Rotated Grey-Box CHL CHL Rotated Grey-Box 0 10 20 30 40 derivation of the Grey Box model. Although the errors of the Measured CHL a (mg L-1) predictor coefficients were normally distributed around the regression estimate (as determined by a ShapiroeWilks test, Fig. 6. Cumulative concentrations of CHL a accounted for by planar surface equations p ¼ 0.96) and modeled residuals were independent (as determined following four successive iterations (Fig. 5D), and ‘rotation’ and ‘offset’ as a function of by the DurbineWatson statistic > 2), the dependent variable (CHL measured concentrations. For ‘rotation’, the summation of the scaled estimates was ‘centered’ via the GRG2 algorithm onto measured concentrations (see section 3.2). The a) displayed a non-normal distribution and heteroscedasticity dashed line represents a ‘desired’ 1:1 relationship. (p ¼ <0.001, as determined by a Spearman rank correlation test D.F. Millie et al. / Environmental Modelling & Software 38 (2012) 27e39 35 between residual and measured concentrations). A scatter plot of south Florida extends from May to October (the time period model residuals as a function of predicted CHL a concentrations during which water quality data was collected in Sarasota Bay) and (data not shown) depicted asymmetry around the ‘zero line’, is typified by heightened PRECIP, resulting in the introduction of thereby signifying a condition of nonlinearity. nutrient-enriched, freshwater inflows from the adjacent water- Non-linear transformations (e.g. square/fourth-root, loga- sheds into brackish coastal waters. Reflecting the increased meta- rithmic) were applied to the data distributions in attempts to bolic activity and resource utilization by phytoplankton to greater stabilize the variance. Although logarithmic transformation of the TEMPs and nutrient enrichment, CHL a concentrations in Florida’s dependent variable provided for homoscedasticity of CHL estuarine waters often are greatest during warm, summer months a (p ¼ 0.84), MLR models incorporating transformed variables (e.g. Millie et al., 2004). For such a seasonal relationship in Sarasota performed (only) as equally or underperformed the regression Bay, a mathematical expression that suitably accounted for the model utilizing non-transformed variables (r2 z 0.60, p 0.001; magnitude and dynamics in CHL a as a function of SAL and TEMP models not shown), whilst failing the test for normality of predictor was formulated. Subsequent iterations selected predictor variables coefficients about the regression (p < 0.001). known to influence (or be influenced by) phytoplankton biomass/ Based upon model verification statistics (Table 3), the holistic production (PAR, pH, TURB) and variables (WNDSPD, WNDDIR, ANN and the Grey-Box clearly outperformed the MLR. Values for CURDIR) denoting physical forcing factors that impact (localized) RMSE, RI and AAE were greatest for the parametric model, with near-surface biomass accumulations (Paerl, 1988; Paerl et al., 1998; corresponding values for the Grey-Box between that of the MLR Millie et al., 2009, 2010). From the CHL response surfaces incor- model and the holistic ANN. Conversely, the greatest and least ME porating these (pairs of) predictor variables, mathematical values occurred for the ANN and the MLR model, respectively, expressions also were derived. denoting these models to have the greatest and least efficiency in An initial premise for Grey-Box derivation was that environ- modeling (i.e. agreement between modeled:measured concentra- mental predictors were permitted equal opportunities to explain tions). Although AE similarly is a measure of prediction accuracy, the variation in CHL concentrations. The number of predictor values for all models approximated zero. Stow et al. (2003) stated variables available for network training/validation decreased (in that such AE values “.can be misleading because negative and pairs) with successive iterations. Accordingly, scaling factors for positive discrepancies can cancel each other.”, indicating that response surfaces were derived (via sensitivity analyses) to account RMSE and AAE are more interpretable statistics because they for the lesser amount of CHL a attributable to predictors in subse- “.accommodate the shortcoming of the average error by consid- quent iterations. Intuitively, absolute values of wi.n would decrease ering the magnitude rather than the direction of each discrepancy.” as the number of response surfaces increased. The lesser amount of model prediction accounted for by progressive iterations was ‘ fi ’ 4. Discussion illustrated by the increasing lack of t (i.e. increased scatter) between modeled:measured CHL concentrations and the depar- Considered to be universal approximators capable of learning tures from a 1:1 modeled:measured relationship within the third any deterministic function (Horkik et al., 1989), the multilayer feed- and fourth iterations (refer to Fig. 4). The iterative adjustment of the forward networks utilized for the Grey-Box model provided CHL concentrations accompanying the decreasing number of a systematic ‘bottom-up’ estimation of microalgal abundance from predictors continually afforded the most sensitive predictors to fl select environmental predictors. With this approach, the iterative have the greatest predictive in uence and thereby, progressively complexity of prediction potentials among candidate predictors improved upon model prediction. If a scaling factor had not been fi was deconvolved and by means of pedagogical knowledge extrac- utilized, the surface equation generated for the rst iteration would ’ fi ’ tion (i.e. the output surfaces), the step-wise effects of (pairs of) have over t the data, and not allowed predictor variables in ‘ ’ predictors upon the intrinsic variance and magnitude of CHL subsequent iterations to holistically account for the reddened (i.e. ‘ ’ a dynamics were depicted. Importantly, this approach afforded acute/chronic variability) and white (natural variability) noises in comprehensible quantitative translations (via discrete mathemat- CHL a data left unexplained as iterative remainders (see Rohani ical equations) for iterative predictive outcomes, as well as a set of et al., 2004). In founding the scaling factor upon sensitivity anal- algorithmic rules across iterations. ysis, it also must remembered that it is possible for only one (or fi The Grey-Box initiated with empiricism, yet moved towards a few) predictors to signi cantly impact the response variable (i.e. a semi-analytical formulation. The sensitivity analysis for the a sensitivity an order of magnitude greater than other candidate holistic ANN (utilizing all candidate predictors) denoted SAL and predictors). In such instances, the derived scaling factor(s) from the TEMP to be the variables having the greatest influence upon CHL a. opening iterations would provide for the remaining predictors (i.e. fi The subsequent response surface depicted a curvilinear relation- surfaces) to explain an insigni cant fraction of the response vari- ship between CHL a concentrations and these ‘most sensitive’ able. In contrast, if all variables were holistically identical in terms predictors, with the greatest concentrations occurring at the lower of their impact upon CHL concentrations (i.e. a relative equal and upper data ranges for SAL and TEMP, respectively. Such rela- sensitivity for predictors), nearly equivalent-scaling factors would tionships were intuitive; the ‘wet’ season in tropical-temperate result, with the response variable explained equally well by successive output surfaces. Not surprisingly, a diminishing return in terms of prediction Table 3 occurred with the reduction of variable pairs from the pool of Model verification statistics e root mean square error (RMSE), reliability index (RI), potential predictors in progressive iterations. To illustrate this, CHL fi average error (AE), average absolute error (AAE), and modeling ef ciency (ME) for a concentrations for the training and cross-validations data sets direct comparison of parametric multiple linear regression (MLR), the holistic arti- ficial neural network (ANN), and the Grey-Box model (refer to sections 2.4 and 3.3). were sorted in order of ascending values and plotted (cumulatively for successive iterations) as a function of modeled values (Fig. 7). Model type Statistic The initial scaled surface incorporating the predictors, SAL and RMSE RI AE AAE ME TEMP, contributed the majority (albeit somewhat a dynamic MLR 4.23 1.90 0.02 3.34 0.67 proportion) of relative prediction throughout the range of the ANN 2.34 1.34 0.05 1.74 0.90 modeled response. Relative estimations arising from the second Grey-Box 3.66 1.50 0.00 2.83 0.75 (iterative) scaled surface increased disproportionately with 36 D.F. Millie et al. / Environmental Modelling & Software 38 (2012) 27e39
The predictive accuracy of the Grey-Box clearly was related to the non-linear optimization methodology chosen to derive the ‘best fit’ mathematical expressions for the iterative surfaces of response/predictor variables. The GRG2 algorithm utilized to derive equation constants and factors is a gradient-descent approach known for its robustness and reliability. However, Gill et al. (1981) questioned whether the GRG2 algorithm truly reaches a global optimum in nonlinear problems, whereby errors in the gradient descent affect performance. In addition, although all ‘best fit’ equations suitably addressed the generated (and relatively simple) curvilinear output surfaces within this demonstration, more complex response surfaces likely would require a specialized optimization technique (e.g. the LevenbergeMarquardt nonlinear least squares fitting algorithm; see Moré, 1978). Similarly, the selection of equations for the mathematical generalization of output surfaces was limited to power, polynomial, and rational representations. For instances where highly developed output surfaces are generated, different and more sophisticated surface Fig. 7. Cumulative contribution of iterative equations within the estimated Grey-Box equations should be considered. model across the range of measured CHL concentrations for the training and cross- Traditional sensitivity about-the-mean analysis (i.e. noting the ‘ validation data subsets. The one hundred twenty-one concentrations (i.e. exem- variation in model output attributable to deviations of individual plars’, see section 2.2) were sorted from least to greatest and numbered in increasing value to denote differential impacts of scaled equations upon model prediction across input variables across their sample range, while other inputs are the range of measured concentrations. held to their respective sample means) was used to extract the ‘cause and effect’ relationships between predictor variables and the measured concentrations, denoting a nonlinear relationship evi- modeled CHL a concentrations. Sensitivity analysis was chosen for denced by slight contributions at lesser concentrations and signif- this demonstration due to the ease in its computation, the icant contributions at the greatest concentrations. From this, SAL simplicity of its comprehension, and its universal application and TEMP appeared to provide the principal and most consistent within statistical/mechanistic modeling (Saltelli et al., 2004; Yeung prediction across lesser CHL concentrations, whereas pH and TURB et al., 2010). Importantly, the use of sensitivity analysis afforded were primary predictors for the greatest concentrations (also a systematic means to exclude input variables from the Grey-Box demonstrated by the differences in the scaling of the y-axes in that did not contribute significantly to overall prediction, thereby Figs. 2C/D and 3C/D). The third and fourth iterations continued to reducing model complexity and improving model quality improve upon model performance (in terms of the model error); (Weckman et al., 2009). Complementary, yet more complex however, the relative contribution of the selected predictors for methods for determining variable influences and/or decision util- these iterations to model prediction was minimal. ities from networks do exist (see Gevery et al., 2003; Olden et al., Although successive iterations of the Grey-Box improved model 2004; Weckman et al., 2005, 2009) and might have been utilized performance - in both the training/cross-validation and testing data for the derivation of the Grey-Box model. Nonetheless, from sets, maximum values for the determination coefficient between whichever methodology utilized, the selection of input variables modeled:measured values were reached by the second iteration and the derivation of scaling factors for successive Grey-Box iter- (i.e. after inclusion of four predictor variables in the model). It must ations requires quantifiable rankings of predictive importance for be remembered that the coefficient of determination is a metric model inputs. denoting the proportion of the response variation explained by The ANN and Grey-Box models outperformed MLR, proving to regressors in a model for a conditional probability distribution, or be an attractive substitute (to parametric linear models) for iden- ‘what is the value of a response variable, y, given a specific value of tifying environmental influences upon CHL a within Sarasota Bay. the variable x, with the distribution parameters, q’;(P (xjy, q). For This greater performance is not surprising. In theory, an ANN environmental data sets, response and predictor variables should encompasses linear regression and due to the introduction of be considered as multivariates in a joint probability distribution, or a model architecture suited for reproducing the non-linear, ‘what is probability of two or more things (y and single/multiple x) complexities/dynamics of a biotic response to environmental happening together’; P (x,yjq). Accordingly, the determination forcing (see Sugihara et al., 1990; Austin, 2002; Oksanen and coefficient alone would not adequately describe the complexity of Minchinb, 2002), should perform as well, or better than MLR predictor/response relationships and it would be more appropriate models (Gonzalez, 2000). However, one should not solely focus on to utilize error metrics and scaling factors for evaluating model whether MLR appears to underperform the network models (based performance between successive iterations. Such was the case for upon interpretation of performance metrics). Depending upon the the third iteration of both the training/cross-validation and testing regression model, CHL a concentrations displayed hetero- data sets, when coefficient values for the fitted equation decreased scedasticity and/or modeled residuals signified a condition of non- only slightly (from that of the second iteration), but error metrics linearity. In a strict sense, such conditions invalidate the underlying and values of wi.n continued to improve (i.e. decrease). A fifth assumptions for linear regression testing (see Reckhow et al., 1990; iteration (data not shown), in which urea nitrogen (UR-N) and Osborne, 2002; Osborne and Waters, 2002), providing for a para- dissolved oxygen (DO) were selected as predictor variables, did not metric model lacking statistical merit. improve significantly upon the (cumulative) results following the The Grey-Box should be considered, at best, an iterative fourth iteration; relative improvements in values of w (from 0.06 to approximation of a conventional ANN. Although the Grey-Box 0.04) and model error (from 46.06 to 44.44 MSE and 5573 to 5377 possessed the positive attributes associated with ANNs (i.e. non- SSE) were relatively minimal. Accordingly, this fifth iteration was linear modeling capabilities, no requirement of a normal proba- deemed unnecessary and only four iterations were included in the bility distribution for data, etc.), a ‘less than ideal’ 1:1 mod- Grey-Box model. eled:measured relationship resulted for the final, non-rotated D.F. Millie et al. / Environmental Modelling & Software 38 (2012) 27e39 37 cumulative plot. This can be attributed to the partial utilization of routine usage of ANNs within studies of ecological assessment and candidate predictors (i.e. incorporating eight, rather than 15 vari- problem solving. The Grey-Box derivation particularly is pertinent ables) and the resulting loss of information encapsulated among to the development of data-driven models predicting a species’ auto-correlated variables (refer to Fig. 5D; c.f. Rohani et al., 2004). presenceeabsence, abundance dynamics, and/or distribution based Rotation and offset corrected the model for instances when the upon system-specific abiotic/biotic, geographic and/or historic iterative summation plot under- and over-estimated (or conversely, influences (akin to ecological niche modeling, Peterson, 2001, over and under-estimated) data within its lower and upper ranges, 2006; Peterson and Kluza, 2005; Carrick, 2011). The quantitative, respectively, with the final presentation approximating a (desir- declarative knowledge provided by the Grey-Box could provide able) 1:1 modeled:measured relationship. In corroborating the time-specific predictions of a species occurrence, with response iterative modeling technology with datasets distinct from the Sar- surfaces affording comprehensible projections of species presence/ asota Bay dataset (e.g. Millie et al., 2006a), the Grey-Box consis- quantities within the data boundaries of autecological and/or tently provided greater and lesser prediction accuracies than MLR synecological predictors. Within this context, Grey-Box formula- and holistic ANNs, respectively. tions are relevant to applications attempting to identify biotic response(s) to environmental stress and disturbance thresholds 5. Summary and recommended Grey-Box applications (see Clements and Rohr, 2009; Baker and King, 2010). Brenden et al. (2008) noted that accuracies in identifying such thresholds are Data sets arising from dynamic ecological systems often require affected by multiple factors and, in order to circumvent difficulties unconventional numerical approaches, necessitating robust and in interpretation of results, advocated the use of robust, quantita- adaptive analyses for data manipulation/minimization, trend tive approaches that included plotting and visualization of data. analysis, and information synthesis (Wood, 2010; Recknagel, 2011; The Grey-Box approach accomplishes this. Michener and Jones, 2012). Because of their robust capability for Additionally, Grey-Box derivations appear advantageous for identifying and reproducing the inherent complexities underlying coastal monitoring programs where the recent, increased reliance chaotic, non-linear systems, ANNs have become popular for the upon autonomous data acquisition has resulted in the generation of statistical-based modeling of ecosystem structure and function. copious amounts of auto-correlated data that users must synthe- However, ANNs require little (if any) expert knowledge for their size and ultimately interpret (see Paerl et al., 2005; Millie et al., application and do not exhibit a declarative knowledge structure; 2006a; Reed et al., 2010). Initially, the Grey-Box could afford the as such, many scientists consider such models to be numerical identification and minimization of pertinent predictor variables enigmas. (for a specific modeled response), with qualitative/quantitative The holistic ANN and the Grey-Box model outperformed MLR in functionalities generated for single (or multiple) locale(s). Derived identifying and reproducing the non-linear, environmental influ- ‘definitions’ of biota-environmental relationships from specific ences upon CHL a within Sarasota Bay. The architecture of the Grey- locales then could be incorporated into stand-alone mechanistic Box provided the benefit of the ANN modeling structure (albeit, at models projecting ecological structure and/or estimating devia- a lesser predictive performance), while utilizing a systematic tions from predictability over larger geographic scales (after Millie ‘bottom-up’ curve-fitting approach. This ultimately afforded et al., 2006b, 2011). a cumulative mathematical formulation of iterative, multivariate nonlinear models. The derivation of an explicit empirical quanti- Acknowledgements tative expression corrected for the lack of declarative knowledge ‘ (within conventional ANNs) and allowed the Grey-Box to bridge Funding for this research was provided by the Florida Depart- ’ the gap between white- and black-box models in both mathe- ment of Environmental Protection, Florida Coastal Management matical transparency and performance. Program (through grants from the National Oceanic and Atmo- The generalization of prediction via the Grey-Box, while spheric Administration [NOAA] Office of Ocean and Coastal utilizing a systematic decrease in predictor variables (in successive Resource Management under the Coastal Zone Management Act of iterations) and in more comprehensible manner than that of the 1972, as amended, Award #s NA10NOS4190178 and ANN, was impressive. However, the derivation of Grey-Box can be NA11NOS4190073) and by the Oceans and Human Health Initiative cumbersome and manually intensive due to the experimentation of NOAA’sOffice of Global Programs, Center for Sponsored Coastal required, the generation of output response surfaces, the derivation Ocean Research. Reference to proprietary names are necessary to fi of iterative weights, and the tting of mathematical equations to report factually on available data; however, Palm Island Enviro- output surfaces. As such, the usage of the Grey-Box technology Informatics LLC, Loyola University New Orleans, the State of Flor- might be problematic for some applications. Nonetheless, if ida, the Ohio University, Central Michigan University, and NOAA a certain statistical approach is employed just because it is neither guarantee nor warrant the standard of a product and imply uncomplicated and/or provides for handling in a simple, exact no approval of a product to the exclusion of others that may be manner, one runs the risk that the selected methodology disregards suitable. The statements, findings, and recommendations or discounts information pertinent to fully understanding the expressed herein are those of the authors alone. This work is fi system of interest (Malmgren, 2000). Accordingly, bene ts of the contribution #1616 of NOAA’s Great Lakes Environmental Research declarative knowledge and/or rule sets afforded by the Grey-Box Laboratory and contribution #19 of the Institute for Great Lakes outweigh the aforementioned detriments and advocate its Research, Central Michigan University. application. Quantitative knowledge extraction from ANNs modeling biotic responses to environmental forcing plays a vital role in the inno- References vative application of statistical models for ecosystem conceptuali- Austin, M.P., 2002. Spatial prediction of species distribution: an interface between zation, potentially leading to discoveries of (previously ecological theory and statistical modeling. Ecol. Model. 157, 101e118. unrecognized) variable relationships and interaction across Bachmann, R.W., Hoyer, M.V., Canfield Jr., D.E., 2001. Evaluation of recent limno- system- and organismal scales (Browne et al., 2003; Millie et al., logical changes at Lake Apopka. Hydrobiologia 448, 19e26. Bachmann, R.W., Jones, B.L., Fox, D.D., Hoyer, M., Bull, L.A., Canfield Jr., D.E., 1996. 2011; Recknagel, 2011; Michener and Jones, 2012). It is imagined Relations between trophic state indicators and fish in Florida (USA) lakes. Can. J. that the Grey-Box model will fuel a greater acceptance for, and Fish. Aquat. Sci. 53, 842e855. 38 D.F. Millie et al. / Environmental Modelling & Software 38 (2012) 27e39
Baker, M.E., King, R.S., 2010. A new method for detecting and interpreting biodi- Millie, D.F., Pigg, R.L., Fahnenstiel, G.L., Carrick, H.J., 2010. Algal chlorophylls: versity and ecological community thresholds. Methods Ecol. Evol. 1, 25e37. a synopsis of analytical methodologies. In: American Water Works Association, Barciela, R.M., Garcia, E., Fernández, E., 1999. Modelling primary production in Manual M57, Algae. American Water Works Association, Denver, Colorado, a coastal embayment affected by upwelling using ecosystem models and arti- pp. 93e122. ficial neural networks. Ecol. Model. 120, 199e211. Millie, D.F., Fahnenstiel, G.L., Dyble, J., Pigg, R., Rediske, R., Litaker, R.W., Tester, P.A., Brenden, T.O., Wang, L., Zhenming, S., 2008. Quantitative identification of distur- 2009. Late-summer phytoplankton in western Lake Erie (Laurentian Great bance thresholds in support of aquatic resource management. Environ. Manage. Lakes): bloom distributions, environmental influences, and toxicity. Aquat. Ecol. 42, 821e832. 43, 915e934. Browne, A., Hudson, B., Whitley, D., Ford, M., Picton, P., Kazemian, H., 2003. Millie, D.F., Weckman, G.R., Paerl, H.W., Pinckney, J.L., Bendis, B.J., Pigg, R.J., Knowledge extraction from neural networks. In: Proc. 29th Ann. Conf. IEEE Soc., Fahnenstiel, G.L., 2006a. Neural network modeling of estuarine indicators: Roanoke, Virginia, pp. 1909e1913. hindcasting phytoplankton biomass and net ecosystem production in the Neuse Carrick, H.J., 2011. Niche modeling and predictions of algal blooms in aquatic (North Carolina) and Trout (Florida) Rivers. Ecol. Ind. 6, 589e608. ecosystems. J. Phycol. 47, 709e713. Millie, D.F., Pigg, R., Tester, P.A., Dyble, J., Litaker, R.W., Carrick, H.J., Fahnenstiel, G.L., Cattaneo, A., 1987. Periphyton in lakes of different trophy. Can. J. Fish. Aquat. Sci. 44, 2006b. Modeling phytoplankton abundance in Saginaw Bay, Lake Huron: using 296e303. artificial neural networks to discern functional influence of environmental Chan, W.S., Recknagel, F., Cao, H., Park, H.-D., 2007. Elucidation and short-term fore- variables and relevance to a Great Lakes Observing System. J. Phycol. 42, casting of microcystin concentrations in Lake Suwa (Japan) by means of artificial 336e349. neural networks and evolutionary algorithms. Wat. Res. 41, 2247e2255. Millie, D.F., Carrick, H.J., Doerong, P.H., Steidinger, K.A., 2004. Intra-annual variability Clements, W.H., Rohr, J.R., 2009. Community responses to contaminants: using basic of water quality and phytoplankton within the St. Lucie River Estuary (Florida, ecological principles to predict ecotoxicological effects. Environ. Toxicol. Chem. USA): a quantitative perspective. Estuar. Coast. Shelf Sci. 61, 137e149. 28, 1789e1800. Moré, J.J., 1978. The Levenberg-Marquardt algorithm: implementation and theory. Dodds, W.K., Smith, V.H., Lohman, K., 2002. Erratum: Nitrogen and phosphorus Lect. Notes Math. 630, 105e116. relationships to benthic algal biomass in temperate streams. Can. J. Fish. Aquat. Murray, K., Conner, M.M., 2009. Methods to quantify variable importance: impli- Sci. 59, 865e874. cations for the analysis of noisy ecological data. Ecology 90, 348e355. Dodds, W.K., Smith, V.H., Lohman, K., 2006. Nitrogen and phosphorus relationships to Oksanen, J., Minchinb, P.R., 2002. Continuum theory revisited: what shape are benthic algal biomass in temperate streams. Can. J. Fish. Aquat. Sci. 63,1190e1191. species responses along ecological gradients? Ecol. Model. 157, 119e129. Florida Department of Environmental Protection (FDEP), 2006. Water Quality Olden, J.D., 2000. An artificial neural network approach for studying phytoplankton Assessment Report: Sarasota Bay and Peace and Myakka Rivers. Division of Water succession. Hydrobiologia 436, 131e143. Resource Management, Bureau of Watershed Management, Tallahassee, Florida. Olden, J.D., Jackson, D.A., 2002. Illuminating the “black box”: understanding vari- Gevery, M., Dimopoulos, I., Lek, S., 2003. Review and comparison of methods to able contributions in artificial neural net- works. Ecol. Model. 154, 135e150. study the contribution of variables in artificial network models. Ecol. Model. Olden, J.D., Joy, A.K., Death, R.G., 2004. An accurate comparison of methods for 160, 249e264. quantifying variable importance in artificial neural networks using simulated Gilbert, P.M., Harrision, J., Heil, C., Seitzinger, S., 2006. Escalating worldwide use of data. Ecol. Model. 178, 389e397. urea e a global change contributing to coastal eutrophication. Biogeochemistry Osborne, J., Waters, E., 2002. Four assumptions of multiple regression that 77, 441e463. researchers should always test. Available at: Pract. Asses. Res. Eval. 8 (2) http:// Gill, P., Murray, W., Wright, M.,1981. Practical Optimization. Academic Press, London. PAREonline.net/getvn.asp?v¼8&n¼2 Goh, A.T.C., 1995. Back-propagation neural networks for modeling complex systems. Osborne, J., 2002. Notes on the use of data transformations. Available at: Pract. Artif. Intell. Eng. 9, 143e151. Asses. Res. Eval. 8 (6) http://PAREonline.net/getvn.asp?v¼8&n¼6 Gonzalez, S., 2000. Neural Networks for macroeconomic forecasting: a compli- Oussar, Y., Dreyfus, G., 2001. How to be a gray box: dynamic semi-physical mentary approach to linear regression models. Canadian Department of modeling. Neural Net. 14, 1161e1172. Finance Working Paper No. 2000-07, Ottawa, Ont., Canada. Paerl, H.W., 1988. Nuisance phytoplankton blooms in coastal, estuarine, and inland Gurbuz, H., Kirvak, E., Soyupak, S., Yerli, S., 2003. Predicting dominant phyto- waters. Limnol. Oceanogr. 33, 823e847. plankton quantities in a reservoir by using neural networks. Hydrobiologia 504, Paerl, H.W., Dyble, J., Pinckney, J.L., Valdes, L.M., Millie, D.F., Moisander, P.H., 133e141. Morris, J.T., Bendis, B., Piehler, M.F., 2005. Using microalgal indicators to assess Heffernan, J.B., Liebowitz, D.M., Frazer, T.K., Evans, J.M., Cohen, M.J., 2010. Algal human and climatically-induced ecological change in estuaries. In: Bortone, S.A. blooms and the nitrogen-enrichment hypothesis in Florida springs: evidence, (Ed.), Estuarine Indicators. CRC Press, Boca Raton, FL, pp. 145e174. alternatives, and adaptive management. Ecol. Appl. 20, 816e829. Paerl, H.W., Pinckney, J.L., Fear, J.M., Peierls, B.L., 1998. Eco-system responses to Horkik, K., Stinchcombe, White, M., 1989. Multilayer feedforward networks are internal and watershed organic matter loading: consequences for hypoxia in universal approximators. Neural Networks 2, 359e366. the eutrophying Neuse River Estuary, North Carolina, USA. Mar. Ecol. Prog. Ser. Hu, Y.H., Hwang, J.-N., 2001. Introduction of neural networks for signal processing. 166, 17e25. In: Hu, Y.H., Hwang, J.-N. (Eds.), Handbook of Neural Network Signal Processing. Peck, M.S., Leffler, A.J., Flint, S.D., Rywl, R.J., 2003. How much variance is explained CC Press, Boca Raton, Florida, pp. 1e30. by ecologists? Additional perspectives. Oecologia 137, 161e170. Jeong, K.-S., Recknagel, F., Joo, G.-J., 2003. Prediction and elucidation of population Peterson, A.T., 2001. Predicting species’ geographic distributions based on ecological dynamics of the blue-green algae Microcystis aeruginosa and the diatom Ste- niche modeling. Condor 103, 599e605. phanodiscus hantzschii in the Nakdong River-Reservoir system (South Korea) by Peterson, A.T., 2006. Uses and requirements of ecological niche models and related a recurrent artificial neural network. In: Recknagel, F. (Ed.), Ecological Infor- distributional models. Biodiv. Inform. 3, 59e72. matics; Understanding Ecology by Biologically-Inspired Computation. Springer- Peterson, A.T., Kluza, D.A., 2005. Ecological niche modeling as a new paradigm for Verlag, Berlin, pp. 195e213. large-scale investigations of diversity and distribution of birds. U. S. Department Johannet, A., Vayssade, B., Bertin, D., 2007. Neural networks: from black box toward of Agriculture, Forest Service Gen. Tech. Rep. PSW-GTR-191. transparent box application to evapotranspiration modeling. Proc. World Acad. Principe, J.C., Euliano, N.R., Lefebrve, W.C., 2000. Neural and Adaptive Systems: Sci. Eng. Technol. 24, 162e169. Fundamentals Through Simulation. John Wiley and Sons Inc., New York. Karul, C., Soyupak, S., Cilesiz, A., Akbay, N., Germen, E., 2000. Case studies on the use Reckhow, K.H., Clements, J.T., Dodd, R.C., 1990. Statistical evaluation of mechanistic of neural networks in eutrophication modeling. Ecol. Model. 134, 145e152. water-quality models. J. Environ. Eng. 116, 250e268. Jeong, K.-S., Kim, D.-K., Jung, J.-M., Kim, M.-C., Joob, G.-J., 2008. Non-linear autor- Recknagel, F., 2011. Ecological informatics: a discipline in the making. Ecol. Inform. egressive modelling by Temporal Recurrent Neural Networks for the prediction 6, 1e3. of freshwater phytoplankton dynamics. Ecol. Model. 211, 292e300. Recknagel, F., French, M., Harkonen, P., Yabunaka, K.-I.,1997. Artificial neural network Lasdon, L., Waren, D., Jain, A., Ratner, M., 1978. Design and testing of a generalized approach for modeling and prediction of algal blooms. Ecol. Model. 96, 11e28. reduced gradient code for nonlinear programing. ACM Trans. Math. Soft. 4, 34e49. Reed, R.E., Burkholder, J.M., Allen, E.H., 2010. Current online monitoring technology Lee, J.H.W., Huang, Y., Dickman, M., Jayawardena, A.W., 2003. Neural network for surveillance of algal blooms, potential toxicity, and physicalechemical modeling of coastal algal blooms. Ecol. Model. 159, 179e201. structure in rivers, reservoirs, and lakes. In: American Water Works Association, Lek, S., Guegan, J., 1999. Artificial neural networks as a tool in ecological modeling, Manual M57, Algae. American Water Works Association, Denver, Colorado, an introduction. Ecol. Model. 120, 65e73. pp. 1e24. Maier, H.R., Dandy, G.C., Burch, M.D., 1998. Use of artificial neural networks for Rohani, P., Miramontes, O., Keeling, M.J., 2004. The colour of noise in short modeling cyanobacteria Anabaena spp. in the River Murray, South Australia. ecological time series data. Math. Med. Biol. 21, 63e72. Ecol. Model. 105, 257e272. Saito, K., Nakano, R., 2002. Extracting regression rules from neural networks. Neural Malmgren, H., 2000. Artificial neural networks in medicine and biology; a philo- Net. 15, 1279e1288. sophical introduction. Opening lecture at the Artificial Neural Networks in Saltelli, A., Tarantola, S., Compolongo, F., Ratto, M., 2004. Sensitivity Analysis in Medicine and Biology (ANNIMAB)-1 conference, Göteborg, Sweden, May 13e16. Practice: A Guide to Assessing Scientific Models. John Wiley & Sons, Ltd, West ISSN: 1652-0459. Available at: http://gupea.ub.gu.se/handle/2077/19466. Sussex, England. Michener, W.K., Jones, M.B., 2012. Ecoinformatics: supporting ecology as a data- Sarnelle, O., 1992. Nutrient enrichment and grazer effects on phytoplankton in intensive science. Trends Ecol. Evol. 27, 85e93. lakes. Ecology 74, 551e560. Millie, D.F., Fahnenstiel, G.L., Weckman, G.R., Klarer, D.M., Dyble Bressie, J., Scardi, M., 2001. Advances in neural network modeling of phytoplankton primary Vanderploeg, H.A., Fishman, D., 2011. An ‘enviro-informatic’ assessment of production. Ecol. Model. 146, 33e45. Saginaw Bay (Lake Huron USA) phytoplankton: characterization and modeling Scardi, M., Harding Jr., L.W., 1999. Developing an empirical model of phytoplankton of Microcystis (Cyanophyta). J. Phycol. 47, 714e730. primary production: a neural network case study. Ecol. Model. 120, 213e223. D.F. Millie et al. / Environmental Modelling & Software 38 (2012) 27e39 39
Stow, C.A., Roessler, C., Borsuk, M.E., Bowen, J.D., Reckhow, K.H., 2003. Comparison Engineering System Design, Neural Networks, Evolutionary Programming, and of estuarine water quality models for total maximum daily load development in Artificial Life, vol. 15. ASME Press, New York, pp. 761e766. Neuse River Estuary. J. Wat. Resour. Plan. Manag. 129, 307e314. Weckman, G.R., Millie, D.F., Ganduri, C., Rangwala, M., Young, W., Fahnenstiel, G.L., Sugihara, G., Grenfell, B., May, R.M., 1990. Distinguishing error from chaos in 2009. Knowledge extraction from the neural ‘black box’ in ecological moni- ecological time series. Phil. Trans. R. Soc. Lond. B 330, 235e251. toring. J. Indust. Syst. Eng. 3, 38e55. Teles, L.O., Vasconcelos, V., Pereira, E., Saker, M., 2006. Time series forecasting of Wood, S.N., 2010. Statistical inference for noisy nonlinear ecological dynamic cyanobacteria blooms in the Crestuma Reservoir (Douro River, Portugal) using systems. Nature 466, 1102e1104. Artificial Neural Networks. Environ. Manag. 38, 227e237. Yeung, D., Cloete, I., Shi, D., Ng, W., 2010. Sensitivity Analysis for Neural Networks, Weckman, G.R., Millie, D.F., Ghai, V., Ganduri, C., 2005. A comparison of knowledge Neural Computing Series. Springer-Verlag, Berlin/Heidelberg. extraction techniques from an artificial neural network in ecological moni- Young, W., Weckman, G., 2009. Using a heuristic approach to derive a grey-box toring. In: Dagli, C.H., Buczak, A.L., Enke, D.L., Embrechts, M.J., Ersoy, O. (Eds.), model through an Artificial Neural Network knowledge extraction technique. Intelligent Engineering Systems Through Artificial Neural Networks. Smart J. Neural Comput. Appl. 19, 353e366.