A Decade of Progress in Iterative Process Control

ADECADEOFPROGRESSINITERATIVE PROCESS CONTROL DESIGN : FROM THEORY TO PRACTICE

Michel Gevers ∗,

∗ CESAME, Universit´eCatholiquedeLouvain Bˆatiment EULER, 4 av. Georges Lemaitre, B-1348 Louvain-la-Neuve, Belgium E-mail: [email protected]

Abstract: One of the most active areas of research in the nineties has been the study of the interplay between system identification and robust control design. It has led to the development of “control-oriented identification design”, the paradigm being that, since the model is only a tool for the design of a controller, its accuracy (or its error distribution) must be tuned towards the control design objective. This observation has led to the concept of “iterative identification and control design” and, subsequently, to model-free iterative controller design, in which the controller parameters are iteratively tuned on the basis of successive experiments performed on the real plant, leading to better and better closed loop behaviour. These iterative methods have found immediate applications in industry; they have also been applied to the optimal tuning of PID controllers. This paper presents the progress that has been accomplished in iterative process control design over the last decade. It is illustrated with applications in the chemical industry. Copyright ⃝c 2000 IFAC

Keywords: identiﬁcation for control, validation, iterative controller tuning

Acknowledgements 1. INTRODUCTION

Iterative process control is a control design method- The work reported herein is the result of collab- ology that has emerged in the nineties as a result orations with a large number of co-authors over of intense research efforts aimed at bridging the the last ten years. I would like to acknowledge gap between system identification and robust con- B.D.O. Anderson, P. Ansay, R.R. Bitmead, X. trol analysis and design. In order to give the reader Bombois, B. Codrons, F. De Bruyne, G.C. Good- an idea about how wide this gap was, we quote win, S. Gunnarsson, H. Hjalmarsson, C. Kulcsar, from a keynote lecture delivered at the 1991 IFAC J. Leblond, O. Lequin, L. Ljung, B. Ninness, A.G. Symposium on Identification (Gevers, 1991) 1 Partanen, M.A. Poubelle, G. Scorletti, L. Triest, P.M.J. Van den Hof, V. Wertz, and Z. Zang. The last ten years have seen the emergence of robust control theory as a major research activity. During the same period, research in system identification has dwindled, and it might be tempting to believe that most of the theoretical questions in 1 The author acknowledges the Belgian Programme on identification theory have been resolved for some Inter-university Poles of Attraction, initiated by the Bel- gian State, Prime Minister’s Office for Science, Technology and Culture. The scientific responsibility rests with the 1 Our extensive quote should of course not be construed author. as approval of the ideas expressed in that paper. time. The surprising fact is that much of robust tification of the model error, and whatever tools control theory is based on prior descriptions of that were available were totally incompatible with model uncertainty or model errors which classical the frequency domain uncertainty descriptions re- identification theory has been incapable of deliver- quired for robust control analysis and design. ing. Conversely, until recently identification theo- However, the most crucial manifestation of the rists have not spent much effort in trying to pro- “identification/control gap” was not so much duce the accurate uncertainty bounds around their these technical incompatibilities, but rather the estimated models that their robust control design total absence of synergy between the two parts of colleagues were taking for granted. It is as if, until the design: identification design and model-based afewyearsago,thecontroldesigncommunityand control design. The prevailing philosophy at the the identification community had not been talking time was: “First identify a model with a method much to each other. The gap between the surre- that also allows the estimation of error bounds on alistic premises on which much of robust control this model; then design a controller based on this design theory is built and the failure of identifica- model and its error bounds.” The problem is that tion theory to deliver accurate uncertainty bounds an identification method whose sole merit is to in the face of unmodelled dynamics has brought deliver error bounds on a restricted complexity to light major deficiencies in both theories, and a model may well produce a nominal model and sudden awareness from around 1988 of the need an uncertainty set that are ill-suited for robust to understand better the interactions between both control design. theories. Due to a lack of understanding of the interplay Surely, a natural place to search for an under- between identification and robust control, most standing of the interactions between identification of the earlier work focused on producing suitable and robust control design is in the adaptive con- estimates of model quality (or uncertainty), and trol community. Indeed, adaptive control combines on bridging the gap between identification and the design of an on-line identifier with that of a robust control. The most obvious manifestation of control law...... However, in their pursuit of this gap, and the one that triggered most of the an adaptive controller possessing some degree of research activity in the early nineties, was the real- robustness to unmodelled dynamics, the adaptive ization that robust control theory requires a priori control community split into two camps. There hard bounds on the model error, whereas classical were those who focused upon the specification of identification theory delivers at best soft bounds, more and more sophisticated control law design i.e. confidence ellipsoids in a probabilistic sense. procedures in the hope that a super control law This led to the development of new identification would be able to overcome the vagaries of any theories that were called “control-oriented” only identification method. And there were those who, because they delivered model uncertainty descrip- by cajoling and forcing the parameter estimates to tions that were compatible with those required by conform to certain boundedness conditions what- robust control theory. The question of whether ever the control law,wereaimingatdeveloping the identified models and their uncertainty de- asuperidentifierabletoleapoveranyreason- scriptions were likely to deliver high performance able controller in a single bound. An essential controllers was not addressed, at least initially. feature of adaptive control, however, is that the identification is performed in closed loop and that It later became clear that the great ‘hard-versus- the controller therefore impacts on the estimated soft’ debate was not the real issue. To quote from model and on its quality (i.e. its error with respect another plenary lecture (Gevers, 1993): An iden- to the true system). It is therefore to be expected tification and control design method that leads to that the separate designs of the identifier and aclosedloopsystemthatisstablewithprobability of the controller without regard for the effect of 99% is of course just as acceptable as an H∞- the control law on the identified model, or of the based design that leads to a ‘guaranteed stable’ identified model on the robustness of the control closed loop, but that is based on prior error bounds law, may not lead to a maximization of the global that cannot be verified. However, even though the robustness of the identifier/controller schema. ‘hard-versus-soft’ question proved to be a non- issue, numerous other technical stumbling blocks This last sentence was to become the program for had to be conquered before robust control analysis much of the research activity in the nineties: going and design could be applied to models identified from separate designs to a synergistic design. In from data, rather than just to models and model 1990, any observer of the scene was aware of the uncertainty sets obtained from prior assumptions. many different technical inconsistencies between the newly emerged robust control theory and the To summarize, the intense research effort of the more classical prediction error identification the- nineties on identification for control has pursued ory. For example, prediction error identification two major objectives: theory had very little to offer in terms of quan- • Obtaining better estimation procedures for tification and robust control, but they do not the quantification of the model uncertainty constitute this synergy. They are the technical for identified models; in particular, produce building blocks. Indeed, identification for control uncertainty descriptions that are compatible is a “design” problem, as its name indicates. We with robust control theory. now explain how to give meaning to this problem, • Understanding the interaction between iden- and why the solution leads to iterative controller tification and model-based control in order design. to produce control-oriented identification design guidelines. 2. FROM IDENTIFICATION FOR CONTROL In this paper, we shall mainly focus on the TO ITERATIVE IDENTIFICATION AND progress accomplished in identification for control CONTROL design,i.e.thesecondissue.Thislineofresearch has led to such important new concepts as it- 2.1 The setup erative model-based controller redesign, cautious model and controller updates, and eventually it- All through this paper we consider the situation erative model-free controller redesign. But before where there is a “true system”which,forthe we venture in this direction, let us first briefly sake of analysis, we assume to be linear time- elaborate on the question of model uncertainty, invariant. For the sake of simplicity, we also con- if only to clearly distinguish it from the question sider a single-input single-output system only in of identification for control. this paper. Thus, the true system is represented The quantification of the model error is of course by averyimportantobjective,whateverthegoal S : yt = G0(z)ut + vt, (1) of the identification step that has produced this model. A reputable engineer should never deliver where G0(z)isalineartime-invariantcausalop- aproducttohisclientwithoutsomestatement erator, y is the measured output, u is the control about the quality of that product, whether it be input, and v is noise, assumed to be quasistation- amachinetool,ameasurementdevice,orady- ary. namical model. When the product is a model, and when the client is a robust control designer, then We now consider the situation where we can this client expects a model quality statement that perform experiments on this system with the is compatible with his/her robust control design purpose of designing a feedback controller. We tools. There is no sense telling the robust control also consider that, most often, the system is engineer that the bias error on the delivered model already under feedback control, and that the task is characterized by some complicated minimiza- is to replace the present controller by one that tion formula, and that the noise-induced error is achieves better performance. This situation is characterized by ellipsoidal confidence regions on representative of very many practical industrial ameaninglessparametervector,ifallthecontrol situations. engineer can handle for his robust control design It is also typical of many industrial applications is a frequency domain error bound. When that that the system to be controlled is very complex, happens - and this is exactly what did happen and that it would therefore require a very high ten years ago - then the robust control engineer order dynamical model to represent the system leaves the room in disgust and starts developing a with high fidelity. Any model-based control design new identification theory which he calls ‘control- procedure would therefore lead to a very high oriented’, only because it can deliver model error order controller, since the complexity of a model- bounds that are compatible with existing robust based controller is of the same order as that of control theory. the system. The practical situation, considered in All through the nineties, we have witnessed a this paper, is where we want the to-be-designed tremendous activity, on the part of both com- controller to have low complexity. munities, in the area of model quality estimation and model uncertainty description, with a view of bridging the technical incompatibilities between 2.2 In search of a low complexity controller the two theories. We cannot possibly hope to reference the hundreds of relevant papers. One of There are many ways of obtaining a low-complexity the better surveys of this line of research, up to controller for a high order system. These include the middle of the decade, can be found in (Ninness identification, model reduction, or controller re- and Goodwin, 1995). duction, in open or in closed loop, etc. A compar- ison between identification methods and reduction The results on model quality are necessary for methods, on an industrial example, can be found the construction of a synergistic design of iden- in (Bendotti et al.,1998). Here we consider the rather obvious idea which et al.,2000).Weshallnotelaborateonthemin consists of identifying a low order model from data this paper, where we focus on iterative designs. collected on the real system, from which a model- based controller is then computed. Given that the low order model cannot possibly represent the 2.3 Matching identification and control objectives true system over the whole frequency range, it will have a systematic error called the bias error, We have described the context in which we op- in addition to the inevitable noise-induced error erate, and we have introduced the motivation for called the variance error.Thisbiaserror-and identification for control. We now show that the indeed the total error - must be taken into account matching of identification and control objectives in the control design; hence the importance of leads to iterative identification and control design. producing methods for the estimation of model To illustrate the need for iterative design, we errors. But what is even much more important take the simplest possible control design objective: than estimating the model error a posteriori is model reference control. Thus, consider the true to design the identification in such a way that system (1) and suppose we have identified a model the bias error does not harm the performance ˆ △ ˆ of the controller that will be designed on the G(z) = G(z,θ)ofG0,fromsomeparametrizedset basis of this approximate model. This is based on of low order models {G(z,θ) | θ ∈ Dθ}.Consider the observation that one can design a high per- acontrollaw formance controller with a model that has large u C z r y , error with respect to the real system (i.e. a very t = ( )[ t − t] (2) wrong model), as long as this model represents with high accuracy the dynamical features of the and assume that our control design objective is true system that are essential for control design. to design C(z)suchthattheclosedlooptransfer For example, it is essential that the model be very function from vt to yt is some prespecified S(z). accurate around the crossover frequency of the to- Then, given a model Gˆ(z), the controller C(z)is be-designed closed loop system, but the error in computed from its steady state gain can be huge. 1 = S(z), (3) The idea of tuning the bias error for control design 1+Gˆ(z)C(z) is at the core of “identification for control”. It is an identification design problem, whose objective We compare the real closed loop system of Fig- is to produce, within a specified class of restricted ure 1 with the designed closed loop system of complexity models, a nominal model whose bias Figure 2, with both loops driven by the same error distribution is tuned towards the control de- reference signal rt. sign objective. As we shall see later in this paper, vt this can only be achieved through a succession of model and controller iterations; hence the itera- rt + ut G yt tive schemes that have emerged in identification - C 0 for control. The tuning of the bias error has led to iterative schemes for the estimation of a ‘control-oriented’ Fig. 1. Actual closed loop system nominal model. However, to fully take advantage of robust control theory, one must develop an ‘identification for control’ theory not just for the rt + uˆt ˆ yˆt nominal model, but also for the uncertainty re- - C G gions around this nominal model. Indeed, robust control is a model-based design methodology in which the controller is designed on the basis of Fig. 2. Designed (or nominal) closed loop system anominalmodeltogetherwithanuncertainty region around the nominal model: see e.g. (Zhou Now, staring at Figures 1 and 2, and remembering et al.,1995).Itisthetaskofmodelvalidationto that there are no disturbances, one observes that: construct an uncertainty region around a nominal model. When the model and its uncertainty region G0C 1 are to be used for robust control design, then this yt = rt + vt validation step must also be tuned towards the 1+G0C 1+G0C control objective. This is a much harder problem C C ut = rt − vt, for which few results are presently available. In the 1+G0C 1+G0C prediction error framework, recent results have GCˆ been obtained in (Gevers et al.,1999)and(Gevers yˆt = rt. (4) 1+GCˆ The ‘control performance error’ 2 ,definedasthe (Zang et al.,1995),(VandenHofandSchrama, error between the actual and the designed out- 1995). This identification error is typically a norm puts, is given by: of the following error: y − yˆ = S(z,θ)[y (θ) − G(z,θ)u (θ)], (8) G C GCˆ t t t t y − yˆ = 0 − r t t 1+G C ˆ t ! 0 1+GC " where the data filter S(z,θ)isproportionaltothe 1 sensitivity function of the design loop (compare + vt (5) 1+G0C with (3)) and is now also θ-dependent. After some straightforward manipulations, this As a consequence, the approach suggested in all can be rewritten as known ‘identification for control’ schemes is to perform identification and control design steps in yt − yˆt = S(z)[yt − G(z,θˆ)ut]. (6) an iterative way, whereby the i-th identification Equation (6) can be seen as an equality between step is performed on filtered closed loop data acontrolperformanceerroronthelefthandside collected on the actual closed loop system with (LHS) and a filtered identification error on the the (i−1)-th controller operating in the loop. This right hand side (RHS). Indeed, the RHS is a corresponds to an i-th identification step in which the following filtered prediction error is minimized filtered (by S(z)) version of the output error yt − with respect to θ,forfixedθî−1: G(z,θˆ)ut,whereut and yt are collected on the actual closed loop system of Figure 1. Thus, it ˆ appears that if θ is obtained by minimizing the yt − yˆt = S(z,θi−1) Mean Square of the RHS of (6), i.e. by closed ×[yt(θî−1) − G(z,θ)ut(θî−1)]. (9) loop identification with a filter S(z), then this will minimize the Mean Square control performance We refer the reader to (Gevers, 1993), (Bitmead, error. In other words, apparently (6) shows that 1993) and (Van den Hof and Schrama, 1995) for we get a perfect match between control error and details and for a survey on such iterative schemes. identification error. However, life is more subtle An interesting question is whether these iterative and complicated. Indeed, the controller C(z)is identification and control schemes converge to the also a function of the model parameter vector θ via △ (3). Since the data collected on the real closed loop minimum of the achieved cost over the set C = system of Figure 1 are a function of C(z), they are {C(G(z,θ)) ∀θ ∈ Dθ} of all certainty equivalence also dependent on θ.Hence,amoresuggestiveand controllers. This corresponds to asking whether by correct way to write (6) is as follows: successively minimizing over θ the mean square of the prediction errors defined by (9) one will yt − yˆt = S(z)[yt(θ) − G(z,θ)ut(θ)]. (7) converge to the minimum of

Even though the RHS of (7) looks like a closed △ 2 loop prediction error, it cannot be minimized J(θ) = E{S(z,θ)[yt(θ) − G(z,θ)ut(θ)]} . (10) by standard identification techniques, because θ This question was analyzed in (Hjalmarsson et appears everywhere and not just in G(z,θ). We al.,1995b)formodelreferencecontrol;itwas have illustrated the fact that with the simplest shown there that the iterative identification and possible control design mechanism, namely Model control schemes do not generically converge to the Reference Control, one can apparently equate a minimum of the achieved control cost. ‘control performance error’ to an ‘identification error’, but that this identification error cannot be This does not mean that iterative identification minimized by standard identification techniques and control schemes have failed. In fact, the idea because the parameter vector appears in more of using available data, collected on the actual than just the model. In other words, we know closed loop system, to obtain a model that is that, to make the control error small, we should better suited for the design of a new controller, minimize the RHS of (7) with respect to θ,butwe has found immediate and widespread applications don’t know how to do this. because it is easy and intuitively reasonable. In typical applications large numbers of closed loop For optimization-based control design criteria, the data are flowing into the control computer, and it control performance criterion also defines an iden- makes a lot of sense to use these data to replace tification criterion that one would want to mini- the existing controller by one that achieves better mize with respect to the model parameters: see performance. In addition, the theoretical work on iterative model and controller adjustments has 2 It was called that way in (Astr¨˚ om, 1993), (Astr¨˚ om and shown that, in order to compute a new controller Nilsson, 1994) because, if the closed loop transfer function with better performance, the optimal experiment of the actual system was equal to the reference model S(z), is to perform closed loop identification. Thus, no this error would contain only the noise contribution. special experiments are required, and there is no the identification step altogether. This idea led need to “open the loop” in order to design the new to a gradient-based algorithm for the iterative model and the corresponding new controller. optimization of the parameters of any restricted complexity controller (Hjalmarsson et al.,1994), Thus, this is one area where the transfer of which was later called IFT, for Iterative Feedback technology from theoretical research to applica- Tuning. In the next section we describe the IFT tions has been extremely fast. The first applica- algorithm and some of its more recent develop- tions of control-oriented identification and itera- ments. tive model-based controller tuning were reported within months after the theoretical results had been produced. Representative examples can be 3. IFT: A MODEL-FREE ITERATIVE found in (Partanen and Bitmead, 1995), (Schrama CONTROLLER RETUNING METHOD and Bosgra, 1993), (de Callafon et al.,1993), (de Callafon, 1998). The practical impact of it- 3.1 Introduction to the IFT algorithm erative closed loop identification and controller redesign has been assessed in (Landau, 1999a), The key feature of the IFT algorithm is that an where some interesting observations are made on unbiased estimate of the gradient of a control the distinction between this batch-like mode of op- performance criterion is computed from signals eration and the more classical theory and methods obtained from closed loop experiments with the of adaptive control. present controller operating on the actual sys- The guidelines that emerged during the nineties tem. For a controller of given (typically low-order) for the application of iterative identification and structure, the minimization of the criterion is then control schemes were supported by intense re- performed iteratively by a Gauss-Newton based search that brought to light two essential features. scheme. For a two-degree-of-freedom controller, three batch experiments are performed at each • The benefits of closed loop identification step of the iterative design. The first and third when the model is identified with a view of simply consist of collecting data under normal designing a new controller: see e.g. (Liu and operating conditions; the only real experiment is Skelton, 1990), (Schrama, 1992a), (Hakvoort the second batch which requires feeding back, at et al.,1994),(Leeet al.,1995),(Hjalmarsson the reference input, the output measured during et al.,1996),(Landau,1999b). This pro- normal operation. Hence the acronym Iterative duced a revival of interest in the design of Feedback Tuning (IFT) given to this scheme. For closed loop identification methods: see e.g. aone-degree-of-freedomcontroller,onlythefirst (Hansen et al.,1989),(VandenHofand and third experiments are required. No identifica- Schrama, 1993), (Van den Hof et al.,1995). tion procedure is involved. • The need for cautious adjustments between successive model and controller updates, in The optimal IFT scheme was initially derived in order to guarantee closed loop stability or (Hjalmarsson et al.,1994).Givenitssimplicity, performance improvement with the new con- it became clear that this new scheme had wide- troller: see e.g. (Schrama, 1992b), (Bitmead ranging potential, from the optimal tuning of sim- et al.,1997),(Andersonet al.,1998)and ple PID controllers to the systematic design of (Anderson and Gevers, 1998). controllers of increasing complexity that have to meet some prespecified specifications. In particu- Despite its practical successes, the failure of all at- lar, the IFT method is appealing to process con- tempts to establish convergence of iterative iden- trol engineers because the controller parameters tification and control schemes was a major worry, can be successively improved without ever open- more from a theoretical than from a practical ing the loop. In addition, in many process control point of view. Indeed, in practice it was observed applications the objective of the controller design that major improvements in performance of the is to achieve disturbance rejection. With the IFT closed loop systems were obtained within the first scheme the tuning of the controller parameters for few iterations, after which the improvements were disturbance rejection is driven by the disturbances very minor. Divergence typically occurred only if themselves; there is no need for the injection of a one continued to iterate beyond these initial steps. persistently exciting reference signal. It is the analysis of (Hjalmarsson et al.,1995b) Since 1994, much experience has been gained with that revealed the reason for the possible diver- the IFT scheme: gence. This analysis led the authors to reformu- late the iterative identification and control de- • It has been shown to compare favourably sign scheme as a parameter optimization prob- with identification-based schemes in simu- lem, in which the optimization is carried directly lation examples: see (Hjalmarsson et al., on the controller parameters, thereby abandoning 1994), and its accuracy has been analyzed in (Hjalmarsson and Gevers, 1997). • It has been successfully applied to the flexible may be defined as the output of a reference d transmission benchmark problem posed by model Td,i.e.y = Tdr,butfortheIFTmethod I.D. Landau for ECC95, where it achieved knowledge of the signal yd is sufficient. The error the performance specifications with the sim- between the achieved and the desired response is plest controller structure (Hjalmarsson et al.,1995a). △ y˜(ρ) = y(ρ) − yd (12) • It has been tested on the flexible arm of Cr(ρ)G 1 the Laboratoire d’Automatique de Grenoble = 0 r − yd + v. 1+C (ρ)G 1+C (ρ)G (Ceysens and Codrons, 1995), on a ball- # y 0 $ y 0 on-beam system (De Bruyne and Carrette, If a reference model is used this error can also be 1997), for the temperature control of a water written as tube and for the control of a suspended plate (Molenaar, 1995). Cr(ρ)G0 1 y˜(ρ)= − Td r + v.(13) • It has been adapted to linear time-invariant 1+Cy(ρ)G0 1+Cy(ρ)G0 # $ MIMO systems (Hjalmarsson and Birke- land, 1998) and to time varying, and in This error consists of a contribution due to incor- particular periodically time-varying, systems rect tracking of the reference signal r and an error (Hjalmarsson, 1995). due to the disturbance. With IFT the following • It has been applied by the chemical multi- quadratic control performance criterion is used: national Solvay S.A. to the tuning of PID N N controllers for a number of critical control 1 J(ρ)= E (˜y (ρ))2 + λ (u (ρ))2 (14) loops for which opening the loop or creating 2N t t !t=1 t=1 " limit cycles for PID tuning was not allowed. % % Here we present the fundamentals of the IFT but any other differentiable signal-based criterion algorithm, and we then review the performance can be used. In (14) E[·]denotesexpectationw.r.t. achieved by the scheme at S.A. Solvay. the weakly stationary disturbance v.Theoptimal controller parameter ρ is defined by

ρ∗ =argminJ(ρ), (15) 3.2 The basic control design criterion ρ

We present here a basic version of the IFT al- The errorsy ˜t(ρ)andut(ρ)appearinginthecrite- gorithm; we refer to (Hjalmarsson et al.,1998) rion can be filtered by frequency weighting filters for a more complete derivation and discussion. Ly and Lu to give added flexibility to the design: We consider the unknown true system (1), to be see (Hjalmarsson et al.,1998). controlled by a two degrees of freedom controller: Let T0(ρ)andS0(ρ)denotetheachievedclosed ut = Cr(ρ)rt − Cy(ρ)yt (11) loop response and sensitivity function with the controller {Cr(ρ),Cy(ρ)},i.e. where Cr(ρ)andCy(ρ) are linear time-invariant transfer functions parametrized by some param- Cr(ρ)G0 1 nρ T0(ρ)= ,S0(ρ) .(16) eter vector ρ ∈ R ,and{rt} is an external 1+Cy(ρ)G0 1+Cy(ρ)G0 reference signal, independent of {vt}:seeFigure3. v Given the independence of r and v, J(ρ)canbe ru y written as Cr G0 - N 1 2 J(ρ)= (yd − T (ρ)r ) (17) 2N t 0 t t=1 Cy % & ' N 1 1 + E {S (ρ)v}2 + λ E (u (ρ))2 . Fig. 3. Block diagram of the closed loop system 2 0 2N t !t=1 " ( ) % To ease the notation we will from now on omit the The first term is the tracking error, the second time argument of the signals; however, we shall term is the disturbance contribution, and the last use the notation y(ρ), u(ρ)fortheoutputand term is the penalty on the control effort. the control input of the system (1) in feedback with the controller (11), in order to make explicit the dependence of these signals on the controller 3.3 Criterion minimization parameter vector ρ. Let yd be a desired output response to a reference We now address the minimization of J(ρ)given signal r for the closed loop system. This response by (14) with respect to the controller parameter vector ρ for a controller of specified structure. It j reference signals by {ri }, j =1, 2, 3, and the cor- j is evident from (12) that J(ρ)dependsinafairly responding output signals by {y (ρi)}, j =1, 2, 3. complicated way on ρ,ontheunknownsystemG0 Thus we have and on the unknown spectrum of {v}.Toobtain the minimum of J(ρ)wewouldliketofinda r1 r, solution for ρ to the equation i = yielding (20) 1 1 y (ρi)=T0(ρi)r + S0(ρi)vi ; ∂J r2 r y1 ρ , 0= (ρ)(18)i = − ( i) yielding (21) ∂ρ 2 1 2 y (ρi)=T0(ρi)(r − y (ρi)) + S0(ρi)vi ; N N 1 ∂y˜t ∂ut r3 = r, yielding (22) = E y˜t(ρ) (ρ)+λ ut(ρ) (ρ) . i N ∂ρ ∂ρ 3 3 !t=1 t=1 " y (ρ )=T (ρ )r + S (ρ )v . % % i 0 i 0 i i ∂J j If the gradient ∂ρ could be computed, then the Here vi denotes the disturbance acting on the solution of (19) would be obtained by the following system during experiment j at iteration i.These iterative algorithm: experiments yield an exact realization ofy ˜(ρi): ∂J 1 d ρ = ρ − γ R−1 (ρ ). (19) y˜(ρi)=y (ρi) − y , (23) i+1 i i i ∂ρ i while it is shown in (Hjalmarsson et al.,1998)that Here Ri is some appropriate positive definite matrix, typically a Gauss-Newton approximation of ∂y △ 1 ∂C ∂C the Hessian of J,whilethesequenceγi must obey (ρ ) = r (ρ ) − y (ρ y3(ρ ) ∂ρ i C (ρ ) ∂ρ i ∂ρ i i some constraints for the algorithm to converge to r i +# $ alocalminimumofthecostfunctionJ(ρ): see * ∂C + y (ρ )y2(ρ ) (24) (Hjalmarsson et al.,1994). ∂ρ i i , Such problem can be solved by using a stochastic is an unbiased estimate of ∂y (ρ ). approximation algorithm of the form (19) such as ∂ρ i suggested in (Robbins and Monro, 1951), provided The three experiments described above generate ∂J the gradient ∂ρ (ρi)evaluatedatthecurrentcon- corresponding control signals: troller can be replaced by an unbiased estimate. In order to solve this problem, one thus needs to 1 1 u (ρi)=S0(ρi) Cr(ρi)r − Cy(ρi)v , generate the following quantities: i 2 1 2 u (ρi)=S0(ρi) -Cr(ρi)(r − y (ρi)) −. Cy(ρi)vi , (1) the signalsy ˜(ρi)andu(ρi); 3 3 ∂y˜ ∂u u (ρi)=S0(ρi) -Cr(ρi)r − Cy(ρi)vi . . (2) the gradients ∂ρ(ρi)and ∂ρ (ρi); ∂y˜ - . (3) unbiased estimates of the productsy ˜(ρi) ∂ρ(ρi) These signals can similarly be used to generate the ∂u and u(ρi) (ρi). estimates of the input related signals required for ∂ρ 1 the estimation of the gradient (19). Indeed, u (ρi) The computation of the last two quantities has is a perfect realization of u(ρi), always been the key stumbling block in solving 1 this direct optimal controller parameter tuning u(ρi)=u (ρi), (25) problem. The main contribution of (Hjalmarsson et al.,1994)wastoshowthatthesequantities while can indeed be obtained by performing experiments on the closed loop system formed by the ∂u △ 1 ∂Cr ∂Cy 3 actual system in feedback with the controller (ρi) = (ρi) − (ρi) u (ρi) ∂ρ Cr(ρi) ∂ρ ∂ρ {Cr(ρi),Cy(ρi)}.Thisisdoneasfollows. +# $ * ∂C + y (ρ )u2(ρ ) (26) ∂ρ i i , 3.4 The IFT algorithm ∂u is an unbiased estimate of ∂ρ (ρi). At iteration i of the controller tuning algorithm, An experimentally based estimate of the gradient △ of J can be formed by taking the controller C(ρi) = {Cr(ρi),Cy(ρi)} operates on the actual plant. We then perform three experiments, each of which consists of collecting a ∂J (ρ )= (27) sequence of N data. Two of these experiments ∂ρ i (the first and third) just consist of collecting data * N 1 ∂y ∂u under normal operating conditions; the second is a y˜ (ρ ) t (ρ )+λu (ρ ) t (ρ ) . N t i ∂ρ i t i ∂ρ i real (i.e. special)experiment.WedenoteN-length t=1 / 0 % * * The next controller parameters are then computed atemperaturecontrolproblemforthetrayofa by replacing, in the iteration (19), the gradient of distillation column. Here we present the results the cost criterion by this estimate: obtained with a temperature regulation problem for a tray of a distillation column. An application −1 ∂J to a flow control problem in an evaporator is ρi+1 = ρi − γiR (ρi)(28) i ∂ρ presented in (Hjalmarsson et al.,1998). * The controller used was an industrial 2-degree- where {γ } is a sequence of positive real numbers i of-freedom PID controller where the derivative that determines the step size, and where {R } is i action is applied to y only, and where a first order asequenceofpositivedefinitematrices.Thekey filter is applied to y in order to limit the gain of the ∂J feature of our construction of ∂ρ (ρi), and also the controller at high frequencies when the derivative motivation for the third experiment, is that this action is used. The time constant of this filter estimate is unbiased: 1 is chosen as Td/8, Td being the derivative time ∂J ∂J constant. The PID regulator parameters were E (ρ ) = (ρ ), (29) iteratively tuned using the IFT scheme, with the ∂ρ i ∂ρ i ! " following design choices: Gauss-Newton direction, * step-size γi =1∀i,controlweightingλ =0, As a result, the controller parameters converge un- sampling period of 15 seconds, rd = yd =0during der reasonable conditions to a stationary point of 5hours.Thedeadtimeandthetimeconstantsof the performance criterion, provided the sequence the process were unknown. of controllers along the way are all stabilizing: see (Hjalmarsson et al.,1998). Figure 4 presents temperature deviations with respect to setpoint in a tray of a distillation column, Anumberofimplementationissuesaswellasde- over a 24-hour period, first with the original PID sign choices are addressed in detail in (Hjalmarsson parameters, then with the PID controller obtained et al.,1998).Theyconcernthechoiceofthestep after 6 iterations of the new scheme. Figure 5 size γi and of the matrix Ri in (28), the choice shows the corresponding histograms of these devi- of frequency weighting filters, the elimination of ations over 2-week periods. The control error has possibly unstable controllers in the filtering opera- been reduced by 70 %. tions (24) and (26), the enforcement of integral ac- Control error over 1 day − initial tuning Control error over 1 day − after 6 iterations tion, the attenuation of the effect of disturbances, 6 6 as well as the simplification that occurs in the case 4 4 of a one-degree-of-freedom controller, where the 2 2 third experiment is not necessary. One interesting 0 0 design parameter is the step size, which deter- −2 −2 mines how much the controller changes from one Temperature −4 Temperature −4 iteration to the next one. Before implementing −6 −6 0 5 10 15 20 0 5 10 15 20 anewcontrolleronecancompareitsBodeplot hours hours with that of the previous one, and possibly reduce the step size if one feels that the change is too Fig. 4. Control error over a 24-hour period before large. Similarly, one can predict the effect of a tuning and after 6 iterations of IFT new controller on the closed loop response and on the achieved cost using a Taylor series expansion: Control error over 15 days − initial tuning Control error over 15 days − after 6 iterations 20 20 see (Hjalmarsson et al.,1998). 18 18 16 16

14 14

12 12

% 10 % 10 4. APPLICATIONS OF IFT IN CHEMICAL 8 8 6 6

PROCESS CONTROL 4 4

2 2

0 0 −5 0 5 −5 0 5 The IFT scheme has been applied by the chemical Temperature Temperature multinational Solvay S.A. for the optimal tuning Fig. 5. Histogram of control error over 2-week of industrial PID controllers operating on a range period before tuning and after 6 iterations of of diﬀerent control loops. In each of these loops, IFT PID controllers were already operating. Important performance improvements were achieved using Figure 6 shows the Bode plots of the two-degree of the IFT method, both in tracking and in reg- freedom controller (Cr,Cy)beforeoptimaltuning ulation applications. The reductions in variance (full line), after 3 iterations of the IFT algorithm achieved after a few (typically 2 to 6) iterations (dashed line) and after 6 iterations (dotted line). of the algorithm ranged from 25 % in a ﬂow The gain was too low and the derivative action regulation problem in an evaporator, to 87 % in underused. PID : Cr PID : Cy 60 60 N N 1 2 2 40 40 Jm(ρ)= E (˜yt(ρ)) + λ (ut(ρ)) . 20 20 2N

Gain [dB] Gain [dB] !t=t0 t=1 " 0 0 % % −20 −20 −6 −4 −2 −6 −4 −2 10 10 10 10 10 10 We say in such case that a mask of length t0 is put Frequency [1/s] Frequency [1/s] 100 100 on the transient response of the tracking error. 50 50 Often it is not known a priori how much time 0 0 Phase [deg] Phase [deg] −50 −50 is required to achieve a setpoint change without

−100 −100 −6 −4 −2 −6 −4 −2 10 10 10 10 10 10 overshoot. In such case, one can perform the IFT Frequency [1/s] Frequency [1/s] iterations by initially applying a long mask, and Fig. 6. Bode diagram of the two-degree-of-freedom then gradually reducing the length of this length controller before tuning (full), after 3 itera- of this mask until oscillations start occuring. tions (dashed) and after 6 iterations of the We illustrate this idea with an example presented algorithm (dotted). in (Lequin et al.,1999).Considertheplant 1 Table 1 shows the measured cost J with the 6 G s ( )= 2 successive controllers, as well as the predicted s +0.1s +1 value of the cost, calculated at each iteration One wishes to tune a PID controller in order with the new controller parameters, as explained to achieve a settling time of 20 seconds for the above. The prediction was good except for the 2nd closed loop system. The initial PID parameter iteration which was perturbed by an abnormal values were taken as K =0.025, Ti =2and disturbance. Td =1,yieldingtheverysluggishresponseshown Iteration Cost Next cost in Figure 7. (measured) (predicted) 1 0.80 0.36 2 1.00 0.59 3 0.57 0.35 4 0.37 0.18 5 0.22 0.15 6 0.14 0.11 Table 1 : Calculated and predicted cost

Fig. 7. Closed loop step response with initial PID parameters 5. MINIMIZING THE SETTLING TIME WITH IFT The classical IFT criterion was then applied with adesiredresponseshownindottedlineinFig- ure 8, with the achieved response shown in full The criterion (14) is well suited when the objective line on that same figure. This response is very is to follow a specific reference trajectory, but is unsatisfactory, in large part due to an unfortunate not so appropriate if the objective is to change the choice of initial parameters. output from one setpoint to another one. Indeed, in such case the goal is typically to reach the new setpoint with a minimum settling time, and one does not care about the transient trajectory, provided it does not produce too much overshoot. By constraining the output to follow some particular reference trajectory yd during the transient, one puts too much emphasis on the transient phase of the response at the expense of the settling time at the new setpoint value. To cope with this situation Lequin observed in (Lequin, 1997) that one can add nonnegative Fig. 8. Optimal closed loop step response (full) obtained with the classical IFT criterion and weighting factors to each element ofy ˜t and ut in the criterion (14). A simple way to obtain using the desired response (dashed) asatisfactoryclosedloopresponsetoadesired setpoint change is then to set the weighting factors The IFT criterion was then applied with a mask ony ˜t to zero during the transient period and to of decreasing length, with an initial length of 80 one afterwards, while the weights on the control seconds, and with the same initial parameters. At are put to one everywhere: every iteration of the IFT scheme, the length of the mask was decreased by 20 seconds, until a Astr¨˚ om, K.J. and J. Nilsson (1994). Analysis of mask of length 20 was reached. This led to the aschemeforiteratedidentificationandcon- closed loop response shown in Figure 9. trol. In: Proc. IFAC Symp. on Identification. Copenhagen, Denmark. pp. 171–176. Bendotti, P., B. Codrons, C. M. Falinower and M. Gevers (1998). Control oriented low order modelling of a complex pwr plant: a compari- son between open loop and closed loop methods. In: Proc. 37th IEEE Conference on Deci- sion and Control.Tampa,Florida.pp.3390– 3395. Bitmead, R.R. (1993). Iterative control design approaches. In: Proc. 12th IFAC World Fig. 9. Optimal closed loop step response obtained Congress.Vol.9.Sydney,Australia.pp.381– with the IFT criterion using masks of de- 384. creasing length Bitmead, R.R., M. Gevers and A.G. Partanen Observe the dramatic improvement of the closed (1997). Introducing caution in iterative con- loop response. troller design. In: 11th IFAC Symp. on Sys- tem Identification.Fukuoka,Japan.pp.1701– 1706. 6. CONCLUSIONS Ceysens, B. and B. Codrons (1995). Synthèse itérative de controleurs sans identification. Iterative redesign of controllers using data col- Technical report. Masters Thesis. Université lected on the operating closed loop system has Catholique de Louvain. emerged as a new, powerful and successful con- De Bruyne, F. and P. Carrette (1997). Synthetic trol design methodology, as a result of signifi- generation of the gradient for an iterative con- cant progress accomplished in the nineties on the troller optimization method. CD-ROM of 4th understanding of the interplay between identifi- European Control Conference, paper THA- cation and control design. Most of the schemes F2. are based on model and controller updates; they de Callafon, R.A. (1998). Feedback Oriented iden- require safeguards such as cautious changes be- tification for enhanced and robust control: a tween successive controllers. The study of these fractional approach applied to a wafer stage. ‘identification for control’ schemes has somewhat PhD thesis. Delft University of Technology. surprisingly also led to iterative schemes that are de Callafon, R.A., P.M.J. Van den Hof and entirely model-free. M. Steinbuch (1993). Control-relevant identification of a compact disc pick-up mechanism. In this paper we have focused on the design of In: Proc. 1993 IEEE Conf. on Decision and the nominal model and/or controller via these Control.SanAntonio,Texas.pp.2050–2055. iterative schemes, since these have given rise to Gevers, M. (1991). Connecting identification the more practical design methods, well suited and robust control: A new challenge. In: for process control applications. We have barely Proc. IFAC/IFORS Symposium on Identi- touched upon the vast amount of progress accom- fication and System Parameter Estimation. plished on model uncertainty estimation, and have Budapest, Hungary. pp. 1–10. completely left aside our very recent theoretical Gevers, M. (1993). Towards a joint design of iden- work on model and robust controller validation. tification and control?. In: Essays on Control: Perspectives in the Theory and its Applica- tions, Birkhauser, New York (H.L. Trentel- 7. REFERENCES man and J.C. Willems, Eds.). Birkhauser. Anderson, B.D.O. and M. Gevers (1998). Funda- New York. pp. 111–151. mental problems in adaptive control. Perspec- Gevers, M., X. Bombois, B. Codrons, F. De tives in Control, a Tribute to Ioan DoréLan- Bruyne and G. Scorletti (1999). The role dau. of experimental conditions in model valida- Anderson, B.D.O., M. Gevers and C. Kulcsár tion for control. In: Robustness in Identifica- (1998). Caution in iterative modeling and tion and Control - Proc. of Siena Workshop, control design. In: IFAC Workshop on Adap- July 1998 (A. Garulli, A. Tesi and A. Vicino, tive Control and Signal Processing.Glasgow. Eds.). Vol. 245 of Lecture Notes in Control Astr¨˚ om, K.J. (1993). Matching criteria for control and Information Sciences.Springer-Verlag. and identification. In: Proc. 2nd European pp. 72–86. Control Conference.Groningen,TheNether- Gevers, M., X. Bombois, B. Codrons, G. Scorletti lands. pp. 248–251. and B.D.O. Anderson (2000). Model validation for control and controller validation: a Lequin, O. (1997). Optimal closed loop PID tun- prediction error identification approach. sub- ing in the process industry with the Itera- mitted for publication to Automatica. tive Feedback Tuning scheme. In: CD-ROM Hakvoort, R.G., R.J.P. Schrama and P.M.J. of European Control Conference, paper TH- Van den Hof (1994). Approximate identifica- A-H6.Brussels,Belgium. tion with closed-loop performance criterion Lequin, O., M. Gevers and L. Triest (1999). and application to LQG feedback design. Au- Optimizing the settling time with Itera- tomatica 30,679–690. tive Feedback Tuning. In: Proc. 14th IFAC Hansen, F., G. Franklin and R. Kosut (1989). World Congress, paper I-3b-08-3.Beijing, Closed-loop identification via the fractional P.R. China. pp. 433–437. representation: Experiment design. Proc. Liu, K. and R.E. Skelton (1990). Closed loop ACC. identification and iterative controller design. Hjalmarsson, H. (1995). Model-free tuning of con- 29th IEEE Conf on Decision and Control trollers: Experience with time-varying linear pp. 482–487. systems. Proc. 3rd European Control Confer- Molenaar, D. (1995). Model-free data-driven opti- ence pp. 2869–2874. mal tuning of controller parameters: A practi- Hjalmarsson, H. and M. Gevers (1997). Frequency cal guide with application examples. Techni- domain expressions of the accuracy of a cal report. Report LiTH-ISY-R-1722, Depart- model-free control design scheme. In: Proc. ment of Electrical Engineering. Linköping 11th IFAC Symp. on System Identification. University, Sweden. Vol. 1. Fukuoka, Japan. pp. 135–140. Ninness, B. and G.C. Goodwin (1995). Estimation Hjalmarsson, H. and T. Birkeland (1998). Itera- of model quality. Automatica 31(12), 32–74. tive Feedback Tuning of linear time-invariant Partanen, A.G. and R.R. Bitmead (1995). The MIMO systems. submitted to CDC 98. application of an iterative identification and controller design to a sugar cane crushing Hjalmarsson, H., M. Gevers and F. De Bruyne mill. Automatica 31,1547–1563. (1996). For model-based control design, Robbins, H. and S. Monro (1951). A stochas- closed-loop identification gives better perfor- tic approximation method. Ann. Math. Stat. mance. Automatica 32,1659–1673. 22,400–407. Hjalmarsson, H., M. Gevers, S. Gunnarsson and Schrama, R.J.P. (1992a). Accurate identification O. Lequin (1998). Iterative Feedback Tuning: for control: The necessity of an iterative theory and applications. IEEE Control Sys- scheme. IEEE Trans. on Automatic Control tems Magazine 18,26–41. 37,991–994. Hjalmarsson, H., S. Gunnarsson and M. Gevers Schrama, R.J.P. (1992b). Approximate identifica- (1994). A convergent iterative restricted com- tion and control design. PhD thesis. plexity control design scheme. In: Proc. 33rd Schrama, R.J.P. and O.H. Bosgra (1993). Adap- IEEE Conf. on Decision and Control.Or- tive performance enhancement by itreative lando, Florida. pp. 1735–1740. identification adn control design. Int. Jour- Hjalmarsson, H., S. Gunnarsson and M. Gevers nal of Adaptive Control and Signal Processing (1995a). Model-free tuning of a robust regu- 7(5), 475–487. lator for a flexible transmission system. Eu- Van den Hof, P.M.J. and R.J.P. Schrama (1993). ropean Journal of Control 1(2), 148–156. An indirect method for transfer function es- Hjalmarsson, H., S. Gunnarsson and M. Gev- timation from closed loop data. Automatica, ers (1995b). Optimality and sub-optimality 29 6,1523–1527. of iterative identification and control design Van den Hof, P.M.J. and R.J.P. Schrama (1995). schemes. In: Proc. American Control Confer- Identification and control - closed-loop issues. ence 95.Vol.4.pp.2559–2563. Automatica 31,1751–1770. Landau, I.D. (1999a). From robust control to Van den Hof, P.M.J., R.J.P. Schrama, R.A. adaptive control. Control Engineering Prac- de Callafon and O.H. Bosgra (1995). Iden- tice 7(9), 1113–1124. tification of normalized coprime plant fac- Landau, I.D. (1999b). Identification in closed- tors from closed-loop experimental data. Eu- loop: a powerful design tool (better design ropean Journal of Control 1,62–74. models, simple controllers). In: Proceedings of Zang, Z., R.R. Bitmead and M. Gevers (1995). the workshop ”Future directions in Systems Iterative weighted least-squares identification and Control Theory”(Org: M. Athans and J. and weighted LQG control design. Automat- Senteiro).Lisboa/Cascais,Portugal. ica 31(11), 1577–1594. Lee, W.S., B.D.O. Anderson, I.M.Y Mareels and Zhou, K., J.C. Doyle and K. Glover (1995). Ro- R.L. Kosut (1995). On some key issues in bust and Optimal Control.PrenticeHall,New the windsurfer approach to adaptive robust Jersey. control. Automatica 31(11), 1619–1636.