Introduction to Chemical Home Page Title Page

Engineering Mathematics Contents

JJ II

J I Vitaly Alexandrov Department of Chemical and Biomolecular Engineering Page1 of 128

University of Nebraska-Lincoln Go Back

Full Screen July 22, 2020 Close

Quit Home Page

Title Page

Contents

JJ II

J I

Page1 of 128

Go Back

"Simplicity is the ultimate sophistication." Full Screen

− Leonardo da Vinci Close

Quit Contents

Preface8

Recommended Literature8

Home Page 1 Mathematical Modeling in Chemical Engineering in a Nutshell9 Title Page

1.1 Governing Equations...... 10 Contents

1.2 Building a Realistic Model...... 12 JJ II

1.3 Initial and Boundary Conditions...... 14 J I

Page2 of 128 1.4 Simplification...... 15

Go Back Linearization...... 16 Full Screen Use of Symmetry and Geometry Simplification...... 19 Close Decoupling Equations...... 20 Quit Lumping...... 22 1.5 Solving the Governing Equations...... 22

1.6 Model Testing...... 25

2 Linear Algebra 27

2.1 Vector Spaces...... 27 Home Page

2.2 Scalar Product...... 31 Title Page

2.3 Vector Product...... 35 Contents

2.4 Linear Maps, Matrices and Determinants...... 41 JJ II

J I 2.5 Eigenvalue Problem and Matrix Diagonalization...... 50 Page3 of 128 2.6 Hermiticity and Unitarity...... 54 Go Back 2.7 Numerical Aspects of Linear Algebra...... 58 Full Screen 2.8 Case Study: Flow Graphs in Multicomponent Flash Separation.. 59 Close

2.9 Case Study: Protein Phosphorylation...... 61 Quit 3 Fourier Calculus 64

3.1 Fourier Series...... 65

3.2 Fourier Transform, Convolution and Correlation...... 70

3.3 Laplace Transform and Transfer Function...... 77 Home Page

4 Differential Equations 83 Title Page

4.1 Classification of Differential Equations...... 83 Contents

4.2 Separable ODEs...... 84 JJ II

J I 4.3 Logistic Equation, Population Growth and Tragedy of the Commons 85 Page4 of 128 4.4 Systems of Linear First-Order ODEs...... 88 Go Back 4.5 System Stability...... 92 Full Screen 4.6 Harmonic Oscillator and Green’s Function Method...... 94 Close

4.7 Numerical Methods for Solving ODEs...... 100 Quit 4.8 Partial Differential Equations (PDEs)...... 103

4.9 Case Study: Unsteady Heat Conduction and the Method of Lines. 103

5 Nonlinear Dynamical Systems: From Fireflies to Reactors 109

5.1 Prelude...... 109 Home Page

5.2 Nonlinear Algebraic Equations...... 111 Title Page

5.3 Case Study: Antoine Equation for Nonlinear Flash Separation... 112 Contents

5.4 Nonlinear Dynamics, Steady States and Multistability...... 113 JJ II

J I 5.5 Lotka-Volterra Model: Biological and Chemical Oscillators..... 114 Page5 of 128 5.6 Case Study: Glycolytic Metabolic Oscillator...... 119 Go Back 5.7 Case Study: Michaelis-Menten Enzyme Kinetics...... 122 Full Screen

Close

Quit Preface

Albert Einstein said that "The most incomprehensible thing about the world is that it is comprehensible." It is comprehensible quantitatively through mathemat- ics. What is even more remarkable is that many areas of mathematics started off as purely abstract theories (e.g., geometry, number theory, group theory) turned out to be able to explain the natural phenomena. Take the number π (the first letter of the Greek word "perimetros"), why does this irrational number help ratio- nalize so many real phenomena with no (direct) relation to circle’s circumference? Home Page Joseph Fourier developed a mathematical technique (Fourier analysis) in the 19th century to describe heat transfer. The human inner ear (cochlea), however, has Title Page performed Fourier transforms to process sounds for ages. We know that bees are smart enough to build the honeycomb in the form of regular hexagons. However, Contents humans (mathematician Thomas Hales to be precise) only in 1999 proved the Hon- eycomb Conjecture stating that a regular hexagonal grid is the best way to divide JJ II a surface into regions of equal area with the least total perimeter. J I

I strongly believe that it is essential to develop skills to understand mathemati- Page6 of 128 cal ideas using abstract formalism. This empowers you with the ability to ana- lyze diverse phenomena through the prism of the same fundamental mathematical Go Back concepts. Abstraction makes things simpler and reveals counter-intuitive. On the other hand, specific examples enable immediate connections with practice helping Full Screen build physical intuition. So, in these lecture notes I have tried to strike a balance between theory and applications. For generality purposes, I have not confined the Close pool of examples solely to chemical engineering, despite the title. Moreover, it should be stressed that there is no mathematics for Chemical Engineering. The Quit thing is that Nature does not know what we decided to call Chemical Engineering, Physics or Biology. In fact, the modern chemical engineering is so broad that the mathematical repertoire of a good chemical engineer should be quite extensive. Therefore, it is wise to develop a more universal understanding of mathematics as fundamental concepts resurface in many areas.

These lecture notes should be considered as a companion to the lectures that I teach at the University of Nebraska-Lincoln. To adhere to the lecture notes style, formal mathematical proofs are left out as they can be easily found elsewhere. The Home Page notes are not supposed to substitute exercises, but rather complement them. By solving a lot of practice problems by yourself, you will be able to translate your Title Page passive learning into active understanding of the subject. Remember that nobody made it to the NHL by watching ice hockey on Youtube. Since a lot of problems Contents of practical relevance cannot be solved analytically, homework assignments pro- vide ample opportunities to practice numerical modeling. But bear in mind what JJ II Eugene Wigner had to say: "It is nice to know that the computer understands the problem. But I would like to understand it too." In computing, the GIGO J I (Garbage In, Garbage Out) principle works particularly well. Page7 of 128 The covered topics include linear algebra, Fourier calculus and Laplace transforms, Go Back ordinary and partial differential equations, nonlinear dynamical systems. Before diving into specifics of mathematical methods, I outline a general framework for Full Screen mathematical methods applied in chemical engineering. To get the most out of the class it is recommended to read the appropriate section before coming to a class. Close Also, understand all the examples, solve all practice problems by yourself and subsequently analyze the provided solutions, complete all homework assignments. Quit Recommended Literature

General mathematics: 1. K. F. Riley, M. P. Hobson, S. J. Bence, "Mathematical Methods for Physics and Engineering" Cambridge University Press 2006 (all-in-one: a complete treatise on mathematical methods).

2. C. Henry Edwards and David Penney, "Elementary Differential Equations Home Page with Boundary Value Problems" Pearson 2007 (very good treatment of ODEs with lots of solved problems). Title Page

3. Alexander Altland and Jan von Delft, "Mathematics for Physicists" Cam- Contents bridge University Press 2019 (an ultimate textbook on undergraduate mathematics, except probability). JJ II

J I Chemical engineering mathematics and numerical modeling: Page8 of 128 4. Kevin Dorfman and Prodromos Daoutidis, "Numerical Methods with Chem- ical Engineering Applications" Cambridge University Press 2017 Go Back (geared to practical applications with lots of MATLAB problems). Full Screen 5. Kenneth Beers, "Numerical Methods for Chemical Engineering: Applications in MATLAB" Cambridge University Press 2019. Close

(more theoretical treatment of mathematical modeling) Quit 1. Mathematical Modeling in Chemical Engineer- ing in a Nutshell

The goal of this chapter is to outline a general framework for mathematical mod- eling in chemical engineering. It may be worth coming back to this chapter while studying the following topics to better connect the general principles described here with specific mathematical approaches and examples. To construct a math- ematical model of a physical phenomena two types of equations need to be com- Home Page bined - conservation laws (material-independent) and constitutive relations (material-dependent) - to obtain the governing equations. Title Page

REMARK: Regarding scientific models in general a famous statement by George Contents Box is in order: "All models are wrong, but some are useful." What it really means is that all models and even fundamental laws of nature as we know them now are JJ II inaccurate. For example, the Newton’s second law (F = ma) with mass m being constant was long believed to be applicable to any objects. Einstein demonstrated J I that this is wrong (or better inaccurate) − mass is not constant and increases when Page9 of 128 the object speeds up. The Newton’s law becomes more and more inaccurate as the speed of an object approaches the speed of light. Einstein figured out the science Go Back behind this phenomenon in his theory of special relativity. The Newton’s law of motion, however, is accurate enough when dealing with slow-moving objects. Full Screen

Even though the speed may be far smaller that the speed of light, the phenomenon Close complexity can still be too high to model it well such as in turbulent flows. Turbulent flows, however, are everywhere − the flow of wind and rivers, fluid flow Quit in pipes, pumps, turbines, blood flow in arteries. Despite a great progress, we still don’t quite understand the underlying Navier-Stockes equations. In fact, Feynman has called turbulence the most important unsolved problem of classical physics, while the Clay Mathematics Institute in 2000 made this problem one of its seven Millennium Prize problems in mathematics.

Thus, when you try to model a phenomenon using some governing equations, it is important to realize that any mathematical description is approximate. It is crucial to know the accuracy and limitations of your model. The model should Home Page allow you to capture the most important features of the phenomenon under study and make predictions that you can verify with experiments. Title Page

1.1. Governing Equations Contents Conservation laws for mass, energy and momentum in the system are called JJ II mass, energy and momentum balance equations. They provide relationships be- tween the system and its surroundings. These equations can also contain external J I sources (of mass, energy and/or momentum). Page 10 of 128 Constitutive relations describe the system response to external stimuli such as Go Back fields and forces. Many constitutive relations are phenomenological. Examples include the stress and strain relation in the Hook’s law (F = −kx) or the ideal gas Full Screen law (pV = nRT ).

Close Some constitutive relations are physical laws. For the transport phenomena (of mass, energy or momentum) the constitutive relations can be written in general Quit differential form as Flux = Coefficient × Gradient. (1.1) where coefficient is the characteristic of a material. For example, thermal transport equation (Fourier’s law) is in the form (Heat Flux) = (Thermal Conductivity) × (Temperature Gradient). In the case of diffusion the equation (Fick’s law) is written as (Diffusion Flux) = (Diffusion Coefficient) × (Concentration Gradient). To find the total flux through an area, the above differential equation needs to be integrated. You can think of the gradient (of temperature, concentration, etc.) as Home Page the driving force and the flux as the system response. Title Page Let’s see how it comes together on a simple one-dimensional example. Contents Example 1.1. Build a mathematical model for one-dimensional heat conduction through a non-moving liquid with a source term. The system variables and param- JJ II eters are: ρ - material density, cp - heat capacity, k - thermal conductivity, T - temperature, q - heat flux, A - cross-sectional area, and S - energy source strength. J I

Page 11 of 128 Solution: The problem is first formulated considering a small shell ∆x and small period of time ∆t. By taking the limit we arrive at the differential form of a Go Back mathematical model.

Full Screen 1) Energy balance (conservation law): accumulation: ρcp∆TA∆x Close conduction in: (qA)|x∆t conduction out: (qA)|x+∆x∆t Quit energy production: SA∆x∆t The balance equation reads:

accumulation = in − out + source ∆T q| − q| ρc = x x+∆x + S p ∆t ∆x Taking the limits ∆x → 0 and ∆t → 0 we get the differential equation: ∂T ∂q ρc = − + S (1.2) Home Page p ∂t ∂x Title Page

2) Fourier’s law of heat conduction (constitutive relation): Contents ∂T q = −k (1.3) x ∂x JJ II

J I 3) Combining conservative and constitutive equations: Page 12 of 128 ∂T ∂2T ρc = k + S (1.4) p ∂t ∂x2 Go Back

Full Screen

Close 1.2. Building a Realistic Model Quit A number of modifications to the above model could be introduced. First, the coefficient of thermal conductivity k is a scalar for isotropic materials, however, it becomes a tensor if the system has directional anisotropy. The fluid density ρ can be constant (incompressible fluid) or variable (compressible fluid). Similarly, equations involving fluid viscosity can be considered using either constant (New- tonian fluid) or variable (non-Newtonian fluid) viscosity depending on the applied force.

Also, there could be multiple processes in the system at the same time. For exam- ple, in addition to heat conduction considered above we can include convection. Home Page Thus, we have to add the convection term (Newton’s law of cooling) to obtain

∂T Title Page q = −k + h(T − T ) x ∂x ∞ Contents where h is the coefficient of thermal convection. JJ II The importance of these two contributions to the heat transfer could vary depend- ing on the system. This can be described by a dimensionless number called the J I Nusselt number (Nu): Page 13 of 128 convective heat transfer hL Nu = = conductive heat transfer k Go Back where L is the characteristic length. Clearly, the larger Nu, the more important Full Screen heat transfer through convection. Close The described approach is analogously applied when analyzing the relative im- portance of processes other than heat transfer. Just to name a few dimensionless Quit numbers: Reynolds number (Re): inertial resistance uD Re = = viscous resistance ν where u is the flow speed, D is the diameter of the tube, and ν is the kinematic viscosity of the fluid.

Prandtl number (Pr): Home Page momentum diffusivity ν P r = = Title Page thermal diffusivity α where ν is the kinematic viscosity, and α is the thermal diffusivity. Contents

Damköhler number (Da): JJ II

chemical reaction rate J I Da = mass transport rate Page 14 of 128

Go Back 1.3. Initial and Boundary Conditions The solution of a differential equation involves constants of integration. Their Full Screen values can be determined from boundary conditions for space variables and initial Close conditions for a time variable. The number of initial and boundary conditions is determined by the highest degree derivative in each independent variable. For Quit instance, to solve Eq.(1.4) we need two boundary conditions (the values of T specified at two different locations x) and one initial condition (one value of T at time t). These conditions are defined by the problem at hand. For the example described by Eq.(1.4) typical conditions could be

T (x = a) = Ta; T (x = b) = Tb; T (t = 0) = T0

One type of initial conditions, however, is worth singling out. For example, the 00 0 general solution to y + p(t)y + q(t)y = 0 has the form y = c1y1 + c2y2, where c1 0 and c2 are the two initial conditions (one for y and the second for y ). If we solve Home Page the equation for the following two initial conditions at some time t0 Title Page y1(t0) = 1, y2(t0) = 0 0 0 y1(t0) = 0, y2(t0) = 1 Contents we will obtain the so-called normalized solutions Y1 and Y2. The advantage of this is that the general solution to the equation with any initial conditions JJ II 0 y(t0) = a, y (t0) = b is simply given by J I y = aY1 + bY2 (1.5) Therefore, if we need to solve the differential equation for multiple initial condi- Page 15 of 128 tions, we can solve it once to get the normalized solutions and then the general solution to any set of initial conditions is obtained through a simple multiplication Go Back using Eq.(1.5). Full Screen

Close 1.4. Simplification Quit After a physically reasonable model is built and initial/boundary conditions are specified, some legitimate simplifications could be introduced. The goal of this step is severalfold:

1. Help analyze the phenomenon more clearly. 2. Enable analytical solutions and/or by-hand estimates. 3. Reduce computational burden for solving the governing equations.

Some common strategies to simplify a mathematical model are briefly discussed below. Home Page

Title Page Linearization Many real systems are non-linear. However, linearization of the governing equa- Contents tions is a powerful way to simplify a model. This is because i) linear equations are much easier to solve, ii) the machinery of linear algebra can be efficiently applied, JJ II iii) non-linearity can be recovered by combining a set of linear elements, iv) even if a linear model is not accurate enough, it can still prove to be useful, e.g., by J I providing the order-of-magnitude estimation. Page 16 of 128

The simplicity of linear equations relative to nonlinear counterparts is by virtue Go Back of the superposition principle: Full Screen Lfˆ (αx1 + βx2) = αLfˆ (x1) + βLfˆ (x2) (1.6) Close where Lˆ is any linear operator (algebraic, differential), and α, β are constants. This means that for all linear systems the overall output (response) is equal to Quit the sum of the responses that would have been caused by each input (stimulus) individually. It is this fundamental property of linear systems that enables the construction of solutions to complicated inputs from solutions to simpler ones. On the contrary, nonlinear systems are those in which the change of the output is not proportional to the change of the input. This may lead to highly complex behavior of even simple nonlinear systems. The Navier-Stokes equation in fluid dynamics, van der Waals equation of state in chemistry, the Lotka-Volterra equations in biology are all examples of nonlinear equations.

The idea of linearization is clear from the Taylor series expansion. The Taylor Home Page series for the sine function around a point is written as:

x3 x5 x7 Title Page sin x = x − + − + ... = x + O(x3) 3! 5! 7! Contents Thus, within the linear approximation sin (0.1) = 0.1. Since the accurate value is 0.099833..., the error due to linearization at x = 0.1 is less than 0.2%. For instance, JJ II the oscillatory motion of a non-linear pendulum is described by the second-order ODE J I d2α g = − sin α dt2 L Page 17 of 128 Note that the solution to even such a simple equation cannot be expressed in terms of elementary functions. However, it can be easily solved if approximated Go Back by a linear ODE in the case of small angles α by Full Screen 2 d α 2 = ω α Close dt2 with g being the acceleration of gravity, L - the length of string, and ω = pg/L - Quit the angular frequency of oscillations. Linearization is part of a more general strategy called perturbation theory. It states that a small change (perturbation) in the problem induces a small change in the solution. To illustrate how it works let us consider a simple algebraic example, however, the method can be applied to a variety of problems including differential equations. The equation x2 − πx + 2 = 0 (1.7) has two exact solutions: r π π2 Home Page x = ± − 2 = 2.254...; 0.887... 2 4 Title Page Let’s now compute the square roots using perturbation approach. Since π is close to 3, we can write π = 3+. If π was exactly 3, then the roots would be x = 2 and Contents x = 1, which is relatively close to the exact solutions. This is our starting point ("unperturbed" solution). Let’s now "perturb" π from 3 to 3+ and estimate how JJ II the solutions x = 2 and x = 1 will be modified. Thus, the equation becomes x2 − (3 + )x + 2 = 0 (1.8) J I while expanding x we get Page 18 of 128

2 x = x0 + x1 +  x2 + ... Go Back Truncating this series at the linear term (linear perturbation theory) we can re- Full Screen write Eq.(1.8) as

2 Close (x0 + x1) − (3 + )(x0 + x1) + 2 = 0 and after re-grouping the terms we arrive at Quit

2 2 2 (x0 − 3x0 + 2) + (2x0x1 − 3x1 − x0) +  (x1 + x1) = 0 (1.9) 2 The leading-order term x0 − 3x0 + 2 = 0 gives x0 = 2; 1, which is equivalent to π = 3 when taking  = 0 in Eq.(1.9). The next order term (2x0x1 − 3x1 − x0) = 0 leads to x1 = −1 if we plug in x0 = 1. Therefore, x = x0 + x1 = 1 −  and for  = π − 3 it gives x = 1 − (π − 3) = 0.8584.., to be compared with the exact value of 0.887... For the second root x0 = 2 we obtain x1 = 2, and therefore x = 2 + 2 = 2.283.., to be compared with the exact value of 2.254... If we aim to obtain more accurate estimates of the roots starting from the unperturbed values of x = 1 and 2, we then need to go beyond linear approximation and include next terms in the expansion. Home Page

Title Page In the study of non-linear dynamical systems linearization provides a way to assess the system stability around equilibrium points (steady states). In the correspond- Contents ing chapter on dynamical systems we will demonstrate this in a quantitative way by invoking what is called the Jacobian linearization approach. JJ II

Use of Symmetry and Geometry Simplification J I

Symmetry is one of the most powerful concepts in science and engineering. The Page 19 of 128 use of symmetry enables remarkable simplifications to a problem. The classical example is the use of curvilinear coordinates instead of Cartesian coordinates for Go Back systems with cylindrical or spherical symmetry. For example, when analyzing fluid flow in a pipe, the use of cylindrical coordinates (x = ρ cos φ, y = ρ cos φ, Full Screen z = z) greatly simplifies the solutions of the Navier-Stokes equation. Mathemati- cally, this is equivalent to a coordinate transformation represented by a matrix Close (Jacobian matrix). In the next section we will also discuss that if we are deal- ing with a system of equations then the best coordinate system is the one that Quit decouples equations. The model simplification can also be achieved by using a simpler geometry/shape of the system. For example, when dealing with rough surfaces, it may be not a bad approximation to treat the surface as perfectly flat. Another common geometric simplification is the assumption that the system under study is semi-infinite in one direction. Considering as an example heat transfer through a wall, two boundary conditions can be specified: ∂T (x = 0) = 0; T (x = L) = T ∂x L Home Page

Assuming the wall is semi-infinite, the boundary condition at x = 0 can be con- Title Page verted to lim T = T0 x→−∞ Contents where T0 is the initial temperature of the wall. This new boundary condition is equivalent to assuming that the temperature far from the exposed surface does JJ II not change from the initial value. This is a valid approximation at short expo- sure times. The advantage of this semi-infinite model is that it has a solution J I expressed as a single function rather than a slow-converging series. Page 20 of 128

Decoupling Equations Go Back

It is quite obvious that the simplest form of a quadratic n × n matrix is diagonal: Full Screen   λ11 0 ... 0 Close    0 λ22 ... 0  D =  . . . .  (1.10)  . . .. .  Quit  . . .  0 0 . . . λnn For example, the determinant of such a diagonal matrix is simply the product of its diagonal elements: det D = λ11λ22 . . . λnn. The other advantage of such representation is that the system of linear equations for this matrix of coefficients D is written as  y = λ x + 0x + ··· + 0x  1 11 1 2 n   y2 = 0x1 + λ22x2 + ··· + 0xn (1.11) ...    yn = 0x1 + 0x2 + ··· + λnnxn Home Page

It means that each dependent variable (y) depends only on one independent vari- Title Page able (x). Such a system of equations is said to be decoupled and can be easily solved. For instance, for the system of first-order ODEs the solutions are readily Contents λnx obtained as yn = cne . JJ II In the section about matrices, we will discuss a central result of linear algebra stating that any Hermitian (real symmetric) matrix is diagonalizable. Therefore, J I the corresponding system of equations can always be decoupled. We will show that the diagonalization procedure is equivalent to the change of variables that Page 21 of 128 can be obtained by solving the eigenvalue problem. Moreover, independent variables in the new decoupled system usually have more clear physical meaning. Go Back Importantly, most matrices appearing in science and engineering are of exactly this type (Hermitian) guaranteeing diagonalizability and decoupling. Bear in mind Full Screen that any nth-order ODE can be transformed to a system of n first-order ODEs (by introducing new variables such as z = y0). Moreover, in many cases PDEs can be Close converted to a set of ODEs, for example, by using the method of lines. This shows a hierarchy of connections between linear algebra and differential equations. Quit Lumping Lumping is a type of averaging approach used to simplify a model that involves a large number of species or processes by reducing the state space of the system. For example, in a lumped mechanism "similar" chemical species may be grouped ("lumped") together to be described as a single entity, thereby decreasing the number of equations.

1.5. Solving the Governing Equations Home Page

This is the topic of subsequent chapters where we will discuss the details of the Title Page methods. However, two concepts related to numerical analysis of the governing equations are worth briefly mentioning here, viz. stiff systems and conditioning. Contents

The first relates to the fact that different processes in the mathematical model JJ II can be characterized by orders-of-magnitude different time scales (stiff systems). This is common in chemical kinetics (different reaction rates), mixed processes J I such as reaction-diffusion, batch distillation (a wide range of relative volatilities), Page 22 of 128 process control (controllers with different stiffness parameters). This makes it chal- lenging for numerical integration as ideally different time steps would be necessary Go Back for slow and fast evolving terms (think of e−t vs. e−1000t).

Full Screen To characterize such stiff systems the stiffness ratio can be introduced as the ratio between the constants determining slow and fast terms. For example, for a system Close λnt of first-order ODEs where time-dependences in solutions yn = e determined by the eigenvalues λn, the stiffness ratio can be defined as the following ratio between Quit the real parts of largest and smallest eignevalues in the spectrum:

Re(λmax) Re(λmin)

The other key concept is conditioning. The question we may ask is how sensitive a system is to small perturbations in the input values. This is important since our input is never perfectly accurate and may vary because of approximate nature of Home Page experimentally measured data or due to round-off errors in simulations. Thus, we want to know if 1% error in the input would lead to 2% or 200% error in the Title Page output. This is directly related to the predictability of a mathematical model. To get a feeling about such sensitivity of the output to the input, the so-called Contents condition number is introduced. In the corresponding section we will show that for the linear system of equations Ax = b, the condition number for the matrix A JJ II characterizing the system can be estimated using matrix norms as J I

cond(A) = A−1 kAk ≥ A−1A = 1 Page 23 of 128

Go Back A system with a small condition number is said to be well-conditioned, whereas the one with a condition number much larger than one is called ill-conditioned. Full Screen It is important to point out that conditioning is an inherent property of the sys- tem and not of the applied numerical algorithm that is characterized by its own Close approximations/errors. To reduce a condition number for the given system (with matrix A) an iterative transformation of A called pre-conditioning can be ap- Quit plied. It can be shown that the best conditioned matrices are unitary (orthogonal) matrices. Unitary (orthogonal) matrices U are related to Hermitian (real symmet- ric) matrices H through U = exp(iH). Therefore, similar to Hermitian, unitary matrices are also diagonalizable.

We will later discuss the concept of system sensitivity in the context of non-linear dynamical systems and chaos. These systems are characterized by a huge sensi- tivity of the output to the initial conditions. This is reflected in the metaphorical term the butterfly effect coined in 1979 by Ed Lorenz in his paper "Predictabil- Home Page ity: Does the Flap of a Butterfly’s Wings in Brazil Set Off a Tornado in Texas?". In the context of chemical engineering it may mean that a slight change in the Title Page system temperature or concentration could lead to an explosion of the reactor.

Contents The other classical example of the issue of conditioning and sensitivity is so-called Wilkinson’s polynomial. Polynomial equations arise in many fields of science JJ II and engineering. In relation to numerical analysis, one of the most important ar- eas is solution to the eigenvalue problem leading to the characteristic polynomial J I det(A − λI) = 0. It was demonstrated by Wilkinson that the roots of this polyno- mial could be very sensitive to perturbations in the polynomial coefficients. Let’s Page 24 of 128 illustrate this following Wilkinson on a simple characteristic polynomial Go Back 1 +  − λ   det 1 3 = 0 (1.12) 4 1 + 2 − λ Full Screen

This can be explicitely written as Close

2 λ − (2 + 1 + 2)λ + (1 + 1)(1 + 2) − 34 = 0 (1.13) Quit with the roots given by (2 +  +  ) ± (( −  )2 + 4  )1/2 λ = 1 2 1 2 3 4 (1.14) 1,2 2 −6 It can be estimated that random perturbations of order 10 in all i lead to changes of the same order (10−6) in the roots. However, independent perturbations of order 10−6 in the coefficients result in perturbations of order 10−3 in the roots. The latter situation can often arise due to rounding errors in the coefficients. Home Page

The next question is can we remove the ill-conditioning by a simple transformation Title Page of the variables (pre-conditioning)? The answer is yes if the transformation itself is performed to high accuracy. The above example illustrates that even if the original Contents problem (1.12) is well-conditioned and quite simple (quadratic), an extreme ill- conditioning may be introduced by solving its characteristic polynomial (1.13). JJ II

1.6. Model Testing J I Once the model is built, it is important to test it. The first step is to solve Page 25 of 128 the equations for limiting cases to check the behavior of a model under different variable values. The idea here is to search for the breaking points, rather than Go Back avoid them. It is good practice to particularly cover corner and boundary cases such as using extreme (maximum and minimum) operating parameters or values Full Screen just outside of typical operating parameters. As a rule of thumb one should not expect a reasonable behavior of the system at conditions under which the model Close was not tested. The more testing is performed, the more predictable a model is. Quit Remember that small mistakes can lead to huge problems. It is worth reminding about the Mars Climate Orbiter - a satellite developed by NASA to explore Mars. The satellite launched in 1998 was lost due to a simple unit conversion problem - some engineers used the metric system (meters-kilograms), while the others used the US system (feet-pounds). It was the lack of testing, not the lack of knowledge, that cost $190 millions.

Home Page

Title Page

Contents

JJ II

J I

Page 26 of 128

Go Back

Full Screen

Close

Quit 2. Linear Algebra

REMARK: In most applications in engineering, we are dealing with a mathemat- ical structure familiar from high school called Euclidean vector space (discussed below), which is a finite-dimensional space over the real numbers. Nevertheless, one should realize that this space is a specific case of a more general notion of a space. In mathematics, the most general notion of a mathematical space relating points (mathematical objects whose nature could be very different, e.g., numbers, Home Page functions, etc.) through a set of axioms (rules) is called the topological space.

By fixing a set of rules, we precisely define the mathematical structure of a space Title Page (each having a different name). A manifold is a generalization of Euclidean vector space (any space that is Euclidean only locally). For example, a sphere like the Contents Earth can be considered locally flat, i.e., Euclidean. If we add a property of dif- ferentiability, we obtain a differentiable manifold that allows us to do calculus on JJ II manifolds. For example, our Universe is not a vector space, but a four-dimensional (with 3 space coordinates and one time coordinate) manifold. In chemical engi- J I neering most of the time you don’t worry about those details. However, being able to see a bigger picture for any topic could be useful. Page 27 of 128

Go Back

2.1. Vector Spaces Full Screen We learn in pre-school that a scalar is characterized by a single number (magni- Close tude). Examples of scalar quantities include the temperature, mass and volume of a body, time, energy, concentration, etc. On the other hand, a vector is described Quit by three numbers (components) in a regular 3-dimensional (3D) Euclidean space encoding both magnitude (distance) and direction. Examples of vectors include velocity, acceleration, force, momentum, etc. The numbers defining vectors x can be either real (3D vector space over the field of real numbers, x ∈ R3) or complex (x ∈ C3). A generalization to an arbitrary number of dimensions n leads to the definition of the real (complex) standard vector space Rn (Cn) as the set of all n-component objects x:

    Home Page  x1   2   x   n ≡ x =   x1, x2, ..., xn ∈ (2.1) Title Page R  .  R   .        xn  Contents

Such column vectors can alternatively be represented as the transpose of row JJ II vectors x = (x1, ..., xn)T . J I

REMARK: Scalars and vectors are members of a more general class of objects Page 28 of 128 called tensors. For example, in 3D space a scalar is the tensor of zero-order with 0 1 3 = 1 component, a vector is the tensor of first-order with 3 = 3 components, Go Back whereas a matrix mapping a vector to a vector is the example of a second-order 2 tensor with 3 = 9 components. Tensor analysis enables the most general descrip- Full Screen tion of linear algebra objects and their component-by-component transformations upon the change of a basis. Although in many cases a simpler analysis of a physical Close phenomenon in terms of scalars and vectors is sufficient, it could be often beneficial to work with tensors (e.g., stress tensor in fluid mechanics). Quit Definition 2.1. Vector space over a number field F (F = R or C) is defined as a triple (V, +, ·) comprised of a set of vectors V , a rule of vector addition

+ : V × V → V, (v, w) → v + w (2.2) and a rule of multiplication of vectors by scalars · : × V → V, (a, w) → a · w ≡ aw (2.3) F Home Page

Title Page Here, the Cartesian product × of two sets A and B is defined as A × B ≡ {(a, b) | a ∈ A, b ∈ B} (2.4) Contents containing all pairs (a, b) formed by elements of A and B. JJ II

J I The defined above triple (V, +, ·) satisfies the following vector space axioms: Page 29 of 128

1) The addition of vectors (V, +) forms a mathematical structure called group: Go Back i) v, w and v + w ∈ V (closure); Full Screen ii) (v + w) + z = v + (w + z) (associativity); Close

Quit iii) there exists the neutral element of addition 0 (null vector) such that v + 0 = 0 + v = v; iv) there exists the inverse element of a vector −v (negative vector) such that v + (−v) = 0.

This group is also Abelian: v + w = w + v (addition is commutative).

2) Multiplication by scalars satisfies the following rules for a, b ∈ F and v, w ∈ V : i) (a + b)v = av + bw (distributivity of scalar addition); Home Page ii) a(v + w) = av + bw (distributivity of vector addition); Title Page iii) (ab)v = a(bv) (associativity of scalar multiplication); Contents

JJ II iv) 1v = v (neutral element in F). J I

In 2D space each vector could be represented by two numbers, v ↔ (v1, v2)T . Page 30 of 128 However, depending on the metric system (e.g., meters, centimeters, inches, etc.) these numbers will be different. But this scaling (multiplication by a scalar) does Go Back not change the behavior of vectors. Moreover, even within the same metric system, the vector components will change if, e.g., we rotate the vector. This necessitates Full Screen to choose a basis set in which all vectors will be fixed through their components. A basis provides the concrete realization of a vector. Close

Quit Definition 2.2. A basis of the n-dimensional vector space V is a set of complete and linearly independent vectors in V meaning that any vector v ∈ V can be uniquely represented as a linear combination of basis elements,

1 2 n v = v1a + v2a + ... + vna (2.5)

Linear combinations such as this appear very frequently in linear algebra and are often substituted using the Einstein summation convention as follows:

n Home Page 1 2 n X i i A1B + A2B + ... + AnB = AiB ≡ AiB (2.6) i=1 Title Page

This pair of indices is called dummy indices, while their summation is called a con- Contents traction of indices. This representation of indices (with indices upstairs and down- stairs) is called covariant index representation as opposed to non-covariant JJ II representation for which both indices appear downstairs: J I v = v1a1 + v2a2 + ... + vnan (2.7) It is clear that some bases could be more convenient to work with (e.g., orthonormal Page 31 of 128 basis, to be discussed below after we introduce the scalar product). Go Back

Full Screen 2.2. Scalar Product Close It should be pointed out that the general definition of vector spaces introduced above does not contain any notion of vector length and direction. Therefore, we Quit can introduce one more attribute to the vector spaces, so-called Euclidean scalar product, that helps define the norm (geometric length) of a vector. Definition 2.3. A scalar product (aka inner or dot product) is a function of two vectors in Rn that produces a scalar (number) and is defined as

n n T 1 1 2 2 n n h, i : R × R → R, (v, w) → hv, wi ≡ v w ≡ v · w ≡ v w + v w + ... + v w (2.8)

Three key properties of the scalar product are Home Page i) hv, wi = hw, vi (symmetry); Title Page ii) hav, wi = ahv, wi and hu + v, wi = hu, wi + hv, wi (linearity); Contents iii) hv, vi > 0 for all v 6= 0 (positive definiteness). JJ II

A vector space equipped with a scalar (inner) product is called an inner product J I space or a Euclidean vector space. The definition introduced by Eq.(2.14) constitutes the algebraic formulation of the scalar product. The norm of a vector Page 32 of 128 can be defined as p p i Go Back kvk = hv, vi = viv (2.9)

The norm of a vector defines its geometric length. The vector v is normalized Full Screen to unit if kvk = 1. A unit vector is obtained by dividing the vector by its norm (normalization): Close v ˆv = (2.10) kvk Quit The angle between two vectors can be defined through the following expression

hv, wi = cos(∠(v, w))kvkkwk (2.11) This can be viewed as the geometric interpretation of the scalar product, viz. as the length (norm) of one vector (kvk) multiplied by the projection of the other vector onto the first one (cos(∠(v, w))kwk).

Any vector v can be decomposed into the parallel (projection) and perpendicular Home Page components as v = vk +v⊥. It can be easily proved using the vector decomposition picture that Title Page vk = ˆwh ˆw, vi (2.12) Contents v⊥ = v − ˆwh ˆw, vi The abstract definitions of a scalar product, norm and angle between two vectors JJ II introduced above can be also applied to other vector objects such as functions. However, unlike familiar vectors from Euclidean geometry in this case visualization J I of geometric angles is not possible. Page 33 of 128

Example 2.1. a) Find the angle between the vectors a = (3,4)T and b = (7,1)T . Go Back

T T Full Screen b) Consider the vectors c = (−1, 1) and d = (1, 2) . Decompose c = ck + v⊥ into components parallel and perpendicular to d, respectively. Close

Solution: Quit a) Using Eq. (2.11)

a · b 3 · 7 + 4 · 1 1 π cos(∠(a, b)) = = √ √ = √ ⇒ ∠(a, b) = kakkbk 9 + 16 · 49 + 1 2 4 b) According to Eq. (2.12)     (c · d) −1 · 1 + 1 · 2 1 1 1 Home Page ck = d = = kdk2 5 2 5 2 Title Page −1 1 1 1 −6 c = c − c = − = ⊥ k 1 5 2 5 3 Contents 1 −6 Consistency check: c · c = 1 [ · ] = 1 [1 · (−6) + 2 · 3] = 0. Also, check JJ II ⊥ k 5 2 3 5 the result graphically by drawing the corresponding vectors. J I

Example 2.2. a) Find the flux of the vector field F = xyi + 4x2j + yzk through Page 34 of 128 the oriented surface S defined as z = xey, 0 ≤ x < 1, 0 ≤ y < 1. Go Back Solution: A unit normal vector (remember that the unit vector is perpendicular to the tangent vector) is computed as Full Screen

∇S 1 Close nˆ = = (−eyi − xeyj + k) k∇Sk p(1 + x2)e2y + 1 Quit The total flux is ZZ ZZ −eyi − xeyj + k F · nˆds = (xyi + 4x2j + yzk) · (p(1 + x2)e2y + 1)dA = p 2 2y S A (1 + x )e + 1 ZZ Z 1 Z 1 (−4x3ey)dA = − 4x3eydxdy = 1 − e A 0 0

Home Page

Definition 2.4. If a set of basis vectors ei is i) unit normalized, keik = 1, Title Page and ii) pairwise orthogonal, hei, eji = 0 for i 6= j, then such a basis set is called orthonormal basis. The two conditions (normalization and orthonormality) can Contents be combined into a single expression as JJ II hei, eji = δij (2.13) J I where δij is called the Kronecker delta (symbol). The orthonormal bases are much Page 35 of 128 easier to work with and they can be obtained through an iterative procedure called Gram-Schmidt orthonormalization. Go Back

Full Screen

2.3. Vector Product Close Similar to the scalar product Eq.(2.14), let’s first provide the algebraic definition of the vector product. Quit Definition 2.5. A vector product (aka outer or cross product) is a function of two vectors in R3 that produces another vector and is defined as

3 3 3 × : R × R → R , (v, w) → v × w (2.14)

To provide the algebraic formula for the vector product, let’s introduce the so- called Levi-Civita symbol Home Page   1, if (i, j, k) cyclic  Title Page ijk = −1, if (i, j, k) anti-cyclic (2.15)  0, otherwise Contents where the indices (i, j, k) are cyclically or anti-cyclically ordered if they are ob- tained from the ordered triple (1, 2, 3) by an even or odd permutation, correspond- JJ II ingly. For example, (2, 1, 3) is anti-cyclic (going counter-clockwise) as it is obtained by one permutation, whereas (2, 3, 1) is cyclic (going clockwise) as it is obtained J I through two permutations. When applied to the orthonormal basis set of vectors, Page 36 of 128 it can be written:

ei × ej = ijkek (2.16) Go Back Notice similarity with the expression for the scalar product of two orthonormal Full Screen basis vectors Eq.(2.31). If ijk = 1, the basis is called positively oriented, and if ijk = −1, negatively oriented. The Levi-Civita symbol can be straightforwadly 3 n Close generalized from R to R . Similar to the Kronecker delta δij, the Levi-Civita symbol ijk is often used to simplify mathematical expressions and we will see it again when defining the determinant of a matrix. Quit With this definition the vector product between two vectors can be written as

i j i j i j v × w = (eiv ) × (ejw ) = v w (ei × ej) = v w ijkek (2.17) Thus, each component is given by

k i j (v × w) = v w ijk (2.18)

Example 2.3. Compute the vector product between a = (3, 2, 1)T and b = (1, −1, 1)T . Home Page

Solution: Using Eq.(2.18) we get Title Page

     2 3 3 2     3 1 a b − a b 2 · 1 − 1 · (−1) 3 Contents 2 × −1 = a3b1 − a1b3 =  1 · 1 − 3 · 1  = −2 1 2 2 1 1 1 a b − a b 3 · (−1) − 2 · 1 −5 JJ II Therefore, c = a × b = (3, −2, −5)T . J I

Page 37 of 128 The geometric interpretation of the vector product can be also deduced from the above algebraic definitions. According to Eq.(2.16), the vector product v × w Go Back is perpendicular to the parallelogram spanned by v and w. By definition, the magnitude of the vector product, i.e. its norm kv × wk, is equal to the geometric Full Screen area spanned by v and w and can be written as Close kv × wk =kvkkwk sin φ (2.19) Quit where φ is the angle between v and w. This can be also shown using the Levi- Civita notation by computing kv × wk2 = (v × w) · (v × w). The scalar and vector products are often combined into a triple product as i j k (v × w) · u = ijkv w u (2.20) where as usual the Einstein summation is used for pairs of upper and down indices. From the definitions provided above, it is clear that the triple product gives the volume of the parallelepiped spanned by the three vectors.

REMARK: Eq.(2.16) is only valid in the context of "flat" Euclidean geometry; Home Page however, physical space is not Euclidean. Our Universe is a four-dimensional (3 time coordinates + time) "curved" space for which the Pythagorean theorem is Title Page true only locally (e.g., when you measure distances in your backyard neglecting the Earth curvature). Thus, a more general Riemannian geometry of curved spaces is Contents necessary when dealing with long distances (e.g., in general theory of relativity) to which Euclidean geometry becomes an approximation over short distances. In JJ II this case, a generalization of Eq.(2.16) can be written as lk J I ei × ej = det (g)ijlg ek (2.21) where glk is the so-called metric tensor defined as a scalar product between pairs Page 38 of 128 of basis vectors glk = hel, eki and det (g) is the determinant of the metric tensor. It can be seen that for Euclidean space gii = 1 (for all i-s), gij = 0 and det (g) = 1, Go Back i.e., gij = δij, and therefore Eq.(2.21) reduces to the Euclidean version (2.16). Full Screen

In Euclidean spaces the metric tensor can be associated with the Jacobian matrix Close J. If we are dealing not with an orthonormal basis set {ei} but a general basis k Quit {vi} = {ekJi }, then the elements of the metric tensor can be written as k l T k l T gij = hvi, vji = hek, eliJi Jj = (J )i δklJj = (J J)ij (2.22) where we contracted k and l indices via Einstein summation. The connection can be seen if we represent the differential for the old Cartesian coordinates (x) in terms of new curvilinear coordinates (q):

 ∂v  dv = e i dqj (2.23) i ∂qj

If we now take the scalar product between two differentials (defining the elements of the metric tensor), we get Home Page

 ∂v ∂v  g = dv · dv = he , e i i j dqkdql (2.24) Title Page ij i j i j ∂qk ∂ql Contents If we denote ∂v ∂v i = J k, j = J l (2.25) ∂qk i ∂ql j JJ II the connection between the metric tensor g and Jacobian matrix J becomes ij J I evident. Due to the orthogonality of ei basis vectors only diagonal elements will survive (i = j) resulting in diagonal metric tensors for flat spaces. Page 39 of 128

According to Eq.(2.22), the determinant of the metric tensor Go Back

T T det (g) = det (J J) = det (J ) det (J) (2.26) Full Screen p Therefore, det (J) = det (g) relating the determinants of the Jacobian matrix Close and metric tensor. Let’s illustrate the described connection on the following ex- ample. Quit Example 2.4. Find the relationship between the metric tensor and Jacobian ma- trix using polar coordinates.

Solution: Polar coordinates (r, φ) are related to Cartesian coordinates (x, y) as x = r cos φ, y = r sin φ Let’s compute the elements of the metric tensor in polar coordinates using Eq.(2.24) recalling that off-diagonal elements must be zero: Home Page 3 X ∂vi ∂vi g = Title Page kl ∂qk ∂ql i=1 Thus, Contents ∂x ∂x ∂y ∂y g = + = cos2 φ + sin2 φ = 1 rr ∂r ∂r ∂r ∂r JJ II ∂x ∂x ∂y ∂y 2 2 2 2 2 gφφ = + = r sin φ + r cos φ = r (2.27) ∂φ ∂φ ∂φ ∂φ J I 1 0  =⇒ g = =⇒ det (g) = r2 Page 40 of 128 kl 0 r2 Now, let’s compute the Jacobian matrix defining the transformation from Cartesian Go Back to polar coordinates (see (2.25)): Full Screen ! ∂x ∂x cos φ −r sin φ J = ∂r ∂φ = =⇒ det (J) = r (2.28) ∂y ∂y sin φ r cos φ Close ∂r ∂φ

Therefore, as we demonstrated above for the general case det(J) = pdet(g) = r. Quit 2.4. Linear Maps, Matrices and Determinants In mathematics the terms map, function, and transformation are often used inter- changeably. However, sometimes maps are thought of as more general and applied to any sets of elements. For example, functions are typically used when dealing with numbers, whereas a function of a function is called a functional. Both func- tions and functionals are examples of maps. Depending on the context maps can have various properties such as smoothness, differentiability, linearity. Home Page Definition 2.6. A rule F assigning to each element a from set A (a ∈ A) an element b from set B (b ∈ B) is called a map: Title Page F : A → B (2.29) Contents Definition 2.7. An inverse rule F −1 realizing the backward transformation from b ∈ B to a ∈ A is called an inverse map: JJ II

F −1 : B → A (2.30) J I

Page 41 of 128 An important class of maps (functions) bears the property of linearity and plays a fundamental role in linear algebra. In the case of vector spaces introduced above, Go Back linear maps are defined as follows. Full Screen Definition 2.8. A map F : V → W between two vector spaces V and W is called a linear map if Close F (av + bw) = aF (v) + bF (w) (2.31) Quit where scalars a, b ∈ R and vectors v ∈ V , w ∈ W . If map acts only within one vector space, then V = W . A linear map acting on a vector x can be represented by the matrix A to yield the new vector y:     a11x1 + a12x2 + ... + a1nxn y1      a21x1 + a22x2 + ... + a2nxn   y2  Ax = y ⇐⇒  .  =  .  (2.32)  .   .   .   .  am1x1 + am2x2 + ... + amnxn ym Home Page Note that an n-component vector x can be considered as a 1 × n matrix. Title Page Example 2.5. Multiply the following vectors (matrices) by matrices: Contents 1 2 1 1 · 1 + 2 · 2  5  · = = 3 4 2 3 · 1 + 4 · 2 11 JJ II

1 2 5 6 1 · 5 + 2 · 7 1 · 6 + 2 · 8 19 22 · = = J I 3 4 7 8 3 · 5 + 4 · 7 3 · 6 + 4 · 8 43 50 Page 42 of 128 3
1 2 3 · 1 = 1 · 3 + 2 · 1 + 3 · 2 = 11 Go Back 2 3
· 3
= 3 · 3 = 9 Full Screen

Close

Let’s introduce the following operation for A called matrix transposition: Quit Definition 2.9. In a real vector space Rn exchanging rows and columns of a matrix A defines the transpose matrix, AT ,

1 2 1 3 A = −→ AT = 3 4 2 4 while in a complex vector space Cn transposition followed by complex conjugation defines the adjoint matrix, A†, Home Page 1 + 2i 3  1 − 2i 2 + i A = −→ A† = AT = 2 − i 2i 3 −2i Title Page

Contents T It can be easily checked that for a symmetric matrix (aij = aji) A = A : JJ II 1 2 1 2 A = −→ AT = ⇒ A = AT 2 3 2 3 J I

† while for a complex symmetric matrix (aij = aji, called Hermitian matrix) A = A : Page 43 of 128

1 + 2i 2 + i 1 + 2i 2 + i A = −→ A† = ⇒ A = A† Go Back 2 − i 2 2 − i 2 Full Screen Above, we introduced an inverse map (2.30) as an inverse transformation from a vector b to a vector a. When the map is represented by a matrix A, an inverse Close map will be represented by its inverse matrix A−1. If such a matrix A−1 exists, −1 −1 then A is said to be invertible. As a result of the definition, A A = AA = I Quit where I denotes the unit matrix:   1 0 ... 0   0 1 ... 0 I = . . . . (2.33) . . .. . . . . 0 0 ... 1

Home Page How to find an inverse matrix? There is no swift method to do so for general n × n matrix and developing efficient matrix inversion algorithms is an active area Title Page of research. Currently, the best algorithm for matrix inversion has computational complexity O(n2.373). This is better than the method usually applied for matri- Contents ces of small dimensions called Gaussian elimination, which has O(n3). Let’s illustrate how Gaussian elimination works on a 2×2 system. JJ II

Example 2.6. Solve the following 2×2 linear system of equations and find the J I corresponding inverse matrix using Gaussian elimination: Page 44 of 128

x1 + 2x2 = 4 Go Back 2x1 + x2 = 1

Solution: The given system can be written as Full Screen

1 2 x  4 Close 1 = 2 1 x2 1 A x y Quit Let’s augment the matrix A by y to obtain the augmented matrix:

 1 2 4  2 1 1

Gaussian elimination allows one to obtain upper (or lower) triagonal or diagonal matrix through a succession of transformations with rows and columns that don’t change the solution of the system. The allowed transformations are: i) swapping rows (or columns); Home Page ii) multiplication of a row (or a column) by a non-zero scalar; iii) adding a scalar multiple of any row (or column) to another row (or column). Title Page For our system, let’s multiply the first row by -2 and add it to the second row to get the upper triangular matrix Contents   1 2 4 JJ II 0 −3 −6 J I Then, let’s multiply the second row by 2/3 and add to the first row to get the diagonal matrix Page 45 of 128  1 0 0  0 −3 −6 Go Back Therefore, the solution is given by Full Screen

1 · x1 + 0 · x2 = 0 Close 0 · x1 − 3 · x2 = −6 Quit x  0 =⇒ 1 = x2 2 Now, let’s apply the same Gaussian elimination approach to obtain the inverse matrix A−1. To this end, we recall that AA−1 = A−1A = I. Therefore, if we convert the augmented matrix with I on the right to the one with I on the left, we −1 will obtain A . Let’s do that for the above example denoting row 1 as r1 and row 2 as r2: !     2 1 2 1 2 1 0 r =−2r +r 1 2 1 0 r1= r2+r1 1 0 − −−−−−−−−→2 1 2 −−−−−−−→3 3 3 2 1 0 1 0 −3 −2 1 0 −3 −2 1 Home Page ! 1 1 2   r2=− r2 1 3 1 0 − 3 3 −1 1 −2 −−−−−−→ 2 1 =⇒ A = − Title Page 0 1 3 − 3 3 −2 1

The obtained result can be compared with the known mnemonics for 2×2 systems Contents for which the matrix inverse is computed by swapping the diagonal elements, chang- ing signs of the off-diagonal elements and dividing by the matrix determinant: JJ II 1  d −b 1  1 −2 A−1 = = − det A −c a 3 −2 1 J I

Page 46 of 128

In addition to Gaussian elimination as a way to solve small systems of linear Go Back equations, we can also mention the Cramer’s rule. Definition 2.10. Cramer’s rule states that a unique solution to Ax = b can be Full Screen found as det (A ) Close x = i (2.34) i det (A) Quit where the matrices Ai are obtained by substituting the i-s column in the original matrix A by the vector b. Example 2.7. Solve the following system of equations using the Cramer’s rule: ( 3x1 + 5x2 = −7

x1 + 4x2 = −14

Solution. In matrix form the given system can be written as Home Page 3 5 x   −7  Ax = b ⇐⇒ 1 = 1 4 x2 −14 Title Page

3 5 det (A) = = 3 × 4 − 5 × 1 = 7 Contents 1 4

−7 5 42 JJ II det (A1) = = −7 × 4 − (−14) × 5 = 42 =⇒ x1 = = 6 −14 4 7 J I 3 −7 −35 det (A2) = = 3 × (−14) − (−7) × 1 = −35 =⇒ x2 = = −5 1 −14 7 Page 47 of 128

Go Back Two important invariants of a square matrix are trace (tr) and determinant (det). These two are scalar quantities meaning that they do not change upon Full Screen transformation of a basis set in which a matrix is defined. Close Definition 2.11. The trace of a matrix A is the sum of the matrix diagonal elements: X Quit tr(A) = aii (2.35) i There are two main ways to define the matrix determinant. The first is the Leibniz rule involving the Levi-Civita symbol and Einstein summation: Definition 2.12. The determinant of a matrix A is defined as: det (A) =  ai1 ai2 ...ain = i1i2...in a1 a2 ...an (2.36) i1i2...in 1 2 n i1 i2 in where subscript indices are used to denote columns and superscript indices to de- note rows (or vice versa).

Home Page For example, for a 2×2 matrix: Title Page 2 −2 det A = = 2 · 3 − (−2) · 1 = 8 1 3 Contents

The second (alternative) is the Laplace rule involving the matrix minors: JJ II Definition 2.13. The determinant of a matrix A is defined as: J I X i+j det (A) = (−1) aijMij (2.37) Page 48 of 128 j where the minor Mij is the determinant of (n − 1) × (n − 1) matrix obtained by Go Back crossing out row i and column j of matrix A. Full Screen For example, for a 3×3 matrix: Close 1 −2 2 3 2 2 2 2 3 det A = 2 3 2 = 1 · −(−2) · +2 · = 1 · 1 + 2 · 2 + 2 · 2 = 9 1 1 0 1 0 1 Quit 0 1 1 | {z } | {z } | {z } M11 M12 M13 Geometrically, the matrix determinant equals the volume of the parallelepiped spanned by the column vectors of the matrix A.

Several properties of the matrix determinants are listed here for reference:

1. The determinant is invariant under transposition:

det (AT ) = det (A) (2.38) Home Page

Title Page 2. The determinant of the matrix inverse

1 Contents det (A−1) = (2.39) det (A) JJ II 3. The matrix A is invertible if det (A) 6= 0 (see Eq.( 2.39)). J I 4. The determinant of a product of matrices equals the product of their deter- minants: Page 49 of 128 det (AB) = det (A) det (B) (2.40) Go Back 5. The determinant of a matrix is invariant under a change of basis set: Full Screen Eq.(2.40) Eq.(2.39) det (A0) = det (T AT −1) = det (T ) det (A) det (T −1) = det (A) (2.41) Close

Quit 2.5. Eigenvalue Problem and Matrix Diagonalization In the above, we have mentioned several times that the simplest representation of a matrix is diagonal. For example, the following two matrices ! 2 0 1 19 9 A = ,A0 = 4 0 1 10 4 11 have the same traces, determinants and solutions to Ax = y, but the matrix A is obviously much easier to work with. In the present section we will demonstrate Home Page an important result of linear algebra that the diagonal matrix A can be obtained from A0 via so-called similarity transformation Title Page −1 0 A = T A T (2.42) Contents where T is the matrix corresponding to a transformation to the so-called eigen- basis. The eigenbasis is obtained by solving the eigenvalue problem as alluded JJ II to below. Note that in general the transformation T of a basis set can be anything we want, but if we aim to get the diagonal matrix representation like the matrix J I A, then T must represent the eigenbasis. Page 50 of 128

The eigenvalue equation is defined as a linear map A acting on a vector v to Go Back give the same vector multiplied by a scalar λ: Av = λv (2.43) Full Screen

Vectors v that satisfy this property are called eigenvectors and the corresponding Close scalars λ are called eigenvalues. A set of all eigenvalues for the matrix A is called the eigenvalue spectrum. To solve this equation, we can re-write it as Quit

(A − λI)v = 0 (2.44) This equation has a solution only if the corresponding matrix determinant is zero:

det (A − λI) = 0 (2.45) This determinant written explicitly can be represented as a polynomial

n n−1 det (A − λI) = p(λ) = (−λ) + (−λ) tr(A) + ... + det (A) (2.46) which is called the characteristic polynomial for the eigenvalue Eq.(2.43). Home Page Example 2.8. Find the eigenvalues and eigenvectors for the following matrix Title Page  0 1  A = −2 −3 Contents

Solution. Let’s solve the characteristic equation for A: JJ II

0 − λ 1 2 det (A − λI) = = −λ(−3 − λ) + 2 = λ + 3λ + 2 = 0 J I −2 −3 − λ Page 51 of 128 =⇒ λ1,2 = −1; −2 Let’s find the eigenvectors corresponding the above eigenvalues: Go Back

Full Screen Av1 = λ1v1 =⇒ (A − λ1)v1 = 0       0 − λ1 1 1 1 v11 Close =⇒ v1 = = 0 =⇒ v11 = −v12 −2 −3 − λ1 −2 −2 v12 Quit  1  =⇒ v = c 1 1 −1 where c1 is an arbitrary constant. Doing the same for the second eigenvalue λ2 = −2 we get  1  v = c 2 2 −2 where c2 is an arbitrary constant. If solved in MATLAB, the constants c1 and c2 will be chosen in such a way that the norm of each eigenvector is unity, viz. in the case of λ , v = ( √1 , − √1 )T . 1 1 2 2 Home Page

Let’s now demonstrate the point with which we have started this section, viz. that Title Page the non-diagonal matrix A can be diagonalized using Eq.(2.42) if T is composed of eigenvectors of A. As a result, we will get the diagonal matrix with eigenvalues Contents of A being on the main diagonal. Thus, we want to show that JJ II   λ1 0 ... 0  0 λ2 ... 0  J I −1   T AT = D =  . . . .  (2.47)  . . .. .  Page 52 of 128  . . .  0 0 . . . λn Go Back if T = (v1, ..., vn). Notice that this similarity transformation leads to a decoupled (diagonal) system. Let’s demonstrate this on the 2×2 matrix A from Ex.(2.8). We Full Screen found that  1 1  Close T = (v , v ) = =⇒ 1 2 −1 −2 Quit −2 −1  0 1   1 1  −1 0  λ 0  T −1AT = − = = 1 1 1 −2 −3 −1 −2 0 −2 0 λ2 There is an interesting theorem proved by S. Gershgorin in 1931 that enables evaluation of the eigenvalues of a matrix without explicitly solving the eigenvalue problem. Theorem 2.14. (Gershgorin’s disk theorem) Every eigenvalue λ of matrix Ann satisfies the following inequality: X | λ − Aii |≤ | Aij |= ri (2.48) j6=i Home Page where ri is called the radius of the Gershgorin’s disc. In other words, every eigen- Title Page value of A lies within the union of the Gershgorin’s disks D(aii, ri) with the centers at aii and radii ri. Contents

Example 2.9. Estimate the eigenvalues of the following matrix using the Gersh- JJ II gorin’s disc theorem:   6 0.5 0.1 J I A = −1 7 −0.1 1 0 3 Page 53 of 128 Solution. When computed exactly, the three eigenvalues of A are ≈ 6.52 ± 0.52i, Go Back and 2.96. The Gershgorin’s disc theorem gives the following estimates: λ1 = D1(a11 = 6, r1 = 0.6) = 6 ± 0.6, λ2 = D2(a22 = 7, r2 = 1.1) = 7 ± 1.1, and Full Screen λ3 = D3(a33 = 3, r3 = 1) = 3 ± 1. There is exactly one eigenvalue in disk D3 (which is disjoint from the union D1 ∪ D2) and two eigenvalues in D1 ∪ D2. Close Moreover, if the theorem is applied to AT , then a tighter bound can be obtained for λ3 = D3(a33 = 3, r3 = 0.2) = 3 ± 0.2. Clearly, diagonal matrices will have their Quit eigenvalues on the diagonal and the disks will have radii zero. Also, the smaller off-diagonal entries, the smaller disk’s radii. REMARK: In the next section we will discuss that if a matrix is Hermitian (symmetric), it is categorically diagonalizable. Most matrices appearing in phys- ical applications are of this type, thus allowing diagonalizability and decoupling. However, it can be shown that if a matrix has a degenerate spectrum (i.e., has at least two equal eigenvalues), then this matrix is not diagonalizable. Never- theless, it is possible to construct a matrix that will be the closest to a diagonal Home Page representation, which is called the Jordan form. We will not discuss it here. Title Page

Contents 2.6. Hermiticity and Unitarity In this section we will briefly discuss two important classes of linear maps - Her- JJ II mitian and unitary. Why these types of maps (matrices) are so important? One of the axioms of quantum mechanics is that any physical observable (mea- J I surable) is represented by a corresponding Hermitian linear map. Each single measurement must give an eigenvalue of the Hermitian opera- Page 54 of 128 tor. The main property of unitary maps is that they preserve the scalar product Go Back between vectors (i.e., their norms).

Definition 2.15. A linear map Hˆ : V → V of a complex (real) scalar product is Full Screen called a Hermitian (symmetric) map if Close hHˆ v, wi = hv, Hˆ wi (2.49) Quit The corresponding matrix representation is called a Hermitian matrix. Using the component representation of vectors along with Einstein summation in an m m k orthonormal basis hei, eji = δij as (Hv) = Hk v , the above definition (2.52) can be written as m k j k l j Hk v δmjw = v δklHjw (2.50) m † m Recalling that for the adjoint matrices Hk = (H )k and multiplying both sides by δik we get ik † m † i i δ (H )k δmj = (H )j = Hj (2.51) Home Page T i i The same procedure applied to a symmetric matrix leads to (H )j = Hj. There- fore, we can write an alternative definition Title Page Definition 2.16. A linear map Hˆ : V → V of a complex (real) scalar product is called a Hermitian (symmetric) map if Contents

† i i T i i (H )j = Hj ((H )j = Hj) (2.52) JJ II

J I Example 2.10. The following matrices are Hermitian (A) or symmetric (B): Page 55 of 128  −1 1 + i 3 1 1 4  Go Back A = 1 − i 2 i ,B = 1 2 −2 3 −i 1 4 −2 3 Full Screen

Close The most important properties of Hermitian (symmetric) matrices without proofs are summarized below: Quit 1. Hermitian (symmetric) matrices are categorically diagonalizable. Such diag- onalization can be done by a unitary matrix via similarity transformation. 2. Eigenvalues of Hermitian (symmetric) matrices are always real. 3. The determinant of a Hermitian (symmetric) matrix is real.

Definition 2.17. A linear map Uˆ : V → V of a complex (real) scalar product is called a unitary (orthogonal) map if Home Page

hUˆv, Uˆwi = hv, wi (2.53) Title Page

Contents Similar to Hermitian maps, in an orthonormal basis the scalar product between two vectors can be written as hv, wi = uiδ wj. Therefore, the definition (2.53) ij JJ II can be also written as m k j l k l Uk v δmjUl w = v δklw (2.54) J I m † m Recalling that for the adjoint matrices Uk = (U )k and multiplying both sides by δik we get Page 56 of 128 † i j i (U )jU = δl (2.55) l Go Back T i i The same procedure applied to an orthogonal matrix leads to (O )j = Oj. In the non-covariant index representation these two conditions are written as Full Screen

† T UijUjl = δil,OijOjl = δil (2.56) Close

† T where Uij = Uji and Oij = Oji, or in matrix form Quit

† † T T U U = I = UU ,O O = I = OO (2.57) Vector rotations, reflections and their combinations are all orthogonal transforma- tions represented by orthogonal matrices.

Example 2.11. Show that the following matrix describing a rotation of vectors by the angle φ is orthogonal:

cos φ − sin φ R(φ) = sin φ cos φ Home Page Solution.To show this, we need to demonstrate (2.57). Let’s find the inverse of the matrix R(φ) that describes the rotation by (−φ) noting that cos (−φ) = cos φ Title Page and sin (−φ) = − sin φ: Contents  cos φ sin φ R−1(φ) = R(−φ) = = RT (φ) − sin φ cos φ JJ II

T T Therefore, R R = RR = I. J I

Page 57 of 128 Two main properties of unitary matrices are as follows: Go Back

iφj Full Screen 1. The eigenvalues of a unitary matrix are complex numbers λj = e with unit modulus (| λ |= 1) where real φj ∈ [0, 2π). Eigenvalues of orthogonal matrices are ±1. Close

2. The determinant of a unitary matrix is a complex number with unit modulus. Quit The determinant of an orthogonal matrix is ±1. 2.7. Numerical Aspects of Linear Algebra It was mentioned above that the Gaussian elimination has the computational com- plexity of O(n3) that makes the approach impractical for large n. A key problem with direct approaches such as Gaussian elimination is that they typically do not use any properties of matrices to find the solution faster. If a matrix is banded, the Gaussian elimination can be modified to reduce the number of steps as most entries are zeros. In general, however, it would require the same number of steps for both dense and sparse matrices. Home Page

One class of methods to solve large systems of equations more efficiently is called Title Page iterative methods. The main idea is to start with some approximation to the so- lution (initial guess), and then iteratively improve it. The process is stopped when Contents the difference between successive approximations meets some threshold value. Let’s illustrate how this works on the most basic version of iterative methods JJ II called the Jacobi method. Here, we will use a simple 2 × 2 example that can be easily generalized to larger systems: J I ( 2x + x = 6 Page 58 of 128 1 2 (2.58) x1 + 2x2 = 6 Go Back (0) (0) The exact solution is x1 = x2 = 2. Let’s start with an initial guess x1 = x2 = 0 and use the following formulas to compute successive approximations: Full Screen

 (n−1) (n) 6−x2 Close x1 = 2 (n−1) (2.59) (n) 6−x1 x2 = 2 Quit Table below lists the obtained results for the first 7 iterations: n = 1 2 3 4 5 6 7 x1 = x2 = 3 1.5 2.25 1.875 2.0625 1.9688 2.0156

2.8. Case Study: Flow Graphs in Multicomponent Flash Separation The flow graphs (diagrams, networks) are useful in analysis of a variety of Home Page processes described by systems of linear algebraic or differential equations. The nodes of a graph represent the variables and the connecting branches (arrows) Title Page represent the coefficients relating the variables. Depending on the context a flow graph could represent a chemical reaction network, distillation column, electrical Contents network, traffic or data flow, process control, etc. For example, a webgraph is used to compute the PageRank of the WWW pages (Google search) ranking them according to the importance of website pages. JJ II J I In chemical engineering the most common source of systems of linear algebraic equa- Page 59 of 128 tions is mass and energy balances. Here, we will consider a simple 3-component flash Go Back separation system as illustrated by the pro- cess flow sheet on the left. For the sake Full Screen of example, the feed could contain hexane, heptane and octane. In your homework you Close will also consider similar systems that in addition to a separator involve chemical re- Quit actions between components occurring in a reactor plus recycle. To make the system even more realistic a heat exchanger can be added to account for heat effects as a result of chemical reactions. In this case mass and energy balance equations need to be coupled when building a mathematical model.

i i Here, F stands for the inlet mass flow rate of stream i (e.g., in kg/hr) and nj is the mass fraction of species j in stream i. The goal is to compute the mass flow rate for each outlet stream that can be denoted as x , x , x . Let’s write down the 1 2 3 Home Page corresponding mass balances:

 Title Page x + x + x = 10        1 2 3 1 1 1 x1 10 0.9x1 + 0.1x2 + 0.2x3 = 0.7 · 10 ⇐⇒  0.9 0.1 0.2 x2 =  7  Contents  0.06 0.8 0.1 x 2 0.06x1 + 0.8x2 + 0.1x3 = 0.2 · 10 3 JJ II Let’s apply the Cramer’s rule (2.34) to solve it. J I det (A ) 3.6 x = 1 = = 7.4074 1 det (A) 0.486 Page 60 of 128

det (A ) 0.9 Go Back x = 2 = = 1.8519 2 det (A) 0.486 Full Screen det (A3) 0.36 x3 = = = 0.7407 det (A) 0.486 Close When plugging into the first equation of the system we get 7.4074 + 1.8519 + Quit 0.7407 = 10. In the homework, you will also practice the Gaussian elimination, Jacobi and Gauss-Seidel methods. 2.9. Case Study: Protein Phosphorylation Proteins carry out a variety of biological functions such as signaling and trans- port. To perform a function, proteins need to be activated. One of the common ways to change protein activity is through attaching a phosphate group, a pro- cess called phosphorylation. This attachment is catalyzed by enzymes called kinases. A sequence of protein phosphorylation events can be represented as a chemical reaction network: Home Page A −→k1 B −→k2 C −→k3 A (2.60) Title Page where the letters represent the same protein differing by the number of phosphate groups attached. Let’s apply the linear algebra formalism to analyze this network. Contents First, let’s write down the stoichiometry matrix:

−1 0 1  JJ II S =  1 −1 0  (2.61) 0 1 −1 J I

Page 61 of 128 Let’s now use it to express the rates of change of the protein concentrations (fluxes) in matrix form: c  k 0 0  c  Go Back d A 1 A cB = S  0 k2 0  cB (2.62) dt Full Screen cC 0 0 k3 cC assuming the first-order reactions and isothermal conditions (ki do not depend on Close temperature). Let k1 = k3 = 1 and k2 = 3. Eq.(2.62) is equivalent to Ax = b if b = dc/dt, x = c and A = SK. We learned before that Ax = b has solutions if Quit and only if det (A) = 0. We get

−1 0 1

det (A) = det (SK) = 1 −3 0 = 0 (2.63)

0 3 −1 Let’s now solve the eigenvalue problem for this system:

det (SK − λI) = (−k1 − λ)(−k2 − λ)(−k3 − λ) + k1k2k3 = Home Page 2 −λ(λ + (k1 + k2 + k3)λ + k1k2 + k1k3 + k2k3) Title Page This leads to three solutions (steady states): Contents λ1 = 0 JJ II p 2 −(k1 + k2 + k3) ± (k1 + k2 + k3) − 4(k1k2 + k1k3 + k2k3) 5 3 λ2,3 = = − ± i 2 2 2 J I We will see in the following sections that existence of complex solutions means os- cillatory behavior (think of the Euler formula). But since the real part is negative Page 62 of 128 (-5/2), the oscillations are decaying to a steady state. We will also consider re- actions beyond first-order such as autocatalytic reactions that result in non-linear Go Back dynamics and will employ linearization around the steady states to analyze their stability. Full Screen

Close

Quit Additional Literature

1. G. Strang, "Computational Science and Engineering" Wellesley-Cambridge Press 2007.

Home Page

Title Page

Contents

JJ II

J I

Page 63 of 128

Go Back

Full Screen

Close

Quit 3. Fourier Calculus

You learned in calculus that the Taylor expansion allows representation of a func- tion locally (at a particular point) through a series of real or complex expansion coefficients. Fourier calculus, on the other hand, allows representation of a pretty much any function globally (on an interval) as a (infinite) sum of simpler building blocks. These building blocks (basis functions or vectors in the language of linear algebra) are complex exponentials (or sin and cos functions related to complex exponentials via the Euler formula). In high school and later you spent quite Home Page some time studying the properties of trigonometric functions. The Fourier theo- rem demonstrates that these harmonic functions are essential. In the language of Title Page mathematical physics it can be said that any periodic function is equivalent to a set of independent harmonic oscillators serving as simple building blocks to con- Contents struct a function. It may seem that the Fourier transform is a nice mathematical trick to simplify the solution to a problem. But the remarkable fact is that the JJ II human inner ear actually performs a Fourier transform of the incoming sound. J I 00 0 When studying non-homogeneous ODEs such as y + py + qy = f(t) you learn Page 64 of 128 that they are solved particularly easy if the input function f(t) is exp, sin or cos. Fourier analysis shows that since f(t) can be approximated by a sum of such Go Back harmonic functions, then due to the superposition principle the solution (response) y(t) to an ODE with f(t) can be represented as a linear combination of solutions Full Screen (eigensolutions) to more simple ODEs. Close When generalized from a function defined on a finite interval to a function defined on the entire real axis, Fourier series becomes a Fourier transform. The Laplace Quit transform technique, as alluded to below, is a special case of the Fourier transform inheriting all of its main properties.

Remark: Jean-Baptiste Joseph Fourier developed his theory when considering a classical chemical engineering problem of heat propagation in solid bodies (the heat equation). Fourier analysis can be considered as one of the greatest results in mathematics being now used in a variety of fields including the solution of DEQs, compression of acoustic and visual data, imaging. To make the Fourier decompo- sition much faster, in practice, Fourier analysis is applied in the framework of the Home Page so-called fast Fourier Transform (FFT) algorithm. This changes the complexity of the algorithm from O(N 2) (brute-force application of the definition) to O(N log N) Title Page (in FFT). Contents

JJ II 3.1. Fourier Series Given a complex-valued function f(x) defined on a finite interval I of width 2L, J I the representation of f(x) as a sum Page 65 of 128

1 X ikx f(x) = f˜ e (3.1) Go Back 2L k k Full Screen ˜ is called the complex Fourier series of f(x). Here, fk are Fourier coefficients and exp(ikx) are Fourier modes. Fourier coefficients give weights (amplitudes) of Close πn each mode in the overall Fourier expansion. The sum runs over all values k = L where n ∈ Z. This choice of k is dictated by periodic boundary conditions so Quit that ei2Lk = ei2πn = 1. Since Fourier analysis is often applied in connection with wave-like phenomena, k is often called the wave number, its inverse λ = 2π/k - the wave length, and n/2L - the harmonic frequency.

It can be shown that Fourier coefficients can be computed as follows Z ˜ −ikx fk = f(x)e dx (3.2) I

It follows from the Euler formula e±ix = cos(x) ± i sin(x) that Home Page

eix + e−ix eix − e−ix Title Page cos(x) = , sin(x) = (3.3) 2 2i Contents Therefore, Fourier series can be alternatively written as trigonometric Fourier series ∞ JJ II a0 X πnx πnx f(x) = + [a cos( ) + b sin( )] (3.4) 2 n L n L n=1 J I with coefficients computed as Page 66 of 128

Go Back 1 Z L a0 = f(x)dx L 0 Full Screen 2 Z L πnx an = f(x) cos( )dx (3.5) Close L 0 L Z L 2 πnx Quit bn = f(x) sin( )dx L 0 L It can be shown by applying the above formulas that if the function is even, i.e., f(−x) = f(x) being symmetric with respect to the y-axis such as cos x or x2, ∞ Fourier series will only contain coefficients in front of cos functions ({bn}n=1 = 0). Likewise, if the function is odd, i.e., f(−x) = −f(x) being symmetric with respect to the origin of coordinates such as sin x or x3, Fourier series will only ∞ contain sin terms ({an}n=1 = 0). This allows further simplification of Fourier series calculations.

Example 3.1. Find the Fourier series for the following "sawtooth" function de- Home Page fined on the interval I = [0,1] (see Fig.1(a)): Title Page ( −x, x ∈ (0, 1 ) 2 Contents f(x) = 1 1 − x, x ∈ ( 2 , 1) JJ II

Solution: According to the complex exponential formula (3.2), the Fourier coef- J I ficients are computed as follows: Page 67 of 128 Z 1 Z 1/2 Z 1 ˜ f0 = f(x)dx = (−x)dx + (1 − x)dx = 0 0 0 1/2 Go Back

Full Screen Z 1/2 Z 1 f˜ (k 6= 0) = (−x)e−ikxdx + (1 − x)e−ikxdx = k Close 0 1/2 Z 1/2 Z 1 Z 1 − xe−ikxdx − xe−ikxdx + e−ikxdx Quit 0 1/2 1/2 Using integration by parts we arrive at e−ik/2 f˜ (k 6= 0) = k ik Thus, according to Eq.(3.1) f(x) can be represented as   −ik/2 " i(2x−1) −i(2x−1) # 1 X e ikx 1 h i(x− 1 ) −i(x− 1 )i e e f(x) = e = e 2 − e 2 + − + ... 2 ik 2i  2 2  k Home Page and using Eq.(3.3) the final Fourier series can be re-written as ∞ Title Page 1 1 1 3 X sin [n(x − 1 )] f(x) = sin (x − ) + sin (2x − 1) + sin (3x − ) + ... = 2 2 2 3 2 n n=1 Contents Since the given f(x) is an odd function, its Fourier series contains contributions only from sine functions. JJ II

J I It is interesting to note that although the function f(x) gets better approximated over the entire interval with increasing the Page 68 of 128 number of Fourier modes, this is not the case fo the boundary regions. It is seen in Fig.1 that even using 20 Fourier harmonics Go Back in the representation of f(x) leads to a sizable "overshooting" of the function near the jump discontinuities (the so-called Gibbs Full Screen phenomenon). It can be shown that the inclusion of more Close terms will not help as the height of the excess peak will remain at a level of O(10%) of the function value. This is the reason why, for example, sharp edges Quit in visual data appear as blurred (see the picture on the right), and additional smoothing techniques need to be employed in practice. Home Page

Title Page

Contents

JJ II

J I

Page 69 of 128 (a) Sawtooth function. (b) Square function.

Figure 1: Fourier representation of the sawtooth and square functions using dif- Go Back ferent number of Fourier modes (with n = 1 and 20 in both cases). Full Screen

Close

Quit 3.2. Fourier Transform, Convolution and Correlation To generalize the Fourier representation of a function defined on the finite 2L in- terval to the one defined on the entire axis, the discrete Fourier sum is transformed to a Riemann sum that becomes an integral in the limit L → ∞:

1 X 1 X 1 Z ∞ (...) = δk (...) → (...)dk (3.6) 2L 2π 2π k n −∞ Home Page Therefore, the Fourier expansion can be written as Title Page 1 Z ∞ f(x) = f˜(k)eikxdk (3.7) 2π −∞ Contents called the inverse Fourier transform, while JJ II Z ∞ ˜ −ikx f(k) = f(x)e dx (3.8) J I −∞ is called the Fourier transform of the function f(x). The alternative short-hand Page 70 of 128 notations Ff(x) = f˜(k) and F −1f˜(k) = f(x) are often used. If the function at hand is multi-variable, a simple generalization from a scalar to a vector (x → x Go Back and k → k) is made with Fourier integrals becoming multi-dimensional. It is important to note that the alternative convention can be introduced by altering Full Screen the signs in the exponents when convenient. Close

For instance, often in applications the independent variable x is time t. In this Quit case the above integrals can be written as 1 Z ∞ f(t) = f˜(ω)e−iωtdω 2π −∞ Z ∞ (3.9) f˜(ω) = f(t)eiωtdt −∞ where ω is a frequency-like variable and the convention for the signs in the expo- nents is changed. Home Page Example 3.2. Find the Fourier transform for the exponential function f(t) = e−a|t| (with a > 0). Title Page

Contents Solution: Z ∞ Z 0 Z ∞ JJ II f˜(ω) = e−a|t|e−iωtdt = eate−iωtdt + e−ate−iωtdt = −∞ −∞ 0 1 1 2a J I = + = 2 2 a − iω a + iω a + ω Page 71 of 128 When plotted, it can be noticed how the sharp peak in the original function gets smoothed out via the Fourier transform. Go Back

The following properties of the Fourier transform are important in applications: Full Screen

Close • Derivative formula: the Fourier transform converts the derivative of a function, dxf(x), into multiplicative factors: Quit

(Fdxf)(k) = ik(Ff)(k), dk(Ff)(k) = −i(Fxf)(k) (3.10) Sequential application of this formula leads to expressions for higher-order derivatives. Since derivatives are linear operators, this allows one to simplify a differential equation (e.g., in time domain t) by converting it to a simpler algebraic analogue (in frequency domain ω). Then solving the algebraic equation and through an inverse Fourier transform (back to the time domain) obtaining the solution to the original equation. • Fourier convolution theorem - the Fourier transformation turns a con- volution (complicated) into a multiplication (simple) Home Page

F(f ∗ g)(k) = Ff(k)Fg(k) (3.11) Title Page

and vice versa Contents 1 F(fg)(k) = (Ff ∗ Fg)(k) (3.12) 2π JJ II Here, the convolution of two functions f and g is defined as Z J I (f ∗ g)(x) = f(x − y)g(y)dy (3.13) Page 72 of 128 The convolution operation is commutative, f ∗ g = g ∗ f. Go Back • Exponential shift formula: Full Screen F(eiaxf(x)) = f˜(k − a); F −1(e−iakf˜(k)) = f(x − a) (3.14)

Close Before applying the above formulas to specific examples, let’s discuss the physical Quit meaning of convolution. Here, we will consider three examples - playing drums, radioactive dumping and spectral line broadening. 1) Playing drums. If you hit the membrane of a drum by a drum stick for the first time, it produces a vibration. When you do that the second time, the vibration from the first strike will already decay somewhat. Therefore, you will hear a sound which is compounded (convolved) of two signals - the decayed first strike and the new one. If you keep beating the drum, you will hear a sound consisting of the current (last) beating and a sum of previous decayed impacts. In mathematical terms, if g(t) is the force exerted on a drum at time t, then g(t)dt will be the force during the impact time dt. Let’s suppose that the drum decaying function is f(u) where u is the elapsed time after the strike. So, if we listen to the drum vibration Home Page at a point in time τ after the hit at t, then the decayed signal will have the value f(u = τ − t). Thus, the compound effect of two strikes will be f(τ − t)g(t)dt and Title Page to obtain the signal from all impacts we need to integrate this to get Eq.(3.13). Contents

2) Radioactive dumping. Radioactive decay is the loss of activity of a radioactive JJ II compound with time. It is also called exponential decay as the decaying function is exponential. To be precise, the change in activity A (decay rate) is proportional J I to the current activity: dA(t) = −λA(t) (3.15) Page 73 of 128 dt where λ is the decay constant being characteristic of a material, while the solution Go Back −λt is given by A(t) = A0e with A0 being the initial activity of a material. For example, the half-life of the 14C isotope (time period when half of the radioactive Full Screen material has decayed into another nuclide) is 5,730 years. This property is used in radiocarbon dating to determine the age of an organic material since after its Close death the isotope ratio 14C/12C will keep decreasing according to the radioactive decay of 14C. Quit Radioactive dumping is the storage of radioactive material. The question we ask here is what activity the storage will have at some time τ if we keep dumping the material at some rate r(t) given that the material activity A(t) decays exponen- tially. Again, the signal (radioactive activity) that will be measured at some point −λt τ will be a convolution of r(t) and A0e .

3) Spectral line broadening is another example of convolution between functions. In spectroscopy, the spectrum consists of different broadening components (e.g., Home Page thermal Doppler effect, collisions) with the measured signal being a convolution of a true signal with broadening functions. To enhance spectral resolution, the Title Page inverse Fourier transform (deconvolution) using Eq.(3.11) can be applied to obtain the deconvolved signal as Contents g(x) = F −1[F(f ∗ g)(k)/Ff(k)] (3.16) JJ II This can be also viewed as a case of a more general operation with signals called filtering. Specifically, in f ∗ g = h where f is the input signal and h is the J I output signal, g can be interpreted as the digital filter (also called the convolution kernel). For example, such digital filters can remove undesired frequency contents Page 74 of 128 of signals. Go Back It can be noticed from the above examples that convolution should be conceptually related to how two functions (signals) are correlated in time. The correlation of Full Screen two functions is defined, similar to convolution Eq.(3.13), as Close Z ∞ Cf,g(t) = f(t + τ)g(τ)dτ (3.17) −∞ Quit The correlation function measures the similarity between f and g after some lag time t. The Fourier transform of the correlation function is given by Z ∞ −iωt Cf,g(ω) = Cf,g(t)e dt (3.18) −∞ and therefore ˜ Cf,g(ω) = f(ω)˜g(−ω) (3.19)

The correlation of the function with itself is called autocorrelation (Cf,f ) and shows how the original function is correlated with its delayed copy as a function Home Page of delay. Thus, Eq.(3.19) becomes Title Page ˜ 2 Cf,f (ω) =| f(ω) | (3.20) Contents known as the Wiener-Khinchin theorem. Remarkably, the Fourier transform of autocorrelation function Cf,f (t) gives the power spectrum, i.e., distribution JJ II of energy density as a function of frequency. J I One important application of Fourier transform is to solve differential equations. Page 75 of 128 Let’s demonstrate how the method works by solving an ODE and a PDE.

Go Back Example 3.3. Solve the following second-order ODE using Fourier transform: 00 −z + z = f(x). Full Screen

Close Solution: Let’s use Eq.(3.10) and linearity of Fourier transform operation, F(f + g) = Ff + Fg, to convert the given ODE to an algebraic equation: Quit k2z˜(k) +z ˜(k) = f˜(k) with solution f˜(k) z˜(k) = 1 + k2 To obtain the original function z(x) we need to apply the inverse Fourier transform. To do this, let’s use the Fourier convolution theorem:

 1  1 1 Z ∞ z(x) = f(x) ∗ F −1 = f(x) ∗ e−|y| = e−|x−y|f(y)dy 1 + k2 2 2 −∞ Home Page

Title Page Example 3.4. Solve the following transport equation using Fourier transform: ∂u ∂2u Contents ∂t + c ∂x2 = 0 ⇔ ut + cuxx = 0 with x ∈ (−∞, ∞), t > 0 and u(x, t = 0) = f(x). JJ II Solution: By applying Fourier transform in the x-variable, the problem is con- verted to J I u˜ + ikcu˜ = 0, u˜(k, 0) = f˜(k) t Page 76 of 128 The solution to this first-order ODE is Go Back u˜(k, t) = e−icktf˜(k) Full Screen By applying the inverse Fourier transform and exponential shift formula we get Close u(x, t) = f(x − ct) Quit 3.3. Laplace Transform and Transfer Function

The Laplace transform is defined analogously to the Fourier transform as Z ∞ Lf(t) = f˜(s) = e−stf(t)dt (3.21) 0 Here, the complex variable s is a frequency-like parameter and t is a time-like Home Page parameter. The Laplace transform inherits a lot of properties of Fourier transform including the main three discussed above - the derivative rule, convolution theorem Title Page and exponential shift formula. The derivative rules for Laplace transform, however, read as (for the first- and second-order derivatives, dt and dtt) Contents

(Ldtf)(s) = sLf(s) − f(0) JJ II 2 (3.22) (Ldttf)(s) = s Lf(s) − sf(0) − dtf(0) J I Thus, if a second-order ODE is to be solved by using Laplace transform, two initial conditions are needed (f(0) and dtf(0)). The Laplace transform technique is well Page 77 of 128 suited to solve initial value problems, but it is typically more difficult to compute its inverse than in the case of Fourier transform. To do so, it is not uncommon to Go Back first apply partial fraction decomposition to be able to convert the fraction into a form available in the Laplace transform table. Full Screen

The Laplace transforms of common functions are easy to derive directly using the Close 1 definition. For example, L(1) = s with s > 0. Thus, by employing the exponential at at 1 Quit shift formula for the exponential function we arrive at L(e · 1) = L(e ) = s−a with s > a. Similarly, by using the backward Euler formula we can find the Laplace transforms for the cosine and sine functions as follows eiat + e−iat 1  1 1  s cos at = ⇒ L(cos at) = + = 2 2 s − ia s + ia s2 + a2 eiat − e−iat 1  1 1  a sin at = ⇒ L(sin at) = − = 2i 2i s − ia s + ia s2 + a2

Home Page Example 3.5. Solve the following equation using Laplace transform: y00 −y = e−t with y(0) = 1, y0(0) = 0. Title Page

Contents Solution: By using Eq.(3.22) and the provided initial conditions we obtain

1 s2 + s + 1 JJ II s2y˜ − s − y˜ = ⇒ y˜ = s + 1 (s + 1)2(s − 1) J I and application of partial fraction decomposition leads to Page 78 of 128 s2 + s + 1 A B C −1/2 1/4 3/4 y˜ = = + + = + + (s + 1)2(s − 1) (s + 1)2 s + 1 s − 1 (s + 1)2 s + 1 s − 1 Go Back

The inverse Laplace transform from s-domain to t-domain gives the solution Full Screen 1 1 3 y = − te−t + e−t + et Close 2 4 4 Quit The first two terms are the transient (particular) solution and the third term is the steady state (solution to the homogeneous equation y00 − y = 0). In chemical engineering the Laplace transform finds a variety of applications be- yond solving differential equations. One of the important applications is the use of the so-called transfer functions together with block diagrams to represent process control systems.

In engineering applications, it is desirable to know the characteristic of a device that determines the output for a range of inputs. This is to say that the intrinsic Home Page properties of a system are deconvoluted from the input. Such a function is called the transfer function of a system (also known as system function). If transfer Title Page functions are known for constituent parts of a larger system, then the correspond- ing combination of transfer functions would describe the behavior of the whole Contents system (represented, e.g., as a block diagram). JJ II Let’s illustrate the concept on the system described by the following ODE: J I 00 0 0 y + py + qy = f(t), y(0) = y (0) = 0 (3.23) Page 79 of 128 which Laplace transform leads to Go Back 1 s2y˜ + psy˜ + qy˜ = f˜(s) ⇒ y˜ = f˜(s) · = f˜(s)˜ω(s) (3.24) s2 + ps + q Full Screen

Its solution is given by the Green’s formula: Close

t Z Quit y(t) = f(t) ∗ ω(t) = f(τ)ω(t − τ)dτ (3.25) 0 Here, ω˜(s) is the transfer function and ω(t) = L−1(˜ω(s)) is the weight function for the given system. These functions are characteristics of the system and not of the input. The Green’s formula (3.25) provides a way to compute the system’s response to any input f(t) once we know ω(t). Since Green’s formula is a special case of convolution, its physical meaning is clear from the examples discussed above.

It is seen from Eq.(3.24) that y˜ =ω ˜(s) if f˜(s) = 1. Since L−1(1) = δ(t) this occurs Home Page when f(t) = δ(t) in the original Eq.(3.23). Therefore, the weight function ω(t) is the response of the system at rest to an impact at t = 0 with unit impulse. Title Page

Now, generalizing the above expressions from the second-order ODE in Eq.(3.23) Contents to a linear ODE of any order, we can write JJ II Lyˆ p(t) = f(t) (3.26) which solution can be obtained by solving J I

LGˆ (t − τ) = δ(t − τ) (3.27) Page 80 of 128 to give Go Back Z t yp(t) = f(τ)G(t − τ)dτ (3.28) Full Screen 0 where G(t − τ) is the Green’s function (same as ω(t − τ) above) and Lˆ is a Close linear differential operator defined as Quit d d2 Lˆ = a + a + a + ... (3.29) 0 1 dt 2 dt2 Thus, instead of directly solving Eq.(3.26), we can solve Eq.(3.27) for the simplest possible input (delta function δ(t − τ)) and then knowing the Green’s function we can readily obtain particular solution yp(t) to any input f(t) through the Green’s formula (3.28). One important advantage of this approach is that it allows the input function f(t) to have discontinuities such as jumps as long as the integral in the Green’s formula exists (i.e., f(t) should be bounded and have finite number of discontinuities). We will demonstrate the application of this approach in the section on differential equations using the example of the second-order ODE (3.23) describing the behavior of a harmonic oscillator. Home Page

Title Page It is instructive to make a comparison between Green’s formula and linear algebra.

Let’s think of the functions y(t) and f(t) in Eq.(3.26) as limits of finite-dimensional Contents vectors y and f. Then, Eq.(3.26) can be written in matrix form as Lˆyp = f and, ˆ−1 ˆ−1 when multiplied from the left by L , leads to y = L f. In the limit, it becomes JJ II Z y(t) = Lˆ−1f(τ)dτ (3.30) J I

When comparing this with the Green’s formula (3.28) we recognize that Page 81 of 128

−1 G(t − τ) = Lˆ (3.31) Go Back

Using this perspective Eq.(3.27) can be associated with Full Screen Z LGˆ (t − τ)ds = δ(t − τ) (3.32) Close

Quit Example 3.6. Find the convolution product t2 ∗ t. Solution: Let’s compute the convolution using two alternative approaches. a) Apply the Green’s formula (3.25):

#t #t Z t Z t τ 3 τ 4 t4 t2 ∗ t = f(τ)g(t − τ)dτ = τ 2(t − τ)dτ = t − = 0 0 3 4 12 0 0 b) Apply the Fourier convolution theorem (3.11) using Laplace transforms:

4 Home Page 2 2 2 1 2 2 −1 2 t L(t ∗ t) = L(t )L(t) = 3 2 = 5 ⇒ t ∗ t = L ( 5 ) = s s s s 12 Title Page

Contents

JJ II

J I

Page 82 of 128

Go Back

Full Screen

Close

Quit 4. Differential Equations 4.1. Classification of Differential Equations Differential equations (DEQs) are equations that involve both functions and their derivatives. Since science and engineering is concerned with how some quantity changes when another quantity is varied (such as time or position), DEQs are ubiquitous. Importantly, DEQs enable quantitative predictions for observed phe- nomena. There are many types of DEQs requiring different strategies to solve Home Page them, either analytically and/or numerically. In this section, we will provide an introduction to the most common types of DEQs and solution methods. Title Page

Two major types of DEQs are ordinary differential equations (ODEs), in which Contents the function depends on a single argument, and partial differential equations (PDEs), in which the function depends on multiple variables. For instance, the JJ II following equations represent the examples of ODE and PDE, correspondingly: J I df(x) = cf(x), c ∈ (4.1) dx R Page 83 of 128

2 2 2 Go Back c ∂xf(x, t) = ∂t f(x, t), c ∈ R (4.2) DEQ is called the n-th order DEQ if the order of the highest derivative is n. Full Screen In practice one rarely encounters DEQs of higher than second order. Moreover, in most cases even non-linear DEQs of higher order can be well represented by Close a system of linear first-order DEQs thus making a tight connection between the methods of linear algebra and DEQs. Quit To solve DEQs means to find the corresponding function such as f(x) in Eq.(4.1). In this particular case it is easy: f(x) = αecx, where α ∈ R. This solution is called the general solution as it defines the whole family of solutions differring by a constant α. To find a unique specific, particular solution we need to specify the boundary and/or initial conditions. For instance, by specifying f(0) = 1 we fix the solution with α = 1, and by specifying f(0) = 0, we fix the solution with α = 0.

Home Page

4.2. Separable ODEs Title Page An important class of DEQs is called separable DEQs as they can be solved by Contents separating variables and integrating after both sides of the equation are multiplied by an appropriate factor (integrating factor). For a general first-order ODE, we can write down JJ II a(x)f 0 + b(x)f = c(x) (4.3) J I and recasting it to the standard form gives Page 84 of 128 b(x) c(x) f 0 + f = =⇒ f 0 + p(x)f = q(x) (4.4) a(x) a(x) Go Back

If we now multiply both sides by the function u(x) (integrating factor) chosen to Full Screen R satisfy u0 = pu giving u = e p(x)dx, then the equation can be written as Close uf 0 + puf = qu ⇐⇒ (uf)0 = qu (4.5) Quit which can be integrated. Example 4.1. Solve the following ODE by separating variables (the method of integrating factors): xf 0 − f = x3 (4.6) Solution. Written in the standard form 1 f 0 − f = x2 (4.7) x leads to Home Page − R 1 dx − ln x 1 u = e x = e = (4.8) x Title Page Multiplying the standard form by u gives 1 Contents ( f)0 = x (4.9) x JJ II and integration leads to the general solution J I 1 x2 x3 ( f) = + c =⇒ f(x) = + cx (4.10) x 2 2 Page 85 of 128

Go Back

4.3. Logistic Equation, Population Growth and Tragedy of Full Screen

the Commons Close The logistic differential equation Quit df = (α − βf)f = αf − βf 2 (4.11) dt is often employed to model population growth. Unlike a simple growth described df by the exponential solution to dt = kf, the growth rate k for the logistic growth is chosen to be k = (α − βf). This takes care of the fact that when the population f increases, the growth rate k should decline as resources become less abundant.

Instead of solving this equation, let’s show a different strategy, viz. how one can analyze it graphically to get qualitative insights. The equation has two critical points f = 0 and f = α obtained from the steady state condition (α − βf)f = 0. 1 2 β Home Page Let’s analyze the system behavior by sketching the so-called phase line with critical points. Depending on the values of f using the original equation we can Title Page draw the changes in f 0 in the corresponding regions. Also, by making the second derivative equal to zero we can find the inflection point (change of the curvature) Contents 00 0 0 0 α as f = αf − 2βff = f (α − 2βf) = 0 =⇒ fi = 2β . The figures below summarizes the obtained results. JJ II

J I

Page 86 of 128

Go Back

Full Screen

Close Figure 2: A phase line with critical and inflection points (on the left) and solution curves (on the right). Quit An interesting modification to the original logistic equation can be made by intro- ducing an extra term h that removes some population: df = (α − βf)f − h (4.12) dt in this case the equation is called the logistic equation with harvesting. As shown in the figure, hmax denotes the maximum rate at which sustainable har- vesting can be achieved. Beyond this point, the population will extinct. This can Home Page be generalized to any "population" in the context of sustainable development from human population growth, natural resources to economics (revenue versus cost). Title Page

An ecologist G. Hardin discussed this issue in his ar- Contents ticle "The Tragedy of the Commons" in 1968. The tragedy is that people seeking their own personal JJ II gain by acting independently would hurt them- selves in the long term since the actions can be J I in contrary to the best interests of the group as a whole (e.g., by depleting their common natural Page 87 of 128 resources). Go Back

Full Screen

Close

Quit 4.4. Systems of Linear First-Order ODEs In the most general form the system of first-order linear ODEs can be written as

 0    f1(t) a11f1 + a12f2 + ... + a1nfn + r1(t)  0     f2(t)   a21f1 + a22f2 + ... + a2nfn + r2(t)   .  =  .  (4.13)  .   .   0    fm(t) am1f1 + am2f2 + ... + amnfn + rn(t) Home Page which in matrix form can be represented as Title Page f 0(t) = Af(t) + r(t) (4.14) Contents If the driving (forcing) term r(t) = 0 (homogeneous equation), this becomes a familiar eigenvalue problem: JJ II

0 f (t) = Af(t) ⇐⇒ b = Ax (4.15) J I suggesting an exponential solution. Due to a superposition principle, the gen- Page 88 of 128 eral solution is given by a linear combination Go Back X λit f(t) = cixie (4.16) i Full Screen where xi are the eigenvectors and λi are the eigenvalues obtained through solving Close the characteristic equation det (A − λI) = 0.

Quit Let us now consider how we can solve Eq.(4.14) when r(t) 6= 0 (inhomogeneous equation). To this end, let’s take as an example the second-order ODE which is equivalent to a 2×2 system of first-order ODEs if we introduce a new variable z = f 0: f 00 + p(t)f 0 + q(t)f = r(t) (4.17) Again, similar to the first-order ODEs, we can first solve the homogeneous version of this equation by using an exponential ansatz (f = eλt) leading to

λ2eλt + pλeλt + qeλt = 0 ⇐⇒ λ2 + pλ + q = 0 (4.18)

λ1t λ2t Then, the solution to the homogeneous ODE is fh = c1f1 + c2f2 = c1e + c2e , Home Page which is called homogeneous solution. Next, we will show that the general solution to the inhomogeneous ODE (4.17) is the sum of the homogeneous and Title Page particular solutions: fg = fh + fp. To find a particular solution fp, we will apply the method of variation of parameters proposed by Euler. Contents

Theorem 4.1. a) The general solution to Eq.(4.17) is fg = fh + fp, where fh = JJ II c1f1 + c2f2 is the solution to homogeneous equation. b) A particular solution fp can be found as J I Z Z f2(t)r(t) f1(t)r(t) Page 89 of 128 fp(t) = −f1(t) dt + f2(t) dt (4.19) W (f1, f2) W (f1, f2) where the determinant Go Back

f1 f2 W = 0 0 (4.20) Full Screen f1 f2 is called Wronskian. Close

Quit Proof. The point a) follows from the linear nature of the ODE and a superposition principle. As for the particular solution, since we already found the solutions to the homogeneous equation (f1 and f2), let’s try fp = v1(t)f1 + v2(t)f2, where v1 and v2 are the parameters to be varied. Upon differentiation, we get

0 0 0 0 0 fp = (v1f1 + v2f2) + v1f1 + v2f2 (4.21)

0 0 where we make (v1f1 + v2f2) = 0. Then, the second derivative becomes

00 0 0 0 0 00 00 fp = v1f1 + v2f2 + v1f1 + v2f2 (4.22) After substituting these expressions into the original Eq.(4.17) we obtain Home Page ( v0 f + v0 f = 0 Title Page 1 1 2 2 (4.23) v0 f 0 + v0 f 0 = r(t) 1 1 2 2 Contents This is again the familiar linear system of equations Ax = b that can be solved JJ II for the parameters v1 and v2, e.g., by using the Cramer’s rule with det (A) = W :

0 f2r(t) 0 f1r(t) J I v1 = − and v2 = (4.24) W (f1, f2) W (f1, f2) Page 90 of 128 By integrating we get the desired parameters: Go Back Z Z f2r(t) f1r(t) v1 = − dt and v2 = dt (4.25) W (f1, f2) W (f1, f2) Full Screen which we can plug into fp(t) = f1v1 + f2v2, thereby proving Eq.(4.19). Close

Quit The advantages of this approach include i) easy generalization to ODEs of any order; ii) coefficients in Eq.(4.17) need not be constants; iii) the forcing term r(t) can be a more general function than trigonometric, exponential or polynomial, thus being a more general approach than the method of undetermined coefficients. In addition to these two methods of solving ODEs, we have previously considered the method of Fourier transforms, and we will also discuss the Green function method below.

Example 4.2. Solve the following second-order ODE using the method of variation of parameters: f 00 − 2f 0 − 2f = te−t (4.26) Home Page Solution. To solve this ODE, we need to first find solutions to the homogeneous Title Page equation and construct the Wronskian. Solutions to the homogeneous equation f 00 − 2f 0 − 2f = 0 (4.27) Contents

−t 3t are f1 = e and f2 = e . Therefore, JJ II −t 3t f1 f2 e e 2t 2t 2t W = 0 0 = −t 3t = 3e + e = 4e (4.28) J I f1 f2 −e 3e According to Eq.(4.25) Page 91 of 128 Z e3t · xe−t Z x t2 v (t) = − dt = − dt = − (4.29) Go Back 1 4e2t 4 8 Full Screen Z −t −t Z −4t e · xe xe x −4t 1 −4t v2(t) = dt = dt = − e − e (4.30) 4e2t 4 16 16 Close Thus, the particular solution is given by Quit t2 x 1 f (t) = v f + v f = − e−t + (− e−4t − e−4t)e3t (4.31) p 1 1 2 2 8 16 16 and the general solution is −t 3t fg(t) = fh(t) + fp(t) = c1e + c2e + fp(t) (4.32)

We mentioned above that one rarely encounters ODEs of higher than second order. In chemical engineering the common source of such equations is in process control. Nevertheless, any high-order ODE can be transformed to a system of first-order ODEs and solved by the methods discussed earlier. For example, Eq.(4.17) can be Home Page transformed to the following 2×2 system by introducing a new variable: ( Title Page z = f 0 0 (4.33) z + p(t)z = r(t) Contents

JJ II 4.5. System Stability J I Let us here discuss the issue of stability on the example of both first- and second- order ODEs. Let’s first focus on the first-order ODE from Eq.(4.33): Page 92 of 128 0 z + p(t)z = r(t) (4.34) Go Back The solution to the homogeneous equation z0 + p(t)z = 0 is given by Full Screen −p(t)t zh = ce (4.35) which depends on the initial conditions through the constant c and is called the Close transient state. The particular solution is Z Quit −p(t)t p(t)t zp = e r(t)e (4.36) and defines the steady state. This analysis is meaningful if p(t) > 0 for which the transient solution converges to zero over time, and the system is called stable. In the case of a chemical reactor, the variables such as concentrations, temperatures, reaction rates, etc., are changing as a function of time until operation reaches nominal process variables, i.e., the steady state.

The analysis is analogous for the second-order ODE (4.17), where the general solution is f = f + f = (c f + c f ) + f . For this system to be stable it is g h p 1 1 2 2 p Home Page λ1t λ2t required that the transient solution fh = c1f1 + c2f2 = c1e + c2e → 0 for all c and c . Therefore, the system stability depends on the eigenvalues λ as follows: 1 2 i Title Page • λ > 0 (real and positive): the solution grows exponentially (the system is Contents unstable).

• λ < 0 (real and negative): the solution converges to zero (the system is JJ II stable). J I • λ = iω (purely imaginary): the solution is oscillatory according to the Euler formula eiωt = cos ωt + i sin ωt. Page 93 of 128

(a+iω)t at iωt • λ = a + iω (complex): since e = e e the solution behavior depends Go Back on the real part a. For a < 0 the solution is oscillating (due to eiωt) and at converging to zero (due to e ). For a > 0 the solution grows exponentially Full Screen with an oscillatory character (unstable). Close

Quit 4.6. Harmonic Oscillator and Green’s Function Method The most important second-order ODE is the one describing periodic motion of a harmonic oscillator, which is Eq.(4.17). The importance of this archetypical model in science and engineering cannot be overemphasized. A huge variety of oscillatory processes from classical to quantum, from physics to biology, can be analyzed in terms of the harmonic equation. In this class we will only touch on a few examples such as pendulum (mechanics), the Belousov-Zhabotinsky reaction (chemistry), and the Lotka-Volterra equations (biology). The last two are exam- Home Page ples of non-linear oscillators and will be analyzed in the section on dynamical sys- tems. Here, we will discuss the first example (pendulum). Instead of the approach Title Page discussed above, we will analyze the system behavior employing the method intro- duced earlier in the section on Fourier analysis, the Green’s function method. Contents The primary use of this approach is to solve linear non-homogeneous boundary value problems, both ODEs and PDEs. Moreover, when combined with the per- JJ II turbation theory, this method can be also applied to analyze the non-linear DEQs. J I One can think of Eq.(4.17) as the Newton’s law Page 94 of 128 mx00 = −cx0 − kx + r(t) (4.37) Go Back

c k r(t) Full Screen x00 = − x0 − x + (4.38) m m m describing the oscillatory motion of a particle on a spring subject to friction and Close driven by the external force r(t). The first term on the right (friction) is often called damping and is produced by any process that dissipates the system energy Quit (e.g., viscosity of a fluid, resistance of an electrical circuit, scattering of light). This damping force can often be approximated as proportional to the velocity and we will denote the proportionality constant c/m = 2γ. In the case when motion is very fast (e.g., when an object is rapidly moving in water or air), a better approximation would be the frictional drag proportional to the square of the velocity. The second term in the above equation is the restoring spring force 2 (Hook’s law) and we will use the notation k/m = ω0, where ω0 stands for the natural frequency of the spring. The driving force r(t) can be of any type, e.g., sinusoidal r(t) = cos (ωdt), where ωd is called the driving frequency. Home Page Thus, we can write Eq.(4.17) in the form Title Page 00 0 2 x + 2γx + ω0x = r(t) (4.39) Contents and represent it as a 2×2 system of linear ODEs: ( x0 = v JJ II 0 2 (4.40) v = −2γv − ω0x + r(t) J I

Let’s first consider the homogeneous case with r(t) = 0, and then apply the Green’s Page 95 of 128 function method when r(t) 6= 0. The matrix form of the above system of equations with r(t) = 0 reads as Go Back 0 x  0 1  x = 2 (4.41) Full Screen v −ω0 −2γ v in which one recognizes b = Ax. Its two eigenvalues can be found using the Close characteristic equation: Quit −λ 1 2 2 2 = λ + 2γλ + ω0 = 0 (4.42) −ω0 −2γ − λ q q 2 2 2 2 =⇒ λ1,2 = −γ ± γ − ω0 = −γ ± i ω0 − γ (4.43) The general solution is given by       x(t) 1 λ1t 1 λ2t x(t) = = c1 e + c2 e (4.44) v(t) λ1 λ2

The behavior of the solution depends on the relationship between ω0 and γ defining Home Page the eigenvalues λ1,2 in Eq.(4.43). There could be three cases. Title Page • Overdamped Case: friction is strong with γ > ω0. In this case both roots λ1,2 are real. The system decays exponentially without oscillations. Contents • Underdamped Case: friction is weaker than the spring force with γ < JJ II ω0. Both roots are complex with negative real parts. The system decays exponentially to zero with oscillations. J I • Critically Damped Case: γ = ω0. Roots are real and equal (λ1 = λ2 = Page 96 of 128 −γ = −ω0). The system decays faster than in any other case without oscil- lations according to te−ω0t. Go Back

Let us now consider the case when r(t) 6= 0 in Eq.(4.17), i.e., the case of an Full Screen externally driven damped oscillator. Since we have just found the homogeneous solutions, we only need to find a particular solution. In the section on Fourier Close transforms, we have shown that a particular solution to any linear ODE with a forcing term r(t) Quit Lfˆ p(t) = r(t) (4.45) can be found (see Eq.(3.28)) using the Green function as

Z t fp(t) = r(τ)G(t − τ)dτ (4.46) 0 where G(t−τ) is the Green’s function. Thus, once we know the Green function, we can readily compute a particular solution for any given forcing term using Eq.(4.46). To compute the Green function, we can re-write our original ODE (4.39) as Home Page d2 d ( + 2γ + ω2)G(t − τ) = δ(t − τ) (4.47) dt2 dt 0 Title Page Taking the Fourier transform we arrive at Contents 2 2 ˜ (−ω − i2γω + ω0)G(ω) = 1 (4.48) JJ II

˜ 1 J I =⇒ G(ω) = 2 2 (4.49) −ω − i2γω + ω0 Page 97 of 128 One way to find G(t) is by inverting the Fourier transform of the Green function

1 Z Go Back G(t) = e−iωtG˜(ω)dω (4.50) 2π Full Screen and then plugging the obtained result into Eq.(4.46) to find a particular solution. Depending on the driving term, the computation of the integral could be easy or Close hard and may require the use of calculus of complex functions. Quit Let us consider an especially simple case with sinusoidal driving force r(t) = cos (ωdt). In this case using the Euler formula Eq.(4.46) can be written as

Z Z 1 −iωdτ iωdτ −iωd(t−τ) 1 2 [e + e ]e fp(t) = dτ 2 2 dω (4.51) 2π −ω − i2γω + ω0 Here, we will use the completeness relation from Fourier transforms Z dk eikx = δ(x) (4.52) Home Page 2π Title Page re-writing Eq.(4.51) as

Contents 1 Z 1 [δ(ω − ω ) + δ(ω + ω )]e−iωdt f (t) = 2 d d dω = p 2π −ω2 − i2γω + ω2 0 JJ II −iωdt e 1 2 2 Re[ 2 2 ] = 2 2 2 2 2 [(ω0 − ωd) cos (ωdt) + 2γωd sin (ωdt)] −ωd − i2γωd + ω0 (ω0 − ωd) + 4γ ωd J I (4.53) Page 98 of 128

It is seen that the amplitude of this periodic function becomes largest when ωd = Go Back ω0, the phenomenon called resonance. Such resonance phenomena may appear with all types of vibrations - mechanical, acoustic, electromagnetic, nuclear, etc. Full Screen Examples of resonance are everywhere. Imagine you are drinking a nice glass of Close wine and listening to some pleasing music. Then all of a sudden the glass shatters in your hand for no apparent reason. The reason could be a resonance between Quit vibrations of the glass and music frequency. Of course, the reason could be much more trivial, for example, just a tiny crack in the glass producing internal stress. A more serious example of resonance is the magnetic resonance imaging (MRI), a medical imaging technique employed to visualize the anatomy and physiological processes in the body avoiding the use of radiation (like in Computer Tomog- raphy). The application of MRI is based on the physics of nuclear magnetic resonance (NMR) in which nuclei (tiny magnets) such as 1H and 13C are per- turbed by an oscillating magnetic field. This interaction produces a signal when the intrinsic frequency of the nuclei matches the frequency of the external magnetic field (the resonance frequency). NMR spectroscopy is used routinely in analytical and organic chemistry to determine the materials structure. It is interesting to Home Page note that the study of the NMR effect was a truly academic endeavor resulting in two Nobel Prizes in Physics. However, eventually, it led to MRI with a wide range Title Page of applications in medicine and technology. Contents

Let us now briefly discuss the physics of oscillating systems more intuitively (phys- JJ II ically) rather than abstractly (mathematically). We showed that if the friction is weak relative to the spring constant (underdamped case) we should observe oscil- J I lations, but now each successive oscillation should be reduced in amplitude due to the friction. Since the friction is constant, the amplitude should be reduced Page 99 of 128 by the same fraction each cycle. Here, we can recognize the exponential func- tion. If we denote the initial (at time t = 0) amplitude by A, then the amplitude Go Back should change with time as A(t) = e−ct, with c being some constant. Thus, the −ct produced oscillation is described by the function e cos (ωdt), which is what we Full Screen found mathematically. Close What about the resonance? If we look at Eq.(4.53) we see that if there is no friction Quit (γ = 0), then the amplitude of the function should go to infinity at ω0 = ωd. However, if we now turn on a little bit of friction (small γ), then the amplitude should be still very high, but now finite. If the friction is very strong (big γ), then the amplitude will become much smaller, in agreement with Eq.(4.53). The typical resonance curves are schematically shown in Figure on the left. If we have a huge friction near resonance, there should be not enough energy in the spring to oscillate against friction, and the system will slowly go to the equilibrium.

Home Page 4.7. Numerical Methods for Solv- ing ODEs Title Page

Clearly, not all ODEs can be solved by elemen- Contents tary functions and therefore numerical methods need to be applied. For example, the solution JJ II to even such a simple ODE as y0 = x2 −y2 can- not be obtained in terms of elementary func- J I tions. Here, we will consider two computational approaches (Euler and Runge-Kutta) and ana- Page 100 of 128 lyze their main properties for solving ODEs. Go Back The idea behind the Euler method is linearization (again), i.e. the fact that any function can be approximated locally by a linear function. Let us consider a Full Screen first-order ODE with an initial condition: ( Close y0 = f(x, y) (4.54) y(x0) = y0 Quit If we pick some step size ∆x, then we can write the update rules for both x and y variables as ( x = x + ∆x n+1 n (4.55) yn+1 = yn + ∆x · Sn 0 where Sn is the slope, Sn = (yn+1 − yn)/∆x = y = f(x, y).

Example 4.3. Solve the following ODE using the Euler method:

0 2 2 y = x − y Home Page Solution. Let’s choose the initial value for the function to be y(0) = 1 with the step size ∆x = 0.1. Table below summarizes the obtained results for the first few Title Page steps: Contents

n xn yn Sn ∆x · Sn JJ II 0 0 1 -1 -0.1 1 0.1 0.9 -0.8 -0.08 J I 2 0.2 0.82 -0.6324 -0.06324 Page 101 of 128 3 0.3 0.75676

Go Back Thus, y(0.3) ≈ 0.757. Obviously, the quality of this approximation depends on the step size and the given function. Below, we answer a question whether the obtained Full Screen result is under- or over-estimation of the true value. Close

Quit The answer to the question if the Euler method produces too high or too low value depends on the curvature of the function. There are two cases: either the curve is convex (y00 > 0) or concave (y00 < 0). If the curve is convex (like a bowl), then with each step the Euler line segments would increasingly deviate from the true curve thereby producing a lower value. On the contrary, if the curve is concave (like a ball), then with each step the line segments would overshoot the true curve. Thus, for the example considered above we compute that y00 = 2x − 2yy0 and thus at the point (0, 1) we find y00 = 2 > 0 defining the convex curvature. Therefore, in this example the Euler method underestimates the true value.

Home Page It can be shown that the error produced by the Euler method is linearly propor- tional to the step size, err ∼ c∆x, being the first-order method. This, however, can Title Page be improved if instead of one slope Sn an average between two neighboring slopes 2 is taken. In this case, err ∼ c∆x and the method is called the Runge-Kutta Contents method of second-order (RK2) for which the update rules become  JJ II xn+1 = xn + ∆x S1 +S2 (4.56) n n J I yn+1 = yn + ∆x · ( 2 )

Page 102 of 128 Analogously, if the slope is obtained by averaging over four slopes, we arrive at the Runge-Kutta method of forth-order (RK4) for which err ∼ c∆x4. Among Go Back these methods RK4 is most accurate, but it comes at the expense of speed since at each step four slopes need to be computed. In MATLAB, RK2 and 3 are the Full Screen basis for the method ode23 and RK4 and 5 is the basis for ode45. In the case of stiff systems, their modifications such as ode23s should be applied. Close

Quit 4.8. Partial Differential Equations (PDEs) There exists a large variety of PDEs requiring different solution strategies. Here, we will only focus on a few specific forms commonly arising in chemical engineering. We will introduce one numerical approach called the method of lines. The key point here is that this method allows one to convert a PDE into a system of ODEs and then apply the results of linear algebra to solve it. In addition, we will illustrate how the Green function method introduced earlier can be applied to analyze non-homogeneous PDEs with boundary conditions. Home Page

The most common source of PDEs in chemical engineering is unsteady state mass Title Page and energy balances. In a general form, they can be written as Contents ! ∂u ∂2u ∂2u ∂2u = α + + + f(x, y, z, t) (4.57) ∂t ∂x2 ∂y2 ∂z2 JJ II where u can stand for c (concentration) or T (temperature), α is either thermal or J I mass diffusivity, and f(x, y, z, t) could incorporate extra terms such as convection, Page 103 of 128 chemical reactions or heat exchange. For simplicity, here we will set f(x, y, z, t) = 0 and focus on one-dimensional case with u(x, t). The solution to PDE (4.57) is the Go Back function u(x, t) describing the time evolution of c or T spacial profiles as they approach the steady states. Full Screen

Close 4.9. Case Study: Unsteady Heat Conduction and the Method of Lines Quit The method of lines strategy:

1. Discretize the spacial domain into n nodes: x1, x2, ..., xn and consider a time- dependent Ti(t) at each node. This con- verts PDE into a set of n ODEs.

dTi 2. Approximate the derivative dt as a centered finite difference for all interior Home Page nodes: Title Page dT T (t) − 2T (t) + T (t) i = α i+1 i i−1 dt ∆x2 Contents (4.58) 3. Integrate the resulting system of ODEs JJ II to determine T (t) using any method ap- i J I propriate for ODEs. Page 104 of 128

Let’s show how this works on the example of unsteady heat conduction described Go Back by the following equation ∂T ∂2T = α (4.59) Full Screen ∂t ∂x2 subject to two boundary conditions and one initial condition: Close

BC : T (x1, t) = T (xn, t) = 0 Quit (4.60) IC : T (x, 0) = 2 To express a function derivative of any order as a finite difference, such as Eq.(4.58), Taylor series expansion is used:

2 dy 1 d y 2 3 y(xi+1) = y(xi) + ∆x + ∆x + O(∆x ) (4.61) dx 2 dx2 xi xi where ∆x = xi+1 − xi. We can express the first and second derivatives as finite differences as

dy y(xi+1) − y(xi−1) 2 = + O(∆x ) (4.62) Home Page dx 2∆x xi 2 Title Page d y y(xi+1) − 2y(xi) + y(xi−1) 4 2 = 2 + O(∆x ) (4.63) dx x ∆x i Contents where Eq.(4.63) corresponds to Eq.(4.58) for the temperature function. JJ II Thus, we can write down a system of ODEs using a finite difference approximation with the discretization step ∆x as J I  dT1 Page 105 of 128  dt = 0  h i  dT2 T3(t)−2T2(t)  = α 2  dt ∆x Go Back ... (4.64) h i  dTn−1 −2Tn−1(t)+Tn−2(t) Full Screen  dt = α ∆x2   dTn  dt = 0 Close Or in matrix form for the interior nodes: Quit dT α = (−K)T (4.65) dt ∆x2 where the matrix  2 −1 0 0 0 0 ...  −1 2 −1 ... 0 0 0     0 −1 2 −1 0 ... 0  K =   (4.66) ···     0 0 0 ... −1 2 −1 0 0 0 0 ... −1 2 Home Page is called a Toeplitz matrix. As one can see, K is real symmetric (K = KT ) and sparse (most entries are zero). In MATLAB it can be constructed using the Title Page command toeplitz. Contents As we have learned before since K is symmetric it is diagonalizable. Let’s use this fact to carry out some stability-accuracy analysis. Imagine we applied a similarity JJ II transformation and obtained a decoupled system of ODEs: J I dT˜ α = ΛT˜ (4.67) dt ∆x2 Page 106 of 128 where Go Back   λ1 0 ··· 0   Full Screen  0 λ2 ··· 0  Λ =  . . . .  (4.68)  . . .. .    Close 0 0 ··· λn Quit λit is the matrix of eigenvalues λi. We also discussed that the solutions e converge if Re(λi) ≤ 0. Let’s employ the explicit Euler method discussed before for the largest eigenvalue λmax as it determines accuracy. With a new variable defined as αλmax λ˜max = (4.69) ∆x2 we can write dT˜ = −λ˜maxT˜ (4.70) dt The Euler update rule for temperature T with a time step ∆t looks like Home Page ˜max ˜max T˜i+1 = T˜i − λ ∆tT˜i = T˜i(1 − λ ∆t) (4.71) Title Page leading to ˜max n Contents T˜n = T˜0(1 − λ ∆t) (4.72) Thus, it is required that | 1 − λ˜max∆t |≤ 1. Alternatively, we can write JJ II 2 2∆x2 ∆t ≤ = (4.73) J I λ˜max α | λmax | Page 107 of 128 ∆t 2 ≤ (4.74) ∆x2 α | λmax | Go Back Thus, it is a combination of time and spacial discretization steps that determines stability limit. According to the inequality (4.74), to preserve the same level of Full Screen accuracy, if ∆x is decreased by 2, then ∆t have to be reduced by 4. In your Close homework, you will also apply this approach to analyze species diffusion in the presence of a second-order chemical reaction (the diffusion-reaction PDE). This Quit will result in a non-linear equation that needs to be linearized and then solved by the method of lines. Additional Literature

1. C. Henry Edwards and David Penney, "Elementary Differential Equations with Boundary Value Problems" Pearson 2007. 2. G Hardin, "The Tragedy of the Commons" Science 162, 1243-1248 (1968).

Home Page

Title Page

Contents

JJ II

J I

Page 108 of 128

Go Back

Full Screen

Close

Quit 5. Nonlinear Dynamical Systems: From Fireflies to Reactors

The study of nonlinear dynamical systems is a fascinating and very rich topic. Its main concepts are finding applications in chaos theory, turbulence, self-organization processes, bifurcation theory, neuroscience. Although we can only touch upon a few key concepts, it would be terribly amiss to entirely skip an introduction to this terrific subject. Home Page

After a brief introduction of the key concepts, we will apply them in the context Title Page of four classical problems to illustrate the main features of complex nonlinear dynamical behavior. Specifically, we will discuss 1) the Lotka-Volterra (predator- Contents prey) model to describe oscillatory dynamics of biological systems, 2) the Belousov- Zhabotinsky reactions to describe chemical oscillations (chemical clocks), 3) the JJ II glycolysis metabolic pathway, and 4) the Michaelis-Menten kinetic model. J I

Page 109 of 128 5.1. Prelude In 1961 Edward Lorenz, a meteorology pro- Go Back fessor at MIT, was developing a numerical model to forecast weather using variables Full Screen such as temperature, pressure and the di- Close rection of the airstream. He noticed that if he accidentally started the simulation us- Quit ing a variable rounded off to a 3-digit num- ber rather than the computer 6-digit value, such as 0.506 instead of 0.506127, the predicted weather patterns would diverge with all similarity vanished after some time (see Figure). Thus, it turned out that a small numerical error could produce a drastically different result even though the model was fully deterministic with deceptively simple equations:  dx  dt = σ(y − x)  dy dt = −xz + ρx − y (5.1)  dz  dt = xy − βz Home Page where σ, ρ, and β are some parameters. Not only this demonstrated that even Title Page accurate modeling cannot lead to precise long-term weather predictions, but also started the new field of deterministic chaos. Contents

We also know that chaos can lead to the spontaneous emergence of order, the self- JJ II organization. For instance, it is known that fireflies in a congregation synchronize their flashing light patterns. This can be observed, e.g., in Great Smoky Mountains J I National Park, Tennessee. What is remarkable is that the insects do not need to be intelligent to self-organize. This turns out to be a general phenomenon from fireflies Page 110 of 128 to sleep cycles and heart beating. It appears that the reason is mathematical, viz. synchronization of a collection of coupled harmonic oscillators. Go Back

What is common in both examples is that the observed behavior arises due to the Full Screen nonlinear nature of governing equations. Mathematically, it means the violation Close of a superposition principle characteristic of linear systems in which the whole is the sum of its parts. Nonlinearity leads to cooperation or competition, either Quit it is the motion of a turbulent fluid, the spread of an infectious disease or the predator-prey relationships. 5.2. Nonlinear Algebraic Equations Before considering nonlinear ODEs, let’s first discuss how more simple nonlinear algebraic equations can be solved. Here, we will consider the Newton-Raphson method that in the case of a function of only one variable is also called the Newton’s method.

Let’s denote the multidimensional root of a system of equations as x∗. Taylor expanding a function around an initial guess xk keeping only the linear terms Home Page leads to the matrix form expression Title Page R(x) = R(xk) + J(xk)(x − x∗) (5.2) where J(xk) is the Jacobian to be evaluated at the point xk: Contents   ∂R1 ... ∂R1 ∂x1 ∂xn JJ II k  ∂R ∂R   . . .  J(x ) = ... =  . .. .  (5.3) ∂x1 ∂xn  . .  ∂Rn ... ∂Rn J I ∂x1 ∂xn R(xk) is called the residual vector and R(x∗) = 0. The iterative procedure can Page 111 of 128 be written as Go Back J(xk)(xk+1 − xk) = J(xk)δk+1 = −R(xk) (5.4)

The iterations proceed until the norm of the residual vector becomes small enough. Full Screen

Example 5.1. Find a root for the following system of equations using the Newton- Close 0 0 Raphson method starting from the initial guess x1 = x2 = 0: ( −x1 Quit f1 = e − x2 2 f2 = x1 + x2 − 3x2 Solution. Let’s carry out only the first iteration for demonstration purposes. The residual vector and the Jacobian for the given system of equations are as follows:

 −x1  e − x2 R = 2 (5.5) x1 + x2 − 3x2

−e−x1 −1  J = (5.6) 1 2x2 − 3 According to Eq.(5.4) we need to solve the following system at the point x0 = (0, 0): Home Page

−1 −1 δ  1 Title Page 1 = − (5.7) 1 −3 δ2 0 3 1 1 Contents Using the Cramer’s rule we get δ1 = and δ2 = that are the new values of x . 4 q 4 −3/4 1 2 3 1 2 3 2 The norm of the residual vector kRk = (e − 4 ) + ( 4 + ( 4 ) − 4 ) ≈ 0.23. JJ II

J I

5.3. Case Study: Antoine Equation for Nonlinear Flash Sep- Page 112 of 128 aration Go Back In the linear algebra section we discussed how to solve a system of linear algebraic equations for a 3-component flash separation. Let’s here consider a nonlinear case Full Screen of a binary mixture (e.g., pentane-hexane) where we can consider both the liquid and vapor phases as ideal. The saturation pressure for each component P sat (in Close mm Hg) at temperature T (in ◦C) can be described by the Antoine equation Quit sat Bi log Pi = Ai − (5.8) T + Ci where A, B and C are the component-specific Antoine coefficients. The 2×2 system of equations to be solved is

( sat xhPh = ph sat (5.9) (1 − xh)Pp = pp where xh is the liquid-phase mole fraction of hexane and ph, pp are the partial pressures of hexane and pentane, respectively, in the vapor phase. As part of your homework, this will need to be solved employing the Newton-Raphson method. Home Page

Title Page

5.4. Nonlinear Dynamics, Steady States and Multistability Contents We will focus on the first-order nonlinear ODEs of the following type: JJ II dy = f(y) (5.10) dt J I where t is time and f(y) represent nonlinear functions. Notice no explicit depen- Page 113 of 128 dence on time on the right hand side (autonomous equations). The system’s steady states (or equilibrium points) are given by the solutions y to ss Go Back dy ss = f(y ) = 0 (5.11) dt ss Full Screen

We aim to analyze what happens to the solution under a small perturbation to Close the steady state: y = yss + ∆. We can thus Taylor expand the functions f around the steady state keeping only first-order terms to get: Quit

f(y) = f(yss) + Jss(y − yss) = Jss(y − yss) = Jss∆ (5.12) where Jss is the Jacobian matrix at the steady state:   ∂f1 ... ∂f1 ∂y1 ∂yn  ∂f ∂f   . . .  Jss = ... =  . .. .  (5.13) ∂y1 ∂yn  . .  ∂fn ... ∂fn ∂y1 ∂yn If we now plug Eq. (5.12) into the original ODE (5.10) we obtain d∆ = J ∆ (5.14) Home Page dt ss This is again an eigenvalue problem with the solutions given by Title Page

Λt −1 ∆ = Xe X ∆0 (5.15) Contents where X is the matrix of eigenvectors for Jss, ∆0 is the initial perturbation and the eigenvalue matrix exponential JJ II   eλ1t 0 ··· 0 J I  0 eλ2t ··· 0  Λt   e =  . . . .  (5.16) Page 114 of 128  . . .. .   . . .  0 0 ··· eλnt Go Back

Full Screen

5.5. Lotka-Volterra Model: Biological and Chemical Oscil- Close lators Quit Let’s denote the prey and predator populations as y1 and y2, respectively. Accord- ing to the Lotka-Volterra model, the population changes over time are described by the following system of ODEs: ( dy1 = αy − βy y dt 1 1 2 (5.17) dy2 dt = δy1y2 − γy2 where α, β, δ, and γ are some positive constants. The two steady states obtained 0 0 γ α from y1 = y2 = 0 are yss1 = (0, 0) and yss2 = ( δ , β ). A linearization of the equations gives the Jacobian matrix α − βy −βy  Home Page J = 2 1 (5.18) δy2 δy1 − γ Title Page At yss1 = (0, 0) the Jacobian is α 0  Contents J = (5.19) ss1 0 −γ JJ II and therefore the steady state is an unstable saddle point defined by (eαt, e−γt). J I γ α At yss2 = ( δ , β ) the Jacobian is Page 115 of 128 −βγ ! 0 δ Jss2 = δα (5.20) Go Back β 0 √ √ and therefore the steady state is a center defined by (eit αγ , e−it αγ ). The solu- Full Screen tions are periodic and oscillating on a small ellipse around the equilibrium point. Close Let’s eliminate time from the ODEs by dividing the two equations: Quit dy y δy − γ 2 = − 2 1 (5.21) dy1 y1 βy2 − α which can be written in a separable form as

βy2 − α δy1 − γ dy2 = − dy1 (5.22) y2 y1 Integration of both parts yields

α ln y2 − βy2 = −γ ln y1 + δy1 + c (5.23) where c is a constant of integration. Let’s assume for simplicity α = β = δ = γ = 1 Home Page and exponentiate the above equation to get Title Page 1 1 y1 y2 = ce (5.24) y2 e y1 Contents which leads to −y1 −y2 c = y1e · y2e (5.25) JJ II According to the last equation, the maximum point of each slightly distorted center J I shown in the figure below is in the middle and decreases along both y1 and y2 as −yi yie (i = 1, 2). Page 116 of 128

We can modify the model by introducing a new term which role is to "remove" Go Back (like a net) preys and predators at the same constant rate k: Full Screen ( dy1 = αy − βy y − ky dt 1 1 2 1 (5.26) dy2 Close dt = δy1y2 − γy2 − ky2

γ α Quit In this case, instead of the old critical point ( δ , β ), we obtain a new steady state γ+k α−k at ( δ , β ). This counterintuitive result is called the Volterra’s principle: Home Page

Title Page

Contents Figure 3: Oscillations of prey and predator populations as a function of time (on the left) and the phase plane for the predator-prey equations for different initial JJ II conditions (on the right). J I while we remove both populations at the same rate, the prey population will Page 117 of 128 γ+k α−k increase ( δ ), and the predator population will decrease ( β ). This effect is sometimes called the paradox of the pesticides as it was also reported in the Go Back case of pest-pesticide relationships when application of pesticides would lead to an increased population of pests. Such manifestation of non-linear effects reveals that Full Screen intervention into biologically coupled systems could produce unexpected results. Close

Let’s show how the same mathematical model can be applied to chemical systems. Quit Let’s write a series of consecutive irreversible reactions to yield a product P :

 k A + X −→1 2X  X + Y −→k2 2Y (5.27)  Y −→k3 P where the first two steps are autocatalytic (i.e., X and Y accelerate their own production). Let’s now write the rate of concentration changes for X and Y : Home Page

( d[X] Title Page dt = k1[A][X] − k2[X][Y ] d[Y ] (5.28) dt = k2[X][Y ] − k3[Y ] Contents By introducing new variables as JJ II k2 k2 k1 τ = k3t; x = [X]; y = [Y ]; a = [A] (5.29) k3 k3 k3 J I the system (5.28) can be simplified to Page 118 of 128

( dx Go Back dτ = ax − xy dy (5.30) dτ = xy − y Full Screen which is same as the Lotka-Volterra model discussed earlier. Thus, the change of species concentrations with time will look as shown in the Figure above if we Close associate X with preys and Y with predators. This class of non-linear chemical oscillators are often called the Belousov-Zhabotinsky reactions or chemical Quit clocks. The observed periodic behavior follows a stable limit cycle (attractor). An attractor can represent anything from a point or a curve to even a fractal structure for which the pattern repeats itself at every scale (in this case it is called the strange attractor). Strange attractors often appear in fluid dynamics described by the Navier-Stokes equation. Each attractor has a basin of attraction − the region of phase space from which the system cannot escape regardless of the initial conditions and will eventually fall into the attractor.

Home Page One of the unusual features of nonlinear systems is that the steady states can branch out (bifurcate) into multiple branches with different stability properties Title Page as one or more parameters is varied. The system stability as a function of pa- rameters can be analyzed using a bifurcation diagram. A detailed analysis of Contents such systems is the subject of bifurcation theory and is beyond the scope of this class. In your homework, however, you will familiarize yourself with the emergence JJ II of bifurcations on the example of a non-isothermal continuously stirred tank re- actor. Specifically, you will see that critical points may interchange their stability J I (i.e., unstable → stable and vice versa) if the parameters are varied, a situation called transcritical bifurcation. As you can guess this could have quite serious Page 119 of 128 implications in practice. Go Back

5.6. Case Study: Glycolytic Metabolic Oscillator Full Screen

Close Glycolysis is a crucial metabolic process by which glucose C6H12O6 is converted − + into pyruvate CH3COCOO and a proton H through a sequence of ten enzyme- catalyzed reaction steps. The free energy released during this series of reactions Quit is used to generate the ATP (adenosine triphosphate) molecules that store energy. It is interesting to note that the concentrations of all intermediate metabolites in the glycolytic cycle oscillate with the same period, but different phases.

A simplified mathematical model of glycolytic oscillator can be written as

( dx 2 dt = −x + αy + x y dy 2 (5.31) dt = β − αy − x y Home Page where α and β are some positive constants, x is the concentration of ADP (adeno- sine diphosphate) and y is the concentration of F6P (fructose-6-phosphate) which Title Page are the key intermediate metabolites. Contents Let’s compute the Jacobian matrix for the system: JJ II −1 + 2xy α + x2  J = (5.32) J I −2xy −(α + x2) Page 120 of 128 This can be evaluated at the critical point β (β, α+β2 ) to produce Jss. The character- Go Back istic polynomial to be solved to find the 2 eigenvalues is λ −Tr(Jss)λ+det (Jss) = 0, Full Screen where Close 2 det (Jss) = α + β > 0 Quit β4 + β2(2α − 1) + (α2 + α) Tr(J ) = − ss α + β2 In general for 2×2 systems,

λ2 − Tr(A)λ + det (A) = 0 (5.33)

p Tr ± Tr2 − 4 det λ = (5.34) 1,2 2 where Tr = λ1 +λ2 and det = λ1λ2. The classification of steady states can be sum- marized in a single diagram as shown in the figure. Home Page

Title Page

Therefore, for our glycolytic oscillator the steady Contents β state (β, α+β2 ) is unstable for Tr(Jss) > 0 and stable for Tr(J ) < 0 since det (J ) > 0. The ss ss JJ II borderline is when Tr(Jss) = 0√leading to the requirement β2 = 1 (1 − 2α ± 1 − 8α). This 2 J I defines a curve in (α, β)-plane showing the region of stable limit cycles for the glycolytic oscillator. Page 121 of 128

One may wonder why oscillations? It appears Go Back that oscillation dynamics is a common control mechanism adopted by nature to suppress over-expression or boost expression Full Screen of certain species through complex nonlinear couplings. Such feedback provides living cells with a regulatory mechanism necessary for survival in open fluctuating Close environments. Quit 5.7. Case Study: Michaelis-Menten Enzyme Kinetics The Michaelis-Menten mathematical model was developed to describe the kinetics of the hydrolysis of sucrose into glucose and fructose by the enzyme invertase. With appropriate modifications the model can be applied to analyze a variety of biochemical reactions that can be conceptually viewed as enzyme-substrate in- teractions. These include antigen-antibody binding, protein-ligand interactions, enzymatic biomass conversion, microbial growth. Home Page Schematically, the reaction mechanism can be represented as Title Page kf k E + S )−*− ES −−→cat P + E (5.35) kr Contents where the enzyme, E, forms a complex ES with the substrate, S, which then JJ II releases the product, P , regenerating the enzyme. The rate of product formation can be written as d[P ] J I = kcat[ES] (5.36) dt Page 122 of 128 Assuming that the reaction is at the steady state, we can determine [ES] using Go Back d[ES] = kf [E][S] − (kcat + kr)[ES] = 0 (5.37) dt Full Screen Since the total enzyme concentration is [E] = [E] + [ES], then tot Close

kf [E]tot[S] [ES] = (5.38) Quit (kr + kcat) + kf [S] Substituting this into Eq.(5.36) we obtain

d[P ] [S] = kcat[E]tot (5.39) dt KM + [S]

kr +kcat where KM is the so-called Michaelis constant defined as KM = . kf

The changes in species concentration over time are shown in the figure. We Home Page mentioned earlier that the model is Title Page used to describe the breaking down of sucrose into glucose and fructose, while Contents here we only have one product [P ]. It turns out that glucose is a competi- tive inhibitor that also binds to the JJ II substrate ensuring that the reaction J I slows down. This reaction is not the only example of competitive inhibition Page 123 of 128 as a biological control mechanism. For example, if you accidentally consume Go Back methanol (poor you), it will be broken down by an enzyme in the liver producing toxic formaldehyde and formic acid that can kill you. However, the enzyme binds Full Screen more strongly to ethanol and therefore if you then drink ethanol (lucky you), it will saturate the enzyme helping with methanol intoxication. Close

Quit Additional Literature

1. S. Strogatz, "Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engineering" Westview Press 2015. 2. S. Strogatz, "SYNC: The Emerging Science of Spontaneous Order" Hachette Books 2003.

3. J. Glieck, "Chaos: Making a New Science" Penguin Books 2008. Home Page 4. I. Prigogine, I. Stengers, "Order out of Chaos: Man’s New Dialogue with Nature", Flamingo 1984. Title Page

5. E. Lorenz, "Deterministic Nonperiodic Flow" Journal of the Atmospheric Contents Sciences 20, 130-141 (1963). JJ II

J I

Page 124 of 128

Go Back

Full Screen

Close

Quit Index Antoine equation, 112 correlation, 74 attractor, 119 covariant representation, 31 autocorrelation, 75 Cramer’s rule, 46 autonomous equations, 113 critical damping, 96 Home Page basin of attraction, 119 Damköhler number, 14 basis set, 30 damping, 94 Title Page Belousov-Zhabotinsky reactions, 118 decoupling, 21 bifurcation diagram, 119 derivative formula, 71 Contents bifurcation theory, 119 determinant, 47 bifurcation, transcritical, 119 deterministic chaos, 110 JJ II butterfly effect, 24 eigenvalue equation, 50 J I characteristic polynomial, 51 eigenvalue spectrum, 50 chemical clocks, 118 eigenvalues, 50 Page 125 of 128 competitive inhibitor, 123 eigenvectors, 50 concave, 102 Einstein, Albert,6 Go Back condition number, 23 Euclidean vector space, 31 conditioning, 23 Euler method, 100 Full Screen conservation laws,9 exponential shift formula, 72 constitutive relations,9 Close convex, 102 filtering, 74 convolution, 72 finite difference, 105 Quit convolution kernel, 74 fireflies, 110 flow graphs, 59 Laplace rule, 48 Fourier convolution theorem, 72 Laplace transform, 77 Fourier series, 65 Leibniz rule, 48 Fourier transform, 70 Levi-Civita symbol, 36 Fourier transform, inverse, 70 limit cycle, 119 Fourier, Jean-Baptiste Joseph , 65 logistic differential equation, 85 logistic equation with harvesting, 87 Gaussian elimination, 44 logistic growth, 86 Gershgorin’s disk theorem, 53 Lorenz, Edward, 109 Home Page Gibbs phenomenon, 68 Lotka-Volterra model, 114 glycolytic oscillator, 119 lumping, 22 Title Page Green’s formula, 79 Green’s function, 97 magnetic resonance imaging, 99 Contents manifold, 27 harmonic frequency, 66 map, 41 JJ II harmonic oscillator, 94 map, inverse, 41 hear conduction, 103 map, linear, 41 J I Hermitian map, 54 map, orthogonal, 56 Page 126 of 128 homogeneous equation, 88 map, unitary, 56 matrix, 42 Go Back inhomogeneous equation, 88 matrix, adjoint, 43 integrating factor, 84 matrix, augmented, 45 Full Screen iterative methods, 58 matrix, Hermitian, 43, 54 matrix, inverse, 43 Jacobi method, 58 Close matrix, stoichiometry, 61 Jacobian linearization, 19 matrix, symmetric, 43 Jacobian matrix, 38 Quit matrix, Toeplitz, 106 Jordan form, 54 matrix, transpose, 43 resonance, 98 matrix, unit, 44 Reynolds number, 14 method of lines, 103 Runge-Kutta method, 102 method of variation of parameters, 89 metric tensor, 38 scalar, 27 Michaelis-Menten kinetics, 122 scalar product, 32 self-organization, 110 Newton-Raphson method, 111 separable differential equation, 84 norm, 31 similarity transformation, 50 Home Page normalized solutions, 15 simple growth, 86 nuclear magnetic resonance, 99 stability, system, 92 Title Page Nusselt number, 13 steady state, 93 stiff systems, 22 Contents ordinary differential equations, 83 stiffness ratio, 22 orthonormal basis, 35 strange attractor, 119 JJ II overdamping, 96 superposition principle, 16 symmetric map, 54 J I paradox of the pesticides, 117 system function, 79 partial differential equations, 83 Page 127 of 128 partial fraction decomposition, 77 topological space, 27 perturbation theory, 18 trace, 47 Go Back phase line, 86 tragedy of the commons, 87 Full Screen phosphorylation, 61 transfer function, 79 power spectrum, 75 transient state, 92 Close Prandtl number, 14 triple product, 38 pre-conditioning, 23 turbulence,9 Quit residual vector, 111 underdamping, 96 vector product, 36 vector space, 29 Volterra’s principle, 116 wave length, 66 wave number, 66 weight function, 80 Wiener-Khinchin theorem, 75 Wilkinson’s polynomial, 24 Home Page

Title Page

Contents

JJ II

J I

Page 128 of 128

Go Back

Full Screen

Close

Quit