T OMORROW’S H ARDEST P ROBLEMS

COMPUTATIONAL CHALLENGES IN MANAGEMENT

The authors describe a relatively simple problem that all investors face—managing a portfolio of financial securities over time to optimize a particular objective function. They show how complex such a problem can become when real-world constraints are incorporated into its formulation.

he financial industry is one of the considerations factor into its formulation. We fastest-growing areas of scientific present the basic dynamic portfolio optimization computing. Two decades ago, terms problem and then consider three aspects of it: such as financial engineering, com- taxes, investor preferences, and portfolio con- putationalT finance, and financial mathematics did straints. These three issues are by no means ex- not exist in common usage. Today, these areas haustive—they merely illustrate examples of the are distinct and enormously popular academic kinds of challenges financial engineers face to- disciplines with their own journals, conferences, day. Examples of other computational issues in and professional societies. portfolio optimization appear elsewhere.1,2 One explanation for this area’s remarkable growth and the impressive array of mathe- maticians, computer scientists, physicists, and The portfolio optimization problem economists that are drawn to it is the formida- Portfolio optimization problems are among ble intellectual challenges intrinsic to financial the most studied in modern finance,3 yet they markets. Many of the most basic problems in fi- continue to occupy the attention of financial aca- nancial analysis are unsolved and surprisingly re- demics and industry professionals because of silient to the onslaught of researchers from di- their practical relevance and their computational verse disciplines. intractabilities. The basic dynamic portfolio op- In this article, we hope to give a sense of these timization problem consists of an individual in- challenges by describing a relatively simple prob- vestor’s decisions for allocating wealth among lem that all investors face when managing a port- various expenditures and investment opportuni- folio of financial securities over time. Such a ties over time, typically the investor’s expected problem becomes more complex once real-world lifetime. The prices and price dynamics of goods and financial securities he or she purchases and any constraints such as tax liabilities, loan re- payment provisions, income payments, and 1521-9615/01/$10.00 © 2001 IEEE other cash inflows and outflows determine the investor’s overall budget. MARTIN B. HAUGH AND ANDREW W. L O For expositional clarity, we start with a simple MIT Sloan School of Management and Operations Research Center framework in which there are only two assets

54 COMPUTING IN SCIENCE & ENGINEERING available to the investor: a bond that yields a risk- portfolio immediately after $1.21 less rate of return and a stock that yields a ran- date t. Of course, xt and yt may $1.1 dom return of either 10 percent or –10 percent only depend on the informa- with probability p and 1 – p, respectively. We de- tion available at time t, a re- $.99 note the prices of the bond and stock at date t striction that we impose $1 $.99 with Bt and St, respectively. Without loss of gen- throughout this article. $.9 erality and for notational simplicity, we assume The requirement that xt that S0 = $1 and the riskless rate of interest is 0 and yt are nonnegative means ≥ $.81 percent, so Bt = 1 for all t 0. Finally, suppose that borrowing and that the investor’s horizon spans only three dates, sales are not allowed, a con- t = 0 t = 1 t = 2 t = 0, 1, and 2, so that the possible paths for the straint that many investors evolving stock price St are given as in Figure 1. face. That xt and yt are re- Figure 1. Stock price evolution. Of course, in practice, an investor has many as- quired to be integers simply sets to choose from over many dates, and the reflects the fact that it is not possible to purchase price of each asset can take on many values. For a fractional number of stocks or bonds. The con- illustrative purposes, though, this simpler speci- straint in Equation 3b states that Wt+1 is equal to fication is ideal because it contains all the essen- Wt multiplied by the portfolio’s gross return be- tial features of the dynamic portfolio optimiza- tween dates t and t + 1. tion problem in a basic setting. Nevertheless, We can easily solve this problem numerically even in this simple framework, it will become ap- by using the standard technique of stochastic dy- parent that practical considerations such as taxes, namic programming.4 In particular, because investor preferences, and portfolio constraints V2(W2) = u(W2), we can compute V1(W1) using can create surprisingly difficult computational the Bellman equation so that challenges. =+ Let Ct denote the value of the investor’s con- VW11()Max{} uC () 1 E 122[] VW () (4) C sumption expenditures at date t, and let Wt de- 1 note the investor’s wealth just prior to date-t consumption. We assume that the investor has subject to the constraints in Equation 3. An as- a lifetime function U(C0, C1, C2) defined pect of Equation 4 that makes it particularly easy • over each consumption path {C0, C1, C2} that to solve is the fact that the value function V1( ) summarizes how much he or she values the en- depends on only one state variable, W1. This en- tire path of consumption expenditures. Then, ables us to solve Equation 4 numerically without ignoring market frictions and assuming that the too many computations. Suppose, for example, ∈ + investor’s utility function is time-additive and that W0 = $1,000. Then, because xt, yt , there time-homogenous—for example, are only 1,100 possible values that W1 can take, so the right side of Equation 4 must be evaluated U(C0,C1, C2) = u(C0) + u(C1) + u(C2), (1) for only these 1,100 values. In contrast, when there are market frictions or when the investor the investor’s dynamic portfolio optimization has a more complex utility function, the compu- problem at t = 0 is given by tational requirements increase dramatically, re- flecting Bellman’s “curse of dimensionality.”4 =++ V00() WMax E 0012[] uC () uC ()() uC (2) CCC012,, Taxes subject to Most seasoned investors are painfully aware of the substantial impact that taxes can have on the W – C = x S + y B , t = 0, 1, 2 (3a) t t t t t t performance of their investment portfolio, so Wt+1 = xtSt+1 + ytBt+1, t = 0, 1 (3b) taxes play a major role in most dynamic portfolio ≥ 5–7 Ct 0, t = 0, 1, 2 (3c) optimization problems. ∈ + To see how taxes can increase the computa- xt, yt , t = 0, 1, 2 (3d) tional complexity of such problems, let trading x = y = 0 (3e) 2 2 profits in the stock be subject to a capital gains tax + where denotes the nonnegative integers, and xt in the portfolio optimization problem described and yt are the number of shares of stocks and bonds, earlier. Because this model has only two future respectively, that the investor holds in his or her periods, we do not distinguish between short-

MAY/JUNE 2001 55 term and long-term capital gains. For expositional see that the presence of taxes has made the dy- simplicity, we also assume that we cannot use cap- namic portfolio optimization problem consider- ital losses from one period to offset gains from a ably more difficult. Specifically, in solving Equa- • later one. Although these simplifying assumptions tion 4 numerically, V1( ) is computed for only make the problem easier to solve, it will be ap- 1,100 possible values of W1. In contrast, solving parent that the computations are considerably Equation 8 numerically requires the evaluation • more involved than in the no-tax case. of V1( ) for all possible combinations of {W1, S1, To solve the dynamic portfolio optimization N0,0}, of which there are 1,001,000! problem with taxes, we use dynamic program- (We can verify this by noting the one-to-one ming as we did earlier. However, the value func- correspondence between {W1, S1, N0,0} and {C0, S1, tion is now no longer a function of wealth only, N0,0}, and counting the possible combinations of but it also depends on past stock prices and the {C0, S1, N0,0}. Assuming, as before, that W0 = number of shares purchased at each of those $1,000, we see that there are 1,001 possible choices prices. In other words, the value function is now for C0. If C0 = i, then there are 1,000 – i possible path-dependent. If we use Ns,t to denote the num- values for N0,0. For each combination of (C0, N0,0), ber of shares of stock purchased at date s ≤ t and there are two possible values of S . This means that 1000, 1 that are still in the investor’s portfolio immedi- in total, there are 2∑ (, 1 000−=i ) 1 , 001 , 000 combinations i=0 ately after trading at date t, then we can express of {W1, S1, N0,0}.) the portfolio optimization problem as Even in a simple two-period two-asset model, =++the portfolio optimization problem with taxes be- V00() WMax E 0012[] uC () uC ()() uC (5) CCC012,, comes considerably more complex. Indeed, in the subject to T-period N-asset case, it is easy to see that by  t −1  date T, there are O(NT) state variables, and if  τ −− = Wt – Ct – Max 0,∑ (SSNt s )( st,,−1 N st ) each state variable can take m distinct values, then  =  s 0 there will be O(mNT) possible states at date T. For xtSt + ytBt, t = 0, 1, 2 (6a) an investor with a 20-year horizon, an annual Wt+1 = xtSt+1 + ytBt+1, t = 0, 1 (6b) trading interval, and a choice of 25 assets, and as- t suming that the state variables take on only four ∑ N distinct values at the end of the horizon, the xt =,st, t = 0, 1, 2 (6c) s=0 number of possible states at the end will be of the 301 C ≥ 0, t = 0, 1, 2 (6d) order 10 . (The number of distinct values that a t state variable can take on depends, of course, on ≥ ≥ ≥ N0,0 N0,1 N0,2 0 (6e) the precise nature of the state variable. For ex- ≥ ≥ N1,1 N1,2 0 (6f) ample, for a binomial state variable that is not ∈ + xt, yt , t = 0, 1, 2 (6g) “recombining,” the number of distinct values it T x = y = 0 (6h) can take at date t is 2 ; if it is recombining, this 2 2 is reduced to T + 1. We use four distinct values where τ is the capital gains tax rate. When t = 2, only for illustrative purposes; in most practical the value function depends on (W2, S1, S2, N0,1, applications, the number is considerably larger.) N1,1), and we obtain the relation

VWSS2(,,, 212 N 0111,, , N ) Preferences   1  Another important aspect of portfolio opti- =−uW Max0,(∑τ S − S ) N  (7) mization problems is the objective function that 221 ss,    s=0  represents the investor’s preferences. Tradition- ally, these preferences are represented by time- so that date-2 consumption is simply W2 less any additive time-homogeneous utility functions, capital gains taxes that must be paid. At t = 1, the which yield important computational advantages value function depends on (W1, S1, N0,0), and we because they imply that the value function at can write the Bellman equation as date t does not depend on the investor’s con- sumption choices prior to date t. VWSN(,, ) 11100, Unfortunately, the assumptions of time-additiv- =+ Max{}uC()1122120111 E [] V ( W ,,, S S N,, , N ) ity and time-homogeneity seem to be inconsistent C .(8) 1 with the empirical evidence on the consumption If we compare Equation 8 with Equation 4, we and portfolio choices of investors. For example, in-

56 COMPUTING IN SCIENCE & ENGINEERING dividuals tend to grow accustomed to their level of date t = 1 is 700.) consumption over a period of time, implying that In practice, these types of constraints are of- preferences depend not only on today’s consump- ten imposed on investment funds to reduce tion level but also on levels of past consumption.8 transaction costs and the risk of churning. When Commonly known as habit formation, such such a constraint is imposed, the value function • preferences imply that the value function Vt( ) is is no longer a function of only Wt but also of the a function of (C0, C1, ..., Ct–1) in addition to any cumulative number of shares transacted up to other relevant state variables. As in the case with date t. By creating path dependence in the value taxes, these problems quickly become intractable function, constraints can substantially increase as the number of time periods increases. Of the computational complexity of even the sim- course, closed-form solutions are available for a plest portfolio optimization problems. few highly parameterized models of habit for- mation,9 but in general these models must be solved numerically.10 There are many other em- Possible solution techniques pirical regularities of investors’ preferences that The most natural technique for solving dy- can induce path dependence in the value func- namic portfolio optimization problems is sto- tion, and for each of these cases, the computa- chastic dynamic programming. However, this tional demands quickly become intractable. approach is often compromised by several fac- tors such as the curse of dimensionality when too many state variables are involved. In general, Portfolio constraints practical considerations such as taxes, transac- When constraints are imposed on a portfolio tions costs, indivisibilities and optimization problem, their impact on the prob- integer constraints, non-time- lem’s computational complexity is not obvious. additive utility functions, and Closed-form solutions On the one hand, some unconstrained problems other institutional features of that admit closed-form solutions fail to do so financial markets tend to cre- are rarely available for once constraints are added. In practice, however, ate path dependencies in port- closed-form solutions are rarely available for re- folio optimization problems— realistic portfolio alistic portfolio optimization problems, with or this increases the number of without portfolio constraints. We must solve state variables in the value optimization problems. such problems numerically, in which case, im- function. Such problems are posing constraints can sometimes reduce the difficult to solve in all but the number of computations because they limit the simplest cases, with computa- feasible region over which we must evaluate the tional demands that become value function. An example of this is the impact prohibitive as the number of time periods and of the constraints in Equation 3 on the basic assets increases. portfolio optimization problem. In that case, we Here, we briefly outline an alternative that imposed the constraint that xt and yt are non- might produce good approximate solutions to negative integers, eliminating the possibility of otherwise intractable portfolio optimization borrowing or shortselling. This implies that only problems. This approach—called approximate a finite number of values for W1 are possible, and dynamic programming, neuro-dynamic pro- as a result, the number of computations needed gramming, or reinforcement learning—has had to evaluate V1(W1) is greatly reduced. much success recently in solving challenging dy- On the other hand, in some cases constraints namic optimization problems in several contexts, can greatly increase the number of computations, including .11–16 Although despite the fact that they limit the feasible set. there are many different algorithms that we This typically occurs when the constraints in- could categorize as “approximate dynamic pro- crease the problem’s dimensionality. For exam- gramming,” we will look at just one such algo- ple, in the basic portfolio optimization problem, rithm: approximate value iteration. consider imposing the additional constraint that Suppose the optimal value function at date t the cumulative number of shares transacted— of a T-period dynamic optimization problem is ∈ n both purchased and sold—up to date t is bounded given by Vt(Xt) where Xt is an n-dimen- by some function, f(t). (If 500 shares were pur- sional vector of state variables. We assume that chased at t = 0 and 200 shares were sold at t = 1, because of the computational intractabilities, it is • the cumulative number of shares transacted as of impossible to determine Vt( ) exactly. Therefore,

MAY/JUNE 2001 57 we define a parameterized class of functions value function, Vt. Deriving a lower bound is typ- ically straightforward—the sequence {Vt: t =1,...,T} defines a feasible trading strategy; therefore, the {}VX˜ ();:ββ∈ p (9) tttt value of this strategy, which we can estimate through simulation, is a lower bound for V0. which is called the approximation architecture. However, deriving an upper bound for V0 is gen- ˜ We then select our estimate Vt of Vt from this erally not so straightforward. One possibility is β p class of functions by selecting some t from , to try to apply stochastic duality theory, which the space of p-dimensional real-valued vectors. has already been studied extensively in the con- Now suppose that we have applied the back- text of portfolio optimization.17 In addition, ward recursion of the Bellman equation to ob- other researchers have successfully employed ˜ ˜ tain estimators VVtT+1,..., . Then we can use an duality theory in conjunction with approximate approximate Bellman equation to compute an dynamic programming to construct lower and estimator of the value function at time t so that upper bounds on the prices of American op- tions.18 A similar approach might also work for ˆ ()=+ ˜ βˆ  portfolio optimization problems, which we are VXttMax E ttt gX() V t+1() Xt+1; t+1  (10) investigating in ongoing research.

where gt(Xt) is the reward at time t, possibly repre- senting consumption as in our earlier examples, and where the maximization is with respect to here are many other approximate dy- the decision variables (suppressed for notational namic programming solutions, in- ˆ • simplicity). Computing Vt ( ) often entails exten- cluding algorithms based on approx- sive Monte Carlo simulations because it is gener- imate policy iteration and on ally not possible to compute expectations over a Q-learning.T 11 These approaches generally share high-dimensional space. In these cases, we esti- the common feature of resorting to function ap- ˆ • mate Vt ( ) at a fixed number of “training points” proximation and simulation techniques to deal ∈ n (P1, ..., Pm), where each Pi represents a possi- with computational intractabilities. As comput- ble realization of the state vector Xt at date t. Once ing power continues its remarkable growth, we we have estimated VPˆ () for i = 1, ..., m, we obtain believe that these techniques will become in- ~ ti • Vt( ) by solving the following least-squares problem: creasingly important in addressing many of the challenges of financial computing over the next m 2 few decades. ββˆ = arg Min ∑()VPˆ ()− VP˜ (); ttititβ . (11) t i=1 Acknowledgment ˜ With Vt now determined, we then proceed to This research was partially supported by the MIT ˜ compute Vt −1 in a similar fashion and continue Laboratory for Financial Engineering. in this manner of approximate value iteration ˜ until we find V0 . The computational require- References ments of this algorithm might be considerably 1. M. Haliassos and A. Michaelides, “Calibration and Computation less demanding than if we were to solve the of Household Portfolio Models,” Household Portfolios, L. Guiso, problem exactly. This is because we now only es- M. Haliassos, and T. Japelli, eds., MIT Press, Cambridge, Mass., timate the value function at a small representa- 2000. tive subset of the state space rather than com- 2. M. Haugh and A. Lo, “ and Derivatives,” Quanti- tative Finance, vol. 1, no. 1, Jan. 2001, pp. 42–75. puting it exactly at all points in the state space. 3. R. Merton, Continuous-Time Finance, Basil Blackwell, Oxford, UK, Once we implement an approximate dynamic 1990. programming algorithm, we obtain an estimator 4. D. Bertsekas, Dynamic Programming and Optimal Control, Vol. I, ˜ {}Vt of the value function. The natural question Athena Scientific, Belmont, Mass., 1995. that follows is whether this estimator is “good.” 5. D. Bertsimas, A.W. Lo, and G. Mourtzinou, Tax-Aware Multiperiod Although some theoretical results partially answer Portfolio Optimization, MIT Sloan School of Management, Cam- bridge, Mass., 1998. this question,11 it is often difficult in practice to 6. P.H. Dybvig and H.K. Koo, Investment with Taxes, Olin School, determine the accuracy of an approximate solu- Washington Univ., St. Louis, 1996. tion to a particular problem. One possibility is to 7. G.M. Constantinides, “Capital Market Equilibrium with Personal try to derive lower and upper bounds on the true Tax,” Econometrica, vol. 51, no. 3, May 1983, pp. 611–636.

58 COMPUTING IN SCIENCE & ENGINEERING 8. D. Kahneman, P. Slovic, and A. Tversky, Judgment Under Uncer- tainty: Heuristics and Biases, Cambridge Univ. Press, Cambridge, EDITORIAL UK, 1982. 9. G.M. Constantinides, “Habit Formation: A Resolution of the Eq- CALENDAR uity Premium Puzzle,” J. Political Economy, vol. 98, no. 3, June 1990, pp. 519–543. JANUARY/FEBRUARY 10. J. Heaton, “An Empirical Investigation of Asset Pricing with Tem- porally Dependent Preference Specifications,” Econometrica, vol. Usability Engineering in Software 63, no. 3, May 1995, pp. 681–717. Development 11. D. Bertsekas and J. Tsitsiklis, Neuro-Dynamic Programming, Athena When usability is cost-justified, it can be integrated into the Scientific, Belmont, Mass., 1996. development process; it can even become one of the main 12. F. Longstaff and E. Schwartz, “Valuing American Options by Sim- drivers of software development. How can you avoid conflict ulation: A Simple Least Squares Approach,” to be published in between your usability and development staff and build even Rev. of Financial Studies. stronger teams? 13. B. Van Roy, “Temporal Difference Learning and Applications in Fi- nance,” Computational Finance, Y.S. Abu-Mostafa et al., eds., MIT Press, Cambridge, Mass., 1999. MARCH/APRIL 14. J. Moody et al., “Performance Functions and Reinforcement Global Software Development Learning for Trading Systems and Portfolios,” J. Forecasting, vol. What factors are enabling some multinational and virtual 17, 1998, pp. 441–470. corporations to operate successfully across geographic and 15. J.N. Tsitsiklis and B. Van Roy, Regression Methods for Pricing Com- cultural distances, while others struggle and fail? Software plex American-Style Options, Laboratory for Information and De- development is increasingly becoming a multisite, multicultural, cision Systems, MIT, Cambridge, Mass., 2000. globally distributed undertaking. 16. R. Neuneier, “Optimal Asset Allocation Using Adaptive Dynamic Programming,” Advances in Neural Information Processing Sys- tems, D. Touretzky, M. Mozer, and M. Hasselmo, eds., MIT Press, MAY/JUNE Cambridge, Mass., 1996. Organizational Change 17. I. Karatzas and S. Shreve, Methods of Mathematical Finance, Today’s organizations must cope with reorganization, process Springer-Verlag, New York, 1998. improvement initiatives, mergers and acquisitions, and ever- 18. M. Haugh and L. Kogan, Pricing American Options: A Duality Ap- changing technology. Through experience reports, case studies, proach, Wharton School, Univ. of Pennsylvania, Philadelphia, and position papers, we will look at what organizations are 2001. doing and can do to cope.

Martin B. Haugh is a PhD candidate in operations re- JULY/AUGUST search at MIT. Starting January 2002, he will be an as- Fault Tolerance sistant professor in industrial engineering and opera- We used to think of fault-tolerant systems as ones built from tions research at Columbia University. His technical parallel, redundant components. Today, it’s much more complicated. Software is fault-tolerant when it can compute an interests include financial engineering and operations acceptable result even if it receives incorrect data during research. He received his BSc and MSc in mathemat- execution or suffers from incorrect logic. How do you plan for ics and statistics at University College, Cork and his this? How do you build it in? MSc in applied statistics at the University of Oxford. Contact him at the MIT Operations Research Center, SEPTEMBER/OCTOBER 1 Amherst St., E40-149, Cambridge, MA 02139; Software Organizational Benchmarking [email protected]. How do you decide what to benchmark and how much detail is necessary? How do you identify the right information sources? Andrew W. Lo is the Harris and Harris Group Professor of Finance and director of the MIT Laboratory for Fi- NOVEMBER/DECEMBER nancial Engineering. His technical interests include fi- Lightweight Processes nancial engineering, computational finance, risk man- Do lightweight processes such as eXtreme Programming and agement, stochastic processes, nonparametric statistics, adaptive software development lower the ceiling or raise the optimization, data mining, and computational learn- floor? Here’s a look at new flexible and agile methods as well as ing. He received his BA in economics from Yale and his some fresh techniques that are applicable to any software methodology. MA and PhD in economics from Harvard. He belongs to the IEEE, the American Economic Association, the Amer- ican Finance Association, the American Statistical Asso- ciation, the Econometric Society, the Institute of Math- ematical Statistics, and the Society of Industrial and Applied Mathematics. Contact him at the MIT Sloan School of Management, 50 Memorial Dr., E52-432, Cambridge, MA 02142; [email protected].

MAY/JUNE 2001