Model Risk Management

Nav Vaidhyanathan Director, Head of Model Risk Management Wintrust Financial Corporation

Global Association of Risk Professionals

August 2014 These materials reflect the views and opinions of the speaker and are not intended to reflect the views, policies or practices of Wintrust Financial or its affiliated companies. They are for informational purposes only.

The views expressed in the following material are the author’s and do not necessarily represent the views of the Global Association of Risk

Professionals (GARP), its Membership or its Management.

2 Agenda

Model Risk Management is a broad, evolving, and an expanding topic. This presentation will cover only select components at a high level.

. Inherent risk (model tier) and residual risk of model . A framework for model monitoring . Model risk aggregation framework . Model validation for stress test (e.g., DFAST) models

The following can potentially be considered as self-evident truths for Model Risk. Given everything else equal, . All models have limitations . Model Risk is related to model limitations . Model Risk increases with more models (this might fall in the grey area) − E.g., two Tier 1 models potentially pose higher risk than 1 Tier 1 model (everything else being the same, i.e., similar limitations, similar validation status, similar materiality, similar findings, etc.) . Model Risk for Tier 1 model > Model Risk for Tier 2 model . Model Risk for validated model ≤ Model Risk for unvalidated model (for the same model) . More findings ⇒ more Model Risk (same category and level of findings) . Certain types of findings will have higher Model Risk than others . Models with deteriorating performance pose higher Model Risk . Model Risk is exhibited by the variance between model output and observed actual (this is just one of the ways that Model Risk exhibits itself)

* may not be axioms in the purest sense of the term

Inherent Risk and Residual Risk of Model

Residual Risk Rating could be High

driven by: Rating • Model validation findings: Moderate

• Category Residual Low • Level Risk • # of Open Findings Low (Tier 3) Moderate High (Tier 1) • Model performance / Inherent Risk Rating (Model Tier) monitoring

Inherent Risk Rating could be driven by: Model owners can reduce residual • Materiality risk – by improving models, • Type of model improving controls, and/or • Used for regulatory / financial reporting or critical business remediating findings decisions • Feeds into another Tier 1 model?

Model owners cannot (usually) reduce inherent risk

. Residual Risk Rating (High, Moderate, Low) is based on the Residual Risk Score range

. E.g., RRS > 200 ⇒ High, RRS ≤ 100 ⇒ Low . If the model is deemed unfit for use, the RRR is High . The Residual Risk Score Range to Residual Risk Ratings mapping needs to be monitored periodically . The ranges can be determined by typical observations from model validation and model monitoring exercises . If a model is unvalidated, the residual risk rating is HIGH

. Model monitoring should evaluate whether changes (including anticipated) in products, exposures, activities, clients, or market conditions necessitate adjustment, redevelopment, or replacement of the model . Ongoing activities include – review of input data and assumptions, outcome analysis, model stability checks, review of overlays and expert judgments, benchmarking, etc. . Key considerations − Model performance monitoring › Pros and cons of model performance measures › Model type › Model use › Type of measure − Periodicity of monitoring − Thresholds for model recalibration or rebuild − From performance measure to model performance rating

Impact of Failure (e.g., on Capital)

Tier 1 Tier 2 Tier 3

Validated … …

Yes No

High Residual Moderate Low Residual As part of an RCSA Risk Residual Risk Risk process, the impact of the model failure is estimated by model owners

The end nodes in the For DFAST projection models, for Severely Adverse Scenario, the impact on capital can be assessed decision tree can be using backtesting during Great Recession period for all models (adjust the variance by severity of the used for aggregating scenario compared to the Great Recession) model risk By looking at the distribution, a conservative starting point for capital can be chosen for each model Tier

. Are estimates conditioned on macroeconomic scenarios? . Do the models produce outcomes that show separation and order between the baseline, adverse, and severely adverse scenarios? . Are the scenario and assumptions consistent across all models? . Are exposures modeled using appropriate levels of segmentation? . Qualitative elements of the model . Relevance of third party vendor models to company specific characteristics, especially when using proxy portfolio . Sensitivity analysis around assumptions . Use of weak models vs. expert judgment . For models not validated, compensating controls and conservativeness . Documentation . Capital buffer for model risk

Global Association of Risk Professionals

111 Town Square Place 14th Floor Jersey City, New Jersey 07310 U.S.A. + 1 201.719.7210

2nd Floor Bengal Wing 9A Devonshire Square London, EC2M 4YN U.K. + 44 (0) 20 7397 9630

www.garp.org

About GARP | The Global Association of Risk Professionals (GARP) is a not-for-profit global membership organization dedicated to preparing professionals and organizations to make better informed risk decisions. Membership represents over 150,000 risk management practitioners and researchers from banks, investment management firms, government agencies, academic institutions, and corporations from more than 195 countries and territories. GARP administers the Financial Risk Manager (FRM®) and the Energy Risk Professional (ERP®) Exams; certifications recognized by risk professionals worldwide. GARP also helps advance the role of risk management via comprehensive professional education and training for professionals of all levels. www.garp.org.

Michelle McCarthy Managing Director, Risk Management Nuveen Investments

Global Association of Risk Professionals

August 2014 Disclaimers

These materials reflect the views and opinions of the speaker and are not intended to reflect the views, policies or practices of Nuveen Investments or its affiliated companies. They are not intended to provide investment advice and are for informational purposes only.

The views expressed in the following material are the author’s and

do not necessarily represent the views of the Global Association of

Risk Professionals (GARP), its Membership or its Management.

2 Contents

. Differences between models and their uses at banks vs. the buy side

. What should be in scope for model risk management activity in the asset management business?

. Potential tiering of asset management models, and effect on model risk management activities

. OCC 2000-16 . OCC 2011-12, FRB SR 11-7 . The case of AXA Rosenberg brought the conversation to the buy side, without much guidance on how bank practices can and should differ from asset managers’ practices

. Asset managers are not computing regulatory capital  Bank-owned asset managers may need to provide operational risk capital calculations, but this is not in service of the asset management business . We rarely use valuation models as the sole source of valuation; we frequently use external sources . Many asset management models are an input to a decision, but not the sole driver of that decision

. Includes models that are used for investing or hedging Drive investing activity to some extent Value key assets . Does not include calculators; model must provide estimates and involve uncertainty to require model risk management Good change controls always an important operational control, but calibrations and validations should be reserved for models that estimate uncertainty Would not include compliance systems Would not include price-to-yield calculations Would not include simple screens for fundamental stock characteristics Would not include pure index replication models without meaningful uncertainty . Does not include analyses that help develop an investment thesis—only the tools used to execute the thesis, or to value assets once purchased

. A potential tiering of asset management models:

Tier 1 Tier 2 Tier 3 Tier 4 Models that: • are tightly coupled to trading but permit • apply a screen to a universe of • may support a certain • have high potential direct human intervention, and/or affect a stocks, using some measure of amount of decision impact on company results small portion of the business uncertainty, such as a measurement making, but with many • are broadly applied across the • result in trading with little other input of the dispersion of some other inputs to these entire company than direction from a model (quant fundamental characteristic (low price- decisions outside the • are understood and strategies) to-earnings ratios, for example) model. maintained by a relatively • provide hedge ratios for using • compare characteristics of securities • are outside the narrow range of experts at the derivatives, or to balance proportions to assist the portfolio manager in company’s areas of key company of fixed income instruments, in finding securities that he or she business risk • Result in instant electronic portfolios deems desirable • are understood by a trading with little or no ability • value smaller, less significant balance broad range of business to intervene, and/or affecting sheet items than those captured in Tier people a large portion of the portfolio 1

Such as: • A strategy that strongly adheres to • Using Barra or Yieldbook to see • Measures of portfolios • Statistical arbitrage model rebalancing if Barra or Yieldbook where a portfolio is overweight, in that are not used to • High frequency trading model measures exceed a threshold order to select trades that will balance drive action • A model that is rigorously • An economic indicator model that is the portfolio better • Index replication adhered to, is key to a broad the basis of an investment strategy • Economic indicator screens to models that approach set of strategies and is • High frequency trading models provide general investment signals 100% replication disclosed as a critical feature • Models used for delta hedging or asset • Index replication models with tilts of the investment process allocation rebalancing • Risk oversight models • Significant valuation models • Minor valuation models where third • Valuation models where third party where third party sources are party sources are unavailable valuations are available unavailable

Tier I Tier II Tier III Tier IV Investment Models Fundamental Business Models x x Stock Selection Screen x x Quantitative Stock Selection x x Portfolio Construction x x Index Replication with Tilts x Index Replication x Statistical Arbitrage x x High Frequency Trading x x Leverage x Hedging x Economic Models Growth x x Inflation x x Industry x x Credit x x Liquidity x x x Risk Models Capital Loss x Liquidity x x Ex Ante Tracking Error x x Ex Post Tracking Error x Duration, Beta, Factor Exposure Limits x x Valuation Models Third Party Pricing Services x x Pricing Model - Third Party Developed x x x Pricing Model - In-House Developed x x x

. Inventory . For material models (Tiers 1 to 2) Initial independent validation ─ Where independence may or may not require the different reporting lines set forth for banks: effective challenge Documentation (model theory, assumptions, limitations, code, user instructions) Change validation Periodic formal, documented calibration Periodic re-validation given results of calibration . Model management for Tier 3 models could be lighter in terms of frequency, intensity, formality of independence . The challenge of third party vendor models

Global Association of Risk Professionals

111 Town Square Place 14th Floor Jersey City, New Jersey 07310 U.S.A. + 1 201.719.7210

2nd Floor Bengal Wing 9A Devonshire Square London, EC2M 4YN U.K. + 44 (0) 20 7397 9630

www.garp.org