Model Governance

inancial modeling is increasingly default or delinquency used in the important to the banking industry, loan approval process and risk Fwith almost every institution pricing now using models for some purpose. ■ risk models measure Although the use of models as a manage- risk, monitoring earnings exposure ment tool is a significant advance for the to a range of potential changes in industry, the models themselves repre- rates and market conditions sent a new source of risk — the potential for model output to incorrectly inform ■ Derivatives pricing models estimate management decisions. asset value, providing a methodology for determining the value of new or Although modeling necessarily involves complex products for which market the opportunity for error, strong gover- observations are not readily available nance procedures can help minimize by In addition, models play a direct role in determining regulatory capital ■ Providing reasonable assurance requirements at many of the nation’s the model is operating as intended; largest and most complex banking ■ Contributing to ongoing model organizations. Some of these institutions improvement to maintain already use value-at-risk models to deter- effectiveness; and mine regulatory capital held for market risk exposure.1 At institutions adopting ■ Promoting better management the Basel II capital standards when final- understanding of the limitations ized, financial models will have a much and potential weaknesses of a model. expanded role in establishing regulatory This article briefly discusses the use capital held for all risk types. of models in banking and describes a Not all models involve complex mathe- conceptual framework for model gover- matical techniques or require detailed nance. In addition, the article suggests code. This possible areas of examiner review when does not, however, diminish their poten- evaluating the adequacy of an institu- tial importance to the organization. For tion’s model oversight, controls and vali- example, many banks use dation practices. that capture historical performance, current portfolio composition, and exter- Use of Models in the Banking nal factors to calculate an appropriate Industry range for the allowance for loan and lease losses. Although at first glance this Fundamentally, financial models may not appear to be a “model,” the describe business activity, predicting output from such spreadsheets directly future or otherwise unknown aspects contributes to preparation of the institu- of that activity. Models can serve many tion’s reported financial statements, and purposes for insured financial insti- some controls are necessary, given the tutions, such as informing decision seriousness of any potential errors. making, measuring risk, and estimat- ing asset values. Some examples: Model Governance ■ Credit scoring models inform deci- sion making, providing predictive Institutions design and implement information on the potential for procedures to help ensure models achieve their intended purpose. The

1 Institutions with $1 billion or more in trading assets are subject to the 1996 Market Risk Amendment to risk- based capital regulations.

4 Supervisory Insights Winter 2005 necessary rigor of procedures is specific ■ Bank staff or external parties with to each model. An institution’s use of appropriate independence and expert- and reliance on a model determines its ise periodically validate that the importance and, in turn, establishes the model is working as intended. level of controls and validation needed ■ Internal audit tests model control for that model. For some simple spread- practices and model validation pro- sheet models, controls and validation cedures to ensure compliance with may consist of a brief operational pro- established policies and procedures. cedures document; password protection on the electronic file; and periodic review by internal audit for accuracy Supervisory Review of Models of the data feeds, formulas, and output reporting. While procedures will vary, With the industry’s growing reliance certain core model governance princi- on , regulators are ples typically will apply at all institu- devoting additional attention to model 3 tions (see Figure 1): governance. Examiners do not typically review controls and validation for all ■ The board establishes policies provid- models, but instead select specific ing oversight throughout the organiza- models in connection with the supervi- tion commensurate with overall sory review of business activities where reliance on models. model use is vital or increasing. ■ Business line management2 provides The evaluation of model use and adequate controls over each model’s governance often becomes critical to use, based on the criticality and the regulatory assessment of risk in the complexity of the model. reviewed activities. For example, many

Figure 1

Model Risk Governance Framework Board and Line of Business Senior Management Management Controls Change Model Control Inventories Data Integrity Controls

Policies Oversight Model Documentation Model

Roles and Outcome Responsibilities Validation Analysis

Developmental Process Evidence Verification Internal Model Validation Audit Resources

2 Providing for appropriate controls may be the responsibility of senior management at smaller organizations. 3 OCC Bulletin 2000-16, “Risk Modeling,” (May 30, 2000) is the primary source for formal regulatory guidance on model governance available at www.occ.treas.gov/occ_current.htm.

5 Supervisory Insights Winter 2005 Model Governance continued from pg. 5

banks have completely integrated the ing model validation (see Figure 2). Such use of credit scoring models into their reviews also would consider findings of retail and small business lending. the bank’s internal audit staff relative to Model results play a significant role in these areas. , contributing to the deci- sions to make loans and price loans for credit risk. Model results also typically Model Oversight are used to assign credit risk grades to When evaluating board and senior loans, providing vital information used management oversight, examiners in and the determina- typically tion of the allowance for loan and lease ■ losses. Therefore, examiner assessment Review model governance policies of credit risk and credit risk manage- to determine (1) if the policies are ment at banks that use integrated adequate for the bank’s level of model credit scoring models requires a thor- use and control, and (2) if validation ough evaluation of the use and reliabil- procedures used for individual models ity of the scoring models. comply with established policies; and ■ Although the supervisory review of Review the bank’s model inventory model use and governance may some- for accuracy and completeness. times require quantitative or information Model policies: A single board- technology specialists for some complex approved policy governing models models, examiners can perform most may suffice for many banks, although model reviews. Even when specialists are those with greater reliance on financial used, model review does not occur in modeling may supplement the board- isolation; the specialist’s evaluation of approved policy with more detailed mathematical theories or program policies for each line of business. Such coding is integrated into the examiner’s policies typically assessment of model use. Regulatory ■ review typically focuses on the core Define a model, identifying what components of the bank’s governance components of management informa- practices by evaluating model oversight, tion systems are considered subject examining model controls, and review- to model governance procedures;

Figure 2

Suggested Framework for the Supervisory Review of Models

Regulatory Evaluation Regulatory Examination Regulatory Review of Model Oversight of Model Controls of Model Validation

Model Developmental Policies Documentation Evidence

Inventories Data Process Integrity Verification

Security and Outcome Change Control Analysis

6 Supervisory Insights Winter 2005 ■ Establish standards for controls and tions and potential weaknesses, and validation, either enterprise-wide (2) operating procedures; minimum standards or, alternatively, ■ Review data reconciliation procedures varying levels of expected controls and business line analysis of model and validation based on model criti- results; and cality and complexity; ■ Evaluate security and change control ■ Normally require verification of procedures. control procedures and independent validation of model effectiveness By conducting their own review of before a model is implemented;4 and model documentation and controls, examiners gain a stronger understanding ■ Generally define the roles of manage- of the model’s process flow. This under- ment, business line staff, internal standing enables examiners to test the audit, information technology staff, findings of the bank’s validation and and other personnel relative to internal audit review against their own model development and acquisition, observations. use, controls, and validation responsibilities. Model documentation: Documenta- tion provides a thorough understanding Model inventories: Banks of any size of how the model works (model theory) or complexity benefit from maintaining and allows a new user to assume respon- an inventory of all models used. The sibility for the model’s use (operational inventory should catalogue each model procedures). Each model should have and describe the model’s purpose, iden- appropriate documentation to accom- tify the business line responsible for plish these two objectives, with the level the model, indicate the criticality and of documentation determined by the complexity of the model and the status model’s use and complexity. Generally, of the model’s validation, and summa- elements of documentation include: rize major concerns identified by valida- tion procedures or internal audit review. ■ A description of model purpose and Periodic management attestation to the design. accuracy and completeness of the model ■ Model theory, including the logic inventory is a strong practice to help behind the model and sensitivity to ensure that the inventory is appropri- key drivers and assumptions. ately maintained. ■ Data needs. Model Control Practices ■ Detailed operating procedures. When examining controls around indi- ■ Security and change control vidual models, regulators procedures. ■ Review model documentation for ■ Validation plans and findings of (1) discussion of model theory, with validations performed. particular attention to model limita-

4 Banks may sometimes face compelling business reasons to use models prior to completion of these tasks. For example, trading of certain complex products often relies on rapidly evolving models. Management may, in some instances, decide the potential return from such activities justifies the additional risk accepted through the use of a model that has not been validated. In such cases, management should • Specifically approve the temporary use of an unvalidated model for the product. • Formalize plans for a thorough validation of the model, including a specific time frame for completion. • Establish limits on risk exposures, such as limiting the volume of trades that are permitted before vali- dation is completed.

7 Supervisory Insights Winter 2005 Model Governance continued from pg. 7

Data integrity: Maintaining data program to authorized users and appro- integrity is vital to model performance. priate information technology personnel. Much of the information used in a Control can be maintained by limiting model is electronically extracted or physical or electronic access to the manually input from source systems; computer or server where the program either approach provides opportunity resides and by password protection. for error. Business line management is The institution should have backup responsible for the regular reconcilia- procedures to recover important model- tion of source system information with ing programs in the event of technologi- model data to ensure accuracy and cal disruption. completeness.5 Change control may be necessary Data inputs need to be sufficient to only for complex models. Such proce- provide the level of data consistency dures are used to ensure all changes and granularity necessary for the model are justified, properly approved, docu- to function as designed. Data lacking mented, and verified6 for accuracy. sufficient granularity, such as product- Events covered by such procedures or portfolio-level information, may be include the addition of new data inadequate for models that use drivers inputs, changes in the method of data and assumptions associated with trans- extraction from source systems, modi- action-level data. For example, the fications to formulas or assumptions, robustness of an interest rate risk and changes in the use of the model model designed to use individual output. Typically, proposed changes security-level prepayment estimates are submitted for approval by business could be compromised by the use of an line management before any alter- average prepayment speed for aggregate ations to the model are initiated. To mortgage-backed securities held in the maintain up-to-date documentation, investment portfolio. staff may log all changes made to the model, including the date of the Security and change control: Key change, a description of the change, financial models should be subject to initiating personnel, approving person- the same controls as those used for nel, and verification. other vital bank software. Security controls help protect software from When model importance and complex- unauthorized use or alteration and ity are high, management may choose from technological disruptions. Change to run parallel models — prechange and control helps maintain model function- postchange. Doing so will assist in ality and reliability as ongoing enhance- determining the model’s sensitivity to ments occur. the changes. Changes significantly affecting model output, as measured Some level of security control is gener- by such sensitivity analysis, may trigger ally appropriate for all financial models. the need for accelerated validation. Security controls limit access to the

5 For example, the regular verification of data integrity for a value-at-risk model likely would include the following: • Reconciliation of trading account exposures in source information systems with model inputs to ensure that all trading positions are being captured and accurately incorporated into the model. • Reconciliation of model outputs with model inputs to ensure all data inputs are being appropriately used, with particular attention to handling missing, incomplete, or erroneous data fields that serve as risk drivers in the computation of value-at-risk for each trading position. 6 Optimally, all changes to models should be verified by another party to ensure the changes were made accu- rately and within the guidelines of the approval. This does not constitute validation, but merely verification that approved changes were made correctly.

8 Supervisory Insights Winter 2005 by loan officers and loan review staff, but Model Validation also look at original financial statements Validation should not be thought of as a and other documents to verify the loan purely mathematical exercise performed was properly underwritten and risk by quantitative specialists. It encom- graded. Similarly, examiners review devel- passes any activity that assesses how opmental evidence, verify processes, and effectively a model is operating. Valida- analyze model output not to validate the tion procedures focus not only on model, but to assess the adequacy of the confirming the appropriateness of bank’s ongoing validation (see Figure 3). model theory and accuracy of program code, but also test the integrity of Components of Validation: model inputs, outputs, and reporting. ■ Developmental evidence: The review Validation is typically completed before of developmental evidence focuses on a model is put into use and also on an the reasonableness of the conceptual ongoing basis to ensure the model approach and quantification tech- continues to perform as intended. The niques of the model itself. This review frequency of planned validation will typically considers the following: depend on the use of the model and its • Documentation and support for the importance to the organization. The appropriateness of the logic and need for updated validation could be specific risk quantification tech- triggered earlier than planned by substan- niques used in the model. tive changes to the model, to the data, or •Testing of model sensitivity to key to the theory supporting model logic. assumptions and data inputs used. Examiners do not validate bank •Support for the reasonableness and models; validation is the responsibility of validity of model results. the bank. However, examiners do test the •Support for the robustness of effectiveness of the bank’s validation scenarios used for stress testing, function by selectively reviewing various when stress testing is performed. aspects of validation work performed on ■ individual models.7 When reviewing vali- Process verification: Process verifica- dation, examiners tion considers data inputs, the work- ings of the model itself, and model ■ Evaluate the scope of validation work output reporting. It includes an evalu- performed; ation of controls, the reconciliation of ■ Review the report summarizing valida- source data systems with model tion findings and any additional work inputs, accuracy of program coding, papers needed to understand findings; and the usefulness and accuracy of model outputs and reporting. Such ■ Evaluate management’s response to verification also may include bench- the report summarizing the findings, marking of model processes against including remediation plans and time industry practices for similar models. frames; and ■ Outcome analysis: Outcome analysis ■ Assess the qualifications of staff or focuses on model output and report- vendors performing the validation. ing to assess the predictiveness of the This process is analogous to regulatory model. It may include both qualitative review of bank lending. When looking at and quantitative techniques: loan files, examiners do not usually rely • Qualitative reasonableness checks exclusively on the review work performed consider whether the model is

7 This review may require the use of quantitative specialists, depending on the complexity of the model.

9 Supervisory Insights Winter 2005 Model Governance continued from pg. 9 Figure 3

Ongoing Validation

Process Verification Developmental Outcome Analysis Evidence

Model Output Reports to End Users Model: Various Extracted Theory Source Data Drivers and Assumptions Systems Calculations Model Output Data Used in Validation

Mapping of source systems to Model structure and Back-testing/benchmarking model inputs performed to assumptions are results used to improve model. ensure accuracy and optimized. completeness.

generally producing expected Optimally, validation work is performed results. by parties completely independent from • Back-testing is a direct comparison the model’s design and use. They may be of predicted results to observed an independent model validation group actual results. within the bank, internal audit, staff with • Benchmarking of model output model expertise from other areas of the compares predicted results gener- bank, or an external vendor. However, for ated by the model being validated some models with limited importance, with predicted results from other achieving complete independence while models or sources. maintaining adequate expertise may not always be practical or necessary. In such cases, however, management and inter- Expertise and independence of nal audit should pay particular attention model staff: The criticality and to the appropriateness of scope and complexity of a model determine the procedures. level of expertise and independence necessary for validation staff, as well as Validation work can incorporate combi- the scope and frequency of validations. nations of model expertise and skill The more vital or complex the model, levels. For example, management may the greater the need for frequent and rely on the bank’s own internal audit detailed validations performed by inde- staff to verify the integrity of data inputs, pendent, expert staff. adequacy of model controls, and appro- priateness of model output reporting, The complexity of some models may while using an outside vendor with require validation staff to have special- model expertise to validate a model’s ized quantitative skills and knowledge. theory and code. The extent of computer programming in the model design may require Third-party validation: Vendors are specialized technological knowledge sometimes used to meet the need for and skills as well. a high level of independence and

10 Supervisory Insights Winter 2005 expertise. They can bring a broad findings into their assignment of super- perspective from their work at other visory ratings to the bank. financial institutions, providing a useful For example, regulatory guidelines source for theory and process bench- forrating the sensitivity to market risk marking. When using external sources component under the Uniform Financial to validate models, appropriate bank Institutions Rating System include an personnel should determine that assessment of management’s ability to vendor review procedures meet policy identify, measure, monitor, and control standards and are appropriate to the exposure to changes in interest rates or specific model. market conditions.8 Any significant Banks sometimes use third parties for examiner concerns with the effective- validation when they purchase vendor ness of a model used to measure and models. The validation of the model monitor this risk, such as the failure to theory, , assumptions, and validate the model or a lack of under- code for purchased models can be standing of model output, would have complicated, as vendors sometimes are some negative effect on the rating. unwilling to share key model formulas Conversely, if the model improves inter- and assumptions or program code with est rate risk management, this would be clients. In such cases, vendors typically positively reflected in the rating. supply clients with validation reports Other component ratings also can be performed by independent parties. Such influenced by model use, such as the work can be relied on if management has evaluation of credit scoring models’ adequate information to determine the effects on loan underwriting proce- scope is adequate and findings are appro- dures and credit risk management in priately conveyed to and acted on by the assigning an asset quality rating. The model vendor. Management may also management component rating also increase its comfort with vendor-supplied may be influenced if governance pro- models through a greater emphasis on cedures over critical models are weak. regular outcome analysis. However, management cannot rely exclusively on a The use of financial modeling in vendor’s widespread industry acceptance the banking industry will continue to as evidence of reliability. expand. By necessity, supervisory attention to the adequacy of governance practices designed to assess and limit Supervisory Evaluation of associated model risk also will increase. Model Use and Governance Bank management is responsible for Robert L. Burns, CFA, CPA establishing an effective model gover- Senior Examiner nance program to recognize, understand, Potential bank governance practices and limit the risks involved in the use of and supervisory activities described in these important management tools. The this article are consistent with existing examiner’s role is to evaluate model use regulatory guidance, but represent the and governance practices relative to the thoughts of the author and should not institution’s complexity and the overall be considered regulatory policy or importance of models to its business formal examination guidance. activities. Examiners incorporate their

8 Relative to the evaluation of a bank’s sensitivity to market risk, the FDIC Manual of Examination Policies states, “While taking into consideration the institution’s size and the nature and complexity of its activities, the assess- ment should focus on the risk management process, especially management’s ability to measure, monitor, and control market risk” available at www.fdic.gov/regulations/safety/manual/section7-1.html#rating.

11 Supervisory Insights Winter 2005