Model Risk Management Quantitative and qualitative aspects

Financial Institutions www.managementsolutions.com Design and Layout Marketing and Communication Department Management Solutions - Spain

Photographs Photographic archive of Management Solutions Fotolia

© Management Solutions 2014 All rights reserved. Cannot be reproduced, distributed, publicly disclosed, converted, totally or partially, freely or with a charge, in any way or procedure, without the express written authorisation of Management Solutions. The information contained in this publication is merely to be used as a guideline. Management Solutions shall not be held responsible for the use which could be made of this information by third parties. Nobody is entitled to use this material except by express authorisation of Management Solutions. Content

Introduction 4

Executive summary 8

Model risk definition and regulations 12

Elements of an objective MRM framework 18

Model risk quantification 26

Bibliography 36

Glossary 37 4 Model Risk Management - Quantitative and qualitative aspects MANAGEMENT SOLUTIONS I n t r o d u c t i o n In recent years there has been a trend in financial institutions Also, customer onboarding, engagement and marketing towards greater use of models in decision making, driven in campaign models have become more prevalent. These models part by regulation but manifest in all areas of management. are used to automatically establish customer loyalty and engagement actions both in the first stage of the relationship In this regard, a high proportion of bank decisions are with the institution and at any time in the customer life cycle. automated through decision models (whether statistical Actions include the cross-selling of products and services that algorithms or sets of rules) 1. are customized to suit the client’s needs, within the framework of CRM 5. First, the last few years have seen an increase in the use of automated electronic platforms that execute trade commands which have been pre-programmed by time, price or volume, and can start without manual intervention, a system known as algorithmic trading. As an example, an automated trade command that took place on May 6, 2010 resulted in a 4,100 million-dollar “flash crash” of the New York Stock Exchange, which fell more than 1,000 points and recovered to the same value in only 15 minutes 2.

Second, partly encouraged by Basel 3 regulations, banks are increasingly using decision models (consisting of statistical algorithms and decision rules) in their origination, monitoring and credit recovery processes. Thus, whether or not a loan is viable is determined by estimating the probability of default (PD) of the client. Similarly, banks monitor customer accounts and anticipate credit deterioration using automatic alert 1MIT (2005). 5 models, pre-classify customers and determine their credit 2SEC (2010). 3BCBS (2004-06). limits; and, in credit collections, they develop statistical profiles 4Capital asset pricing model. of delinquent customers in order to apply different recovery 5Customer relationship management strategies.

In the commercial area, customers are able to select a product’s characteristics (loan amount, term and purpose, insurance coverage, etc.) and the system makes a real-time decision on viability and price. In many cases, the model asks the customer a number of questions and proactively makes the offer that best suits the customer (doing this manually would be a slow and complex process).

The use of valuation models for products and financial instruments has become widespread in financial institutions, in both the markets and the ALM business. Some classic examples are Black- Scholes, CAPM 4 and Monte Carlo valuation models.

Another area where the use of models is more and more frequent is fraud and money laundering detection. Bank and regulators alike use models that identify fraudulent or money laundering-oriented transactions, which requires combining statistical customer profiling models (know your customer - KYC), transaction monitoring rules and black lists. 6 Model Risk Management - Quantitative and qualitative aspects MANAGEMENT SOLUTIONS u p l i w w H e M 4 4 4 4 4 O c a o c M w m t t T p d t f I r e ( T a c M a t i n n F o i h h h h o o a q n p r h h x s p s s e r r o t r h h i d

o o o e o r e e a e o o t p e k p m n g h t u d p i e o v t m w a s i i

i d d d h t d

j c

i t c c o s e s

. m a c e T D A e e a O A h d e i v

n r g U O m o

m e

b

u d u p r e e i e e e s e e e o e h h u 1 o l g r i h f x x r s a e s e g u b u c f i b . l o d f e s f i r e s o

o

l l l v r l q S r u a a i a )

p t

l s f v t g e e t v m

i p

e

i e t s s i a t v s a s . e t p j u e

e r r i c m e y n d a d . t i n c l l u r o e h e e t e c

x h t s , u r . o d i i i o

r i t s c i e r r

r m

a r h s s s h t b o o n r e e e e a

e e e e c

e a r h m l e p i . n m r b , i t n

r y o e r s k k o s l s i

s g e n t g f l t l r u g o n

n

m d t

y e a e u r o e u a

.

i s t

o s

r k h u i e c u p s a

t n

e o a m m t t a e n r v

c

c h e v s

n t T s ( n l s s s m

o o a » w

f h t s e i

p s t

m c o a n n t u e e e m y f a

n e i a d s m

i e h i

c 8 e

t s e t m n

e o a

t

t u n r t l n . i o

s s f

d g t e l s t m p , a

e e t

l

h e i e n d e b h d e

t y

s

e

g f d e a h

o

s a f y o t i i u r t t r e v o e s . i t p b o s r t e d o m n n

e y e

i n e s

r i t , e y n

e e n i b d t t

e a d

a a s r s d s f m o u l o a e

a s

r

i i c t c c i h

t c l

l

C t t r k , g r t

n n t u e p n d m i e i

u s a

b o s f n

. l n s a n i u h a e t n , e n a e u i p m u . o s r o

c u c e c s f g l n t

e e e w r a t m b s r e s d o n e

e d c i c i

a s i i r i d l d e e m d c i o i t n e s d

d n t l n s u o n o o

o l t d r m m u d t m o l

a i i o d o i

b u e e a i q

a n b i i n e n

i g

g e s d d t m o n o j p z e n c n

t r o b s s m l r

d e e r p s d i e

u - y g o g s

o

s c n e e i e l n n k

m m g t g t

c n m a t y ; n c

i o s e n p

a

e a

a d

: r

s

d

h n e e l e e t a -

i n e t s p

c t

r t o

g

p

f i s l c t p t s i h r m p i e h a r d t e e s r

e e h (

e l n

t o

s

a h e o

u e s s l i i c e ,

r

l p i i e l i k p l l q d l e s

f

f s

9 a a e o c n m e e h t g o m s h a a e e i i s r o r n . c i i i k

l

r n i

o n o n c n u

6

e e s n , f y g m f c n k

e a o r q

e o u c p c ;

d

a i o F p p

s t

t

s t r a u e i d m i o i n g t e u a n n u a n o o u n r n n

e e n r h o e c e u e r i l l l f d s e r

t n v e l , l o i i e m s y r

, f a o t r o n d n g s a g

d u i c n a r f n l

e

t

e i a

s o c t v d s a x i n 7 e n t t h a s m u n v t r t o e

t , b e d c t . g s

s i t n t h m h

a n l

h i u o c

a

a s t s e e w r l a a

i v t a n i

r t r t v ,

a i o l s i s o e o e t

d c a h c a r t a a e

e

f i g t e c l « d a o t - i

s u f e h s i t d l i

t

f n h i r t a l d n n c h e c d o

t r y c s C m n u i e o

d i i i

a n r i o h a a a

R a u f v e n a

m r

c e d a u

c n

i m m u d e n f c f i n e v e b t t m n n n n e e l o e n e n s r c s t e h

s

o i , s i r

s e l d g i d a i t t

e r i g o

s s o d e

o m

d s t d r s r

o

A , g m p i e i o o s l e r m s i

e e

u

e e

n n m u e n n

m d

n e e u y a a h t o n w n u t o L

f m i a n r v n l s e v c m s a n , t h

l p n x e

e s a u u v

M

c e c t s i n

t e

e e t o o o e c f i n p n t a s a l ( e a e d t w p e u i n h m a t

i . o

u r l M d y g r t r o s f t r , m a n v s r s t g r o n y

r p

r v

a i r k s d

c o o S a

t e

i t n a a t g r o a a n ( d e e

p i . t R m d , e g h

i a e s u p o O l y n

r e c t

s i f t l n i e x s

c s i

b m M a n , k e i f c t o i a n a d s

n e t r l l f c

d

o n t i i C t m ) a e

e p l s

i . i i t

m a t t c l r

a i s d o

e ) c

m o s m t ) e a

d e

g a m

m o

o C s

f

a a n s c

h b – r i k l e a c m t e o n t f n

n m n a u e

r i ; r n

r h h a o p k r ,

i i e h

c a n o a e s ) y

r

t s a t s o t w l d t a d m

n s e t ’ a f r t

i

r

n i

t i , i n , s t

e – s i s a g o u

c

m

n ( o

o o i s o a o i e

r

y a ; h t s

w v k F

n r

t o h

d u t i g c n u s n s h

n

u m v d u n

t n s n

e ’ o i e t w e g k e r u a a t s l h v h l e , t u e d i s g o

c - a a e d

a d r w s e p r e n

s : e c r m r , s h e o s s h i f

h r l

e t

r

e ) y c u t

h

o u d w r e l h b o e i s 1 e o f a

s

h y s o a l c 0 a o s t u r e t d

a d a r e u n n e f h

t h e l h e d a s k i r r t s h e

l t n d i

e n n e i t e c a n n e s i d d l h l r l 1 1 9 8 7 6 J e v t 4 4 4 t u b R A d n i c c e t I O F S 1 0 h m P h h a b x M I o e s e o e b s a e r g e M C l p i a r u o t e e r r d o i m t g

t p C o a d e

s i t o a

t u a e . d a

m M u A D I p w i r e m s u - a . l i o t

e r x t n p F e i i n u e r d m i b g x i c l a o s r l a d

e f e l e a i r n , t a r p g a o

s s a e n

m

a l

a n e r d r e s

v a a s t l n e t r e i t

a c d

u a r

m o t s

i e p c o o

o p i

( h a

i y m x g

c k i i t t o 2 n c l l e s f n d r o f d l r n

u l

i h a e c e t

e

0 p a i a e u e r c a n

o r d l i m u b n

2 , e t 1

r i c d e n i

l o p t a l s n r s c s t s m d i

0 c 1 e c p s d i i h k s e h o s i s r t l n

n n i 1 s e - s . i u

i i b s d n u o t o e r i t 1 s r i

t n a n 2 e k s v n g g a W s a

i i

t s

o c 2 a o l u o s . s L i p

s ,

t g

e t l t S

n )

t s

t T m o u a

e n c

i

a m . r w $ e i o p h o n s

b

n h i o n t

a c k n v e m g 7 n

m , v l

i h a l a l d d u u

t o n i e t g s o 0 i

n d i h i t

h e v e h

c o t n t e

n a o 0 e s f t f

h r d i t a

e

r a t

n f a w o

i , e s o a a

d v h

s d l t

y d e

h o n l e l u w

U n t r l u r i

e a W g d a u s i

i e r r i i a i l u e e

s S l s s o r

o m m s r

f l

l a h e d w

t n i h l i

r f l

(

p u i t g v i c o i

f 2 s a i n b n o i o m n a i

r r i d a

n s m p n a u t e c i e 0 u m e l i t n

) t a h l r t e k 2 s ,

1

h t e e i c

a r s e

i

s b

s v m i

n t a 0 k n a i

m t 3

n c s o i d

c s o q e n h c d 0 h j f

f c

a t ) r c

J e

t . b i t d i r

a 8 j m h n i u s i t r o t u a c i i h n u r

n z a c e . o

s a h o p y e e n

e

l a

e s i d s i d , m

t a c c o n l

t n u

e t r

l f s g i y n w

. p O m e

r t u a d

, e i

d

t r g g u

t

2 n u e a t a h l T r t u s e i e l e j h m 0

e i s

h a o h l i u w C

l s e r f n v c t d

1 p a i k l i i t t g r

e

h v h e s s t s

c d 3 s e o a t y i i e

s r r n t

i o t o m k

; e a e i

c i n a r r

d d m q r t ( o u s s q r a a

i e A s a t n

k c e

i a s k i m o u , u a n i t v r i V u l

l n e

t e

o k m w s i

a s e a s f n b c i i s A o

q a g e r . n

r o B t c : n t q h

u a a i s e o e d n s u n a i I v a

t s i )

S n

u m g a c

i

, m ( e m s n s m t l a

t l

t n h a d o (

s g

i

i e . o 2 w n d d r ( i

t g e s e n e

m w m i m

0 m e a s

w t t t t

e n p

d a n e 1 a p a

i i o t a s f m g f p s 3 i o y t e h s i r

t r p l i r i

v i

e k ) o

n o c

a t n o o a d t i i ; o e a e d o

e s c h v s a o v n t i c r t e t

n r c

t e h

d i i r , e e t t : i s t e p

n n i l

r i d w i a d i

o i t

t i e

f

o o

n x c t h r $ t o g v E r i n e h

n l f o i s e e a 6 e u n B

b a

t s

i

t t t

r .

, f c d

l A e a r k 2 d c h

i l

h a i a a n h c a u a s

n . e t

s e ( b p c e c

A i s k n u 2 h a

s e t o c i f t

p p 0 l

a V i t a e r i a p l o r t d a i n 1 d i e r e u r o o l

A o e o r 3

l y t

o i d s n c a o

c

t ) h n s e s i z b

. a n t t i

n t n s 1 s i o a i

s o c 1 e

n n o i

o

t n b a h ) i g f f n , l e Fig. 1. Model cloud: the size of each term is proportional to the number of models whose objective is this term

7

Source: average of several financial institutions 8 Model Risk Management - Quantitative and qualitative aspects MANAGEMENT SOLUTIONS E x e c u t i v e

s u m m a r y This section is intended to summarize the main conclusions particularity that it cannot be removed, only mitigated reached on model risk management in financial institutions through critical analysis or “effective challenge”. (which are elaborated on in the appropriate sections of this document). 8. Regulators expressly indicate that, while expert modeling, robust model validation and a duly justified though not excessively conservative approach are necessary elements Model risk definition and regulations in model risk mitigation, they are not sufficient and should not serve as an excuse for the industry not to continue to 1. The use of mathematical models by financial institutions in work towards improving models. many areas is rapidly gaining ground. This brings significant benefits (objectivity, automation, efficiency, etc.) but also 9. With regard to model risk organization and governance, entails costs. regulators do not prescribe a specific framework, but they do address the need to establish a clear distinction 2. Among these costs is model risk, understood as the loss between model “ownership” 15 , “control” 16 and (economic, reputational, etc.) arising from decisions based “compliance” 17 roles. on flawed or misused models. 10. The Board of Directors is ultimately responsible for 3. Thus understood, model risk may arise from three basic approving a framework for model risk management (MRM) sources: data limitations (in terms of both availability and and should be regularly informed about any significant quality); estimation uncertainty or methodological flaws in model risk to which the entity could be exposed. the model’s design (volatility of estimators, simplifications, approximations, wrong assumptions, incorrect design, etc.); 11. Finally, regulators emphasize that the fundamental and inappropriate use of the model (using the model principle in model risk management is “effective challenge”, outside its intended use, failure to update and recalibrate, understood as critical analysis by objective, qualified etc.). people with experience in the line of business in which the model is used, who are able to identify model limitations 9 4. There has been little regulatory activity on model risk and, and assumptions, and suggest appropriate changes. with one exception, regulations in this area refer almost exclusively to the need to make valuation adjustments in 12. This comprehensive approach to model risk is novel in the derivatives, the requirement to cover all risks in the internal industry, and the expected trend for the coming years is capital adequacy assessment process (ICAAP 12 ) or the use of that the industry will gradually take it on board and the Basel III leverage ratio as a mitigating factor of model implement the prescribed practices, as the most advanced risk when estimating risk-weighted assets for the institutions are already doing. calculation of capital requirements via internal models 13 .

5. The exception relates to the Supervisory Guidance on Model Risk Management 14 published by the OCC and the U.S. Fed in 2011-12, which, for the first time, accurately defined model risk and provided a set of guidelines establishing the need for entities to develop a Board- approved framework to identify and manage this risk (though not necessarily quantify it).

6. These guidelines cover all phases of a model’s life cycle: development and deployment, use, validation, governance policies, control and documentation by all involved; the documentation aspect is particularly emphasized due to its 12 Internal Capital Adequacy Assessment Process. importance in effective model validation. 13 BCBS (2010-11). 14 OCC-Fed (2011-12). 15 Ownership: refers to the model “owner” or user. 7. Among the main requirements is the need to address 16 Control: function that validates the model and sets limits on its use. 17 Compliance: includes the execution of processes that ensure the other two roles model risk with the same rigor as any other risk, with the are performed according to the established policies. 1 Model Risk Management - Quantitative and qualitative aspects 0 MANAGEMENT SOLUTIONS 1 E 1 1 1 1 3 7 6 4 5 l . . . . . e B m a d m i W i o T i a R i a t i A f W t d u m s I W t t c t f m n n n n n n r i h h h o h e l s n n h n i a f p o i u a a s c s s

v t s a a a t i i h p

r e a a p t n d d e t t a m c t d e d s k e t t s e l e o n n t h t t

i h h h e a u i i

s u n k e

o r l

t n a

e i t t M l m c e M a a e n w k e c r n i t

e e d s v u u m m c t r f r r

a o , g g

t e

w

r h r i g e

a e e e e e R i t t o t a i a t c s T b t o s m n l e e e i e i i l d h a g g s a ) s e

s M l i n n a o r o o

e a l , r

n

p m m

r n V

s g n

e

a a a , b s t s p

e a a a r c n n e n u a

t o s i t a o b t k s e r r a i t r e h l g t s o l e e l a n ’ ’ z

k s

d d a

h i o a l s s

i n v a m p u t a

p f a i n u n n i i e a n n , d t

r r

d t s n e

l e p r i

m m t e n t t i e t w m r a e o a t t e

g z

e a e n

o o a

r o i e c p g ,

c i q c c f l t , a o d l

( a

n o o

h l t g i h o

e s e r h M t o n M m r e l k u y t n u m o i n l

o

s c

o d d i t o

n i t a g t

e r )

a s c t a o a

i r o

’ c R r o

R b

o i g v s r e e r s n o t o

u o s y g a l h e i r l . n M e a f M l u

b M

r a e

u a l l e , ,

n d

f a m r

a m

s s s

e

c b d c o

a u w c l d i n e F i p a a a l R .

n d e

n ) n t

a l l o h r

p i n o d o i

a

i s n y j y l e p i n M i l i g

n f g b z s

n l

n , z

e a o d d c r

d

o c r d l

m e a o a v b o a a l

y a a s

i s

t h e l s v e d e t c t s s z t e , f a r i n u n

i n m s t y

h y a h r c i d t o a : t

a l e l a e i n n t i o

t a c i

c

s h

e o s i n g ( e n e t z n g n

d b t h i t e e t

h e a n m t b h d e h n i a a e r v a n

o c g o

e

w o s e n ) m ,

s s w y b

e

n t

g r e

e

a , i r e m B v e

o e e

v

a

m

i n m

o M

y t y t a

e n o d s e a o h w e s r o e t n n a h h

g

r l

e v c m

u n

o M o r n i B o i

l a R p a d k

a M i e e o n t

a

n . a g a n a

d t a f c ) u i s r s

M n p s

a e r l t e l , t

h

a d u g M l e o

c l r

u m R e e s k s o h

d n t r c , n b

e n e

b o

l

u o i d h t o . l

a f r r b a r o a n R

d t

t a y M h m o c u v a o r i e v i i e v i t , w t t n y f t n M e

d s s

e e i d e s n o e s

s

e g a t l a n

i D g s

v : i i o h

p s

a ( e r t s r e c R l x r

b

s o n l s g f r

s o m e f - n ,

e

r

i e s o t , p o l

i i

u ( o c l r v g f r i

s e

v s n l s i

t e v o a o e

u e a n i o f l s u e a a , n a u M k k

a a n i

t e n

t w t n p l c n c a p s n t i t r , b

o c l t

l c o e n

l m O c e t

n t h i i R i s e l c n c

r

t n o . d b l t r o i i o s

e d t ; c i o i t a M a n n c o l y t f y

s o , f e a a i y r i ( f , f h h

e e i n v l

d

o e g

h

m b s R

i i f r i t t

c n t o

t c n n a e e t d i

c f s d h i u n

i h w i ) o h c a y o

r e n f n m l s h

e o i e i

e c

s a s r

e a a n n v

o n k r n i t a f g , i i i l f

i n o m

u h p e s s e o g w , a d n r i c t (

m m M m

n k

a

o M a u I r r n e r l , g

r g

t a n e

i u i c a

d e i n

u o R c t o o o r n c p k n i M

w R i t n d h e c a e d o

g a t M d d d g e o g c O a i

e

n t o o l n h o ( r e a e e

r s

a l i s a n d , t ) e r d

t e l t n l l l n i , t h l k

r l l a a r e o d l e s s o e

i l t o l o s s l e k s f l 2 M 1 1 0 8 9 . . . o B s q i d r m v d d i t n f d o M i a m a m t T o r m b m m d m t d o e e o h o a n n h e r u e u e o o e s e e o s g o a o e e a l d

r m

r e d d y o w f v p g l t c c b a f o p i n o d n n a d i d d u c e u o a i u u c n

u i l l e u e a

e s i u t w n e a o e t e l r e e t m

n i n n m m t g a

i u i t

e

c r g i g m

s r l n f m l l o i t o o d i - i d d

t s

y h f t e r n e i o o n o h v u e s e e i i n i r

, e n s a

i e e n o c p e

.

o h c a n

g

d m t n

n n p

m a e

a c m i s p a k n T h g

n i l a l f o t s a g e s n t t e

i d ) i o c

t h s e s e t c e n d

s

a a o ; e h

u n e l i s

t t d f

a

n c q e

a o n n a e a

t t r a i y

a a d o , o ( l n i r t n h a

i i o n d r t z w n

n n t d

s ( ( t e c o o i u

e u f t

e p c 2

3 o

a e i i m s e

w d r h d

a g p e t n o n n l i b c d l c e o t o ) ) m a

n t h c d l

n

r u

r c i n

o i y h

e

c s

q m E M c t t o o o t i

o e t p b

o n t l s o e c o e i

r l

h b

i a s u a n o

i f n

a c g t a d k r y l r i o

q s i n f u t

r o e h t e t t i f n e o s p m s n

t r

i

i g n o d i i a e u t q r a m i c

s n e u ,

a e n o g t m o

f

i c h f c e p i a r e e w

e w r o a u d u m

g t f c n a t e

r i i e o a a t c

n y l p s l t m h d p a l a

y c l l t

i h

s

t x h l v t h d t y

o i m p t i ( t i n r y t i

e l i , i n i m i a n o h i c e r a

r e e . n o o e e e

u s o e c

a t l m d a g e i d g n u

n s s

r

n p i t h c s n r p n b m s e o l i f

p

, ;

f o

d t

t u s i t

p

i o

o h r m

i m a d t a e o a ( c r p u

c i o n u t i c o

s o i c o a d f d o a c r g i a i n

r n o n u

u e o g r o n f ) o l d

r m n t t k n e u e t y a u o

d m t n c t t ; s e m d

n c t i d e n

o r l ) (

l

t y o n r

e e

e s , a

d s o ,

c e

e s l t e

o o

m d

t i o e n

r w e s n

m n s h t a h h m s

l d o

t l t d w e b t h

w n o f

n

t s n o c e a a r e o r h i i

e

a s t e i e e m i s o p m t w e s i l d

l

i a i t s u e v

i s e e n v w l n

l t i i

m c y b

l e k n

s t

k r t c h h e l a e a d d e a r a t t h

d i

h u

n o j

o t i o l y v o m n t t a

s l

e a g g t . s o i

i i

o e r i e r g o t

h i o w t f t d t

d k i c n e e e o t k p b

n

o s

r

h i u r r n

t y v c h a a

g s ; g

m n i

r m m e o e m l i d

o i r l r

a r .

t

n y v a a s

l g a t i f o i

q c u i m

e n l e s

t o t e l a n s d o a e o n i r ) o e a f i r u k l g d

n

l n

i

c l

d n

n

a d a t o i w y o s s f i l b , r d i h z a t a h n

o a

r : n

t l i k

e d m f t

a ( o r s p e t i e s

, l g p e p (

w h

d d

s t e , l i k e a t 1

m t m

t d

r o h

e f i e u u e i r m a ) e a h l m o l

n ) e o

e

l

n t a e m

f r t d a i n a b o

r c r

h D f n l d p e i s i

t o t e r

e s y e t n y s f e e h m u

s e e a e o i o u d d k

p c

s i f

n

b e r t r d t c t s a y m r s t

s e o h i e h a o i t i

e i e a

e

r

n s l e v a

t d d a a e e r

o s e a l , i o r e a i i n

n s o e , z n n i n i d

c e s n d d l n a e g d e k h s . s l : 21. In order to explain this process, a study has been conducted in which model and parameter risk associated with credit risk models is estimated.

22. To illustrate the estimation of data deficiencies, the first exercise examines the impact of insufficient information in the most predictive variables of a scoring model for consumer mortgages. It is observed that the presence of errors in the model’s most predictive variables can either significantly increase the default rate or raise the opportunity cost (reduce the volume of acquired business) if default is fixed at the same level.

23. The second exercise is aimed at analyzing model risk arising from estimation uncertainty, using the confidence intervals of the estimators. To do this, the confidence intervals of a scoring model, the PD calculation and the LGD calculation (separately and combined) are used as a starting point to analyze how the capital requirement for a mortgage portfolio may become underestimated through the combined effect of uncertainty in the score estimators, PD calibration and LGD estimation.

24. Finally, the third exercise analyzes model misuse through a model (a scoring model, in this case) that was not updated for 12 months after it was first developed. This is done by comparing the model’s predictive power at the time of construction with that observed a year later, and assessing the impact the decision not to update it had on default as well as on opportunity cost (lower business acquisition levels). The observed outcome is that failure to update the model under analysis for 12 months reduced predictive power by several percentage points. The consequence of this reduction in predictive power was an increase in the default rate, or alternatively a reduction in the volume of 11 acquired business (i.e. an increase in the opportunity cost) with default remaining at the same level.

25. In summary, model risk can have a very significant quantitative impact that can result in management making poor decisions or even underestimating the institution’s capital requirement or provisions. It is therefore desirable to have a MRM framework in place and, where appropriate, develop robust model risk estimation techniques aimed at implementing suitable mitigation techniques. 1 Model Risk Management - Quantitative and qualitative aspects 2 MANAGEMENT SOLUTIONS M o d e l

r i s k

d e f i n i t i o n

a n d

r e g u l a t i o n s What is a model? As can be seen, the concept of model is broader than what a partial interpretation – more related to the mathematical When analyzing model risk, the first question that may arise is algorithm in a strict sense – would suggest, and includes expert what a model is and what it is not. models, among others.

According to the Fed and the OCC 18 , the term “model” refers to However, this definition leaves some room for interpretation. “a quantitative method, system, or approach that applies For example, under a definition such as the above, the statistical, economic, financial, or mathematical theories, following would no doubt have to be considered as models: techniques, and assumptions to process input data into quantitative estimates.” It has three components (Fig. 2): 4 An algorithm for the calculation of value-at-risk (VaR) in market risk using either a Monte Carlo or a historical 4 The inputs, which introduce hypotheses and assumptions simulation. in the model. 4 A score to calculate the probability of default (PD) of the 4 A method to transform the inputs into estimates. This loans in a portfolio using logistic regression. method will generally rely on statistics to produce quantitative estimates. 4 Mechanisms for the valuation of exposures, assets, instruments, portfolios, derivatives, etc. 4 A reporting component that converts the estimates into useful business information. Conversely, under this definition, it would be questionable whether the following could be considered as models: Always according to the Fed and the OCC, “the definition of model also covers quantitative approaches whose inputs are 4 Any simple data aggregation: sums, averages, standard partially or wholly qualitative or based on expert judgment, deviations, financial ratios, etc. provided that the output is quantitative in nature.” A constant annual growth projection based on a single 4 13 historical year-on-year change, without any other analysis being conducted. 18 OCC- Fed (2011-12).

Fig. 2. Model components 4 A decision-making mechanism based solely on the value of insufficient sample size. An example of this would be using a variable; for instance, a simple acceptance or rejection appraised value at the date a contract was formalized rule based on the LTV value. rather than at the latest date when feeding a model, because the latest date has not been stored in the data It is, however, for each institution to decide the scope of what bases. should constitute a model and therefore would be affected by model risk and the policies defined in connection with it. This 2. Estimation uncertainty or model error in the form of scope will at times be unclear and will require a degree of simplifications, approximations, flawed assumptions or subjective judgment. incorrect model design. Any of these can occur at any point in model development, from design to implementation, Once the scope has been defined, a decision will also have to which can lead to erroneous results that are not consistent be made as to the types of model that are to be analyzed (risk, with the purpose and intended use of the model. They commercial, financial projections, etc.). include estimator uncertainty (reflected in the confidence intervals, which are often calculated but tend not to be used), and also the use of unobservable parameters, the Model risk: nature and sources lack of market consensus on the model’s functional form, and computational difficulties, among others. Models are simplified representations of reality. This simplification is inevitable given the complexity of the 3. Model misuse, which includes using the model for purposes relationship between variables, and in any case it is a source of other than those for which it was designed (such as risk that needs to be identified, analyzed and managed like any developing a rating model based on a specific portfolio and other risk in an institution. applying it to a different portfolio, for instance from another country), and not re-estimating or re-calibrating Model risk, according to the Fed and the OCC, is defined as “the the model in a long time. potential for adverse consequences from decisions based on incorrect or misused model outputs and reports” 19 . S

N If data deficiencies are explicitly added to this (as they result in O I

T incorrect model output), the sources of model risk may be U L

O classified into three groups (Fig. 3): S

T N E

M 1. Data deficiencies in terms of both availability and quality, E G

A including data errors, lack of critical variables, lack of N

A historical depth, errors in the feeding of variables or M

14

s t c e p

s 19 a

Ibid. e v i t a t i l a u q

d n

a Fig. 3. Model risk sources

e v i t a t i t n a u Q

-

t n e m e g a n a M

k s i R

l e d o M Regulatory context However, most regulations either only mention model risk tangentially or model risk can be understood to be included in “other risks”.

To date, there are few specific regulations covering model risk, 4 Other mentions: this encompasses other minor references and these regulations tend to be non-specific in both their to model risk, which treat it as being implicit in other definition and the expected treatment. In addition to the regulatory elements. The most notable case is the Basel guidance document issued by the Fed and the OCC, which will Committee’s treatment of model risk mitigation through be analyzed in detail, there are some regulatory references to the application of the leverage ratio, though it does not go model risk that can be characterized into three types (Fig. 4): further into detail 24 .

4 Valuation adjustments: these are regulations governing the need to prudently adjust the valuation of specific products (especially derivatives) to cover any potential model risk. The main regulatory milestone, however, took place in 2011-12 with the publication of the Supervisory Guidance on Model Risk Although Basel II already indicated 20 the need to make Management 25 by the U.S. regulators. In this document, the these adjustments, the main regulatory milestone in this concept of model risk and the need for institutions to have a direction occurred in 2013 with the publication of the EBA’s framework in place to identify and manage this risk is clearly RTS on Prudent Valuation, which for the first time provided defined for the first time. specific methodological guidelines on how to perform these adjustments. This is the only case where the explicit quantification of model risk is covered in the regulations, as will be detailed further in this document.

4 Buffer capital linked to ICAAP 21 : both Basel II in its second pillar and some local regulators 22 transposing this regulation address the need to hold capital for all risks considered significant by the organization, as part of their internal capital adequacy assessment process (ICAAP). 20 BCBS (2004-06): “699. Supervisory authorities expect the following valuation adjustments/reserves to be formally considered at a minimum: […] where This is also the case in other ICAAP-related processes, such appropriate, model risk.” 21 as the U.S. stress testing exercise known as CCAR 23 , where Internal Capital Adequacy Assessment Process. 22 See, for example, Bank of Spain (2008-14). the regulator also suggests the possibility of holding a 23 Comprehensive Capital Analysis and Review. 24 capital buffer for model risk (though it does not require it), BCBS (2010-11): «16. […] the Committee is introducing a leverage ratio requirement that is intended to achieve the following objectives: […] introduce 15 which in fact some entities do in practice. additional safeguards against model risk and measurement error by supplementing the risk-based measure with a simple, transparent, independent measure of risk.” 25 OCC-Fed (2011-12).

Fig. 4. Main regulatory references to model risk To achieve this, the standard proposes a set of guidelines or validation; and compliance is the set of processes that principles for action on model risk, structured by area (Fig. 5), ensure the ownership and control roles are performed in which can be summarized as follows (Fig. 6): accordance with the established policies.

1. Model risk should be managed like any other risk; banks 8. The Board of Directors is ultimately responsible for the should identify its sources and assess its magnitude in model risk management framework, which it must approve, order to manage it. and should be regularly informed about any significant model risk to which the entity is exposed. 2. Model risk cannot be eliminated, only mitigated by good management. A combination of expert modeling and 9. The key principle in model risk mitigation is effective robust validation, while necessary, is not sufficient to challenge: critical analysis by objective parties who are eliminate model risk. qualified and experienced in the line of business in which the model is used, and are able to identify the limitations 3. Consequently, there should be a framework for model risk and assumptions and suggest appropriate improvements. management (MRM) in place, approved by the Board of Directors. In summary, regulators are encouraging institutions to design model risk management frameworks that formalize the criteria 4. A well supported conservative approach regarding inputs, to be followed in model development and implementation, outputs and model design is an effective tool, but should ensure their prudent use, establish procedures to validate their not be an excuse not to continue to work towards the performance and define policy governance and applicable continuous improvement of models. documentation criteria.

5. Prudent use of models may include duly justified This comprehensive approach to model risk is new in the conservative approaches, model stress testing, or a possible industry, and the expected trend for the coming years is that capital buffer for model risk. However, the regulator also institutions will gradually adopt it, as the more advanced warns that using too many conservative elements may lead organizations are already doing. to model misuse. S N O I

T 6. There are many sources of model risk and institutions U L

O should pay special attention to the aggregate model risk S

T resulting from their combination. N E M E G

A 7. While the choice of a suitable organizational model is at the N

A discretion of the institutions, a clear distinction needs to be M made between the roles of model ownership, control and 16 compliance; ownership involves knowing the model risk to

s t

c which the institution is subject; control deals with limit e p s a

setting and follow-up, as well as independent model e v i t a t i l a u q

d n

a Fig. 5. Structure and scope of Fed and OCC’s rule on MRM

e v i t a t i t n a u Q

-

t n e m e g a n a M

k s i R

l e d o M Fig. 6. Summary of the Fed and OCC guidelines on model risk management

4A clear definition of model purpose (statement of purpose) should be provided. 1 4Data and other information should be appropriate, should undergo rigorous quality testing as well as checks to Development and ensure its use is justified, and any approximations should be thoroughly documented. implementation 4There should be testing procedures to assess the accuracy, robustness, stability and limitations of the model, as well as a behavior test for different input values. 4The entire development process should be documented in detail.

4User feedback provides an opportunity to check model performance. 2 4Reports should be made using the estimator and model output values for different sets of inputs in order to Use provide indicators of model precision, robustness and stability. 4A prudent use of models may include duly justified conservative approaches, stress tests or a possible capital buffer for model risk. 4However, even the regulator warns that excessive use of conservative elements may lead to model misuse.

3 4The validation process should be embedded in a framework that: - Applies to all models, whether developed in-house or by third parties. Validation - Applies to all model components (including inputs, estimation and outputs). - Is commensurate with model use, complexity and materiality. - Requires independence from model developers and users. 4Validators should have knowledge and expertise and should be familiar with the line of business in which the model is used, and Internal Audit should verify the effectiveness of this framework. 4All models should be reviewed at least once a year and whenever they undergo material changes.

4The MRM framework, which must be reviewed annually, is developed by Senior Management and approved by 4 the Board, which should be regularly informed about any significant model risk. Governance, 4The three basic roles in MRM are: ownership (use of the model), control (model risk measurement, limit setting policies and and monitoring) and compliance (compliance with policies by the other two roles). control 4There must be a full inventory of all models in use. 4It is advantageous to have external resources perform validation and compliance tasks.

5 4The model should be documented with a level of detail that would permit a third party to understand the operation, limitations and key assumptions of the model. Documentation 4Where third party models (vendor models) are used, it should be ensured that adequate documentation is provided so that the model can be properly validated. 17

Common sources of model risk Model risk is present in all stages of a model’s life cycle: development and implementation, monitoring, validation and audit; and stems from three main sources: data, estimation uncertainty and error, and model use. Some of the common problems that generate model risk are summarized below, by source type. Data Estimation Use - Error in data definition - Estimator uncertainty - Using the model for purposes for which it was - Error in the mapping of data with the data - Model does not properly depict reality not designed sources - Mathematical assumptions are inadequate - Differences between regulatory and - Insufficient data feeding frequency - High sensitivity of expert adjustments management practices - Problems arising from data provisioning - Use of unobservable parameters - Extending the model’s use beyond its original scope (new products, markets, segments, etc.) - Data migrations - Lack of market consensus - Model not effectively used in practice - Accuracy of proxies (margin of error) - Computational difficulties - Model not re-estimated or re-calibrated in a - Insufficient sample - Non-use of confidence intervals long time - Insufficient historical depth - Outdated model due to parameter - Unapproved changes introduced in the model - Lack of critical variables decalibration, expert adjustments that are not updated, variables with no discriminating - Differences in model definition and uses capacity, etc. between the commercial and the risk area - Model instability - Model lacks credibility with the user - Lack of comprehensive documentation - Insufficient analytical capabilities - Use of new methodology not supported by academic research - Reduced Validation Unit independence from model developers 1 Model Risk Management - Quantitative and qualitative aspects 8 MANAGEMENT SOLUTIONS E l e m e n t s

o f

a n

o b j e c t i v e

M R M

f r a m e w o r k MRM Policy 4 Model risk quantification: methodology for model risk estimation, according to its nature and classification. Several international financial institutions have developed a number of key principles for model risk management (MRM), which are summarized and put forward in this section as possible best practices. The Board of Directors is ultimately responsible for the approval All of these organizations believe it necessary to create a MRM of the MRM framework. Also, and in line with the guidelines framework that should be approved at the highest level and to established by the OCC and the Fed, the Board should receive establish model ownership, control and compliance as the key regular reports on the implementation of the MRM policy and roles in model risk management. must be informed of any model risk that may have a material impact on the institution. Thus, best practice puts model risk on This framework should be embodied in a written and explicit the same level as any other significant risk for the institution. policy that has been approved by the Board of Directors and includes four main elements (Fig. 7): From an organizational viewpoint, model risk management is characterized by three components: 4 Organization and governance: description of roles and responsibilities, in particular establishing the Model Risk 4 Cross-cutting nature: model risk management affects Management Function (MRM) as a reference for all several areas within an institution, including the Business matters in this area. lines, Risk, Finance, Internal Audit and Technology.

4 Model management: guidelines on model classification, 4 Roles: MRM policy should define the role of each area in development, monitoring, documentation, inventory and model risk management. In line with OCC and Fed reporting. guidelines, three roles are usually defined:

- Ownership: this role lies with the model’s end-user 4 Model validation and change management: guidelines 19 for the review of models, the approval of changes, and areas, which are responsible for ensuring the model is on waivers required for using the model prior to properly used as well as for reporting any errors or approval. inconsistencies; the role of ownership also pertains to

Fig. 7 . Elements of a policy for model risk management (MRM) 2 Model Risk Management - Quantitative and qualitative aspects 0 MANAGEMENT SOLUTIONS 2 4 d t t I 6 t h h C e

s h e o c h i

s e e t M M m r m t t c U - - o e f e o h w n o

- - R . u

o R s a a

e S i t n o i t p n m V a W p l s i n r M d r . h d n t k ,

C l u t C i i c a f f p a e

v o

a e f i n n a a e u u h

g e w a

n o t o n l u O

s o o o f o l b r n g i i F c d l a n l

o e i l r e n z d e

r f d f n f C l n m d o h l l u e r R s f i i o i e k i

c e v r r u i n s n s e i c

l m c i t R c b e a

e . n i e i p l t p r e b d ,

p

n r d s i e n t e g

l

t i t m r T O

a e t ’ d o c F d i e o e r k o

i h a i s l e o d e g

o h . a l h l n i s t

i a

l

r 2

n i n

p , a d e

e n

t o : i M

c

6 t

n n i e f i t g t

l o b b c u e n d n s i . o d p

a e l h l a h a u

e s d I

a l

n d a M y y p

n c e n m l r o

m r n r

n e e t i s

s b a m

n

o t o : e

d d e n

t

s t r

t c d

s t R b

a r i o y e b o I a h e : v h l a e m h t e a n i e e n

s e

M r f y

r a

d g e e

d w r e l e t v o a

i e c c

i n t

y

e o m c b n d e s s s i e e

e t

e

e t l o a t s h

C a o f a

u

t h r d

e

w e h l m l n r a l m

r

t n a o

l s o t o n t o t d n a p e d R g a

h

e

o i h r

h i i t V n t s e o p m o i m o

t n a e l i r e d o e r r e n t R t e e s a h a i a n

o n l m r : d

v a

t

e v i n k

d i p r t n

e w m m l c

n o V t l s r

e

i e

e

m d .

d ( t

d w t e e e t e m l k f d r a f n i i i l h r i a o w

u n t i e u a p n s c s a

c i n o l e n o t a h l n a e d i

p n e n n e

t n l t e o o d a a d p s r g r y t

i a

e s o

c c c n b e o k c a t n o

r n

a i e

e

s s r o h l t n y e y c n t a . a r d n y d t m

o c u l n a

i i d e v a .

I e i c o c

s o

e f e

o o r n

l o r o d s e r f a l t o e t e i

e i n e - n n u u b B r e w r

n l h i r e , r r s v

r s y v i

s

t s p o d n u e i e t

w

e w . n b o ( e n

l i l

f s e h M t m i t l r d y T a c e s i u t l m y c t a e h h i r h g e o e i

t

r h s y e

n h

n u t a R r o i o e a s i u i a a

p

e s o e i s c c I t c e c e i

n M s t

d u

n n s . s c

h

m h d h s n s t

e c m

i s o i e

T e t o i r t n a i p t e )

o t

s

z o

o e r

i m i e

h s ( u l o m m b

e a

s e s a d g

r i o n l ) r c l n r n i n

d s

i s s y e . c t n s i a c a i s n r e r . p u r i t

s

i

i i

e e d l o y w t a n o t e

t f n e

i r k o l e p o i s i s a h t i o

l e n i c t n s c , g a o

e

s s

) z

t r h A e n l . u t r

, s s b

e . n i o a

l e r

a

l

E o t e l e

, a o u p

t y I

e

t B c

c n o m t u t o t a r t i n n i

r d

s e

s e o h u o r s s d

a m m s r

d

t

e a

i

m

s o

o s i n n n h w t t l t s s p y i i h o a p h f s o m d n

e

a

i r o l a

r e n a e t e r l e o M i e y h n r i n m s

o

t s l t a e e d e t s R s i t s a n n a

M

i l i d g i s l n n y T d T n w m 1 2 3 4 h h . . . . e i e f a e e e f l i e l n

d O e t u I a t v T t D l i d

M M e n n r h h h a a p i x

s a - - - e o b e v g t c v s e e e e l p R R t p o n i c e r l

j A i P C l e u a y i i d s u t n i e l M M i r r r

u t l m a n r n h o

, p i c o

a c d i v

i w d e i u o m i a h n t o g

e n n p t t n v e i e f f p o e s f t i t i e y

t u r n : d c e r v

n e o e e

k

a e l a e a r d a p r o a i l n s d e u y d n e d t n t

n u e r m n

e l i i v o o i c

i : d . c l y d n

t

o e s

a n

d

t o i r a d a t t M r t e m

i e n e v a o s n n i e h g y e o i n b t e o w n

s o i r g l i o . d l s d a e

o l n d j s

e o c e n a

g . l e d

o d r s

p i

d v

r t s

m n n u g

d e i

c f i u i h r t e

e n r a s e o p : t s d

k n t e u o

r e e

l a s n h

i l e a l t a e

i n

t e v s r p l

n i s n i d

a l : o h f i u n n r

o a e d

d l

n o s o t s g a t

i f w e s d c n n

m s s h

o s

g r r a l

a e r m l

s

l

t n t

o - u m l f e

n t m t o n e

o h l i u m d o o o h u o o u d p u d m f d d e o p r e e f

e

a w n l e o s o

h t

d e d v . m t s l

m a i t m h a s

r d o

n s a h h l

e

e a t p

t s d b e : t v e o r a

o o

o l e o l s

t p h o g

e e

s e l t d s o d d o

d i u s h i e t r n p c a

e n e

b c

o e o o a e l l

s n o t v m d s a l g l g

l j l l h

v h a

s e h s i u

e o t

r r u z d

e b m e e h s o

e i n p c i l g a a s s i e d d e n

e e u

t d t o M n k y t e i a r i v

o

f t i i l v e

b f a

l u d l r n i o d

e c u i o l o r e h

e l r c

e s

t t y n o

l i m s c d r d i s h o a n y i

e

n

; e o e n . h d , a

e s

a p d e

o a a d o d r f i t

i n o l v n s ,

m

t c n r c d

n i

. a f o R d

c o e c a

a t c o t y e l

u e e a i

n

h l l l m o

r s b a f . l

u n t n m

s

s m o d a k r

S t i e o

t d c s d t n t h

r

d o r

e O

o e s

l

d i e e g p u m e e l e n h m n

a d e f x d o c

a r v f g o t t p t o e o p l t i e r

u h e e e

c u u

u u l l r d c r t

y l t d s r e e i o e s l o o r e r s s e

d i

e r e p i f s i k s p r

w n l

d

t

s s e ( s o . t w h e

M l , e i h , r t

y s

t d e s i h a

i e h e n R t t

s e h h r f

p O a o i e s u ) r k . t 5. No redundancy: prior to the development of a model, the As it can be appreciated, the construction and maintenance of modeler should confirm that there is not an existing model a model inventory requires a considerable effort by that may cover the user’s needs. organizations, but is an essential element of model risk management. As regards model development and management components in an MRM framework, there are four key areas: model Model Tiering inventory, tiering, documentation and follow-up. They are detailed as follows. It is good practice to classify models according to the risk their use involves for the entity. Among other factors, the Model Inventory thoroughness required in model documentation, the need for validators to approve changes or the frequency and rigor of the Banks should have a complete inventory of all existing models, follow-up depend on this classification or tiering. with an aim to facilitate model risk governance and management and to keep a record of all uses, changes and the In a normal tiering process, the owner of a model suggests a approval status of each model. level of risk for the model, but the MRM function approves it and has the last word in this respect. The model inventory should: Model tiering is a partially subjective process that seeks to 4 Include all entity models within the perimeter of what the reflect the criteria through an institution estimates the risk of MRM framework considers “model” and in all areas: Risk, each model. As an example, tiering may depend on the Commercial, Finance, etc. following factors (Fig. 8):

4 Include information on each model’s tier, its 4 Materiality, which reflects the economic consequences of a documentation, its review status, intended and real uses, possible error or wrong use of the model, and that each possible waivers that are being applied, and any other owner quantifies in the most appropriate terms: exposure information that the MRM function considers relevant for or balance-sheet volume of affected assets, margin at risk, its good governance. reputational impact metrics, etc.

4 Be supported by a proper technological tool which includes 4 Sophistication, which expresses the model’s level of a single and centralized repository for the whole entity and, complexity: highly complex mathematical formulation, ideally, an interface that permits dialogue between the dependency on a large number of input variables, observed owner (including the developer), the validator and the stability of parameters, numerical approximations to auditor of the model. analytical expressions (e.g. stochastic differential equations), new algorithms or algorithms for which there is 4 Keep track of all versions, changes, waivers, documents, no academic evidence of performance and stability, etc. considerations from the validators and the supervisor 21 (when applicable), and expected dates to review and 4 Impact on decisions, reflecting to what extent model results update the model. have an influence on decision processes that are sensitive

Fig. 8. Model tiering example 2 Model Risk Management - Quantitative and qualitative aspects 2 MANAGEMENT SOLUTIONS o E w d T M t c t d c c I s t r h u a o o a h w o o

a i i o e n p t c m n s e c c n n h d

h

e t

u u c w f i w r f M n s b t n p e o a o r e e

r l f m m h o m e a e h h r v c s e r - - - g u r R l

i

c e s

l t

t i e o r a r D e e t u M t

r o s i o s e r w h d

h t e r n s n n o l t i e o d r r E d I o D u s r e a f

e

e n p h e a e d h f t t

r k d i s x c n e e a t t t e n e u

s a a

s i t l p a e

e i r t o o u i

s i l p s e . a i p o t d t t n t s n e e c ’

r n o y n

s r

i m i i o D

p a i b y l n a a s t

s o o s r c y d v

g o r

i u e n r

t

u u t p i t r n o e t t ,

n n a e e t e t t i

w i r t e i i l t a e p o s l a i t r

m

c o

n e

e n m ,

p t i n i t d n u o m

u s l r o g

u s

p

o m h n p , g t d a c h i

g

e

e t v

c e n s n e m a i o b w n r , a o o

r i e f o

n h e w

e t o n e n t , e a t

a t r n y

r m

o p t e m d e u w t

n i t h e n i t

e t c n

a h o n f s c

m e , l n a . l

t i e

a , t

d p l

s

d h d b e l e o e

a e n i

H n n o h t e o

f

i m s l o n i

y n n s m a t r n s

e d

f i c i v f e i

r a

h i , r t t

g d t o t t

o b

n t h a e t a r e d

l e n i t

a o t o e e s y h h r e l o o n n e h h l d n

n l r

i

o h

n

s t

o l p n e e

n r r b c f - a t p e e

e h c e : d r o o o

c i

h w r

a

a

g a t e n

y

p r r M e a e i r r d s m r r u h l e r o

s r

e o i s

c

e r e x f h p m t d t e e

u l w e

R o e c t d u a i r d i i a m

n a e i u e a v s c v

n e

M r t n o o . y l f i

d i r

n . a e s

a a l u n a s i s i o o e i

y

d u n f

e

l t l s a

l d t e l t s

i f d

f o h i i

s

i t e l . a e d i n n f j d u

u f t e o

d o o h u a e f p o f o n l d t e c n a e c n s e n

n

e l s

l m a

c k

e r ’ c

h t t c d c c c b t

t s a

. e i i h f e o i n m i t e t t

y o

a e e s f r e f a a d

i i y i f e s i m e

t i l o v r n n r

e m s s o o

o h u s u c c d

s e d n i

t d t , n d p t n t c a c t e r p

h o e . h

a e o e p

l a u

. p e l

g s b e d

a o d c i t t

e f T n a e l n m i

t v e c y i

a t

s t e p

h r s h m v w

e s s a e .

m e t p t l i

e t r e t e e u e s s o y l r

o e h o i r

n u

r

y

e t d , p p o n a a f e s u

a e d u t

s o o n

e s r r p c

a n a l

l o e e t f o e m u r d r t e r r o t i p t

d

r l s v o

e v i r m

e i e s

t r o

r o r o e

i i

i n

a h

v d o a m s e u s i t n n

d o s n o ’ e o i h f s

v . s s o

m r l e d p

d n r t T i s e y u e o a n n , h l e e r h

d ’ a o n m a t o

s s e l a e l h

d o f , l v b i r f y r e i e t e e y s h l d c k e t r A 4 4 4 4 4 4

m U T M M D T v d e p a o m f m i i e ( o d r f t b p c o n m e o r e e a a s e e x x b f e e a e l e s d a s

a o o s c o e l

s n l e t s c t e r s t p n r l j q t p i e u u h n r o t d d e e a d t a e d h c c a n r a c l h u l

e l m n s r w m c ’ e i s r u e e e ’ p

a n t o h s c l s m s o e i

c u t s e m o l l l t f p

t l t

o t o e m i

p , e i m n . a o d d e i d l v i c m

a l

s t o t d l o l u w o t e t n c d r a o o o e i i

s l a e a i m n o e d o l

n r n e g y o :

l s

l y g o d c r

c n r a i b o s s n f e , t , d , t k i b n u s

i

r i u

e

c n e

h u i e a t e y g c r

s s s

k p e n r s m o a s c s r e e d d c o u a a l

, t a y

, s i

e : e u

a l p f c

a n r n

q c l l a

d , t i

c j e

d

: r

r i i t e l s u . u i c t r

a e

a n n p g u i r

f t o o a n i e i h o a b n h o i s n s s s n n s d p

t a p b l n t r t e e p c

t e r v s o , r i

a v s a n a

u o i a t i t

m t o e d

s i i a e f l , g r t i i n e h b h n r t

i t i t n

o r s e n i s , d r t o c i o e y

n

o e o d e a e t s f l y s d

d n p a s e y n e

i o d f m d v

s n n u

, r c t

a c t e

o c p

. t n e m f i e

i o r

q , a

i m m a m a w t r a

t e s o r u r s d p t s a f o t a u t i s o s h e h n

n c t i l p e i : e p

o u e t i n e o e a t

a c l n o l t e t h i

n i t d h

r l s

l d m n a i n e

f m o o t u e i o

i o t o e t f e

o

m t n ) d

:

m u

f m f l f

y i

t n

o f e d r a i d n

a n m

t d u

h t e t s m

o e n , t f e n , s a a n e s

h

s e

r e h s i t t p d e o s

e t , d r t n n e n d o e p s o h

o

e t k r g x e s d e c w

,

d c t

d

u u o o g

f o

e t t d p d a r a u t l e o l o

i r i e c h t a n d u e a t p r s m t a l m e i l a p s r

t a o l t d

i i t e e s d o o e t

s t i d o l c i i h e n r i d a e o a b i

a r e o

l e r r b t e t o n e t o e r e t

a d

e i i n

u a n

h t u a h c l l o f o s f

a

g r t

e

h

o t o s t

s a c a t a o

s w i d i n r m t i h y w

t s o i e u r l r o e

l

t f f o f n e

u t i l ,

i e h o o

n

c f

b e d t o t n p i o a t n m s a e a t n

s h i r r r a h a r

m d c s r t h c r p a i i

s i a c a e e i l t l i a d s d

o e r h e s

o

u c l a p n e e i t t o t r

p

, a e d p

l a d i r i n

n h

r i c r r d l t s n o t d e s i p t

e i e i l t t h o i s d

h t , e e a a

k

t r a t n

i w e . l d h l k j n o e n e e

c d y e

:

u i l d . a s h

a e n i

. r c e n i t

i s

o t m i n p f t e p a t i

s t t e h t

o n

e a u l y o g t s p i i r d y o n f l n a g , e , r f l

i l

o d t c a i v

c i t a e t a l t e h i a e a e o w e n h n t n d e t

t y d l l d e c a d i i i v , ’ o n o y s n e n g n d s :

Summing up, documenting models requires a substantial effort - Analysis of sample stability through the evolution of by the entity but is key to facilitating the update, follow-up, variables. validation and audit processes, as well as the supervisory review. Consequently, it is receiving increasingly more - Metrics on predictive power development, such as the attention from entities and regulators. area under the ROC curve, the powerstat or the Kolmogorov-Smirnov distance, among others. Model Follow-up - Assessment of model behavior using techniques such as Finally, it is essential to have a decision-model performance volatility or predictivity analysis of each variable, monitoring system in place that allows for the early detection backtest of obtained vs. expected results or analysis of of deviations from target and for remedial or preventive action residuals. to be taken. - Comparison of results with alternative models or The follow-up should be carried out with a frequency that is industry benchmarks. proportional to the model risk (measured through its tier), and should consist of a series of alerts and objective criteria that will 4 Decision strategies: the behavior of decision rules make it possible to determine when a model needs to be accompanying the statistical model is analyzed; for rebuilt, recalibrated or decommissioned. Also, the final user’s instance, minimum conditions for acceptance, exclusive feedback is essential in model monitoring, as it is the most and restrictive rules or the cut-off point. effective source for identifying deviations with respect to the model’s expected behavior. 4 Expert adjustments: the impact of expert adjustments to the model is analyzed; for instance, the manual decision Model monitoring should be comprehensive, in the sense that when the automatic judgment assigned by the decision it should not be limited to the statistical or mathematical strategy is modified (override). algorithm, but should instead monitor all elements employed in the decision (which can include decision rules, expert For example, sound monitoring of a decision model in credit adjustments, strategies and any element that participates in risk would enable the assessment of many aspects, such as the the final decision) jointly. This is especially relevant and a usual model’s predictive power, the cut-off point efficiency in terms source of model risk, because it is the model as a whole that of default, the binding character of the automatic decision makes the final decision. strategy (scoring + rules), the proper functioning and implementation order of the decision rules, the percentage of Therefore, a comprehensive follow-up may examine the manual decision and, accordingly, the adequacy of the decision following elements, among others: authority, override and the classification of their reasons, the opportunity cost (denied or withdrawn requests), or the use of 4 Statistical model: the performance of the statistical other criteria besides statistical criteria (e.g. profitability and 23 algorithm is analyzed through: costs) in decisions, among others. 2 Model Risk Management - Quantitative and qualitative aspects 4 MANAGEMENT SOLUTIONS t e ( t r p T d i M c w f T y 2 1 v A t t o e h h h e a h o x . . e s r c i h o q l e o r F o

l p a a

e c c l p i o t i a o r d u d r p r n g o i e

l m S i T h

s o u u t

f e n w 1 e . i a o a r g r c h r -

i i r ------9 a u g r i s o d s e o i l ) t n e o r - e e o p f

.

t s

i T M Q Q Q D M g h v t u q

n

m i d s e o d p P i n s a r n t o i t e a , h

p o u s

o o o

e u u u n n

e a e c n u i a g t s u c a l t

t l , n e

l c u a a a i h d e m n : t i

e h

t a r s c s h

d n s

c s w h u n t i i l n l a t d g

e , ’

a i l y t e o s h i i n t

o d a i e h y a t t o t m

c i q t s

h g h

i c i

s b n t o a y t e f o s a n

e d

i e d

y d

l t

u o t t i e e n t a i

t e e

n u e i r o l g

h e

h a s o o o t s s o i o n v a r l m a . e

n v

l d o l

n t

h

e e e c v ( f d

a d a o C l n n g d f u s i e F e

o t

u

a r e v o i

a t n l s t M

w q t r i d o a i u r s i f o

i c h g s c s e h l n : i m e g u a

d f d e t

i u t n

r d o

o

m e a i a d e y

i i g s e t a a l . l i a f n

a n d o a

s o d n e o , l c m l t u p .

a 9

t

u t

l e

s d g e o l r k t n i t e n w i

g

c t i l i e t ) n l p i e v m u l i o q

p f e

h a s

e s n u . a v t t

a c i q a

e t e n e n m n k t a M y e u o l e d t e m v e t o s y n a u r

c s i

t

d d

s

e

a p l a i

o

e r e

d s o f

i h i d t y r b i r

w e n u o e t n n t

o c h ( e s a o l n i a d e

a y t

e u y s n a r a a . k s n t l t c c o i r n s n l v

l k g e

g m l t h e s t l l n s i o o h t

a

y u

y h b m e

d e r a l e a o

s d c a m d

n f w ,

t e

r e l

d : t u

n

m

o

d t h . d t

a m t e

t

i a q t n

a - o i o e t o d e h n h e

a m

n n l u o e u n t n s y v o o d s n

e t p t e

t d t p s

d i u t s

i e o r a d f l e .

r

o

m l n , s s o e s t r v

v

e o e k l m t l o e , b e s

o h

s p a l

a b t l e m y m a

s f r l

b q u

p h o b t l s l o r e o

m , e y i i h i t

e s u

t e d a m o u d e n t e c l

a s o

e a e i p

n a e t i a d u r l t a t n d p s

t d

r e t

c y r n p t t c s t

s o e o p v v

e o e s n i s

. i a

i o

t r o ) b l a o b o f f g r n h s

s s

l . o t r

a a v r y t n o s e n p S e l , e e n t o i d

i v r c e i

l s f

o l

e n

r e e b a u y e e s e o t r s p u r k i n n t s e d i

h n

u l l s l

l

m c c c r t s e d y r

l i m s v p

o o c h t f e o h o , t i t

2

f s h i i i t o u e a c h m u e , i e w r a o p i k o

7 c e t e c l n c m

l e w l s n s l i i . e d

l e - i i h e s g

e c

e

o r a u

t n h f e o a s

s a o

n a n b o h l

u p l . s d t : i l c d

l e t n

t g y g r e e i l u s

o i l

r

e s

a o d y e o h m e t e r s v l n h e n u n

e

e t m s o d e d

c s c h r t a d t

t e

r o e u n a

i e r

a c d s t a d i l s t e e . k

. l

3 4 5 6 7 8 2 M 7 I ...... n a n

t a h H h d I V f i m R v t F b m I m n s O w d n n n g u i h t a r s e a e a i o e e e r a o f d t f e

o o e l e l r g v s l e

r l l f c e a i n e i r p

u q , t e e d d

d p

d e i d u a k s r d s h d d n p e e u r e e p n a o u

n a

e m s c e o a s a d n r e e e l l t s a

n t p i o s

f c n

t d c i i r z i e t c n n e s g l o o

o s o t e d h u

i t a o

u e , p n n c e d c i r

f n

r n s e M b n t a e o r s m

r f y v e r t e i

f

f m

i d e t i ) s o s e a p r : r o i f t n i l n .

s

e p o h c n

u t i e d t w a n r a t t d e o

c a h i d m o e o r n h s y n

e : i l g

e e t r i a

l t e

c a : c u c e a t h d n

. e w

a : n

n i

h t

t

n e t

u : r m l n v

t t t n

s h d i c d

q i h e s t

d h o n t s a

t e n d e h e s i

t

y

u h t s e

n b n l i a V o

u , s

a i e i

t p a

a t a

i

d t p

e s e p a

a

n i r m r i r u S n e l b n

s a d c

e t p l i

r p o s c i f t s s h

d t e s e t b d

s i t i l r i o i

p a i i u i c t n o

i

n o o

v t s o e a a t i c u r t e m a e

t n e i u v g f t t r n a n c a n o u s t

c i i

n t e s i t

m i n n

c i o o

a o u i n t w o p i n i

d f t s i o i s a n l r n f

i p f i

o n y o e e g s

c h i

n n l n o q a

t a

u i z a n n : a s

f a

h n g

d f

i n t t u u e s t i n t e i

t s r

t

i g s

i s h

s d d t e n d t a u n n a n i m i h

h h o

l

t e r l q . c e n f o r

e c g e i

e e o u t n e a

f o t u l

e

y w V h

n i M r e s i s u

t

i e d o . a

e c t n c p a e e t

g p n I l d o t R t o e a n n d l

o i a i o

i t o o w

n l r v P c c

d

p i h s n t n r e r s s r o e y r d e i ; o a e a g s e r s s

i t n l

e

t o a i l t i h s i l u ( p

c i a i b s e b e i t d d t a r p i f o y

e n u e r

d r s e l m l c e e . t o o e s e n i

, i r s o i a h a

v p z e a

n c z

r

o s u f , n

a f e

n e e e e

i t o d i u d a t o n d t n

l h s d n s h o o r i i n n e

p n d s n o s i M

d a p p r t l v c d

t o a u g n

d t h o i t i t r i m a n

h t l n

n e

d i i t

e s t u l l s o s p g a h e i e s

i h e h k d f t o n l a t i

e v s o n e i b . o o

a n o r

m e s r s t t i n u t e

n

l

l h i i w t i o . e u l

o

t h d t o

i f s i s h n e e t

u

t b o h e s l , e d r 9. Documentation: descriptive information on different aspects should be kept updated by the Validation function: As was previously mentioned, while model risk quantification is - Assessment, measurement and follow-up not required by regulations, in practice some entities are methodologies, as well as statistical model already including specific quantitative techniques in their MRM methodologies. framework for model risk mitigation purposes.

- Validation reports, including those written by Internal These techniques are applied to: Audit and those related to the validation process, with a clearly indicated conclusion (approved, approved on 4 Data, through sensitivity to error in variables and even to condition - waiver - or disapproved. the absence of key data input for model execution.

- Historical record of the changes done in internal 4 Estimations, by using output sensitivity to the volatility of systems, including the validation system itself. estimators.

10. Audit: the Validation function itself must be reviewed by 4 Use, by monitoring changes in predictive power and other Internal Audit, which needs to analyze its work and follow-up metrics. implemented controls, as well as to give its opinion on the degree of actual independence of this unit. The following section explains how this quantification may be performed, together with an exercise that illustrates its 11. Vendor models: models developed by third parties (vendor practical application. models) present additional difficulties for their validation, as the documentation may not be complete and the construction data may not be available. In these cases, the entity should request all necessary information and apply the same procedures to validate the model, and should do so with the same rigor as if it was internal, limited only by legal requirements.

25 2 Model Risk Management - Quantitative and qualitative aspects 6 MANAGEMENT SOLUTIONS M o d e l

r i s k

q u a n t i f i c a t i o n

Model risk quantitative management cycle 4 Mitigation of model risk identified and quantified by applying the appropriate measures, which will depend on Beyond specific aspects required by the EBA for the valuation the nature of each source. of derivatives (model risk AVAs 28 ) , as was previously mentioned, it is a fact that model risk quantification is not yet Even though eliminating model risk is not possible, explicitly required by regulators, which is why no significant implementing an approach that combines a rigorous risk progress has been made by the institutions in this field so far 29 . management structure, as prescribed by the Fed and the OCC, with prudent and detailed quantification such as that described Notwithstanding the above and, for management purposes, may be an effective strategy to mitigate it. some institutions have started to work on the model risk management cycle from a quantitative perspective, with an aim to support qualitative model risk management as Motivation and approach of the study recommended by the Fed and the OCC regulations. Seen in this light, a model risk quantitative management cycle would In light of the scant practice observed in model risk comprise three phases (Fig. 10): quantification in the industry, only undertaken by some institutions, it was considered of interest to carry out an 4 Identification of model risk sources and classification exercise in this connection. For this reason, a study was according to the previously mentioned sections: designed with a view to quantifying this risk in credit risk parameters and models. - Data deficiencies. - Estimation uncertainty or model errors This exercise is composed of three parts, which coincide with - Model misuse. the three model risk sources described:

4 Quantification of the model risk inherent to each source, 4 Data deficiencies: in this first phase, the impact of the lack using a methodology based on model output sensitivity to of information in a model’s most predictive variables will be potential fluctuations (in the inputs or the estimators) that 27 characterize the uncertainty associated to the source. 28 Model risk additional valuation adjustments (AVAs), detailed in EBA (2013). 29 The capital buffer is an exception in the CCAR exercise in the US, which some entities are implementing.

Fig. 10. Model risk quantitative management cycle 2 Model Risk Management - Quantitative and qualitative aspects 8 MANAGEMENT SOLUTIONS h W p 4 4 4 4 4 T D 4 4 h a a i a v r e t a h e

e a c c a t e a w w s m ( t m h t t T p C A A A M E a t b c b o ( w a T e e r b t s R t m e h h h h h

c

o a h n n s n

s h h r s s x a p a a a e e a t

y t b d i i i o O r g e o o m r r s i e e a e t t t l u h c t l t n a d d s a r s e e l

p p e s o c e e l i i i e o

e i h h u s t d t a r

u r l

e m m m r d i C , l b s

u t m m c u e h 1 b a a o o

i r t i

s e y f B s a c t i s

m e w m l t d e n t s g y t a a p t 3 r , l l

l h a 4 a s o z r r e n e i r a h c a a I

1

I a s l h a

r o o a g

p u t t S t i a u i c

e

t o w m a

e e

t o d w u t d t t s

v o e m e t m g u t h v s e o t d r p p

o e g i l o o i s d c 3 t r i

i r n m e

e l t . o t

t r t f i a 0 o n a e r o

a p e r t

t a e r m e t r n e l e r r g

a : r p u m ; i n e l d

y l f o n r i r a

g o i s f s s s n i o n r t y l l t

r g z o s o f l o s i a a t f d l e e

b

o u s e

.

r

o h f

o a v e e

r

e o

e y c r t

c o d r d u u d g l t o n o f

w i

m i i d t ) s e r b i d e i e o

m r a n r c

r c e e a

d s b o

e d

s n e u l e e r c u

e t

e o

a

p q a

o t k p l w i e f r

d e

m c h a e e o c o o

s t e l l e

c o c : t

n

e a r n m t e

l

l

r l u m o t x a i o . n c r h c l e i a

n w e t o s f i u ’ d s o t

f n l o d h

e a b d e v s l a i i T i r

i h f r

m a

t a r s e l r

n

a n o t e r

r a

a p t p

i n d o

i e

i a t

t i m

e o t t e p l n b c e a l y 7 n b t

a ( s t n x h a h r e n

n a y o c r s

a e c l

t

h h f

o

n i m o r m o 8 t e o u g u e i a l e

t d e p y z c i n e o a e

d e

n r e s i l p o n s e m h o i %

u t

m o y u t e

l s a p s

l

s e

l

s r g t

e c d a o m y a

i i u i o u

f e i t e i t i f ,

n h e d p u n n m t b o m n e s r o t

s e n g t

y )

m f i i r p ,

n r t t d f m o s . m

c e h d e p

. o i

u n t r

. m t

r h d s t e

a e p t s I s l

t

o i c a g t t . m r

f o p I

e

e p d T d o a s s

a h m

n o

e h i i n f

a 4 o

e r o T t r a f s v p s e s i a

m o s h l l r m e m a e i e n

o t p o n e %

d e r l c o t v e g o e

e m r r

t e c a i o r i t l t d o r

m r g s i o o

r r i o s h

, e

d

f d i e p n a e a s r i n f t

. b

u r d o m t p e n i f o

b

o t o t

e r r r o e k c

i s l

w h i

l t e a t h n u h i

p r n

c 2 o t p g o c m a e l u d d . r

t h n v

t r a ) l f e t

a a u s o e t

0 c i i i e n t

o i o e , o V w

. o t

a e

a l l e v l g

t m

i t

c s y o r t w e t l

l

u i a 0

v d p d t w h r a h r l l

c i a d e o h u P

o

e

i e d

p m t s r f

i , e o t m i e

r o o

i o e i t e y a

0 o f D t i n r e n r r i i a p t s t n i p e r s i

n o n n a

e

i r r

o e b

n h h 0 o p o ,

l n o s w

a o n t w u m , t l r

t t o l t i l g

g

a

w u

r 0 f e l o

i n o r n o f i o i t t v L d h d g r u t e t a i i w i o m o i t r

h

o i s d b l s w h G i r

t w l f y t f

e e g e

m

L s s s l l d m l

l

u :

i : r a m

h e

e d

e s u e

, a i k p i

i e e

G

b l D b a o i b

e i o n t t e w s f

p

t n e r o a n n

a e t t

e s f a h s p t g u o e u m e q r D

e o

, g

o i d

n e

g r a r l

a

a c

i c

s r o e t r i e

l s d r

i d t

l u i . t f

a r n a r e h o e n t a i

h d n r e

l s

e h M

i c n g

m d

d a

e s i e a i n m a n

s c t

c

r v c n o d a d e

l a t e m s i

t a

i d e e l e n c - a s t e o s o n v o a i s s , n h d e l w

e

g o e i

s o d c v t c

t i t C e t c s s t o l n o t n e t n c t

i e . y s a u h y i t u r e o u a e o h s n s r e

o t

s C f l d o n t s t e s i t

p , i z e

a m m a i

n

c l n b s r e

f e e

i r n u o o t c l F r a n r i m

e a m v e t r t i

o o u r o c s e

s

u a g n s

m r t m i e

C d P s s u a

k t l t e a n b y m i a l f t o I t a i t

g e d i o a r l D

. a c e t i m n i . e m e r m d s i y

a t u d o

o t y

t n T

t n r u - e e

a m s r

s d n i

i c l o h n e

n t o o o a p e p p

o c , o

d a r h e

t w d w l f i

t d o

w t n a l n r

s

g w

l o d t f i e e t i u

a t a i

i d o e r a i h o I s , s h h a i f h l o l

I . n

t p i t t l h

l I l u i n o e e n l , n e e I . R n e h l d t i l l f r d p B t g , 4 4 4 T M S t i q C q p 3 3 r c r 1 0 m e e o o R A u h u u r o s p n e

l o p m e p l a r a a a n c f

e o d p i

e F Q A r p c f p r o u c a e p c b d r 4 k n l n d a s s m i i n a a m e e a i e o o u e e l i t e s e 0 c b n e o r r u v

t n s t u s t n i i t a v s n n e e t i a r e p t n n t e q

l % n a e f c r s s i e f u i a r u l a t t , e t s - r a

n

d d i t m

y e i i t r f s e a e u m i

s f i o . m c

n n l a n o l f r a

u i

v . n g t i t i i

l t t r t r o g i

d

a e e l d f i c c p m o y t h l a o e i

i s v e v t h l n a f

t s f t t i l

n t a e e t t d , a u d

e e y

t s e a i f

i t u

t

i e

i i i p r i n e o e i r

y w n n t o e d t o r i s n v v i r p t a a t d o n d

t v l h l a l e o e f

i i d

t . b n e e c n

s y

i

( , g n m i f c

n i i e s u t a t s o a

d n t d i t i e e

i , a u

s n i y e o . n g

a

p v n h

h l n t n p t g

n e . n t

s c ( y a a

s

e

i t l u a t

h

2 g o

a i g

d o . o

t d t n ’ e t c , y t i e r

n i

a s c

r h n l s

i % t h n a e h w t s e m a n v a p

d s

i s h d h

, b i a

t c a e e

t

c t g e

a e

s t

f 1

e m ) h e m r 1 n f d h h a t b e e r l e r

s r o . a

l a

i e c a 5 h

r o s i e i

5 x s p i i e c

e

l m o s l o r m

a o q t a b u a a e %

t e a a e t m % d

i

i m o e f d p s d o p t

d u r o s a n e

t s d p

m r

a b i

b c i

e

t 9

i e e i n

t e e r

s n i d o s o r

e t t a d t l n o

o e a t s k 0 t e l t

l h s o a k r o

e n f c i

o a r i

e o s t c r s l

c .

m % r d r

s l e i d a

m

r i t o e i a : ) e

f i r c i s h

e n

n e

r s , s u

n q e m a e e i

o

, a r n u e r n e k s k s

t

o i

s e s e c a v c u l m t d n r e f h c c q t p t

s

c

e d h d o t

e a s a f a m u i t i u e

o m t a e i

l u o

r m P e s o s e e b

n r o u i t i d e

e u i s r

n m i i r e a o o

e D f i p l

o e s i n d r h i r m a

y t r

m

n e

y n e e 1 t

m f a i t

r o d

b a

e t

e 6 n n

M o

h i c

g i e a m

2 o e e e n a h t n u

e t y e 7 g i a p

e v r r f

o

r h n d s i t R

t a

s

m l t m % e o a e l n r t

i

a t h h

u r i , f

e t

h o s M v e r o

h b n b m b s e t r r

t e t o s o a

e o

e t s u i a o r o

i e h o r t a u e y

s

(

o n m r s t d p n

a a i

f c s t l m 4 u

s e p

i , f n l s

r u

v r

m y

u l m n e t o t i o u % b t e a t n i e y

n d p

i h o t

f h l

m p a d t m o r m b t

e

y

d m o

a a

o h s

d l d

m i i h g e t ’

n u o u c a

s s s y t

l

t h b r u e e e e t 1

l m

e o d e

i t o

n m r

t d o i a o o m ( e

v a i i n w s o f i 0

u m d s 7

f n n d y d a w e e a c b n b 8 i

n d e % a e n t e % o

o

s e u e r f y u a p e %

s d d t h

i m e a t i i l u d r r o s n r

l i t o . e b t

n

e ) o c

r t a k t r t u t e r a , y n w

T

f g n r o e e

e g

e t i a a a l e s u

r s , l w v

s e

h

c

e a

e t

: s c a

d d u e a n q t d f i

k d b s e i

r e t t c i f i n

o i n t r g s

s d s u e m m t e i

o d e l r r e a q

s m i e s e a

h

e t e f c

u s a c f i o

t

t n i u a

a n

o t a i a

r i n t e d n i ) t a s r a d s e h

i i t t

u

s h b d , n

r

o u t i i t t s i

a t L t o f e o i i l e o u e e t n o f c

t t G t t d n y d n a r r a r e

h n b

D r s t

.

t y i e i b

o e f v o y e r Quantification of uncertainty: confidence intervals

The estimation of sample parameters is normally accompanied by - A large amount ( ) of random numbers is simulated, the obtainment of confidence intervals, which provide a more according to an even distribution between 0 and 1, ~ (0,1). realistic estimation of the parameter since they represent a range or - For each { .. }, a full set of estimators for the logistic set of values within which the real value of the estimator lies with regression is simu푛lated through the inve𝑥rse of its respective a given probability (established by the level of confidence). As a distribution functions, = . 푋 𝑈 result, an interval with a level of confidence 1 – for a parameter is represented as: - The entire푘 p∈ or1tf푛olio is scored w훽ith each set of estimators.

This way, the distribution o훽f the𝐹 s–1co𝑥re for each portfolio record is α 𝜗 obtained, from which it is possible to estimate confidence intervals, volatilities and other measures for the quantification of the model There are two gen[₁,eral a]p wprhoearceh 𝑃es( ₁ for the es) t=im 1a-t훼 ion of confidence risk shown by the natural uncertainty in the estimation of a logistic intervals: regression. - Parametric, in which a known distribution of the estimator is Confidence intervals of central tendency estimators (non- assumed. parametric methods) - Non parametric, in which no knowledge about the distribution To establish the uncertainty associated to an estimator that has is assumed. been calculated using a central tendency estimator (such as a mean or a median, used in the calculation of LGD in credit risk, for Below are several common techniques for the obtainment of example), non-parametric techniques that do not require knowing confidence intervals of both parametric assumptions (taking a or assuming a distribution for the estimator are normally used. One logistic regression as an example) and non-parametric assumptions of the most common techniques is bootstrapping, which is based on (such as the calculation of central-tendency estimators, such as the random sampling and basically consists of generating n size mean or the median). samples to obtain the distribution function of the selected Confidence intervals of the logistic regression estimators estimators (mean, median, etc.) from all the generated samples. (parametrical methods) In particular, the steps to develop the confidence intervals of the Generally, logistic regression is used to relate discrete binary selected estimator for the calculation of a estimator would be the responses (such as the non-payment or payment of a debt) with a following: set of explanatory variables, which is the reason why this is the - Given the parameter construction sample composed of most frequently used algorithm in scoring and rating models. This observations, = ( , … ), a random훽 sample is generated to relationship may be expressed as follows: replace the original sample to obtain a bootstrap sample * = ( = ( , … ). 푛 - Then, the stati𝑥stic o𝑥f₁ inter𝑥est for the bootstrap sample * = ( *) Where is the vector of explanatory variables used in the model, is generated. is the probability that the respon𝜋se being predicted actually 𝑥 𝑥 𝑥₁ 𝑥 happens, conditio푙표n𝑔ed𝑖𝑡 t(o𝜋 ), =th a푙표t 𝑔is , = 훼+,훽 ´𝑥 is the - Steps 1 and 2 are repeated times being sufficiently large. 1-𝜋 훽 훽 𝑥 intersection at the origin of the curve, and is the vector with the Generally, it is suggested that reaches values over 1,000 𝑥 32 slopes of the variables. iterations for the calculation of intervals with confidence levels 𝜋 푌 between 90% and 95%. 𝑁 𝑁 29 When modeling through 𝑥a logistic 𝜋re= g𝑃(푌ress=io1 n|𝑥), th훼e and 𝑁 parameters, known as weights or estimator훽s, are obtained. Thus, - After the iterations, a bootstrap estimator sequence the main sour𝑥ce of model risk in a logistic regression is the error * , …, * will be generated for the statistic distribution to be made when estimating its weights, and confidence intervals may studied. 훼 훽 be used to quantify it. The distributi𝑁on of bootstrap estimators will enable the calculation 훽 ₁ 훽  Wald’s confidence intervals, also called normal confidence of a percentile confidence interval for ( – ) 100%, among other intervals, are based on the fact that estimators show a normal things. This interval will be given by the quantiles ( ( ), ( )) of asymptotical behavior, and admit a closed expression. As a result, bootstrap replications with and . Wald’s confidence interval for the estimator is 훽 1 훼  simply: 푞 푝₁ 푞 푝 𝑁 푝₁= 훼/2 푝=(1– 훼)/2 100 (1- 훼)% 훽 where is the percentile of the/ standard normal distribution, is the maximum likelihood estimator of , and is the standard error esti퐼ma=ti o훽 n o±f 𝑧 .- 𝜎̂ From th𝑧e confid1e0n0 c푝%e intervals obtained, it is easy to get the empirical dis훽tribution of the score for each of the logs in훽 the 𝜎̂ sample, which is the key element to훽 calculate the model error that is being assumed. The steps to do this would be as follows: - For each estimator, the function of the normal distribution is considered, centered in the weight estimator and with standard deviation equal to the estimator of its standard error. 훽 𝐹 =𝑁 훽, 𝜎̂ 32 Efron and Tibshirani (1993). 3 Model Risk Management - Quantitative and qualitative aspects 0 MANAGEMENT SOLUTIONS T a a F p p r I 4 4 4 4 4 T P m q n i n r r h h s u o e a

e o o F k t d e e r e i i

n h m v c

r g d

a b t a s a h c p p p W I I M M

s t r f f f t e i h t e . o t h t o c c s i n , e e

o y r r r w

s 1 r

a i n o o

h t

e e e e c c

r c n o v s

s f g w t 1 I a h

5 i o t o d d i e e

d d d e o u h t c : e n n s . r o

a s e

%

u

t

p p n S e e s u t i i i n l e l e

m

s w D g r c c c h t

o a t u t t g l l t t

t t i

: v x s

o t t t h

r e

g p m e n e s p w - ( h B A c u

h i i i e e e a i o

o

e v v v

d i o t . a e m a k ( : p c r c : r

c n e o

f e e e t

b d t s c s

t t d e t t h s t u f o o y a . b a

e

h h e - e a

i h t a o e e v v v

r t p t p s n i i l a y f n w , l e f i e n t

h - e c p b e a a a

r z y

m

o e s i

a o a t d

a o o e

t n t e r r r

o ,

t t s d s

i i

f

s c f h d i i i n a r I a n m n d o

d a c e o a a a c f u e I e

o o u

e i

l a i a o

d t e d o d b b b e c n p e

l s f c p f

n c

p

b d l a r a f u

c

r w

m e l l l r y u e o i t o p c i a e p s e e e e u

r u r

c c i s l s d t t r i r t a o r n e e

b , s

o o n e s i l o h e -

r r e a a e r i

e s t

k r a r o g i e e s a r e m d a s e s t n a e n

n e

e ) n e t e a s

a c

s d f

, s n e l

i u t o d .

a

t n e f e h e

c u c o r o n

s u f h

u

h m d l v r

n i n t s

p i c

p

o l d p

x l c r o i e e f c i h e t s

e o t n e i

n f e o o r i o e i e p t q e

s i w i

e n d n e r t p p v n

t m y c l i e d

r l h s p d o u l

n r ( g t s , o g t

r

t g e o b

e

F a

m r c s r r , e g a e t e o a t i

l

r a

w e t u i i n f t h o

u t f n r f a d f n g a n d i w u h r i u h r d

r o s

e s m n s s g o t t d

. s o i t e n e i c e

t b n e i r i o e

t h n i 1 m e c f ( b

m l m

h

i n e

e r i o ? y

y g F

d t e e e t 1

s w a e u x

g y s

i i i

f

w t e e s n ) t n v e g n r

t c

r n m s

. h

i f h s n e c r h d f e

e t t t i g o s . i o T e c a

n

a h i h x o e m e m v u e

w

1 i r

r o w u u p o s t e e t e

o e s a o i

l 2 e m

n h m o e

l l t n o o t d s i

r

u d d l t ) s i m p

m l u r g e s v

p

v w : i

o y t

o

o

r e n i

o n

i m t t

i n v a l

o i n o d m t n a e

d h h i c 9 m r o t o e t h

d g c d e t r r g e t h e e

e e t o d f l p

e e h h p e l

e

u o

l

o i

t t o . t d f d a

e s

e e (

l a l h b m h m f o h l m . t ,

s b q e i e r

s c y o a e y a n l t e c e f l o l u o t e t w o p l

:

t o e

a

e , a

m 1 t i

m

a s o d e s u m n o a w h t w i o t . l n h

f 0

e s o

l b t r r o

I

t i t

e e m

a

e e 5 m e l n m s o t t e s , a d g e

s ) o a t g t

r m o , o

s c o t

?

e i r

2 m a n h u k d o o d

d o n e e a r e r r e e o d e e ) d e d t l l , l s s

d e d p o t l f 4 A v m v a o s o c F r

l c o u i i d g a k a W o p o 1 o i s a s m . e t b e l . f b r r

n

s 4 1

h a l i e e l t e - g o t 2

0 e l b d e h

a p .

m h

i s )

t m n n F i e e i , i

h n c a

n i - o

a

a r t i s s r o a e u s g f l n s i e

e t

t

v d r l v t

m

t d

t c i e e t e

t h

e g i d h x o b

h p n

o r k e e i v o e n y n y e , l r e d y

i a

c u

s t

e

a b e

4 s e m i r h t a s b s l i i p r 0 u l

e a g e r

m u l o t

% a e i s e b r

n h n c p e d i l

d o n l s s i

s t e g e r f e ( o u i ) u d e i e i

n

s . l c . d l t

f

s e e t s , m g h a a f

s e s e l . k i : p n

e c

f n t r t C e u m a i

t p h e h h e e b c l

u t d e e r o e a e n p u i o l n f

p u

d t

t i o m s f o v g

n

c e l e r i t p i e e f n a g e e o c o l

p s

i e e t s t s

n o

d . t e s i

r o

s

n h b t t t e r

s h r h o h b o v e l t

t e

v n e a e y r a u w

d d s o i

l n n b d 1 t i

e i e l d i t h i u f n e . u e t h f a 0 a e a m y d f

s u o 5 t t a

u

i

l h d e c n . u t b e u

l

o e e r t e t

y t l a o

t

s f h s r

t m t

a t t b a s r e h e h

u a

t i t o v i e v s e e t r a l s t o

s e

. i t m u

e

t o n

l a O h

b u s q r p e n u p r e i m y e u d g r p d l

t e e i o a 1

i

e i n b n d b r p

l . m t

9 y a i u t u t l c i 8 o h

l n e s o t

t

m

i e i h i d ( t s t n v i y h e t

. o e e e b a s d . y t

s i e t l 31 3 Model Risk Management - Quantitative and qualitative aspects 2 MANAGEMENT SOLUTIONS s e e t 4 4 4 4 T P h i h s n m a F r t e d o i i m u g r

c c s L t s c c U c c T e t c 1 a b F F S F , u s

. h h e i t l a o o a a a a G c b r i r i h s 0

o e m s a 1 n a m g o o

e e n t l l l p l c n m , i i a c 3 D t I t c i c c t i 0 a n s h m m o

b h s m u i i I . i s s o i u u u e o

o t

0 l o

l b g i

: S w r e t r l

l e a a t t n n l l l r

y t a d 0 n a u a a n

a a a i i h t t d U i r s l h t n v a d t , m g t n h h

t t ’ i s l

t t t

t h e s

P i n e y a i o e i y a e e e e

o r i i t e e o n t m

a e

m e n D , d e n , g s n

r d y i d d d

n

s t n r m x a , s e a t c c

a c , d

h c

e . . , p a

h t t e a e t s

r

u o o n l t a i e s o

W k o o i h

e p

t e o g h

s r e t z p s n n n

s i i n e c r

h

r e

e m t l r o r c

g e i o u t e i e i i c f f e s c d e o i t t t c n i e h t o i i n s r r i e s r r

n r f a a

h d d u p r i v a e a a

e e g o

i f e r t

t r B m l p n g c

o e e i p l i e i i r f

g t i t l

s a t n m i u s s e o I e r p i e o i i n n a r h

y

S o a t n n a t c e

a g c m r r

n r s i c r a c c i

i

s a o t l t o n

e a e n o a t s m t c

o t c a e e e l a i a t t o d i e , l ’

r k m t l a d n e y m

e n s h

1 n

i d c c i y t a

d P i s i i n p n

h

r u d s n n u 0 h d e f

s u P

o l t D a : t s f

i i t t i g

o

e n , . t t

l e a

d r t d y D l r n o i n 0

t o a t e e a f s o

a m n i c i

o

o e

m i 0

t b n s

r

t i r r t l s

t q m e s a c n e h r i n

v v

e e t 0 u h u t t g f c i s r o o n e u d

r a a t y o h n c d

t l

t e t a

i i m i t n d a d s a a . a e l l

n h i b r v s ,

l

h

i e o e

t m n n

c L s e f o n i

i

a i a e u c o n i L t i o e t n t u G o

n l d o t f

n l i

y a t G , d e u e t

v r

i s s

n l n i s t s

t f D d y

t n a c o s t i , l e t y D e d e h s c c h

t a

h t t

o o t 1 s

l i n i

y r o o a w e e t i e , s e s s n t ,

e v

m r 0

x c s i h . r r i ,

f u

t

o i

s

i

c a P c e o e i

, P m n h o t T w t P e 0 n a

n a f a r l h a D h p , u D g h

s

c D

r 0 g t c s r h d c p i P e i

, e e r s o a m

i r s

c 0 t e a s

m s i

e c D

a i i

e

r h a c M r o L t e

i

r r

p f t a t n a n p p s w a n t h e e

o G o d r e i c i l

a n o d l

a a i t c i d ( m

o r a , r t d n

a D b a i F a

d n o c r m o

m h n

k y l c e g a r i L r l n

t

i t

u n g e

e t e t , e a a b L

l m e G t e

i y h o

t m t n

s e s .

o s t p G e r d c

u , h

D i t e f t C 1 a p

e e i

o i s a r

i D , i n s o t e t

m n 3 t

t m

t a

u i c l h n a e k c t n i e

d i

i s ) r

o t a m c c h b l e d o e , w . o

r a l g h

e

t a f p o n i e t

e r T n

t d l e s s l a e h h t a p u

i i p o e s o a d t o e t

t i e i c P t a o i

c t i t n

l . n n t a

o t r s

a l h D L r h r u d

a . n

g n s l i i

G a i ’ s t s a

t s s , r i e k D . t s e e W A 4 4 4 4 ( c 8 a F a n % s i i u s F g

d t c i , i h m s

g . F T T A m s 8 f c e

a d

m i e

i . 1 a i h h l % g u n

s a m n u e 1

o p 4 l

a e e u i

a 4 m a a e

t l v b ) u i

d

e . t r i

t : s l u u

e i g e r e S l l s a e v o u i y a e n n e

t o a

n l e n u e o , t u t c

s c c

t l

r o l i o a t t p

u e b a e

o e e e t s h o n n

l

o l t d s m r r q n . y o d t d e

f i t t e

o

s u

a a r t

o e - f c t o r c

h o n i x i o i i s v o d f f r v o n n r e

e

h e e

n t o p i e t r m m t t n

h o h c d m y y m f w

o i u e

i e w s

, o b w d s

i i

e e n o p c

t n n d s i u e s

o h n n h r

d

i

b e e n m o b n L P t e t e i e s c l h

y c G r c l D b

u

d c l e u

h r

e e a

i l d o D 2 i

r n t - n

s e

c t r s i

c o u e o %

s t

w k : a c f c e

d a a w k l c f f , l e d a .

e s i a p i i s

a 9 t n b r a n t p i p

c h i m

t m 0 i t r t r i f

t m i a t

a i b y a a t %

u a a s i o o a l

c t n l y i l a

i a

i n , l f c v i t n v

o

t t s

t 7

t c o a e t g y i i

t

h n o o h t o % u r h

n

h i n u i e

f n e a u n n a c s r . e

p t

o e o l t d f

i c t d

t i o

h r m s u n s o e o h n

v c h m r r l d u r e

a

d o e ( e

o t m

l i d t e o r h

e s d n u i c e u o t v

s v e g

a p l i l c

t e w d m e t

e s u i a r i u

m p

s e a n s u u b a a n l t v e i p

r i p t s a e e c b m e i

e e p r t o

e t

) y c

o i d h

r a r a o u f c

o o r t o

e 4 s t a n s b

a n r a o

d

% p

f s c i y e s c r o n o

a i a

i s . h c t u d

l t w c p , a l o y .

h o p e t i l n

u t h

w i r

t a n t

l e e o b d o l s

d i y t s Part III: Model misuse - If, conversely, when applying the model at the end of the 12th month, a cut-off point were fixed so as to keep The last exercise seeks to quantify the impact of model risk unchanged the business volume obtained by the model arising from misuse (Fig. 15). Misuse may include applying a at construction (type II error), how much would the model to a population other than the one used to construct default accepted by the model increase? the model (for example, applying a scoring model built on the basis of a mortgage portfolio from a specific country to the The findings are as follows (Fig. 16): same portfolio from another country, or to the portfolio resulting from a merger with another entity). A special and not 4 A decline in predictive power of over 10% (8 ROC points) is so evident case of inappropriate use would be applying a observed after 12 months. model to the same portfolio after a long period of time without verifying that the model is still applicable, especially if there has 4 When the model is applied 12 months after construction, been an economic cycle change that could have substantially maintaining the default rate as it was at the time of transformed the population. construction reduces the business volume obtained by 15% (i.e. the opportunity cost is multiplied by 1.15), whereas To reflect this case, the following exercise is based on a scoring maintaining the business volume obtained by the original model, and seeks to quantify the impact of not having updated model multiplies the default rate by 1.67. it for 12 months. To do this: Therefore, the third exercise shows that the model risk arising 4 The model is applied to the portfolio at the end of each of from misuse (in this case, not having updated it) may have a the 12 months following its construction. significant impact on default and on the volume of acquired business. 4 The actual default levels shown in the following year are used to measure the model’s predictive power, which appears to decrease over time. Fig. 16. Third exercise results: What-if analysis of Default vs. Opportunity cost 4 Two what-if analyses are performed by moving the cut-off point to answer to the same questions as in the first exercise:

- If, when applying the model at the end of the 12th month, a cut-off point were fixed so as to keep constant the default rate that the model accepted at construction (type I error), how much would the opportunity cost

increase, measured as the business volume that is not 33 obtained?

Fig. 15. Summarized approach of the third exercise 3 Model Risk Management - Quantitative and qualitative aspects 4 MANAGEMENT SOLUTIONS i 4 4 4 4 4 4 4 4 4 4 4 4 A M m T t q I a s ( s s n a h u e l h u s

w c

i a a n c

p e t o a c c

t h i a s

a t w f v f p d m q O D U A S P e M E Q E S r c n r g o a

q I d i o o u h a c y

o n a c f s t t s t e u i e t n a c a r s e c u i u f f e o r r s c o i r r l a t t i c l d t m e t r e a r v e f s t e a c f e e

v

i a

i a t h a a u r i t u e d b f m t i d i a m

i d

i c a o l i s e s n n c l o o

b a l h n o s t n i p c

r l r f p e e i i

t

s t s a e a y d d l u f

g e g e t d o r o a l e e a r g f i t o y l c

l

i i i p y m c

v t m

e t e s i o

i n c e r c s t t d

n , i u a a ,

a f i p r c

e a t m

e e

l e h t i h x s o a r y i d t n e o t n i i l o e c u p

a o i c c m s s n c d h t , o e

m t e . c m k n s

s t a f s t l e a n n e a a t . i t i

a c

n t

i

i u n u a t

o e t

i l r e t l s s t a o l t r

t e c s o r o b a e o l e r y

i s h t m h i l i a n

h y u n i n n

o f s e e n g n t o l i f d p f t n k l s n a r c

s , e o k

e a f p e

i

s t d u

i b

r o g n i , t a n e t u o c s

i h p s o

n t s

i d

i w n p h

t a o m l

o u n a m m t

l . i d t r t o

d o r o f

p c o

v o y

s e n m o t s f T l l b

o r f h d i s

f y

u s o a n s o

c e f , u , f o

l o

g a

d r h c u

a v t m

y o

e s m

c a e

o t e l

s t d

d e t o d h u n o s y m e

e m c i c

r

l r g

a s s n q s s d i o x e e e a D a a n

e e o t

d m h k s s n e

p o i i j a p p u a o t

e l t o c r k t . o e c l u

t

d

a s i s

s s a

i r e r i s e n s p r e l a i e u e d a o d

m d n

t s n t

e . i

t a e i

(

o c a

s o r i a n d

l a r l t t g u t t e f w e e

t d l a

e c i s i t i i u o

s

h a , b d

s s u t a

e l f n d o i l p

n i u n t Q m e . c s h n

i p i

n r k a f e

l i e i t n r t e a v y r t

k o l l l p y o

i e p u t i a g t u a r t y i o c i e n s d c o , e s e t i

f o n e s n e

t t

a

u o s i t o

k y y e q v n r d c n

a t

s h t i f

t n c e s c u l n v

n t

i u i c o

o i a u t i i s

s t a b n

l s e o o

o n d n t e c a s o m a u

h s v p r

r i a y n r r g e c f f i n o n o s c g e

e r . n n a d n

a e

o i n

m

y d o e n a e e

t t f s n n

n s

l c t t m g e g c c h t a u t

i . y

m c

e ) c

r a e t c d e

i h a c

o u : a a l

f e a n f o

h

r

o o

l a n . a i o o d v y a e r r p b m u v s s

n c n m n d e d d m s a i a t n r

o

t a d a u n t c t m i u o

d l p l e s i o s t l h p t r d a r y i l f i o g o o r m

i d u h e i n e l i l f n d o r s e s t s e r

t e d e

n a s k s d

a e e g a e m m i t i

d a s i e t o r n e u e r

r , e c t d w

s e

e

t

t

x u e d i o t n n w o l

o

l n

m o a o

. d ’ i o l t o t p l n l

t s e n n o a s r d

s

c h

e n h h

r i d

e

e o i

g . b l

s e n v c a

s e t f a

i

t e e n r c e s d r i n i o r o

a

y n h p l t e n v a n o o

t e e

f s

e f l p

f v r

e e u p i i q n f i t u o e m

s f

a c i t d l e o a m h u

d s

, t

u y u d

r l t l i m

r r p k o a s l l r e t e

a e h

i t o o e

t o m o

t s

s t o n t f p a r

t o h e d o s w o w , p

t k h d e u h t m f c l t h

a d i i

i r

i o

d i s e t r i r e i a i c e o n s t s o e e e n e p o o s l a i r n a

x n s o o l s n r a c g s r u u d t ’ b p c . i . o s a i t t a n n

s

t g b e e t l l e l r k s e y . d h ) h l l s

r , . e ,

e t 4 4 4 4 4 t P T R G R o p a a e a h u f n h a i e i b h r l g t e e c o t

g o e l a e o i o i h a t g u r s r r a r

m o

t

e t a l e u o u r e a e u

l t v s a a u s r y r s i g

e e e

o f t

s t h m t w e h

r . i

n e a

u n g y d

a o

r n i

m g o t n

s r d m o d d

v n u v b a e

e e i b r e o e n e n l p e r

n s x n f

f n g g l s t o o p t o i a a u

i t o r l s r s y o n n l e p e r y o t i y r c t e

n s s e w i i i

e s n t n r a r g o e

l

v - l i g e y o

f u

t m t i

c v

i

s f t h v p a o h

i h e

m o e a l , f f a

r

e o

l

d n i y t n i m n o

r d h i e

, g

n

m d c d e a e o p l e s e t u

s e d l o t m

i o l p f i d t n s e d t o e y e

o g i u l t e n r m r

c o d t t

i l g a i h o o

i

a e u e o i n

i r

n l l t n g p s n a l ’ d h y s c e t h t r s

e

l

i o d ’ a p d u t

o s r

o d

p f r

d u i n i m r s e n

u p e r e p . k d

i c r o q

d n

r s o i m t o d c u e g e i v o d t e c e o t

i i t n t n u v i l n d h s i s . e n g c t i

, e e

o

t g

w a l p

i t n s f o

h n o i . l i

r

i t n e m d m w s h

m t a

a e i t n t h k

r s f

d i o e

a n o n r g n d .

Propagation of uncertainty in model concatenation

The propagation of uncertainty is an unavoidable effect of Examples statistical models that arises when a model is fed with the estimation provided by another model, which can amplify the This table shows the variances of different functions of the real A B error. variables A, B with standard deviations , , covariance and constants a,b . Since most statistical models can be constructed as a For example, the result of credit scoring models is an input composition of these functions, inferring from them how the variable for the calibration of the probability of default (PD), which volatility of the original variables is transferred to the final result of 𝜎 𝜎 in turn feeds capital consumption models. The combined statistical the combined model is immediate. uncertainty at each step may cause high volatility in the final result (the capital, in this case) due to model risk. In statistics, the most common method for measuring uncertainty is the use of the absolute error of variables ∆ and of their standard deviation, . If variables are correlated, their covariance must also be considered for the calculation of the error propagation. 𝑥 Propagation of err𝜎or in linear combinations Let ƒ ( , ... , ) be a set of functions which are linear combinations of variables ,... , with combination coefficients , ... , , ( ... ); that is:  𝑥₁ 𝑥 푚 푛 ƒ 𝑥₁ 𝑥 𝐴₁ 𝐴 푘=1 푚

Then, the variance-covariance matrix of ƒ is given by: As can be seen, the error volatility can quickly increase just by  = 𝐴 𝑥 = 𝐴𝑥  concatenating two statistical models, which has serious ₌₁ implications when quantifying model risk in a decision system and reinforces the need to establish reasonableness controls in the intermediate results.   Where is th𝑐표e 𝑣 va=ri a n c e - c o 𝐴var𝑐i표a𝑣nc𝐴e m=at r𝐴ix𝑐 표o𝑣 f𝐴 the set of variables.    The previous expressions represent the error propagation of a set of varia𝑐b표l𝑣es in the function to which they belong. When the𝑥 errors on the set of variables are uncorrelated, the previous expression simplifies to:

35 𝑥  Propagation of error𝑐 표in𝑣  n=o n - l𝐴 in(e𝜎 a2)r co 𝐴mbinations  When ƒ is a non-linear combin a ti o n of the set of variables , an interval propagation could be performed in order to compute intervals which contain all consistent values for the variables. In a probabilistic approach, the function ƒ must be linearized by 𝑥 approximation to its first-order Taylor series:

ƒ ƒ ƒ ƒ

 ≈ 0 + 𝜕  𝑥 ≈ 0 + 퐽𝑥  where ƒ denotes the partia𝜕l𝑥 d erivative of ƒ with respect to the -th variable and where i s t h e Jacobian matrix of ƒ. Since ƒ is a constant, it does not contribute to the error on ƒ. Therefore, the propagation of error follows the linear case above, but replacing  the line𝜕ar co𝜕e𝑥fficients, and by the partial derivatives ƒ 𝑖 퐽 ⁰ and ƒ :

𝐴 𝐴 𝜕 𝜕𝑥   𝜕 𝜕𝑥 𝑐표𝑣 = 퐽𝑐표𝑣(𝑥)퐽  3 Model Risk Management - Quantitative and qualitative aspects 6 MANAGEMENT SOLUTIONS B d C B B A C B R C B d s g B a N B 2 A B i B B n y n 0 e a a a a a a a a a l e e o o o c d e t o s s s s s s s n n n 1 d v c f r e m m n w e t b a i e e e e e e i o i 1 k k k e

r v q v s I u s a l l l l l l n m m

r m m a b e

e u f o o a l d C I C C C C l l a e o t t

I d r n a i i f f r

p i s

o o o o o l g t t t F r

v I e p

c d

S S t t t m F

v l m m m m m e ( r l e I e g e e e y o 2 p p

a n a r n 2 s r

e e m i u p a r m m m m m 0 m l a a A t

c 0 t i

m N s l e l 1 d o i i N N f a s e e e n n 1 t i i i i i e o r 0 o t t t t t s a a t n

m e e e n 4

w t t t t t o l e o , ( ( t . t i w e e e e e w w

t

a 2 2 i o i s D f g e r 9 r a o o s e e e e e

t 0 0 y s e o s s s C n

t t e i r n

m (

l l 0 0 v o i i o o o o o k r f S a t i e e o c c

n k i r 8 7 a n . o r n n n n n e p e s e t t e n a

- ) t t t a

f -

p w m n t i m . 1 a . i e e B B B B B

t a

h l o a C V t t

t a 4 r r s a a a a a b S e

e e n d

o a l P ) ) e N N n n n n n

e m e

w .

v M l m

B r n G p k k k k k i t G o o r a o d o t a b d ; i i i i i r e . .

p n n n n n n l u c a

s r o r e e - a 6 4 e e r c k g g g g g i e t J m u h r d e s

i e u s

v l

( ( o f u p

S S S S S 2 h e s S J d i o n I e n s I a

r u u u u u 0

l e e (

r n e r i e e F

n y r I n 0 p p p p p

p n e D C

i d m t m r 2 u s 6 e l e e e e e s s t a A a o

k e 0

i a s ) o r r r r r e o m ( v t .

c A v v v v v

m 1 r 2

m n n e r o e T u y i i i i i P e 3 e 0 d s s s s s t

n

h b

m 2 a V

. ) w 2 i i i i i 1

r a

o o o o o

e e n 0 t t e a 0 e 3 e n o o

n n n n n h r 1 a r t I s 0 )

n d R

r s

,

e 2 g 1 i v

5 c ( ( ( ( ( k l i t S B

2 2 2 2 2 i 0 , o e a r C

) . I e

t n e n .

J 0 0 0 0 0 0 l m n

U a a n i u U d o t 0 0 0 0 1 5 d

t p t e s ( n e . i 6 5 5 4 0 i p ) a

2

e t s i r . b n 2 e t ) - - - -

t d t n 0

V i . 2 1 0 1 a i a . ) T t i

n o

a 0 c a . B

l a ) ) 6 1 C n e m

s . . t a l n 6 S a l

) )

k s r e t i B B l . . C ,

s d t i i

t o

s

u t t I B r a a a j e : o a n

a e u

u d e n a s s n a B l p n t r t

l t n e e e n d s C d i e a i e i i

o a l l l o e e t w d c e a s r a o a

n l ) n n k f s

r r

m o . l b o I

d e g a s I t o r I a r

h t s m r k : ( f O

i n o : 2

e A

o

l o A i T k u 0 o t n f i C t 0 n w n

a e t 9 d l g h - e , e t e T S M o O ( M f C M E v h E E B a E 2 i h a s f B B B e r v n a f t h f r 0 o I a a

s e t l f e c s A A A T t a o a i p i e n d 0 d

h e d u n c

n p J

- n S s s 5 a i ( ( e d C e e o r a t n c 2 2 s : m , l i g ) s / l

t

o E i t i m .

B g 0 0 o F ( /

n a i

e i R o I a B o a A w e e 1 1 .

f R l t m e

i C n

n f S n s y s

u d 4 3 s t B w

A n

o k

M

h

y

) ) e t e & a ( M ) T ( t . . d

w o 2 m

s e i

n r M n

A i

a n M R t o v b a E 0 m a

t . d e y C m p

T e i l a f x 0

n s t o s

S m

a o S b h p R h n c 6 A 6 H a o i d o t

s h m a e e , r i a ) g . o d r e a

r e s l . o 2 . a y s

g u

a e e n d l v i p p G l e a l o

n 0

n e t m , u a C

V i t

c r i u P 1 D n g p m L i o n r v r o a h , e i 0 r o

o

o e e e e n d c ( l R m u e e n . i l C

p l n c e

d s e l C . i d s n S R e t

n i F m

d d a J . l a s

o ( y e t e i r R T e . . 2 i

o n . t

e s m n o M p i

o ( C e ) i 0 t n t e u 1 o . t n o t e v f 1 ) m e s

. 9 e / n

V r i m t

3 ( r a M e e 9 t o

h 2 e i a ( ) s

s w

3

n o S . 0 e g a l o u (

s u ) I 2 i 1 f

. k i u n m r . n t C

o a

0 T e 0 i t h A l

g n u t h n a E 1 . p m ) e i

n l g . r t o e m H

1

e a

( r i F

e i

o n S - I e

c . m C s i e n R

1 n E n t n , D n t o

r t u 2

a t p C C d - a g c r a

m ) f a l ( o l ) o y n i e . i f v e A

n n n s

d a S e n a b e m a

M g d g n u s o l u s n o n y s

-

u e d p f c A o p o R p d s

n r t l

t e i f e ) k o t o

C e i s h

B t o a r ) g A a

r l g o a v . e o o i t t n n u c g t a i i

m

f a o s y

i C y d l

r e t o a o s r

n d / o m F

J . d t n t I m r .

i T r n

o y

P

n t , o e G o

C t h o

r v a g s G d e y f .

e d a p s

a

r

i H u G t

I l t e n e n t B s h i y i a d o e l r d a s d o e -

. r u s a v F l a

r

o

t S e t m u R e i n s s i t E r s o t a

s c n a i u C . t t n n e r o r i r

n

k

a t e o t a r o e g p h s s n n t s . e d Glossary

ALM: Assets and liabilities management. Model Risk Additional Value Adjustment (AVA): necessary adjustment for the proper valuation of positions in the trading Capital buffer: Capital surcharge, whose purpose is to ensure that book. an entity is able to absorb the losses derived from its activity in periods of stress. Monte Carlo simulation: technique used to approximate the probability of an event by carrying out multiple simulations using CCF: Credit conversion factor. random variables.

EAD: Exposure at default. It has an ‘off balance sheet’ component OCC (Office of the Comptroller of the Currency): US federal agency (commitments, etc.) which requires certain assumptions to be whose mission is to regulate and supervise all national banks, made. For instance, it could be said that for a credit facility, EAD is federal bureaus and foreign bank agencies. The OCC’s goal is to equal to the amount drawn down + CCF x available credit amount, ensure that the supervised organizations operate in a safe and where CCF is the credit conversion factor. sound manner and in compliance with legal requirements, including the fair treatment of customers and their access to the EBA (European Banking Authority): an independent EU authority financial market. whose overall objective is to maintain financial stability in the EU and to safeguard the integrity, efficiency and functioning of the Override: manual decision that contradicts the result of a statistical banking sector. The EBA was established on 1 January 2011 as part model. of the European System of Financial Supervision (ESFS) and took over the Committee of European Banking Supervisors (CEBS). PD: probability of default.

Fed (Federal Reserve System): central bank of the United States ROC curve (receiver operating characteristic): curve used to analyze founded in 1913 to provide the nation with a safer, more flexible the predictive power of a binary output model. It represents the and more stable monetary and financial system. Over the years, its relationship between the type I error (incorrectly classifying role in banking and the economy has expanded to include adverse events) and the type II error (incorrectly classifying activities such as conducting the nation's monetary policy, favorable events). supervising and regulating the banking institutions or providing financial services to depository institutions. RWA: risk weighted assets; on or off balance sheet exposure weighted by the risk involved for the institution, calculated Flash crash: a very rapid, deep, and volatile fall in security prices according to the methods established by the regulator. occurring within an extremely short period of time. For example, 37 the flash crash occurred on 6 May 2010 in the United States. Scoring/rating: model that assigns a score to each rated item (facilities/counterparties) according to their credit quality. If what is Gini index or powerstat: metric that serves to quantitatively analyze being rated is a facility-customer binomial, it is a scoring model; the discriminant power of a binary output model, based on the but if counterparties are to be rated, it is a rating model. classification it makes of adverse and favorable events. Stress test: simulation technique designed to determine the ability IRB (Internal Rating Based): advanced method for estimating of an entity to withstand an adverse financial situation. In a broader regulatory capital based on internal rating models. To gain access, sense, it refers to any technique used to assess the ability to deal institutions must fulfill a number of requirements and be with extreme conditions, and it can be applied to entities, authorized by the supervisory authority. portfolios, models, etc.

Kolmogorov-Smirnov distance: non-parametric test used to Type I error: statistical term referring to the error that occurs when compare the similarity between two probability distributions. It the null hypothesis is true, but it is rejected. uses the maximum of the absolute difference between the empirical and the estimated distribution. It is used as a predictive Type II error: statistical term referring to the error that occurs when power metric in binary output models. the null hypothesis is accepted even though it is false.

KYC (know your customer): relevant information on customers VaR (Value at Risk): statistical technique used to quantify the level obtained for various purposes, such as regulatory compliance on of financial risk assumed over a period of time at a given fraud, money laundering, terrorist financing or corruption. confidence level.

LGD: Loss given default. It is equal to 1 – the recovery rate. What-if analysis: simulation of the impact of one or more specific According to BIS II, paragraph 468, it must be calculated scenarios for the input variables on the outputs of a process. considering economic downturn conditions.

LTV: relationship between the outstanding amount of a loan and the value of the associated collateral (loan to value). Used for collateralized loans, mainly mortgages. 3 Model Risk Management - Quantitative and qualitative aspects 8 MANAGEMENT SOLUTIONS e x p e c t a O t i u o r n

a s , i

m a n

i d s

t b o e

c e o x m c e e e

d t h

o e u i p r r

a t c r r l u i t e n s n t e e t r s d s ' l c M L M C r j o j M R R J R J i t I I M M c T i T A S t W i a c M u a u n t m n n u o e u a a o o n o e o u & & r s u a m v a i v

f a a a a a a c b i g

v r

s a v m l t d l

p o t i a r D D b p i e l e n u c h n n n n n n h . s e h e a e i e o l n l l

r l o c e r s o a e

. l

n t p n p a a a a a a e r e n m r

r s L g M M

a

i t o . i r k v i m i n G g g g g g g s . m . o i c

r a t i m c a n a c m f c f

z m o i s e

a a a a C r e e e e e n e

i n a n F t . m r i g G a a

n a e r

t c a i e n n l m c o m m m m m m s y a C s . d c v m

i t s

n a n e n

. i i o n

a

a a r a

i l G e . o a t

a a s o f o n n u a s e e e e e c e v M t s

s g g n t u o s

s n . . b a s s i p a o

n , n n n n n n l c a m c

l o

s

e e , f n @ c c t e c t d o b

e r a i l e i c f v s t t t t t t i

a i a l u r r r l o a s i

- a

o t i

o m r s y e r e c

c m S S S a a n S d e 1 a l , c s a a l a a c a r

i n a . o j e n n i o o o o s s e n s t r g d

i . t t a s d t o n l s ) i s t i a

e

m e

s n e u d d d l l l l

a n t

o w i l , M M c

u u u u

n o A

o e @ c s c r b s n

f

l @ i f p a A ' t t t t o l C M s f t p e i e t

n

p a a

e o i n m

t i i i i o t @ o n n t a m o n o o o o l s o s l h n n s j h p l e t a i a o r r e i g n

) n v n n n n

s m h n r a a a , e d a s l r n c s

n

e s i o i e e a k a M s s s

s

g g ) t s s e i k o e p c ( t s e d . r r n

r u p e r

F

i h d e e P P D s o i u

i n o l a

s y n a s r s a o t d a 1 g i

m m @ e

a a n f e n s l i -

i , n i g T l

, n , i t c a n

r h b

u i r r

3 s M l n i b a e c a e m d r e M a a t e e n t t . g

k a l 0

c . e n o n n a y i g c c o l t t f n n c a

t i n , o

s d 0 u a i e i

l m t

h c m e e o e s e m n n t t n l l O n o m k n

k d i s n

i

r r n n m m p n 1 a g t S S a o

s

i

r p . r a

c ( e - n t c o l

8 r , e g t o o g

r f S

g e i a t f o o e r t u g I i t l

s l l e i o a t n n n o h n n o e i u u o c m f

d n r e m n r o e s s a a g m i t f h a t t n

o e c s f

t u i s i i b f t m t i S n i z o o a

e i t

m s i e s c e e a t r a u o i a o l , o n i n n u a o e s e

g n

n o c t s . n e c l . l t n c s s r s t n i u t y t d o i n o

t o i a i

n o i

S c c o a , t ( c v a g

a

m l n 9 m p e i e a o C

. l n i n o l i - c , t

s s . e

u l

o a i p s c y c o n m d u e s , n s

b , o n o

m o ,

n M s

n t . (

E F a l m i E F r d m

n s c i o o n i m i t c a u C u s

n o e h n f P p

e n k r R f b l a n e m e s r t o e r a s a C n o i o r g

t m , r n n p h g c

p s c , s c d o y

g i i e a e

a R a i e a a e

i r i a a s ,

m e t s t g s n B l

s s l s

8 i n s s s , s e c a C d

e i .

e s t d n i r i n

, r a n n e s v

i u g N i n n , r t

i z

t c c v

R

a s T g h t e t i , i h t ) c u l s e s , i o e e o k r b e s n d a l

Design and Layout Marketing and Communication Department Management Solutions - Spain

© Management Solutions. 2014 All rights reserved

Madrid Barcelona Bilbao London Frankfurt Warszawa Zürich Milano Lisboa Beijing New York San Juan de Puerto Rico México D.F. Bogotá São Paulo Lima Santiago de Chile Buenos Aires