EBA REPORT

ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES EBA/REP/2021/08

EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

Contents

List of abbreviations 3 Executive summary 4 1. Background 6 2. Mystery shopping activities of National Competent Authorities 10 2.1 Objective, subject matter and product scope 10 2.2 Methodologies used 11 2.3 Follow-up actions after the mystery shopping 16 3. Lessons learned 18 3.1 Main benefits 18 3.2 Main challenges 19 3.3 Good practices identified by National Competent Authorities 19

2 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

List of abbreviations

EBA European Banking Authority ESAs European Supervisory Authorities EU European Union FI Financial institution MS Mystery shopping NCAs National Competent Authorities

3 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

Executive summary

Following the review of the founding Regulations of the three European Supervisory Authorities (ESAs), the European Banking Authority (EBA) received a number of additional consumer protection mandates in Article 9(1) of its founding Regulation, which includes the mandate to ‘coordinate mystery shopping activities of competent authorities, if applicable’. The mandate has been applicable since 1 January 2020.

As a first step to fulfil said mandate, the EBA collated mystery shopping (MS) activities by National Competent Authorities (NCAs) with a view to sharing experience, learning valuable lessons and identifying good practices for the benefit of the EBA and NCAs that use or intend to use MS in the future.

The report covers initiatives in respect of products that fall within the scope of action of the EBA’s consumer protection mandate, i.e. retail banking products and services (mortgages, deposits, payment accounts, payment services and electronic money). At this stage, only a limited number of NCAs have carried out such MS activities in their jurisdiction.

In particular, the report summarises the most common approaches taken by the NCAs based on the information collated, primarily for the period from 2015 to 2020. It does so by reviewing three key characteristics of the MS activities: i) the objective, subject matter and product scope, ii) the methodologies used by NCAs and iii) the follow-up actions after the MS.

In terms of the objective, subject matter and product scope, the report highlights that in most cases MS is part of a standard on-site supervision or a targeted/thematic investigation carried out by NCAs. MS represents a useful tool to gather information about financial products and services and/or obtain relevant evidence of the activity performed by financial institutions (FIs) or their conduct towards consumers and usually complements the information obtained through the general on-site inspection activity. Although some MS activities focus on specific topics, MS often applies to a wide range of subject matter and targets mainly pre-contractual information for consumers, mortgage credits and basic payment accounts.

With regard to the methodologies used, divergent practices exist among the NCAs regarding how to carry out MS exercises, but responses suggest that for many NCAs that carried out MS, it can be done on site or via the internet and is currently limited to the pre-contractual phase and the collection of information on the practices of the sellers for potential consumers.

Finally, regarding follow-up actions, NCAs indicated that the results of the investigations have been made public via the publication of reports without identifying the institutions that were covered by inspection activities. Some NCAs indicated that they also have the power to publish warnings addressed to specific FIs and issue sanctions to ensure FIs comply with the relevant legal provisions.

4 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

Regarding the lessons learned, NCAs are of the view that MS allows them to obtain faster results and encourages FIs to take corrective actions where regulatory shortcomings were identified and to improve their level of compliance with the applicable regulatory standards to the benefit of consumers’ protection. Some challenges remain, however, regarding the collection of information/evidence which differs from one country to another (e.g. the risk of exposure of individuals during the performance of the MS, constraints regarding the recording of the MS activity). Among the good practices identified by the NCAs, most of them concern common methodology aspects (e.g organising training of NCAs’ inspection/supervision agents including guidance for the preparation of MS and proper documentation of all activities after the execution of MS, identifying target customer profiles and defining agreed ‘rules’ of consumer’s behaviour etc.).

Knowledge about these lessons learned and the good practices identified will guide the EBA’s coordinating mandate on MS in the future.

5 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

1. Background

1. In September 2017, the European Commission put forward proposals to amend the ESAs’ founding Regulations1, as part of its review of the European System of Financial Regulation, which consists of the ESAs and the European Systemic Risk Board.

2. On 21 March 2019, a political agreement was reached by the European Parliament, the European Council and the European Commission on the review of the ESAs, including a new MS mandate as part of its ‘Tasks related to consumer protection and financial activities’. Article 9(1)(g) of the revised EBA Regulation (2019/2175) now provides that ‘the Authority shall take a leading role in promoting transparency, simplicity and fairness in the market for consumer financial products or services across the internal market, including by: […] coordinating mystery shopping activities of competent authorities, if applicable’.

3. The new Regulation was published in the Official Journal of the EU on 27 December 2019. The changes introduced in the EBA Regulation have been applicable since 1 January 2020. From that day onwards, the EBA has been expected to fulfil its new mandate as one of the elements of its objectives of regulatory and supervisory convergence, integrity and transparency of financial markets as well as customer and consumer protection.

4. To fulfil its remit, the EBA started to collate MS activities of NCAs and store them in an EBA internal repository available to NCAs to inform themselves of the work done by other authorities and share experience. As part of its coordinating mandate, the EBA arrived at the view that it would be of added value to take an approach similar to the approach the EBA took to fulfil its education mandate, i.e. to assess the initiatives in a repository and learn valuable lessons and good practice which would benefit the NCAs that use or intend to use MS in the future.

5. The EBA’s report on MS activities of NCAs (hereafter ‘the report’) is the result of this reflection and represents a first step in the EBA’s fulfilment of this mandate. It is based on an internal repository of MS activities that were undertaken by NCAs, primarily since 2015. The report also provides an opportunity for other national or international organisations interested in MS to learn about, and possibly build on, the work carried out by NCAs.

6. The report is split into three chapters. Chapter 1 sets out the background which includes the scope of the report in the context of the wording of the EBA mandate on MS and the general regulatory remit of the EBA, as well as the methodology used by the EBA to collate the

1 For the EBA Regulation, see Regulation (EU) No 1093/2010 of the European Parliament and of the Council of 24 November 2010 establishing a European Supervisory Authority (European Banking Authority), amending Decision No 716/2009/EC and repealing Commission Decision 2009/78/EC(OJ L331, 15.12.2010, p.12); Regulation (EU) No 1094/2010 of the European Parliament and of the Council of 24 November 2010 establishing a European Supervisory Authority (European Insurance and Occupational Pensions Authority), amending Decision No 716/2009/EC and repealing Commission Decision 2009/79/EC (OJ L 331, 15.12.2010, p. 48–83); Regulation (EU) No 1095/2010 of the European Parliament and of the Council of 24 November 2010 establishing a European Supervisory Authority (European Securities and Markets Authority), amending Decision No 716/2009/EC and repealing Commission Decision 2009/77/EC (OJ L 331, 15.12.2010, p. 84–119).

6 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

initiatives. Chapter 2 summarises the most common approaches taken by the NCAs based on the information collated in a repository. It does so by reviewing three key characteristics of the MS activities: i) objective, subject matter and product scope, ii) methodologies used by NCAs and iii) follow-up actions after the MS. Chapter 3 presents the lessons learned and good practices identified by the NCAs from the most relevant initiatives.

7. The scope of the report reflects the EBA’s scope of action on consumer protection and the extent to which NCAs have carried out MS activities. Several NCAs mentioned that they currently do not yet have explicit competence or a legal mandate to carry out MS in their jurisdiction and therefore do not perform any MS activities. Moreover, some NCAs reported that discussions are currently taking place at national level on the possibility of adding such powers to relevant competent authorities’ mandate, for some of them as part of the implementation of the Consumer Protection Cooperation Regulation2.

8. A large number of NCAs indicated that, in their respective jurisdiction, MS is also carried out by other entities, such as publicly funded organisations and state bodies that conduct consumer research in the areas of financial services, and consumer protection organisations, or that MS is conducted as part of academic research.

9. At this stage only a limited number of NCAs have carried out MS in their jurisdiction for products and services in the scope of action of the EBA’s consumer protection mandate. Some NCAs reported that they have carried out MS activities but in areas that are not in the scope of action of the EBA’s consumer protection mandate or that they only supported the MS activity carried out by the financial market authority. The MS activities reported by those NCAs concern investment products and services and/or insurance products and services with the exception of an NCA which reported an MS activity concerning ‘bank and non-bank bureaux de change’.

10. The task assigned to the EBA in the mandate – to ‘coordinate’ – is also reflective of the EBA’s role, objectives and tasks, because, as an EU agency with an understanding of the work carried out by the NCAs, the EBA is best placed to coordinate such work. In addition, Article 31 of its founding Regulation confers on the EBA a ‘general coordination role between competent authorities’, which includes ‘facilitating the exchange of information between [them]’. Finally, the EBA has been mandated to become active regarding MS with the wider objective of ‘promoting transparency, simplicity and fairness in the market for consumer financial products or services across the internal market’.

11. Given the remit of the EBA, the focus of the report is inevitably limited to MS carried out by NCAs only. The report therefore does not take into account any MS that may be carried out by other National Supervisory Authorities or other organisations, such as the activities carried out by national and EU organisations other than NCAs (e.g. European Commission, competition authorities etc.) or organisations from non-EU countries.

2 Regulation (EU) 2017/2394 of the European Parliament and of the Council of 12 December 2017 on cooperation between national authorities responsible for the enforcement of consumer protection laws and repealing Regulation (EC) No 2006/2004, OJ L 345, 27.12.2017, p. 1–26.

7 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

12. In addition, the report is necessarily focused on initiatives in respect of products that fall within the scope of action of the EBA’s consumer protection mandate, i.e. retail banking products and services (mortgages, deposits, payment accounts, payment services and electronic money).

13. It should finally be noted that the report is based on only a subset of MS activities from the wider range of MS activities undertaken by the NCAs. The initiatives included in the report are those that fall within the scope of the EBA’s work in MS and have been submitted by the NCAs according to their judgement of the relevance of the activities. This approach is intended to ensure that the report focuses on the most relevant work for the purpose of the EBA’s mandate on MS.

14. In order to develop this report, the EBA collated MS activities of NCAs that fall within the scope of action of the EBA that NCAs have carried out in the past, and saved them in a repository that the EBA intends to update on a regular basis, potentially every two years.

15. Most of the MS activities of NCAs reported in the repository have been launched in the past six years. To date, the repository consists of 14 MS activities carried out by NCAs, of which 8 MS activities fall within the EBA’s consumer protection scope of action. To obtain a comprehensive overview of MS activities of NCAs, the EBA has applied a broad understanding of MS activities to be included in the repository. In this regard, the following understanding of MS is used for the purpose of the repository and consequently for the report:

‘Mystery shopping’ is understood as an undercover research approach used by NCAs, or market research companies that they may have used, to measure quality of customer service and/or gather information about financial products and services and the conduct of FIs towards consumers. MS may include the use of individuals who may act as potential or actual customers and who are trained and briefed to experience and measure key phases of a product’s lifecycle and compliance with particular requirements. They report back their experiences in a detailed and objective way. They perform specific tasks, for example reviewing how staff perform against pre-determined standards during an interaction with a customer. That interaction may occur at the pre-contractual, contractual or post-contractual phase and may involve purchasing a product/service, asking questions, or registering complaints. MS enables supervisors to carry out an assessment, in concrete situations, rather than relying on documents kept by firms, on- site interviews, or surveys.

16. The above understanding of MS may differ from MS definitions or MS mandates of some NCAs.

17. The repository holds a number of elements of information on each MS activity, including, for example, the features and content of the activities, the methodology used by each NCA, follow- up actions undertaken as well as lessons learned.

18. The repository presents, for each MS activity, a uniform set of information. This standardisation is required for the EBA to be in a position to provide a comparable overview of the MS activities of NCAs. However, this setup comes at the expense of some loss of detail for each initiative, including the nuanced ways in which NCAs tailor some of their MS practice.

8 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

19. NCAs submitted to the EBA’s repository those activities that they deemed most useful for the purpose of the report and that are also within the regulatory remit of the EBA. As a result, the repository is a subset of the wider range of MS activities undertaken by the authorities.

9 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

2. Mystery shopping activities of National Competent Authorities

20. This chapter presents the MS practices most commonly used by the NCAs as reported in the MS activities collated in the repository. To this end, the chapter focuses on several characteristics presented in the repository: i) objective, subject matter and product scope, ii) methodologies used by NCAs and iii) follow-up actions of NCAs after the MS. For each of the three key characteristics the chapter provides some background, explains how the characteristic is understood and presents a general view of the most common approaches used by the NCAs in the activities submitted.

2.1 Objective, subject matter and product scope

21. Many NCAs that carried out MS indicated that in most cases MS is part of a standard on-site supervision or a targeted/thematic investigation carried out by NCAs.

22. NCAs explained that one of the objectives of MS is to complement the information obtained through the general on-site inspection activity and to verify that the service provider grants the necessary pre-contractual or contractual information to consumers. MS carried out by NCAs included, for example, gathering information directly from the points of sales about pre- contractual information and explanations offered to verify whether the applicable rules are complied with, or, for instance, whether prior supervisory instructions that the authority had issued to a particular FI were subsequently applied. Following the increasing digitalisation of the relationships between financial providers and customers, several MS activities performed by NCAs concerned mobile and online marketing practices.

23. According to many NCAs that carried out MS, MS represents a useful complementary tool to gather information about financial products and services and/or obtain relevant evidence of the activity performed by FIs or their conduct towards consumers.

24. The MS activities aim also at assessing compliance with existing or new legal requirements or NCAs’ recommendations. For example, MS was carried out regarding the provision of banking products and services through digital channels and the monitoring of the implementation of FIs’ obligation to disclose information related to credit moratoria on their websites.

25. Some NCAs also reported that certain MS exercises were initiated following complaints received or a suspicion of infringements and unfair commercial practices. For example, MS was carried out following a complaint on difficulties for foreigners to open payment accounts with basic features in the country. In addition, some NCAs explained that most of the MS carried out was part of annual plans of inspections.

10 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

26. An analysis of NCAs’ MS activities in the repository reveals that MS concerns a wide range of subject matter. Although some MS activities focus on specific topics, MS often applies to more than one type of subject matter. In addition, some MS carried out by NCAs covers all types of financial products, not just banking products, which reflects the fact that some NCAs are integrated supervisors across the banking, insurance and securities sectors and therefore cover all of these sectors.

27. The products and services that were subject to MS within the EBA’s consumer protection scope of action from the NCAs are the following:

- consumer credit/personal loans: examination of the obligations (information, advice) associated with an application for a consumer credit, to verify whether they are provided with misleading information; - home loans and mortgage credit: compliance with pre-contractual information and obligations by the creditor, for instance, provision of the European Standardised Information Sheet; - basic payment accounts: conditions of access, legal limits to the fees and expenses charged, characteristics of debit cards associated with this type of account, provision of information etc.; - payment services: examination of payment services information, provision of the information document on charges and fees; - car loans and leasing products: Annual Percentage Rates advertisements, commercial communications related to car loans and leasing products or car loan simulators available on the websites of FIs.

28. According to the information collected in the repository, MS carried out by NCAs targeted mainly pre-contractual information for consumer and mortgage credits and basic payment accounts.

2.2 Methodologies used

29. This section describes the methodologies used by NCAs when carrying out MS in their jurisdiction regarding i) general approach, geographical reach and format of MS activities carried out and ii) providing a complete overview of the preparation of MS. The section refers also to iii) the type of information collected as part of the MS exercise.

General approach, geographical reach and format for MS carried out

30. Based on the responses received, divergent practices exist among the NCAs regarding how to carry out MS exercises, in particular on whether it can be done through on-site inspection at the branch only, or also via the internet, which for certain NCAs corresponds to ‘off-site inspection’:

11 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

- ‘on-site inspection in the context of MS’ corresponds to analysis of different risks, internal control systems, business models, conduct or governance, at the premises of an inspected FI, usually within a predefined scope and timeframe and with a view to assessing the information provided by sales staff; - ‘off-site inspection in the context of MS’ corresponds to collection of information via the internet or by phone and analysis of documents and information provided by FIs, which includes analysis of contracts, an internet watch on advertisement etc.

31. Furthermore, the responses suggest that for many NCAs that carried out MS it can be done on site or via the internet.

32. Many NCAs that carried out MS mentioned that MS, in their jurisdiction, is limited to the pre- contractual phase and the collection of information on the practices of the sellers for potential consumers. The pre-contractual phase includes, for example, collecting evidence on how a financial product is sold, and verifying that no misleading information is provided to consumers and that all required pre-contractual documents are provided. It could involve, for instance, looking at information provided to the consumer though a standardised information sheet, looking at advertisements, promotions, information, standard brochure and/or basic information about a product/service, filling in an online form available for consumers on a specific website etc. NCAs explained that the use of MS for the pre-contractual phase via an ‘on-site inspection’ implied in particular the following steps, for example concerning payment accounts with basic features:

- evaluating the suitability of the product offered by the FI as an answer to a request made by the mystery shopper (e.g. the mystery shopper requested information about opening a bank account); - assessing the adequacy of information provided by the FI to the mystery shopper; - identifying any other irregular practices (e.g. irregular commissions, non-provision of required documentation); - verifying if the payment account with basic features was available in the branches, whether its features were well explained and presented, and if any written information was provided to the mystery shopper.

33. However, in addition to the pre-contractual phase, some NCAs explained that they use MS during the contractual phase too, in order to test the knowledge of/advice given by the staff, the information provided to the consumers, explanations about the features and risks etc. Some NCAs explained that sometimes it involves mystery shoppers opening real accounts, signing a loan agreement or signing an insurance contract, or simply checking the sales process by collecting information passively as a potential customer.

34. Many NCAs that carried out MS indicated that on-site MS is done in a sample of branches of FIs with a total number of FIs targeted by MS which could in certain cases reach up to 24 FIs. NCAs explained, however, that the number of FIs covered by the MS activities depends on the size of the market in each country and the goal of the MS. Some NCAs, for example, explained that MS

12 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

might also be conducted in the framework of a targeted investigation which could focus on one institution and could concern one specific issue identified by the NCAs. Similarly, in most cases, and depending on resource availability and the geographical coverage of the entity’s sales activity, the MS activity covers the entire jurisdiction of the NCAs.

Preparation of the mystery shopping

35. In general, NCAs referred to the following preparatory steps taken prior to conducting MS activities for on-site and off-site MS:

- study phase to determine the scope and the objectives: the NCA will decide on the MS goals and analyse the legislations and regulations which are applicable and the other relevant information about the products and services, complaints, issues identified or topics to be monitored; - prior analysis of FIs: collect information about the supervised entity, the characteristics of the products and services, e.g. price lists (fees, interest rates); FIs’ website information about the products and services, such as a standardised information sheet, fee information document, terms and conditions applicable; - more specifically for on-site MS: - prepare the inspection programme: describe the objectives and methodology of the inspection, define the duration of the action and the inspection team that will perform the exercise; some NCAs explained that they rely on the programmes of the investigations and the relevant provisions of the NCA’s legal act which contains the main pillar of each MS activity;

- prepare the field of work: identify places, branches; identify the FIs that will be analysed; select the mystery shoppers; ensure the necessary logistical planning;

- development of an evaluation form to be filled in by mystery shoppers: define mystery shopper profiles, inspection guide and ‘Facts Identified during Mystery Shopping Exercise’ (‘FIMS’) document;

- training of the mystery shoppers/briefing the staff: some NCAs explained that before each MS, the NCA organises a personal briefing with every mystery shopper. Mystery shoppers/staff are informed on key concepts of financial products and conduct of business rules (‘train the trainers’ principle), on the objectives and methodology, definitions, visit process, questionnaire and documentation. The NCAs build scenarios that mystery shoppers/staff have to learn. NCAs require substantial preparation, for example the supervision officers need to prepare a document describing the fictional interest in the product offered, how mystery shoppers or the staff will introduce themselves to the party involved, what information they should target, and the role that they would play. However, not all NCAs provide training.

13 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

- pilot visits for assessing the functioning of the defined process: mystery shoppers and/or staff go through the whole MS as a simulation exercise before the mystery shoppers start the MS. Every mystery shopper is expected to be familiar with the mystery shopper’s guide, where the exact rules of MS are outlined.

Execution of the mystery shopping

36. All NCAs that conducted MS indicated that so far they have directly executed the MS and it has not been outsourced to any auxiliary agent despite the legal possibility for some NCAs to outsource such activity to an external company.

37. As reported by NCAs in the repository, the organisation and structure of the MS activity vary among the NCAs (unit, joint unit, inspection/supervision team, outsourcing etc.). The number of individuals involved in MS differs from one NCA to another and depends on the nature of the investigation and the objective of the project. Furthermore, the responses highlighted that MS inspections are usually part of a specific programme designed according to internal rules. For example, certain NCAs reported that in certain cases the team leading the work on MS is made up of a minimum of two dedicated staff members (e.g. a manager and an associate), while in other cases it involved more staff (up to 20 staff involved in total) who are divided into teams or come from different parts of the NCA. In other NCAs, MS is always part of on-site supervision. As such, no uniform approach appears to exist across the NCAs.

38. According to the responses of NCAs, the execution of MS responds to the following steps depending on whether MS takes place at the FIs’ premises (on site) or via the internet (what certain NCAs call off site):

- for MS taking place on site: - the NCA’s inspection/supervision agent performs the MS exercise according to the inspection guide or the defined profile/guidelines, asking for information about a specific banking product or service which they plan to acquire (e.g. deposit or credit);

- the NCA’s inspection/supervision agent evaluates the information given by the branch representative and the compliance thereof with the applicable rules (e.g. adequacy of information disclosure concerning the product/service; verifying that the consumer was not discriminated against based upon race, skin colour, religion, national origin, sex or marital status; verifying the level of technical competence);

- following the exercise, the NCA’s inspection/supervision agent should register the information and the documentation provided by employees;

- the MS might be followed (or not) by on-site inspections called ‘accredited inspection’, during which the inspection/supervision agent could fill in an inspection fact sheet, which identifies non-compliant observed facts. Some NCAs reported that the NCA’s inspection/supervision agent gives immediate feedback to the FI on the outcome of the MS exercise.

14 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

- for MS taking place via the internet: the NCA’s inspection/supervision agents browse websites, assess the information disclosed to banking customers and collect the information (e.g. adequacy of information disclosure concerning the product/service; compliance with the duties of information to be provided). - for both on-site and off-site MS: the NCA’s inspection/supervision agents draft the final report of the inspection, which includes a description of the outcome of the exercise and, if any infringement is detected, proposals for the adoption of enforcement measures (namely, recommendations or specific orders).

39. According to some NCAs a distinction should be made between a so-called ‘light’ and ‘heavy’ version of MS. A ‘heavy’ version of MS is when there is i) a personal interaction between the NCA’s inspection/supervision agent and the FI, and ii) a personal tailored response from the FIs. In these cases the NCA’s inspection/supervision agent will not actively reveal his/her identity and that he/she is a supervision officer of an NCA. A heavy MS will take more time to prepare (at least two weeks) and to perform a heavy MS approval from the NCA’s head of supervision division is needed in advance. All other forms of MS are regarded as ‘light’ and can be performed within one working day. Based on the NCAs’ responses, the duration of MS activities varies significantly among NCAs from one month to more than one year.

Type of information collected during the mystery shopping

40. As mentioned in the ‘Mystery Shopping Survey Methodology’ in the Annex to the final report of the European Commission on the ‘Mystery shopping survey on territorial restrictions and geo-blocking in the European Digital Single Market’3, depending on the market evaluated and the research objectives MS projects can focus on interactive evaluations, observations, audits or a combination of these, which can lead to the collection of both objective and subjective information, and interaction (or not) with a staff member, requiring sometimes a mystery character. As stated in the EC MS survey methodology:

- interactive evaluations imply that an interaction is needed between the mystery shopper and another person. Usually interactive evaluations are used to assess the level of service given but can, however, also be applied to gain information (separately from the service level) such as the price of a product/service, which aspects are mentioned by default and which ones have to be prompted etc.;

- observations imply that there is no interaction between the mystery shopper and another person. The mystery shopper just observes what is being told, what is happening or what is present. Like interactive evaluations, observations can imply the collection of both objective and subjective information with a mystery aspect in order not to influence the data collected;

3 See European Commission Mystery shopping survey on territorial restrictions and geo-blocking in the European Digital Single Market: Final Report published in 2016 (p. 162–163)

15 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

- audits most resemble observations to a large extent but differ in the fact that they hardly leave room for subjective interpretation and should be seen as a very detailed assessment, an inventory which does not impose any mystery character to reveal the information. It is present or it is not present.

41. The table below summarises the above-mentioned differences between interactive evaluations, observations and audits:

Interactive evaluation Observation Audit Interaction with a staff member × Requires a mystery character × × Objective information can be × × × collected Subjective information can be × collected

Inventory approach ×

42. Most NCAs that carried out MS activities collected ‘objective’ information which includes, for example, documents/information provided directly to the consumers, contracts, pictures (photographs of advertising, posters, print screen of websites etc.), online advertisements, product terms and conditions, fee information document, signed employee testimony, information recorded on a recording device by a mystery shopper etc. Other NCAs, in turn, also use ‘subjective’ information in addition to ‘objective information’, such as an assessment of the conduct of staff (e.g. behaviour, attitude). By way of example, this can include the degree of knowledge of the entities’ staff about the pre contractual information that has to be provided to consumers applying for a mortgage loan or certain credit cards. However only a limited number of NCAs use subjective information.

2.3 Follow-up actions after the mystery shopping

43. All NCAs indicated that the results of the investigations have been made public via the publication of a report, which for certain NCAs is the bi-annual statistical report of the NCAs or represents high-level findings published in a circular. These reports, however, usually do not identify the institutions that were covered by MS.

44. Some NCAs highlighted that MS is part of on-site supervision and the results of MS are considered as ‘complementary evidence’.

45. Half of the NCAs which carried out MS also mentioned that they have the power to publish warnings addressed to FIs. In addition, some NCAs can also issue sanctions on FIs to ensure they comply with the relevant legal provisions, and for eliminating the issues detected. Those sanctions include in particular: - ordering the FIs to terminate the infringement, within a prescribed time limit. The FIs will be obliged to notify the NCA of the measures implemented;

16 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

- banning or imposing conditions regarding the pursuit of the activity or the supply of products and services involved in the infringement, until the infringement is eliminated;

- imposing a consumer protection fine;

- prohibiting in advance some planned commercial practices which are presumed to lead to unfair commercial practices.

46. Some NCAs explained that imposing such formal (legal) measures (e.g. a fine) on a firm obliges the NCA to publish the respective formal decision. In that publication, the investigation of the NCA is also published with some details, which means that any MS activities that have been performed during the investigation would most likely also be disclosed in said publication.

47. In addition, certain NCAs mentioned that, following the issuance of specific orders and recommendations to the management bodies of the FIs inspected, a number of bilateral contacts take place with the compliance officers of those institutions in order to explain the conclusions of these exercises and to analyse the corrective measures proposed. Other NCAs indicated that they do not organise any meetings with the firms to present the results or discuss with them but give a copy of the MS inspection report, which contains a record of the finding, to the firms inspected.

17 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

3. Lessons learned

48. This chapter reports NCAs’ insights into MS by presenting the lessons learned and good practices gained by NCAs when carrying out MS activities. The chapter presents the benefits and challenges identified by NCAs.

49. It also includes the good practices identified by NCAs. The good practices are not intended to be exhaustive or representative, not least because they are based on only the subset of activities included in a repository.

3.1 Main benefits

50. Most NCAs are of the view that an MS exercise is a useful component and a complementary tool to fulfil their supervisory and/or subsequent enforcement objectives, even if not every on- site supervision will include MS. Some NCAs specified that they use MS when necessary and unless they can otherwise provide evidence via on-site inspections.

51. For certain NCAs, MS sometimes represents the only way to efficiently and effectively monitor certain market practices. MS allows NCAs to carry out a comprehensive analysis of the compliance with applicable requirements of the practices and conduct of FIs when providing financial services to consumers (such as selling practices, information disclosure and advice provided to consumers) and that cannot be easily identified during usual on-site or off-site inspections. Some NCAs highlighted that if the NCA had requested as a supervision/inspection officer similar information to that obtained via MS, there is a good chance that NCAs would not have been able to receive similar information.

52. MS helps NCAs to get first-hand experience of the interaction between consumers and FIs, evaluate whether the FI's staff have an adequate level of knowledge and competence and verify whether specific training or information has been provided to sales staff, especially as regards new legal or regulatory acts.

53. MS also allows NCAs to obtain faster results. For example in the COVID 19 pandemic context, some NCAs were able to draw conclusions in a short period of time, in a situation which was considered critical, in order to ensure compliance with the measures taken to protect banking consumers from the effects of the COVID 19 pandemic. MS also encourages FIs to take corrective actions where regulatory shortcomings were identified and to improve their level of compliance with the applicable regulatory standards to the benefit of consumers’ protection.

54. In addition, some NCAs mentioned that MS is used to complement and support thematic reviews together with other forms of data collection such as focus groups, surveys etc.

18 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

3.2 Main challenges

55. A variety of challenges have been identified by NCAs in their responses, which are the following:

- Some requirements oblige NCAs’ inspection/supervision agents or mystery shoppers to disclose their real identity information when entering into a contract with an FI (identity card, identification via electronic ID, pay slips, accreditation needed to access the complete files etc.). MS is therefore harder to use in the contractual and post-contractual phase if no commercial relationships can be entered into without checking the consumer’s identity. The NCAs’ investigation/supervision agents would be indeed forced to disclose their real identity, which would impact the MS process. MS seems to be more suitable for the pre-contractual phase. A pre-contractual investigation would rather be gathering information on mandatory disclosure.

- The risk of exposure of individuals during the performance of the MS (in particular during the on-site inspection) when the MS is conducted by the NCA itself: due to the size of the country some NCA staff are immediately identified.

- As with other instances of interpreting social observations, NCAs as observers need to avoid subjective bias.

- Risk for MS to capture only one singular experience/one-off occurrence as the mystery shopper captures a snapshot at a particular point of time; according to the feedback received from FIs by NCAs, some MS exercises do not always reflect the usual practice of the FIs.

- Constraints regarding the recording of the MS activity for subsequent supervisory purposes, e.g. because the recording is legally not permitted, was technically impossible, or, where it was possible, it proved to be of insufficient audio quality.

- Limited time and human resources to carry out MS exercises alongside other supervisory work: conducting MS involves guidance, training of NCA staff, potential recruitment of mystery shoppers etc., which creates more difficulties for smaller countries with less human resources.

56. Such challenges also demonstrate that the collection of information/evidence might differ from one country to another due to divergent national practices.

3.3 Good practices identified by National Competent Authorities

57. Among the good practices identified by the NCAs, most of them concern common methodology aspects, such as:

- organising training of NCAs’ inspection/supervision agents which will include setting up detailed ‘mystery shopper’ profiles and roles for the on-site visit as well as guidance for the preparation of MS and proper documentation of all activities after the execution of

19 EBA REPORT ON THE MYSTERY SHOPPING ACTIVITIES OF NATIONAL COMPETENT AUTHORITIES

MS (e.g. filling in an inspection fact sheet immediately after the MS exercise to guarantee the correctness of the output);

- organising briefing (before MS) and debriefing meetings (after MS) with the NCA’s inspection/supervision teams to ensure consistency in the approach;

- suggestion to visit branches of FIs with a team of at least two people to facilitate a comparative analysis between the different points of view and more in-depth analysis with less operational risk;

- necessity to identify target customer profile and define agreed ‘rules’ of consumers behaviour;

- conducting MS covering several FIs and in several FI branches to ensure a wider range of information is collected (e.g. to prevent any one-off occurrence, to ensure a level playing field or to prevent any infringement by one specific FI).

58. Knowledge about these lessons learned and the good practices identified will guide the EBA coordinating mandate on MS in the future.

20

EUROPEAN BANKING AUTHORITY Tour Europlaza, 20 avenue André Prothin CS 30154 92927 Paris La Défense CEDEX, FRANCE Tel. +33 1 86 52 70 00 E-mail: [email protected] https://eba.europa.eu