Who Guideline on the Implementation

Working document QAS/19.783 January 2019 Draft document for comments – prepared by EMP/RSS

2 WHO GUIDELINE ON THE IMPLEMENTATION

3 OF QUALITY MANAGEMENT SYSTEMS

4 FOR NATIONAL REGULATORY AUTHORITIES

5 (January 2019)

6 DRAFT FOR COMMENTS

Please send any comments you may have to Dr S. Kopp, Group Lead, Medicines Quality Assurance, Technologies Standards and Norms ([email protected]), with a copy to Ms Sinéad Jones ([email protected]) by 30 March 2019.

Medicines Quality Assurance working documents will be sent out electronically only. They will also be placed on the Medicines website for comment under “Current projects”. If you have not already received our draft working documents, please send your email address (to [email protected]) and we will add you to our electronic mailing list.

9 © World Health Organization 2019 10 . 11 All rights reserved. 12 13 This draft is intended for a restricted audience only, i.e. the individuals and organizations having received this draft. The draft 14 may not be reviewed, abstracted, quoted, reproduced, transmitted, distributed, translated or adapted, in part or in whole, in any 15 form or by any means outside these individuals and organizations (including the organizations' concerned staff and member 16 organizations) without the permission of the World Health Organization. The draft should not be displayed on any website. 17 18 Please send any request for permission to: 19 20 Dr Sabine Kopp, Group Lead, Medicines Quality Assurance, Technologies Standards and Norms, Department of Essential 21 Medicines and Health Products, World Health Organization, CH-1211 Geneva 27, Switzerland, fax: (41 22) 791 4856, email: 22 [email protected]. 23 24 The designations employed and the presentation of the material in this draft do not imply the expression of any opinion 25 whatsoever on the part of the World Health Organization concerning the legal status of any country, territory, city or area or 26 of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted lines on maps represent approximate 27 border lines for which there may not yet be full agreement. 28 29 The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or 30 recommended by the World Health Organization in preference to others of a similar nature that are not mentioned. Errors and 31 omissions excepted, the names of proprietary products are distinguished by initial capital letters. 32 33 All reasonable precautions have been taken by the World Health Organization to verify the information contained in this draft. 34 However, the printed material is being distributed without warranty of any kind, either expressed or implied. The responsibility 35 for the interpretation and use of the material lies with the reader. In no event shall the World Health Organization be liable for 36 damages arising from its use. 37 38 This draft does not necessarily represent the decisions or the stated policy of the World Health Organization. 39 Working document QAS/19.783 Page 2

40 SCHEDULE FOR DRAFT WORKING DOCUMENT QAS/19.783: 41 42 WHO GUIDELINE ON THE IMPLEMENTATION OF 43 QUALITY MANAGEMENT SYSTEMS FOR 44 NATIONAL REGULATORY AUTHORITIES 45

Activity Date

QMS workshop in Zimbabwe. Complete self-benchmarking by end 9-13 Oct 2017 of workshop. Conduct survey and summarize countries' expression of needs for WHO guidelines on QMS implementation.

Prepare and present concept paper to recommend the development 16-20 Oct 2017 of the 'WHO guidelines on implementation of quality management systems (QMS) for national medicines regulatory authorities'.

Develop the TOR for drafting group members, including the 15 Nov 2017 selection criteria and invitation letter to join the drafting group.

Contact selected countries and call for experts to join the drafting 15 Nov 2017 to 15 Jan 2018 group according to selection criteria and follow up on communications. Select and appoint drafting group.

Select and appoint drafting group based on meeting specified criteria 15 Jan 2018 and invite them officially to join the group, WebEx conference, and first face-to-face drafting group meeting.

Kick-off drafting group meetings (WebEx). 13-15 Feb 2018

First face-to-face meeting with drafting group, Tunis, Tunisia. 12-16 Mar 2018

QMS workshop in Burkina Faso. Complete self -benchmarking by 16-20 Apr 2018 end of workshop. Run survey and summarize countries' expression of needs for WHO guideline on QMS implementation.

Produce first draft (V1) of the guidelines. 29 Jun 2018

Review of draft V1 by CRS, rapporteur and ISO expert. 30 Jun to 10 Jul 2018

Deliver final V1 for review by drafting group. 11 Jul 2018

Conduct WebEx meetings with drafting group. 7-9 Aug 2018

Drafting group members submit comments on V1. 31 Jul 2018

Address comments from drafting group and produce V2 for review 31 Aug 2018 by CRS and hand over to TSN.

ECSPP meeting. Update on status of guidelines development. 15 Oct 2018 Working document QAS/19.783 Page 3

Submit to TSN for Public Consultation 1. 31 Oct 2018

Public consultation 1. 31 Oct – 31 Dec 2018

Address and incorporate comments from public consultation 1, 31 Dec 2018 produce draft to share with drafting group, address comments from drafting group, prepare (V3) of the guidelines.

Provide opportunity for the drafting group to comment on current 1 Feb 2019 version via WebEx.

Organize and conduct informal consultation with international 10 Feb 2019 stakeholders and drafting group (three-day meeting)

Informal consultation on V3, Tunis, Tunisia. mid-March 2019

Post-meeting edits/clean up and submit to TSN for public consultation 2 (V4).

Public consultation 2. 15 Mar to 15 Jun 2019

Collate comments and circulate for final review by drafting group. 15 Jul 2019

Hand-off to TSN for review by ECSPP in advance to annual meeting 31 Jul 2019 in October where the guidelines will be presented for adoption.

Review guidelines for possible endorsement. Oct 2019

Address all comments and requests from ECSPP, produce final 31 Dec 2019 version of guidelines (V5). 46

Working document QAS/19.783 Page 4

47 Table of Contents 48 Abbreviations ...... 6

49 1. Introduction ...... 7

50 1.1 Background ...... 7

51 1.2 Basis for the guideline ...... 8

52 1.3 Objective ...... 8

53 1.4 Scope ...... 8

54 1.5 Instructions for using the guideline ...... 9

55 2. General considerations ...... 10

56 3. Definition of terms ...... 12

57 4. Translation of ISO Standard 9001:2015 to the specific needs of NRAs ...... 15

58 4.1 Requirements ...... 15

59 4.2 High level structure of ISO 9001:2015 ...... 15

60 4.3 Clause by clause guidance for NRAs on the requirements for ISO Standard 9001:2015 ...... 18

61 Clause 0 Introduction ...... 18

62 Clause 1. Scope ...... 22

63 Clause 2. Normative references ...... 22

64 Clause 3. Terms and definitions ...... 22

65 Clause 4. Context of the organization ...... 23

66 Clause 5. Leadership ...... 45

67 Clause 6. Planning ...... 49

68 Clause 7. Support ...... 54

69 Clause 8. Operations ...... 64

70 Clause 9. Performance evaluation ...... 75

71 Clause 10. Improvement ...... 84

72 5. QMS implementation methodology ...... 87

73 6. Considerations to ensure integrated implementation of QMS in NRA ...... 90

74 References and further reading ...... 92

75 Authors and acknowledgements ...... 94 Working document QAS/19.783 Page 5

76 Appendix 1. Integration of QMS into the WHO Global Benchmarking Tool ...... 95

Working document QAS/19.783 Page 6

79 Abbreviations

80 NOTE: This section will be updated in the final stages of guideline development. 81 82 CAPA Corrective action and preventive action 83 CI Continual improvement 84 DI Documented information 85 GBT Global Benchmarking Tool 86 GCP Good Clinical Practice 87 GMP Good Manufacturing Practice 88 GRP Good Regulatory Practice 89 ICT Information and Communication Technology 90 IMS Integrated Management System 91 ISO International Standards Organization 92 KPI Key Performance Indicators 93 LIMS laboratory Information Management System 94 MA Marketing Authorization 95 M and M Monitoring and Measurements 96 MC Market Surveillance and Control 97 MOF Ministry of Finance 98 MOH Ministry of Health 99 MRM Management Review Meeting 100 MS Member States 101 NCL National Control Laboratory 102 NRA National Regulatory Authority 103 PDCA Plan, do, check and act 104 P and S Products and Services 105 QMP Quality Management Principles 106 QMS Quality Management System 107 SF Substandard and Falsified 108 SLP Summary Lot Protocol 109 SOP Standard Operating Procedure 110 TM Top Management 111 TRM Technical Review Meeting 112 VL Vigilance (one of the NRA regulatory functions) 113 WHO World Health Organization Working document QAS/19.783 Page 7

114 1. Introduction

115

116 1.1 Background 117 118 Implementation of the Thirteenth World Health Organization (WHO) General Programme of Work 119 (2019-2023) as adopted by the Seventy-first World Health Assembly (2018) and the WHO Leadership 120 Priorities, has attracted much international public health attention to the theme of Universal Health 121 Coverage (UHC) and increased access to safe and effective medical products. 122 123 Several World Health Assembly (WHA) resolutions, including WHA67.20 (2014), mandate WHO to 124 provide support to its Member States (MS) in strengthening national regulatory systems for medical 125 products. It recognizes that “effective regulatory systems are an essential component of health system 126 strengthening and contribute to better public health outcomes; that regulators are an essential part of the 127 health workforce, and that inefficient regulatory systems themselves can be a barrier to access to safe, 128 effective and quality medical products” [1]. Accordingly, WHO’s vision is for all MS to have a 129 regulatory system that ensures medical products and other health technologies in the market meet 130 internationally recognized standards of quality, safety, and efficacy to facilitate access to these products. 131 132 National Regulatory Authorities (NRAs) are responsible for facilitating access to safe, quality and 133 effective medical products within the respective MS and for consistently demonstrating that the services 134 they provide meet legal and regulatory requirements; they deliver effective and efficient services; and 135 they can evaluate performance and make improvements. A quality management system (QMS) can 136 ensure that the products or services an NRA provides consistently meet statutory and regulatory 137 standards and meet customers’ expectations. A QMS provides opportunities to enhance customer 138 satisfaction; address context-associated risks and opportunities for continuing improvement; 139 demonstrate conformity to specific QMS requirements; and assure the quality, safety and efficacy of 140 medical products. 141 142 In 2015, WHO developed and launched the Global Benchmarking Tool (WHO GBT). This tool assists 143 regulators worldwide in evaluating the developmental status of their regulatory system and its related 144 functions. The GBT includes one indicator that assesses the NRAs’ level of development with respect 145 to QMS.1 Benchmarking results of 43 low and middle-income countries indicate that most NRAs need 146 to establish and implement, or if already established, enhance and maintain QMS.

1 Appendix 1 describes the relationship between QMS and the WHO GBT.

Working document QAS/19.783 Page 8

147 QMS implementation is challenging for NRAs due to the diversity of NRA organizational structures, 148 the different levels of NRA development and the number of regulatory functions that need to be 149 addressed. Several international guidelines on QMS have been published; however, none of these 150 focuses specifically on NRAs. Other existing guidelines are field specific [11-20]. At the request of 151 MS, WHO developed this document to provide tailored guidance to NRAs on QMS implementation. 152

153 1.2 Basis for the guideline 154 155 ISO Standard 9001:2015 ‘Quality management systems- Requirements’ is a well-known international 156 standard published by the International Organization for Standardization (ISO). The standard is 157 applicable to both products and services and provides requirements for establishing a QMS that can be 158 applied to any organization, public or private, big or small, and to a variety of fields. The WHO GBT 159 QMS sub-indicators are based on this standard. Accordingly, ISO standard 9001:2015 offers a practical 160 model to establish and implement QMS for NRAs [7]. 161

162 1.3 Objective 163 164 The objective of this guideline is to assist NRAs to develop, implement or improve quality systems 165 using ISO Standard 9001:2015, and subsequent updates, as a basis. The expectation is that this will 166 increase the reproducibility of the quality and consistency of the outputs (products and services), 167 customer focus and stakeholder satisfaction. 168

169 1.4 Scope 170 171 This is an overarching guideline that can be applied across all regulatory functions, including 172 registration and marketing authorization, vigilance, market surveillance and control, licensing 173 establishments, regulatory inspections, laboratory access and testing, clinical trials oversight and lot 174 release. 175 176 This guideline, for practical and illustrative purposes, only provides examples for the following four 177 NRA functions: 178 179 • Registration and marketing authorization (MA) 180 • Vigilance (VL) 181 • Lot release (LR) Working document QAS/19.783 Page 9

182 • Market surveillance and control (MC). 183 184 Each of these functions was selected for different reasons: MA is a critical function, but no specific 185 guidance is available; VL is the weakest function as evidenced by the results from WHO benchmarking 186 of NRAs; LR is the vaccine-specific regulatory function and requires particular attention; and MC was 187 selected because sub-standard and falsified medical products are a major issue in developing countries. 188 Although the examples are specific to these four functions, it is important to note that the principles can 189 be applied to any regulatory function. 190 191 This guideline can be utilized by all institutions responsible for regulatory oversight of medical products 192 including the national control laboratory (NCL) and any other agency or institute involved in regulatory 193 oversight, as well as customers and other stakeholders. 194

195 1.5 Instructions for using the guideline 196 197 This guideline provides interpretation for implementation purposes specific to NRAs. It does not 198 duplicate the text from the ISO Standard 9001:2015, therefore it is important to read this guideline in 199 conjunction with ISO Standard 9001:2015 and its supporting documents. 200 201 1.5.1 Required documents 202 203 As a first step, NRAs should have the following three documents on hand. 204 205 • ISO Standard 9001:2015 is the standard upon which this guideline is based. 206 • ISO Standard 9000: 2015 provides QMS-related vocabulary (terms and definitions) and describes 207 the fundamentals and principles of QMS [8]. 208 • ISO/TS 9002:2016 provides generic guidance (not specific to NRAs) on how to apply ISO Standard 209 9001:2015 by describing individual clauses and giving examples of steps any organization can take 210 to meet the requirements [21]. 211 212 1.5.2 Recommended documents 213 214 The additional documents listed below may be of interest to NRAs but are not required. 215 216 • Quality management principles (ISO brochure) [34] 217 • ISO Standard 9001:2015 for small enterprises – What to do? Advice from ISO/TC 176 9

Working document QAS/19.783 Page 10

218 • ISO/IEC 17020 covering requirements for the operation of various types of bodies performing 219 inspection including regulatory inspections [22] 220 • ISO/IEC 17025: 2017 is the existing standard providing general requirements for the competence of 221 the testing and calibration laboratories [23] 222 • ISO 19011:2018 Guidelines for auditing management systems [24] 223 • ISO 31000:2018 Risk management- Guidelines [25] 224 • ISO 9004: 2018 Quality management - Quality of an organization - Guidance to achieve sustained 225 success [26] 226 227 1.5.3 Accessing the documents 228 229 Complete information on QMS-related ISO standards, free brochures and publications and links for 230 purchase of these documents are available on the ISO website https://www.iso.org/iso-9001-quality- 231 management.html. Alternatively, ISO standards can also be purchased from the national standard body 232 in the NRA’s country. 233 234 1.5.4 Guidance for NRAs on the requirements for ISO 9001:2015 235 236 This document provides guidance on how the ISO Standard 9001:2015 requirements can be applied to 237 QMS implementation for NRAs. Section 4.3 of this guideline is a clause by clause correlation to 238 Clauses 0 to 10 of ISO Standard 9001:2015. NRAs are advised to use the following step-wise approach: 239 240 1. Review the clause in ISO Standard 9001:2015. 241 2. Refer to the corresponding clause and accompanying guidance in ISO/TS 9002:2016. 242 3. Refer to the corresponding clause in section 4.3 of this guideline, which contains guidance and 243 examples specific to NRAs. 244 245 NRAs are referred to ISO Standard 9000: 2015 [8] for terms and definitions. Definitions of some 246 important terms are included in Section 3 of this guideline.

247 2. General considerations 248 249 Use of this guideline is voluntary. NRAs are free to use this guideline or to choose other methods for 250 implementing QMS. 251 Working document QAS/19.783 Page 11

252 NRAs from any country can use this guideline regardless of the NRA’s size or organizational structure. 253 WHO considered three NRA organizational models (centralized, decentralized and discrete)2 and 254 examined whether the differences in their organizational structure impacted the approach to QMS 255 implementation. WHO concluded that regardless of the organizational structure of the NRA, each 256 institution involved in the regulatory oversight of medical products should establish its own QMS for 257 the products and services it provides. 258 259 Preferably, all regulatory institutions in a single country should follow the same standard for 260 consistency and coherency, ideally ISO Standard 9001:2015, since this standard provides the basis 261 (principles and requirements) for adaptation and implementation of QMS to any field. 262 263 Certain ISO 9001:2015 standard elements are systemic while others are functional. This guideline 264 discusses them in an integrated manner tailored to the specific needs of NRAs. 265 266 WHO does not provide QMS certification services and cannot issue QMS certification or conduct 267 official QMS audits of NRAs. However, as part of the regulatory systems strengthening program, WHO 268 can provide technical support for benchmarking. 269 270 Good Regulatory Practices (GRP) provide a means for establishing sound, affordable, and effective 271 regulation of medical products as an important part of health system strengthening. GRP are a set of 272 practices applied to the development, implementation and maintenance of controls, including laws, 273 regulations and guidelines, to achieve a public policy objective. There are nine GRP principles [6]. 274 275 • Legality: Regulation should have a sound legal basis and should be consistent with existing 276 legislation, including international norms or agreements. 277 • Impartiality: Regulation and regulatory decisions should be impartial to be fair and 278 to avoid conflicts of interest, unfounded bias or improper influence from stakeholders. 279 • Consistency: Regulations should be clear and predictable; both the regulator and the 280 regulated party should understand the behaviour and the conduct that are expected and the 281 consequences of noncompliance. 282 • Proportionality: Regulations and regulatory decisions should be proportional to the risk and 283 should not exceed what is necessary to achieve the objectives. 284 • Flexibility: Regulations should not be prescriptive; they should allow flexibility in

2 A more detailed description of the NRA models is provided in practical help box 1.

Working document QAS/19.783 Page 12

285 responding to a changing regulated environment and different or unforeseen circumstances. 286 287 • Effectiveness: Regulations should produce the intended result. 288 • Efficiency: Regulations should achieve their goals within the required time, effort and cost. 289 • Clarity: Regulations should be accessible to, and understood by, the users; 290 • Transparency: Regulatory systems should be transparent; requirements and decisions should be 291 made known to affected parties and, where appropriate, to the public in general. 292 293 These principles may be used while framing the quality policy and objectives of the NRA and 294 compliance achieved through the implementation of QMS. A WHO guidance document on GRP is in 295 development [6].

296 3. Definition of terms

297 298 NOTE: This section will be refined and updated in the final stages of guideline development. 299 300 The definitions provided below apply exclusively to the terms used in this guidance document. 301 302 Terminology and definitions in this document that are specific to NRAs are those of WHO [32, 33]. 303 Additional terms related to QMS can be found in ISO Standard 9000:2015 [8]. 304 305 A 306 • Activities- project management smallest identified object of work in a project 307 • Assessment- Systematic, independent and documented process for obtaining assessment evidence 308 and evaluating it objectively to determine the extent to which assessment criteria are fulfilled 309 • Audit- Systematic, independent and documented process for obtaining objective evidence and 310 evaluating it objectively to determine the extent to which the audit criteria are fulfilled 311 312 B 313 • Batch- A defined quantity of product processed in a single process or series of processes and 314 therefore, expected to be homogeneous 315 316 C 317 • Certification- The term applied to third party attestation related to products, processes, systems or 318 persons 319 • Competence- Ability to apply knowledge and skills to achieve intended results. Commitment 320 • Conformity- Fulfilment of a requirement 321 • Continual improvement- Recurring activity to enhance performance 322 • Control- The taking of all necessary actions to ensure and maintain compliance with the criteria 323 established in the statutory and regulatory requirements 324 • Corrective action- Action to eliminate the cause of nonconformity and to prevent recurrence Working document QAS/19.783 Page 13

325 • Counterfeit- А counterfeit medicine is one which is deliberately and fraudulently mislabelled with 326 respect to identity and/or source 327 • Customer- Person or organization that could or does receive a product or a service that is intended 328 for or required by this person or organization 329 • Customer satisfaction- Customer’s perception of the degree to which the customer’s expectations 330 have been fulfilled 331 • Customer service interaction of the organization with the customer throughout the life cycle of a 332 product or a service 333 334 D 335 • Defect- Non-fulfilment of a requirement related to a specified use 336 • Documented information- information required to be controlled and maintained by an organization 337 and the medium on which it is contained 338 339 E 340 • Effectiveness Extent to which planned activities are realized and results achieved 341 • Efficiency Relationship between the result achieved and the resources used 342 343 F 344 Feedback- customer satisfaction opinions, comments and expressions of interest in a product, a service 345 or a complaints-handling process 346 347 G 348 • Good manufacturing practice That part of quality assurance which ensures that products are 349 consistently produced and controlled to the quality standards appropriate to their intended use and 350 as required by the marketing authorization 351 • Good regulatory practice Set of practices that are to be applied to the development, implementation 352 and maintenance of controls – including laws, regulations and guidelines – to achieve a public policy 353 objective 354 • Governance Refers to the different ways that organizations, institutions, businesses and 355 governments manage their affairs. Governance is the act of governing and thus involves the 356 application of laws and regulations, but also of customs, ethical standards and norms 357 358 K 359 • Key performance indicator- A quantifiable measure used to evaluate the success of an organization, 360 employee, etc. in meeting objectives for performance 361 362 M 363 • Management system- System to establish policy and objectives and to achieve those objectives 364 • Measurement management system- Set of interrelated and interacting elements necessary to achieve 365 metrological confirmation and continual control of measurement processes 366 • Medical products- A term that includes medicines, vaccines, diagnostics and medical devices 367 368 N 369 • National Regulatory Authority- WHO terminology for national medicines regulatory authorities. 370 NRAs should promulgate and enforce medicines regulations 371 • Non-conformity- Non-fulfilment of a requirement 372 373 P 374 • Procedure- Specified way to carry out an activity or a process 375 • Process- Set of interrelated or interacting activities that use inputs to deliver an intended result 376 • Process approach- Any activity or set of activities, that uses resources to transform inputs into 377 outputs can be considered a process 13

Working document QAS/19.783 Page 14

378 • Product- Output of an organization that can be produced without any transaction taking place 379 between the organization and the customer 380 • Provider (Supplier)- Organization that provides a product or a service 381 382 Q 383 • Qualification Action of proving that any premises, systems and items of equipment work correctly 384 and lead to the expected results 385 • Quality Degree to which a set of inherent characteristics of an object fulfils requirements 386 • Quality assurance Part of quality management focused on providing confidence that the quality 387 requirements will be fulfilled 388 • Quality characteristic Inherent characteristic of an object related to a requirement 389 • Quality control- Part of quality management focused on fulfilling quality requirements 390 • Quality management- Management with regard to quality 391 • Quality management system Part of management system (set of interrelated or interacting elements 392 of an organization to establish policies, objectives, and processes to achieve those objectives) with 393 regard to quality 394 • Quality manual- Specification for the quality management system of an organization 395 • Quality planning Part of quality management focused on setting quality objectives and specifying 396 necessary operational processes and related resources to fulfil the quality objectives 397 • Quality policy Overall intentions and direction of an organization related to quality as formally 398 expressed by top management 399 400 R 401 • Regulation- A written instrument containing rules having the force of law. Regulatory requirement- 402 Obligatory requirement specified by an authority mandated by a legislative body 403 • Release- Permission to proceed to the next step of the process or to the next process 404 • Requirement- Need or expectation that is stated, generally implied or obligatory. Generally implied 405 means that it is a custom or common practice for the organization, its customers or other interested 406 parties, that the need or expectation under consideration is implied 407 • Review- Determination of the suitability, adequacy and effectiveness of an object to achieve 408 established objectives 409 • Risk- Effect of uncertainty 410 411 S 412 • Service- Output of an organization with at least one activity necessarily performed between the 413 organization and the customer 414 • Statutory requirement- Obligatory requirement specified by a legislative body 415 416 V 417 • Validation Confirmation, through the provision of objective evidence, that the requirements for a 418 specific intended use or application, have been fulfilled. 419 420 Working document QAS/19.783 Page 15

421 4. Translation of ISO Standard 9001:2015 to the specific needs of

422 NRAs

423 424 4.1 Requirements 425 426 This section of the guideline requires NRAs to have access to three standards: ISO 9001:2015, ISO 427 9000:2015 and ISO/TS 9002:2016 [21] (see 1.5.1 above). 428 429 4.2 High level structure of ISO 9001:2015 430 431 ISO Standard 9001: 2015 contains an introduction section (Clause 0) and 10 clauses as presented in 432 Table 1. Clauses 1 to 3 set the stage for the requirements. Clauses 4 to 10 represent the actual 433 requirements, with Clause 4 providing an overview of considerations regarding the context of the 434 organization and how to apply the process approach. These considerations are addressed in detail in 435 Clauses 5 to 10. Table 1 provides an overview of the structure of ISO Standard 9001:2015 and briefly 436 describes the intent of each clause.

Working document QAS/19.783 Page 16

437 Table 1. Structure of ISO 9001:2015, its clauses and brief description of intent for each clause 438 0 Introduction Describes benefits of QMS, quality management principles, concept of process approach and Plan-Do-Check-Act (PDCA), concept of risk-based thinking and relationship with other management system standards. 1 Scope Provides purpose of QMS for an organization (i.e. NRA) 2 Normative ISO Standard 9000:2015 should be used as the reference standard which defines the terms used in ISO Standard 9001:2015. references

3 Terms and Terms and definitions are given in ISO Standard 9000:2015. definitions

4 Context of the 4.1 To determine issues (strengths and areas for improvement) internal and external to the NRA which may affect its ability to organization meet the expected results 4.2 To determine the interested parties (stakeholders) and capture their needs and expectations relevant to the QMS 4.3 The organization (i.e. NRA) to decide the scope (boundaries) of the QMS 4.4 Provides a template for process approach (PDCA) and documented information needed for QMS 5 Leadership 5.1 Responsibilities/actions of/by top management (TM) to demonstrate leadership and commitment towards QMS, including customer focus 5.2 Development of a quality policy by TM and ensuring its application 5.3 Definition of roles, responsibilities and lines of authority by TM 6 Planning 6.1 Determining risks and opportunities (using information from 4.1 and 4.2) and planning actions on risks and opportunities 6.2 Establishing quality objectives and making plans to achieve them 6.3 Planning for changes, if any, in QMS 7 Support 7.1 Providing resources for QMS (people, infrastructure, measuring equipment and organizational knowledge) 7.2 Ensuring staff are competent 7.3 Ensuring people are aware of quality policy, quality objectives, importance of their contributions to the effectiveness of QMS and knowing the consequences for not doing work as per QMS 7.4 Establishing internal and external communication processes 7.5 Creation and control of documented information (procedures and records) 8 Operation 8.1 To address operational planning and control Working document QAS/19.783 Page 17

8.2 Requirements for products and services (P and S) covering communication with customers, developing and reviewing requirements for P and S and to document changes to P and S requirements 8.3 To develop processes for designing and developing P and S 8.4 To develop processes for procurement of the right P and S 8.5 To carry out provision of services under controlled conditions, including post-delivery activities 8.6 To ensure authorized release of P and S 8.7 To ensure outputs (products, services or other) which are not conforming are controlled 9 Performance 9.1 Monitoring, measurement, analysis and evaluation (Check part of PDCA) covering plan for monitoring and measurements evaluation (M and M) of P and S, processes and system and for analysis and evaluation of M and M data and establishing a process for obtaining customer feedback for assessing the degree of customer satisfaction 9.2 Process of planning and conducting internal QMS audits and reporting results internally 9.3 Management review covering purpose of review, inputs to be considered by TM and outputs of review with decisions and actions relating to opportunities for improvement, changes needed in QMS and resource needs 10 Improvement 10.1 To determine opportunities for improvement with focus on enhancing customer satisfaction 10.2 Nonconformity and corrective action. Actions to control or correct a nonconformity should be taken promptly, this can be achieved by containing the problem while investigations continue to eliminate its cause to avoid its recurrence 10.3 Using outputs from 9.1.3 and improvement decisions taken during management review to initiate continual improvements Annex A Clarification of new structure of the standard, terminology in the standard and concepts Annex B Other international standards on quality management and QMS developed by ISO/TC 176 Bibliography Useful list of supporting ISO standards and websites 439

Working document QAS/19.783 Page 18

440 4.3 Clause by clause guidance for NRAs on the requirements for ISO

441 Standard 9001:2015 442 443 Clause 0 Introduction 444 445 This clause describes the benefits of QMS; quality management principles; the concept of process 446 approach and Plan-Do-Check-Act (PDCA); the concept of risk-based thinking; and relationship with 447 other management system standards. 448 449 Clause 0.1 General 450 451 ISO Standard 9001:2015 has been adopted by more than 130 countries, it is internationally accepted 452 and has become a world benchmark for good management practice. More than one million certificates 453 of conformity to ISO Standard 9001 have been issued worldwide. ISO Standard 9001:2015 is not a 454 product standard but a system standard. It gives “what” an NRA should do for its QMS and leaves the 455 “how” to be decided by the NRA. 456 457 Benefits for NRAs using this ISO standard include: 458 459 • the possibility to standardize operations, leading to uniformity; 460 • availability of up-to-date manuals, instructions and procedures; 461 • clarity and transparency of responsibilities; 462 • systematic training and development of human resources; 463 • structured and smooth vertical and horizontal communication; 464 • in-built process performance monitoring and improvement mechanism; 465 • systematic processing of customer feedback; 466 • standard system of detections, investigation and correction of errors; 467 • system for addressing customers’ complaints; and 468 • a means to demonstrate the ability to consistently provide quality products and services. 469 470 QMS are influenced by the different policies, objectives, diverse work methods, resource availability 471 and administrative practices specific to each NRA. Therefore, the details of each QMS will be different 472 in each NRA. While the detailed method of QMS implementation is important, what matters most is 473 that the QMS yields effective, consistent and reliable results. The QMS must be as simple and 474 understandable as possible to function properly and meet the quality policies and objectives of the NRA. Working document QAS/19.783 Page 19

475 An NRA may decide to have its QMS assessed for certification or not. Regardless of whether it seeks 476 assessment and/or certification by a third party, the NRA will still benefit from the implementation and 477 maintenance of an effective QMS. 478 479 The ISO Standard 9001:2015 standard adopts the “process approach”, which enables the NRA to plan 480 processes and its interactions. It also incorporates the concepts of the PDCA cycle and risk-based 481 thinking. Risk-based thinking enables an NRA to identify factors that could cause its processes and 482 QMS to deviate from the planned results; put in place preventive controls to minimize negative effects; 483 and leverage opportunities as they arise. 484 485 Clause 0.2 Quality Management Principles 486 487 ISO Standard 9001:2015 supports the application of the seven quality management principles (QMPs) 488 described in ISO Standard 9000:2005 which are applicable in the context of NRAs. 489 490 • Customer focus. The primary focus of quality management is to meet customer requirements and 491 to strive to exceed customer expectations. 492 • Leadership. Leaders at all levels establish unity of purpose and direction and create conditions in 493 which people are engaged in achieving the organization’s quality objectives. 494 • Engagement of people. Competent, empowered and engaged people at all levels throughout the 495 organization are essential to enhance the organization’s capability to create and deliver value. 496 • Process approach. Consistent and predictable results are achieved more effectively and efficiently 497 when activities are understood and managed as interrelated processes that function as a coherent 498 system. 499 • Improvement. Successful organizations have an ongoing focus on improvement. 500 • Evidence-based decision-making. Decisions based on the analysis and evaluation of data and 501 information are more likely to produce desired results. 502 • Relationship management. For sustained success; organizations manage their relationships with 503 relevant interested parties, such as providers. 504 505 An ISO brochure titled ‘Quality management principles’ provides the full text of the QMPs. The 506 brochure can be downloaded from http://www.iso.org/iso/pub100080.pdf at no charge. ISO 9000:2005 507 and the ISO brochure, contain supporting information on the QMPs, including the rationale, key 508 benefits and possible actions associated with each QMP. These principles also provide a sound basis 509 for establishing quality policy and quality objectives. 510 19

Working document QAS/19.783 Page 20

511 Clause 0.3 Process approach 512 513 ISO 9001:2015 advocates the use of a process approach for the development, implementation and 514 enhancement of the effectiveness of a QMS, with the aim of increasing customer satisfaction by meeting 515 their requirements. Understanding and managing the interconnected processes in a regulatory system 516 contributes to the NRA’s effectiveness and efficiency in achieving its quality objectives and intended 517 results. In addition, this approach helps NRAs identify the management capacity needed to produce the 518 desired outputs. 519 520 The NRA should identify the following elements for each process:

521 • the main inputs to the process, for example: information, legal requirements, national and/or 522 regional government policies, materials, energy, human and financial resources; 523 • the desired outputs, for example, the characteristics of the product/service to be provided; 524 • controls and indicators needed to verify the process performance and/or results; and 525 • interaction with other processes (outputs from one process typically form inputs into other 526 processes).

527 PDCA 528 529 The regulatory system, the QMS and related processes can be managed using the Plan-Do-Check-Act 530 cycle (PDCA) cycle with an overall focus on risk-based thinking to leverage opportunities and prevent 531 undesirable results. ISO 9001:2015 provides the following brief description of the PDCA process: 532 533 • Plan: establish the objectives of the system and its processes, and the resources needed to deliver 534 results in accordance with customers’ requirements and the NRA’s policies, and identify and 535 address risks and opportunities; 536 • Do: implement what was planned; 537 • Check: monitor and, where applicable, measure processes and the resulting products and services 538 against policies, objectives, requirements and planned activities and report the results: 539 • Act: take actions to improve performance, as necessary. 540 541 Clauses 6 to 10 each focus on one stage of the PDCA cycle: 542 543 • Clause 6 –Planning – Plan 544 • Clause 7 – Support – Do Working document QAS/19.783 Page 21

545 • Clause 8 – Operation – Do 546 • Clause 9 – Performance evaluation – Check 547 • Clause 10 – Improvement – Act

548 Figure 1. ISO Standard 9001:2015 clauses viewed in relation to the PDCA cycle.

549 550 551 Risk-based thinking 552 553 According to the ISO standard 9001:2015, risk-based thinking is an essential component of QMS. Risks 554 and opportunities should be identified during the planning stage. 555 556 Risk is the effect of uncertainty, and any uncertainty can have positive or negative effects on one or 557 more objectives. Uncertainties can emerge due to changes in the operational environment, political 558 decisions, lack of information or unknown information or a variety of aspects. NRAs should plan and 559 implement actions to address risks and opportunities to prevent negative effects and improve results. 560 561 Opportunities can arise due to situations favourable to the achievement of a desirable result. For 562 example, a change in the structure of the NRA can create opportunities to improve the efficiency in the 563 organization. It can also carry some risks. Actions taken to leverage the opportunities should also 564 include consideration of the associated risks. 565 566 21

Working document QAS/19.783 Page 22

567 0.4 Relationship with other management system standards 568 569 It is important to note that ISO Standard 9001:2015 was developed following the same high-level 570 structure used in all ISO management systems standards. This structure (10 clauses with the same 571 headings) facilitates the integration between different standards enabling NRAs to develop an integrated 572 management system (IMS) if they wish to implement other management system standards. The 573 following standards may be of interest to NRAs. 574 575 • ISO Standard 37001:2016 anti-bribery management systems (24) 576 • ISO/IEC 27001:2013 information security management systems (25) 577 578 ISO Standard 9004:2018 (Managing for sustained success of an organization) (9), provides guidelines 579 which NRAs may use for initiating improvements in the QMS. 580 581 Clause 1. Scope 582 583 The scope explains the purpose of the standard. This clause states that the ISO 9001:2015 requirements 584 are for a QMS, and not for products or services. It also indicates that ISO Standard 9001:2015 is 585 intended to be generic and applicable to all organizations, regardless of their type, size, or the products 586 and services they provide. 587 588 By implementing this standard, the NRA can demonstrate its ability to consistently provide products 589 and services that meet customer, statutory and regulatory requirements and can enhance customer 590 satisfaction. This guideline aims to provide guidance to adapt the ISO Standard 9001:2015 591 requirements to the needs of NRAs with respect to all of its regulatory functions, including the 592 supporting processes. 593 594 Clause 2. Normative references 595 596 ISO Standard 9000:2005 - Quality management systems- Fundamentals and vocabulary is an integral 597 part of ISO Standard 9001:2015 and is the source for the definitions of the terms used in ISO Standard 598 9001:2015. 599 600 Clause 3. Terms and definitions 601 602 As all terminology required for the use given in ISO Standard 9000:2005, no additional terms are Working document QAS/19.783 Page 23

603 included here. Specific NRA-related terminology (not included in the ISO standard) is listed and 604 definitions are those provided in the relevant WHO documents. 605 606 The ISO 9000 family of standards uses generic terms to describe the relationship between the parties 607 involved. For the purposes of this guideline the term “organization” means NRA. “External providers” 608 are people or companies from whom the NRA receives products and services (e.g. suppliers). 609 “Customers” are people or organizations who receive products and services from the NRA. 610 611 ISO’s “Online Browsing Platform” can be used to search for information on terms and definitions 612 included in ISO 9000:2005, see: https://www.iso.org/obp/ui/

613 Clause 4. Context of the organization 614 615 4.1 Understanding the organization and its context 616 617 Guidance 618 619 The intent of this clause is to understand the external and internal issues relevant to the NRA’s purpose 620 and strategic direction that can impact its ability to achieve the planned quality objectives of its 621 QMS. 622 623 There are many sources of information about internal and external issues that can affect the effective 624 implementation of the QMS. The issues are categorized as either related to statutory or regulatory 625 requirements. Statutory issues are considered for both internal and external cases. They provide the 626 boundaries within which the QMS can be implemented while complying with national laws (pieces of 627 legislature). Regulatory issues relate to professional regulatory bodies for personnel, materials, 628 environmental, financial and other areas that affect internal and external implementation of the QMS. 629 There are many sources for information about external and internal issues, such as internal documented 630 information and meetings, national and international press, websites, publications from national 631 statistics offices and other government departments, professional and technical publications, 632 conferences and meetings with relevant agencies, meetings with customers and relevant interested 633 parties, and professional associations. 634 635 External and internal issues can change, and therefore should be monitored and reviewed. The NRA 636 can conduct reviews of its context at planned intervals and through activities such as management 637 review.

Working document QAS/19.783 Page 24

638 639 An NRA must /should understand the context to provide the foundation for determining the scope of its 640 QMS, quality policy, quality objectives, and risks and opportunities. 641 642 Practical help box 1. Guidance to assist in the interpretation of clause 4.1 643 644 Understanding the NRA and its context 645 646 NRAs can be organized according to different models: 647 648 • In the centralized model, all regulatory functions are under the same organization and TM. 649 • In the decentralized model, a central office is generally located in the capital city and subsidiary offices 650 in states or provinces. The roles and responsibilities of these offices can be different; some functions may be 651 carried out at central level while others are delegated to the decentralized offices. 652 • In the discrete model, different institutions are responsible for different regulatory functions. Each of 653 them reports independently, usually to the Ministry of Health. 654 655 The specific characteristics of the NRA in question must be carefully analyzed when considering the context of 656 the organization. WHO concluded through discussions with the drafting group, that independently from the 657 organizational structure of the NRA, each institution involved in the regulatory oversight of medical products 658 should establish their own QMS in accordance with their specific processes. 659 660 Examples of internal/external issues 661 662 Internal issues to be taken into consideration include: 663 664 • resource factors, including infrastructure, governance, environment for the operation of the processes, 665 organizational knowledge, workforce and financial considerations; 666 • human aspects such as competence of persons, organizational culture and values, relationships with unions; 667 • operational factors such as process capabilities, performance of the quality management system, customer 668 evaluation; and 669 • factors in the governance of the organization, such as rules and procedures for decision making or 670 organizational structure.

671 External issues to be taken into consideration include: 672 673 • macro-economic factors such as money exchange rate predictions, economic situation, inflation forecast, 674 credit availability; 675 • political factors such as political stability, public investments, local infrastructure, international trade Working document QAS/19.783 Page 25

676 agreements; and 677 • technological factors such as new sector technology, materials and equipment, patent expirations, 678 professional code of ethics. 679 680 NRAs can use tools such as strengths, weaknesses, opportunities and threats analysis (SWOT) and political, 681 economic, social, technological, legal, environmental analysis (PESTLE) to identify issues. Alternatively, simpler 682 approaches can be useful, depending on the size and complexity of the NRA’s operations, such as brainstorming 683 and asking "what if" questions. 684 685 4.2 Understanding the needs and expectations of interested parties 686 687 Guidance 688 689 The intent of this clause is to ensure that the NRA considers the requirements of relevant interested 690 parties, beyond just those of its direct customers. NRAs should focus only on parties that can have a 691 direct or indirect impact on the NRA’s ability to provide products, and services that meet requirements 692 customer’s and statutory and/or regulatory requirements. The NRA may consider external and internal 693 issues (decided under clause 4.1) for determining relevant interested parties. 694 695 The NRA should have a robust system in place to monitor and review the relevant requirements of its 696 interested parties at planned intervals. The information resulting from these activities should be 697 considered when determining the scope of the QMS (see 4.3) and for determining risks and 698 opportunities (see 6.1). 699 700 Practical help box 2. Guidance to assist in the interpretation of clause 4.2 701 702 Interested parties and examples of their requirements 703 704 Examples of NRAs interested parties include: 705 706 manufacturers, researchers, sponsors for new product development, civil society, consumers, patients, healthcare 707 providers, distributors, exporters, importers, wholesalers, pharmacists, government partners (MOH, MOF, other), 708 parliament members and commissions, national and international pharmacopoeias, health system in general and 709 immunization program in particular, provincial NRA offices in the case of decentralized models, other institutions 710 with regulatory responsibilities as in the case of the discrete model. 711 712 713 25

Working document QAS/19.783 Page 26

714 Examples of requirements include: 715 716 availability of affordable medical products of assured quality, safety and efficacy, effective and efficient service, 717 legality, transparency, good communication skills, confidentiality, courtesy, compliance with laws, regulations 718 and requirements and responsiveness, good governance, impartiality, clarity, consistency and flexibility. 719 720 4.3 Determining the scope of the QMS 721 Guidance 722 723 The scope for the QMS should be established based on the following information: 724 725 • the external and internal issues as determined by the requirements of clause 4.1; 726 • the requirements of relevant interested parties (such as regulators and customers) as determined 727 in accordance with the requirements in clause 4.2; and 728 • the products and services provided by the NRA. 729 730 In determining the scope of the QMS, the NRA shall also establish the boundaries of the QMS by 731 considering issues such as: the infrastructure of the NRA; the NRA’s different sites/offices and 732 activities; and centralized, decentralized or externally provided functions, activities, processes, products 733 and services. 734 735 The NRA should carefully review each individual requirement within a clause to determine whether it 736 is applicable. Some or all the requirements in a clause may be applicable. NRAs should not decide a 737 clause is not applicable without careful consideration of each requirement. The documented scope 738 should include details of the products and services covered as well as justification for any requirements 739 determined to be not applicable. 740 741 The scope should be maintained as documented information using whatever method meets the NRA’s 742 needs, such as in the quality manual or a website. 743 744 Practical help box 3. Guidance to assist in the interpretation of clause 4.3 745 746 Defining the scope of a QMS in an NRA 747 748 The NRA should determine the scope of the QMS based on the services to be provided, requirements of interested 749 parties, processes, infrastructure, and activities and resources available for each organization. The scope should 750 match the roles and responsibilities of the NRA and address all regulatory functions. In the case of a decentralized Working document QAS/19.783 Page 27

751 or discrete NRA, if a certain institution is responsible for vigilance and another institution is responsible for 752 inspections; the scope for each institution should cover the services each one provides. Both institutions should 753 establish a QMS, preferably using the same standard, ideally ISO Standard 9001:2015. 754 755 Example of scope statement: 756 757 The Quality Management System at country X National Regulatory Authority (XNRA) covers all the procedural, 758 executive and supervisory functions of the X National Regulatory Authority in order to ensure the safety of food, 759 the safety and quality of the human and animal medicines, and the safety and efficiency of medical devices and 760 supplies through the establishment of an effective regulatory body in all sectors of the XNRA and all of its 761 branches in the country. 762 763 Excluding: 764 765 1. All the technical procedures and tests carried out in the laboratories, which will be covered through the 766 application of quality management system for the laboratories based on the international standards ISO Standard 767 17025, 768 769 2. All procedures for sampling during the inspection of establishments and at the ports of entry as well as 770 inspection tools used in this regard, will be covered by the implementation of the quality system of inspection ISO 771 Standard 17020. 772 773 4.4 QMS and its processes 774 775 Guidance 776 777 This clause provides a template for the process approach (PDCA). It focuses on the processes needed 778 for the QMS in accordance with ISO Standard 9001:2015 and the related documented information 779 needed. 780 781 When referring to the processes required by NRAs to carry out the different functions, it includes not 782 only the processes for service provision, but also the processes needed for the effective implementation 783 of the system, such as internal audits, management review and others (including processes that are 784 performed by external providers). 785 786 A process is a set of interrelated or interacting activities that use inputs to deliver intended results. 787 788 a) The NRA should determine the inputs required (what is required for the implementation of the

Working document QAS/19.783 Page 28

789 processes as planned) and the outputs expected from its processes (either by the customers or the 790 subsequent processes). Inputs and outputs can be tangible (e.g. materials, components or equipment) 791 or intangible (e.g. data, information or knowledge); 792 793 b) When determining and organizing the sequence and interaction of these processes, different methods 794 can be used such as process maps or flow diagrams (see figure 2 as example); 795 796 c) To make sure that processes are effective (i.e. deliver the planned results), the process control criteria 797 and methods should be determined and applied, criteria for monitoring and measurement can be process 798 parameters, or specifications of services; performance indicators related to quality objectives or other; 799 800 d) The NRA should determine the resources needed for processes such as people, infrastructure and 801 environment for the operation of the processes, organizational knowledge etc; 802 803 e) The NRA should assign the responsibilities and authorities for its processes by first determining the 804 activities of the process and then determining the persons who will perform the activity; 805 806 f) The NRA should ensure that any actions needed to address risks and opportunities associated with 807 the processes are implemented; 808 809 g) The NRA should analyse and evaluate monitoring and measuring data (see c above); and implement 810 any changes needed to ensure that these processes consistently achieve their intended results; and 811 812 h) The NRA can use the results of analysis and evaluation (see ‘g’ above) to determine the necessary 813 actions for improvement. Working document QAS/19.783 Page 29

814 Figure 2. Example of interaction of processes

815 816 Business processes of the NRAs are shown in the centre of figure 2 (operation box) which includes four 817 processes of NRAs related to MA, VL, MC, & LR. Horizontally, the customer requirements are 818 captured to the left of the operation box and, to the right, delivery of products and services to the 819 customer is shown. Vertically, top management, at the bottom, provides its leadership and commitment 820 for QMS to achieve its intended results. Monitoring and measurement data of the processes/services 821 and customer feedback data, when analysed and evaluated, provide information on performance of 822 QMS. Output of performance evaluation can be used for initiating improvement of QMS/services. Two 823 vertical bars on left and right of the figure demonstrate that the QMS is based upon the context of the 824 organization and planning of QMS has been done based upon the context information. 825 826 MA, LR, VL and MC are used as models throughout this guideline; however, a similar approach can 827 be taken to address the other regulatory functions. Practical help boxes 4 to 8 and Table 2 illustrate the 828 processes for carrying out each of the above-mentioned functions. It is worth noting that it is not the 829 purpose of the process flows provided in figures 3 to 6 to represent a recommendation about the steps 830 required to exercise each of these functions. These are just provided as examples to explain the 831 relationship between the processes, the related inputs/outputs, monitoring points and established 832 controls, indicators used, resources needed including roles and responsibilities, authorities and risks and 833 opportunities for improvement described in the help boxes. Different NRAs can have different ways 834 of approaching the functions and, hence, the steps involved and the relationships between steps and/or 835 inter related processes may differ. 836 29

Working document QAS/19.783 Page 30

837 Practical help box 4. Guidance to assist in the interpretation of clause 4.4 838 839 Characteristics of the processes and their interrelationships involved in the MA function 840 841 Figure 3. Processes and their interrelationships in conducting the MA function

842 843 844 Inputs 845 846 Laws, regulations, mandate, guidelines, access to laboratory for sample testing, market authorization applications, 847 site master files, outcome of inspections, including: good manufacturing practice (GMP), good clinical practice 848 (GCP), good distribution practice (GDP), others 849 850 Steps 851 852 Dossier screening, dossier evaluation (quality, safety, efficacy), laboratory analysis, inspection report, expert 853 committee review and decision making, final approval by Top Management and update of the list of registered 854 medicines. 855 856 Outputs 857 858 Screening results (checklist), dossier acceptance letter, dossier review reports, rounds of questions to the 859 manufacturer and other pertinent communications, test results, inspection reports and certificates. Update of 860 database, committee decision and MA approval or rejection. 861 862 Working document QAS/19.783 Page 31

863 Main processes 864 865 Receipt of application, screening, evaluation, including cycles of questions and responses, granting MA or 866 rejection 867 868 Interacting processes 869 870 Laboratory analysis, GMP inspection, review by expert committee, email communications, meeting minutes, IT 871 platform, official files, letters, meetings (expert meetings) 872 873 Examples of criteria and methods in place to ensure effective operation and control of processes: control 874 points and performance indicators 875 876 Criteria that must be monitored to ensure that processes are properly executed will be based on the criticality of 877 the processes and steps of the processes. Guidelines and SOPs will define elements such as the target evaluation 878 timeframe. Control points will be defined, and indicators chosen in such a way as to allow to monitor parameters 879 of performance. 880 881 Control points 882 883 Screening and dossier review 884 885 Performance indicators 886 887 Key performance indicators (KPI) that measure the actions and events that lead to a result as well as the frequency 888 of the evaluation must be established. The KPIs, particularly if they are carefully developed, represent an excellent 889 tool to monitor performance of the NRA. When setting KPIs use a quantitative method whenever possible and 890 determine appropriate numerators and denominators. [27]. 891 892 Examples of performance indicators include 893 894 Percentage of applications that have been screened within the specified timeline. 895 896 Compliance with defined review timeline 897 898 Quality of the evaluation reports, e.g. evaluation report assessed by three evaluators of different level of seniority 899 yields similar results 900 901 Number of new products listed in the register in a year in relation to the number of applications received 902 31

Working document QAS/19.783 Page 32

903 Potential indicators for other possible control points 904 905 Compliance with overall timeline for registration 906 907 Customer satisfaction evaluated through complaints, surveys, questionnaires, percentage of approved appeals, 908 others 909 910 Use of internal audits to assess performance 911 912 913 Working document QAS/19.783 Page 33

914 Practical help box 5. Guidance to assist in the interpretation of clause 4.4 915 916 Characteristics of the processes and their interrelationships involved in the VL function 917 918 Figure 4. Processes and their interrelationships involved in conducting the VL function

919 920 921 Example of inputs, steps and outputs of processes and processes interrelationships 922 923 Inputs 924 925 Information received from patients, health professionals, international vigilance (VL) networks, industry, media, 926 risk management plans, clinical trials or PMS, suspect product, adverse event reporting, risk management plan, 927 post- market surveillance, clinical trial data. 928 929 Steps 930 931 Receipt, analysis, conclusion, reporting, feedback 932 933 Outputs 934 935 Communication of outcome (positive or negative), regulatory measures, alerts, recalls, risk minimization plan, 936 medical product information provided to patients, health professionals, international VL networks, industry, media 937 and feedback to reporting source 938 939

Working document QAS/19.783 Page 34

940 Examples of criteria and methods in place to ensure effective operation and control of processes: control 941 points and performance indicators 942 943 Criteria that must be monitored to ensure that processes are properly executed will be based on the criticality of 944 the processes and steps of the processes. Guidelines and SOPs will define elements such as the target evaluation 945 timeframe. Control points will be defined, and indicators chosen in such a way as to allow to monitor parameters 946 of performance. 947 948 Criteria for monitoring performance can include 949 950 • Structural indicators should measure systems and physical infrastructure; 951 • Assessments/evaluations/reviews should be timely (according to severity of signals); 952 • Evaluation should address all relevant aspects of the VL system (quality of the evaluation); 953 • The evaluation strategy should include outcomes that can be realistically measured, to avoid inaccurate or 954 misleading data; 955 • Indicators should provide an assessment of current PV documentation and resource compliance with 956 regulatory VL expectations and requirements. 957 • KPI should be re-evaluated to assess their relevance as indicators, and targets can be re-set when deemed 958 appropriate. 959 • As a consequence of monitoring VL System performance, corrective and preventive measures must be 960 implemented, resulting in continuous improvements to the VL System. 961 962 Control points 963 964 Triage/prioritization, data collection and verification, coding of adverse event descriptions, 965 quality of case causality assessment, timeliness, dissemination. 966 967 Performance indicators 968 969 Key performance indicators (KPI) that measure the actions and events that lead to a result as well as the frequency 970 of the evaluation must be established. When setting KPIs use a quantitative method whenever possible and 971 determine appropriate numerators and denominators. [27]. 972 973 Examples of performance indicators include 974 975 Examples of performance indicators include the Number of vigilance inspections performed against planned based 976 on prioritization criteria for inspection. 977 Working document QAS/19.783 Page 35

978 Number of Adverse Drug Reaction reports received from healthcare professionals, from the media (data collection 979 mechanisms). 980 981 Percentage of fatal adverse drug reactions analyzed within target timeline, percentage of serious adverse drug 982 reactions analyzed within target timeline. 983 984 Number of complaints addressed vs. total number of complaints received by the VL department. 985 986 Number of recalled products “controlled” vs recalled products. 987 988 Internal audit findings.

Working document QAS/19.783 Page 36

989 Practical help box 6. Guidance to assist in the interpretation of clause 4.4 990 Characteristics of the processes and their interrelationships involved in the MC function 991 Figure 5. Processes and their interrelationships involved in conducting the market surveillance 992 and control (MC) function 993 994 Start GDP Sampling Plan 995 Medical Product in Market Control of import/ export Market Complaint 996 Internet Sales Inspection report 997 Sampling 998 999 Control 1000 Testing point 1001 Test 1002 report 1003 1004 Summary No Identify SF 1005 report Control 1006 point 1007 Yes 1008 End 1009 Identification of Products & Personnel 1010 involved in SF 1011 1012 1013 Involvement of NRA, Intelligence, 1014 Enforcement, Police, Whistle Blow, 1015 Manufacturer, Distribution, Retailer 1016 1017 1018 Recall & Communication of SF to all Stake Outcomes of MC 1019 Reconciliation of holders: NRA, All within NRA, MAH, activities shared with Control 1020 Quantity Supply chain, User (Patient), Healthcare NRAs and other point 1021 professional, Other NRA, Public domain stakeholders Control 1022 Disposal of Recalled point 1023 products 1024 Report of defective product/ Rapid Alert Notification Working document QAS/19.783 Page 37

1025 MC requires the NRA (in collaboration with other relevant authorities e.g. customs) to ensure that substandard 1026 and falsified (SF) products do not enter/or are removed from the national market. It mandates the NRA to ask for 1027 all transactions relating to importation and/or exportation of consignments of medical products to be conducted 1028 by licensed entities and that good storage and distribution practices be followed. 1029 1030 Example of inputs, steps and outputs of processes, processes interrelationships 1031 1032 Inputs 1033 1034 Market complaint, market intelligence, inspection reports, sampling plan outcome, feedback from import and 1035 export activities and internet pharmacy. 1036 1037 Steps 1038 1039 Risk-based sampling, testing, identifying SF, decision on recall and communication to all stakeholders, including 1040 all relevant parties within NRA, market authorisation holder, supply chain, health care professionals, patients, 1041 international organisations others. 1042 1043 Outputs 1044 1045 Identification of SF, alerts, recalls, communication to all stakeholders and database for the SF. 1046 1047 Examples of criteria and methods in place to ensure effective operation and control of processes: control 1048 points and performance indicators 1049 1050 Control points 1051 1052 Sampling and testing, identification of SF, recalls and related reconciliation and effective communication to 1053 stakeholders 1054 1055 Performance indicators 1056 1057 Key performance indicators (KPI) that measure the actions and events that lead to a result as well as the frequency 1058 of the evaluation must be established. When setting up KPIs use a quantitative method whenever possible and 1059 determine appropriate numerators and denominators. [27]. 1060 1061 Examples of performance indicators include 1062 1063 Number of consignments received through the port of entry. 1064 Number of samples drawn against planned. 37

Working document QAS/19.783 Page 38

1065 Number of samples sent for testing and number tested. 1066 Time taken to generate test report against the target timeline. 1067 Time taken to evaluate suspected products against the target timeline. Working document QAS/19.783 Page 39

1068 Practical help box 7. Guidance to assist in the interpretation of clause 4.4 1069 1070 Characteristics of the processes and their interrelationships involved in the LR function 1071 1072 Figure 6. Processes and their interrelationships involved in conducting the LR function

1073 1074 1075 Inputs 1076 1077 Cover letter, summary lot protocol (SLP), samples, marketing authorization specifications, information on adverse 1078 events, surveillance data (test results on samples retrieved from the market). 1079 1080 Steps 1081 1082 Screening documents, request testing, perform testing, evaluation of SLP and testing results, decision, refer for 1083 review by technical committee. 1084 1085 Outputs 1086 1087 Notification of rejection, lot release certificate. 1088 1089 1090 39

Working document QAS/19.783 Page 40

1091 Main processes 1092 1093 Screening documents (cover letter and SLP), evaluation of SLP, review by the technical committee, decision- 1094 making process. 1095 1096 Interacting processes 1097 1098 Sample testing can be considered an interacting process if this is contracted out to a third-party laboratory. 1099 Otherwise, it is a process that must be performed to yield an output. 1100 1101 Examples of criteria and methods in place to ensure effective operation and control of processes: control 1102 points and performance indicators 1103 1104 Criteria that must be monitored to ensure that processes are properly executed will be based on the criticality of 1105 the processes and steps of the processes. Guidelines and SOPs will define elements such as the target evaluation 1106 timeframe. Control points will be defined, and indicators chosen in such a way as to allow to monitor parameters 1107 of performance. 1108 1109 Control points 1110 1111 Evaluation of the SLP. 1112 Expert committee review (of the report). 1113 1114 Performance indicators 1115 1116 Key performance indicators (KPI) that measure the actions and events that lead to a result as well as the frequency 1117 of the evaluation must be established. When setting KPIs use a quantitative method whenever possible and 1118 determine appropriate numerators and denominators. [27]. 1119 1120 Examples of performance indicators include 1121 1122 Compliance with evaluation timelines. 1123 Check inputs in laboratory information management system (LIMS), checklist, outputs - percentage of outputs 1124 verified/validated in quality review. 1125 Trend analysis for test results. 1126 Percentage of timely reviews by the Expert review committee. 1127 1128 1129 Working document QAS/19.783 Page 41

1130 Potential indicators for other possible control points 1131 1132 Customer satisfaction evaluated through complaints, surveys, questionnaires. 1133 Use of internal audits to assess performance. 1134 1135 Practical help box 8. Guidance to assist in the interpretation of clause 4.4 relating to resources 1136 1137 Resources, roles and responsibilities, and authorities required to ensure adequate performance of processes 1138 for delivery of quality services by NRAs (common to all functions). 1139 1140 Resources, roles and responsibilities, authorities 1141 1142 Human resources should be allocated in line with the processes to be executed as well as the workload. Each 1143 employee has a job description and needs to be trained and qualified to perform his/her job. Roles and 1144 responsibilities and lines of authority should be detailed in the job description and organizational chart. Each 1145 process should have appropriate staffing and managers responsible for it. Staff performance, including 1146 performance of managers, should be evaluated regularly and re-training provided as needed. 1147 1148 Human resources 1149 1150 May include receptionist, administrative staff, screening officer, case investigation experts, evaluators, leadership, 1151 process supervisors, laboratory analysts, IT staff, human resources staff, expert committee members, regulatory 1152 inspectors, housekeeping staff, driver, other. 1153 1154 Proper infrastructure should be in place to carry out the activities (processes), e.g. if adequate laboratory 1155 infrastructure is not in place, consider contracting the service of a qualified laboratory. 1156 1157 Examples of aspects to be considered in terms of infrastructure, including facilities, IT, financial resources, 1158 documentation and work environment. 1159 1160 Infrastructure 1161 1162 • Adequate work space 1163 • Equipment as needed 1164 • Fully established lab or access to a contracted laboratory 1165 • Means of transportation 1166 • IT system, computers and software, databases, archiving system 1167 1168

Working document QAS/19.783 Page 42

1169 Financial resources 1170 1171 Financial resources to buy appropriate equipment, secure its maintenance and procure consumables. Computer 1172 systems and databases need to be validated. Hiring and retaining a sufficient number of qualified staff requires 1173 competitive salaries 1174 1175 Documentation System 1176 1177 Required documents include: strategic direction, vision and mission, laws and regulations, quality policy, 1178 guidelines, lot release policy, SOPs, forms, instructions and checklists. The NRA should establish a system for 1179 documentation preparation, review and approval as well as documentation control, revision and recordkeeping 1180 (see also guidance under clause 7.0). 1181 1182 Work environment 1183 1184 Social, physical and psychological factors all contribute to establishing an environment conducive to quality work; 1185 e.g. non-discriminatory, non-confrontational, stress-reducing, physically comfortable (lighting, temperature, 1186 ventilation, others). 1187 1188 During the planning stage, the NRA should address risks and opportunities in accordance with the 1189 requirements set forth in 6.1. 1190 1191 Table 2. Risks and opportunities affecting MA, LR, VL and MC 1192 1193 NOTE: Text is presented in plain format below to ensure it has line numbers consistent with the rest of 1194 the document, thereby facilitating use of the comment form during public consultation. The table will 1195 be appropriately formatted in the final stages of guideline development. 1196 1197 Marketing authorization 1198 1199 Transparency of NRAs and their work is one of the principles of GRP. Posting as much information as 1200 possible on the internet helps NRAs increase transparency. This information can include the 1201 registration procedure steps and timelines, related regulations and guidelines, charts indicating actual 1202 level of compliance with the target timelines, evaluation reports, others. Posting sensitive information 1203 on the web (e.g. performance charts) constitutes a risk of potential complaints or criticism, but at the 1204 same time offers opportunities for improvement, advocacy of the work performed, reliability, others. 1205 Working document QAS/19.783 Page 43

1206 The evaluation process should be properly monitored and evaluated, including the experts who conduct 1207 the evaluation. Failure to do so can lead to the risk of granting a MA based on an insufficient or 1208 inadequate data package. Risks usually also entail opportunities. In this example, if an NRA does not 1209 have the adequate expertise / resources to evaluate a certain product, reliance on other agencies can be 1210 an option. 1211 1212 Lot release 1213 1214 Tests to be performed as part of lot release must be appropriately validated, including the equipment 1215 used (properly calibrated), the consumables properly tested, released and used before expiry, qualified 1216 analysts to perform the tests who are regularly re-qualified and test performance monitored. Failure to 1217 meet all these requirements lead to the risk of either releasing a lot that does not meet the requirements 1218 or rejecting a lot that meets the specifications. Identification of the specific constraints may also bring 1219 about opportunities to improve the planning for procurement of consumables or equipment that may be 1220 required. 1221 1222 IT failures pose a risk for timely delivery of the service (release of lots), it raises at the same time an 1223 opportunity for renewing the system (hardware/software). 1224 1225 Vigilance 1226 1227 VL function carries a risk of potentially missing out on important signals because of underreporting or 1228 poor analysis and interpretation of the reports. The consequence is harm to the public and loss of 1229 reputation for the NRA. Lack of, or poor communication about, the safety of a product that has been 1230 suspected (as a result of analysis of the signals) may result in public panic and loss of trust on the NRA. 1231 Such failures can offer at the same time an opportunity for improvement and strengthening of the system 1232 1233 There is an increase in the number of registered new medicines (biologicals, biosimilars, others) so that 1234 a robust vigilance system needs to be in place. To establish a robust VL system, a database is developed 1235 to monitor implementation of all the approved Risk Management Plans (RMPs) and to measure their 1236 effectiveness. 1237 1238 The data base is also used for all medicines subject to additional post-marketing monitoring to track 1239 potential safety concerns. Vigilance inspection is one of the tools used to monitor and maintain the VL 1240 system within local companies and agents. 1241

Working document QAS/19.783 Page 44

1242 Market surveillance and control 1243 1244 • Risk of failure to test the sampled products due to limited capacity for testing e.g. reagents, 1245 standards, staffing, other. This may lead to failure in identifying products that have been damaged 1246 in the distribution/ storage chain. This may adversely impact the reputation of the NRA. At the 1247 same time, such an event provides an opportunity to convince TM of the constraints and the need 1248 for resources to prevent a recurrence. 1249 • Lack of expertise for SF case investigation poses a serious risk of missing SF products. It provides 1250 an opportunity to establish or review methods and training of staff in medicine production facilities. 1251 • Unclear communication strategy may lead to mis-communication or missing communicating an SF 1252 to certain relevant stakeholders. This may adversely impact the reputation of the NRA. It provides 1253 an opportunity to review the procedures and personnel responsible for communication of such 1254 events. 1255 • In case a recall of a product is mandated, there is a risk not to recall and dispose the whole batch. 1256 Recall from remote areas may be challenging. It provides an opportunity to empower the national 1257 vigilance system, involve and commit other institutions in the dissemination of regulatory measures 1258 and recall products that are damaged or do not meet the required quality standards or are SF. 1259 1260 Documented information for QMS 1261 1262 The strategic direction of the NRA, mission, vision, policies, quality objectives, as well as procedures 1263 and other information, should be documented (4.4.2). In addition to this, NRAs will also need 1264 Documented Information to support its operations. As per definition, documented information is 1265 information required to be controlled and maintained by NRAs. 1266 1267 At several places ISO Standard 9001:2015 requires: 1268 1269 a) maintaining documented information (which means a manual, procedure, instruction, checklist, 1270 vision/mission/policy/objectives statements, guidelines, specifications, drawings, websites, circulars, 1271 government orders etc.); and 1272 1273 b) retaining documented information (which means records, reports, minutes of the meetings or 1274 any document which provides evidence that the activity has been performed as per applicable 1275 criteria/methods etc.). 1276 1277 Some of the documented information to be retained may be formal (validation reports, audit reports, Working document QAS/19.783 Page 45

1278 others) and other informal (meeting minutes). NRAs should have such documented information (see 1279 clause 7.5 for details of DI). 1280 1281 Practical help box 9. Guidance for interpretation of clause 4.4 relating to documented information 1282 1283 High-level NRA documentation can include the legal basis, regulations, decrees, strategic plan, vision and 1284 mission, overall objectives of the organization, quality objectives, quality policy, quality manual, others. 1285 1286 Lower-level NRA documents can be divided into two categories: documents and records. 1287 1288 Examples of documents include SOPs, instructions and forms and checklists. Examples of records include 1289 assessment reports, inspection reports, test results, application submissions from manufacturers, marketing 1290 authorization dossiers, SLPs, correspondence, trending data and its analysis, validation and qualification protocols 1291 and reports, calibration data, training plans and records, analysts and evaluators qualification information, 1292 maintenance program, training program and records, internal audit plans and reports, corrective and preventive 1293 action (CAPA) plans and compliance reports, others. 1294 1295 Documentation and records must be controlled. A system should be in place whereby documents are reviewed, 1296 authorized and approved. Newer versions replace older ones which become obsolete. Documentation should not 1297 only be maintained, but also retained for established periods of time, which are defined by each authority 1298 according to established rules (generally not less than five years for some documents and not less than 10 years 1299 for others). A documentation control system should be established to ensure that all relevant areas have the 1300 required documented information and that only the latest (current) version is available at any point in time. (See 1301 also clause 7.0). 1302 Clause 5. Leadership 1303 1304 5.1 Leadership and commitment 1305 1306 Guidance 1307 1308 The intent of this clause is to ensure that NRA TM demonstrate leadership and commitment by taking 1309 an active role in engaging, promoting, communicating and monitoring the performance and 1310 effectiveness of the QMS. 1311 1312 The clause requires TM to demonstrate leadership and commitment with respect to the QMS. To 1313 achieve this, TM should comply with conditions (a-j) listed in the standard. In brief, TM is accountable 1314 for the effectiveness of the QMS and its integration into the core processes of the NRA by supporting 1315 the critical characteristics of the QMS as defined and described in ISO Standard 9001:2015. This 45

Working document QAS/19.783 Page 46

1316 includes promoting the use of the process approach through the PDCA cycle and risk-based thinking to 1317 ensure that the quality policy and quality objectives are compatible with the context and strategic 1318 direction of the NRA; ensuring that the required resources are available to support staff to contribute to 1319 the effectiveness of the system; supporting other managers in their respective roles to demonstrate their 1320 leadership; and promoting improvement. Experience shows that leadership and commitment are 1321 essential requirements for successful implementation of a QMS. 1322 1323 An important element of the ISO Standard 9001:2015 standard is its emphasis on customer focus. In 1324 this respect, TM should demonstrate its leadership and commitment to the QMS by continually 1325 identifying the needs and expectations of its customers (Drug importers, Drug manufacturers, Drug 1326 outlet operators, Patients, Medical Practitioners, Research institutions etc), as well as ensuring that the 1327 NRA fulfils applicable statutory and regulatory requirements. 1328 1329 In many cases, a focus for on-time delivery performance and on customer complaints can provide 1330 information on any actions that might be necessary to achieve or improve customer satisfaction. 1331 1332 The NRA also needs to ensure that appropriate actions are implemented to address risks and 1333 opportunities that can affect customer satisfaction. To increase customer satisfaction, innovation and 1334 best practices can be introduced into the NRA processes. 1335 1336 5.2 Quality Policy 1337 1338 Guidance 1339 1340 Two key aspects are covered in this clause, namely the development of a quality policy and 1341 communicating the quality policy to the NRA personnel. 1342 1343 The quality policy is a powerful and highly visible statement of intent towards quality services by the 1344 TM signed by the Director or Head of the NRA (in the case of a discrete NRAs, this responsibility may 1345 fall in the hands of the Ministry of Health). 1346 1347 While establishing a quality policy, the NRA TM should keep in view its purpose and strategic direction 1348 (mission, vision, guiding principles and core values). Good regulatory practices (see 2.0 General 1349 considerations of this guideline) and quality management principles (see clause 0.2) can also be used 1350 for establishing commitment of the NRA TM towards quality services. 1351 Two commitments should come out clearly in the statement of quality policy: Working document QAS/19.783 Page 47

1352 1353 • Commitment to satisfy customer or stakeholders requirements as well as applicable statutory and 1354 regulatory requirements; and 1355 • Commitment to continual improvement of the QMS. 1356 1357 The policy should also provide a framework for setting quality objectives (which means any claims in 1358 the quality policy should be measurable when converted into objective). 1359 1360 TM should ensure that the quality policy is communicated, understood and applied by persons of the 1361 NRA, so they are able to contribute to the effectiveness of the QMS. The policy can be communicated 1362 by different methods such as via noticeboards, screensavers, by the organization’s website, or during 1363 routine meetings. In addition, the NRA can make the quality policy available, as appropriate, to relevant 1364 interested parties such as external providers (service providers/suppliers), partners, customers and 1365 governmental agencies, for example, by displaying it on NRA’s website. 1366 1367 Practical help box 10. Guidance for interpretation of clause 5.2 1368 1369 Example of quality policy for NRAs from countries X, Y and Z. 1370 1371 1) Country X National Regulatory Authority (XNRA) quality policy. (Approved by XNRA management) 1372 1373 XNRA is committed to meet the needs and expectations of customers through continual improvement of its 1374 processes and quality services by implementing QMS effectively according to ISO Standard 9001:2015 1375 requirements. We will ensure quality, safety and/or efficacy of food, medicines, cosmetics and medical devices 1376 in compliance with the XNRA Drug Act 1:2006. We should establish objectives at system and departmental level 1377 that ensure that the requirements of this policy are met. Top Management is committed to providing the necessary 1378 resources to ensure maintenance and continuous improvement of QMS. 1379 1380 Executive Director signature 1381 “Together we protect public health” 1382 1383 2) The YNRA is committed to protect the health of people of the country and fulfil its duties with professional 1384 and scientific rigor, while ensuring safety, efficacy and quality of Allopathic, Homeopathic and Herbal 1385 medicines, vaccines, and biological products according to the Drugs Act and Rules and future amendments. 1386 1387 YNRA should work in effective, transparent and timely manner, ensuring implementation of Quality Management 1388 System and to ensure its continuing improvement. 1389 To meet our commitment, we must: 47

Working document QAS/19.783 Page 48

1390 1391 — Foster a team approach. 1392 — Emphasize appropriate training for all employees. 1393 — Recognize each employee's responsibility for quality. 1394 — Provide regulations with timely written corrective actions. 1395 — Earn recognition of our quality process and progress. 1396 — Provide a framework for establishing and reviewing quality objectives. 1397 — Develop and achieve Quality Improvement Goals. 1398 — Maintain our honesty and integrity by following our Code of Conduct. 1399 — Review and renew this Quality Policy on a regular basis. 1400 1401 3) “ZNRA is committed to provide quality services in response to customer needs and expectations. We should 1402 strive to balance the interests of our stakeholders without compromising quality, safety and/or effectiveness 1403 of food, drugs, cosmetics and medical devices by managing the Authority with utmost professionalism. We 1404 commit ourselves to comply with requirements of the ISO 9001:2008 standard and continually improve 1405 effectiveness of Quality Management System. We should manage and provide resources for continuous 1406 improvement of our services to ensure customer satisfaction”. 1407 1408 5.3 Organizational roles, responsibilities and authorities 1409 1410 Guidance 1411 1412 The NRA TM will need to establish specific responsibilities and authorities for the assigned roles and 1413 ensure that persons of the NRA understand and are aware of their assignments. These could be 1414 communicated through job descriptions, work instructions, duty statements, organization charts, 1415 manuals, procedures, others. Adequate resources are required to match the needs. 1416 1417 Items a) and b) in the clause - the QMS conforms to the requirements of the ISO Standard 9001:2015 1418 [7] and processes are delivering the intended outputs - describe roles to be assigned to each process 1419 owner (managers), while items c) to e) in the clause - reporting on QMS performance, promoting 1420 customer focus and maintaining the integrity of the QMS when changes are made - describe roles to be 1421 assigned to specific persons. Although certain responsibilities are delegated, the overall responsibility 1422 and accountability for the QMS remains with TM. 1423 1424 1425 1426 Practical help box 11. Guidance for interpretation of clause 5.3 Working document QAS/19.783 Page 49

1427 1428 Ideally, NRAs will have a QMS unit in place, or a responsible officer as a minimum within each Unit, Department 1429 or Directorate, as management representative or QMS coordinator who can report on QMS performance, promote 1430 customer focus and maintain the integrity of the QMS when changes are made. 1431 1432 Responsibility for ensuring that the QMS conforms to the requirements of the ISO Standard 9001:2015 and 1433 processes are delivering the intended outputs should be included in staff job descriptions and assessed during 1434 performance evaluation. Staff should be trained in QMS (conferences, meetings, online platform). Trainings 1435 should be relevant to the regulatory functions and reflected in documented information. 1436 1437 Example of country YNRA 1438 1439 The Deputy Director of the agency has been designated as the representative of the agency in quality management. 1440 His responsibilities and authority include: 1441 1442 • To ensure that the necessary processes for quality management are established, implemented and maintained. 1443 • To inform senior management of system operation, including the needs for improvement. 1444 • To promote awareness of customer requirements at all levels of the organization. 1445 1446 The responsibility of the management representative includes relationships with external parties on matters related 1447 to the system. He is the designated Quality Assurance Manager. 1448

1449 Clause 6. Planning 1450 1451 6.1 Actions to address risks and opportunities 1452 1453 Guidance 1454

1455 The intent of clause 6.1 is to ensure that when the NRA plans its QMS processes, it identifies its risks 1456 and opportunities and plans actions to address them. The purpose of this clause is to prevent 1457 nonconformities, including errors in outputs, and to determine opportunities that might enhance 1458 customer satisfaction or achieve quality objectives.

1459 When determining risks and opportunities, the NRA should focus on enhancing desirable effects, 1460 preventing or reducing undesired effects (through preventive actions or risk reduction). This is adopting 1461 a "risk-based approach" and the NRA should consider the application of this approach to all processes 1462 required for its QMS.

Working document QAS/19.783 Page 50

1463 The NRA can choose the methods of risk determination that suit its needs. The simpler approaches 1464 include techniques such as structured brainstorming, "what if?” method, consequences/probability 1465 matrices, etc. For guidance, the NRAs may refer to the international standard ISO/IEC 31010 that 1466 provides a list of risk assessment tools and techniques. 1467 1468 When examining opportunities, potential risks to the QMS associated with them should also be 1469 determined; the results of such determinations should be used when making decisions on whether to 1470 implement the opportunities. 1471 1472 The application of risk-based thinking can also help the NRAs to develop a proactive and preventive 1473 culture focused on doing things better and improving how work is done in general.

1474 Once the risks and opportunities are identified, actions must be planned to address them. Actions are 1475 planned, implemented, analysed and evaluated to assess their effectiveness.

1476 The actions taken to address risks will depend on the nature of the risk (its probability/frequency and 1477 severity), for example:

1478 a) The risk can be avoided by no longer performing the process where the risk can be encountered 1479 (risk is terminated). 1480 b) The risk can be eliminated by assisting persons in the organization with less experience or by 1481 capacity building (risk is treated). 1482 c) The risk can be shared by out sourcing the process or taking an insurance cover (risk is transferred). 1483 d) The risk can be accepted, and no action taken, based on its potential effect or the cost of the needed 1484 action (risk is tolerated). 1485 1486 The above alternative actions are also termed as 4T (terminate, treat, transfer or tolerate) methods of 1487 treating the risks. 1488 1489 Practical help box 12. Guidance for interpretation of clause 6.1 1490 1491 Example of actions to address risks and opportunities for lot release process 1492 1493 It is foreseen that there will be an increased demand for release of batches of a certain vaccine the following year. 1494 The analysis of risks and opportunities to meet the increase in demand includes an assessment of the current 1495 situation (process capacity), an analysis of the risks of attempting to meet the demand under the present conditions, 1496 and the opportunities that arise from this new situation. The release process requires analysts to test the vaccine 1497 batches, reviewers to go through the SLP, and professionals to prepare the report to be reviewed by the Technical Working document QAS/19.783 Page 51

1498 Committee, plus the final approval by the Head of Agency. The risks associated with addressing the increased 1499 demand include the fact that testing as well as review capacity may not be sufficient, and since the Technical 1500 Committee meets only once per month, the response capacity may not be enough. 1501 1502 As part of the planning process, the team needs to assess the risk of releasing lots that do not meet specifications 1503 on one hand; and the risk of not releasing a lot that meets the specifications due to limited capacity on the other. 1504 In addition, the likelihood (probability) of failure in service providing, the impact on the quality of the products 1505 released, the estimated frequency at which errors could happen, the potential impact on customer’s satisfaction 1506 and the credibility of the institution (seriousness) in case of not meeting the increased demand, or in case the 1507 service provided is inadequate, should all be considered. 1508 1509 The analysis of risk and opportunities provides projections for changes to be introduced in the system to effectively 1510 and efficiently address the increased demand. For example, if surge in testing capacity cannot be implemented or 1511 is not economically feasible, a new prioritization mechanism based on knowledge of the different products to be 1512 released and record of the manufacturers could be put in place. 1513 1514 Through this analysis, it may be estimated that by increasing the staff by one analyst and increasing committee 1515 meetings to two per month, the demand could be appropriately addressed (opportunity for increased resources). 1516 1517 6.2 Quality objectives and planning to achieve them 1518 1519 Guidance 1520 1521 The intent of this clause is to ensure that the NRA establishes quality objectives and plans appropriate 1522 actions to achieve them. Quality objectives should be established for relevant functions, levels and 1523 processes, as appropriate, to ensure the effective deployment of the NRA’s strategic direction (plans) 1524 and its quality policy. Whenever possible, they should be SMART objectives (Specific, measurable, 1525 achievable, realistic and time bound. The following table provides guidance on implementation of 1526 bullets ‘a’ to ‘g’ of clause 6.2 1527 1528 Table 3. Guidance for development of quality objectives 1529 Requirement Intent with example

a. Be consistent with the quality policy Use commitments made in quality policy for setting quality objectives e. g. setting objectives on continual improvement of QMS as committed in quality policy

b. Be measurable Define quantity or period e. g. processing time of customer request will be reduced from 2 to 1 day

Working document QAS/19.783 Page 52

c. Address applicable requirements For example, if certain regulatory requirement relating to product/service are applicable, setting objectives for that

Using WHO GRP for setting objectives

d. Be relevant to conformity of products For example, ‘On Time and In Full’ delivery of service, setting and services and enhanced customer targets for achieving higher level of customer satisfaction satisfaction

e. Be monitored Means being reviewed for progress being made in achieving the quality objective; this could be carried out through analysis of process monitoring and customer feedback data and comparing results with set targets

f. Be communicated For example, through circulation of minutes of meetings internally and to external interested parties viz suppliers by signing agreements

g. Be updated as appropriate Potential or actual changes that can impact on the ability to achieve quality objectives need to be considered and action taken as necessary, to ensure new issues or requirements are addressed.

1530 1531 A plan should be in place to ensure that the set objectives will be met. The planning exercise includes 1532 determining the actions that will need to be taken, the resources that will be required (e.g. human and 1533 financial to purchase equipment and the required supplies), assigning responsibilities to staff for specific 1534 tasks, determining timelines for completion of each step and deciding means to be used for measuring 1535 and evaluating whether the objectives have been achieved or not (see also clauses 9.1 to 9.3). 1536 1537 Practical help box 13. Guidance for interpretation of clause 6.2 1538 1539 Example of mission, vision and quality objectives for an NRA 1540 1541 XNRA mission statement 1542 1543 The mission of XNRA is to protect and promote public health by ensuring quality, safety and/or efficacy of food, 1544 medicines, cosmetics and medical devices. 1545 1546 XNRA vision statement 1547 1548 The vision of XNRA is to provide the best regulatory services to ensure the quality of food, drugs and cosmetics 1549 in the southern hemisphere by 2020. 1550 1551 XNRA quality objectives 1552 Working document QAS/19.783 Page 53

1553 The quality objectives of XNRA are established in line with the goals outlined in the XNRA strategic plan for the 1554 period of 2015-2020. These objectives are: 1555 1556 a) Maintain good governance and management of the agency with view at ensuring continuing improvement of 1557 QMS. 1558 b) Continuously improve the quality of service through regular training of staff, monitoring of performance and 1559 monitoring compliance with set review timelines. 1560 c) Strengthen laboratory services by conducting the validation of all test methods used and the qualification of 1561 staff involved in such tests by 2020. 1562 d) Strengthen cooperation and collaboration with relevant organizations and government agencies. 1563 1564 Planning to achieve quality objectives 1565 1566 The achievement of XNRA quality objectives should be through implementation of specific actions as detailed in 1567 the current XNRA strategic plan. 1568 1569 XNRA determines and provides resources (including human, financial, infrastructure, technology, work 1570 environment and organizational knowledge needed to establish, implement, maintain and continuously improve 1571 QMS. The resource requirements are defined through budgeting and other business management processes 1572 including planning and management review. 1573 1574 TM is ultimately responsible for quality of XNRA services by ensuring the resources, systems and processes 1575 needed to implement and improve QMS and for undertaking management review meetings. All employees are 1576 responsible for the quality of their work and implementation of the policies and procedures applicable to the 1577 processes they perform. 1578 1579 The quality objectives should be achieved by 2020 and will be evaluated by undertaking quality internal audits 1580 and analyzing performance data for continual improvement of the system with the overall aim of meeting 1581 customers’ needs and expectations. 1582 1583 Another example - ZNRA Quality objectives 1584 1585 Objective 1: The rate of counterfeit and substandard food, medicine, cosmetics and medical devices circulating 1586 in the country reduced by 50% by June 2020 1587 Objective 2: Customer satisfaction for services offered by ZNRA increased by 80% for both internal and external 1588 customers from 63% and 66% respectively by June 2020 1589 Objective 3: ZNRA self-sustained financially from 60% to 80% by June 2020 1590 Objective 4: 90% of human resources recruited and retained by June 2020 1591 6.3 Planning of changes 1592 53

Working document QAS/19.783 Page 54

1593 Guidance 1594 1595 The intent of this clause is to determine the need for changes to QMS to adapt to changes in the NRA’s 1596 context/business environment, as well as to ensure that any proposed changes are planned, introduced 1597 and implemented in a controlled manner. 1598 1599 The purpose of planning the change is to maintain the integrity of the QMS and ensure the NRA’s 1600 ability to continue to provide conforming products and services during the change. 1601 1602 The need for changes can result from, changing needs of customers and other relevant interested parties, 1603 new products to be evaluated to grant market authorization, changing process methods to improve 1604 trends in non-conforming outputs, using new information and communication technology (ICT) for a 1605 service or process, outsourcing important processes, persons in key roles leaving (either due to 1606 retirement, job change or other), or moving to online service provision. 1607 1608 The NRA should consider the availability of resources and necessary allocation or reallocation of 1609 responsibilities for any change. This could be done by assigning persons to a team to manage the 1610 change, or by delaying the change until the right resources are available. 1611 1612 Practical help box 14. Guidance for the interpretation of clause 6.3 1613 1614 In planning for an increased demand for lot release the following year, a risk analysis determines that the NCL 1615 will need to add one more SLP evaluator, increase the technical committee meetings to twice per month, and 1616 prioritize lot testing. These measures entail a change in the processes which must be planned, documented, 1617 integrated to the QMS and properly monitored. In this manner, the organization is considering the potential impact 1618 of the change, the availability of resources, and the allocation or reallocation of responsibilities thereby conserving 1619 the integrity of the QMS. 1620 1621 1622 1623 1624 1625 1626 Clause 7. Support 1627 1628 7.1 Resources Working document QAS/19.783 Page 55

1629 1630 Guidance 1631 1632 The intent of this clause is to ensure that the resources necessary for the establishment, implementation, 1633 maintenance and continual improvement of QMS are available to the NRA for its effective operation. 1634 1635 In determining the resources that need to be provided, the NRA should consider the current capabilities 1636 of its internal resources (e.g. people, capability of equipment, organizational knowledge) and any 1637 constraints (e.g. budget, number of resources, schedule). A decision should then be made on the 1638 resources needed, including those to be sourced externally, and the necessary actions taken to ensure 1639 the resources needed are provided; this applies to all resources listed under sub-clauses of resources 1640 from 7.1.2 to 7.1.6 of the Standard. 1641 1642 Three important categories of resources need to be considered; people, infrastructure and environment 1643 for the operation of processes. To plan for adequate resources in quality and quantity, three steps are 1644 to be followed: 1645 1646 a) Determine what resources are needed (number of people and level of competence required, utilities, 1647 facilities, equipment including hardware and software needed as well as good working conditions; 1648 physical such as temperature control, level of lightening, etc and human conditions to ensure an 1649 adequate work environment). 1650 b) Plan how and when these are going to be provided. 1651 c) Plan for the means to ensure that the resources provided are maintained (periodic preventive 1652 maintenance) and controlled as needed. 1653 1654 Monitoring and measurement resources 1655 1656 The intent of the clause is also to ensure that the NRA determines and provides suitable resources 1657 (measurement equipment, instruments, etc.) to ensure valid and reliable monitoring and measuring 1658 results when evaluating the conformity of its products and services. 1659 1660 Monitoring implies critical observation, supervision and checks to determine the quantitative or 1661 qualitative status (or both) of an activity, a process, a product, or a service. Measurement considers the 1662 determination of a quantity, magnitude, or speed, by using suitable measuring resources. This can 1663 include the use of calibrated or verified equipment that is traceable to national or international 1664 measurement standards. For services, it can include the use of known and validated models for service 55

Working document QAS/19.783 Page 56

1665 feedback, for example social service models. 1666 1667 In determining the criticality of monitoring and measurements to ensure valid results, the NRA should 1668 determine what needs to be monitored and/or measured for its processes, products and services. It 1669 should then determine the resources needed for this monitoring and measuring; ensuring its 1670 suitability/fitness for the purpose. Resources used should be maintained for their continuing fitness. 1671 1672 Documented information to be retained can include schedules outlining how often checks are needed 1673 to ensure valid results as well as reports of results/outcomes. 1674 1675 Measurements need to be traceable (to national and/or international measurement standards) when it is 1676 a requirement or when the NRA determines it to be necessary to have confidence in the validity of the 1677 measurement results. 1678 1679 If measuring equipment is used to verify conformity to requirements and to provide confidence in the 1680 validity of measurement results, the NRA should consider how the measuring equipment is verified 1681 and/or calibrated, identified with calibration status, safeguarded from adjustments, stored, used and 1682 maintained. 1683 1684 If measuring equipment is found to be unfit for the intended purpose, the potential impact on compliance 1685 with measurement requirements should be reviewed and necessary actions taken. The results of such a 1686 review can also indicate that no action is required or, alternatively, that a service needs to be performed, 1687 products in stock need to be investigated, relevant customers must be informed, or even that a product 1688 recall is required. The level of action needed depends on the conformity of products and services. 1689 1690 Organizational knowledge 1691 1692 The clause also focuses on the need to maintain the knowledge determined as necessary, by the NRA, 1693 for the operation of its processes and to achieve conformity of products and services, as well as to 1694 encourage the acquisition of necessary knowledge based on changing needs and trends. 1695 1696 Organizational knowledge is the specific knowledge of an organization coming either from its collective 1697 experience or from the individual experience of its persons. This knowledge is or can be used to achieve 1698 the organization’s intended results. 1699 1700 The NRAs should consider how to determine and manage the organizational knowledge required to Working document QAS/19.783 Page 57

1701 meet NRA’s present and future needs. Persons and their experience are the foundation of organizational 1702 knowledge. Capturing their experience and knowledge can generate synergies leading to the creation 1703 of new or updated organizational knowledge. 1704 1705 In determining, maintaining and making available organizational knowledge, NRAs can consider: a) 1706 learning from failures, near miss situations and successes; b) gathering knowledge from stakeholders, 1707 experts and partners; c) capturing knowledge that exists within itself. 1708 1709 The tools for maintenance and distribution of organizational knowledge can include the intranet, 1710 libraries, awareness sessions, newsletters, others. 1711 1712 Practical help box 15. Guidance for the interpretation of clause 7.1 relating to organizational 1713 knowledge 1714 1715 The NRA should safeguard the knowledge necessary for its operation and achievement of conformity of products 1716 and services. It should also encourage gaining new knowledge to meet its current and future needs. 1717 1718 Example of measures taken by the NRA of country Y to maintain and update organizational knowledge: 1719 1720 a) YNRA has developed, and updates as needed, detailed job descriptions of personnel responsible for key 1721 processes in the chain that leads to their outputs (products and services), 1722 b) YNRA carries out initial training of new staff and refreshment training of staff at different levels with 1723 new information relevant to their respective positions to keep their competence up to date, 1724 c) Recruitment of new staff is based on job description, the position posted on the website, and candidates 1725 subject to a test and interview before decision-making, 1726 d) In case of staff turnover, and whenever possible, an overlap between the staff leaving the position and 1727 the new staff is sought, so that knowledge is properly transferred to the new staff and opportunity is given to the 1728 incoming person to practice under advice of the person leaving the position, 1729 e) In case of retirement, succession is properly planned through timely recruitment of successor 1730 f) Organizational knowledge refers not only to processes for service delivery, it also includes the broader 1731 perspective (e.g. knowledge of mission, vision, quality policy and objectives, strategic plan and strategic 1732 objectives of the NRA, understanding the context, internal and external issues, customers’ expectations, statutory 1733 and regulatory requirements, relationship with customers and suppliers and with other relevant organizations or 1734 agencies, others). To ensure that this knowledge is maintained and properly communicated to personnel internally, 1735 YNRA organizes meetings at regular intervals where issues are discussed; a newsletter is produced and circulated 1736 through the intranet on monthly basis; in case of urgency email communications are sent to all relevant personnel,

Working document QAS/19.783 Page 58

1737 g) YNRA provides opportunities for training of staff outside the NRA, by participating in technical courses 1738 and through attendance to scientific meetings. Personnel that benefits from such opportunities are required to write 1739 a meeting or training report and to deliver a lecture to colleagues in the NRA for knowledge/ information sharing. 1740 h) Staff benefitting from fellowships abroad are required in addition to g) to stay in the NRA for a time 1741 equal to the double of the duration of the fellow ship. Monthly seminars are organized in which personnel share 1742 experiences from their work with others (e.g. a rejected market authorization application, information on an 1743 innovative product, feedback from the field regarding safety profile of recently registered and commercialized 1744 vaccines). 1745 1746 7.2 Competence 1747 1748 Guidance 1749 1750 The intent of this clause is to identify the necessary competence required to perform individual roles 1751 and responsibilities and to ensure persons carrying out work are competent, based on training, education 1752 and/or experience. The term ‘persons’ includes managers, existing employees, temporary employees, 1753 sub-contractors and their employees, and outsourced persons. 1754 1755 Competence is the ability to apply knowledge and skills to achieve intended results. Demonstrated 1756 competence is sometimes referred to as ‘Qualification’ or ‘licensed person’. 1757 1758 Competence requirements can be determined by, for example: 1759 1760 • Specified performance criteria 1761 • Awareness of specified requirements and acceptance criteria 1762 • Knowledge of processes and controls operated by the organization. 1763 1764 When a person does not meet, or no longer meets, the competence requirements, then actions should be 1765 taken; such as, for example: 1766 1767 • Mentoring the employee 1768 • Providing training 1769 • Simplifying the process so that the person can carry it out successfully 1770 • Reassigning the employee to another position. 1771 Evaluation of competence can be done in several ways, including: 1772 Working document QAS/19.783 Page 59

1773 • Regular supervisor or manager evaluation of persons performing tasks and the operation of 1774 processes; and 1775 • Benchmarking against service performance requirements. 1776 1777 Appropriate documented information that provides evidence of an employee’s competence includes, 1778 e.g. diplomas/degrees, completion of training, resumes, performance reviews and licenses should be 1779 retained. 1780 1781 Practical help box 16. Guidance for interpretation of clause 7.2 1782 1783 The practical help box 15 refers on point c) to the selection and recruitment of staff. The selection process for 1784 recruitment is critical to identify persons competent for the job, the requirements of the position, the acceptance 1785 criteria and the position specific knowledge are reflected in the job description published on YNRA website. The 1786 selection process includes a test and one or more interviews before a decision is made regarding the best candidate. 1787 This process is likely to successfully identify competent candidates. YNRA invests in maintaining competence 1788 of its employees, and in reducing turnover to the maximum possible extent. It has provisions to update knowledge 1789 of personnel through regular refreshment training, participation in scientific/technical meetings and other means. 1790 Competence records are kept as part of the process of acquiring competence. 1791 1792 In case a person no longer meets the requirements of the position, YNRA offers mentoring and retraining, and as 1793 a last option reassigns the person to a different position. There are instances where reassignment of a person to a 1794 different position is not the result of failure but of the initiative of the person to gain experience in a different area 1795 of expertise. Rotation of personnel is also a regular practice in YNRA to facilitate the acquisition of additional 1796 skills/ competencies and to provide incentives. 1797 1798 7.3 Awareness 1799 1800 Guidance 1801 1802 The intent of this clause is to ensure that persons are aware of the quality policy, relevant quality 1803 objectives, their contribution to the effectiveness of QMS and the implications of not conforming to 1804 QMS requirements. 1805 1806 Persons can demonstrate their awareness in day-to-day activities by distinguishing between what is 1807 acceptable and what is not, and by taking appropriate action when processes, products and services do 1808 not meet agreed specifications. 1809

Working document QAS/19.783 Page 60

1810 Depending on the nature of the work that the persons perform, the actions for creating awareness can 1811 vary. Awareness can be created through regular review meetings, gathering feedback and ensuring this 1812 feedback is made known to relevant persons. 1813 1814 NRA staff should be aware of the quality policy and objectives, their role and contribution to an 1815 effective QMS and the implications of not conforming to the QMS requirements. A common practice 1816 is for the TM to post the mission, vision, quality policy, and quality objectives in the entrance of the 1817 NRA or other strategic locations for all staff and customers to see. 1818 1819 Practical help box 17. Guidance for interpretation of clause 7.3 1820 1821 To ensure that personnel are aware of the quality policy and relevant objectives, ZNRA posts them at the entrance 1822 of the building, in every bathroom, library and every meeting room. In addition, ZNRA has distributed slogan 1823 bottoms stating “I adhere to QM policy and objectives” for every single worker in the organization. ZNRA also 1824 placed posters of the Quality Management principles and the benefits of the QMS in improving performance and 1825 implications to the health of the public if not conforming to QMS. ZNRA TM speaks in QMS terms. 1826 1827 7.4 Communication 1828 1829 Guidance 1830 1831 The intent of this clause is to establish the process of internal and external communications. NRAs 1832 needs to decide what needs to be communicated and who needs this information, to determine the most 1833 effective communication method and timing; including who provides the communication. 1834 1835 The NRA should identify those parties with whom they should communicate, to ensure the effective 1836 operation of the QMS, such as customers, suppliers, experts, ministry of health, media and other 1837 stakeholders. 1838 1839 More formal communication might be required for external interested parties, such as reports, invoices 1840 or service level agreements, press briefs, etc. Internal communications can use methods such as regular 1841 department meetings, briefing sessions, email or the intranet. More formal methods, such as written 1842 reports or minutes of the meetings etc., can also be required for internal communication, depending on 1843 the nature of the information and how critical the issues are that need to be communicated. 1844 It is also common for an NRA to designate a specific communication officer who has been trained on 1845 what, how, when, and to whom communicate depending on the matter. Communications outside the Working document QAS/19.783 Page 61

1846 NRA, e.g. the media, should be carried out exclusively by communications-trained officers designated 1847 by NRA TM. 1848 1849 Practical help box 18. Guidance for interpretation of clause 7.4 1850 1851 Example of internal communication requirements at YNRA 1852 1853 As a minimum internal communication is done through the following channels: 1854 1855 • Meeting with Ministry-Joint Secretary (once in a quarter) 1856 • Senior Staff meetings (monthly) 1857 • Full Departmental meetings (weekly) 1858 • Daily operational meeting of the departments and administrative areas 1859 • Email, internet platform and/or telephone. 1860 1861 Example of communication with the customer at YNRA 1862 1863 Communication with customers is maintained through: 1864 1865 • Internet platform, fax, email or post 1866 • Surveys and interviews are used to obtain customers feedback 1867 • Meetings and exchanges between specialists and managers 1868 • Complaints and grievances are handled in quality management following the instructions for handling 1869 complaints and grievances. 1870 1871 7.5 Documented information 1872 1873 Guidance 1874 1875 The intent of this clause is to put the documented information into two categories; information that 1876 needs to be maintained and information that needs to be retained. The wording “maintain documented 1877 information” means the information contained in documented procedures, manuals, forms, checklists 1878 etc. The other examples are QMS scope statement, quality policy and quality objectives statements. 1879 These are popularly called ‘documents’. 1880 1881 The wording “retain documented information” means ensuring that information that is used to provide 1882 evidence about whether a requirement has been fulfilled needs to be kept/ retained. These are popularly 1883 called ‘records/ evidences. 61

Working document QAS/19.783 Page 62

1884 1885 In general, ISO Standard 9001:2015 is not prescriptive in terms of the extent of documented information 1886 needed. This will vary from organization to organization depending on the size and complexity of their 1887 operations and processes; customers, statutory and regulatory requirements; and the competence of the 1888 persons involved.

1889 The following is the typical structure of documented information for QMS: 1890 1891 • Quality manual – a high level document providing intentions and commitments of the NRA about 1892 each requirement of ISO Standard 9001:2015 and providing references to lower level documents 1893 such as procedures etc. The manual could also include statement of scope of QMS, quality policy 1894 and quality objectives. 1895 • System procedures – such as procedures for risk determination and risk control, maintenance of 1896 infrastructure, maintenance and calibration of monitoring and measuring resources, creation and 1897 control of documented information, internal audit, management review, customer complaints and 1898 feedback, corrective action etc. 1899 • Standard operating procedures (SOPs) – for operational processes for each of the regulatory 1900 functions 1901 • Forms/formats/templates – as needed in the above procedures 1902 • Records – as evidence of demonstrating conformance to the prescribed requirements 1903 1904 While creating and updating documented information (DI) for QMS, an appropriate identification and 1905 format is used, and that DI is duly reviewed and approved. 1906 1907 The Identification and description of DI may be assured by the title, date, author, or reference number 1908 (or a combination of these), the format for the DI can be hard copy, electronic or both. It could also be 1909 in more than one language, based on the culture of the organization. 1910 1911 The method for the review and approval of DI should be decided, e.g. having an identified person with 1912 the authority to review and approve the DI or having one or more reviewers and one person who takes 1913 the responsibility for approval. 1914 1915 Documented information should be available in a suitable format and be adequately protected. The DI 1916 should also be in a form that is suitable for the intended use, for example, a written technical service 1917 agreement for an external service provider, or process parameter information in electronic format that 1918 can be used/downloaded at the process interface internally. Working document QAS/19.783 Page 63

1919 1920 Controls on DI include its availability, distribution and protection (for example from loss of data), its 1921 confidentiality, improper use and unintended changes. This can be done in many ways, including in 1922 electronic systems with ‘read-only’ access and specified permissions to access, password protection or 1923 identification (ID) entry. Information security issues and data backup should also be taken into 1924 consideration. 1925 1926 The control of documented information also must address distribution, access, retrieval and use, storage 1927 and preservation, control of changes, retention and disposition of DI. 1928 1929 Documented information can change and develop as an organization develops its QMS. There is also 1930 a need to consider how historical documented information is maintained, stored and retrieved as 1931 necessary for subsequent use. 1932 1933 The retention time for documented information could be a statutory or regulatory requirement, a 1934 contractual requirement, or can be determined by the NRA. 1935 1936 Documented information of external origin such as customer’s given DI (application files, SLPs), 1937 government orders, national/regional/international standards, calibration reports given by outside 1938 laboratories etc., as necessary for QMS should be identified appropriately and controlled in line with 1939 other DI. 1940 1941 When documented information is retained as evidence of conformity (records), it should be protected 1942 from unintended alterations, e.g. in case of soft copies, only giving controlled access (‘read only’) to 1943 such information. 1944 1945 Practical help box 19. Guidance for the interpretation of Clause 7.5 1946 1947 Examples of documented information that needs to be maintained by NRAs 1948 1949 Organizational chart, job descriptions, list of personnel and their roles and responsibilities, operational processes 1950 flow charts, mission and vision policy statements, strategic plans, QMS scope statement, quality manual and 1951 quality objectives, standard operating procedures, instructions, forms among others. 1952 1953 1954 Examples of documented information that needs to be retained by NRAs 1955 63

Working document QAS/19.783 Page 64

1956 Marketing authorization files, summary lot protocols, certificates of compliance/ non-compliance, lot release 1957 certificates, records of test results, reports, annual product review reports, testing methods and related validation 1958 reports, reports of adverse events following immunization and case investigation reports, complaints and related 1959 reports, personnel qualification records, personnel training records, personnel health records, internal/external 1960 audit plans and related reports, management review meetings’ agenda and minutes, validation, qualification or 1961 calibration records or reports among others. 1962 1963 More information on good data and record management can be found in the recently published WHO guideline. 1964 (28) 1965 1966 Clause 8. Operations 1967 1968 8.1 Operational planning and control 1969 1970 Guidance 1971 1972 The intent of this clause is to ensure that NRAs plan, implement and control the operational processes 1973 (MA, VL. MC and LR processes) that are necessary to meet the requirements of service provision, 1974 including any externally provided (outsourced) processes. 1975 1976 The following are the components of the plan: 1977 1978 • Determining requirements for products and services - consider customer, statutory and regulatory 1979 requirements, organizational requirements including requirements relating to relevant interested 1980 parties (stakeholders). 1981 • Establishing criteria (methods/procedures/KPIs) for the control of processes and acceptance of 1982 products and services consider; a) risks and opportunities; b) quality objectives; c) requirements for 1983 products and services. 1984 • Determining what resources are needed and if the current resources suffice. 1985 • Planned and potential unintended changes, and how these changes can affect the operations. 1986 • Determining documented information that needs to be maintained and that which needs to be 1987 retained 1988 1989 The output of the above planning should be used as inputs to operations. It should be kept in suitable 1990 format and media for those who need to use it. Practical help boxes 5 to 8 give examples of the details 1991 of the processes involved in MA, LR, VL and MC, and possible KPIs, and required resources. Working document QAS/19.783 Page 65

1992 1993 8.2 Requirements for products and services 1994 1995 Guidance 1996 1997 The first part of the clause refers to communication with customers and focuses on five communication 1998 areas: 1999 2000 a) detailed communication of the products and services offered so that the customer understands the 2001 requirements, to be provided to the customer through the website, pre-submission meetings, 2002 scientific advice, telephone or other, 2003 b) clear communication on how the customer can contact the NRA in case of questions or other 2004 services, or how the NRA would contact the customer in case of questions or other, 2005 c) establishing channels to gain information from customers such as concerns, complaints, positive 2006 and negative feedback, for example a web-based platform, phone calls, surveys, etc., 2007 d) inform customers how the customer property (documents, samples, dossiers etc.) is handled, where 2008 appropriate, and 2009 e) ensure that the NRA is proactive in communicating with the customer about possible contingency 2010 actions that can be taken, if the need occurs, such as natural disasters, epidemics, shortfall of staff 2011 or others. 2012 2013 Proactive communication enables the customer to understand what the NRA can or intends to provide 2014 and enables the NRA to understand or confirm the needs and expectations of the customer and bring 2015 greater transparency and public accountability. 2016 2017 The second part of the clause refers to determining the requirements of products and services, which in 2018 the case of NRAs are mostly statutory and regulatory requirements such as the Food and Drugs Act, the 2019 pharmacopoeia, monographs, etc.; but also, timelines for service delivery, fees charged for service, 2020 hours for customers service, acceptable waiting/or response time, etc. This information should be 2021 transparently communicated to customers and the public in general, usually through the NRA website. 2022 2023 The NRA should review the commitments it makes to a customer and ensure it can meet them. The 2024 review allows to reduce the risk of issues arising during operations. 2025 2026 The NRA should ensure that request for service received from customer is complete and is in conformity 2027 with service requirements. When there is a difference between the requirements for products or services

Working document QAS/19.783 Page 66

2028 as requested by customer and the one prescribed by the NRA, the same should be communicated to the 2029 customer and resolved before processing the request. Any verbal request/change in the requirements, 2030 either by the NRA or by the customer, should be confirmed before service is provided. 2031 2032 When the requirements for products and services are changed due to any reason, the NRA should take 2033 measures to inform all relevant interested parties. 2034 2035 The NRA should retain evidence of the results of the revisions to the requirements of products and 2036 services and any new requirements for the products and services that are provided. 2037 2038 Practical help box20. Guidance for the interpretation of Clause 8.2 2039 2040 Example of communication with customers 2041 2042 a) Customer Communication 2043 2044 Communication with customers aims at collecting information about their needs and expectations, as well as the 2045 reception of doubts, suggestions and complaints about the current work process and the engagement of 2046 stakeholders. 2047 2048 Phases: 2049 2050 - meetings with internal customers - managers and their teams - to identify problems and propose solutions; 2051 - meetings with regulated sector and other stakeholders to identify difficulties with the products and services 2052 offered by the regulatory authority and seek suggestions for improvements; 2053 - meeting with managers, to present the methodology and the schedule of actions. 2054 2055 Products: 2056 2057 - user's journey map: tool to identify all the points of contact of a user with a product or service and understand 2058 their needs, feelings, desires and pains related to the product or service, to promote the necessary improvements; 2059 - communication plan: a tool that establishes strategies for communicating with customers and other stakeholders 2060 involved during all stages of the process improvement initiative and raising awareness of the new way of working. 2061 2062 2063 2064 b) Determination and critical analysis of requirements 2065 Working document QAS/19.783 Page 67

2066 To determine the requirements, it is necessary to understand the challenges, make the diagnosis and immerse in 2067 the problem, through five stages. 2068 2069 Phases: 2070 2071 - definition of the scope of processes; 2072 - collection of quantitative and qualitative data; 2073 - mapping the current situation of the processes; 2074 - definition of performance gains; 2075 - definition of the team committed to the initiative. 2076 2077 Products: 2078 2079 - planning the initiative; 2080 - quantitative and qualitative analysis of processes; 2081 - flowcharts and checklists AS IS (as it is). 2082 2083 c) Changes on requirements 2084 2085 When changes of requirements occur, regardless of the reason, process documentation is reviewed, changes are 2086 recorded as well as communicated to all those involved in the chain. 2087 2088 Phases: 2089 2090 - meetings with customers to understand the dynamics of the necessary changes; 2091 - revision of the documentation (scope, schedule, actors, communication plan); 2092 - communication to all those involved. 2093 2094 Product: 2095 2096 - new planning of the process; transformation initiatives (scope, schedule, actors, communication plan). 2097 2098 8.3 Design and development of products and services 2099 2100 Guidance 2101 2102 The intent of this clause is to ensure that NRAs establish, implement and maintain a design and 2103 development process in order to ensure that new products and services meet requirements. The design 2104 and development process define the characteristics of the products and services. 67

Working document QAS/19.783 Page 68

2105 2106 The design and development of products and services consists of a set of processes that use ideas or 2107 requirements for a product or service. These ideas or requirements can come from customers, end- 2108 users, regulations, the organization or other interested parties including WHO. The ideas or 2109 requirements are processed to develop more detailed requirements that finally define the characteristics 2110 of the product or service. An example of an instance where an NRA may need to go through design 2111 and development process is in case it decides to perform a regulatory function that was not in place 2112 until then, or a new process within a function. For example, NRA from country X does not inspect 2113 clinical sites and is now able to introduce this new process within their activities to regulate clinical 2114 trials. To introduce this process, they will follow international guidance, however specificities of the 2115 process to be followed will be designed in-house. If an organization only uses ideas or requirements 2116 provided by regulation, customers or end-users, without adding more detail, it does not have design and 2117 development activities. In such cases, an explanation as to why the NRA is not applying clause 8.3 in 2118 its QMS can be included in the QMS scope statement (see clause 4.3). 2119 2120 The design and development of products and services requires several phases or steps 2121 2122 • Design and development planning: to determine the necessary design and development activities 2123 and tasks. This plan should include, required process stages; design inputs, design review, design 2124 verification and design validation, resource needs; as well as a clear definition of roles and 2125 responsibilities. 2126 • Design and development inputs: determines the inputs for design and development projects. These 2127 inputs need to be unambiguous, complete, and consistent with the requirements that define the 2128 characteristics of the product or service. It is important to consider the functional and performance 2129 requirements, statutory and regulatory requirements as well as additional standards or codes of 2130 practice. 2131 • Design and development controls: to ensure that once the inputs have been determined, the design 2132 and development activities and controls are implemented in accordance with the planning, to ensure 2133 process is effective. 2134 • Design and development outputs: to ensure that design and development outputs (service provision, 2135 standard operating procedure or service provision manual) give the necessary information for all 2136 the processes needed to provide intended products and services (including information to be 2137 provided by service recipients, service provision process, and post-delivery activities, if any). 2138 • Design and development changes: to determine, review and control changes made during or after 2139 the design and development process. Changes can arise during the design and development process 2140 (because of design review, verification or validation activity), after the release and approval of the Working document QAS/19.783 Page 69

2141 design and development outputs and during implementation of the same, as a result of monitoring 2142 customer satisfaction and interested parties’ feedback or as result of changes, or new, statutory and 2143 regulatory requirements. 2144 2145 Once these development phases are completed, a review by person(s) who are not involved in design 2146 activity) is required to ensure that design and development planning stages and the output of each stage 2147 are in place. Verification (comparing the new design with a similar proven design) and validation 2148 (testing under intended user conditions) activities are essential for controlling the design and 2149 development process and need to be implemented effectively. 2150 2151 NO EXAMPLE AVAILABLE AS YET 2152 2153 8.4 Control of externally provided processes, products and services 2154 2155 Guidance 2156 2157 The intent of this clause is to control processes, products and services that are provided by an external 2158 provider. External providers could include Government’s central procurement agency, suppliers of 2159 products and services, experts and consultants or someone to whom the NRA decides to outsource a 2160 process. 2161 2162 The NRA is responsible for ensuring that externally provided processes, products and services conform 2163 to requirements (e.g. through incoming goods inspection, or surveillance of an outsourced service 2164 provider). 2165 2166 The NRA should clearly identify its requirements (specifications) for the product and service to be 2167 purchased to ensure that externally provided processes, services or products do not have a negative 2168 effect on its operations or on customer satisfaction. 2169 2170 The NRA should ensure, its requirements are complete, clear and address any potential issues. It should 2171 clearly communicate the requirements and controls to be applied to the external provider and both 2172 parties should agree as to what is required. This understanding of requirements is usually reflected in 2173 a technical service agreement and/or through a purchase order/contract. 2174 2175 The NRA needs to determine and apply criteria for the evaluation, selection, monitoring of performance, 2176 and re-evaluation of external providers. The type and extent of control to be exercised is based on how

Working document QAS/19.783 Page 70

2177 much effect the externally provided process, product or service can have on the conformity to 2178 requirements of the NRA’s products or services. The NRA should determine which specific controls 2179 are to be implemented to an external provider. Control activities that may be considered include 2180 inspections, certificates of analysis or testing, second party audits, evaluation of statistical data and 2181 performance indicators. 2182 2183 The NRA should maintain up-to-date information related to its external providers, evaluated on their 2184 ability to comply with purchasing requirements both in terms of conformity of the product and service 2185 provided and delivery performance. A list of providers could serve as a basis for external provider 2186 selection and management of relation with current external providers. 2187 2188 Practical help box 21. Guidance for the interpretation of clause 8.4 2189 2190 For the MA process, the NRA has as external provider with the figure of "Third Authorized Party” (TAP), which 2191 are persons authorized by the NRA to perform a preliminary review of marketing authorization dossiers and to 2192 issue, if applicable, a Favorable Technical Reports (FTR) for registration, modification or extension of medicines 2193 MA based on compliance with the requirements established by the Ministry of Health in the corresponding 2194 regulations for the completion of procedures. 2195 2196 The process of selection of a TAP by the NRA is done through an announcement published by the Ministry of 2197 Health in which the requirements that must be fulfilled by interested candidates are established. The authorization 2198 to act as a TAP has a validity of two years. TAPs can be individuals or companies. 2199 2200 The NRA recognizes the technical competence of the TAP ensuring that the process of evaluation and release of 2201 an FTR is managed effectively in accordance with the provisions of the current regulations through the 2202 establishment of policies, responsibilities and activities to be fulfilled by the TAP. The TAP is also subject to 2203 controls for evaluation and monitoring. 2204 2205 The controls applied by the NRA fall under the following categories: 2206 2207 • Technical supervision 2208 • Supervision of the records reviewed by the TAP 2209 2210 Verify the technical qualification, training and experience of the personnel involved in the review. 2211 Verify the documented training system that ensures competencies in the technical aspects. 2212 Verify that there are adequate tools, references and bibliography that allow technical activities. 2213 Review of the technical procedures and the homologation criteria of the reviewers (TAP). 2214 Working document QAS/19.783 Page 71

2215 Follow up on the corrective or preventive actions derived from the non-conformities detected in the supervision 2216 visits. 2217 2218 TAP verification actions allow to detect FTR with inconsistencies or non-compliance with the regulatory 2219 provisions. 2220 2221 The qualification of the TAP has been assigned a level of confidentiality from I to IV, being I "highly reliable" 2222 and IV "not reliable", which will provide the level of review (reduce, regular or strict) by the NRA of the 2223 procedures entered with FTR of the TAP. 2224 2225 2226 8.5 Production and service provision 2227 2228 Guidance 2229

2230 The intent of this clause is for the NRA to establish controls to ensure that the intended results are 2231 achieved (products and services), by reducing the potential for errors/nonconforming outputs. The 2232 clause also focuses on the preservation of data and physical property, traceability, control of changes 2233 and the responsibility for post-delivery activities.

2234 Items a) to h) of clause 8.5.1 provide suggested controls to be applied to service provision to ensure that 2235 the criteria determined in clause 8.1 are met. These include documented information about the 2236 procedures used and the results of monitoring activities including measurements if applicable; checks 2237 to ensure that the necessary infrastructure is in place as well as availability of competent personnel, that 2238 processes are validated, actions taken to prevent human error including appropriate training of personnel 2239 provided, and controls are in place for release, delivery and post-delivery activities including 2240 confirmation that authorized personnel for these activities is in place. 2241 2242 To properly monitor the status of product and service provision throughout the service provision 2243 process, products and services should be identified (reference number of service request, batch number 2244 for drugs, code no of product, others) and traceability ensured. Product and service identification 2245 prevents unintended mix up of the service requests and allows tracing of the events for processing 2246 service requests, updating customer about status of service delivery, investigation of customer 2247 complaints, etc. 2248 2249 The NRA, due to its mandated role and responsibilities, has access to property that does not belong to 2250 the NRA, but which is under the NRA’s control; this property can be tangible or intangible. Examples 71

Working document QAS/19.783 Page 72

2251 include marketing authorization dossiers, SLPs, vaccine samples for testing, intellectual property or 2252 personal data, others. The NRA should make provisions to ensure the protection of such property. 2253 2254 The actions taken to protect it, will depend on the type of property. The owner of the property should 2255 be clearly identified and made known within the NRA. Protection of data could be ensured through a 2256 specific password protected electronic location or file with restricted access to store customer’s 2257 intellectual data, patent information, performance and sales figures, etc. Data integrity can be ensured 2258 by regular back-ups and virus protection, storage of magnetic media (e.g. video tapes, audio tapes and 2259 computer disks) in a non-magnetic environment, others. 2260 2261 When the NRA takes control of the property its verification is important (e.g. state or physical condition, 2262 accuracy of personal data, completeness of the dossier). 2263 2264 The customer or external provider should be accurately informed if property is lost, damaged or 2265 otherwise found to be unsuitable or incapable of use. This will require to be documented. 2266 2267 Outputs from different processes and products and services, should be preserved from damage or loss 2268 at all stages during service provision. The NRA should determine which are the outputs, products and 2269 services that can deteriorate or degrade and implement appropriate preservation methods. 2270 2271 In case that changes occur during service provision, these must be reviewed and controlled. 2272 2273 There may be numerous reasons why changes occur; for example, a change initiated by an external 2274 provider (e.g. delays in getting expert’s opinion or test reports from external labs), due to internal issues 2275 (e.g. critical equipment failure, internet connectivity issues) or to an external issue (e.g. new or modified 2276 customer or statutory and regulatory requirements). 2277 2278 Changes must be controlled, and the relevant documented information retained. Examples include: a) 2279 minutes of the review activities; b) description of the change; c) details of the person(s) or a customer 2280 authorizing the change. 2281 2282 The responsibility of the NRA does not end with product and service delivery, the clause also 2283 emphasizes the need to determine the post-delivery activities in which the NRA is engaged. For this, it 2284 should consider if the post-delivery activity is part of a contractual requirement or is a regulatory 2285 requirement such as marketing authorization renewals, approval of changes (variations), market 2286 surveillance or to address a potential complaint from customers (customers dissatisfaction). Working document QAS/19.783 Page 73

2287 2288 Other examples of post-delivery activities include: 2289 2290 • Engagement with customers to determine if the products or services were to their satisfaction 2291 through customer feedback, resolving customer complaints, customer compliments, media reports 2292 etc; and 2293 • Customer access to on-line information required after delivery. 2294 2295 Practical help box 22. Guidance for the interpretation of clause 8.5 relating to preservation of 2296 physical property 2297 2298 Example of infrastructure required for the preservation of vaccines until their expiry 2299 2300 An NRA may receive vaccine samples for visual inspection or testing during the lot release process, and in some 2301 cases also during the marketing authorization evaluation process, although at this stage this is not required. 2302 2303 In case samples are requested, these need to be properly stored and kept until the expiry date. The NRA needs 2304 adequate infrastructure to keep vaccines at 2-8°C during the whole shelf life. Adequately validated and regularly 2305 monitored refrigerators are needed. Back up measures are required such as an alarm system (ideally centralized), 2306 which ensures that in case of electricity failure or break down of the equipment, a responsible officer is 2307 immediately informed. A back up refrigerator or electricity generator, depending on the source of the failure, 2308 must be available for such situations 2309 2310 8.6 Release of products and services 2311 2312 Guidance 2313 2314 The intent of this clause is to ensure that products and services are checked for conformity for all 2315 applicable requirements, at appropriate stages of the service provision process, before they are released 2316 for delivery to the customer, for example, issue of market authorization certificate/letter. 2317 2318 Approval by a relevant authority may be required when all checks for conformity have not been 2319 satisfactorily completed - in some cases, this could be the customer. 2320 2321 The release of products and services should be suitably documented (DI). The DI should include 2322 evidence that the product or service conforms to all acceptance criteria and be traceable to the person 2323 authorized to release products and services.

Working document QAS/19.783 Page 74

2324 2325 The person(s) who authorize(s) final release of the product or service should be suitably defined by, for 2326 example, their job description or authority level. 2327 2328 Practical help box 23. Guidance for the interpretation of clause 8.6 2329 2330 Products of XYN Drug Authority include reports, certificates, licences, permits and authorization letters. These 2331 are checked by the respective supervisors and signed by the Executive Director or other senior officer(s) 2332 authorised by the Governing Board to do so as per section XX of the National FDA Act, before delivery to the 2333 applicant or entity that requests for them. The list of authorised persons (Authorised Persons to Release the XYN 2334 Drug Authority products to applicants) is updated from time to time and communicated to all staff via the posted 2335 on the XYN Drug Authority Intranet. 2336 2337 The release of reports, certificates, licences, permits and delivery to the applicants does not proceed until the 2338 requirements have been satisfactorily met (e.g. certificates for GMP compliance are not issued until the evidence 2339 of corrective and preventive actions taken by the manufacturer are submitted and evaluated by the XYN Drug 2340 Authority and found to be satisfactory). 2341 2342 8.7 Control of non-conforming outputs 2343 2344 Guidance 2345 2346 The intent of this clause is to prevent non-conforming outputs from progressing to the next stage or to 2347 the customer. 2348 2349 There are different ways to control non-conforming outputs: 2350 2351 • Correcting (rework, repair) the nonconformity to ensure it does conform 2352 • Removing the nonconformity from the process entirely (rejecting or scrapping the output/product) 2353 • Obtaining authorization for release under concession 2354 2355 The extent of control that an organization needs to take depends on the nature of the nonconformity and 2356 its potential effects. 2357 2358 If the nonconformity is discovered after it has progressed to the next stage, or been delivered to the 2359 customer, the NRA should take appropriate actions to prevent unintended use or undesired 2360 consequences, and take measures such as issuing a re-call, suspension, re-processing, eliminating or Working document QAS/19.783 Page 75

2361 reducing the NC to acceptable level (concession). In case of concession, authorization should be given 2362 by the appropriate person(s) or, where relevant, the customer. 2363 2364 The NRA should ensure that the documented information retained includes details of the 2365 nonconformity, the actions taken to correct, mitigate or communicate it, any concessions obtained (e.g. 2366 agreement with the customer that the product or service could be used despite the nonconformity) and 2367 who authorized the actions taken. 2368 2369 Retaining documented information on the above ensures that processes are improved and optimized; 2370 corrected work instructions, processes and procedures are detailed for future use. 2371 2372 Practical help box 24 Guidance for the interpretation of clause 8.7 Control of non-performing 2373 outputs 2374 2375 When a nonconforming output is detected before or after delivery to the customer, it is registered by the process 2376 owner and an investigation form e.g. complaint investigation in-process form (in case of market/customer 2377 complaints); or the OOS investigation form (in case of the QC Laboratory); or the corrective action request (CAR) 2378 form (for others, e.g. arising out of quality audits); is raised for investigation to be initiated in order to find out the 2379 root cause or assignable cause. Correction and corrective action are then taken by the respective process owner. 2380 2381 A non-conforming output may include any of the following items that is found to have an error, mistake or defect 2382 before or after delivery to the applicant (customer): 2383 2384 a) Marketing authorization certificate, GMP certificate, import/export permit, manufacturing licence, licence to 2385 sell drugs, or a laboratory test report or certificate of analysis, 2386 b) Published adverse event report, 2387 c) Clinical/field trial assessment monitoring report, 2388 d) Promotional material vetting report. 2389 2390 In all cases, correction and corrective action are taken and the certificate, permit, licence or report that has an 2391 error, mistake or defect is either cancelled or withdrawn (without prejudice), or both and replaced with a corrected 2392 one. However, the validity period and the applicable conditions remain the same. 2393 2394 2395 2396 Clause 9. Performance evaluation 2397 2398 Practical help box 25 Guidance for the interpretation of clause 9 75

Working document QAS/19.783 Page 76

2399 2400 A monitoring and evaluation framework that tracks process activities, targets, key performance 2401 indicators, and outputs is used to monitor progress of processes. Performance reports (quarterly, semi- 2402 annual, and annual) are made and their information analyzed and used as input in management reviews. 2403 2404 9.1. Monitoring, measurement, analysis and evaluation 2405 2406 Guidance 2407 2408 The intent of this clause is to ensure that NRAs conduct monitoring, measurement, analysis and 2409 evaluation to determine if the intended results are being achieved. The NRAs should determine what 2410 needs to be monitored and measured (according to the characteristics of processes, products, services 2411 and risks involved) and the methods to be used to analyse and evaluate the performance and 2412 effectiveness of QMS. The NRAs should also determine how and when the monitoring, measurement, 2413 analysis and evaluation will be carried out, and the resources that will be needed for the same. 2414 2415 The NRAs should decide on what documented information relating to monitoring, measurement, 2416 analysis and evaluation will need to be retained as evidence of the results. 2417 2418 One way of monitoring performance is through feedback from customers. It allows to evaluate the 2419 degree of customers’ satisfaction and to determine opportunities for improvement. The NRA may 2420 choose to seek feedback from a selected population of customers or from every customer at the end of 2421 a transaction. Means to obtain feedback are offered through the social and published media such as 2422 web sites and message boards, opinion surveys and compliments or complaints. 2423 2424 The NRAs should be able to determine the degree of customer satisfaction after the results are analysed 2425 and evaluated; and act based on this information. This information should be an input to management 2426 review (9.3) and can be used for determining if actions are necessary to improve customer satisfaction. 2427 2428 The results of monitoring and measurement (data and information) must be analysed to determine if 2429 processes, products and services meet requirements and whether there are any needed actions and 2430 opportunities for improvement. The purpose of analysis and evaluation of data from monitoring and 2431 measurement activities include assessing the level of customer satisfaction, assessing whether plans are 2432 being met, assessing performance of external providers, how successful the NRA has been in addressing 2433 risks and opportunities, status of performance and effectiveness of QMS and need for improvements. 2434 Working document QAS/19.783 Page 77

2435 Data sources that could be used for analysis and evaluation include the monitoring of customer 2436 perception (9.1), feedback from media and the public in general (9.1), monitoring quality objectives 2437 (6.2), timeliness of service delivery (8.5), data related to corrective or preventive actions taken (8.7), 2438 non-compliances detected during internal audits (9.2), others. 2439 2440 The output from analysis and evaluation is generally in the form of documented information such as 2441 trend analyses or reports and becomes an input to management review (see clause 9.3). 2442 2443 EXAMPLE NOT AVAILABLE AS YET 2444 2445 9.2 Internal audit 2446 2447 Guidance 2448 2449 The intent of the clause is, for the management of NRA, to obtain information through internal audits 2450 about continued conformance and effectiveness of QMS. 2451 2452 The internal audits are conducted at planned intervals to verify if the NRA’s activities and processes 2453 continue to meet the prescribed system as defined in NRA’s documented information (e.g. Quality 2454 manual, quality policy, quality objectives, procedures, instructions, risk control plans and other plans), 2455 if the QMS continues to meet the requirements of ISO Standard 9001:2015 standard and if QMS is 2456 effectively implemented and maintained. 2457 2458 Internal audits should be planned, results reported and timely actions on the audit findings taken. 2459 2460 Items a) to f) under clause 9.2.2 of the ISO Standard 9001:2015 provide details on how audits must be 2461 planned. The planning process includes developing the audit programme (calendar of audits over a 2462 period of time - for example, one year - which also means setting the frequency of audit of each area 2463 and related processes over a one-year time), defining the criteria to be used (standard, requirements), 2464 the scope of the audit and the methodology to be used during the audit (interviews, documented 2465 information review, results, trends, etc.). Auditors should be selected, usually from sectors within the 2466 regulatory agency not affected by that particular audit (cross functional audit) who have been trained in 2467 QMS auditing. Sometimes external auditors may be used if independence within the NRA is difficult 2468 to ensure. Corrective actions must be taken promptly to address the findings of the audit. Documented 2469 information of the programme and audit results should be maintained. 2470

Working document QAS/19.783 Page 78

2471 Responsibility for planning of audits lies with the person who is directing and managing the internal 2472 audit process (audit manager) such as QMS coordinator or some other person as authorized by 2473 management of NRA. 2474 2475 While developing the programme, the audit manager should apply risk-based thinking and consider 2476 how often the process is performed as well as its maturity and complexity, whether any changes in the 2477 process were introduced, or other changes affecting the NRA, the process performance, results from 2478 previous audits and history of complaints. 2479 2480 After each internal audit is completed, the results should be reported to relevant management. Based 2481 on these results, appropriate correction and/or corrective actions can be necessary. Typically, 2482 organizations establish a time to respond and correct nonconformities to ensure they are fixed in a timely 2483 manner. 2484 2485 During the audit, auditors might bring up a potential weakness in the QMS, which may represent 2486 opportunity for improvement. Such information can help management to decide if it is appropriate to 2487 initiate action for improvement. 2488 2489 It is important that management of NRA fosters an open-minded culture where quality audits are 2490 perceived as a means to improve performance and not to assign blame for any non-conformities found. 2491 2492 Practical help box 25. Guidance for the interpretation of clause 9.2 2493 2494 Planning internal audits by ANRA, the NRA from Country A 2495 2496 Scope of the auditing programme: legal services, internal audit, procurement management, communication and 2497 public education, finance and accounts, information and communication technologies and statistics, human 2498 resources and administration, planning monitoring and evaluation, food inspection and enforcement, food 2499 registration food risk assessment, clinical trials control and pharmacovigilance, medicines and complementary 2500 products inspection and enforcement, medicines registration, medical devices, diagnostics and cosmetics control 2501 Timelines: November and December 2018 2502 2503 Processes being audited clearly defined, methodology applied is interviews and documented information review. 2504 Audit duration: One day for each section/unit. 2505 Auditors: Two internal auditors (cross functional) are selected based on expertise. 2506 2507 ANRA is a decentralized authority with five zonal offices. The auditing programme is the following: Working document QAS/19.783 Page 79

2508 2509 Scope: The five zonal offices 2510 Processes: QMS, premises licensing, inspection and enforcement, import and export control, registry, customer 2511 complaints and procurement and finance 2512 Audit duration: One day for each zonal office 2513 Timelines: October 2018 2514 Auditors: Two internal auditors (cross functional) are selected based on expertise. 2515 2516 Summary of the procedure for conducting internal audits by ANRA 2517 2518 The purpose is to provide details on how internal audits should be conducted to check and evaluate the efficacy 2519 and effectiveness of the QMS for continual improvement. 2520 2521 The quality manager (QM) appoints the auditors and prepares the audit programme for HQ and Zone offices. The 2522 QM informs auditors and auditees in writing about the programme. The audit team prepares the audit checklist 2523 based on previous audit findings and QMS documentation. During the audit it is the responsibility of the team 2524 leader to open the audit and to verify attendance. The audit team should collect and verify information for specific 2525 processes, procedures, functions, sites, areas and activities. It records the findings and prepares the audit report. 2526 The audit team leader closes the audit. Non-conformities are categorized in minor, major and opportunity for 2527 improvement. The audit report is reviewed and agreed upon by auditors and auditee. This is then forwarded to 2528 the QM. The auditee should prepare a CAPA and the QM will take measures for timely follow up. If corrective 2529 or preventive actions are properly implemented, the audit is closed, otherwise another follow up form is opened. 2530 2531 The audit reports, corrective and preventive actions reports, attendance register, checklists and audit programme 2532 should be kept at QM office and maintained for a period of five years and then destroyed by tearing, shredding, 2533 burning or other appropriate means. 2534 2535 9.3 Management review 2536 2537 Guidance 2538 2539 The intent of this clause is to ensure that NRA’s top management conducts periodic review of 2540 performance of its QMS. The purpose of such review is to determine if NRA’s QMS continues to be 2541 suitable (fitting the purpose), adequate (sufficient), effective in achieving the intended results and 2542 continues to be aligned with the strategic directions of the NRA. 2543 2544 Management review should be conducted at planned intervals; this could be daily, weekly, monthly, 2545 quarterly, semi-annually or annually, depending on the situations facing the NRA. If various levels of

Working document QAS/19.783 Page 80

2546 the management carry out management review activities, the results should be made available to its top 2547 management for final decision/approval. 2548 2549 Management reviews could be a standalone activity or in a combination of related activities (e.g. 2550 strategic planning, business planning, annual meeting, operations meetings, other management 2551 reviews). Working document QAS/19.783 Page 81

2552 Table 4. Management Review Meetings (MRM) inputs and outputs 2553 INPUTS OUTPUTS Review implementation of actions to be taken • Decisions and actions related to opportunities for improvement (10.1) following previous review meetings • Decisions and actions related to changes Updated analysis of internal and external context of required in the QMS (6.3) NRA (4.1) • Need for additional resources to implement improvement initiatives and changes Performance and effectiveness of QMS (9.1) suggested in QMS and for other areas where through: resources (including human resource) are not adequate (7.1). Customers satisfaction (9.1), feedback from • Progress towards quality objectives (6.2) interested parties (4.2), implementation of quality • Management review meeting minutes to be retained as documented information and objectives (6.1), monitoring processes through KPIs communicated in an adequate way to all (8.5), conformity of products and services (8.6), concerned. • Outputs from MRM will be inputs for the status of non-conformities including response to following meeting complaints and corrective or preventive actions (10.2), monitoring and measurement results (9.1), audits outcome (9.2) and performance of external providers (8.4) Adequacy of resources (7.1) Data about the effectiveness of the actions to confront risks and opportunities (see 6.1) Opportunities for improvement (10.1 and 10.3) 2554 2555 Practical help box 25. Guidance for the interpretation of clause 9.3 2556 2557 Country A NRA Summary Procedure for Management Review Meetings 2558 2559 The QM should call management review meetings at a minimum once per year but should attempt to conduct 2560 them quarterly. Attendees must include the Director General, Directors of Units, Management representative, 2561 Legal counselor and any other personnel as deemed necessary by the management to attend. The QM should 2562 record the minutes of the meeting. Analysis of data presented (according to table 4) should be performed to look 2563 for areas of improvement. Improvement items and follow up actions should be implemented as Action items. The 2564 agenda should include status of action of previous MRM and changes in internal and external issues that are 2565 relevant to the QMS 2566 2567 Some NRAs have two types of review meetings: 2568 2569 1. Technical review meetings (TRM) membership is QM focal points in different technical units. They take 2570 place frequently (usually monthly) and focus specifically on technical aspects such as KPIs, implementation 81

Working document QAS/19.783 Page 82

2571 of corrective or preventive actions, conformity of products and services, monitoring and measurement results 2572 including trends, audit outcomes and performance of external providers 2573 2574 2. Management review meetings membership is as for the (TRM) plus higher-level management as indicated 2575 for Country A above. These take place on quarterly, bi-annual or annual basis. Outputs from the TRM are 2576 inputs for MRM as well as other QMS related aspects not addressed in the TRM (see table 4) 2577 2578 2579 2580 2581 2582 2583 2584 2585 2586 2587 2588 2589 2590 2591 2592 2593 2594 2595 2596 2597 2598 2599 2600 2601 2602 2603 2604 2605 2606 Working document QAS/19.783 Page 83

2607 XYN Drug Authority Management Review Flow Chart Quality Management Department Top Management

Schedule Management Review Meeting & Issue Analyze Performance, Agenda Results & Trends

Management Review Inputs (Agenda) Suitability & 1) The status of actions from previous Adequacy management reviews; 2) Reports on process performance, conformity of services and the adequacy of resources (including effectiveness of processes; monitoring and evaluation results and trends;

changes in external and internal issues; Suitable & Not suitable or resources available e.g. human resource, Adequate Not adequate time, equipment and technology used, financial, etc.) with respect to the following key drug regulatory processes: a) Assessment and registration of medicines; b) Inspection and licensing/certification of • pharmacies, medicine shops • pharmaceutical manufacturers (Good Manufacturing practice); c) Control of pharmaceutical imports and exports; d) Pharmacovigilance; e) Clinical Trials; Agree & Implement No System Changes f) Vetting of drug promotional materials; System Changes g) Post-marketing surveillance; and h) Enforcement.

3) Report and trends on quality objectives (extent to which objectives related to customer satisfaction, e.g. service delivery objectives have been met) for the key drug regulatory processes listed in 2 above and all the support processes, e.g. Finance and administration, human resource, procurement, legal services, information technology, internal audit, quality management, and public relations; 4) Report and trends on client/customer complaints (market complaints, including appeals) Management Review Outputs 5) Information from the recent customer • Opportunities for improvement; satisfaction survey report; • Changes to the QMS; 6) Internal quality audit results (from Internal quality • Resource needs. audit reports, including second party audits); Documentation & Records 7) Report on nonconformities and corrective actions; 8) Performance of external providers; 9) Effectiveness of actions taken to address risks and opportunities for all key drug regulatory processes and all support processes; 10) Identifying opportunitiesDocumentation for improvement & Records for all processes.

Working document QAS/19.783 Page 84

2608 Clause 10. Improvement 2609 2610 10.1 General 2611 2612 Guidance 2613 2614 The intent of this requirement is to ensure that the NRA determines opportunities for improvement, 2615 plans and implements actions to achieve the intended results and to enhance customer satisfaction. 2616 2617 Improvements can help the NRA to keep meeting customer requirements and expectations by 2618 improving its products and services, correcting or preventing undesired effects, and improving the 2619 performance and effectiveness of QMS. 2620 2621 There are different methods to conduct improvement, such as correcting existing nonconformities and 2622 taking actions on their causes to prevent recurrence, or making small-step-ongoing improvement 2623 activities or through breakthrough projects leading to innovation, revision and improvement of existing 2624 processes or the implementation of new processes; 2625 2626 10.2 Non-conformity and Corrective Action 2627 2628 Guidance 2629 2630 The intent of this clause is to ensure that the NRA manages nonconformities, and implements corrective 2631 action, appropriately. 2632 2633 Non-conformity means ‘non-fulfilment of a requirement’ related to a product, service, process or QMS. 2634 These requirements may come from the customers, from relevant interested parties, from statutory and 2635 regulatory requirements, or they may be internal requirements defined by the NRA in its policies, 2636 manuals, procedures, quality objectives, etc. 2637 2638 A non-conformity could be identified from customer complaints or from non-conforming outputs, 2639 problems arising from relevant interested parties, audit results, the effects of unplanned changes, etc. 2640 2641 The immediate action needed is to control or correct any non-conformity. This can be achieved by 2642 containing the problem while the investigation continues. For example, making customers aware of a 2643 non-conformity and to provide information about the potential or actual effects on the product provided Working document QAS/19.783 Page 85

2644 or service delivered and also correcting the situation. 2645 2646 To make corrections and introduce corrective or preventive actions, the NRA should follow the 2647 following steps: 2648 2649 1. Review and analyse the non-conformity to determine its cause by using methods such as, 5-why 2650 method or cause-and-effect-analysis diagrams or simply by brainstorming with a cross functional 2651 team. Examples of typical root causes include lack of understanding of requirement, lack of 2652 resources, process not well-defined, etc. 2653 2654 2. Determine the extent of the actions that need to be taken to eliminate the cause determined at 1 2655 above. There might be instances where the cause of the non-conformity cannot be eliminated, 2656 therefore the NRA should consider taking actions to detect and minimize the effects of the non- 2657 conformity if it were to occur again. 2658 2659 3. Implement any needed actions as decided at 2 above. This may include making changes in 2660 process/procedure, providing resources, retraining persons, ensuring better adherence to defined 2661 process, etc. It should be ensured that corrective action taken in one area should not cause adverse 2662 effects in another area of the NRA. 2663 2664 4. Review the effectiveness of corrective or preventive actions taken by confirming (through 2665 evidence) that the actions have been implemented. This may be accomplished by observing the 2666 performance of processes or reviewing documented information or verifying during internal audits 2667 that the same non-conformity is not repeated. This review should be done after a reasonable time 2668 needed to implement corrective action has elapsed. 2669 2670 5. After the review of corrective or preventive actions, the NRA should consider whether there is a 2671 new risk or opportunity that was not determined during planning (see 6.1) and planning should be 2672 updated as necessary. 2673 2674 The NRA should retain documented information showing what correction or corrective or preventive 2675 actions were taken, including the nature of the non-conformity (e.g. nonconformity statement); 2676 examples include corrective action forms or databases and evidence demonstrating that actions have 2677 been taken. 2678 2679

Working document QAS/19.783 Page 86

2680 Practical help box 26. Guidance for the interpretation of clause 10.2 2681 2682 An example of dealing with nonconformity using a Corrective Action Request Form. 2683 XYN Drug Authority Corrective Action Request (CAR) Form

2684 (To be used to request for corrective action of a nonconformity in the NRA quality system) 2685 Nonconformity Root Cause Corrective The steps that have or will be Timeline (from 5-Why Root Action taken for the demonstration Cause Analysis of effectiveness of the actions Form attached) taken 1.1 There was no The parameters to Develop a format 1. Developed a format with Oct 2018 evidence of be monitored and for monitoring the key parameters as listed the tool to be used quality objective monitoring of the a) Products had not been clearly indicating quality objective in b) Annual and Quarterly the Post Marketing established. the parameters. targets Nov 2018 Surveillance 2. Create a database on the department, NRA server for contrary to the monitoring the quality requirements of the objectives and train the standard. NRA staff in using it.

2686 A root cause analysis must be attached. See example below: 2687 XYN Drug Authority Root Cause Analysis Form Directorate/ Department / Unit/ Area: Inspection & Licensing Dept. Representative: Dr. Brian Harvey Date: 7th Nov 2018 Category of Problem: Nonconformity Nonconforming output Market Complaint Other (please specify)

2688 (Check applicable box above by double clicking on it) 2689 Problem / Issue Why 1 Why 2 Why 3 Why 4 Why 5 1.1 There was no Why was there no Why was Why were the quality Why was the system evidence of evidence of monitoring monitoring of the objectives not yet of monitoring the monitoring of the of the quality quality objectives communicated to the quality objectives not quality objectives objectives? not yet started? relevant personnel at well established? in for the all levels? Inspection and Licensing (I&L) Because the Because the quality parameters to be processes Because the quality objectives had not Because the system of monitored and the objectives had just been been communicated monitoring them was tool to be used had developed and to the relevant not well established. not been monitoring them had personnel at all established. not started. levels. This is the root cause that must be taken to the CAR form above to determine the corrective action 2690 2691 Note: Although this technique is called “5 Whys,” you may need to ask the question fewer or more times than five before you 2692 find the root cause of the problem or non-conformity. 2693 Working document QAS/19.783 Page 87

2694 10.3 Continual improvement 2695 2696 Guidance 2697 2698 The intent of this clause is that NRA should continually improve the suitability, adequacy and 2699 effectiveness of its QMS. 2700 2701 Continual improvement can include actions to increase consistency of process outputs and products and 2702 services; improve process capability and reduce process variation. This is done to enhance the NRA’s 2703 performance and benefit its customers and interested parties. The results from analysis (9.1) and 2704 evaluation and management review (9.3) are used to decide whether continual improvement actions are 2705 needed and what they should be. 2706 2707 Examples of CI include reducing errors, rework, complaints, non-conformity, breakdown of equipment, 2708 delays of promised services etc. and improving customer satisfaction, improving employees’ 2709 involvement, etc. 2710 2711 There are several methodologies and tools that NRA can consider to initiate continual improvement 2712 activities, for example, Kaizen, benchmarking, use of self-assessment models. etc. 2713 2714 Practical help box 27. Guidance for the interpretation of clause 10.3. 2715 2716 As part of continual improvement, XYN Drug Authority uses trending of quarterly, semi-annual and 2717 annual performance of the key drug regulatory process and all support processes; and from results of 2718 management review, to determine areas of underperformance and to identify any opportunities for 2719 improvement.

2720

2721 5. QMS implementation methodology 2722 2723 For the successful implementation of QMS, full commitment of Head of NRA (top management) will be 2724 necessary with respect to provision of timely resources (human and others) for implementation of QMS 2725 and by demonstrating his/her leadership, commitment and customer focus (see guidance under clause 5.1 2726 of this guideline) through all stages of the implementation of QMS. 2727 2728 A systematic way of implementing the QMS will include the following steps:

Working document QAS/19.783 Page 88

Step Activity Refer guidance Responsibility under clause within the NRA A. Documenting QMS 1. Appoint a Core Team (CT) with members from various - Head of the NRA functions of NRA with one person as team leader, who subsequently could be designated as QMS Coordinator 2. Persons in CT should fully understand the QMS Full guideline CT requirements either through study of this guideline or undergo a formal training on the subject 3. Develop current Context statement (SWOT analysis) of 4.1 CT NRA or use one, if already available 4. Determine and document requirements (needs and 4.2 CT expectations) of interested parties/stakeholders (both external and internal) relevant to QMS 5. Determine and document the Scope of QMS (could be 4.3 CT and Head of whole NRA or specific functions) with NRA’s products and NRA services within the scope listed in it. If any of the requirements of ISO 9001 is not applicable, provide its justification within scope statement 6. Develop and document Quality Policy, keeping in view the 5.2 CT and Head of purpose (vision and mission), context and strategic direction NRA of NRA. Policy statement could be communicated through display within NRA office(s) or otherwise communicated to all, for its understanding and application. 7. Develop and document QMS related responsibilities and 5.3 CT and Head of authorities at different levels of NRA staff and NRA communicate to all concerned. 8. Use information from step 3 and 4 above, as input, to 6.1 CT determine risks and opportunities and develop risk control plan. 9. Develop and document measurable and time bound quality 6.2 CT and Head of objectives including plan for monitoring and achieving NRA them and communicate quality objectives to all concerned. 10. Carry out a gap analysis with respect to support processes 7.1, 7.2, 7.4 CT covering human resources, infrastructure (equipment, hardware, software, facilities etc.), process environment (heating, lighting etc), measuring equipment, organizational knowledge and communication and fill the gaps, if any. Develop new or harmonize existing Standard Operating Procedures (SOPs) for control of measuring equipment, organizational knowledge, training and communication 11. For internal support services provided by viz 7.1, 7.2, 7.4 CT administration, HR, ICT systems, maintenance, logistics, procurement etc, it is good to develop and practice Service Level Agreements (SLAs) covering service standards (time lines) and responsibilities of each party (internal service provider and service recipient) 12. Conduct gap analysis to assess the extent to which the 8.1, 8.2, 8.3, 8.4, CT existing NRA policies, procedures/manuals and practices 8.5, 8.6 and 8.7 relating to regulatory functions (MA, VL, MC, LR or others) are in line with service provision processes (8.1 to 8.7) of ISO 9001 and harmonize existing SOPs or develop Working document QAS/19.783 Page 89

additional processes and related SOPs and SLAs with customers. Also integrate risk control plan in the relevant SOPs. 4.4.1 Guidance under clause 4.4.1 will facilitate to harmonize SOPs with ISO 9001 requirements. 13. Develop quality system procedures (QSPs) for monitoring 9.1.2, 9.2, 9.3, CT of customer satisfaction, internal audit, management 10.1, 10.2 and review, complaints handling, correction and corrective 10.3 actions, improvement; and put them in practice. 14. Develop and document a Quality Manual (QM) stating as to 7.5, 6.3 QMS Coordinator how NRA intents to meet each requirement of ISO 9001 with scope and quality policy statement (at 5 and 6 above) included into it. All other documents viz SOPs, QSPs, SLAs, forms/formats/templates referred in QSPs/SOPs/SLAs may be added as annexes to QM or kept separately as standalone folders. All above documented information (DI) including records could be either in hard or soft version. A QSP for creation, updating and control of DI will also be needed and also a QSP on Planning for changes. B. Practicing QMS 15. It is good practice that documents as they get developed are - QMS Coordinator communicated to all concerned and put into implementation and all concerned mode. 16. Formal awareness sessions may be held by CT for people to 7.3 CT/QMS understand and apply the policies, objectives, SOPs, QSPs, Coordinator and SLAs etc and if necessary train people on how to use new all concerned QSPs/SOPs/SLAs 17. Monitoring of products, services and processes should 9.1.1 & 9.1.3 CT/QMS continue to happen against defined KPIs, risk control plan, Coordinator and and through monitoring of applicable quality objectives. all concerned The monitoring data should be analysed and evaluated. 18. After formal implementation of QMS, for at least a period 9.2, 10.2 & 9.3 QMS of 3 months, an internal audit, followed by and related QSPs Coordinator, all corrections/corrective actions on the audit findings and concerned and management review should be carried out Head of NRA 19. After each management review there will be follow up 9.3, 10.1, 10.2 & QMS Coordinator actions on the decisions taken during review and taking 10.3 and related forward improvements where ever identified during review. QSPs 20. Steps 17 to 19 are ongoing - All concerned 2729 2730 It will be realistic to give a time frame of 9 to 12 months for completing all the above steps well. 2731 2732 If the NRA considers necessary to take help of a QMS consultant for implementation of ISO 9001 QMS, 2733 the NRA may appoint one. 2734 2735 2736 2737

Working document QAS/19.783 Page 90

C. Certification of QMS Although not mandated by ISO 9001, If the NRA management wishes to obtain a third-party certification, the NRA may select an accredited Registrar/Certification body (data available on ISO website) at an appropriate time, for example during step 18 above.

The selected certification body (CB) will first examine the NRA’s documents (quality manual, QSPs, SOPs & other documents) for their conformity to ISO 9001. Thereafter once NRA has completed all activities satisfactorily (i.e. up to step 19 above), the CB will arrange an audit of the NRA’s QMS and based upon the audit results will issue the certificate. Certification is generally valid for a period of 3 years and for the maintenance of certification, annual surveillance audits are also carried out by same CB after certification. 2738 2739

2740 6. Considerations to ensure integrated implementation of QMS in

2741 NRA 2742 2743 Implementing ISO Standard 9001:2015 in departments or institutions of the NRA is feasible; however, 2744 integration of several functions of NRAs into one comprehensive and effective system represents a 2745 challenge. Executive coordination between different departments or offices in the decentralized model 2746 or institutions in the discrete model is critical and challenging. This is not just a question of regulatory 2747 affairs because the challenges are widespread. Capacity building and strengthening is essential for 2748 coordinating efforts between institutions involved in achieving good implementation of drug policies, 2749 frameworks, others. 2750 2751 Potential mechanisms that can help in QMS implementation: 2752 2753 • Strong coordination mechanism is established including communication, 2754 • High level support from TM for QMS implementation, 2755 • Assembly of a high-level executive committee to enforce understanding and commitment to QMS 2756 implementation and maintenance, 2757 • Empowerment of the NRA by the MoH with authority to drive QMS implementation, 2758 • Sustainability of QMS would be facilitated if part of the legal framework, e.g. a decree/mandate- 2759 or other legal means supported it, 2760 • Including QMS in the national medicine policy, 2761 • Including responsibility for contributing to QMS in staff job description, Working document QAS/19.783 Page 91

2762 • Training all staff in QMS via courses, conferences, meetings, online platform. Trainings should be 2763 relevant to the regulatory functions, 2764 • Creation of a QMS unit in the NRA or as a minimum appointment of a QMS responsible officer 2765 with the appropriate level of authority, 2766 • Engagement of all stakeholders, 2767 • Creation of a portal for information-sharing among different stakeholders which would allow NRAs 2768 to share their procedures for QMS implementation, documentation, models, frameworks, tools. It 2769 would provide an opportunity for QMS networking among NRAs, 2770 • Coordination of KPIs between different functions so that all in the NRA speak the same QMS 2771 language, integrated and responsive to achieving the strategic plan and the establishment and 2772 maintenance of the QMS 2773 • Monitoring of the process flow between each party, e.g. MA sharing data with NCL and other 2774 functions as needed, 2775 • WHO recommendation for the TM to enforce QMS for all parties of the NRA, 2776 • As a part of the enforcement concept, and for some NRA models, creation of a high level (e.g. in 2777 the MOH) QMS unit with an external audit team, 2778 • Creation of a technical unit with one representative from each organization or department which 2779 meets at regular intervals for coordination, communication and data analysis, 2780 • Practice by many vaccine producers is to have a two-level system: a management review team 2781 which is high-level management who meet once a year and a quality management team which is 2782 more technical. The technical team meets and discusses KPIs monthly or quarterly. A similar 2783 approach could be considered by NRAs.

Working document QAS/19.783 Page 92

2784 References and further reading 2785 NOTE: References listed in this section, and their numbering throughout the guideline, will be 2786 corrected and updated in the final stages of guideline development. 2787 2788 [1] WHA67.20 Resolution http:// http://apps.who.int/gb/ebwha/pdf_files/WHA67/A67_R20-en.pdf 2789 Accessed on 30 September 2017. 2790 [2] WHO Expert Committee on Specifications for Pharmaceutical Preparations. Fiftieth Report. WHO 2791 Technical Report Series 996, 2016; pp 3-4. 2792 [3] Establishment of the WHO Expert Committee on Biological Standardization 2793 http://www.who.int/biologicals/expert_committee/en/ (last accessed 20.03.13).

2794 [4] WHO Expert Committee on Biological Standardization. Forty second Report. Guidelines for 2795 national authorities on quality assurance for biological products. WHO Technical Report Series 822, 2796 1992; Annex 2. 2797 [5] WHO Expert Committee on Biological Standardization. Forty-fifth Report. Regulation and 2798 licensing of biological products in countries with newly developing regulatory authorities. 2799 WHO Technical Report Series 858, 1995; Annex 1. 2800 [6] Good Regulatory Practices: Guideline for National Regulatory Authorities for Medical Products. 2801 WHO/DRAFT/ September 2016. 2802 [7] International Organization for Standardization, ISO 9001:2015. Quality management systems- 2803 Requirements Fifth edition 2015-09-15. Reference number: ISO 9001:2015 (E) 2804 [8] International Organization for Standardization, ISO 9000:2015. Quality management systems- 2805 Fundamentals and vocabulary. Edition 2015 2806 [9] International Organization for Standardization, ISO 9004. Quality of an Organization -Guidance to 2807 achieve sustained results. ISO 9004:2018 2808 [10] International Organization for Standardization, ISO 19011: 2018 provides guidance on auditing 2809 management systems. Second edition 2011-11-11 2810 [11] Guide to the Implementation of a Quality Management System for National Meteorological and 2811 Hydrological Services (2013 edition), World Meteorological Organization, WMO-No. 1100, 2013, 2812 http://www.wmo.int/pages/prog/hwrp/qmf-h/documents/ext/wmo_1100_en.pdf , accessed on 30 2813 September 2017 2814 [12] Guidance for the Implementation of a Quality Management System in Drug Testing 2815 Laboratories: https://www.unodc.org/documents/scientific/QMS_Ebook.pdf , accessed on 30 2816 September 2017. 2817 [13] EDQM, Quality Management Documents https://www.edqm.eu/en/quality-management- 2818 guidelines-86.html. Accessed on 30 September 2017. Working document QAS/19.783 Page 93

2819 [14] ICH Harmonised Tripartite Guideline. Pharmaceutical Quality System Q 10. 2820 Current Step 4 version, 4 June 2008 2821 [15] Quality Management Training for Blood Transfusion Services. 2822 http://www.who.int/bloodsafety/publications/who_eht_05_03a.pdf, accessed on 30 September 2017 2823 [16] Quality systems requirements for national good manufacturing practice inspectorates. 2824 http:// http://apps.who.int/medicinedocs/documents/s22112en/s22112en.pdf, accessed on 30 2825 September 2017 2826 [17] WHO good practices for pharmaceutical quality control laboratories. 2827 http://www.who.int/medicines/areas/quality_safety/quality_assurance/GoodpracticesPharmaceuticalQ 2828 ualityControlLaboratoriesTRS957Annex1.pdf, accessed on 30 September 2017 2829 [18] A new evolution for quality management in the automotive industry. 2830 https://www.iso.org/news/2016/08/Ref2109.html, accessed on 30 September 2017 2831 [19] Guidelines on Quality Management for Multidisciplinary Occupational Health Services. WHO 2832 European Centre for Environment and Health, Bilthoven 2833 http://apps.who.int/iris/bitstream/10665/108268/1/E68239.pdf, accessed on 30 September 2017. 2834 [20] Manual on the Quality Management System for Aeronautical Information Services. International 2835 Civil Aviation Organization. First Edition-2010 2836 https://www.icao.int/APAC/Meetings/2011_AAITF6/QMS%20manual%20formated%20for%20edit 2837 %20and%20submission%20to%20DOC%20Control%20_MH_.pdf, accessed on 30 September 2017. 2838 [21] ISO Technical Specification ISO/TS 9002: 2016 “Quality management systems –Guidelines for 2839 the application of ISO 9001:2015” 2840 [22] ISO/IEC 17020:2012 Conformity assessment- Requirements for the operation of various types of 2841 bodies performing inspection. 2842 [23] ISO/IEC 17025: 2017 General requirements for the competence of testing and calibration 2843 laboratories 2844 [24] ISO 19011:2018 Guidelines for auditing management systems 2845 [25] ISO 31000:2018 Risk management- Guidelines 2846 [26] ISO 9004: 2018 Quality management - Quality of an organization - Guidance to achieve 2847 sustained success 2848 [27] ISO 37001:2016 Anti-bribery management systems- Requirements with guidance for use 2849 [28]ISO/IEC 27001:2013 Information security management- Requirements 2850 [29] ISO 14001:2015 Environmental management systems- Requirements with guidance for use. 2851 [30] Request for Quality Metrics- Guidance for Industry. U.S. Department of Health and Human 2852 Services Food and Drug Administration. Center for Drug Evaluation and Research (CDER). Center 2853 for Biologics Evaluation and Research (CBER). July 2015. Pharmaceutical Quality/CMC. Current 2854 Good Manufacturing Practices (CGMPs)

Working document QAS/19.783 Page 94

2855 [31] Guidance on good data and record management practices. WHO Technical Report Series 996, 2856 annex 5; 2016. 2857 [32] Guidelines on Clinical Evaluation of Vaccines: Regulatory Expectations. WHO/Draft/29 2858 October 2015 2859 [33] Quality Assurance of Medicines terminology database. List of terms and related guidelines 2860 http://www.who.int/medicines/services/expertcommittees/pharmprep/20160302_QASterminologyDB. 2861 pdf 2862 [34] Quality Management Principles. ISO brochure. 2863 https://www.iso.org/files/live/sites/isoorg/files/archive/pdf/en/pub100080.pdf 2864

2865 Authors and acknowledgements 2866 2867 Mr Y. Al-Nujaym, Saudi Food and Drug Authority, Saudi Arabia; Dr O.A.M.A. Badary, National 2868 Organization for Drug Control and Research, Egypt; Ms G.F. Ferreiro, Centro para el Control Estatal 2869 de Medicamento, Equipos y Dispositivos Médicos, Cuba; Ms A. Julsing, Medicines Control Council, 2870 South Africa; Ms Y. Lee, Ministry of Food and Drug Safety, Republic of Korea; Dr R. Lino de Brito, 2871 Agencia Nacional de Vigilancia Sanitaria, Brazil; Ms L. Margaryants, Scientific Centre of Drug and 2872 Medical Technology Expertise, Armenia; Ms G. Mkomagi, Tanzania Food and Drug Authority, 2873 Tanzania; Ms M. Muñozcano Quintanar, Comisión Federal para la Protección contra Riesgos 2874 Sanitarios, Mexico; Mr G. Muthuri Francis, Pharmacy and Poisons Board, Kenya; Ms A. Olivares, 2875 Comisión Federal para la Protección contra Riesgos Sanitarios, Mexico; Mr P. Osatapirat, Thailand 2876 Food and Drug Administration, Thailand; Ms H. Qorani, Jordan Food and Drug Administration, 2877 Jordan; and Dr C. P. Alfonso, Mr S. Arora, Dr R.O.A. Dehaghi, Dr N. Dellepiane, and Ms S. 2878 Ramirez, World Health Organization, Switzerland. Working document QAS/19.783 Page 95

2879 Appendix 1. Integration of QMS into the WHO Global Benchmarking Tool 2880 The WHO Global Benchmarking Tool (GBT) is used to assess the level of implementation of QMS in NRA. The QMS indicator consists of 14 self-scored 2881 sub-indicators to identify the degree of QMS implementation and the existing gaps across the NRA. The equivalence of the 14 sub-indicators in the GBT 2882 with the ISO 9001: 2015 requirements is shown in Table 1. 2883 The concept of maturity level (ML) from ISO 9004:2009 into the stratification of QMS scores is incorporated in the GBT and has been implemented for 2884 some time by the Benchmarking of the European Medicines Agencies (BEMA). Maturity levels take into account the criticality of indicators and the minimal 2885 required capacity of a regulatory system to control and implement proper oversight of health products. 2886 2887 The maturity level indicates the status at which each sub-indicator performs. As maturity level one addresses the legal framework of the regulatory system, 2888 the QMS gap analysis starts at maturity level two. In maturity level two, an organization operates in a reactive mode with an evolving national regulatory 2889 system that partially performs essential regulatory functions. In maturity level three, an organization has a stable, well-functioning and integrated regulatory 2890 system which is accompanied with essential capacity implemented in all functions. Maturity level four implies a regulatory system that performs at advanced 2891 level with continual improvement which support all implemented regulatory functions. The basis for climbing the maturity level ladder for robust regulatory 2892 systems strengthening is for NRAs to fill the gaps at the lowest level before taking a step up. Maturity level three is currently being considered as the target 2893 performing level of national regulatory authorities for implementation of the WHA Resolution 67.20. 2894 2895 Table 1. Equivalence of the 14 GBT sub-indicators with the ISO 9001: 2015 requirements Sub- ISO 9001:2015 Description Requirements ML indicator Clause Requirements for documentation management as well as RS05.07 7.5 Documented information 2 traceability of regulatory activities are established Top management demonstrates commitment and leadership to RS05.01 5.1.1 Leadership and commitment 3 develop and implement QMS

Quality policy, objectives, scope and action plans for establishment Quality policy, objectives, RS05.02 4.3, 5.2.1, 5.2.2, 6.2 3 of the QMS are in place and communicated to all levels scope, and action plans

Organizational chart, roles and responsibilities to establish the Organizational roles, and RS05.03 5.3 3 QMS are defined and in place responsibilities

Working document QAS/19.783 Page 96

Enough competent staff is assigned to develop, implement and Human resources and RS05.04 7.1.2, 7.2 3 maintain the QMS competency

The externally provided products and services relevant to Control of externally provided RS05.09 8.4 3 regulatory activities are controlled through established mechanisms process, products and services

Internal and/or external audits of the QMS are established and RS05.11 9.2 and ISO 19011 Internal audit 3 conducted at planned intervals The regulatory authority establishes required mechanisms to RS05.05 10.3 Continual improvement 4 continually improve the QMS The NRA has identified its regulatory processes, determined their RS05.06 interactions and defined the methods needed to control these 4.4, 8.3, 8.5.1 Operation 4 processes

External and internal issues including relevant potential risks are Actions to address risk and RS05.08 6.1, 4.1 4 defined and assessed periodically for proper risk mitigation opportunities

A mechanism to evaluate the satisfaction of internal and external RS05.10 customers and other interested parties is in place for system 9.1.2, 9.1.3 Customer satisfaction 4 improvement Corrective actions, and actions to address risks and opportunities, Non-conformity and corrective RS05.12 are implemented and documented and their effectiveness is 9.1.3, 10.2 4 action verified Top management reviews and documents the organization’s QMS RS05.13 9.3 Management review 4 at planned intervals (management review) A mechanism is established to evaluate and demonstrate the RS05.14 7, 7.2 Training 4 effectiveness of training activities 2896