D e v e l o p i n g F u r t h e r t h e I R R S P r o c e s s P a g e | 1

2013

IAEA, NSNI, RAS

LUX, Ivan

[DEVELOPING FURTHER THE IRRS PROCESS] Discussions points stemming from the Targeted Consultancies and the analysis of past IRRS missions D e v e l o p i n g F u r t h e r t h e I R R S P r o c e s s P a g e | 2

Contents D e v e l o p i n g F u r t h e r t h e I R R S P r o c e s s P a g e | 3

DEVELOPING FURTHER THE IRRS PROCESS Discussions points stemming from the Targeted Consultancies and the analysis of past IRRS missions

Introduction In order to develop further the IRRS process and - for reflecting this in the official guidance – to elaborate a new edition of the IRRS Guidelines, a series of Targeted Consultancies have been conducted in 2012. For similar reasons the results and data of the major IRRS missions conducted between 2006 and 2011 have been analysed. The IAEA Nuclear Safety Action Plan requires the definition of effectiveness criteria of the IRRS peer review. Partly to answer this requirement a set of performance indicators have been introduced to measure the effectiveness and efficiency of a mission1. The documents summarizing the results of the above consultancies and analysis contain conclusions and proposals that that seem to be advantageous from the point of view of developing further the IRRS process. The issues to discuss are divided into four chapters (General, Preparations for a mission, Conduct of a mission, and Follow-up of a mission). Each chapter contains as many sections as the number of issues belonging to the subject of the chapter. For easier orientation among the various input results each section begins with references to the related Targeted Consultancy (CS) recommendations as given in the consultancy report; references to the conclusions of the IRRS Analysis report (if applicable) and references to the related Performance Indicator and effectiveness criterion (if applicable) in the report entitled ‘On the Efficiency and Effectiveness of the IRRS Missions’. All these report are available from the shared area web space referred to in footnote 1. The issues summarized in this document are such that in the opinion of the author need consent and common decision by the IAEA units having roles in the IRRS process. In each of the issue discussed here one or more propositions are offered for discussion and acceptance. All propositions are supported by factual arguments and/or recommendations by the Targeted Consultancies. The reader is encouraged to consult the underlying documents referred to footnote 1 and bring up further issues for discussion that are not covered by this report but are considered needing consensus decisions. I. General

I.1 IRRS module structure Related CS recommendations: 1.07, 5.03, 6.01 Related analysis conclusions: C14, C20, C23, C29, C30 The present structure of the IRRS modules is well elaborated and tested, yet there is room for further refinement in order to increase efficiency. Two proposals are discussed below.

1 The Terms of Reference and the reports of the consultancies, the report on the analysis and the report on the effectiveness and efficiency performance indicators all are available from the Shared Area website http://gnssn.iaea.org/NSNI/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2FNSNI%2FShared %20Documents%2FOPEN%20Shared%20Files%2FTargeted%20Consultancies%20on%20IRRS%2FConsultancy %20No%2E%2010 D e v e l o p i n g F u r t h e r t h e I R R S P r o c e s s P a g e | 4 a) Revision of Module 11 and the list of facilities and activities Recommendation 1.07 of Targeted Consultancy No. 1 says: “The topics covered by module 11 should be standardized and be part of the core modules for full scope IRRS missions. The usual number of experts in this module exceeds the associated risk”. In recommendations 5.06 it is suggested that “all components technically belonging to the review and assessment, previously listed among additional areas in Module 11, should be incorporated into the core Module 6”. According to recommendation 6.01  The area of transport safety should be included in the list of “Facilities/activities”, … and should be dealt with in the same way as other facilities and activities in this list  Only the additional areas “control of medical exposures” and “control of chronic exposures (radon, NORM and past practices) and remediation” should be considered as optional ones in Module 11 Accordingly Proposition No. 1: the list of facilities and activities should include:  Nuclear power plants;  Research reactors;  Fuel cycle facilities;  Waste management facilities;  Radiation sources facilities;  Decommissioning activities;  Transport activities; whereas Module 11 should consist of:  Control of medical exposures;  Control of chronic exposures (radon, NORM and past practices) and remediation. b) Possible merging of modules

Analysis of past missions shows that module 2 (Global Number of observationsNuclear Safety from initial Regime) missions raises considerably less issues and concerns that the other Recommendationsmodules.Suggestions TheGood practices case is similar with module 8 (Enforcement). Several such data are 140 quoted in the IRRS Analysis report (cf. the conclusions listed above), the case is 120 illustrated here with

s 100 n

the number of observations in major missions as given in Figure 1.o i t a v r

e 80 s b o

Figure 1: Number of the various observations in IRRS missions f o

r 60 e b m

Thus it seems to be justified not to keep these modules separateu but merge them to a larger

N 40 module. 20

Proposition No. 2: module 2 (GNSR) should be merged with0 module 1 (Responsibilities I II III IV V VI VII VIII IX X XI and functions of the government) Module number Proposition No. 3: module 8 (Enforcement) should be merged with module 7 (Inspection)

I.2 Team size and assignment Related CS recommendations: 1.12 – 1.18, 2.09 – 2.13, 2.25, 3.02, 3.03, 4.04, 4.05, 10.31 Related Performance Indicators: P1, P6 D e v e l o p i n g F u r t h e r t h e I R R S P r o c e s s P a g e | 5

Almost all Targeted Consultancies have dealt with the size and module assignment of the IRRS teams (see the referred recommendations above). The long refinement process resulted in a Standard Team Assignment scheme as shown in Table 1. a) Team size The Standard Assignment Scheme in Table 1 defines the estimated size of the IRRS teams for missions to countries with small, medium and large nuclear programmes as well as for follow-up Team sizes missions. These numbers attempt to cover all min ave max

s 25 possible needs and are meant to be guiding r ones e b

that may be adjusted to the specificities of a m 20 e m mission. The IRRS practice so far followed shows that 15 m a

these numbers are realistic (sometimes e t

f 10 o

overestimating) and indeed may cover most of the r e possible needs. This is demonstrated in Figure 2, b 5 where m the average, minimum and maximum team u sizes

N 0 of the past major missions are shown by bars large med small fuand the proposed numbers by red vertical lines. programme size Figure 2: Team sizes in past missions Proposition No. 4: Teams of the future missions should be recruited considering the indicative team sizes given in Table 1 and based on the specificities of the host country. Co-operation of members of Groups C and D (core regulatory activities) and the Facility and Activity Group needs considerations based on the specificity of the actual missions. In most cases besides the general issues and the overall responsibility, also one of the facilities or activities (usually the most important one) is also taken by Groups C and D, the review results of the other facilities and activities are provided them by the Facility and Activity Group. b) Forming reviewer groups Another achievement of the Standard Assignment Scheme above is that the reviewers are expected to work in groups in order to be as effective as possible. Footnotes below the table clarify details of the assignments. The numbers of experts in the groups were elaborated based on past experience and common sense and have been tested in recent missions. These numbers are indicative and averages, they may be adjusted/regrouped according to the specific needs of the missions. Proposition No. 5: Teams of future IRRS missions should be recruited and organized with consideration to the Standard assignment scheme in Table 1 and to the specificities of the hosting regulatory body. Recommendation 10.31 supports this proposition by stating that IAEA should hand over to the host country the Standard Team Assignment and the Standard Mission Schedule tables in order to assist in the compilation of the mission schedule. D e v e l o p i n g F u r t h e r t h e I R R S P r o c e s s P a g e | 6

Table 1: Standard assignment of the modules to groups and guidance on the number of experts in an IRRS mission

Team activitiesa) Smallh) Medium Large Follow-upg) Team Lead (TL, DTL, TC, DTC)b) 3 3 3 3 Group A Governmental R&F (mod 1) 1c) 2c) Global nuclear safety regime (mod 2) Interface with security (mod 12) 3c) 2c) Group B Responsibilities of the regulatory body (mod 3) 2 3 Management system (mod 4) Development of regulations and guides (mod 9)f) Group C Authorization (mod 5) 2 2 Review and assessment (module 6) 2 2 Group D Inspection (mod 7) 2 2 Enforcement (mod 8) Group E 2 2 Emergency preparedness (mod 10) Group F Other optional thematic areas (mod 11) 2 1 1 - control of medical exposures - control of chronic exposures (radon, NORM and past practices) and remediation 3 Facilities/Activitiesd) Nuclear power plants Radiation sources applications Research reactors - 2 5 Fuel cycle facilities Waste management facilities Decommissioning Transport of radioactive material Fukushimae) - 1 1 - Temporary, should be included in standard modules Administration 1 1 1 1 Total 11 17 22 11 a) Numbers of experts include IAEA staff, b) DTC is assigned to a Group, c) DTL supports Group A d) For small nuclear programs the Facilities and Activities are to be reviewed by Groups C – F.  Experts allocated to Fac./Act.s in medium and large missions may need to work together with the experts in Group F.  For large programs the number of reviewers may be less than 5 according to the needs of the mission.  Usually experts in Group C and D are expected to cover the general issues and the nuclear power plants. e) The Fukushima module should be reviewed also for small programs and in follow-up missions even if there is no dedicated reviewer to that f) Groups C through E review the respective contents of the regulations and guides (provided it is in the scope of the mission) and report to Group B g) Numbers of experts of follow-up missions are relevant for large nuclear countries and should be adjusted for small non- nuclear and medium size nuclear countries according to the actual needs. h) Definitions: Small: No NPPs, few Facilities/Activities Medium: Less than 5 NPP units (in construction, operating, to decommission), few Facilities/Activities Large: 5 or more NPP units, and/or many Facilities/Activities I.3 Mission schedule Related CS recommendations: 1.28, 1.46, 2.37, 10.31 Recommendation 1.28 states: For an optimal effectiveness there should be a detailed schedule of the mission including all interviews, visits, meetings and deadlines for report writing. Recommendation 10.31 was quoted above. Consultancy No. 1 provided a detailed mission schedule that was further refined by most of the subsequent consultancies. The resulting Standard Mission Schedule is given in Appendix I. This schedule has been tested in recent initial IRRS missions and was found rather effective. Proposition No. 6: The IRRS missions should be planned according to and should then follow as far as practicable the Standard Mission Schedule as presented in Appendix I. I.4 Structure of the Guidelines One of the major objectives of the series of Targeted Consultancies was to collect suggestions for increasing the effectiveness and efficiency of the missions that most naturally need to be included into a revised version of the IRRS Guidelines. This objective was fully reached by the consultancies and a great number of valuable recommendations and suggestions have been obtained from the experts, most of them having very practical values. It seems to be expedient that the IRRS Guidelines summarize the basic principal facts and data and the practical hints separated. Proposition No. 7: The novel edition of the IRRS Guidelines shall be prepared in two volumes. The first volume shall contain the information present in the actual Guidelines after removing practical hints and guidance from therein. Volume No. 2 is dedicated to practicalities.

I.5 Definition of findings Related CS recommendations: 3.09, 5.02 Recommendation 3.09 says: It is important to highlight in the mission report also good achievements, which do not formally meet the definition of a Good Practice. In recommendation 5.02 it is put forward that: Good Practice is defined as outstanding performance, significantly over that of anybody else. Such a definition, if strictly followed, does not take into account that regulatory bodies are working in significantly different environments (e.g. some of them are in a development phase, and some are operating with significantly limited resources). The consultants suggest that some additional criteria be introduced for appreciating strong features in regulatory performance. There is an obvious need from both the hosts and the reviewers for some kind of appreciation by the peer review of excellent performance of the host country even if this performance is not unique and therefore does not qualify for the status of a Good Practice. We might call it e.g. the “Sign of Excellence”. Proposition No. 8: Sign of Excellence, acknowledging excellent (although not unique) performance of an IRRS host country should be defined and used in future IRRS missions. Exact definition for this shall be elaborated by IAEA.

I.6 Use of web space and IRRS website Related CS recommendations: 1.35, 1.36, 4.03, Several recommendations relate to the use of web and internet. Recommendation 4.03 says: The IAEA should consider opening a portal which contains all mission-related materials to help reviewers in preparing for the mission. For example, this could include a model IRRS mission report, ARM material, IRRS guidelines, CNS report, Lessons Learned reports from previous IRRS missions. An IRRS web-site is under development that is meant to satisfy all needs and requirements related to the web-appearance of IRRS information. Proposition No. 9: The IRRS web-site should be demonstrated to and discussed by the IAEA experts active in the IRRS process.

I.7 Basic IRRS Training Related CS recommendations: 1.19, 1.20, 2.04 Targeted Consultancy No. 1 in its recommendation 1.19 says: IRRS Basic Qualification Training should be developed by the IAEA and required to be completed by each new expert prior to participating in his/her first mission. According to recommendation 2.04 new IRRS team members should complete the appropriate IRRS training module before attending a Mission. As a first step the concept of a Basic IRRS Training (BIT) has been elaborated. The general part of the concept is given below.

The BIT Concept Purpose: To provide the basic information on preparing and conducting IRRS missions to the experts expected to take part as reviewers, to the IAEA staff involved in IAEA missions, and to the Liaison Officers of future IRRS host countries. Format: The Training Course is composed of a series of uniform MS Power Point presentations. Each presentation covers a given subject/topic and is about 30 minutes long. The presentations shall be sufficiently detailed to be also used as a refreshing material after the course. The presentations shall make use of tools and forms (figures, animations, other illustrations) that highlight the main messages and make these messages easy to remember. Each presentation shall summarize the subject in the last 1-2 pages with examples and illustration of great impact (e.g. funny or humorous). For more see “Structure” below. Scope: The series of presentations shall cover the entire IRRS process (including its basis, the IAEA Safety Standards), a historical overview of IRRT and IRRS, practical guidance on preparing and conducting initial and follow-up missions and the data and lessons learned from the past missions. Details of the contents of the presentations shall be elaborated in the specific part of the Concept. Basis: The presentations shall primarily be based on the following documents  Relevant IRRS Safety Standards (as listed in the IRRS Guidelines)  IRRS Guidelines (Edition 2012)  Previous IAEA presentations on the IRRS process  IRRT and IRRS Missions since 1998 in EU Member States (ENSREG WG1, 2009)  IRRS Mission Highlights (Report prepared for the 2011 Washington Conference)  IRRS Targeted Consultancy Report No. 1 through 10 (TargetedConsultancy_No_n_Report.docx, n=1,2,…10)  Summary of the Recommendations from the Consultancies - compilation (ConsRecomSummary.xlsx)  Findings of the IRRS Missions to Major Nuclear Countries, 2006-2011 - compilation (MissionFindings.docx)  Good Practices Identified by IRRS Missions, 2006-2011 - compilation (GoodPractices.docx)  Analysis of Data and Results from IRRS Missions to Major Nuclear Countries, 2006-2011 – report (IRRS Analysis.docx)  On the Efficiency and Effectiveness of the IRRS Missions, Draft v5 – report (IRRS_Effectiveness_Criteria_v5.docx)  Prompt evaluations of the IRRS missions – brief reports on the recent missions (PromptEval_XXX.docx, XXX stands for the abbreviation of the host country) Structure: The individual presentations represent distinct Chapters of the training and shall include the following Sections (whenever they are adequate to the subject): 1. Introduction  Outline and short description of the subject to be presented  Context with and embedment into the entire Training Course  Main messages of the Chapter (2-3 compact statements) 2. Guidance (or Main Body)  Structured presentation of the relevant part(s) of the IRRS Guidelines (or of the document(s) adequate to the subject of the Chapter) – theoretical and factual information, practical issues come in Section 5  The main messages discussed in more depth 3. IAEA Safety Standards  Presentation of the structure and list of the related IAEA Safety Standards with explicit reference to the respective documents on the IAEA website  Brief discussion of the related GSR Part 1 Requirements  Brief discussion of the requirements of other related IAEA SSs  Brief discussion of the related IAEA Safety Guides 4. Illustration  Data, facts, findings, figures and other visual means derived from and illustrating the results of previous missions that are related to the subject 5. Practicum  Practical hints for the reviewers/hosts as derived from the IRRS Guidelines and from the Targeted Consultancy Reports 6. Summary  Short summing up of the entire Chapter  Repetition of the main messages in the framework of a high impact illustration/ visualization The specific contents of the various Chapters shall be developed if the concept is accepted. The Training Course shall include 12 Chapters and shall be arranged in the order of the topics in the IRRS Guidelines. Proposition No. 10: The Basic IRRS Training (BIT) Course should be elaborated according to the concept above. I.8 Steering Committee on IRRS Process In order to take advantage of the experience of senior IRRS experts it seems expedient to set up a Steering Committee on IRRS Process. The proposed Terms of Reference of the Committee is given below. ToR of the Steering Committee Vision: The vision of the Steering Committee on IRRS Process (hereinafter referred to as the ‘Committee’) is that an effective and efficient IRRS process is in place to assist Member States in their efforts for continuous improvement of nuclear safety. Mission: The mission of the Committee is to advise IAEA and the Member States in the development of the IRRS process. Aims: The Committee is aimed at  Providing expertise in matters affecting the effectiveness and efficiency of the IRRS missions;  Offering advice in issues discussed or debated by the IRRS community; and  Taking initiatives for further development of the IRRS process. Composition: The Committee shall be composed of senior international experts with broad and extensive IRRS experience. Members of the Committee shall be invited by IAEA. Invitation of three Committee members shall be initiated by NSNI RAS, two members by NSRW RIT and one member by IEC. The Committee shall elect a Chairperson for three years. The Chairpersons shall be responsible for:  Leading the meetings of the Committee;  Preparing the work programme and planning the meetings;  Assessing the effectiveness and efficiency of the Committee’s activity. The Chairperson will be assisted by the IAEA Secretariat. This assistance is provided by NSNI RAS. Meetings and Working Method: A regular one day meeting of the Committee is held parallel to the yearly IAEA General Conference in Vienna. At the initiative of any Committee member or of the IAEA Secretariat an extraordinary meeting can be called together between two regular meetings. Further meetings by electronic communication (audio-conference, video-conference, e-mail communication) can be organized at request of any member or of the Secretariat. The Agenda of a meeting is determined by the propositions from the Committee members and from the Secretariat. The Agenda is prepared by the Secretariat and is sent for finalization to the Committee members not later than four weeks prior to the meeting. The Committee is free to elaborate a Working Method Document for its own use. Meetings are summarized in Minutes. The Minutes of Meeting (MoM) are composed by the Secretariat and are sent to the Committee members for comments and approval. The MoM shall include:  Conclusions, suggestions, proposals, deliberations taken by the Committee;  Actions with deadlines and responsible persons as decided by the Committee;  Tentative date of the next meeting, including the topics of an extraordinary meeting if so decided. Proposition No. 11: The Steering Committee on IRRS Process as defined in the ToR above shall be established. I.9 Revision of Fukushima Module Related CS recommendations: 3.04, 3.05, 4.04, 5.01, 6.01, Recommendation 3.04 states that: Fukushima issues should be reviewed also for small nuclear programs. In recommendations 4.04 it is suggested that Fukushima topics be also addressed in follow-up missions. Recommendation 5.01 states that: The special Fukushima module should be considered as temporary addition. Its components should be incorporated into the core modules at the earliest opportunity In order to see the possibility and necessity of such changes the mission Fukushima-related Conclusionsreport chapters dedicated to the regulatory positive negative neutral implications of the Fukushima accident 100% have

been analysed for all eight IRRS s missions n

o 80% i that included this chapter in its unified s form. u l c

The details of the analysis results are n 60% o C reported in a document entitled f ‘Analysis o

r 40% of the Results from the Tailored IRRS e Module b m on the Regulatory Implication of the u 20% Fukushima Accident’. The report is N 0% 2 available from the shared area web-site referred to ins1 footnotes2 I 1II. III IV V VI VII VIII IX FigureX aver 3: Distribution of the Fukushima conclusions Modules

The conclusions formulated in these chapters on the prompt reactions, future plans and IRRS module-wise activities related to the accident were sorted into three groups according to their overall contents:  Positive conclusion: evaluates positively the activity of the regulatory body in the given field by referring to outstanding achievements  Neutral conclusion: does not commend neither raises concerns, usually records that the regulatory body acted as expected  Negative conclusion: points to non-conformances or gives important recommendations for improvement Figure 3 shows the results of the grouping. The fractions of positive conclusions are shown by blue bars, that of the negative ones by red bars, the neutrals are given by green bars. On the horizontal axis s1 and s2 refer to the conclusions in the prompt and longer term actions and the roman numbers refer to the numbers of modules. The last column shows the overall average. Accordingly, the average ratios of the positive, negative and neutral conclusions are 28/3/64, respectively, which means that the overwhelming majority (97%) of the conclusions refer to situations where no concern is to be raised. The conclusions were further analysed by their contents, in more details. A number of unified conclusion subjects were defined and the statements in the conclusions were sorted under these unified conclusion subjects. Figure 4 below summarizes the unified conclusions and their distribution among the module-wise conclusions. It is seen that e.g. 40 out of the approximately 160 statements in the conclusions stated that the regulatory body acted appropriately. Similarly, in 27 statements it is acknowledged that the existing status with respect to the given module is appropriate to handle issues similar to the accident. In altogether 13 cases out of the 160 were further actions suggested. The results may even be better illustrated if the ten unified conclusion subjects are condensed into coarse groups of similar quality as given in Table 2 below.

2 Shared Area Documents Unified conclusions No. of unified conclusions 1 - no concern raised 50 2 - existing status appropriate e

c 40 n

3 - necessary action recognised a r a

4 - committed to act e 30 p p

5 - further actions planned a

f 20 o 6 - further actions initiated r e

7 - appropriate action taken b 10 m

8 - exemplary action taken u N 0 9 - stress test 1 2 3 4 5 6 7 8 9 10 10 - further action suggested Unified conclusion sequence no.

Figure 4: Unified conclusion subjects in the module-wise Fukushima conclusions The coarse groups unite those unified conclusions, which express similar activity statuses. The four coarse groups include those conclusions that say that in the given module  the actual status is acceptable – ‘nothing to do’;  what is to be done is planned or started – ‘will be done’ ;  what has to be done was performed – ‘done’;  actions are expected – ‘to be done’.

Unified conclusions sum 60 1 - no concern raised nothing to do 49 2 - existing status appropriate 50 3 - necessary action recognised 40 nothing to do 4 - committed to act will be done 32 30 will be done 5 - further actions planned done 6 - further actions initiated 20 to be done 7 - appropriate action taken 10 8 - exemplary action taken done 54 9 - stress test 0 10 - further action suggested to be done 13 distribution of conclusions

Table 2: Coarse grouping of unified conclusions Figure 5: Distribution in the coarse groups

Figure 5 summarizes in visual form the number of conclusions in the coarse groups. It reflects the fact that in about 70% of the cases the conclusions state that either there was no issue (nothing to do), or the issues were properly handled (done). In 22% of the cases the action has been considered or initiated (will be done) and only in about 9% of the cases remained something to be done. The results above suggest the following conclusions:  In the overwhelming majority of the cases the nuclear regulatory bodies act according to or above the expectations raised by the implications of the Fukushima accident. Apart from a very few cases no need for further development of the regulatory regime was identified as a consequence of the accident;  The peer review has not revealed important issues related to the implications of the accident in the regulatory regime of the host countries. Review and revision of the module may be expedient;  The method of self-assessment and peer review so far followed for the tailored module on the regulatory implications of the Fukushima accident needs revision in order to take advantage of the analysis of the previous results. Simplification of the method seems justified. As a consequence the following propositions are offered: Proposition No. 12: The peer review and report writing related to the first two subchapters on the tailored module on Fukushima implications should be similar to those for other modules of an IRRS missions. To this end the Standardized Report Template (c.f. III.2) should be extended to these subchapters. Proposition No. 13: The peer review and report writing related to the third subchapters on the tailored module on Fukushima implications should concentrate on describing the module-wise regulatory activities in terms of the unified conclusions as above. If the proposals above are accepted their implementation needs the following actions:  Elaboration of the standard template for the tailored Module  Revision of the guidelines for the tailored Module with emphasis on the guidance on the module-wise reporting  Preparation of a table to be used by the reviewers when proposing their unified conclusions Note that most naturally the proposed changes may not influence the results of future missions on the Fukushima implications, yet they are suitable to make the review process more comfortable for the reviewers and the hosts. II. Preparations for a mission

II.1 Information and preparatory meetings Related CS recommendations: 10.23, 10.25, 10.27, 10.28 Recommendation 10.23 states: After the decision on an IRRS Mission the first step should be a one- day information meeting between IAEA and the host country about 18-24 months prior to the mission. In recommendation 10.25 it is suggested that IAEA should consider to make information meeting compulsory and to modify the agendas of that meeting and of the Preparatory Meeting accordingly. Consideration should be given to run SARIS training (see below) simultaneously with the information meeting. As the explanation for these recommendations the consultants expressed that the purpose of the meeting is to decide on the scope and on the approximate date of the mission. Information should be given on financial issues and on the services supplied by the IAEA; works related to the self- assessment and ideas on the policy discussions should be clarified. Furthermore an information seminar on an IRRS mission should be held by the IAEA at least for the relevant regulatory staff. A successful information meeting sets the scene for the whole RB staff and also in more detail for the senior management. According to recommendation 10.28 in order to shorten the preparatory meeting it will focus on high level remaining issues, such as: the final scope of the review, confirmation of IRRS expertise and the counterparts, policy issues, agenda and logistics. The recommendations above can be summarized in a proposition as below: Proposition No. 14: An information meeting should be introduced into the regular IRRS process that is held 18-25 month prior to the mission. This meeting should assist the host country in its timely preparation to the mission by providing information and by setting basic data. The preparatory meeting, to be held 6-9 months before the mission should then be shorter, involve fewer participants and concentrate on the remaining open issues. The meeting would be a good opportunity to share with the host country standardized information on conducting and hosting an IRRS mission. Recommendation 10.31 supports this proposition by stating that IAEA should hand over to the host country the Standard Team Assignment and the Standard Mission Schedule tables in order to assist in the compilation of the mission schedule.

II.2 SARIS tool Related CS recommendations: 1.04, 2.07, 3.10, 4.02, 6.03, 6.06, 7.03, 10.26 Self-assessment plays a central role in the IRRS process as it constitutes the basis for the preparations of the reviewers to the mission and also for the peer reviews themselves. On the other hand, self-assessment serves two purposes at a time since besides its role in the IRRS process it is also meant to be a tool for the regulators to periodically take a snapshot on its status and the progress made since the previous self-assessment. The dual purposes cannot always be served with equal success. It was a returning issue both in the consultancies and in the feedbacks from IRRS missions that the SAT questionnaires need further development and structuring. The corresponding opinions and suggestions are quoted below. (Note that the comments and recommendations treated here relate to the former SAT, some of them may be outdated due to the development of the new SARIS.) Reviewers’ and hosts’ feedbacks from recent IRRS missions almost unanimously suggest simplification and shortening of the questionnaire for IRRS purposes (c.f. the prompt evaluations of the recent missions available from the shared area referred to in footnote 1). Recommendation 1.04 states: The Self-Assessment Tool (SAT) questionnaires are complicated, repetitive, and not user friendly for the host country or the IRRS team members. The host country spends an inordinate and unwarranted amount of resources to answer all of the questions, especially in the special modules on radiation protection, transports and radioactive sources. … The SAT questionnaires should be revised. According to recommendation 2.07 The IRRS SAT questionnaires should be revised and simplified by IAEA. Recommendation 3.10 proposes that the SAT-concept is modified to better support the IRRS review. SAT should produce two sets of output. One output should be intended for the regulatory body´s internal use and include the answers to the detailed question lists. The other output should be intended to be part of the ARM, focus on assessment and evaluation, and should include the action plan. In recommendation 4.02 it is suggested that the IAEA could consider regrouping the SAT questions in thematically logical groups. By virtue of recommendation 6.06 the Self-Assessment Tool (SAT) questionnaires are complicated, repetitive, and not user friendly for the host country nor for the IRRS team members. The SAT questionnaires are not balanced and do not follow the principle of graded approach… The SAT questionnaires have to be revised and harmonized with the IRRS mission report structure. The complete set of SAT questionnaires has to be finalized by a single team. The most important messages of the opinions and recommendations above is summarized in the proposition below: Proposition No. 15: The current self-assessment questionnaires should be revised, the revised version should be simpler, shorter and should follow the structure of the IRRS mission report. Proposition No. 16: The output of the IRRS self-assessment tool should produce part of the ARM in a structure analogous to that of the mission report. Note also that according to recommendation 10.26: in order to give the host country insight in SARIS tool, IAEA should encourage the host country to participate in a … course on SARIS. It is possible (but not necessary) to have it in connection with the information meeting. II.3 ARM Related CS recommendations: 1.03, 2.08, 2.19, 3.12, 4.01, 5.08, 5.09, 5.12, 10.16, 10.17 The ARM is the basic input document set of an IRRS mission and as such its structuring and preparation are of primary importance. Several consultancies have dealt with this issue; the most important recommendations are quoted below. Recommendation 1.03 states that: the ARM should include summaries for every module highlighting the specific aspects of the regulatory framework of the host country and give an overall perspective on the regulator. The host country should prepare the ARM in the same format and template used by the IRRS team to prepare the mission report. In recommendation 4.01 it is suggested that in order to make the ARM still more efficient  Summaries in the ARM for each module should be structured so that it helps the reviewer in evaluating the compliance with the relevant GSR part 1 requirements specific to that module.  Compliance with other relevant IAEA standard’s requirements should also be demonstrated.  IAEA should develop a template for mapping (see Swiss example) to help the host country in the demonstration. Recommendation 5.12 requires that the regulatory body’s action plan in response to its self- assessment should be considered as an essential part of the ARM. According to recommendation 10.16: IAEA should elaborate a template for the ARM, the host country should prepare the ARM using this IRRS ARM template, while in recommendations 10.17 it is required that the ARM should include executive summaries for every module highlighting the issues arising for those specific aspects of the national infrastructure for safety, and should give the overall information on the identified strengths, weaknesses, opportunities for improvement based on the self-assessment. The recommendations above quite unanimously suggest the following Proposition No. 17: IAEA should elaborate a template for the contents and format of the ARM taking into account the relevant good practices of the past missions. A template for the Action Plan would also be useful. In specific, it seems to be useful to set requirements on the summary of self-assessment: Proposition No. 18: The ARM should contain the summary of the results of the self- assessment for each module following the format of the IRRS missions report. Proposition No. 19: The ARM should also contain the results of the action plan implementation that followed a previous IRRS mission (if there was one).

II.4 Timelines of preparations Related CS recommendations: 10.Appendix Preparations to an IRRS missions is a long process that may take as much as three years. Proper timing of the activities is essential for the success. Consultancy No. 10 has set up the time-table of the preparatory activities, it is found in Appendix II to this document. Proposition No. 20: The time-table in Appendix II should be accepted as the one to be suggested by the IRRS Guidelines to the host countries.

III. Conduct of a mission

III.1 Initial team meeting and team building Related CS recommendations: 1.24, 1.25, 2.14, In recommendation 1.24 it is expressed that experience from prior missions has shown that the preparation phase should be more structured in order to better ensure that every expert is equally well prepared. The importance of preparation for the mission as an essential prerequisite for participating in the team should be reemphasized. Recommendations 1.25 says: There is not enough time available for a team building process prior to the start of the work. The arrival day and the first Sunday could be used for that purpose. The time before the entrance meeting should also be used for the intentional and structured team building process. In recommendations 2.14: The Consultants agreed with the proposal of Consultancy No.1 that additional time should be allocated at the beginning of the Mission for team building to promote efficiency of the team. This should also include rules of conduct and expected behaviours. Additional time should be allocated at the beginning of the Mission for team building. The Standard Missions Schedule in Appendix I takes into account these recommendations. They are summarized in the following explicit proposition: Proposition No. 21: The official program of an IRRS mission should include a team building session on Sunday morning (or the day before the Entrance meeting) prior to the Initial team meeting. Accordingly, the team members are expected to arrive the day before (i.e. on Saturday, if the mission starts on Monday).

III.2 Mission report template Related CS recommendations: 1.33, 1.34, 2.31, 3.14, 3.18, 4.10, 4.16, 4.18, 5.21, 5.22, 6.14, 6.15, 6.17, 7.20, 7.26, 7.30 – 7.33, 9.01 Standardization of the mission report is beneficial for several reasons. It gives guidance to the host country in developing the ARM, to the reviewer in making interviews and in preparing their report contributions and to everyone who wishes to make comparisons of or draw conclusions from several missions. Almost all consultancies have dealt with this question and agreed in the need of a standardized template. The most relevant recommendations are quoted below; most of the recommendations referred to above give details of the template related to the module investigated. Recommendations 1.33 says: The report Template should be developed/revised to guide authors on structure and content – the background, comparison with standards and expectations, conclusions justification to support R, S & GPs as well as form and style. According to recommendation 1.34 harmonisation both within the mission report and with other mission reports should be an important goal of the team. In recommendation 5.22 the consultants support development of the general template of the final report from IRRS missions as important means for improved effectiveness, harmonization and consistency of the missions. Based on the recommendations of the previous consultancies consultancy No. 9 elaborated a detailed mission report template. It is available from the shared area referred to in footnote 1. The template has been tested in a recent mission to Finland and was received very positively (c.f. the Prompt Evaluation Report on the mission also available from the shared area.) Consequently the following proposition seems to be justified: Proposition No. 22: The IRRS missions should use the Standard IRRS Mission Report Template as guiding document for the summaries of the ARM (c.f. Proposition No. 17) as well as for the interviews and the mission reports.

III.3 Balance of findings Related CS recommendations: 1.37, 3.08 Related analysis conclusions: C12 Related Performance Indicators: P11 Recommendation 1.38 states: The focus for balance, harmonization and consistency of R, S & GPs should reflect the main purpose of the mission which is to improve the regulation of nuclear safety for the Host Country and the International Regulatory Community. The recommendation also suggests that in order reaching a realistic review result there must be certain balance between the numbers of various observations of a mission. It is expected that in a mission where a large number of non-compliance with the IAEA safety standards is found the number of good practices will be lower than the average and vice versa. The question has been analysed for the major past missions. Figure 6 illustrates the tendency of findings in a number of missions, the blue line shows the number of recommendations (divided by its average) and the green line shows the number of good practices (again divided by its average). Obviously the two lines have opposite tendencies, in other words the two quantities are negatively correlated and therefore their empirical correlation ratio is expected to be nearly constant over the efficient missions

This observation leads to the definition of the Balance of Findings Performance Indicator. Let N R denote the number of recommendations NG the number of good practices in a mission and let

NAv(R*G) be the average of the product NR* NG (as calculated from the past missions). The PI describing the balance of the findings in a mission is defined as

ρ = √[( NR* NG)/ NAv(R*G)] In Figure 6 the orange line shows the values of this indicator in the past missions. It is apparent that although the numbers of the recommendations and good practices assume substantially differing values in various missions, this indicator shows only moderate changes around unity. It is also

Tendency of findings

Recommendations Good Practices Balance

2.50

2.00

1.50

1.00

0.50

0.00

Figure 6: Recommendations, good practices and their balance in recent missions obvious that whenever the value of this indicator is substantially lower than unity then either the number of findings or the number of Good Practices is too low, whereas if the indicator is much greater than unity then one of the observations types is too high. Proposition No. 23: The Balance of Findings PI should be used as one of the indicators of the effectiveness and efficiency of the IRRS missions. As such it will be part of the prompt evaluation of the missions (c.f. Section IV.1)

III.4 Feedbacks on the mission Related CS recommendations: 1.43, 1.44, 2.02, 2.34, 2.35, 6.05 Related Performance Indicators: P5, P7, P8 In recent missions feedbacks from both the IRRS team members and the host countries were requested in order to evaluate the efficiency and effectiveness of the mission. The feedbacks were obtained via questionnaires on the following issues:  Team members‘ feedback on the ARM;  Team members feedback on the effectiveness of the mission; and  Host country’s feedback on the effectiveness of the mission. Examples of the feedback results are available in the prompt evaluation reports of recent missions uploaded to the shared area referred to in footnote 1. Recommendation 1.44 says that the consultancy participants discussed IAEA’s recent initiative to use surveys of IRRS Team members and the host country as a means of providing feedback on the effectiveness of the IRRS mission. The participants were positively impressed by this initiative and believe it should be continued. Continue the initiative to use surveys of IRRS Team members and the host countries to provide feedback on the effectiveness of the IRRS mission According to recommendation 2.02 IAEA should develop a consistent approach to collect and evaluate feedback from IRRS host countries and participants on Missions to learn lessons. In recommendation 2.34 the Consultants recommend IRRS participants give structured feedback on the conduct and effectiveness of the mission. The Consultants endorse IAEA proposals for developing a process for collecting and evaluating this information. This is to promote the continuing improvement to the IRRS process. Recommendation 6.05 proposes that the document “On the Efficiency and Effectiveness of the IRRS Mission” contain provisions on feedback from the IRRS team members and the host organisation on the IAEA Safety Standards used. The IAEA should develop a system for collection, evaluation and further use of these comments and recommendations Following the recommendations quoted the proposition below seems justified: Proposition No. 24: The questionnaires for feedbacks from the team members and from the host country as proposed in the effectiveness and efficiency report and used in recent missions should be used in future IRRS missions. IV. Follow-up of a mission

IV.1 Prompt evaluation of a mission Related CS recommendations: 1.44 The evaluation of the effectiveness and efficiency of missions may have several advantages:  Feedback is given to the participants (team members, host country, IAEA)  The possible ways of further improvement may be derived  Efficiency and effectiveness of the IRRS process is demonstrated in a transparent way Examples of prompt evaluation of recent missions are reachable from the shared area referred to in footnote 13. Recommendation 1.44 states that the consultancy participants discussed IAEA’s recent initiative to use surveys of IRRS Team members and the host country as a means of providing feedback on the effectiveness of the IRRS mission. The participants were positively impressed by this initiative and believe it should be continued.

3 Shared Area Documents Continue the initiative to use surveys of IRRS Team members and the host countries to provide feedback on the effectiveness of the IRRS mission Accordingly: Proposition No. 25: Prompt evaluation of IRRS mission should be a permanent practice. Proposition No. 26: Evaluation report should be prepared within two month after the missions. IV.2 Report finalization Related CS recommendations: 1.42 The actual version of the IRRS Guidelines requires that “the IAEA Coordinator in conjunction with the IRRS Team Leader, and with appropriate coordination with other team members, will assess the host country comments and draft the final IRRS Report”. Recommendation 1.42 proposes: Provide clarity regarding the expected interaction between the IAEA Team Coordinator, Team Leader and Team Members after receipt of any final comments from the host country. Modify the Guidelines to read as follows: “The IAEA Coordinator and the IRRS Team Leader, with the appropriate coordination with IRRS team members, will produce the final report. This action will include an assessment of any final comments received from the host country.” Proposition No. 27: The IRRS Guideline should clarify in more details the responsibilities and duties of the team members in finalizing the mission report. Proposition No. 28: The team members need to be informed of any change in the mission report occurring or proposed after the end of the mission.

IV.3 Follow-up as a compulsory part Related CS recommendations: 4.22 Recommendation 4.22 proposes that follow up mission should be made a compulsory part of the mission. The IAEA should not conduct a mission unless there is a clear statement from the host country to host a follow up mission. Proposition No. 29: Follow-up should be considered to be a permanent/compulsory/ strongly recommended part of the IRRS process. A tentative date for the follow-up missions should be discussed during the preparatory meeting of the initial missions and should be included into the IRRS mission report

IV.4 Timing of follow-up mission Related CS recommendations: 4.24 Related analysis conclusions: C17 Ratio of open issues in follow- Analysis of the past IRRS missions demonstrated that considerable up portionmissions of the findings is not closed by the time of the follow-up missions. Figure 7 Suggestions showsRecommendations theAll average, minimum and maximum ratios of the open issues to the 0.70 number of findings. It is seen that on average 23% of the findings remains open. 0.60

Based on this observation the analysis concludes that the relatively 0.50 high number of s e

issues remained open suggests that the time between the initial andu follow-up missions may s s i 0.40 l l sometimes be not sufficient to comply with the recommendations or suggestionsa of the initial / s e u

s 0.30 s i In line with this observation recommendation 4.24 says: The consultants suggest that the follow up n e p mission is conducted within 3 years after the original mission o 0.20 Figure 7: Ranges of0.10 open issues in follow-ups

0.00 Proposition No. 30: The time period of requesting a follow-up missionMinimum shouldAverage be Maximum changed from the actual value of “from two to four years” after the initial mission to “from three to five years”. IV.5 Follow-up mission report template Related CS recommendations: 9.02 The Consultancies recommended the use of a standard template form also for the follow-up mission report. It is available from the shared area referred to in the footnotes above. Proposition No. 31: The follow-up IRRS missions should use the Standard IRRS Follow-up Mission Report Template as guiding document for the mission reports. Appendix I: Standard Schedule of IRRS Missions Initial Mission First Week

Time SAT SUN MON WED THU FRI SAT SUN Arrival D 9:00- TM write Report of Team T 10:00 TL and DTL Members C  Disc review 10:00- ussin introductory part 11:00 w g V r and i i impr s t ovin i e g I t s Draft n Team building s i Repo V t meeting: / n rt Re Int i e E t adi Entrance erv s r  Cros Visits P r ng,  5 minutes/TM Meeting iew i v s- R o self-intro s t i Read Cr 11:00- d Draft text to  Refresher s e ing oss 12:00 e u TL training w  TL, - x c s DTL Free day, Social rea e t , TC Tour din r o and c r g DTC . y of read p the ever a Re ythin r g por t t s 12:00- Standing lunch Standing lunch Standing lunch Standing lunch 13:00 Lunch with Lunch Visits Int V I V D Policy Finalisat 13:00- Host erv i n i T Discussions ion of 14:00 iew s t s C the 14:00- Initial Team Intervie s i e i Draft 15:00 Meeting: ws t r t w Report 15:00- 16:00 s

 IRRS process  Main Secretariat objectives edits the  Report writing report Cross- Preliminary reading  Schedule v s r Draft by TM First i / i  Written Report e E t 16:00- observations preliminary Ready 17:00  In-Group findings w P e discussions delivered s R s i E n x t e r r o c d

17:00- Daily Team Daily Team Meeting: Daily Team Daily Team 18:00 Meeting Discussion of findings Meeting Meeting

18:00- Informal Team Dinner Dinner Dinner Dinner Dinner Dinner 20:00 dinner

20:00- Writing of Secretariat edits Report Writing of the Secretariat edits the TM Read Draft 24:00 the report TM write Report report report Initial Mission Second Week

MON TUE WED FRI 9:00-10:00 Common read through Submission of the Final Draft 9:00-10:00 Cross-Reading Individual discussions and finalisation by the TL, DTL, TC and DTC of Rs, Ss and GPs Team read everything Exit Meeting 10:00-12:00 with counterparts 10:00-12:00 Finalisation Submission of the Draft Press Conference to the Host 12:00-13:00 Standing lunch Standing lunch Lunch Lunch 12:00-13:00 T W C, r 13:00-15:00 Policy Discussions 13:00-15:00 D T T Host reads e TC Drafts the Press Release Departure Home C Draft Individual discussions pr 15:00-17:00 of Rs, Ss and GPs ep 15:00-17:00 with counterparts ar e E xe cu tiv Discussion of the e report by the Su team m m ar Discussion of Briefing of the DDG 17:00-18:00 Daily Team Meeting y Executive 17:00-18:00 Finalisation of the press release an Summary d ex it pr es en tat io n

18:00-20:00 Dinner Dinner Dinner 18:00-20:00

20:00-21:00 20:00-21:00 Secretariat includes Secretariat finalises text Free changes 21:00-24:00 21:00-24:00

Follow-up Mission

Time SAT SUN MON TUE WED THU SAT Team 9:00-10:00 Entrance Meeting TM write Report building TL and DTL Finalisation meeting: review 10:00-11:00 introductory part  5 Interviews Exit Meeting minutes/ & Interviews TM self- Submission of the Press Conference 11:00-12:00 Draft text to TL intro Draft to the Host (alternative)  Refreshe r training Standing 12:00-13:00 Standing Lunch Standing lunch Standing lunch lunch Lunch T Written 13:00-14:00 Policy L comments by Discussions fi the Host 14:00-15:00 n a Initial Se Arriv li Team cre H al of s Meeting: Interviews tar o Team Cross e iat st Finalisation of Mem  IRRS - s TC 15:00-16:00 edi r the Report bers process readi t draft ts e  Main ng h s the the a objectiv Interviews e Pres rep d es p s ort s r Rele Departures of Team Members  Report D e ase writing r s a  Schedul e e Written ft Presenting the preliminary Preliminary Draft n final Draft of 16:00-17:00  First findings Report Ready t the Report to observat delivered a the Host ions ti o n Daily Team Discussion of Daily Team Meeting: Daily Team 17:00-18:00 Executive Meeting Discussion of Meeting Summary findings 18:00-19:00 Informal Team Dinner Dinner Dinner Dinner 19:00-20:00 dinner Dinner Secretariat Writing of the edits Report 20:00-24:00 TM Read Draft Free report TM write Report Appendix II: Standard timing table of pre-mission activities Remark: This table provides time sequence of the activities. The table contains major activities only and it can be further developed and broken down into many more steps for actual planning purposes. The time scale is illustrative and may be adjusted to the actual needs and possibilities. T0 is the beginning time of the IRRS mission From/to Durat. Activity Guidance Participants Outcome/deliverables T0-3 y’s/ - Request for IRRS Ch.2 host IAEA plans and initiates the T0-2 y’s mission T0-24 m’s/ 1-2 Information Ch.5 IAEA, host and IAEA provides initial T0-18 m’s days Meeting all information on the mission stakeholders and also training on self- assessment, indication of the time when mission takes place T0-15 m’s/ 2-3 m’s Preparations for Ch.3 Host and Project plan T0-12 m’s self-assessment stakeholders T0-12 m’s/ 3-9 m’s Self-assessment Ch. 3 host and all SA Report, List of non- T0-9 m’s stakeholders – compliances, Action Plan counterparts are responsible T0-9 m’s/ 1-2 Prep Meting Ch.5 IAEA, host and Scope is fixed, Policy Issues T0-6 m’s days all selected, Schedule agreed stakeholders (including visits) T0-3 m’s/ 1 day Sending out ARM Ch. 4 Senior ARM received by the IAEA T0-2 m’s to IAEA management, – Remark: schedule Counterparts, preparation of ARM T0-9 m’s/ Within Resource Ch. 8 Project group Booking of hotel rooms T0 that planning: and assigning rooms at the period regulator T0-5 m’s/ Within Resource Ch. 5 Project group Room for entrance T0 that planning: meeting for IRRS team and period hosts (30+ people) T0-5 m’s/ Within Resource Ch. 5 Project group Room(s) for Exit meeting + T0 that planning: press conference facilities, period PR staff is engaged T0-9 m’s/ Within Resource Ch. 6 Project group Administrative support will T0 that planning: be available all the time period T0-9 m’s/ Within Resource Ch. 6 Project group Emergency exercise T0 that planning: planned and organized period T0-9 m’s/ Within Resource Ch. 7 Staff involved Not allow absences for the T0 that planning: time of the mission period T0-5 m’s/ Within Resource Ch. 6 Project group IT resources including IT T0 that planning: staff are planned and period informed, and supplied or booked for the time of IRRS mission T0-5 m’s/ Within Resource Ch. 6 Project group Communication T0 that planning: arrangements discussed, period analysed and finalized T0-5 m’s/ Within Resource Ch. 6 Project group Catering organized T0 that planning: period T0-5 m’s/ Within Resource Ch. 6 Project group Transport organized T0 that planning: including visits and social period events T0-5 m’s/ Within Resource Ch. 6 Project group Visits agreed with T0 that planning: stakeholders including period licensees T0-5 m’s/ Within Resource Ch. 6 Project group Inspection visits organized T0 that planning: period T0-1 m/ 1-2 hrs Organize Ch. 2 All hosts staff Hosts' staff is informed T0-2 w’s meeting/briefing about the objectives, for all hosts staff conduct, scope of the mission – in general terms T0-2 m’s/ Within Mission planning: Ch. 7 Project group Ensure availability of all T0-2 w’s that relevant documents period T0-2 m’s/ Within Mission planning: Ch. 7 Project group Information flow within the T0-2 w’s that host organization is period planned, senior staff is familiar with that T0-2 m’s/ Within Mission planning: Ch. 7 Project group Efficient re-scheduling of T0 that changes is organized and period means to inform all interested persons are in place