Interoperability Process Reference Manual

1 2 3 4 5 6 7 8 9 Smart Grid Testing & Certification Committee (SGTCC)

Interoperability Process Reference Manual (IPRM)

Version 2.0

2nd DRAFT – November 3, 2011 10 11 12

2 3 4 Interoperability Process Reference 5 Manual 6 (IPRM) 7 13Contents

14CONTENTS...... 2 151. IPRM EXECUTIVE SUMMARY...... 5

16 1.1. PURPOSE AND PROBLEM STATEMENT...... 6 17 1.2. PRESENT TESTING FLOW...... 7 18 1.3. FUTURE TESTING FLOW...... 8 19 1.4. INTENDED AUDIENCE...... 9 202. INTERNATIONAL GUIDELINES FOR TESTING AND CERTIFICATION...... 11

21 2.1. OVERVIEW OF ISO/IEC 17025...... 11 22 2.2. OVERVIEW OF ISO/IEC GUIDE 65...... 13 233. OVERVIEW OF IPRM VERSION 2...... 14

24 3.1. CHANGES FROM VERSION 1...... 14 25 3.2. ORGANIZATION OF THE IPRM...... 15 264. ITCA IMPLEMENTATION OF THE IPRM...... 16

27 4.1. WHAT IS AN ITCA?...... 16 28 4.2. HOW DOES AN ITCA IMPLEMENT THE IPRM?...... 17 29 4.3. RELATIONSHIP BETWEEN ACCREDITATION BODIES, ITCAS, CERTIFICATION BODIES AND TEST LABS. 19 305. BEST PRACTICES FOR INTEROPERABILITY AND CONFORMANCE TEST 31CONSTRUCTION...... 20

32 5.1 GENERAL TEST POLICIES...... 20 33 5.2 TEST SUITE SPECIFICATION (TSS)...... 22 34 5.3 ATTRIBUTES OF A TEST PROFILE IN LIEU OF COMPLETE TSS...... 23 356. CYBERSECURITY TESTING...... 24

36 6.1. INTRODUCTION...... 24 37 6.2. CYBERSECURITY CONCEPTS...... 25 38 6.3. CYBERSECURITY TESTING ENVIRONMENT...... 25 39 6.4. CYBERSECURITY TESTING TYPES AND FRAMEWORKS...... 25 40 6.5. ADDITIONAL CYBERSECURITY TESTING LABORATORY BEST PRACTICES TO CONSIDER...... 29 41 6.6. ROLE OF THE CYBERSECURITY TESTING PROVIDERS...... 30 427. INTEROPERABILITY TESTING AND CERTIFICATION AUTHORITY ROLE AND 43REQUIREMENTS...... 32

44 7.1. INTEROPERABILITY REQUIREMENTS FOR USE BY THE ITCA...... 33 45 7.2. GOVERNANCE...... 33 46 Table 1 – Interoperability Process Governance Requirements...... 36 47 7.3. LAB QUALIFICATION...... 36 48 Table 2 – Interoperability Lab Qualification Process Requirements...... 37 49 7.4. TECHNICAL DESIGN FOR INTEROPERABILITY AND CONFORMANCE PROGRAM DESIGN...... 37 50 Table 3 – Interoperability Technical Design Process Requirements...... 44

8______9 2 10 Smart Grid Testing And Conformity Committee 11 IPRM Version 2.0 – DRAFT November 3, 2011 12 13 Interoperability Process Reference 14 Manual 15 (IPRM) 51 7.5. IMPROVEMENTS...... 44 52 Table 4 – Interoperability Improvements Process Requirements...... 45 53 7.6. CYBER SECURITY...... 45 54 Table 5 – Interoperability Cyber Security Process Requirements...... 47 558. REFERENCES...... 48 569. GLOSSARY OF TERMS...... 49 57A. WORKING GROUP...... 55 58B. DOCUMENT HISTORY...... 56 59

16______17 3 18 Smart Grid Testing And Conformity Committee 19 IPRM Version 2.0 – DRAFT – November 3, 2011 20 21 Interoperability Process Reference 22 Manual 23 (IPRM) 24 60Preface 61

62About the SGTCC 63The Smart Grid Testing & Certification Committee (SGTCC) is a standing com- 64mittee within the Smart Grid Interoperability Panel (SGIP), the organization ini- 65tiated by the National Institute of Standards and Technology (NIST) to coordi- 66nate standards deployment for the Smart Grid. The SGTCC mission is the cre- 67ation of 68  organizational frameworks, 69  methodologies, and 70  documentation 71relating to compliance testing and product certification on Smart Grid interop- 72erability and cybersecurity based standards. 73 74The SGTCC is composed of a broad range of volunteers with expertise in test- 75ing and product certification associated with utilities, vendors, independent 76test labs, accreditation bodies, associations and consortia that operate certifi- 77cation programs, standards bodies, and government. The SGTCC is open to all 78interested individuals with testing and certification expertise and responsibility, 79with leadership provided by an SGIP-elected panel of approximately 30 voting 80participants. 81

82Contact the SGTCC

25______26 4 27 Smart Grid Testing And Conformity Committee 28 IPRM Version 2.0 – DRAFT November 3, 2011 29 30 Interoperability Process Reference 31 Manual 32 (IPRM) 83Questions or comments about this document, as well as general inquiries 84about the SGTCC may be directed to: 85Rik Drummond, SGTCC Chairman ([email protected]) 86or 87Rudi Schubert, SGTCC Program Administrator ([email protected])

33______34 5 35 Smart Grid Testing And Conformity Committee 36 IPRM Version 2.0 – DRAFT – November 3, 2011 37 38 Interoperability Process Reference 39 Manual 40 (IPRM) 41

881. IPRM Executive Summary 89 90The SGTCC has developed and issued this Interoperability Process Reference 91Manual (IPRM) detailing its recommendations on processes and best practices 92that enhance the introduction of interoperable products in the market place. 93These recommendations build upon international standards based processes 94for interoperability testing and certification. 95 96 Implementation of the IPRM by Interoperability Testing and Certification 97 Authorities (ITCAs) will increase the quality of standards-based, secure 98 and interoperable products in the Smart Grid marketplace. 99 100The SGTCC believes that implementation of the IPRM will lead to reduced de- 101ployment costs of Smart Grid systems and devices, and enhanced product 102quality with respect to interoperability and conformance, ultimately providing 103increased end-user customer satisfaction, and confidence to the buyer through 104meaningful certification programs. 105 106The IPRM is a key foundational element of the SGIP Testing and Certification 107Framework. It will enable the adoption of consistent and measurable certifica- 108tion and testing policies and procedures across Smart Grid products (utilizing 109standards) based on the conformance, interoperability, and cybersecurity test- 110ing experience and expertise of SGTCC participants, and the widely accepted 111ISO/IEC 17025 and ISO/IEC Guide 65 international standards for testing labora- 112tory and certification body management systems. 113 42______43 6 44 Smart Grid Testing And Conformity Committee 45 IPRM Version 2.0 – DRAFT November 3, 2011 46 47 Interoperability Process Reference 48 Manual 49 (IPRM) 114The ISO/IEC testing and certification standards provide a solid foundation for 115the development and operation of high quality testing and certification pro- 116grams. The SGTCC also recognized that additional technical requirements and 117best practices are necessary to help assure test program technical depth and 118sufficiency in meeting end user expectations for interoperability and cyberse- 119curity. These additional recommendations are detailed in this IPRM.

1201.1. Purpose and Problem Statement 121 122The IPRM is intended to enhance the validity and consistency of testing and 123certification programs for standards based products in the market place to help 124assure their conformance and interoperability for the end user/buyer. 125 126The SGTCC has identified the need for testing and certification programs with 127uniform quality processes (ISO 9001, Quality Management Systems - Require- 128ments based) across all products based on Smart Grid standards. Additionally 129there is a need for third party assessment and accreditation services to be 130available to help assure that these programs achieve the technical and quality 131expectations of end users. 132 133Trusted 3rd party certification programs require specific and detailed test re- 134quirements, pass/fail metrics, and defined processes to enable a consistent 135and well-understood approach to testing, and the subsequent assessment of 136the test certified results. There can be little left to interpretation as a vague or 137undefined process will lead to inconsistent application and conclusions provid- 138ing negligible benefit to end users. 139

50______51 7 52 Smart Grid Testing And Conformity Committee 53 IPRM Version 2.0 – DRAFT – November 3, 2011 54 55 Interoperability Process Reference 56 Manual 57 (IPRM) 58 1403rd party, independent and trusted certification programs associated with some 141of the Smart Grid standards under consideration for the SGIP Catalogue of 142Standards do exist, however there is significant variability across those pro- 143grams with respect to the depth of testing and the detailed processes, policies 144and practices in place to support the granting of certification. This presents an 145additional dimension to the problem as there is a need for all Smart Grid certifi- 146cation programs to be meaningful and rigorous to increase end user confidence 147in their product decisions related to interoperability between products and sys- 148tems. 149 150The IPRM seeks to address these issues described above and provide the rec- 151ommendations of the SGTCC in structuring robust testing and certification pro- 152grams, and the means to assess the success of ITCAs in their implementation 153of the IPRM recommendations. 154 155

1561.2. Present Testing Flow 157 158The diagram below illustrates the present approach to testing for Smart Grid 159systems and devices. 160

59______60 8 61 Smart Grid Testing And Conformity Committee 62 IPRM Version 2.0 – DRAFT November 3, 2011 63 64 Interoperability Process Reference 65 Manual 66 (IPRM)

Smart Grid Product Testing Flow– Present Model

Phase O

D Define a Standards Develop and Release S

/ Need Standards y r t s u d n I b a L

Validation Testing t s e T s r o

d Develop Products Products Deployed n e V 161

Figure 1 – Smart Grid Product Testing Flow – Present Model 162 163In this model, standards are developed by industry and vendors use these 164standards for their product design and testing. Testing may be done internal to 165the vendor, onsite at a utility or other end user laboratory, or working with an 166independent provider of testing services. 167 168This model is straight forward and useful; however in the absence of a broader 169framework, there are broad variants in the approach and depth of testing pro- 170grams, leading to uncertainty in whether or not the testing is achieving the 171needs of end users. A number of current industry programs offer options for 172certification, usually for conformance to a specification. However a majority of 173programs simply include basic testing and result reporting, and do not go so 174far as to certify conformance or especially interoperability. Thus the end buyer

67______68 9 69 Smart Grid Testing And Conformity Committee 70 IPRM Version 2.0 – DRAFT – November 3, 2011 71 72 Interoperability Process Reference 73 Manual 74 (IPRM) 75 175is not assured of conformance and interoperability quality products in the mar- 176ket place. 177 178

1791.3. Future Testing Flow 180The diagram below illustrates the future testing model proposed by the SGTCC 181for Smart Grid systems and devices. It goes beyond the various basic programs 182currently available introducing a new concept of ITCAs as test and certification 183program owners responsible for implementing a consistent high quality frame- 184work across test programs within their scope of operations. Additionally, ac- 185creditation bodies are introduced to independently validate that ITCA certifica- 186tion bodies and test labs are indeed adhering to the recommended practices. 187This will provide greater confidence to end users in products that exhibit inter- 188operability and conformance. 189

76______77 10 78 Smart Grid Testing And Conformity Committee 79 IPRM Version 2.0 – DRAFT November 3, 2011 80 81 Interoperability Process Reference 82 Manual 83 (IPRM)

Smart Grid Product Testing Flow- FMO

Phase

Support Establishing

O Define a Standards Develop and Release Support Companion ITCA to support D Need Standard Test Criteria S standard/testing

Evaluate State of ITCA Designated as ITCA Designated as Does an ITCA Testing Support for Transitional Accredited Program C Exist?

C Standard Program Participant Participant T G S

YES

A Support IPRM

C New ITCA Formed

T Adoption by ITCA I

Validation Testing s r o

d Develop Products Products Deployed n e V r o

t Development of i Assessment of ITCA

d Independent

e for IPRM

r Accreditation

c conformance

c Services A 190

Figure 2 – Smart Grid Product Testing Flow – Future Model 191

1921.4. Intended Audience 193 194The IPRM is the center of the SGIP’s testing and certification framework, and as 195such affects a broad range of stakeholders. The IPRM most directly affects and 196is operationalized by: 197  Interoperability Testing and Certification Authorities (ITCAs), 198  Certification Bodies (ISO/IEC Guide 65),

84______85 11 86 Smart Grid Testing And Conformity Committee 87 IPRM Version 2.0 – DRAFT – November 3, 2011 88 89 Interoperability Process Reference 90 Manual 91 (IPRM) 92 199  Test Laboratories (ISO/IEC 17025) and 200  Accreditation bodies. 201 System and device vendors are also key stakeholders both in how the IPRM af- 202 fects their products under test, and their internal test laboratory operations. 203 End users and buyers are the customers and should be able to expect interop- 204 erable products in the market place. In addition, the value of IPRM implemen- 205 tation is enhance where these customer understand the standards and associ- 206 ated certifications, and include specific interoperability and certification re- 207 quirements in RFPs. 208 209The IPRM defines the responsibilities within and among the ITCA, SSO (Stan- 210dards Setting Organizations), accreditors, test lab(s) and the certification body 211designed to bring interoperable standards based products to market. The IPRM 212implementation is facilitated through the use of check lists to ensure key areas 213are addressed. It is also important that ITCAs communicate well with end users 214in conveying the value of the certifications that they provide and how they may 215be used in assessing and achieving product interoperability. 216 217These responsibilities require detailed processes, which are left to the individu- 218al ITCA to define. However, the IPRM requires that product certification be is- 219sued by an independent ISO/IEC 65 accredited third party. This follows the logic 220promulgated in the two basic international management guidelines used as a 221foundation for the Framework: ISO/IEC 17025, General Requirements for the 222Competence of Testing and Calibration Laboratories, and ISO/IEC Guide 65, 223General Requirements for Bodies Operating Product Certification Systems.

93______94 12 95 Smart Grid Testing And Conformity Committee 96 IPRM Version 2.0 – DRAFT November 3, 2011 97 98 Interoperability Process Reference 99 Manual 100 (IPRM) 224

2252. International Guidelines for Testing and Certification 226 227The SGTCC extensively investigated and discussed the critical operational pro- 228cesses that independent certification bodies and laboratories need to imple- 229ment to instill end user confidence in interoperable products. It was quickly 230concluded that international standards, particularly ISO/IEC Guide 65, General 231Requirements for Bodies Operating Product Certification Systems, and ISO/IEC 23217025, General Requirements for the Competence of Testing and Calibration 233Laboratories, were so broadly used and supported, that adoption of these stan- 234dards for Smart Grid interoperability was a more prudent approach than devel- 235oping customized criteria that would simply parallel these accepted standards. 236Further, there is already a system of accrediting organizations and processes in 237place that support accreditations of testing and certification functions based on 238these international standards. 239

2402.1. Overview of ISO/IEC 17025 241 242ISO/IEC 17025 is focused on test laboratories and contains requirements that 243labs need to demonstrate that they operate a quality management system, are 244technically competent, and are able to generate technically valid results. It in- 245corporates all requirements of ISO 9001, Quality Management Systems – Re- 246quirements, that are relevant to testing services and facilitates acceptance of 247test results from accredited laboratories. Accreditation bodies apply these re- 248quirements in their laboratory assessments. 249 101______102 13 103 Smart Grid Testing And Conformity Committee 104 IPRM Version 2.0 – DRAFT – November 3, 2011 105 106 Interoperability Process Reference 107 Manual 108 (IPRM) 109 250ISO/IEC 17025 can be applied to any testing lab operation, whether indepen- 251dent (i.e. third-party) laboratories or in-house laboratories operated by manu- 252facturers for their own internal product testing. The advantage of applying 253ISO/IEC 17025 for Smart Grid testing operations is that many labs have already 254pursued and achieved compliance for selected aspects of the services they of- 255fer, and can simply expand their scope of accreditation to encompass new ser- 256vices necessary to support Smart Grid interoperability. This approach will build 257on common best practices used across the testing industry, speeding imple- 258mentation and avoiding unnecessary creation of redundant processes. 259 260ISO/IEC 17025 focuses on two major areas of laboratory operations: 261 1) Management requirements and 262 2) Technical requirements. 263 264The management requirements address issues such as a lab’s documented 265practices (i.e. both administrative and technical), impartiality of the lab in its 266operations, responsibilities for continuous improvement and issues resolution, 267and the active support and involvement of lab management in assuring com- 268mitment to complying with these criteria. 269The technical requirements focus on areas such as ensuring that lab staff are 270competent in performing their testing duties, assuring that the lab environ- 271ment is adequate for services performed, assuring that test plans and other 272necessary operating instructions are documented and available, and that nec- 273essary equipment and software used for testing is calibrated, maintained and 274appropriate for its intended usage. 275

110______111 14 112 Smart Grid Testing And Conformity Committee 113 IPRM Version 2.0 – DRAFT November 3, 2011 114 115 Interoperability Process Reference 116 Manual 117 (IPRM) 276The criteria described in ISO/IEC 17025 are extensive and the brief description 277above simply provides a high level view of some of the key elements that labs 278need to address in attaining accreditation. 279 280The technical scope of accreditation is specific to the selected tests / services 281for which the lab applies for evaluation. Evaluations for compliance can be 282performed by a number of different accrediting bodies, and there are global 283and regional agreements in place that provide for broad acceptance of an ac- 284creditation once attained. 285

2862.2. Overview of ISO/IEC Guide 65 287 288ISO/IEC Guide 65 is for certification bodies and parallels many of the same con- 289cepts applied in ISO/IEC 17025 test laboratories. There are general criteria 290that assure that the organization is non-exclusionary, open and without conflict 291of interest. Documented administrative policies and processes, as well as doc- 292umented technical requirements and specifications for certification, are among 293the required criteria. Criteria are also included to assure that procedures are in 294place to describe the granting of certifications, as well as ongoing mainte- 295nance, extensions and terminations of certifications once granted. Personnel 296qualifications are addressed for those involved in the evaluation and decision 297making process associated with the organization’s certifications. As in the 298case for ISO/IEC 17025, this is only a brief description of highlights associated 299with the more extensive criteria described in the document. 300 301While these international standard guidelines provide a solid foundation for the 302development and operation of high quality testing and certification programs, 118______119 15 120 Smart Grid Testing And Conformity Committee 121 IPRM Version 2.0 – DRAFT – November 3, 2011 122 123 Interoperability Process Reference 124 Manual 125 (IPRM) 126 303the SGTCC also recognized that additional technical requirements and best 304practices are necessary to help assure test program technical depth and suffi- 305ciency in meeting end user expectations for interoperability and cybersecurity. 306These supplemental criteria are described in subsequent sections of this docu- 307ment. 308 309 310 311

127______128 16 129 Smart Grid Testing And Conformity Committee 130 IPRM Version 2.0 – DRAFT November 3, 2011 131 132 Interoperability Process Reference 133 Manual 134 (IPRM) 312

3133. Overview of IPRM Version 2 314This section provides an overview of how this Interoperability Process Refer- 315ence Manual has changed in this latest version and the organization of the doc- 316ument. 317

3183.1. Changes from Version 1 319 Version 1 of the IPRM was released in January 2011. That version provided ex- 320tensive background information considered by the SGTCC in the assessment of 321the state of Smart Grid conformity and interoperability programs, and provided 322extensive information that would be valuable to readers that may have limited 323background in testing program models and approaches. 324 325The general goal of this revision was to enhance the utility of the document to 326support implementation of the criteria and recommendations by an ITCA and to 327structure it in a way to better facilitate assessments of ITCA implementation 328both internal to the ITCA and for external independent assessments. The 329changes in structure and clarity are major. The changes in content are minor. 330 331Fundamentally, Version 2 has an operational focus, while Version 1 provided an 332informational focus. Most of the key informative material from Version 1 has 333been retained in this update. The main body of the IPRM emphasizes the oper- 334ational aspects, while the informational material is provided in a series of sepa- 335rate informational annexes to the document. 336

135______136 17 137 Smart Grid Testing And Conformity Committee 138 IPRM Version 2.0 – DRAFT – November 3, 2011 139 140 Interoperability Process Reference 141 Manual 142 (IPRM) 143 337

3383.2. Organization of the IPRM 339 340The preceding sections provide the necessary background to understand the 341issues addressed by the IPRM, the SGTCC philosophy in addressing the issues, 342the objectives of the IPRM in solving the issue and the basic foundation of in- 343ternational standards upon which the IPRM is constructed. 344The subsequent sections of the IPRM address: 345 346  Application of the IPRM by ITCAs and the SGTCC role in shepherding im- 347 plementation

348  Background considerations on the various types of industry 349 certification/test programs

350  SGTCC recommendations for Interoperability Testing Best Practices be- 351 yond those identified in ISO/IEC Guide 65 and ISO/IEC 17025

352  Recommendations specific to Cybersecurity Testing

353  Detailed Tables of Criteria recommended for implementation by ITCAs

354 355 356 357 358

144______145 18 146 Smart Grid Testing And Conformity Committee 147 IPRM Version 2.0 – DRAFT November 3, 2011 148 149 Interoperability Process Reference 150 Manual 151 (IPRM) 359

3604. ITCA Implementation of the IPRM 361 362The SGTCC strongly advocates the implementation of an Interoperability Test- 363ing and Certification Authority (ITCA) organization to support each Smart Grid 364standard. 365 366The implementation of the IPRM by an ITCA is intended to accomplish several 367goals: 368 369- Increase the buyer’s confidence in the purchase of neutral 3rd party certified 370 interoperable products for their organizations over time

371- Standardize the testing and certification processes, through a set of best 372 practices, used for all smart grid products

373- Provide a basis for an approval process for those organizations following the 374 IPRM to assure the purchasing organizations of quality, audited testing pro- 375 grams. The SGTCC believes that end users purchasing Smart Grid products 376 based on standards will save money and shorten product implementation 377 cycle time by using products that comply with the SGIP TCC Framework.

378 379

3804.1. What is an ITCA? 381An Interoperability Testing and Certification Authority (ITCA) is the program 382management organization providing oversight for testing and certification ac- 152______153 19 154 Smart Grid Testing And Conformity Committee 155 IPRM Version 2.0 – DRAFT – November 3, 2011 156 157 Interoperability Process Reference 158 Manual 159 (IPRM) 160 383tivities associated with one or more standards or specifications, which takes re- 384sponsibility to insure that interoperable products within the scope of the specif- 385ic ITCA program are brought to market. The ITCA coordinates the participation 386of certification bodies and test labs for its program. 387 388An early finding of the SGTCC was that standards that had an associated ITCA 389engaged in test and certification of products to the standard were more rapidly 390implemented and adopted by the market place. 391 392The SGTCC has established the following required practices for ITCAs, certifica- 393tion bodies and test laboratories: 394 395  Certification bodies (CBs) should be accredited to ISO Guide 65, Gener- 396 al Requirements for Bodies Operating Product Certification Systems 397  Test laboratories should be accredited to ISO 17025, General Require- 398 ments for the Competence of Testing and Calibration Laboratories 399  The ITCA should have an agreement with an accrediting organization(s) 400 to assure that Certification Body and Test Lab accreditation is being per- 401 formed in accordance with the ITCA program scheme. 402  An ITCA should have a strong relationship with the SSO associated with 403 the standard for the purpose of feedback towards standard improve- 404 ment and clarification where there may be ambiguities 405 406The requirements for adherence of ITCAs to these internationally recognized in- 407dustry standards is consistent with the practices exhibited in other industry 408programs engaged in testing and certification activities related to critical infra-

161______162 20 163 Smart Grid Testing And Conformity Committee 164 IPRM Version 2.0 – DRAFT November 3, 2011 165 166 Interoperability Process Reference 167 Manual 168 (IPRM) 409structure (e.g. FCC programs for communications networks) and issues impact- 410ing personal safety and security (e.g. OSHA NRTL safety programs). 411

4124.2. How does an ITCA implement the IPRM? 413 414An ITCA begins its implementation of the IPRM and engagement with the 415SGTCC by declaring their intent to participate in the program and implement 416the IPRM recommendations in their program scheme. This step is formalized by 417completion of the ITCA Application Form located on the SGTCC Twiki site: 418 419http://collaborate.nist.gov/twikisggrid/bin/view/SmartGrid/SGTCCIPRMImple- 420mentation 421 422Submittal of the ITCA Application Form indicates that the ITCA has entered a 423transitional phase during which time they are actively engaged in integrating 424the IPRM recommendations in their program. The SGTCC recognizes that imple- 425mentation is a significant investment for the ITCA. The SGTCC will recognize 426those ITCAs that have committed to the program by acknowledging their par- 427ticipation on the SGTCC TWiki site. 428 429As an acknowledged participant in the program, the ITCA must be able to clear- 430ly demonstrate that they are progressing with their implementation activities in 431a timely manner. The SGTCC retains the right to remove an ITCA from the par- 432ticipant program where it concludes that the ITCA is not actively implementing 433the IPRM and/or fulfilling other obligations as described in the program partici- 434pation agreement.

169______170 21 171 Smart Grid Testing And Conformity Committee 172 IPRM Version 2.0 – DRAFT – November 3, 2011 173 174 Interoperability Process Reference 175 Manual 176 (IPRM) 177 435 436In the longer term, it is expected that an ITCA will fully implement the IPRM rec- 437ommendations, utilizing certification bodies that have been independently ac- 438credited to ISO Guide 65 and test laboratories that have been independently 439accredited to ISO 17025. At the time of release of this document, the SGTCC is 440actively engaged in dialogue with accreditation organizations to facilitate the 441availability of these assessment services in 2012, with the key IPRM recom- 442mendations integrated into the independent accreditation criteria.

178______179 22 180 Smart Grid Testing And Conformity Committee 181 IPRM Version 2.0 – DRAFT November 3, 2011 182 183 Interoperability Process Reference 184 Manual 185 (IPRM) 443

4444.3. Relationship between Accreditation Bodies, ITCAs, 445Certification Bodies and Test Labs 446 447The diagram below depicts the relationships across the key elements of testing 448and certification programs in the view of the SGTCC. ITCAs may be structured 449with subtending certification bodies and test labs dedicated to its mission. Al- 450ternatively, the ITCA may serve a dual role, acting itself as the certifying body. 451The SGTCC recognizes that flexibility is necessary in ITCA structure to accom- 452modate the diverse needs across the many technologies that are a part of the 453Smart Grid. While that flexibility and any necessary innovation are encouraged 454by the SGTCC, the one constant required is the need for checks and balances in 455the process such that certification decisions are made by personnel indepen- 456dent of those developing test data. 457

Relationship Diagram between ITCAs, Certification Bodies, Test Labs, Accreditors and the SGTCC

Accreditors

Provides

C guidelines, Recognizes ITCAs C

T practices for implementing

G ITCAs, CBs, recommendations S TLs

ITCA 1

A (Multiple standards C

T and/or certifications I addressed)

ITCA 2 (ITCA provides certification )

s function internally) B

C CBs accredited ( per ISO Guide s

r CB 1 CB 2 CB 3 65 and/or ITCA e i

f certification i

t scheme r e C

s Test Labs b

a accredited to L TL 1 TL 2 TL 3 TL 4 TL 5 ISO 17025 for t

s CB technical e

T scope

186______187 23 188 Smart Grid Testing And Conformity Committee 189 IPRM Version 2.0 – DRAFT – November 3, 2011 190 191 Interoperability Process Reference 192 Manual 193 (IPRM) 194 Figure 3 – Relationships between Accreditation Bodies, ITCAs, CBs, TLs, and SGTCC 458 459

4605. Best Practices for Interoperability and 461Conformance Test Construction 462 463This section provides best practices and guidelines for ITCAs in their develop- 464ment and operation of interoperability and conformance testing programs. The 465recommendations provided in this section were generated based on input from 466experienced testing organizations that have evolved interoperability and con- 467formance programs through lessons-learned in executing tests for both soft- 468ware and hardware applications. 469 470The recommendations may not apply directly to all testing applications; howev- 471er, they should be considered for interoperability and conformance test pro- 472grams as these practices have proven to be valuable in executing a broad 473cross-section of program types. Each ITCA should evaluate how these recom- 474mendations, observations and practices apply to their specific programs, and 475incorporate the recommendations into their programs where applicable. 476 477 5.1 General Test Policies

478 o Product vendors need to know if their products are eligible for 479 testing and certification, and how to prepare for certification. In 480 many cases, product vendors may be required to prepare specific 481 test environments (i.e. GUI applications to access low-level APIs, 482 test scripts, supported browsers, dedicated test hardware, sam-

195______196 24 197 Smart Grid Testing And Conformity Committee 198 IPRM Version 2.0 – DRAFT November 3, 2011 199 200 Interoperability Process Reference 201 Manual 202 (IPRM) 483 ples, etc.) in order to conduct testing of the standard and all un- 484 derlying software. Advanced knowledge of certification processes 485 helps set expectations of vendors to prepare a product for certifi- 486 cation.

487 o Final Test Reports should include at a minimum: 488 . Organization(s) that conducted the tests and location(s) 489 where the tests took place 490 . Test completion dates 491 . Product name / version / release tested 492 . Type of tests (i.e. interoperability or conformance) 493 . Test script version information 494 . Standards version information 495 . Technique(s) used for a test including standards and proce- 496 dures followed 497 . Test profile used or a list of test cases if a complete test 498 profile is not used 499 . Test equipment used, and all equipment traceability state- 500 ments.

501 o Some certification programs specify limits on the length a certifi- 502 cation may be considered valid. Criteria may include expiry dates, 503 and may be dependent on release of new standards or products.

504 o An interoperability testing program shall determine whether or 505 not interoperability has been demonstrated between all products 506 within the scope of the test. They must not assume conformance 507 testing achieves interoperability unless they have significant mar- 508 ket based historical evidence that their tested products are com- 509 pletely interoperable using conformance based methods. The 203______204 25 205 Smart Grid Testing And Conformity Committee 206 IPRM Version 2.0 – DRAFT – November 3, 2011 207 208 Interoperability Process Reference 209 Manual 210 (IPRM) 211 510 ITCA or certification body must not declare they have achieved in- 511 teroperability in the products unless one of the two points above 512 are qualitatively achieved and demonstrated.

513 o A certified interoperable product SHALL be conformant to the 514 standard unless full conformance causes interoperability issues. 515 In such cases, the issue should be reported back to and formally 516 recognized by the ITCA and/or the SSO so corrective action can be 517 taken.

518 o The level of Interoperability and Conformance testing is always a 519 trade-off between cost and test coverage. It is highly recom- 520 mended that the ITCA perform a cost-benefit analysis on the de- 521 gree of coverage associated with the test for both conformance 522 and interoperability against the cost to test. In determining the 523 test coverage, the security and safety concerns along with appro- 524 priate NERC / similar requirements should be considered para- 525 mount in determining the coverage assessment.

526 o Proper test tools produce reliable, repeatable and traceable test 527 results. Such tools require validation processes, test suites, tool 528 documentation, test reports, calibration certificates and other rel- 529 evant artifacts. The validation of the test tools must be per- 530 formed against a defined sample of software and / or hardware 531 implementations under test. Refer to ISO / IEC 17025 for more 532 detail on the use of qualified and calibrated test tools. 533 534 5.2 Test Suite Specification (TSS)

535

212______213 26 214 Smart Grid Testing And Conformity Committee 215 IPRM Version 2.0 – DRAFT November 3, 2011 216 217 Interoperability Process Reference 218 Manual 219 (IPRM) 536 o A common TSS should be established when one or more test labs 537 are deployed to test the same standard and / or profile. If unique 538 test procedures are required to support a test suite, then they 539 should also be defined.

540 o The TSS should be test tool agnostic.

541 o The TSS should be subject to revision control including revision 542 history, revision numbering, and a defect / expansion manage- 543 ment process. The TSS should clearly identify the test purpose, 544 references, resource requirements, test setup, procedures, ob- 545 servable results and possible problems / lessons learned with the 546 test approach. Observables should clearly identify pass / fail / in- 547 determinate requirements and informational elements.

548 o The TSS should clearly define any conventions that will be re- 549 quired to achieve interoperability.

550 o The TSS should restrict cardinality and define the exact attributes 551 and associations required for interoperability.

552 o The TSS should remove or clarify all ambiguities and any areas of 553 the standard that may be interpreted differently between two or 554 more interoperable systems.

555 o The TSS should be a standard and managed as such by an ITCA or 556 SSO. The documentation should include scope, date of issue, revi- 557 sion, change control, and methods to feedback implementer’s re- 558 sults.

559 o The TSS should have accompanying tools to validate data and 560 data structures contained in, or produced by, the test.

561 o Test cases should have clear mappings to feature-sets, use-cases, 562 and requirements. 220______221 27 222 Smart Grid Testing And Conformity Committee 223 IPRM Version 2.0 – DRAFT – November 3, 2011 224 225 Interoperability Process Reference 226 Manual 227 (IPRM) 228 563 o An interoperability testing program shall determine whether or 564 not interoperability has been demonstrated between all products 565 within the scope of the test. The TSS should ensure all areas of 566 the interoperability and conformance testing are sufficiently de- 567 fined and documented such that the test can be repeated.

568 o The TSS should define the test data required to execute the test 569 cases. The TSS should define any test stub required to execute 570 messages that will generate negative responses.

571 o The TSS should identify interoperability issues arising from ambi- 572 guities in the standard, and establish requirements sufficient to 573 prevent those interoperability issues. 574 5.3 Attributes of a Test Profile in lieu of complete TSS

575 576 o Must be a subset of the TSS

577 o Specifies mandatory and optional elements

578 o Specifies all restrictions

579 o Cannot add to the standard, but can only restrict the standard

580 o Define the type of profile (i.e. message, model or implementation) 581 and provide a name for the profile that clearly defines the objec- 582 tive / scope of the profile and the use-cases it is designed to test

583 o Is a companion standard or is submitted to the SSO for progres- 584 sion as a companion standard 585

229______230 28 231 Smart Grid Testing And Conformity Committee 232 IPRM Version 2.0 – DRAFT November 3, 2011 233 234 Interoperability Process Reference 235 Manual 236 (IPRM) 5866. Cybersecurity Testing

5876.1. Introduction

588Most product vendors make a claim as to functionality and/or offered 589cybersecurity controls. Organizations need to have a minimum level of 590assurance that a product's stated function and cybersecurity claim is valid. 591Confidence in a product’s cybersecurity can be based on an impartial cyberse- 592curity evaluation, which includes an analysis of the product and the testing of 593the product for conformance to a set of cybersecurity requirements and con- 594trols. The use of consistent standardized cybersecurity evaluation criteria and 595methodologies contributes to the repeatability and objectivity of the results. 596Cybersecurity testing should be performed in conjunction with interoperability 597tests. As functionality is developed to enable interoperability, new potential 598vulnerabilities could be discovered. By ensuring cybersecurity testing is con- 599ducted with interoperability tests the following objectives are met:

600  Uncover design, implementation and operational flaws that could allow 601 the violation of cybersecurity requirements and controls;

602  Determine the adequacy of cybersecurity mechanisms, assurances and 603 other properties to enforce the cybersecurity requirements and controls;

604  Assess the degree of consistency between the cybersecurity require- 605 ments and controls and their implementation;

606  Identify and locate loopholes that can cause loss of important informa- 607 tion, function, or allow unauthorized access; and

237______238 29 239 Smart Grid Testing And Conformity Committee 240 IPRM Version 2.0 – DRAFT – November 3, 2011 241 242 Interoperability Process Reference 243 Manual 244 (IPRM) 245 608  Identify the cyber security functionality that could impact the interoper- 609 ability of components and systems.

6106.2. Cybersecurity Concepts 611The primary objective of cybersecurity testing is to validate the cybersecurity 612claims. Testing will determine if the product functions as intended and con- 613forms to a defined set of cybersecurity requirements. At a high level, this 614means providing confidentiality, integrity (authenticity and non-repudiation), 615and availability for the information stored on the product and system into 616which the product is integrated..

6176.3. Cybersecurity Testing Environment 618A critical part of conducting cybersecurity testing is the environment in which 619the products are tested. Products should be tested in an environment that 620closely simulates the intended implementation of the product. Cybersecurity 621testing should conform to identified environmental standards and specifica- 622tions. An internal cybersecurity policy should be developed for the product be- 623ing tested which provides the cybersecurity rules under which an implementa- 624tion or cybersecurity testing environment must operate. In order to understand 625the security of an implementation being tested, the rules and assumptions 626about how the implementation must operate need to be specified in particular 627for mitigations and controls outside the boundary of the implementation. Envi- 628ronmental conditions, particularly when outside the normal operating condi- 629tions of a component or system, could cause a potential critical cybersecurity 630breach, so they need to be considered during testing. For example, in some 631extreme temperatures, humidity and radiations, authentication devices can fail 632even when these conditions are not maliciously induced.

246______247 30 248 Smart Grid Testing And Conformity Committee 249 IPRM Version 2.0 – DRAFT November 3, 2011 250 251 Interoperability Process Reference 252 Manual 253 (IPRM) 6336.4. Cybersecurity Testing Types and Frameworks 634The type of cybersecurity tests conducted on a product will depend on the type 635of product, its functionality, operating environment, and both functional and 636non-functional cybersecurity requirements. When intertwined with interoper- 637ability testing, cybersecurity specific test methodologies should be document- 638ed. The interoperability testing results should remain intact and verified after 639security fixes are applied to address issues found during cybersecurity testing. 640Different testing methodologies are used to determine attributes (interoperabil- 641ity, conformance, usability, security, etc.) of an implementation being tested. 642Multiple types of testing techniques should be used, including manual review, 643fuzz testing, static analysis, dynamic analysis, and penetration testing.

644In general, Smart Grid security testing/assessment approaches can be broken 645into the following major categories:

646  Security Conformance/Certification Testing is usually initiated by 647 each vendor, to verify that a vendor product meets an industry estab- 648 lished level of security, and tested by independent third-party labs using 649 the same testing procedures that are pass/fail in nature. Results of 650 these tests are often published and used in vendor marketing cam- 651 paigns.

652  Security Interoperability Testing measures the interoperability of se- 653 curity components between devices and systems from different vendors.

654  Security Architecture Review is a form of paper and pencil exercise 655 to discuss the configurations and security posture of a particular system, 656 which often includes not only that system but how that system is con- 657 nected to other systems and devices. This type of test can also include

254______255 31 256 Smart Grid Testing And Conformity Committee 257 IPRM Version 2.0 – DRAFT – November 3, 2011 258 259 Interoperability Process Reference 260 Manual 261 (IPRM) 262 658 policy review and how it affects the organization and systems the poli- 659 cies govern.

660  Vulnerability Assessments is a form of assessment that uses tools to 661 find known vulnerabilities in systems by analyzing the system version, 662 retrieving its configuration, and comparing them to the vulnerability 663 databases leveraged by the tools. Utility security departments or a con- 664 tracted specialized security firm they hire usually perform this type of 665 assessment.

666  Penetration Testing is a specialized form of assessment where the 667 testing team takes on the role of the attacker and tries to find and ex- 668 ploit vulnerabilities in systems and devices. Testers use the same 669 methodology that attackers often use to identify vulnerabilities in the 670 system. Once a vulnerability is found, the testers attempt to exploit the 671 flaw to gain a foothold in the system and begin the process again to dis- 672 cover additional, lower level vulnerabilities that weren’t previously ex- 673 posed. Penetration testing is distinguished from vulnerability assess- 674 ments by the fact it tests the depth of vulnerabilities instead of simply 675 breadth, focus on discovering both known and unknown vulnerabilities, 676 and provide the testing team with a better understanding of a vulnera- 677 bility’s risk to the business.

678  Security Code Review is a form of assessment performed to identify 679 security flaws in the source code of systems and devices. Vendors usu- 680 ally perform this type of assessment since system source code is not 681 usually provided to the customer. However, in some instances a vendor

263______264 32 265 Smart Grid Testing And Conformity Committee 266 IPRM Version 2.0 – DRAFT November 3, 2011 267 268 Interoperability Process Reference 269 Manual 270 (IPRM) 682 may provide the source code to a third-party assessment laboratory, if a 683 non-disclosure agreement is in place.

684  Compliance Review is a form of assessment that focuses on the exis- 685 tence and execution of security policy. This can be performed to fulfill a 686 formal regulatory requirement such as NERC CIP or an informal, internal 687 assessment to determine compliance with such guidance documents as 688 the NISTIR 7628, security profiles from the Advanced Security Accelera- 689 tion Project (ASAP-SG), or other non-regulatory documents.

690  Verification and Validation (V&V) Testing or Final Acceptance 691 Testing (FAT) is a final step of an electric utilities’ purchasing and de- 692 ployment process, to confirm that the system meets business require- 693 ments and fulfills its intended purpose. This type of assessment can in- 694 clude one or more of the assessments listed above, including penetra- 695 tion testing.

696Each cybersecurity testing laboratory should follow a documented cybersecuri- 697ty testing framework or methodology to perform all of their cybersecurity test- 698ing. Some of the most common cybersecurity testing frameworks are:

699  The Common Criteria for Information Technology Security Evalu- 700 ation (Common Criteria or CC) is an international standard (ISO/IEC 701 15408) for computer security certification. Common Criteria is a frame- 702 work in which computer system users can specify their security function- 703 al and assurance requirements, vendors can then implement and/or 704 make claims about the security attributes of their products, and testing 705 laboratories can evaluate the products to determine if they actually 706 meet the claims.

271______272 33 273 Smart Grid Testing And Conformity Committee 274 IPRM Version 2.0 – DRAFT – November 3, 2011 275 276 Interoperability Process Reference 277 Manual 278 (IPRM) 279 707 http://www.commoncriteriaportal.org/. 708 709  The Information Systems Security Assessment Framework (IS- 710 SAF) testing methodology is designed to evaluate network, system and 711 application controls. 712 http://www.oissg.org/issaf

713  NIST Special Publication 800-53A, Guide for Assessing the Secu- 714 rity Controls in Federal Information Systems and Organizations 715 is written to facilitate security control assessments conducted within an 716 effective risk management framework. Special Publication 800-53A is a 717 companion guideline to Special Publication 800-53, Recommended Se- 718 curity Controls for Federal Information Systems and Organizations. 719 http://csrc.nist.gov/publications/nistpubs/800-53A-rev1/sp800-53A-rev1- 720 final.pdf.

721  The Open Source Security Testing Methodology Manual (OSST- 722 MM) provides a scientific methodology for the accurate characterization 723 of operational security (OpSec) through examination and correlation of 724 test results in a consistent and reliable way. This manual is adaptable to 725 almost any audit type, including penetration tests, ethical hacking, secu- 726 rity testing, vulnerability testing, red-teaming, blue-teaming, and so 727 forth. It is written as a security research document and is designed for 728 factual security verification and presentation of metrics on a profession- 729 al level. 730 http://www.isecom.org/mirror/OSSTMM.3.pdf

280______281 34 282 Smart Grid Testing And Conformity Committee 283 IPRM Version 2.0 – DRAFT November 3, 2011 284 285 Interoperability Process Reference 286 Manual 287 (IPRM) 731  The Penetration Testing Execution Standard (PTES) is a standard 732 designed to provide both businesses and security service providers with 733 a common language and scope for performing penetration testing. The 734 standard defines a baseline for the minimum effort required for a basic 735 pentest, and includes several higher "levels" to provide more compre- 736 hensive activities for organizations with higher security needs. 737 http://www.pentest-standard.org/index.php/PTES_Technical_Guidelines

738  NIST SP800-115, Technical Guide to Information Security Testing 739 and Assessment is a guide to the basic technical aspects of conduct- 740 ing information security assessments. It presents technical testing and 741 examination methods and techniques that an organization might use as 742 part of an assessment, and offers insights to assessors on their execu- 743 tion and the potential impact they may have on systems and networks. 744 http://csrc.nist.gov/publications/nistpubs/800-115/SP800-115.pdf.

745  NIST SP800-142, Practical Combinatorial Testing is a method that 746 can reduce cost and increase the effectiveness of software testing for 747 many applications by testing combinations of parameters that provides 748 highly effective fault detection. 749 http://csrc.nist.gov/groups/SNS/acts/documents/SP800-142-101006.pdf.

7506.5. Additional Cybersecurity Testing Laboratory Best 751Practices to Consider 752Along with the cybersecurity testing concepts, environments, and framework 753considerations described above, the following provides additional cybersecurity 754testing best practices for a cybersecurity testing laboratory to consider:

288______289 35 290 Smart Grid Testing And Conformity Committee 291 IPRM Version 2.0 – DRAFT – November 3, 2011 292 293 Interoperability Process Reference 294 Manual 295 (IPRM) 296 755  The cybersecurity test team members should subscribe to a specific 756 code of ethics. An organization like the National Board of Information 757 Security Examiners (NBISE), https://www.nbise.org, helps to validate 758 hands-on skills and knowledge in order to reliably predict an individual’s 759 future performance and aptitude for cybersecurity.

760  A risk and threat analysis should be provided for the item to be tested. 761 A risk and threat analysis provides information to the test team about 762 actions that may be harmful to an implementation and the sources of 763 the actions. In order to understand the security of the implementation 764 being tested, both the threats an implementation is expected to address 765 as well as the threats that are not addressed need to be known. Particu- 766 larly threats that are intended to be addressed by mitigations external to 767 the implementation need to be documented to properly scope and 768 bound the test to be performed.

769  Existing cybersecurity test programs should be leveraged when possi- 770 ble. Integrating existing cybersecurity test programs into a cybersecuri- 771 ty testing program allows for the expertise and experience from other 772 cybersecurity testing domains to be applied. However, understanding 773 the limits of existing cybersecurity testing programs is required in order 774 to integrate those programs appropriately. By leveraging existing cyber- 775 security testing programs, it may reduce the time and cost of the testing 776 process.

777  Component cybersecurity testing should be included in cybersecurity 778 test programs when possible. When a software or firmware component 779 contains vulnerabilities and is reused throughout a product line, those

297______298 36 299 Smart Grid Testing And Conformity Committee 300 IPRM Version 2.0 – DRAFT November 3, 2011 301 302 Interoperability Process Reference 303 Manual 304 (IPRM) 780 vulnerabilities could be within multiple products. Testing at the compo- 781 nent level ensures that the layered cybersecurity within a product con- 782 tinues to protect the information and processing. Components often re- 783 quire re-interoperability testing when used in a different configuration to 784 ensure proper setup and use.

785  Cybersecurity testing programs should ensure they align with the busi- 786 ness and technical requirements for the enterprise, unit, product, etc. 787 When cybersecurity testing programs align with business, system, and 788 technical requirements, risk can be limited and the product will have cy- 789 bersecurity designed into the components and systems, so it can be part 790 of the core testing and not an add-on “feature” to be tested.

7916.6. Role of the Cybersecurity Testing Providers 792Cybersecurity testing is typically conducted by information system developers, 793system integrators, certification agents, information system owners, auditors, 794inspectors general, and the information security staffs. Cybersecurity testing 795services, whether provided by an element within the customer’s organization 796or a contracted public or private sector entity should demonstrate a variety of 797different capabilities. The cybersecurity testing provider should have a man- 798agement structure that provides the framework for testing teams to conduct 799effective cybersecurity testing. The cybersecurity testing provider should be 800able to perform administrative functions to support the testing teams, protect 801the information received from the customer, and develop and implement stan- 802dard procedures to ensure that all testing teams provide consistent, reliable 803and repeatable testing services. Cybersecurity testing providers assemble ap- 804propriate individuals to make up a testing team. The team members that work 805together to prepare for, conduct, and document the findings of the testing. 305______306 37 307 Smart Grid Testing And Conformity Committee 308 IPRM Version 2.0 – DRAFT – November 3, 2011 309 310 Interoperability Process Reference 311 Manual 312 (IPRM) 313 806Each team is made up of individuals that should collectively have the knowl- 807edge, skills, and abilities to conduct the cybersecurity testing.

808

809Organizational management capability for a cybersecurity testing provider is 810the implementation of a management system that sufficiently enables a ser- 811vice provider to manage, plan, conduct, and assure quality of cybersecurity 812testing services. The cybersecurity testing provider should have an operational 813and management system in place in order to effectively manage and conduct 814cybersecurity testing. The management system framework should include: (i) 815maintaining independence to prevent a conflict of interest when conducting 816the cybersecurity testing; (ii) implementing an effective management structure 817that provides both technical oversight and administrative support; (iii) provid- 818ing the resources to select an effective testing team with the knowledge, skills, 819and abilities to conduct the testing based on the customer’s product’s cyberse- 820curity claims; (iv) protecting customer information collected during the cyber- 821security testing process; and (v) implementing or creating tools, templates, 822and standard cybersecurity testing procedures to ensure each testing team 823provides consistent service. Customers should be confident that the cybersecu- 824rity testing provider’s cybersecurity testing teams have the knowledge, experi- 825ence, and resources to conduct an effective cybersecurity test.

826

314______315 38 316 Smart Grid Testing And Conformity Committee 317 IPRM Version 2.0 – DRAFT November 3, 2011 318 319 Interoperability Process Reference 320 Manual 321 (IPRM) 827

8287. Interoperability Testing and Certification Authority 829Role and Requirements 830 831The ITCA SHALL provide governance and coordination for the maintenance and 832administration of Interoperability Testing Laboratories and Certification Bodies 833in the implementation of testing and certification activities associated with 834specified standards. The ITCA is also expected to cooperate with relevant SSOs 835and user groups for continuous improvement of the program and the standard 836being addressed. 837 838An ITCA SHALL manage the end-to-end processes associated with interoperabil- 839ity testing and certification. It is expected that the ITCA has the appropriate in- 840frastructure in place to support this function. Where a new ITCA is being 841launched in support of a standard, establishment of the following is recom- 842mended: 843 844  Business plan 845  Clear governance structure and IPR policy 846  Testing lab(s) 847  Certification body / bodies 848  Security certificate authority 849 850The following information SHALL be used by the ITCA to improve the interoper- 851ability and cybersecurity of a Smart Grid standards based product.

322______323 39 324 Smart Grid Testing And Conformity Committee 325 IPRM Version 2.0 – DRAFT – November 3, 2011 326 327 Interoperability Process Reference 328 Manual 329 (IPRM) 330 8527.1. Interoperability Requirements for Use by the ITCA 853 The interoperability requirements are comprised of five major categories 854 which will be used by the ITCA to effectively manage the testing and cer- 855 tification organization. Note: Many of the below items are more formally 856 defined in the ISO 65 and 17025 standards. 857 858 The five major categories are: 859  Governance 860  Lab Qualification 861  Technical Design 862  Improvement 863  Security 864 865 The IPRM requirements are written with the key word “SHALL”. However, 866 depending on the ITCA program and relevant standard under considera- 867 tion, some requirements may not be applicable for all situations. 868. 869

8707.2. Governance 871 Governance defines the structures, policies, rules and regulations asso- 872 ciated with the ITCA certification program. A governance process exam- 873 ple would require the ITCA to establish and maintain an independent and 874 vendor neutral testing and certification oversight authority. The follow- 875 ing list of Interoperability Governance Process Requirements provided in 876 Table 1 SHALL be considered governance process requirements for man- 877 aging the interoperability testing and certification programs. 331______332 40 333 Smart Grid Testing And Conformity Committee 334 IPRM Version 2.0 – DRAFT November 3, 2011 335 336 Interoperability Process Reference 337 Manual 338 (IPRM) 878 879 880 Gov- Interoperability Governance Process ern-x Requirement Description Gov-1 An ITCA Certification Program SHALL clearly identify the Standard(s) to which testing or certifications are assessed. The ITCA SHALL provide oversight to provide confidence that implementations of Standard(s) in certified products are indeed interoperable.

Gov-2 DELETED Gov-3 DELETED Gov-4 If an ITCA permits first party testing, the ITCA SHALL clearly define the circumstances under which such testing may be submitted to a certification body and the duties of the certification body in determining the suitability of first party test data. Gov-5 If an ITCA requires third party testing, the ITCA SHALL clearly identify the circumstances under which such testing SHALL be submitted to a certification body and the duties of the certification body in reviewing the third party test data. Gov-6

339______340 41 341 Smart Grid Testing And Conformity Committee 342 IPRM Version 2.0 – DRAFT – November 3, 2011 343 344 Interoperability Process Reference 345 Manual 346 (IPRM) 347 Gov-7 The ITCA SHALL define roles, responsibilities, and resource elements of the interoperability program in documented Certification Program requirements.. Gov-8 The ITCA SHALL specify method(s) for reporting issues to appropriate parties for resolving certification difficulties (vague or inconsistent specifications or test methods, incompatibility with similarly certified products, etc.). Gov-9 The ITCA SHALL maintain a certified product and systems list. This list SHALL be publicly available. Gov-10 The ITCA SHALL maintain a test case reference and modification history list.2 Gov-11 Test Suite Specifications (TSS)3 used for interoperability or conformance testing SHALL be managed in a well-defined, open and formal manner with change control. Gov-12 A common TSS SHALL be established when multiple test labs are deployed to test the same standard and / or profile. If common unique test procedures are required to support this test suite, then they SHALL also be defined. The TSS should be test tool agnostic. Gov-13 All certification bodies operating under the ITCA Cer- tification Program SHALL be accredited as meeting ISO/IEC Guide 65. The accreditation scope SHALL in-

3481 The ITCA should use best efforts in contacting a standards body with respect to a specifica- 349tion; however, it is not their responsibility to resolve issues with the specification. 3502 See Glossary of Terms for definition and explanation of the test case reference list. 3513 See Glossary of Terms for definition and explanation of the TSS. 352______353 42 354 Smart Grid Testing And Conformity Committee 355 IPRM Version 2.0 – DRAFT November 3, 2011 356 357 Interoperability Process Reference 358 Manual 359 (IPRM) clude the International Classification for Standards (ICS) Codes applicable to the technologies for which certification activities are performed. Accreditation SHALL be by an accreditation body that is signatory, in good standing, to the International Accreditation Forum (IAF) multilateral agreement for “Product.” Gov-14 DELETED. Gov-15 If an ITCA has multiple testing laboratories and certifying bodies, processes SHALL be in place to avoid quality differences and assure repeatable testing between the laboratories. Gov-16 DELETED Gov-17 The ITCA SHALL ensure that the test labs and certification bodies maintain their accreditation for participation in the Certification Program. 881 Table 1 – Interoperability Process Governance Requirements

8827.3. Lab Qualification 883 Lab qualification defines the requirements in Table 2 that SHALL be ap- 884 plied by ITCAs when recognizing testing laboratories. It should be noted 885 that additional requirements are further detailed in ISO 17025. 886 Lab-x Interoperability Lab Qualification Process Requirement Description Lab-1 In selecting test organizations, the ITCA SHALL have uniform and transparent procedures for evaluating test labs. Lab-2 The ITCA SHALL define requirements to qualify the personnel involved in the certification and testing processes. 360______361 43 362 Smart Grid Testing And Conformity Committee 363 IPRM Version 2.0 – DRAFT – November 3, 2011 364 365 Interoperability Process Reference 366 Manual 367 (IPRM) 368 Lab-3 The ITCA SHALL require that its test labs be accredited to ISO 17025. The accreditation scope SHALL include the specific standards or specifications against which testing may be performed. Accreditation SHALL be by an accreditation body that is (a) a signatory, in good standing, to the International Labora- tory Accreditation Cooperation (ILAC) mutual recog- nition arrangement; or (b) recognized under the (US) National Cooperation for Laboratory Accreditation (NACLA). Lab-4 DELETED 887 Table 2 – Interoperability Lab Qualification Process Requirements 888

8897.4. Technical Design for Interoperability and Conformance 890Program Design 891 892 The Technical Design for Interoperability and Conformance Program De- 893 sign defines the requirements needed to effectively manage the proce- 894 dures and processes associated with interoperability and conformance 895 testing. 896 897 Tech-x Interoperability Technical Design Process Cate- Requirements Description gory Tech-1 Techni- The ITCA SHALL specify in the TSS those fea- cal tures that are mandatory, and those features

369______370 44 371 Smart Grid Testing And Conformity Committee 372 IPRM Version 2.0 – DRAFT November 3, 2011 373 374 Interoperability Process Reference 375 Manual 376 (IPRM) that are optional. Tech-2 Techni- The ITCA SHALL require and enforce that ven- cal dors declare the optional features implemented in a product. Tech-3 Techni- The ITCA SHALL require that implementations cal of optional features be tested and certified for conformance and inter-operability. Further- more, the ITCA SHALL define common test cases for optional features to be used by all test labs. Tech-4 Techni- Separate digital certificates SHALL be associ- cal ated with products implementing optional features.

Tech-5 Techni- An ITCA SHALL define the record handling and cal retention requirements to be followed by the TL and CB functions, consistent with requirements of ISO 17025 and ISO Guide 65. Tech-6 Inheri- The ITCA SHALL specify the conditions under tance which it will allow for sub-component (e.g., previously certified hardware modules used in developing final products, previously certified software components with well-defined interfaces and dependencies etc.) inheritance in development of final products. However, it is the ITCAs responsibility to ensure that interoperability is maintained. Tech-7 Inheri- The ITCA SHALL maintain a controlled list of tance compatible sub-components that can be inher- 377______378 45 379 Smart Grid Testing And Conformity Committee 380 IPRM Version 2.0 – DRAFT – November 3, 2011 381 382 Interoperability Process Reference 383 Manual 384 (IPRM) 385 ited to build final products. This might include specifying compatible feature-sets. Tech-8 Inheri- When supporting products composed of sub- tance components, the ITCA SHALL define the set of additional tests necessary to ensure interoperability (e.g. integration testing, final performance testing, etc.) Tech-9 Inheri- The ITCA SHALL implement a Compliant Por- tance tion Description (CPD)4 to be used as a guide for assembling a product based on compatible sub-components. Tech- Version The ITCA SHALL have an explicit process in 10 Control place to assess necessity of re-certification against subsequent release versions of a specification, including security. Tech- Version The ITCA SHALL define the level of re-certifica- 11 Control tion required for subsequent release versions of a specification. Tech- Version The ITCA SHALL define a mechanism to identi- 14 Control fy the latest version of a previously certified product or system implementation. This is important in cases where a previously certified product or system has been upgraded to a different version. Tech- Version The ITCA SHALL have a mechanism to enforce 15 Control version control rules such that each product certification clearly identifies the SSO docu-

3864 See Glossary of Terms for definition and further explanation of CPD 387______388 46 389 Smart Grid Testing And Conformity Committee 390 IPRM Version 2.0 – DRAFT November 3, 2011 391 392 Interoperability Process Reference 393 Manual 394 (IPRM) ment and version to which product is certified. Tech- Testing The testing and certification program SHALL 16 - Gen- have common well-defined standardized test eral cases. These test cases should be defined in an open, consensus-driven fashion. These test cases will be used by all test labs approved by the ITCA. Tech- Testing There SHALL be a defined correlation between 17 – Gen- implementations and required testing, com- eral monly called a Proforma Implementation Con- formance Statement (PICS).5 Tech- Testing The testing and certification program SHALL 18 -Gener- maintain a current and upcoming list of appli- al cable test cases to be called a Test Case Refer- ence List. Tech- Testing There SHALL be a Test Plan derived from the 19 – Gen- Test Case Reference List and used by all autho- eral rized test labs. Tests SHALL be identified using the test plan. Tech- Testing The testing and certification program SHALL 20 – Gen- require that a static conformance review6 take eral place prior to testing a product. Tech- Testing The testing and certification program SHALL 21 – Gen- implement validated test tools. Golden refer- eral ence test equipment may be utilized where appropriate.

3955 PICS can be referred as both Protocol Implementation Conformance Statement and Profile Im- 396plementation Conformance Statement. Proforma is being used in this requirement to reference 397both concepts. 3986 See Glossary of Terms for the definition and explanation of a static conformance review. 399______400 47 401 Smart Grid Testing And Conformity Committee 402 IPRM Version 2.0 – DRAFT – November 3, 2011 403 404 Interoperability Process Reference 405 Manual 406 (IPRM) 407 Tech- Testing The TSS SHALL be subject to revision control, 22 – Gen- including revision history, revision numbering, eral and a defect and expansion management process. The TSS should clearly identify the test purpose, references, resource requirements, test setup, procedures, observable results and possible problems / lessons learned with the test approach. Observables should clearly identify pass / fail / indeterminate requirements and informational elements. Tech- Testing The testing and certification program SHALL 23 - Con- assure that defined product test cases cover for- application profiles for specific feature sets mance and functions defined by the specific application profile, and implement interoperability evaluation within that application profile. Tech- Testing Where practicable, the testing and certifica- 24 – Con- tion program SHALL assure that defined prod- for- uct test cases cover all feature sets and func- mance tions. Tech- Testing DELETED 25 – Con- formance Tech- Testing DELETED 26 – Con- for-

408______409 48 410 Smart Grid Testing And Conformity Committee 411 IPRM Version 2.0 – DRAFT November 3, 2011 412 413 Interoperability Process Reference 414 Manual 415 (IPRM) mance Tech- Testing The testing and certification program SHALL 27 – Prod- assure that defined product use cases are cov- uct In- ered in application profiles. Interoperability terop- testing and evaluation SHALL be implemented erabili- within those application profiles. ty Tech- Testing The testing and certification program SHALL 28 – Prod- classify common or major market products ac- uct In- cording to their application profiles, and interop- clude them as part of an interoperability eval- erabili- uation for those specific profiles. The evalua- ty tion SHALL make use of test profiles correlated to those specific applications.7 Tech- Testing The testing and certification program SHALL 29 – Prod- ensure that venues are provided for multi-ven- uct In- dor and multi-product communication and interop- terchange evaluations (e.g. “plug fests”). This erabili- program may be optional for ITCAs correlated ty to standards resulting in application interfaces and not a physical product Tech- Prototyping of draft standards or major revi- 30 sions SHALL be supported via multi-vendor / multi-product testing. The ITCA SHALL solicit for the prototyping of draft standards or major revisions, and organize multi-vendor / multi-

4167 Interoperability testing is tied to market realities. Hence the testing and certification program 417needs to have a mechanism to adopt representative market products as an integral part of in- 418teroperability testing. 419______420 49 421 Smart Grid Testing And Conformity Committee 422 IPRM Version 2.0 – DRAFT – November 3, 2011 423 424 Interoperability Process Reference 425 Manual 426 (IPRM) 427 -product testing. It is recommended that the prototyping take place in the late stages of standards development in order to verify the correctness of the standard, verify the test suites and verify that the anticipated interoperability or conformance testing is debugged. Tech- Testing The ITCA SHALL have a process to select a 31 – Prod- minimum of two distinct reference implemen- uct In- tations as golden implementations or golden terop- units. The selection is usually based on the re- erabili- sults of the interoperability testing. All other ty implementations SHALL be tested against these golden implementations.8 Tech- Testing The ITCA SHALL make appropriate provisions 32 – Prod- for the use of golden implementations in the uct In- testing and certification program to strengthen terop- consistent and standard implementation and erabili- interoperability testing and certification pro- ty cesses. Tech- Testing The golden implementations or golden units 33 – Prod- SHALL be clearly associated with each version uct In- of the standard. Each golden unit is a snap terop- shot (instantiation) of each version of the stan- erabili- dard. ty Tech- Testing The testing and certification program SHALL

4288 The industry prefers three golden units for product testing, but the minimum number of gold- 429en units shall be no less than two golden units. 430______431 50 432 Smart Grid Testing And Conformity Committee 433 IPRM Version 2.0 – DRAFT November 3, 2011 434 435 Interoperability Process Reference 436 Manual 437 (IPRM) 34 – Prod- ensure that critical vendor implementations be uct In- made available to the labs as golden imple- terop- mentations. erability Tech- Testing . DELETED 35 – Prod- uct In- terop- erabili- ty Tech- Testing If an ITCA Certification Program involves 36 – Sys- multiple Smart Grid systems, then the tem In- Program Requirements SHALL support end-to- terop- end testing of Smart Grid systems involving erabili- multiple product implementations. ty Tech- Testing An ITCA SHALL involve all relevant parties to 37 – Sys- define various business logic models for the tem In- end-to-end system testing, and make scenar- terop- ios and test harness systems available for erabili- testing. ty Tech- Testing The testing and certification program SHALL 38 - Perfor- ensure that when functional performance re- mance quirements are defined in an application profile, the performance test profile(s) SHALL be designed to implement test cases for evaluat-

438______439 51 440 Smart Grid Testing And Conformity Committee 441 IPRM Version 2.0 – DRAFT – November 3, 2011 442 443 Interoperability Process Reference 444 Manual 445 (IPRM) 446 ing these requirements. Tech- Testing DELETED 39 – Per- formance Tech- Tools DELETED 40 Tech- Tools The ITCA SHALL ensure that test tools have a 41 complete mandatory feature-set coverage of a standard. In cases where two or more implementations of optional features are available, the ITCA SHALL incorporate those feature-sets in the test tool.9 Tech- Tools The ITCA SHALL define procedures and pro- 42 cesses to validate the use of test tools and reference implementations. Tech- Techni- DELETED 43 cal Lead Tech- Techni- DELETED 44 cal Lead Tech- Techni- DELETED 45 cal Lead

4479 Effective test tools need to be able to test all features and functions of a standard. Some fea- 448tures of a standard may never be supported by certain products; however when a standard is 449published, the industry is free to implement optional feature set in addition to the mandatory 450set; lack of testing capability of optional feature sets hinders interoperable feature set intro- 451duction. Normally, validated test tools have implementations of all features, including optional 452ones as a condition for the tool validation. 453______454 52 455 Smart Grid Testing And Conformity Committee 456 IPRM Version 2.0 – DRAFT November 3, 2011 457 458 Interoperability Process Reference 459 Manual 460 (IPRM) Tech- Techni- DELETED 46 cal Lead Tech – ITCA ITCA shall develop criteria for surveillance to 47 be carried out by its certification bodies. 898 Table 3 – Interoperability Technical Design Process Requirements 899

9007.5. Improvements 901 The Improvements section outlines the controls that will need to be in 902 place to support the interoperability testing processes. 903 Improv-x Interoperability Improvements Process Requirements Description Improv-1 The ITCA SHALL implement monitoring and auditing programs to ensure adherence to its policies. This is in the ISO 65 document. Improv-2 The ITCA SHALL establish a checklist for the auditing of the appointed evaluation laboratories. Improv-3 The ITCA SHALL periodically audit the laboratories at appropriate intervals to ensure laboratories up- hold necessary capabilities. Improv-4 The ITCA SHALL establish an auditing procedure and implement audits to verify that product interoperability is maintained after the product passes the testing and certification programs and enters the market. Improv-5 The ITCA SHALL have processes in place, including corrective and preventative actions, which results in continual improvement of their testing and certifica- 461______462 53 463 Smart Grid Testing And Conformity Committee 464 IPRM Version 2.0 – DRAFT – November 3, 2011 465 466 Interoperability Process Reference 467 Manual 468 (IPRM) 469 tion programs. Improv-6 The ITCA SHALL be in constant communication with the standards writing committees to create a feedback loop. For example, the ITCA should define a process to communicate the TSS test results back to the SSOs and stakeholders. Improv-7 The ITCA SHALL provide a forum for feedback to be received from a stakeholder, interested business party and use case in order to improve its interoperability best practices. Improv-8 It is preferred that ITCAs have a method for actively soliciting interoperability feedback on implementations of the standard in order to achieve some level of customer and user-community satisfaction on that feedback. 904 Table 4 – Interoperability Improvements Process Requirements 905

9067.6. Cyber Security 907 The Cyber Security section outlines the requirements which SHALL be 908 used by the ITCA to validate the security-related components of the in- 909 teroperability testing program. 910 Sec-x Cyber Security Improvements Process Requirements Description Sec-1 The ITCA SHALL define the procedures and processes which will be used to validate interoperability cyber security requirements. Sec-2 The testing and certification program SHALL ensure 470______471 54 472 Smart Grid Testing And Conformity Committee 473 IPRM Version 2.0 – DRAFT November 3, 2011 474 475 Interoperability Process Reference 476 Manual 477 (IPRM) that cyber security functional performance requirements are defined, and test cases designed to evaluate the requirements. Sec-3 Where applicable, the ITCA SHALL have a process in place to select and implement a Digital Certificate Is- suance mechanism that may include the election of a Certificate Authority. The energy service providers can use this certificate for authentication that a given product has actually been certified.10 Sec-4 The ITCA SHALL be responsible for certificate management including issuance, maintenance and policing. The ITCA can choose to outsource this responsibility as long as they remain responsible for the interoperable outcome. Sec-5 The ITCA SHALL implement a process to qualify testing personnel at an appropriate level for their cyber security test training and experience. Sec-6 The ITCA SHALL specifically require a test methodology that includes widely-accepted stress testing processes including static analysis and penetration testing. Sec-7 The ITCA SHALL assure that cyber security models are policy driven, and testing SHALL also be based on policy settings. Sec-8 The ITCA SHALL ensure that processes are in place for vendors to submit threat analysis as part of the certification process. Sec-9 The ITCA SHALL leverage and align with existing secu-

47810 Optional for ITCAs that result in interfaces and not result a physical product. 479______480 55 481 Smart Grid Testing And Conformity Committee 482 IPRM Version 2.0 – DRAFT – November 3, 2011 483 484 Interoperability Process Reference 485 Manual 486 (IPRM) 487 rity test programs. Sec-10 The ITCA SHALL ensure that processes are in place to incorporate component-based cyber security concepts in the testing program. Sec-11 The ITCA SHALL ensure that all business, system, and technical interests are represented in the testing program. 911 Table 5 – Interoperability Cyber Security Process Requirements 912

488______489 56 490 Smart Grid Testing And Conformity Committee 491 IPRM Version 2.0 – DRAFT November 3, 2011 492 493 Interoperability Process Reference 494 Manual 495 (IPRM)

9138. References 914 915NIST Framework and Roadmap for Smart Grid Interoperability Stan- 916dards 917ISO 17000 - Conformity Assessment - Vocabulary and general principles 918ISO 17011 - Conformity Assessment - General requirements for accredita- 919tion bodies accrediting conformity assessment bodies 920ISO 17025 - General requirements for the competence of testing and calibra- 921tion laboratories 922ISO Guide 65 - General requirements for bodies operating product certifica- 923tion systems 924ISO Guide 67 - Conformity assessment - Fundamentals of product certifica- 925tion 926

496______497 57 498 Smart Grid Testing And Conformity Committee 499 IPRM Version 2.0 – DRAFT – November 3, 2011 500 501 Interoperability Process Reference 502 Manual 503 (IPRM) 504

9279. Glossary of Terms 928Accrediting Body – Organization that formally evaluates processes of test 929laboratories or certification bodies with respect to specific standard(s) or speci- 930fication(s). 931Application Profile - A selected subset of the product and / or standard which 932can be used to implement a particular feature set or use case scenario. 933Attestation - Issuance of a statement that fulfillment of specified require- 934ments has been demonstrated. 935Certificate – Unique identifier of a particular product. It applies to both soft- 936ware and hardware products. The certificate can be a physical or digital artifact 937(e.g., X.509 PKI schemes require digital certificates). 938Certification – Third-party attestation related to products, processes, systems 939or persons. 940Certification Bodies (CBs) – The entity responsible for certifying that prod- 941ucts have fulfilled the requirements of a standard or specification. 942Compliance Folder - The set of test evidence, usually including test data, test 943report, product information, and review records. The folder serves as the record 944of an implementation fulfilling all requirements of a certification test program. 945Compliant Portion Description (CPD) – A CPD is a definitive manifest of all 946mandatory and optional features implemented in a certified product. The CPD 947is generally used by product designers to judge: 948  Conformance of an implementation, 949  Completeness of a system composed of pre-certified sub-compo- 950 nents by comparing each of the CPDs of those sub-components. 951  Interoperability of two products based on matching feature sets 952 as described by their respective CPDs. 505______506 58 507 Smart Grid Testing And Conformity Committee 508 IPRM Version 2.0 – DRAFT November 3, 2011 509 510 Interoperability Process Reference 511 Manual 512 (IPRM) 953For example, a designer can compare the CPD with the test requirements to 954determine the level of conformance of a product to a specification. When de- 955signing a product composed of pre-certified sub-components, the respective 956CPDs will serve as selection criteria to design the complete product. The CPD 957also helps to judge the level of interoperability that can be expected from inter- 958actions between two independent implementations. A client service and a 959server function can be reviewed for their expected level of interoperability 960solely based on their respective CPDs. 961Conformance Certification – A third-party attestation that a product con- 962forms to a standard or specification. 963Conformance Testing – Determines whether an implementation conforms to 964the standard as written. This is done by evaluating the implementation with a 965test tool such as an emulator, test harness, golden unit, etc. 966Feature set – A feature set is a particular characteristic of a product based on 967a particular use case scenario. For example: signaling price is a feature set. 968First Party Testing – is when an implementer self-tests their own product. 969This is usually permitted after a technology has matured to where sufficient 970tools and specifications enabling first party testing are available to all vendors. 971Inheritance – Those actions required to evaluate the compatibility of a pro- 972posed inherited design including products, subsystem functions and design re- 973quirements. 974Interoperability – Ability of a product or system to work with or integrate with 975another product or system based on defined business requirements. 976Interoperability Testing – Connects two or more implementations together 977and determines whether they can successfully communicate. Significantly dif- 978ferent from conformance testing, it is often possible for two systems that con- 979form to the standard to be unable to communicate. If they can communicate, 513______514 59 515 Smart Grid Testing And Conformity Committee 516 IPRM Version 2.0 – DRAFT – November 3, 2011 517 518 Interoperability Process Reference 519 Manual 520 (IPRM) 521 980it is possible that they cannot perform any useful functions. These situations 981arise because the implementations have conflicting interpretations of the spec- 982ification, or because they have chosen conflicting options within the standard. 983A particular form of interoperability testing is application testing, in which there 984is a specification for the particular use of standard that can be tested. 985Implementation Under Test (IUT) – The implementation subject to testing. 986Covers System Under Test (SUT) and Device Under Test (DUT) 987Multi-vendor and Multi-product Testing Event – An interoperability test of 988products with other peer products. The outcome of the testing is used to im- 989prove both products and the specification. 990Performance / Protocol / Proforma Implementation Conformance 991Statement (PICS) – Defines all mandatory and optional feature sets of a 992specification that can be used to implement a product. 993Platform level communications protocol - In the IPRM, platform level com- 994munications protocols are integrated products based on standards only associ- 995ated with layers 1 and 2 of the OSI layer. (e.g., Wi-Fi platform) 996Qualified Product Notification (QPN) – A certificate and accompanying ex- 997planatory document issued by the ITCA as a record when a product has fully 998satisfied the requirements of the testing and certification program. The QPN 999details all supported feature sets verified by the program. 1000Record of Work - The material evidence of any work or task, such as test 1001data or test report. 1002Second Party Testing – Testing activities performed by buyers and users. 1003Security Testing – Analyzes whether the implementation correctly makes use 1004of any security features from the standard or other security features available

522______523 60 524 Smart Grid Testing And Conformity Committee 525 IPRM Version 2.0 – DRAFT November 3, 2011 526 527 Interoperability Process Reference 528 Manual 529 (IPRM) 1005in the product. This is the most difficult type of testing program since it must 1006evaluate whether the system has vulnerabilities, which are not always obvious. 1007Standards Setting Organizations (SSOs) - An association whose primary 1008activities are developing, coordinating, promulgating, revising, amending, re-is- 1009suing, interpreting, or otherwise maintaining standards. A Standards Develop- 1010ing Organization is one form of a Standards Setting Organization. Example 1011SSOs including International Organization for Standardization (ISO), Interna- 1012tional Electro technical Commission (IEC), Institute of Electrical and Electronics 1013Engineers (IEEE), American National Standards Institute (ANSI), etc. An SSO 1014can also be an industry trade association that develops industry standards 1015such as the ZigBee Alliance. 1016Static Conformance Review – A review of designed feature sets versus the 1017specified PICS to determine the extent to which the features are supported by 1018the IUT. This is the first step when a product enters a testing program. General- 1019ly the test lab requests that the implementer declare all supported feature sets 1020in a product. This information is used to create the test plan for that product. 1021Testing and Test Control Notation (TTCN) - A formalized test scripting lan- 1022guage used to describe communication protocol test cases per ISO / IEC 9646. 1023Test Campaign - A series of tests for a particular product out of the TSS, 1024based on the running Test Profile group and the Test Plan, derived from the Test 1025Case Reference List. 1026Test Cases – A set of tests to verify a particular feature set. There are many 1027ways to test a feature set, with each of those representing a test case. Gener- 1028ally, a program defines all possible test cases in the test specification docu- 1029ment. 1030Test Case Reference List – A current master list of all tests that are to be in- 1031cluded into a product test plan. This list also indicates the time variable appli- 530______531 61 532 Smart Grid Testing And Conformity Committee 533 IPRM Version 2.0 – DRAFT – November 3, 2011 534 535 Interoperability Process Reference 536 Manual 537 (IPRM) 538 1032cability of each test by reflecting those tests which are no longer valid, and 1033those that are not currently valid but are scheduled to become active in the 1034near future. This helps a product implementer in preparing fully conforming 1035and interoperable products for an upcoming launch. 1036Test Harness - Collection of software, test data, and hardware configured to 1037test a product by operating it under varying conditions and monitoring its be- 1038havior and output. 1039Test Interface - The programmatic application interface to enable communi- 1040cation between a test harness and system or device under test. 1041Test Plan – A Test Plan is a list of applicable tests for a specific product and is 1042derived from the Test Case Reference List. 1043Test Procedure – A stepwise test method of a particular test case. An exam- 1044ple of a test procedure can be the steps needed for an Energy Services Inter- 1045face (ESI) to send price signals, which may include configuring the time infor- 1046mation, updating price tables, etc. 1047Test Profile or Profile - A select subset of a product and / or standard to im- 1048plement a particular test of a feature or a use-case test. Test Profiles evaluate 1049a subset of a TSS and are used to target specific areas of product interoperabil- 1050ity. 1051Test Resource - Any information, equipment, material, and support required 1052to implement testing. 1053Testing – According to EN 45020, testing is defined as “the technical operation 1054that consists of the determination of one or more characteristics of a given 1055product, process or service according to a specified procedure”. 1056Testing Laboratories (TLs) – Test service providers for a standard or specifi- 1057cation.

539______540 62 541 Smart Grid Testing And Conformity Committee 542 IPRM Version 2.0 – DRAFT November 3, 2011 543 544 Interoperability Process Reference 545 Manual 546 (IPRM) 1058Test Suite Specification (TSS) or Test Spec- Consists of a suite of tests, 1059categorized into logical functional areas, such as use cases or well-defined fea- 1060tures. Each test suite consists of many related test cases corresponding to a 1061particular feature set or use case. Test cases would include both valid and in- 1062valid behavior tests. Each test case is further described step-by-step with test 1063procedures and well defined pass / fail / indeterminate criteria, along with ref- 1064erences. 1065Test Suite- A collection of related test cases. A test suite can be put together 1066to test a feature set. A pricing test case would be in a “price test suite” but a 1067messaging test case would be in a “messaging test suite”. 1068Third Party Testing – Testing activities performed by organizations indepen- 1069dent of first or second parties. 1070Use Case - A description of a system’s behavior as it responds to a request 1071that originates from outside of that system 1072

547______548 63 549 Smart Grid Testing And Conformity Committee 550 IPRM Version 2.0 – DRAFT – November 3, 2011 551 552 Interoperability Process Reference 553 Manual 554 (IPRM) 555 1073 1074 1075 1076 1077 1078 1079 1080 1081 INFORMATIONAL ANNEXES

556______557 64 558 Smart Grid Testing And Conformity Committee 559 IPRM Version 2.0 – DRAFT November 3, 2011 560 561 Interoperability Process Reference 562 Manual 563 (IPRM) 1082A. Working Group 1083 1084 Interoperability Process Reference Manual (IPRM) – Working Group #4 Leadership Rik Drummond, Chair Rolf Bienert, Co-Chair Rudi Schubert, Project Manager

IPRM Version 2 Editing Team Members Rik Drummond Drummond Group Rudi Schubert Enernex Dean Prochaska NIST Marianne Swanson NIST Don Heirman Heirman Consultants Sandy Bacik Enernex Kent Donohue Underwriters Laboratories Rolf Bienert TUV Rheinland Listserv Information To send an e-mail to the IPRM working group list: [email protected]

To subscribe to the IPRM working group, go to the following address: http://collaborate.nist.gov/twiki-sggrid/bin/view/SmartGridTestingAndCertificationCommittee select Join SGIP-SGCTCC-IPRM listserv. 1085

564______565 65 566 Smart Grid Testing And Conformity Committee 567 IPRM Version 2.0 – DRAFT – November 3, 2011 568 569 Interoperability Process Reference 570 Manual 571 (IPRM) 572 1086 B. Document History 1087 Revisi Revisio Revision Summary of Changes on n Date By Numb er 1 8/30/11 Rudi First outline of IPRM V2 prepared for IPRM working group Schubert review and comment

2 10/22/11 Rudi IPRM V2 – Official 1st draft for SGTCC comment released Schubert 3 11/3/11 Rudi IPRM V2 – Official 2nd draft for SGTCC comment Schubert released; incorporates most comment received on prior draft, with additional comments tabled for further working group discussion 4 5 1088 1089

573______574 66 575 Smart Grid Testing And Conformity Committee 576 IPRM Version 2.0 – DRAFT November 3, 2011