Essnet “Preparation of standardisation”

Work package 1 Review of current international and methodological handbooks and guidelines

Version 9 September 2011

1. Definition of a statistical standard and of guidance...... 3 1.1. Introduction...... 3 1.2. Current use of the words “standard” (noun or adjective) and standardise/ standardisation in statistics...... 3 1.2.1. Method used...... 3 1.2.2. Some statistical references to standards as “sets of rules to be applied” and the corresponding clues...... 5 1.2.3. The current use of the words standard and standardisation is not fully consistent...... 10 1.3. Taking stock of standardisation in other domains ...... 11 1.3.1. Definition of a standard...... 11 1.3.2. How should this proposal be understood in the ESSnet STAND-PREP ? ...... 13 2. Review and analysis of six methodological manuals...... 16 2.1. A framework for the analysis...... 16 2.1.1. Conceptual scheme of the repository for the standards ...... 17 2.1.2. The IT tool for the management of the repository...... 1 2.1.3. Procedure followed for the definition and refinement of repositories for single handbooks and for the construction of the unified repository ...... 27 2.2. Analysis of the six methodological handbooks...... 28 2.2.1. Eurostat sampling references guidelines...... 29 2.2.2. Recommended practices for editing and imputation in cross-sectional business surveys (EDIMBUS)...... 33 2.2.3. Guidelines for statistical metadata on the Internet...... 41 2.2.4. ESS Handbook for Quality Reports...... 45 2.2.5. Recommended practices for questionnaire design and testing ...... 52 2.2.6. Survey Methods and Practices ...... 57 2.3. Description of the unified repository for methodological standards...... 65 2.3.1. General description of the repository...... 65 2.3.2. Examples of compared analyses of standards derived from different handbooks for the same GSBPM sub-processes...... 72 3. Review of support measures ...... 82 3.1. Introduction...... 82 3.2. Support measures related to standards...... 82 3.3. Overview of support measures...... 83 3.3.1. Motivation for implementation ...... 83 3.3.2. Making the implementation easy...... 84 3.3.3 Supporting right implementation ...... 86 3.4. Summary...... 87 4. Quality of statistical standards and criteria for the assessment...... 89 4.1. Introduction...... 89 4.2. The concept of quality of standards ...... 89 4.2.1. Applicability...... 90 4.2.2. Effectiveness ...... 91 4.2.3. Efficiency...... 91 4.3. Relations between the proposed quality criteria on methodological standards ...... 92 4.4. Main problems connected with standards that may have quality character...... 92 5. Conformity to standards...... 94 5.1. Definition ...... 94 5.2. Conformity assessment is considered during the development of standards...... 94 5.3. Standards in the EU and EU conformity assessment...... 96 5.3.1. Framework for EU standardisation...... 96 5.3.2. Modules for conformity assessment ...... 97 5.4. How to assess conformity to methodological standards in statistics? ...... 101 5.4.1. Problem statement in the ESS...... 101 5.4.2. Proposal for conformity assessment ...... 103 Annex 1 - ISO/ IEC Guide 2 - Standardization and related activities - General vocabulary...... 106 Annex 2 - Extracts referring to standards with another meaning ...... 124 Annex 3 – Review of literature on standards...... 126

2 1. Definition of a statistical standard and of guidance

1.1. Introduction

The definition of a standard in the ESSnet STAND PREP is a concern that is felt to be important by all participants. This issue has been tackled several times (even when preparing the answer to the Eurostat “call for interest”!). This is an issue that deserves serious consideration both for this ESSnet but very probably as well for further discussions in the sponsorship and in the ESSC and PG that will discuss the reports of that sponsorship.

This kind of work is not exceptional in the ESS: for instance, the statistical definition of “enterprise” was a lasting concern for several years in Business statistics. The definition of “quality” required some work as well. For these two examples, there were some pre-existing references in other domains or in other statistical offices that helped settle the debate relatively quickly. For the definition of “enterprise”, the existing definitions in national accounts and accounting standards helped a lot. As for the definition of quality in statistics, papers from the US Bureau of the Census, Statistics Canada and Statistics Sweden provided very useful references

In the case of standards, the ESSnet has taken stock of the literature on the subject. Whereas there are many references on standards in domains other than statistics, we did not find references in statistical organisations that could help to solve the issue quickly.

In this document, we refer to the practice in existing statistical texts using the word of “standard” in order to define the underlying agreed concept for statisticians. This method is used below.

However, since the previous method does not bring useful results, we have then used the ISO vocabulary to propose a definition for the concept of “standard” and the corresponding definition of standardisation.

1.2. Current use of the words “standard” (noun or adjective) and standardise/ standardisation in statistics

1.2.1. Method used

“Standards” are often referred to by statisticians, with different meanings. This is largely due to the fact that “standard” has several meanings in English. We should therefore select for our purpose only texts in which the meaning of “standard” used by the author refers to standards that are the result of the standardisation activity.

In short, standardisation consists in developing standards.

The definition of standardize given by the Merriam-Webster dictionary 1 is:

1 American English 3 1: to reduce to or compare with a standard 2: to bring into conformity with a standard 3: to arrange or order the component items of a test (as of intelligence or personality) so that the probability of their eliciting a designated class of response varies with some quantifiable psychological or behavioral attribute, function, or characteristic

The last meaning is corresponds to some technical work in statistics, and is not relevant here.

The first two meanings of standardise refer to the word standard with the same underlying meaning of this noun. As a first approximation, such standards would be “ a set of rules to be applied ”.

We have to take into account the various meanings of “standard” in English, in order to keep for our analysis only the texts that refer to the meaning of standard we are interested in.

Whereas it might be easier to understand for native English speakers, it can be tricky for non native ones.

If we refer to the Cambridge dictionary Online 2, the definition of standard is :

1 - meaning “usual”: o a pattern or model that is generally accepted. ex: This program is an industry standard for computers. o a song or other piece of music which has been popular and often played over a long period of time 2- in the context of quality : o a level of quality - ex: This essay is not of an acceptable standard - do it again. This piece of work is below standard/is not up to standard. We have very high safety standards in this laboratory. Not everyone judges success by the same standards - some people think happiness is more important than money. Her technique became a standard against which all future methods were compared. o a moral rule which should be obeyed - ex: Most people agree that there are standards (of behaviour) which need to be upheld, but agreeing on them is rather more difficult.

The definition given by the Merriam-Webster dictionary (“definition for English learners”) is:

1 a : a level of quality, achievement, etc., that is considered acceptable or desirable [count] ▪ high/low standards of quality ▪ His work is not up to our standards. [=is not as good as we require it to be] ▪ By modern/today's standards, the house is just too small. ▪ industry standards ▪ Their standards are slipping. [noncount] ▪ His work is not up to standard. = His work is below standard. [=his work is not good enough] — see also double standard , standard of living b standards [plural] : ideas about morally correct and acceptable behavior ▪ She has high moral standards. [=principles] 2 [count] : something that is very good and that is used to make judgments about the quality of other things ▪ This book is the standard by which all others must be judged.

2 British English 4 3 [count] : a fixed official unit of measurement ▪ a standard of weight — see also gold standard 4 [count] : a flag that is used in official ceremonies 5 [count] : a song that has been sung by many different artists ▪ an old standard

The meaning we are interested in is the first meaning listed by the Cambridge dictionary and in the Webster dictionary (reference to “ industry standards ”).

After taking into account these possible meanings of the word “standard”, we have rejected some references in the statistical literature using the word “standard” (see Annex) and kept the ones below. For each of these, we derive “clues” for the definition of the concept of “standard” that is implicit in the references below. These clues are shaded in light grey.

1.2.2. Some statistical references to standards as “sets of rules to be applied” and the corresponding clues

1.2.2.1) In the report of the “Canadian Advisory panel on Telephone public opinion survey quality” (February 2007): Standards are defined as “ Practices that should be requirements for all telephone studies conducted by the Government of Canada ».

Clue 1: We deduce from this definition that: o there are several statistical standards for just the statistical activity of “telephone opinion survey”. The paper lists a series of such standards. There are more or less as many standards as steps in the GSBPM. These sentences are made of a few sentences (generally 5 to 6). Example : for data collection dealing with call-backs, the standard is defined as: “Unless circumstances strongly indicate a different number of call-back attempts and scheduling of call-backs is required, there will be a minimum of eight call-backs made, varying the day and time of each call.”

o these standards are requirements, i.e. compulsory rules that have to be applied.

1.2.2.2) In the “ESS code of practice” (CoP), the word “standard” is used several times.

1.2.2.2.1: It is stated that: “ The head of the statistical authority and, where appropriate, the heads of its statistical bodies have the sole responsibility for deciding on statistical methods, standards and procedures ”.

Clue 2: This would suggest that standards differ from statistical methods (which refer to academic knowledge) and from procedures (which refer to ad hoc organisations of the production process).

1.2.2.2.2: It is also stated that “ European and other international standards , guidelines and good practices must be fully observed in the processes used by the statistical authorities to organise, collect, process and disseminate official statistics .”

5 Clue 3: This would suggest that standards are of a more binding nature than guidelines and good practices. However, since the CoP requires the guidelines and good practice “fully observed” leads to some confusion: what is then the difference between a standard and a guideline? Standards may be relevant for the following activities in NSIs: organisation, data collection, processing, dissemination.

1.2.2.2.3: The overall methodological framework of the statistical authority follows European and other international standards , guidelines, and good practices .

Clue 4: Standards may define the methodological framework. There are international standards, international guidelines and international good practices that refer to three different types of entities.

1.2.2.2.4: Procedures are in place to ensure that standard concepts, definitions and classifications are consistently applied throughout the statistical authority.

Clue 5: there is a conformity assessment of standards inside the NSI.

1.2.2.2.5: Revisions follow standard , well-established and transparent procedures. […] A standard daily time is set for the release of European Statistics.

Clue 6: standards are defined for procedures.

1.2.2.2.6: Statistics are compiled on the basis of common standard s with respect to scope, definitions, units and classifications in the different surveys and sources.

Clue 7: standards include some aspects of the definition of statistics like scope, definition (of characteristics to be collected?), statistical units. Standards are defined as well for classifications…These standards have to be consistent (“common”) for the various statistics and classifications.

1.2.2.2.7: Metadata are documented according to standardised metadata systems.

Clue 8: the products than can be standardised include whole systems, for instance metadata system.

1.2.2.3) In the draft regulation on ESA 2010 , it is written that:

The objective of the Regulation is to revise ESA both concerning the methodology on common standard s, definitions, classifications and accounting rules, and the programme for transmitting for Union purposes accounts and tables compiled according to the methodology.

Clue 9: The ESA includes a methodology (= a set of methods) dealing with standards. These standards have to be distinguished from definitions, classifications, accounting rules.

Article 1 states that: ESA 2010 provides for: (a) a methodology (Annex A) on common standard s, definitions, classifications and accounting rules that shall be used for compiling accounts and tables on comparable bases for the purposes of the Union, together with results as required by Article 3;

6 Articles 3 states that “Member States shall transmit to the Commission (Eurostat) the data and metadata required by this Regulation in accordance with an interchange standard specified by the Commission (Eurostat).”

Clue 10: Standards in statistics will be available for the activity “interchange of data and metadata”.

Paragraph 1.21 states that : The concepts are internationally compatible because: a) The concepts in the ESA are consistent with those in the worldwide guidelines on national accounting, i.e. the SNA; b) For the EU Member States, the ESA is the standard for submitting national accounts data to all international organisations;

Clue 11 : The whole ESA is considered as one single standard.

It is stated as well that :

“The main concepts in the ESA are well-established and fixed over a long period, because: a) They have been approved as the international standard for many years;

Clue 12: The set of all concepts of the ESA (but not the other rules/ statements) is in itself one single standard.

1.2.2.4) The November 2010 conclusion of the Ecofin Council states that :

The Council CONSIDERS that it is desirable - for administrative purposes – to refer to the agreed set of national accounts standard s in the EU when the ESA is revised, in order to reinforce quality, enhance clarity and transparency of communication vis-à-vis the authorities (EFC, Finance Ministers) and other users, and increase efficiency, while reducing burden .

Clue 13: The Esa is not one single standard, but a set of standards.

1.2.2.5) The regulation 223/2009 uses as well the term “standards”:

For the purposes of this Regulation, the following definitions shall apply: 1- statistics means […] 2- ‘development’ means the activities aiming at setting up, strengthening and improving the statistical methods, standards and procedures used for the production and dissemination of statistics as well as at designing new statistics and indicators;

Clue 14: standards may be produced during the development phase in NSIs.

Please note that in this regulation, “ The development, production and dissemination of European statistics shall take into account international recommendations and best practice .

Clue 15: standards set up by the ESS take into account international recommendations.

To guarantee the quality of results, European statistics shall be developed, produced and disseminated on the basis of uniform standard s and of harmonised methods.

Clue 16: Standards should be uniform.

7

1.2.2.6) In the “Communication of the Commission on the vision for the next decade” (404/2009) 3:

In order to guarantee comparability and consistency among the data of all Member States, the output from NSIs is harmonised according to agreed standards .

Clue 17: Since Eurostat describes here the current situation in the ESS (output harmonisation), we assume that such standards include “definitions of statistics” and “methodological handbooks”.

The stovepipe model […] Last but not least, this way of production is highly inefficient and costly, as it does not make use of standardisation between areas and collaboration between Member States.

Clue 18: Logically speaking, this means that standardisation between areas leads automatically to efficiency and diminution of cost.

At the level of the NSIs, statistics for specific domains are then no longer produced independently from each other; instead they are produced as integrated parts of comprehensive production systems (the so-called data warehouse approach) for clusters of statistics. These systems would be based on a common (technical) infrastructure, they would apply as far as possible standardised software , and they would make use of all available data sources which are appropriate in quality.

Clue 19: Software can be standardised.

There are considerable challenges of a technical and methodological nature. Standardisation and integration of formerly separated production processes will demand great efforts and an effective change management.

Clue 20 : The production process of NSIs has to be standardised, i.e. standards on various aspects of the production process have to be available.

1.2.2.7) In the Communication on priority setting in statistics:

“ Eurostat is also sponsoring, together with other international organisations, development and promotion of the Statistical Data and Metadata eXchange format (SDMX), which has the status of an ISO standard . This XML-based standard permits electronic transfers of statistical data and metadata between organisations and to users. SDMX standard s will help enterprises to have a return on the information that they provide to statistical authorities.”

Clue 21: “standard” has the same meaning as in ISO texts.

Direct use of standardised accounting data using electronic tools is another promising way forward. The data are received directly from the enterprise accounts without a special questionnaire. The accounting data can be supplemented by small surveys as in the case of administrative data. The Commission will support Member States by aligning the definitions in the legal acts on those in accounting standard s, by developing the necessary electronic tools, and by devising methods for targeted complementary surveys and corresponding estimation techniques.

3 In the French version, the word « norme » is always used, referring to the ISO-types standards. In the German version, the word « Norme », but also « Standard/ Standardisierung » are used. The two latters do not belong to the ISO official vocabulary, but seem to refer to the same objective. 8 Clue 22: Since accounting standards are defined by standard setting organisations, the word standard is used in the specialised meaning.

1.2.2.8) The Communication of the Commission on the robustness of statistics (211/2011):

Recent experience has shown that it is as important to assess the quality of upstream data as it is to continue assessing the quality of the translation of this data into ESA standards . Preventive measures, such as promoting further standardisation in public accounts , and a wider proactive management of risk assessment are needed, including assessing the adequacy of the quality of upstream data for compiling statistics in accordance with ESA rules.

Clue 23: ESA is referred to as a set of several standards. Definitions are classified under standards.

For instance, a set of quality management standards tailored to the specificities of the EDP compilation process could be agreed, implemented by NSIs and certified by the Commission;

Clue 24: Standards can be certified.

1.2.2.9) Invitation to the ESSnet Preparation of Standardisation: (letter MG 447320 from 22 August 2010)

1 - […]: to increase the efficiency and quality of statistical production by further standardisation of processes and methods .

Clue 25 : standardisation increases efficiency and quality of statistical production

2- prepare the work of the sponsorship. The ESSnet has to deliver the main results: • criteria for the adoption of methodological handbooks as standard s • classification of standard s • description of the procedure for adoption and maintenance.

Clue 26 : existing methodological handbooks are not considered standards by Eurostat. Adoption of a standard is a formal act, defined by a procedure.

1.2.2.10) Annex I of the call for interest for the ESSnet Preparation of Standardisation:

Standardisation requires the contribution of different actors in the statistical process: statistics production, IT development, methodology, metadata and management.

Clue 27 : standards are envisaged for methodology, statistics production, IT development.

The project aims to prepare standardisation in two ways: by reviewing methodological standard s embedded in handbooks and by systemising other standard s. The difference is not only in subject (handbooks and other standard s), but also in angle: the first work package concerns the standards itself, whereas in the second work package the focus is on the process of standardisation .

Clue 28 : methodological standards are expected to be either the methodological manual themselves or part of it. This is not very clear.

9 Although the scope of the standardisation initiative still has to be defined and the proposed action preparing standardisation serves to help define and clarify the scope, it is clear that it concerns statistical standardisation (and not for instance IT or communication standard s).

Clue 29: statistical standards would be analogous to standards in IT or Communication, i.e. ISO/ IEC 4 standards

Work package 1 and 2 will in parallel generate some ideas on future work on standardisation ; e.g. the registration of standard s, coherence checks, standardisation versus flexibility and development of methods, levels of standardisation and priorities for standardisation .

“Describe procedures for adoption and maintenance of the standard s”.

The review document consists of an advice on standardising (parts of) the handbook and of suggestions on necessary changes before standardisation ; it also includes a reflection on the current adoption of the standard and on potential support measures;

Clue 30: it is expected that there would be rules for the adoption of standards, that these standards would be binding. Current practice in the adoption of standards is considered not to be satisfactory.

1.2.3. The current use of the words standard and standardisation is not fully consistent

Some clues are very unclear even in isolation: o clue 17 requires standards to be uniform. o clue 20 mentions that software can be standardised. This would mean that: there exist somewhere standards for software. Existing software has to be brought to conformity with these standards. o clue 24 mentions that standards can be certified. Certification means that an organisation checks that a product is in agreement with some standards, and that this organisation guarantees the conformity. Standards have to be developed and registered by standard setting organisations, along agreed procedures. What is meant by clue 24?

Some clues give contradictory information:

This clue contradicts Why clue n° 7 9 standards do/ do not include definitions or classifications 1 8, 28 The scope of a standard is an elementary step of the statistical process (defined for instance by the GSBPM) in clue 1 and a whole system in clue 8 9, 23 11, 13, 14 The ESA is a set of elementary standards (clues 9, 23) or a single standard (clue 11, 13, 14)

4 16 International standards are available (clue 4) but the ESS standards are only required to take into account recommendations (that would

4 IEC : the International Electrotechnical Commission is in charge of international standardisation in electricity and electronics. 10 be expected to include guidelines and good practice but not standards). 9 18 Clue 9 excludes definitions from the scope of standards whereas 18 includes them 15 22, 29 Standards would be the result of some sort of in-house development (clue 15), in contradiction with the formal standards produced by standard-setting organisations (clue 22). 18 26 Methodological handbooks are (clue 18) or are not (clue 26) considered as standards.

Conclusion : the current use of the word “standard” in the ESS does not refer to a very precise concept. It deals with rules, but their origin, their binding nature, their link with various statistical activities is not precisely defined nor uniform in the various texts (and sometimes in the various paragraphs of one single text).

1.3. Taking stock of standardisation in other domains

1.3.1. Definition of a standard

There is a high convergence in the definitions used either by US standardisation organisations, by EU ones (CEN) and by the ISO/ IEC.

The ISO offers a full glossary, defining consistent elements for the standardisation activity.

The group of statisticians in the ESSnet proposes to use the ISO reference vocabulary in order to ease communication with the outside world 5.

The starting point of the construction is the concept of normative document :

A normative document provides rules, guidelines or characteristics for activities or their results. Note 1: The term “normative document” is a generic term that covers such documents as standards , technical specifications , codes of practices and regulations . Note 2: A “document” is to be understood as any medium with information recorded on or in it. Note 3: The terms for different kinds of normative documents are defined considering the document and its content as a single entity.

A normative document is made of provisions.

A provision is an expression in the content of a normative document, that takes the form of a statement, an instruction, a recommendation or a requirement. A statement is a provision that conveys information. An instruction is a provision that conveys an action to be performed.

5 For a full description, see Annex 1 ISO/ IEC Guide 2 - Standardization and related activities - General vocabulary (for relevance matters, the consortium chose to publish only the English vocabulary)

11 A recommendation is a provision that conveys advice or guidance 6. A requirement is a provision that conveys criteria to be fulfilled. A deemed-to-satisfy provision indicates one or more means of compliance with a requirement of a normative document A descriptive provision is a provision for fitness for purpose that concerns the characteristics of a product, process or service A performance provision is a provision for fitness for purpose that concerns the behavior of a product, process or service in or related to use

NOTE These types of provisions are distinguished by the form of wording they employ; e.g. instructions are expressed in the imperative mood, recommendations by the use of the auxiliary “should” and requirements by the use of the auxiliary “shall”.

A standard is a document, established by consensus and approved by a recognized body , that provides, for common and repeated use, rules, guidelines or characteristics for activities or their results, aimed at the achievement of the optimum degree of order in a given context. Note: Standards should be based on the consolidated results of science, technology and experience, and aimed at the promotion of optimum community benefits.

A code of practice is a document that recommends practices or procedures for the design, manufacture, installation, maintenance or utilization of equipment, structures or products.

Note : A code of practice may be a standard, a part of a standard or independent of a standard.

Standards are classified into various types.

One useful distinction is the following: product standard : standard that specifies requirements to be fulfilled by a product or a group of products, to establish its fitness for purpose. process standard : standard that specifies requirements to be fulfilled by a process, to establish its fitness for purpose.

This distinction is relevant mainly for goods. Goods can be standardised either directly, by a product standard, or indirectly, by a process standard defining its production process. Product process are considered in the literature to be much more compatible with innovation than process standards.

6 "Guidance" has no precise meaning, except that it does not contain any rule. In the ISO glossary, it would be called a "Code of practice". However, a "Code of practice can also be a standard, and such a standard would include no rule at all. The concept of "Code of practice" cannot be exclusive of being a standard as defined along the ISO concept. In particular , a code of practice can be based on a consensus agreement or can be produced in any other way."

The ISO glossary states that: [A] code of practice [is a] document that recommends practices or procedures for the design, manufacture, installation, maintenance or utilization of equipment, structures or products NOTE A code of practice may be a standard, a part of a standard or independent of a standard.

Note: since the wording "ESS Code of practice" is already used for the definition of ethical principles in ESS statistics, it would probably be wiser to use another name. "Documentary guidance" could be kept instead...

12 Statistics are considered to be services 7. service standard: standard that specifies requirements to be fulfilled by a service, to establish its fitness for purpose.

Note: Service standards may be prepared in fields such as laundering, hotel-keeping, transport, car-servicing, telecommunications, insurance, banking, trading.

It is generally expected that service standards will belong to the category of process standards.

1.3.2. How should this proposal be understood in the ESSnet STAND-PREP ?

The group does not propose to change the meaning of existing recommendations or legal texts; it proposes to adopt future texts on the basis of a vocabulary consistent with other domains, i.e. the ISO definition.

For instance, let us take the example of methodological handbooks in the ESS. We have observed that in general: o the activity for which the methodological manuals are developed is sometimes/ frequently unclear: for instance, it is often difficult to decide if the manual is intended for general training of statisticians or for defining a process that has to be implemented. o the adoption procedure is often unclear, or at least is not in line with the usual decision process in the ESS. At some point, such manuals should be adopted by the ESS Committee or even by other kinds of organisation. o the “optimum degree of order in a given context” is not really dealt with. In particular, it is agreed that standards could aim for various degrees of quality of results or for savings. In general, methodological standards do not deal with this issue. Hence, in general, existing methodological manuals would be called normative documents , not standards. Very few of them could be considered as proper standards since their adoption is unclear. The set of ESS guidelines on Seasonal Adjustments (2009) is an exception: it has been adopted by the CMFB and the SPC (predecessor of ESS Committee).

Could it be called a standard?

It has to be considered that standards have to comply with some principles. As mentioned in “The ABC’s of standards activities (US National Institute of Standards and Technology- NIST , 2009) ”:

“To maximize their utility and to prevent the creation of unnecessary barriers to foreign and domestic commerce, the standards making process should be conducted in accordance with the internationally accepted principles of Consensus, Transparency, Balance, Due Process, and Openness. These principles are particularly important for standards likely to be used in technical regulations. These five principles are explained below: Consensus. Consensus means that all views are heard and the resultant standard is generally agreed to by those involved. Consensus is characterized by the absence of sustained opposition to substantive issues. However, it does not necessarily imply unanimity. Transparency . Transparency means: (a) providing advance public notice of a proposed standards development activity; (b) identifying the scope of work to be undertaken; (c)

7 At least they are classified in this category by the CPA, as all information services. 13 providing information on conditions for participation; (d) and providing an opportunity for all interested parties to comment prior to final approval and adoption. Balance. Balance means that no one interest, including the government, should dominate. It should be noted that balance can be affected not only by the number of participants in particular categories but also by the funding source. The provider of the funding in standards development work can sometimes end up dominating the process. This is particularly true if the funding is from a government entity. If funding is to be provided by a government agency or other entity, care must be taken to avoid undue influence on the outcome of the process by the funding provider. Due Process. Due process means that any person (organization, company, government agency, individual, etc.) with a direct and material interest has a right to: (a) express a position and the basis for it; (b) have that position considered; and (c) appeal if adversely affected. Due process ensures equity and fair play in the standards development process. Openness. The standards development process should be to open to participation by all materially affected interests.”

The ESS guidelines on Seasonal Adjustments cannot be considered as a standard of the European Economic Area (EU+EFTA) since they do not apply some of these principles. Neither the private producers of statistics, nor academics were associated to this process. Hence the Transparency/ Openness/ Due process criteria are not satisfied here. They can be satisfied only for the population of the national statistical authorities represented by the ESSC (it does not fully apply to the National Central banks, since the ECB/ ESCB has not adopted it).

The US NIST defines a typology of private standard setting organisations , made of five classes: technical and professional societies, industry associations, standards-developing membership organizations, building code organizations, consortia.

It quotes as well in the public sector two types of standards: o Government regulatory standards are those designed to be used by [US] federal regulatory agencies in rulemaking and related activities. o Federal and Military Specifications , which are official documents used by [US] agencies and by the [US] Department of Defense respectively, to support government procurement.

If we used the same categories, we would consider that the “ESS guidelines on Seasonal Adjustments” were a standard for the European Commission and Member States administration .

In addition, such standards could be classified according to their binding character: o mandatory standards would be those that are adopted by the co-legislators, i.e. those that are incorporated in a regulation or in a directive o voluntary standards would not be adopted by a law-making process.

For instance, the NACE rev.2 classification can be considered as a mandatory standard .

Another important issue in practice is the “scope” of statistical standards that could be developed in the ESS. By “scope”, we mean the activity for which standards would be developed (as mentioned in the definition of a standard, it provides rules for activities or their result) . The ESSnet group has discussed the following issue: should the activity covered be a global one (for instance, the design and production of specific statistics - let us say “Structural business statistics in the manufacturing sector”) or a very focused one (micro-editing of one structural business statistic)?

This concern was linked with the observation that existing methodological manuals can either deal:

14 o with a particular activity defined in the GSBPM (for instance defined by one phase or step in the model) for all kind of statistics. For instance, the manual on Edimbus: editing and imputation of cross sectional business statistics (GSBPM steps 5.3 and 5.4) o with a group of statistics for all steps in GSBPM. For instance, the manual on Handbook on the design and implementation of business surveys (1997)

In order to check for consistency of standards - which is desirable- it was considered if developing standards for all “building blocks” of the GSBPM would be the optimal solution…

Although defining a standard for each elementary activity of the statistical process cannot be excluded (no logical shortcoming in this possibility was identified), it was felt that standards (as defined at the ISO or with an EU directive) would encompass a sizeable number of statements (rules, guidelines, etc) in practice. For instance, the ISO 20252-2006 standard on Market, social and opinion surveys is about 35 pages long and covers all steps in the design, production and dissemination phases. It can be noted as well that the ISO has not chosen such an approach to standardisation. This choice might be due to several reasons: o there are fixed costs for negotiation: so developing a huge number of elementary standards would be costly o in order to cover the whole production process of one statistic, one would have to wait until the whole system of standards is developed.

Adopting such a definition does not mean that statements (as those quoted in paragraph 1.2.2) should be understood using such a meaning of the word “standard”.

Such a choice would be of no consequence on the scope of the work on standardisation in the ESS. There is no ambiguity on this issue, since the ESSnet STAND PREP has no competence on this issue, that can only be dealt with by the Sponsorship on standardisation.

15

2. Review and analysis of six methodological manuals

2.1. A framework for the analysis

The objective of Work Package 1 was to define the concept of standard in statistics, in particular the standards corresponding to “statistical methods”, and to apply these definitions in order to find out, for each of the six selected methodological handbooks, which parts can be considered as standards . Actually, it became soon clear that this work of analysis could yield a value added: the job was not only to excerpt the core of the methodological handbooks, consisting in the provisions that are the basic elements of a standard; but also the relationships with other fundamental items and references that are contained in a handbook, as methodologies, methods, IT tools, classifications, definitions, etcetera. Furthermore, it was also considered the usefulness of linking each provision (or group of provisions) to a given phase or sub-process of GSBPM, in order to have the possibility to collect and compare all provisions available for a given step of the production process of statistical information. The relationships between all the elements cited above are rather complex: it was therefore necessary to model them by defining a conceptual scheme representing all the entities that can be found in a set of methodological handbooks, and the relations connecting these entities. This conceptual scheme was translated in a logical one, the basis for a repository able to contain the knowledge that was to be extracted from the handbooks. It was also necessary to develop an IT tool enabling all partners in the ESSnet to populate the repository in a coordinated and assisted way. To this aim, a procedure was defined, consisting of the following steps: 1. each single handbook was assigned to one or two partners, that conducted the analysis on it; 2. if two different analyses had been carried out on the same handbook, a reconciliation step was performed to get a unique and agreed result; 3. the results of all the analyses have been introduced in a final and unified repository, that contains all the provisions individuated in the six handbooks. This final repository is to be considered as a proof-of-concepts: it is possible to start from the vast amount of handbooks related to the statistical production processes, each of them to be considered as a normative document, and obtain something that can be regarded as an instrument to get information at a detailed level on all rules concerning a particular subject, regardless of the source. This would not be possible if the handbooks had been described only at the aggregated level, without going at the elementary level of the “provision”. For instance, it is now possible to interrogate the repository in order to list all the provisions related to a given GSBPM sub-process obtaining information on the source (a given handbook), the type (indicating the associated degree of freedom), the related methodologies and methods.

16

2.1.1. Conceptual scheme of the repository for the standards

In figure 1, the conceptual scheme of the repository for statistical standards is reported, accordingly to the features of the entity-relationship model 8.

handbook = statistical normative activity (UN) document

1,n 0,n

co ntains refers to

1,1 0,n

structured 1,n 1,1 provision group of contains provisions

0,1 0,1 0,1 0,1

refers to refers to refers to refers to

1,n 1,n 1,n 1,n

methodology classification definition IT tool

0,n

based on

1,n 0,n

0,n method implemented in

Figure 1 – The conceptual scheme of statistical standards in methodological handbooks

8 Chen P.P., “The entity-relationship model—toward a unified view of data”, ACM Transactions on Database Systems (TODS), September 22–24, 1975

17 In the following, we illustrate the meaning of each entity, their attributes and the relationships among entities. To exemplify, we show also the forms dedicated to each entity that are in the software developed for the management of the repository (which is illustrated in detail in par. 2.1.2).

Normative document

In the model described in the E-R scheme, a statistical methodological handbook may be defined, accordingly to the definitions reported in Section 1 of this report, as a “ normative document ” (“a document that provides rules, guidelines or characteristics for activities or their results”).

The attributes related to this entity are: 1. title: the title of the document; 2. version: indication of the version, if available (otherwise, year of publishing); 3. type: the normative document can be: o a “standard” if the document has been established by consensus and approved by a recognized body; o a “pre-standard” if the document is adopted provisionally by a standardizing body and made available to the public in order to gain the necessary experience from its application; o a “technical specification” if it prescribes technical requirements to be fulfilled by a product, process or service; o a “code of practice” if it is a document that recommends practices of procedures o a “regulation” if provides binding legislative rules. 4. level: the area of influence of the document. It can be: o international o regional o national o provincial 5. body: the institution(s) that is (are) responsible for the development of the normative document; 6. consensus: the procedure (if any) followed to reach a consensus, required to declare the normative document as a standard; 7. aim: the purpose of the normative document.

In general, the handbooks we are dealing with can be considered as of the kind “code of practice”.

18

Group of provisions

A normative document must contain at least one group of provisions , in general many of them (cardinality = 1,n) 9. One group of provisions belongs to only one normative document. In the case of methodological handbooks, the identification of the groups of provisions should take into account the following elements: the structure of the handbook (sections and chapters) and the relations with the most detailed level of GSBPM (sub-processes). A group of provisions must refer to only one sub- process in GSBPM, and should be contained in only one chapter (maybe in more paragraphs).

Attributes of the entity “ group of provisions ” are: 1. collocation: indication of the logical (section, chapter) and physical (from page to page) collocation inside the handbook; 2. description: any important feature to be reported; 3. main GSBPM sub-process 10 ; 4. auxiliary GSBPM sub-process; 5. additional GSBMP sub-process.

Provision

Accordingly to the definitions given in Section 1, a provision is …

A group must contain at least one provision , in general many of them (cardinality = 1,n).

Attributes of the entity “ provision ” are: 1. indication of the physical collocation (page) inside the handbook; 2. description: text, as it appears in the document; 3. type: o statement, o instruction, o recommendation, o exclusive requirement, o optional requirement, o deemed-to-satisfy provision, o descriptive provision, o performance provision; 4. sequence: progressive number unique inside the same structured group;

9 In a given relationship between entities A and B, the cardinality is a couple whose first element indicates the minimum number of instances of B linked to one instance of A, while the second element indicates the maximum number. 10 See the paper “Generic Statistical Business Process Model - Version 4.0” (UNECE April 2009)

19 5. condition(s) of application; 6. components of quality 11 : a) the ones referred to the product: o relevance; o accuracy; o timeliness and punctuality; o accessibility and clarity; o coherence and comparability; b) the ones referred to the process: o effectiveness; o efficiency. A provision may refer to: o statistical activities ; o a methodology ; o an IT tool ; o a classification ; o a definition .

Statistical activities

A provision may or may not concern a given statistical activity , or to many of them (cardinality: 0,n). A given statistical domain can be referred by no provision, or by many provisions (cardinality: 0,n). The different statistical activities considered in this scheme are a subset of the complete set that has been for the Global Inventory of UNECE 12 . In particular, this subset contains all the subject matter areas: o D1: demographic and social statistics, o D2: economic statistics, o D3: environment and multidomain statistics,

11 We consider the EUROSTAT quality dimensions, as they have been defined in Eurostat (2009) ESS Handbook for Quality Reports, 2009 Edition. 12 See the Annex I to the paper “Developing a Global Inventory of Statistical Standards” (UNECE, 18 August 2010). 20 with the exclusion of the other areas as: o D4: methodology of data collection, processing and dissemination; o D5: strategic and managerial issues of official statistics. This is why the items contained in D4 and D5 have been judged as out-of-the-scope with respect to statistical standards, or better represented by the GSBPM.

The attributes of the entity “ statistical activity ” are: 1. identifier: see the United Nations set of codes; 2. description: text (see the United Nations set of descriptions).

Methodology

In this context, we can define a “ methodology ”13 as “ a structured approach to solve a problem ”. More specifically, “a set of research methods and techniques applied to a particular field of study ”14 .

A given provision may or may not indicate one methodology (cardinality: 0,1). A given methodology, if present, must be indicated by one provision, in general by many of them (cardinality: 1,n). The attributes of the entity “methodology ” are: 1. identifier: short description; 2. description: extended description of the methodology; 3. condition(s) of application.

Method

In this context, we define a “method” as “ An established, habitual, logical, or prescribed practice or systematic process of achieving certain ends with accuracy and efficiency, usually in an ordered sequence of fixed steps ”15 . A given methodology may or may not indicate one method , in general more than one

13 OECD Glossary of Statistical Terms, available at : http://stats.oecd.org/glossary/search.asp 14 Statistics Canada Glossary : http://www.statcan.gc.ca/edu/power-pouvoir/glossary-glossaire/5214842-eng.htm 15 http://www.businessdictionary.com/definition/method.html

21 (cardinality 0,n). A given method has to be linked to at least one methodology, or to many of them (cardinality: 1,n). The attributes of the entity “ method ” are: 1. identifier: short description; 2. description: extended description of the method; 3. condition(s) of application. Relationships between methodologies and methods are difficult to define in a univocal way. Basically, a methodology should indicate a general approach, that may include a set of methods enabling to follow that approach 16 .

IT tool

In this context, we define an “IT tool” as “ a software application or system, implementing one or more methods ”. A given provision may or may not indicate one IT tool (cardinality: 0,1). A given IT tool, if present, must be indicated by one provision, in general by many of them (cardinality: 1,n). The attributes of the entity “ IT tool ” are: 1. identifier: name or acronym; 2. description: text; 3. organisation. A given IT tool may or may not implement a given method , or many of them (cardinality: 0,n). A given method may or may not be implemented in a given IT tool, or in many of them (cardinality: 1,n).

Classification

We define a classification as a “a set of discrete, exhaustive and mutually exclusive observations, which can be assigned to one or more variables to be measured in the collation and/or presentation of data ”17 A given provision may or may not indicate one classification (cardinality: 0,1). A given classification, if present, must be indicated by one provision, in general by many of them (cardinality: 1,n). The attributes of the entity “ classification ” are: 1. identifier: name or acronym; 2. description: text; 3. version.

16 For instance, in the field of editing and imputation the so-called “ Fellegi-Holt methodology ” is a general approach following the principle of the minimum change . Minimum change can be obtained by adopting the Fellegi-Holt error localisation method (“ For each failed edit record, the Fellegi-Holt approach first proceeds through a step of error localisation in which it determines the minimal set of variables (fields) to impute, as well as the acceptable ranges(s) of values to impute .”) and the Fellegi-Holt imputation method (“ A single donor is selected from passed edit records by matching on the basis of other variables involved in the edits ”).

17 OECD Glossary of Statistical Terms 22

Definition

A definition is “ a statement of the precise meaning of something ”18 . A given provision may or may not indicate one definition (cardinality: 0,1). A given definition, if present, must be indicated by one provision, in general by many of them (cardinality: 1,n). The attributes of the entity “ definition ” are: 1. identifier: name or acronym; 2. description: text; 3. version.

18 OECD Glossary of Statistical Terms 23

2.1.2. The IT tool for the management of the repository

The conceptual scheme has been implemented in a relational database, whose logical scheme is reported in figure 2.

Figure 2 – Structure of the repository

In order to allow each partner in the ESSnet project to populate the repository in an assisted way, a dedicated software was developed. Here we report only the most important characteristics of the software.

General flow of the operations

Running the application, the first form that will appear is the “general menu” form. The dropdown list located just under the handbook image permits to select the institute (or the specific agent) identifier. At the moment the two section buttons are not enabled.

After the identification of the Institute, the system permits to move in one of the two management

24 sections: “Documents” (or handbooks) and “Methods”.

It was suggested to proceed in this way: 1. when analysing a given methodological handbook, start to select the “Documents” section, and proceed to introduce all necessary information related to the entities in the scheme up to the entity “methods” excluded; 2. once a new method has to be introduced, leave the “Documents” section and, from the “Main menu” select “Methods” section; 3. once the methods have been introduced, then get back to the “Documents” section and proceed to link methodologies to methods.

Documents List Form

The documents list form, as the methods one, is divided in two areas. The search frame permits to display the documents by title, type, body, ID, version, consensus, lever and aim. All these criteria are in “and” search condition. By pressing the “New” button, it is possible to insert a new instance for the “Document” entity.

Provisions Groups List Form

Once a “Document” has been introduced, the Provisions Groups List Form appears. Also in this case, by clicking on the “New” button, it is possible to insert a new instance.

25 Provisions Form

For each selected Provision Group, it is possible to define new instances of “provisions”. Once introduced, for each of them the values of the attributes are reported in the lower part of the form.

In this lower part, it is possible also to associate “Methodologies”, “Tools”, “Classifications” and “Definitions”. If one of these objects does not yet exist, by clicking on the corresponding “…” button it will be possible to introduce a new one that from now on will be available. It is also possible to associate one or more “Statistical activities” by clicking on “List” button: the list of statistical activities appear on the upper part of the form, and it is possible to select the ones required.

Methods List Form

From the initial form, it is possible to select the “Methods” form. In this form all the methods introduced so far will appear. The “New” button permits to add a new method. To select the displayed method the user has to click in any cell of the selected method. When a method has been selected, it will be possible to associate the methodologies to which it belongs.

26

2.1.3. Procedure followed for the definition and refinement of repositories for single handbooks and for the construction of the unified repository

In order to analyse the methodological handbooks, each of them was assigned to different Institutes:

Handbook Organisation(s) carrying out the Organisation carrying out the analysis reconciliation Handbook of recommended practices ONS & Insee ONS for questionnaire development and testing in the European Statistical System Eurostat sampling reference guidelines: Destatis - Introduction to sample design and estimation techniques Survey methods and practice (Stat.Can.) Istat & ONS ISTAT Guidelines for statistical metadata on Insee - the Internet ESS handbook for Quality reports GUS &KSH KSH Edimbus: editing and imputation of KSH & Destatis Destatis cross sectional business statistics

In some cases, the analysis (i.e. the introduction into the repository of the information required for each entity in the scheme) has been carried out by a single Institute. In some other cases, the analysis has been performed by two Institutes. That is why we wanted to compare the results, analyse the differences and assess the degree of arbitrariness in the process. In order to build a unique repository for all the handbooks, for the three cases in which the analysis has been done by two partners, to one of them the responsibility for a reconciliation step was assigned. The partner responsible for the compared analysis had to detect: 1. inconsistencies between the two repositories, that could regard • provisions: differences in definition and/or attributes • other objects (methodologies, methods, IT tools, etc.): differences in definitions and/or in links 2. non-correspondences (missing items in one repository with respect to another). The responsible partner had to list all the identified problems, and discuss them with the other partner, in order to agree on the final result, that is a unique repository of validated provisions and other objects, for the given handbook.

Once for each handbook a unique repository has been defined, then it has been possible to build the final, unified repository containing all the information derived by the analysis of the six methodological handbooks. This final repository has been analysed in order to describe its content and also to illustrate with some examples how it is possible to use it in order to obtain useful information regarding provisions coming from different handbooks and dealing with the same field of application.

27

2.2. Analysis of the six methodological handbooks

In this sub-chapter, the analyses on the different handbooks are reported, accordingly to common structures. In the case where only one Institute was in charge to analyse a given handbook (this is the case for handbooks “Eurostat sampling reference guidelines ” and ”Guidelines for statistical metadata on the Internet”), we have 1. a general description of the handbook and a list of the analysed chapters; 2. the number of provisions and of the groups in which they have been considered together; 3. the indication of the GSBPM main sub-processes; 4. a summary of the entities linked to provisions in the handbook (methodologies, methods, IT tools, classifications and definitions), plus general comments to better illustrate the results of the analysis, or the way it has been carried out. In the case where two Institutes have produced two distinct analyses (the remaining four handbooks), the above items are referred to the “reconciled” version of the analysis, but before of this a compared analysis of the common chapters and divergences encountered have been reported. For each common chapter, if different approaches have been followed, they have been outlined, and examples of divergences are reported. This is an important basis to understand the degree of arbitrariness to which the operation of analysis of a methodological handbook, following the adopted conceptual scheme, is subject.

28

2.2.1. Eurostat sampling references guidelines

General description of the handbook and list of the analysed chapters

The one showed in the following table is the handbook “ Eurostat sampling reference guidelines ” analysed by DESTATIS alone.

Table 1. Handbook-attributes

Attributes of the entity “normative document” Title of the document (of the handbook) Survey Sampling and Reference Guidelines, Introduction to sample design and estimation techniques Version: indication of the version, if available 2008 (otherwise, year of publishing) Type: standard, pre-standard, technical Code of practice, Guidelines specification, code of practice, regulation Level: international (=global), regional (e.g.: International European), national, provincial Body: organisation(s) responsible for the Eurostat normative document Consensus: if any, indication of the agreement Eurostat Aim These guidelines give an overview of basic principles and methods of survey sampling. This includes survey planning, survey quality, sampling, estimation and treatment of non-response. Also, references to statistical software (IT tools) are given.

The handbook is composed of 7 chapters, which have been analysed as reported in Table 2.

Table 2. Analysed chapters

Chapters analysed by DESTATIS 1. Introduction 2. Survey planning and reporting 3. Techniques for sample selection and estimation 4. Treatment of nonresponse 5. Software 6. References 7. Web links

29 Only chapters 3 to 5 provided relevant information for the identification of provisions, methodology and methods.

Overall description of the analysed handbook

Table 3. General characteristics of the chapters

Number of Provision Main GSBPM Number of groups identified Provisions :

CHAPTER 3 1 4.1 3 CHAPTER 4 1 5.6 2

Comment.

In order to detect provisions and provision groups inside the analysed document, a general bottom- up-approach was applied. This means that first the provisions were identified, as these are most elementary items according to the data model of the data base. Secondly, these provisions were logically and according to GSBPM sub-processes grouped into provision groups. The appearance within the chapters was considered as not so important. Of course, top-down would also have been possible instead of bottom-up, but in this case the data model should be more flexible and allow for sub-provisions and sub-sub-provisions etc. The importance of a recommendation on the search strategy is that it should (in theory) lead to similar results when the analysis is carried out in parallel by different people (NSIs).

Table 4. Summary of the entities linked to provisions in the handbook

Entity Total number List Methodologies 5 • Probability Sampling • Sample size Allocation to strata • Estimation Techniques • Reweighting in case of unit nonresponse • Imputation Methods 21 • Cluster Sampling (CLU) • Probability-Proportional-to-Size (PPS) Sampling • Simple Random Sampling (SRS) • Stratified Sampling (STR) • Systematic Sampling (SYS) • Equal Allocation • Optimal Allocation (Neyman) • Power Allocation (Bankier)

30 • Proportional Allocation • Horvitz-Thomson estimation • Ratio Estimation • Regression Estimation • Post-Stratification • GREG-Estimation • Direct Reweighting • Empirical Response Homogeneity Groups (RHG) • Explicit Response Probability Modelling • Calibration of weights • Logical Imputation • Real-Donor Imputation • Model-Donor Imputation It-tools 31 see below Classifications

Definitions

Comment.

Also, for the identification of methods and methodologies, the above mentioned bottom-up- approach was applied here. This means that it was started with the detection of methods as the most elementary items in the data model and then the logical grouping of the methods to methodologies was carried out. Also here, a recommendation on the search strategy is considered to be of great importance in order to come to “unique” solutions.

In order to show the linkage of the IT-tools, mentioned in the handbook, to the methods and methodologies, a more complete overview is given.

It-tools

Two large scale statistical programs, SAS and SPSS contain specific modules for sample selection and survey data analysis. In SAS the procedures are included in STAT module from version 8 on. In SPSS there is an add-on module Complex Samples which must be purchased separately, available from version 11 and above. The programming language R is widely used in academic studies and some authors have provided codes for survey sampling and analysis. Those are found on different web-pages containing R program codes, e.g. www.r-project.org/ .

Software for sample selection SAS/Stat v. 8 and above: Proc SurveySelect (www.sas.com) SPSS/Complex Samples v. 11 and above: CSPLAN together with CSSELECT (www.spss.com)

Software for weight derivation SAS add-on programmes: CALMAR2 (INSEE/France), CLAN97 (Statistics Sweden) SPSS add-on programmes: g-CALIB (Statistics Belgium) BLAISE component BASCULA (Statistics Netherlands) 31

Software for editing and imputation SAS/Stat v. 8 and above Proc MI and MIANALYZE (multi-imputation) SAS add-on programmes: BANFF/GEIS (Statistics Canada), CONCORD, DIESIS, QUIS (ISTAT/Italy), IVEWARE (www.isr.umich.edu/src/smp/ive/)

Software for “standard” estimation SAS/Stat v. 9 and above: Proc Surveymeans, Proc Surveyfreq SAS add-on programmes: CLAN97 (Statistics Sweden), POULPE (INSEE/France), GES (Statistics Canada) SPSS v. 11 and above: CSDESCRIPTIVES, CSTABULATE SPSS add-on programmes: g-CALIB (Statistics Belgium) BLAISE component: BASCULA (Statistics Netherlands) SUDAAN ( www.rti.org )

Software for analytical purposes SAS/Stat v. 9 and above: Proc Surveyreg, Proc Surveylogistic, (GLM, MIXED, NLMIXED,…) SPSS v. 11 and above: CSGLM, CSORDINAL, CSLOGISTIC SUDAAN WesVAR ( www.westat.com )

32

2.2.2. Recommended practices for editing and imputation in cross-sectional business surveys (EDIMBUS)

General description of the handbook and list of the analysed chapters

The one showed in the following table is the handbook “Recommended practices for editing and imputation in cross-sectional business surveys” analysed jointly by KSH and Destatis.

Table 5. Handbook attributes

Attributes of the entity “normative document” Title of the document (of the handbook) Recommended practices for editing and imputation in cross-sectional business surveys Version: indication of the version, if available 2007 (otherwise, year of publishing) Type: standard, pre-standard, technical Code of practice specification, code of practice, regulation Level: international (=global), regional (e.g.: Regional (European) European), national, provincial Body: organisation(s) responsible for the ISTAT, CBS, SFSO normative document Consensus: if any, indication of the agreement - Aim Development of a common practical and methodological framework for editing and imputation applications in business statistics

The handbook is composed of 7 chapters, a subset of which have been analysed as reported in table 6 where the chapters in common are highlighted in bold .

Table 6. Analysed chapters

Chapters analysed by Destatis 3. Detection of Errors 4. Treatment of Errors 5. Subsequent Analysis and Estimation

33

Chapters analysed by KSH 3. Detection of Errors 4. Treatment of Errors 5. Subsequent Analysis and Estimation

Comparison of the common chapters and analysis of the divergences encountered

First common chapter: “Chapter 3: Detection of Errors”

The chapter regards the identifying of values that are not acceptable with respect to some pre- defined logical, mathematical or statistical criteria. Since different error types may contaminate the observed data in a sample of units, usually the error detection process consists of a set of integrated error detection methods dealing each with a specific type of error.

The standards individuated by the two NSI’s in chapter 3 can be summarized in table 7.

Table 7. Summary

Destatis KSH Total number of Provision 6 6 Groups Main GSBPM 5.3. Review, validate and edit 5.3. Review, validate and edit

Secondary GSBPM - -

Collocation Chapter 3 Chapter 3 Number of identified 6 statements 5 statements Provisions 11 instructions 11 instructions statements 14 recommendations 15 recommendations instructions 1 deemed-to-satisfy provision recommendations etc. Number of Methodologies 5 7

The main difference in Chapter 3 is that KSH specifies the approaches “Macroediting” and “Fellegi-Holt paradigm” as methodologies while Destatis specifies them as methods. Another difference in this chapter is that the provision contained in the provision group “Detection of errors” refers to a classification in the case of Destatis and to a methodology in the case of KSH.

Second common chapter: “Chapter 4: Treatment of Errors”

The chapter regards the replacement of values that are believed to be in error with more plausible values. The standards individuated by the two NSI’s in chapter 4 can be summarized in table 8 .

34

Table 8. Summary

Destatis KSH Total number of Provision 2 1 Groups Main GSBPM 5.4. Impute 5.4. Impute

Secondary GSBPM - -

Collocation Chapter 4 Chapter 4 Number of identified 2 2 Provisions 2 statements 2 statements statement instructions recommendation etc. Number of Methodologies 2 2

One main difference in chapter 4 is that KSH specifies the whole chapter 4 as one provision group, while Destatis specifies the two sections of chapter 4 “Imputation” and “Interactive treatment” each time as a provision group. Another main difference is the varying level of detail of specifying the methods the methodology “Imputation” is based on. While KSH specifies 12 methods connected to the methodology “Imputation”, Destatis subsumes them in 4 methods. The following table 9 shows the higher level of detail specified by KSH in this case.

Table 9. Example of the different approaches

Destatis KSH Provision group 8 (Imputation) 7 (Imputation) Provision 1 1 Collocation Pages 30-34 Pages 30-34 Description Missing data resulting from nonre- Missing data resulting from nonre- sponse or values flagged as erroneous sponse or values flagged as erroneous during editing may be treated by during editing may be treated by imputation. Imputation consists of imputation. Imputation consists of replacing these values by plausible replacing these values by plausible ones to meet the demands on data ones to meet the demands on data quality for analysis and dissemination. quality for analysis and dissemination.

Type statement statement Sequence none none Methodology Imputation Imputation

35 Description Missing data resulting from non- Missing data resulting from non- response or erroneous data may be response or erroneous data may be treated by imputation. treated by imputation.

Quality dimensions Accuracy and reliability, timeliness Accuracy and reliability, timeliness and punctuality, coherence and and punctuality, coherence and comparability, efficiency comparability, efficiency Methods Donor-based imputation Cold-deck imputation Imputation cells Deductive imputation Model based imputation Donor-based imputation Rule-based imputation Hot-deck imputation Imputation cells Mean imputation Model based imputation Nearest neighbour imputation Random donor imputation Ratio imputation Regression imputation Rule-based imputation

Third common chapter: “Chapter 5: Subsequent Analysis and Estimation”

The chapter regards the estimation and analysis of data which contain imputed values. These values can not be treated as truly observed because the imputations may lead to biased parameter estimates. Additionally sometimes some missing values and outliers are explicitly not treated by the editing and imputation process because it is considered more appropriate to deal with them by special estimators that take missing values or outliers into account; chapter 5 also deals with such estimators. The standards individuated by the two NSI’s in chapter 5 can be summarized in table 10.

Table 10. Summary

Destatis KSH Total number of Provision 2 2 Groups Main GSBPM 6.3. Scrutinize and explain 6.3. Scrutinize and explain

Secondary GSBPM - -

Collocation Chapter 5 Chapter 5 Number of identified 2 2 Provisions 2 statements 2 statements statement instructions recommendation

36 etc.

Number of Methodologies 2 2

The comparison of chapter 5 shows no specification differences at all. Both NSI´s identified identi- cal provision groups, provisions, methodologies and methods.

Concluding remarks on the compared analysis

The compared analysis of the Edimbus-handbook “Recommended practices for editing and imputation in cross-sectional business surveys” of KSH and Destatis revealed altogether only small differences. One reason may be that both partners were guided by the example of ISTAT of filling in the ISTAT model on Chapter 6 of the handbook “Survey Methods and Practices”.

Taken together there were very small differences in stated provision groups, small differences in specified methodologies, bigger but still moderate differences in specified methods, slightly to moderate differences in descriptions of methodologies and methods as well as no differences at all in allocation of generic statistical business process models.

The main reasons for the encountered divergences are the varying level of detail in specifying (imputation) methods and the problem of specifying an approach or an entity as a methodology versus specifying it as a method.

Description of the reconciled version of the common chapters

On the one hand the reconciled version follows KSH in a higher level of detail especially for the methods linked to the methodologies "Detection of outliers" and "Imputation". Additionally the specified method "Deterministic checking rules" of KSH (connected with the methodology "Localizing random errors") is incorporated and the name of the methodology "Edit checks or checking rules" by Destatis is changed to "Detection of missing values" by KSH. On the other hand the reconciled version follows Destatis in specifying "Macroediting" and the "Fellegi-Holt paradigm" as methods. Furthermore the provision group “Detection of errors” or more specifically the contained provision refers to a classification and not to a methodology. In addition the reconciled version of the handbook follows Destatis in specifying chapter 4 as containing two provision groups. The following tables so summarize the final versions of chapters 3, 4 and 5 of the Edimbus-handbook.

Final version of “Chapter 3: Detection of Errors”

Table 11. Summary of the final version of chapter 3

Final shared version

37 Total number of Provision Groups 6

Main GSBPM 5.3. Review, validate and edit

Secondary GSBPM -

Collocation Charter 3 Number of identified Provisions 31 statements 6 statements instructions 11 instructions recommendations 14 recommendations descriptive provision Number of Methodologies 5

Final version of “Chapter 4: Treatment of Errors”

Table 12. Final version of chapter 4

Final shared version Total number of Provision Groups 2

Main GSBPM 5.4. Impute

Secondary GSBPM -

Collocation Chapter 4 Number of identified Provisions 2 instruction 2 statements statement descriptive provision Number of Methodologies 2

Final version of Chapter 5: Subsequent Analysis and Estimation”

Table 13. Final version of chapter 5

Final shared version Total number of Provision Groups 2

Main GSBPM 6.3. Scrutinize and explain

Secondary GSBPM -

Collocation Chapter 5

38 Number of identified Provisions 2 instruction 2 statements statement descriptive provision Number of Methodologies 2

Overall description of the analysed handbook

Table 14. General characteristics of the chapters

Number of Main GSBPM Number of Provision groups identified Provisions :

CHAPTER 3 (common) 6 5.3. Review, validate 6 statements and edit CHAPTER 4 (common) 2 5.4. Impute 2 statements

CHAPTER 5 (common) 2 6.3. Scrutinize and 2 statements explain

Table 15. Summary of the entities linked to provisions in the handbook

Entity Total number List Methodologies 9 Detection of missing values Detection of systematic errors Detection of influential errors Detection of outliers Localizing random errors Imputation Interactive treatment Variance estimation in the presence of imputed data Estimation in the presence of missing values and outliers Methods 29 Analysis of fatal edits Analysis of ratio edits Finite mixture models Macroediting Selective editing Graphical editing Hidiroglou-Berthelot method Median-absolute-deviation rule One-step robustified ratio

39 estimator Regression based models Weighted quantiles Deterministic checking rules Fellegi-Holt paradigm Cold-deck imputation Deductive imputation Hot-deck imputation Imputation cells Mean imputation Nearest neighbour imputation Random donor imputation Ratio imputation Regression imputation Rule-based imputation Analytical methods Multiple imputation Resampling methods Censored and winsorized means EM-Algorithm M-Estimator It-tools - Classifications 1 Detection of errors

Definitions -

40

2.2.3. Guidelines for statistical metadata on the Internet

General description of the handbook and list of the analysed chapters

The one showed in the following table is the handbook “Guidelines for statistical metadata on the interne” analysed only by INSEE.

Table 17. Handbook attributes

Attributes of the entity “normative document” Title of the document (of the handbook) Guidelines for statistical metadata on the internet Version: indication of the version, if available 2000 (otherwise, year of publishing) Type: standard, pre-standard, technical Code of practice specification, code of practice, regulation Level: international (=global), regional (e.g.: International European), national, provincial Body: organisation(s) responsible for the United Nations Statistical Commisssion and normative document Economic Commission for Europe Consensus: if any, indication of the agreement UNECE, National Statistical Offices members of the UNECE, FAO, IMF, UNSD Aim The Guidelines for Statistical Metadata on Internet make recommendations for a further harmonisation accompanying the statistical information on Internet.

The handbook is not formally composed of chapters, however to better organise the analysis we have considered the paragraphs as chapters.

Table 18. Analysed chapters

Chapters analysed by INSEE 1. Metadata issues specific to the Internet 2. Users of the Internet 3. Guidelines for different types of metadata

41

Chapter 1 : “Metadata issues specific to the Internet”

The chapter regards the basic requirements for metadata on the Internet.

The standards individuated can be summarized in table 19 …..

Table 19. Summary

Total number of Provision 1 Groups Main GSBPM 7.2. Produce dissemination products Secondary GSBPM 7.3. Manage release of dissemination products Collocation P7-8 Section II Number of identified 6 Provisions 4 statements statement 2 recommendations instructions recommendation etc. Number of Methodologies 0

The provision group is actually the chapter. The same information is then to be reported.

Chapter 2 : “Users of the Internet”

Table 20. Summary

Total number of Provision 1 Groups Main GSBPM 7.4. Promote dissemination products Secondary GSBPM

Collocation P 8-9 Section III Number of identified 2 Provisions 1 statement statement 1 recommendation instructions recommendation etc. Number of Methodologies 0

42

Chapter 3: “Guidelines for different types of metadata”

Table 21. Summary

Total number of Provision 1 Groups Main GSBPM 7.4. Promote dissemination products Secondary GSBPM

Collocation P 9-12 Section IV Number of identified 12 Provisions 1 statement statement 7 recommendations instructions 2 Decriptive provisions recommendation 1 instruction etc. 1 Requirement Number of Methodologies 0

Overall description of the analysed handbook

Table 22. General characteristics of the chapters

Number of Provision Main GSBPM Number of groups identified Provisions :

CHAPTER 1 (by NSI: 1 7.2. Produce 4 statements INSEE ) dissemination products 2 recommendations CHAPTER 2 1 7.4. Promote 1 statement (by NSI: INSEE) dissemination products 1 recommendation CHAPTER 3 1 7.4. Promote 1 statement (by NSI: INSEE) dissemination products 7 recommendations 2 Decriptive provisions 1 instruction 1 Requirement

43 Table 23. Summary of the entities linked to provisions in the handbook

Entity Total number List Methodologies 0 Methods 0 It-tools 0 Classifications 0

Definitions 2 - Basic definition of the external users of statistics - Definition of external users of statistics according to their level of skills in statistics

44

2.2.4. ESS Handbook for Quality Reports

General description of the handbook and list of the analysed chapters

The one showed in the following table is the handbook “ESS Handbook for Quality Reports” analysed jointly by KSH and GUS.

Table 24. Handbook attributes

Attributes of the entity “normative document” Title of the document (of the handbook) ESS Handbook for Quality Reports Version: indication of the version, if available 2009 (otherwise, year of publishing) Type: standard, pre-standard, technical Code of practice specification, code of practice, regulation Level: international (=global), regional (e.g.: Regional European), national, provincial Body: organisation(s) responsible for the Eurostat, Working group on Quality normative document Consensus: if any, indication of the agreement Aim The general aim of this Handbook (EHQR) is to provide guidelines for preparation of comprehensive quality reports for a full range of statistical processes and their outputs.

The handbook is composed of 3 parts, 18 chapters chapters, one of which have been analysed as reported in Table 254.

Table 25. Analysed chapters

PART II: GUIDELINES FOR QUALITY Chapters analysed by HCSO REPORTS, Chapter 3. Accuracy

PART II: GUIDELINES FOR QUALITY Chapters analysed by GUS REPORTS, Chapter 3. Accuracy

45

Comparison of the common chapters and analysis of the divergences encountered

Common chapter: Part II, Chapter 3. Accuracy

The chapter regards methods and indicators measuring accuracy of different statistical processes.

The standards individuated by the two NSI’s in the chapter 3 can be summarized in table 26.

Table 26. Summary

KSH GUS Total number of Provision 8 7 Groups Main GSBPM 10.1 Quality management 4.1 Select sample 4.4 Finalize collection 6.5 Finalize outputs 5.4 Impute Secondary GSBPM 2.4 Design frame and sample 5.1 Integrate data methodology 5.3 Review, validate and edit 5.8 Finalize data files Collocation Part II, Chapter 3 Part II, Chapter 3 Number of identified 193 11 Provisions statement 118 11 instructions recommendation 75 etc. Number of Methodologies 6 5

KSH and GUS followed completely different approaches: GUS identified some key points of the chapter while KSH tried to identify most of the statements and recommendations included in the chapter.

Table 27. Example of the different approaches

KSH GUS Provision group 2 3 Provision Accuracy for sample surveys Probability sampling Collocation page 35 page 36 Description Sampling can be of two types: probability sampling, meaning that each unit of the frame population has a known, non-zero probability of being selected in the sample, and nonprobability sampling. Type statement statement 46 Sequence - - Methodology - Probability sampling Description Probability sampling, meaning that each unit of the frame population has a known, non-zero probability of being selected in the sample. Quality dimensions Accuracy Accuracy

Methods - -

Concluding remarks on the compared analysis

GUS regrouped the calculation of quality indicators to one provision group, KSH treated the quality indicators as provisions (with an indication „indicator”) in the related thematic provision groups. Division of the chapter to provision groups was also very different: in the case of KSH provision groups cover all the chapter, GUS identified some key points of the chapter. Thus comparison of provision groups is challenging.

Description of the reconciled version of the common chapters

At the Rome meeting it was agreed that a detailed analysis is useful, thus the reconciled version was based on the analysis made by KSH, completed with the approach of GUS to include all quality indicators to one provision group. Similarly, recommendations for general description of different aspects of quality to be included in quality reports were regrouped to one provision group.

Final version of the first common chapter

Table 28. Summary of the final version of the first common chapter

Final shared version Total number of Provision Groups 10

Main GSBPM 10.1 Quality management

Secondary GSBPM 2.4 Design frame and sample methodology

Collocation Part II, Chapter 3 Number of identified Provisions 196 statement 105 recommendation 91

Number of Methodologies 10

47 In the reconciled version more methods and methodologies were identified and were linked to provisions. The detailed analysis was kept, but examples were removed from provision groups. Two special provision groups were created: one for quality indicators to be calculated and one for general description to be included in the quality report.

Overall description of the analysed handbook

Table 29. General characteristics of the chapters

Number of Provision Main GSBPM Number of groups identified Provisions :

PART II, CHAPTER 3 8 10.1 Quality statement : 118 (by KSH) management recommendation : 75 PART II, CHAPTER 3 7 4.1 Select sample statement : 11 (by GUS) 4.4 Finalize collection recommendation :- 6.5 Finalize outputs 5.4 Impute PART II, CHAPTER 3 10 10.1 Quality statement : 105 (common ) management recommendation : 91

As the handbook is about quality reporting the main GSBPM remained “10.1 Quality management”, but it woud fit also in “9. Evaluating”. We agreed that secondary and auxiliary GSBPM would be the GSBPM step corresponding to the referred process, but at the end GSBPM categories were not modified.

Table 30. Summary of the entities linked to provisions in the handbook

Entity Total number List Methodologies 10 Evaluation of coverage errors Assessment of accuracy in sample survey Assessment of bias Consistency studies Detection of coverage errors Estimation of expected value and variance of indicators Evaluation of nonresponse error Evaluation of processing errors Quality assessment for statistical compilations Sampling Methods 36 Analysis of lag structure Assessment of errors in register variables Assessment of respondents errors

48 Comparing datasets Designing and testing of questionnaires Estimation of the revision size Information from data editing Matching with a different register Reinterviewing Replacement, re-sampling and quality adjustment methods for price statistics Subject economic data to accounting rules Coefficient of variation (CV) Estimation of bias Root mean square error (RMSE) Standard error Qualitative assessment of bias Mirror statistics Assessment of coverage errors in business registers Assessment of coverage errors in household survey Assessment of under-coverage Checking frame information Detection of overcoverage Handling of multiple listings Confidence interval Complementing with register data Special data collections Variations over response waves Studies of coding variation Studies of effects of editing Analysis of revision Analysis of statistical discrepancy Data Quality Assessment Framework (DQAF) of the International Monetary Fund (IMF) Measurement of non-observed economy Cut-off sampling Non-probability sampling Other forms of non-probability sampling It-tools - Classifications 6 Errors Non-sampling errors Types of sampling Types of coverage errors Categories of census Register-based statistics Definitions 11 Census Statistical process using administrative sources Statistical process involving multiple data sources Price or other economic index process Quality adjustment for price indexes Statistical compilations Revision

49 Seasonal adjustment Imputation Mistake Code of practice

Based on the last report prepared by ISTAT on the analysis of handbooks, the final analysis should be improved, especially classification of provision groups to GSBPM processes. In the case of this handbook, main GSBPM is “quality management” for all provision groups. As far as secondary and auxiliary GSBPM of provision groups are concerned the main idea was to relate the provision group to the GSBPM it refers to. The problem in our case is that provisions groups identified are too complex, thus they refer to several GSBPM sub processes. In the database only one main, one secondary and one auxiliary GSBPM process can be associated to a provision group. In the table below GSBPM processes are associated to each provisions group.

Table 31. Provisions groups and GSBPM processes.

Provision group Main GSBPM Secondary GSBPM 1. General characteristics of and 10.1. Quality Management 2.4. Design frame and guidelines for accuracy in the QR sample methodology 4.1. Select sample 4.3. Run collection 4.4. Finalize collection 5.2. Classify and code 5.3. Review, validate and edit 5.4. Impute 5.8. Finalize data files 6.5. Finalize outputs 9.2. Conduct evaluation 2. Accuracy for sample surveys 10.1. Quality Management 2.4. Design frame and sample methodology 4.1. Select sample 4.3. Run collection 4.4. Finalize collection 5.2. Classify and code 5.3. Review, validate and edit 5.4. Impute 5.8. Finalize data files 6.5. Finalize outputs 9.2. Conduct evaluation 3. Accuracy for Censuses 10.1. Quality Management 4.3. Run collection 4.4. Finalize collection 5.2. Classify and code 5.3. Review, validate and edit 5.4. Impute 5.8. Finalize data files 6.5. Finalize outputs

50 9.2. Conduct evaluation 4. Accuracy for statistical processes 10.1. Quality Management 5.3. Review, validate and using administrative sources edit 5.4. Impute 5.8. Finalize data files 6.5. Finalize outputs 9.2. Conduct evaluation 5. Accuracy For Statistical 10.1. Quality Management 2.4. Design frame and Processes Involving Multiple Data sample methodology Sources 4.1. Select sample 4.3. Run collection 4.4. Finalize collection 5.2. Classify and code 5.3. Review, validate and edit 5.4. Impute 5.8. Finalize data files 6.5. Finalize outputs 9.2. Conduct evaluation 6. Accuracy for Price and Other 10.1. Quality Management 2.4. Design frame and Economic Index Processes sample methodology 4.1. Select sample 4.3. Run collection 4.4. Finalize collection 5.2. Classify and code 5.3. Review, validate and edit 5.4. Impute 5.8. Finalize data files 6.5. Finalize outputs 9.2. Conduct evaluation 7. Accuracy For Statistical 10.1. Quality Management 5.3. Review, validate and Compilations edit 5.4. Impute 5.8. Finalize data files 6.5. Finalize outputs 9.2. Conduct evaluation 8. Some special issues concerning 10.1. Quality Management 5.4. Impute accuracy 5.8. Finalize data files 6.1. Prepare draft output 6.5. Finalize outputs 9.2. Conduct evaluation 9. Quality and performance 10.1. Quality Management 9.2. Conduct evaluation indicators for accuracy 10. Items to be included in the 10.1. Quality Management 9.2. Conduct evaluation Quality report for accuracy

51 2.2.5. Recommended practices for questionnaire design and testing

General description of the handbook and list of the analysed chapters

The following table describes the “Handbook of recommended practices for questionnaire development and testing in the European Statistical System” analysed by ONS and INSEE.

Table 32. Handbook attributes

Attributes of the entity “normative document” Title of the document (of the handbook) Handbook of recommended practices for questionnaire development and testing in the European Statistical System Version: indication of the version, if available Release year 2006 (otherwise, year of publishing) Type: standard, pre-standard, technical Code of practice specification, code of practice, regulation Level: international (=global), regional (e.g.: Regional European), national, provincial Body: organisation(s) responsible for the European Commission normative document Consensus: if any, indication of the agreement ESS Aim: The recommended practice manual aims at further specifying the requirements of the European Statistics Code of Practice and defines systematic questionnaire design and testing procedures to minimise measurement error.

The handbook is composed of 7 chapters of which ONS and INSEE have analysed two.

A general comment is that the initial ONS and INSEE analyses were very different and therefore it is difficult to provide meaningful comparisons in any detail. Therefore the two NSIs worked collaboratively to produce the final combined analysis, starting from scratch rather than building on the initial analyses.

Comparison of common chapters

First common chapter: Chapter 2. The cognitive model in answering questions

This chapter describes the application of cognitive research to statistical surveys to provide insight into the cognitive dimensions of the response process.

The standards identified by ONS and INSEE in chapter 2 are summarised in table 33.

52 Table 33. Summary

ONS INSEE Total number of Provision 1 1 Groups Main GSBPM 2.3. Design data collection 2.3. Design data collection methodology methodology Secondary GSBPM 3.1.Build data collection 3.1. Build data collection instrument instrument Collocation Chapter 2 Chapter 2 Number of identified 1 10 Provisions 0 statements 7 statements 0 instructions 2 instructions 1 recommendation 1 recommendation Number of Methodologies 0 8

As can be seen from the table, ONS and INSEE took quite different approaches to defining provisions. Additionally, ONS did not complete the methods selection for this chapter. On investigation both NSIs identified ways in which their analyses were flawed and therefore agreed that direct comparisons are not necessary.

Second common chapter: Chapter 3: Questionnaire design

This chapter describes the process of questionnaire design, namely the development of the conceptual frame of the questionnaire, writing and sequencing of questions and establishing the visual design elements of the questionnaire.

The standards identified by ONS and INSEE in chapter 3 are summarised in table 34.

Table 34. Summary for second common chapter

ONS INSEE Total number of 6 1 Provision groups Main GSBPM 2.3. Design data collection 3.1. Build data collection methodology instrument Secondary GSBPM 3.1 Build data collection None instrument Collocation Chapter 3. Chapter 3.

Number of identified 17 58 Provisions 0 statements 47 statements 0 instructions 2 instructions 17 recommendation 9 recommendations Methodologies 16 36

53

As previously, the two analyses are clearly very different. In this chapter, many of the same terms were identified, but at different levels. For example, many provisions identified by INSEE were identified as methodologies by ONS.

Concluding remarks on the compared analysis

This comparison demonstrated that ONS and INSEE were using different approaches to identifying provisions and methodologies. Following discussion, the NSIs agreed that both approaches could be improved and worked collaboratively towards an agreed analysis.

Description of the reconciled version of the common chapters

Final version of the first common chapter

Table 35. Summary of the final version of the first common chapter: Chapter 2

Final shared version Total number of Provision Groups 1

Main GSBPM 2.3. Design data collection methodology

Secondary GSBPM 3.1. Build data collection instrument

Collocation Chapter 2 Number of identified Provisions 1 0 statements 1 recommendation 0 instructions Number of Methodologies 1

Final version of the second common chapter

Table 36. Final version of the second common chapter: Chapter 3

Final shared version Total number of Provision Groups 5

Main GSBPM 2.3. Design data collection methodology

Secondary GSBPM 3.1. Build data collection instrument

Collocation Chapter 3

54 Number of identified Provisions 17 5 statements 12 recommendations 0 instructions Number of Methodologies 14

Overall description of the analysed handbook

Table 37. General characteristics of the chapters

CHAPTER Number of Provision Main GSBPM Number of Groups Identified Provisions `CHAPTER 2 1. The Cognitive Response 2.3 Design Data Collection 1 Recommendation Process Methodology THE COGNITIVE MODEL IN ANSWERING 2. Conceptual Frame QUESTIONS

(Common ONS & INSEE) CHAPTER 3 3. Data Collection mode 2.3 Design Data Collection 16 Recommendations 4. Define Variables and Methodology QUESTIONNAIRE DESIGN Draft a Tabulation Plan 3.1 Build Data Collection 5. Principles for Writing instrument (Common ONS & INSEE) Questions 1.1 Determine needs for 6.Literature Search information.

Table 38. Summary of the entities linked to provisions in the handbook

Entity Total Number List Methodologies 11 CASI - Computer assisted self interviewing Collect and review literature Conceptual and entity relationship scheme Data collection Dealing with sensitive questions Focus group Interviewer assisted methods Need of respondent to appear in a certain way Recall and memory errors Respondent’s interpretation Self interviewing Methods 11 Output objectives

55 Conceptualisation CAPI CASI CATI PAPI Sensitivity effects In depth/qualitative research Social desirability Recall effect Build data collection instruments

56 2.2.6. Survey Methods and Practices

General description of the handbook and list of the analysed chapters

The following table describes the handbook “Survey methods and practices” analysed jointly by ISTAT and ONS.

Table 39. Handbook attributes

Attributes of the entity “normative document” Title of the document (of the handbook) Survey methods and practices Version: indication of the version, if available 2010 (otherwise, year of publishing) Type: standard, pre-standard, technical Code of practice specification, code of practice, regulation Level: international (=global), regional (e.g.: National European), national, provincial Body: organisation(s) responsible for the Statistics Canada normative document Consensus: if any, indication of the agreement Statistics Canada Aim: This manual is primarily a practical guide to survey planning, design and implementation. It covers many of the issues related to survey taking and many of the basic methods that can be usefully incorporated into the design and implementation of a survey.

The handbook is composed of 13 chapters which have been analysed as reported in table 40 where the chapters in common are highlighted in bold .

Table 40. Analysed chapters

Chapters analysed by ISTAT 4. Data Collection Methods 5. Questionnaire Design 6. Sample Designs 7. Estimation 8. Sample Size Determination and Allocation 10. Processing

57

Chapters analysed by ONS 1. Introduction to Surveys 2. Formulation of the Statement of Objectives 3. Introduction to Survey Design 4. Data Collection Methods 5. Questionnaire Design

Comparison of the common chapters and analysis of divergences

First common chapter: Chapter 4. Data collection

This chapter describes the process of gathering information in statistical surveys with particular attention to the different methods of data collection.

The standards identified by ISTAT and ONS in the chapter 4 are summarized in table 41.

Table 41. Summary

ISTAT ONS Total number of Provision 1 1 Groups Main GSBPM 2.3. Design data collection 2.3. Design data collection methodology methodology Secondary GSBPM 3.1.Build data collection 3.1. Build data collection instrument instrument Collocation Chapter 4 Chapter 4 Number of identified 5 6 Provisions 1 statements 2 statements 3 instructions 2 instructions 1 recommendation 2 recommendation Number of Methodologies 3 1

A general comment is that ISTAT and ONS followed a similar approach: both defined a generic provision regarding data collection as a statement, but while ISTAT identified three methodologies (one for data collection in general, one for self-interviewing and one for interviewer assisted techniques) each linked to the related provisions, ONS identified the general methodology linked to all the provisions. As a matter of fact, as shown in the following table, ONS linked all the identified methods to the general methodology, while ISTAT distinguished the methods to be linked to each methodology.

58

Table 42. Example of the different approaches

ISTAT ONS Provision group 1 1 Provision 5 4 Collocation Page 39 Page 43 Description Interviewer assisted methods are very In order to use an useful for survey populations with low interview-assisted literacy rates or when concepts or method sufficient questionnaire are complex budget must be available to pay for the training, hiring and travelling of interviewers Type Recommendation instruction Sequence None None Methodology Interviewer assisted methods Data collection Description The questionnaire is administrated by an Data collection is the interviewer process of gathering the required information for each selected unit in the survey Quality dimension Timeliness, accuracy and efficiency Effectiveness and efficiency Methods : - PAPI - PAPI - CATI - CATI - CAPI - CAPI - CASI

Second common chapter: Chapter 5: Questionnaire design

This chapter describes the process of questionnaire design and other related aspects like the guidelines for wording of survey questions, layout of questionnaires, the management of response error in the interviewing phase, etc.

The standards individuated by ISTAT and ONS in the chapter 5 can be summarized in table 43.

Table 43. Summary for second common chapter

ISTAT ONS Total number of 6 1 Provision groups Main GSBPM 2.3. Design data 2.3. Design data collection collection methodology methodology

59 Secondary GSBPM 3.1 Build data 3.1. Build data collection instrument collection instrument Collocation Chapter 5. Chapter 5.

Number of identified 41 59 Provisions 3 statements 14 statements 20 instructions 31 instructions 18 recommendation 14 recommendation Methodologies 1 2

As it can be seen, the main difference regards the number of identified Provisions Groups. As a matter of fact, ONS defined only one group (Questionnaire design), while ISTAT split the different aspects concerning the questionnaire design treated in the chapter in separated groups. Furthermore, ONS identified two methodologies, Questionnaire design and Questionnaire testing, while ISTAT did not consider Questionnaire design as a methodology itself.

Concluding remarks on the compared analysis

As a general remark it is possible to conclude that the two approaches were quite similar and that the framework of the repository allowed not only to highlight the differences but also to reconcile them. Finally, an observation valid for both the chapters analysed in common is that ISTAT assigned the attribute “statement” in a smaller number of cases than ONS did.

Description of the reconciled version of the common chapters

The analysis of the common chapters led to the final version, combining the two analyses. Regarding Chapter 4, the final version contains the provisions identified by both NSIs, together with all the other provisions defined separately by each. Three methodologies were chosen as this allows a clearer link to the methods. With Chapter 5, the integrated version defines two provision groups; the first for Questionnaire design in general (in which the majority of provisions were grouped) and a second group regarding the Questionnaire design process, which corresponds to 7 provisions to be performed in the specified sequence. Provisions referring to concepts identified by both NSIs were maintained, but when pertaining to the same issue they were defined in a single provision. In general, a number of attributes concerning the type of provision were discussed and sometimes consequently modified.

Final version of the first common chapter

Table 44. Summary of the final version of the first common chapter: Chapter 4

60 Final shared version Total number of Provision Groups 1

Main GSBPM 2.3. Design data collection methodology

Secondary GSBPM 3.1. Build data collection instrument

Collocation Chapter 4 Number of identified Provisions 8 1 statements 3 recommendations 4 instructions Number of Methodologies 3

Final version of the second common chapter

Table 45. Final version of the second common chapter: Chapter 5

Final shared version Total number of Provision Groups 2

Main GSBPM 2.3. Design data collection methodology

Secondary GSBPM 3.1. Build data collection instrument

Collocation Chapter 5 Number of identified Provisions 63 10 statements 26 recommendations 27 instructions Number of Methodologies 1

Overall description of the analysed handbook

Table 46. General characteristics of the chapters

Number of Provision Main GSBPM Number of groups identified Provisions :

61 CHAPTER 1 (by 1- Introduction to 2.3. Design data 16 statements ONS) Surveys collection methodology CHAPTER 2 1- Formulation of the 1.1. Determine needs 2 statements (by ONS ) Statement of Objectives for information 1 descriptive provision CHAPTER 3 1- Introduction to 2.3. Design data 18 statements (by ONS ) Survey Design collection methodology CHAPTER 4 1- Data Collection 2.3. Design data 1 statement (common ) Methods collection methodology 4 instructions 3 recommendations CHAPTER 5 1- Questionnaire 2.3. Design data 10 statements (common ) Design collection methodology 27 instructions 2- Questionnaire 26 recommendations Design Process CHAPTER 6 1- Non-Probability 2.4. Design frame and 2 statements (by ISTAT ) Sampling sample methodology 1 recommendation 2- Probability Sampling CHAPTER 7 1- Estimation 5.6. Calculate weights 3 statements (by ISTAT ) CHAPTER 8 1- Sample size 2.4. Design frame and 1 statement (by ISTAT ) determination sample methodology 1 instruction 2- Sample allocation 4 recommendations for stratified SRS designs CHAPTER 10 1- Data editing 5.3. Review, validate 10 statements (by ISTAT ) 2- Data imputation and edit 17 instructions 3- Evaluation of 5.4. Impute 11 recommendations imputation procedures 1 optional 4- Identification and requirement treatment of outliers

From the analysis of the considered chapters of the handbook it is possible to highlight that the first chapters (1 to 3) are mainly descriptive of survey design and, consequently, the majority of the provisions identified in these pages are statements. In contrast, the chapters covering data collection, questionnaire design, sample size and editing and imputation are more prescriptive, so a great number of recommendations and instructions were found. Finally, chapters 6 and 7 (about sampling designs and estimation) contain a small number of provisions, almost all defined as statements as these chapters are mainly a review of methodologies and methods.

Table 47. Summary of the entities linked to provisions in the handbook

Entity Total number List Methodologies 19 Data collection Self interviewing Interviewer assisted methods Non-probability sampling Probability sampling Design weight Production of simple estimates

62 Estimating sampling error of survey estimates Sample size determination Allocation criteria Allocation methods Selective editing Deterministic imputation Stochastic imputation Fellegi/Holt framework Imputation variance Methods for outliers identification Methods for outliers treatment Methodologies to test questionnaires Methods 77 Haphazard Sampling Volunteer Sampling Judgement Sampling Quota Sampling Modified Probability Sampling Simple random sampling Systematic sample Probability proportional to size Weighting for egual probability sample designs Weighting for unequal probability sample designs Weighting adjustment for nonresponse Using auxiliary information to adjust eights/Post-stratification Using Auxiliary Information to Adjust Weights/Ratio Estimation Using Auxiliary Information to Adjust Weights/Calibration Using Auxiliary Information to Adjust Weights/Generalised regression Using Auxiliary Information to Adjust Weights/Raking Ratio Estimation Using Auxiliary Information to Adjust Weights/Calibration Factors Estimators for Different Types of Data/Qualitative and quantitative data- Population Totals Estimators for Different Types of Data/Quantitative data- Population Average Estimators for Different Types of Data/Qualitative data-Population average Estimators for Different Types of Data/Domains Estimators for Different Types of Data/Small domains Using auxiliary information to Adjust weights/Outliers Sampling variance Standard error Coefficient of variation Margin of error and confidence interval Design effect Estimating the Sampling Variance using Replicated Sampling Cluster sampling Stratified Sampling (STR) Multi-Stage Sampling Multi-Phase Sampling Replicated Sampling Special Topics in Sample Design/Repeated Surveys Special Topics in Sample Design/Entry-Exit Surveys Special Topics in Sample Design/Snowball Sampling Allocation with fixed sample size Allocation with fixed coefficient of variation Proportional allocation Y- proportional allocation Allocation proportional to square-root of N Allocation proportional to square-root of Y Optimum allocation

63 Neyman allocation Optimum allocation with equal variances Top-down approach for selective editing Aggregate Method Graphical Method Questionnaire Score Method (Berthelot and Latouche,1992) Deductive imputation Deterministic Mean Value Imputation Deterministic Ratio/Regression Imputation Sequential Hot-Deck Imputation Sequential Cold-Deck Imputation Nearest-Neighbour Imputation Stochastic Ratio/Regression Imputation Random Hot-Deck Imputation Fellegi-Holt error localisation Fellegi-Holt imputation method Multiple imputation Relative distance from the centre of the data Quartile method Winsorisation Change the weight Robust estimators PAPI CASI CATI CAPI Informal testing Cognitive methods Focus group Interviewer debriefings Behaviour coding of interviewer/respondent interactions Split sample tests Pilot test It-tools 0 Classifications 0

Definitions 0

64

2.3. Description of the unified repository for methodological standards

2.3.1. General description of the repository

The analysis of the unified repository has been carried out through the construction of a set of descriptive tables with the aim to highlight differences and peculiarities of the six handbooks (HB in the following) examined. In these tables the handbooks have been analysed through some synthetic information on the number of identified entities, such as provision groups, provisions, methodologies and listing the statistical activities and the quality dimensions. A further step has been to list all the combination of GSBPM sub-processes indicated for all HB all together, considering the repository as a unique DB.

An important general remark is that the observations deriving from these tables can provide inevitably only a partial view because not all the HBs have been examined thoroughly; nevertheless this overview is useful to highlight some possible deficiencies of the analysis so far carried out, with the aim to improve the future work on the repository.

The first two tables focus on the analysis of provisions and methodologies for each HB, considered in terms of number and type.

Table 1. Counts of entities individuated for each handbook Handbooks # of # of # of # of # of # of # of Provision Provisions Methodologies Methods IT Classifications Definitions Groups tools

Sampling guidelines 2 5 4 20 3 0 0 EDIMBUS 9 35 9 29 0 2 0 Guidelines for 3 20 0 0 0 0 2 Statistical Metadata Quality Report 10 197 15 36 0 6 12 Questionnaire Design 6 17 14 13 0 0 0 and Testing Survey Methods and 15 161 19 76 0 0 0 Practices TOTAL 25 258 28 86 3 8 14

Table 2. Distribution of provisions by type for each handbook Type of provision Total # of Handbooks # of # of # of # of provisions statements instructions recommendations other types Sampling guidelines 0 0 5 0 5 EDIMBUS 10 11 14 0 35 Guidelines for Statistical 20 Metadata 5 1 11 3 Quality Report 105 91 1 197 Questionnaire Design and 17 Testing 0 0 17 0

65 Survey Methods and 161 Practices 64 48 47 2 TOTAL 121 12 121 4 258

A first consideration regards the total number of provision groups (PG) for each HB (Table 1): generally this number is higher for the multi-topic HB, such as Survey Methods and Practices (SMP), with respect to the HB devoted to specific methodological topic. The same consideration can be done for the number of provisions and methods. It is necessary to underline that for Quality Report (QR) a high number of entities has been indicated, even though only one chapter has been analysed. Another observation regards the relation between the number of PGs and Methodologies on one side and provisions and links to methods on the other side. The analysis of the provision types (Table 2) highlights, in general, a prevalence of statements and recommendations, with a substantial balance between them. Also in this context can be underlined a relation between the nature of the HB (multi-topic or specific) and the prevalence of some types of provision (statement or recommendation)

As far as the quality dimensions are concerned, when analysing Table 3 the prevalent indicated dimensions are accuracy and efficiency: in general accuracy is indicated for the 89% of the provisions, with a small variability among HBs, with the exception of Metadata, while efficiency is indicated for the 27.8% of the provisions but with a high variability among HBs. It is worthwhile to note that for QR only accuracy has been analysed.

Table 3a. Percentage of provisions for quality dimensions for each handbook (product)

Percentage of provisions for dimension of quality (product) Handbooks % of % of % of % of % of relevance accuracy timeliness accessibility coherence Sampling guidelines 0.0 80.0 0.0 0.0 0.0

EDIMBUS 0.0 100.0 36.1 0.0 72.2 Guidelines for Statistical 0.0 0.0 5.0 45.0 10.0 Metadata Quality Report 0.5 99.0 0.0 0.0 0.0 Questionnaire Design and 41.2 88.2 5.9 0.0 17.6 Testing Survey Methods and 19.9 87.6 16.1 10.6 14.3 Practices

TOTAL 9.2 89.7 9.4 6.0 12.4

66 Table 3b. Percentage of provisions for quality dimensions for each handbook (process)

Handbooks Percentage of indicated dimensions of quality (process) % of effectiveness % of efficiency Sampling guidelines 20.0 60.0 EDIMBUS 63.9 77.8 Guidelines for Statistical Metadata 20.0 20.0

Quality Report 0.0 0.0 Questionnaire Design and Testing 0.0 35.3 Survey Methods and Practices 30.4 58.4 TOTAL 17.7 31.0

From the analysis of Table 4, where the indicated statistical activities are listed, it is possible to underline that the multi-topic HB refer in general to all statistical activities, while the specific ones, as Edimbus, refer to specific activities both for the survey context and for the survey phase. Moreover, QR indicated all statistical activities but also specific contexts, highlighting that peculiar quality indicators exist for specific survey contexts.

Table 4. List of statistical activities for handbook

Handbook LIST of distinct statistical activities EDIMBUS All statistical subjects (D1+D2+D3) D2 Economic statistics 2.3 Business statistics 4.4 Data editing and data linkage 4.7 Data analysis Guidelines for Statistical 4.4 Data editing and data linkage Metadata

4.5 Dissemination, data warehousing

5.5 Management and development of technological resources (including standards for electronic data exchange and data sharing)

Quality report All statistical subjects (D1+D2+D3) D1 Demographic and social statistics D2 Economic statistics 2.1 Macroeconomic statistics 2.2 Economic accounts 2.3 Business statistics 2.4 Sectoral statistics 2.4.1 Agriculture, forestry, fisheries 2.4.2 Energy 2.4.3 Mining, manufacturing, construction

67 2.4.4 Transport 2.4.5 Tourism 2.4.6 Banking, insurance, financial statistics 2.5 Government finance, fiscal and public sector statistics 2.6 International trade and balance of payments intranet 2.8 Labour cost 2.9 Science, technology and innovation D3 Environment and multi-domain statistics 3.1 Environment Survey Methods and All statistical subjects (D1+D2+D3) Practices D4 Methodology of data collection, processing, dissemination and analysis 4.4 Data editing and data linkage

With regards to the GSBPM model, in Table 5 are listed the GSBPM sub-processes indicated as the main ones. Even though the funding of this list depends on the degree of completeness of the analysis conducted for each HB, as a general consideration, the multi-topic HB cover quite a high number of sub-processes, while the other ones a more limited set. In Table 6 it is possible analyse the combinations of sub-processes indicated in the integrated repository with respect to the number of GPs, provisions, methodologies and methods.

Table 5. List of GSBPM procesess for handbook

Handbooks LIST of distinct GSBPM sub-process

Sampling guidelines 2.4. Design frame and sample methodology

4.1. Select sample

5.4. Impute 5.6. Calculate weights EDIMBUS 5.3. Review, validate and edit

5.4. Impute

6.3. Scrutinize and explain Guidelines for Statistical Metadata 7.2. Produce dissemination products

7.3. Manage release of dissemination products 7.4. Promote dissemination products Quality report 2.3. Design data collection methodology

2.4. Design frame and sample methodology

2.5. Design statistical processing methodology

4.1. Select sample 10.1. Quality Management

68 Questionnaire Design and Testing 1.1. Determine needs for information

1.5. Check data availability 2.3. Design data collection methodology 3.1. Build data collection instrument Survey Methods and Practices 1.1. Determine needs for information

1.2. Consult and confirm needs

1.3. Establish output objectives

2.2. Design variable descriptions

2.3. Design data collection methodology

2.4. Design frame and sample methodology 3.1. Build data collection instrument

4.1. Select sample 5.3. Review, validate and edit 5.4. Impute 5.6. Calculate weights 10.1. Quality Management

69

Table 6. Number of entities for combinations of GSBPM processes in the integrated repository GSBPM sub-process combination # of # of # of # of # of # of # of Main GSBPM Auxiliary GSBPM Additional GSBPM Groups Provisions Methodologies Methods IT classifications definitions tools 1.1. Determine needs for 1.2. Consult and confirm 1.3. Establish output 1 3 information needs objectives 1.1. Determine needs for 1.5. Check data availability 1 1 1 1 information 2.3. Design data collection 2 34 methodology 2.3. Design data collection 2.2 Design variable 3.1. Build data collection 2 63 1 7 methodology descriptions instrument 2.3. Design data collection 3.1. Build data collection 3 12 6 6 methodology instrument 2.4. Design frame and 2 7 3 9 sample methodology 2.4. Design frame and 4.1. Select sample 2 3 2 16 sample methodology 2.4. Design frame and 4.1. Select sample 5.6. Calculate weights 1 3 3 14 2 sample methodology 3.1. Build data collection 2 6 5 5 instrument 3.1. Build data collection 2.3. Design data collection 1 6 5 5 instrument methodology 5.3. Review, validate and 7 48 6 18 2 edit 5.3. Review, validate and 5.4. Impute 5.6. Calculate weights 1 4 2 5 edit 5.4. Impute 3 19 5 19 5.4. Impute 10.1. Quality Management 1 3 1 1 5.4. Impute 5.6. Calculate weights 4.1. Select sample 1 2 1 6 5 5.6. Calculate weights 1 3 3 21 6.3. Scrutinize and explain 2 2 2 4 7.2. Produce 7.3. Manage release of 7.4. Promote dissemination 1 6 dissemination products dissemination products products 7.4. Promote 2 14 2 dissemination products 10.1. Quality Management 7 96 8 19 1 11 10.1. Quality Management 2.4. Design frame and 2.3. Design data collection 1 14 2 2 2 sample methodology methodology 10.1. Quality Management 2.4. Design frame and 2.5. Design statistical 1 8 1 1 sample methodology processing methodology 10.1. Quality Management 2.4. Design frame and 4.1. Select sample 1 79 10 30 2 sample methodology TOTAL 46 436 66 188 7 8 14

71 2.3.2. Examples of compared analyses of standards derived from different handbooks for the same GSBPM sub-processes

With a unique repository containing the results of the analysis carried out on different methodological handbooks, as the one obtained by following the procedure described in par. 2.1.3, it is possible to compare, among other things, all the objects pertaining to a given GSBPM. In particular, it is possible to compare the provisions, that are the elementary items that compose a standard, and all related objects (methodologies, methods, IT tools, definitions and classifications). What is the utility of such a comparison? Let us consider the definition of standard, based on the ISO approach, that has been given in par.1.3.1 (“ A standard is a document, established by consensus and approved by a recognized body , that provides, for common and repeated use, rules, guidelines or characteristics for activities or their results, aimed at the achievement of the optimum degree of order in a given context .”). Accordingly to this definition, it is clear the methodological handbooks that have been taken into consideration cannot be considered as “standards”, at least for the whole European Statistical System. They are rather “ normative documents ” of the kind “ code of practice ”, usually valid locally inside the body that produced them. In case an initiative to produce statistical standards valid for the whole European Statistical System should take place, a recognized body could be instituted, with the aim of defining these standard for each important step of a statistical process, i.e. for each GSBPM sub-process. This body could take into account the most important methodological handbooks currently used inside the National Statistical Institutes and the other international statistical organisations, as a good start for the process of definition of the standards. Each handbook could be analysed following the same procedure adopted here, thus populating a repository with the structure here proposed. Then, for a given GSBPM sub-process, the provisions originated by the different handbooks could be evaluated in order to decide which of them could be promoted to become part of the final “standard” for that sub-process. In the following, we report examples of comparison, one related to “ 4.1. Select sample ”, the other one to “ 5.3. Review, validate and edit ”.

Case 1: GSBPM sub-process ”4.1. Select sample”

In this comparison, we analyse all relevant objects (from the author’s point of view) in the unified repository, which themselves are the result of the analyses of three methodological handbooks: “ESS Handbook for Quality Reports”, “Survey Methods and Practices” and “Survey Sampling and Reference Guidelines, Introduction to sample design and estimation techniques”.

First, we compare the different (relevant) groups of provision that have been defined from the three handbooks:

ESS Handbook for Quality Survey Methods and Survey Sampling and Reports Practices Reference Guidelines

Non-probability sampling Probability Sampling Sampling Probability sampling Sample allocation to strata

In fact, there had been defined more Provision groups for the three handbooks (see the analyses within the previous chapters), but these had not been considered here. The reason for this is that GSBPM sub-process 4.1 was only mentioned as secondary GSBPM sub-process and the contained provisions were not considered as relevant in this context.

Now, we consider the descriptions of the different provisions, separately for the three handbooks. In our case of “sample selection”, the provisions may be stated in the general form: “The statistician should apply one method among a list of recommended methods”. However, the descriptions of the provisions in the unified repository, elaborated by different NSIs, do not really reflect this. The formulations appear to be misleading to some extend. Nevertheless, they are listed here:

ESS Handbook for Quality Reports GROUP OF PROVISIONS PROVISIONS When non-probability sampling is applied, random error can not be estimated without reference to a model of some kind. Furthermore, sampling biases may well be significant and need to be assessed as well. Some further technical points concerning the presentation of sampling errors are: Sampling 1) Non-response should be taken into account, i.e., the sample size should be the effective sample, after deduction of non-response; 2) The original stratification should be appropriate. One type of non-probability sampling that is frequently applied in economic surveys and therefore needs special attention is the use of a cut-off threshold.

73

Survey Methods and Practices

GROUP OF PROVISIONS PROVISIONS Non-probability sampling is a method of selecting units from a population using Non-probability a subjective (i.e., non random) method. sampling Non-probability sampling can be applied to studies that are used as preliminary steps or follow-up steps. Probability sampling is a method of sampling that allows inferences to be made about the population based on observations from a sample. In order to be able to Probability make inferences, the sample should not be subject to selection bias. Probability sampling sampling avoid.

Survey Sampling and Reference Guidelines

GROUP OF PROVISIONS PROVISIONS Probability Sample selection: process of selecting the sampled units from the frame. Sampling Sample Sample allocation to strata allocation to strata

Next, we consider methodologies and methods:

ESS Handbook for Quality Survey Sampling and Survey Methods and Practices Reports Reference Guidelines Methodologies Methods Methodologies Methods Methodologies Methods Probability- Non- Modified Proportional- Cut-off Probability Probability Probability to-Size sampling sampling Sampling Sampling (PPS) Non- Sampling probability Other sampling forms of Stratified Volunteer non- Sampling Sampling probability (STR) sampling Cluster Judgement Sampling Sampling (CLU)

74 Simple Random Quota Sampling Sampling (SRS) Special Topics Systematic in Sample Sampling Design/Snowball (SYS) Sampling Modified Power Probability Allocation Sampling (Bankier) Optimal Haphazard Sample Allocation Sampling allocation to (Neyman) strata Proportional Cluster sampling Allocation Equal Systematic Allocation sample

Probability proportional to size Replicated

Sampling Simple random

sampling Probability Special Topics sampling in Sample

Design/Entry- Exit Surveys Special Topics in Sample

Design/Repeated Surveys Multi-Stage

Sampling Stratified

Sampling (STR) Multi-Phase

Sampling

At this level, one can identify overlaps of methods. In the unified repository, multiply mentioned methods and methodologies should be eliminated. The following table gives an example of such a consolidation:

75

Methodologies Methods Probability-Proportional-to-Size (PPS) Sampling Stratified Sampling (STR) Cluster Sampling (CLU) Simple Random Sampling (SRS) Probability sampling Systematic Sampling (SYS) Replicated Sampling Special Topics in Sample Design/Entry-Exit Surveys Special Topics in Sample Design/Repeated Surveys Multi-Stage Sampling Multi-Phase Sampling Cut-off sampling Modified Probability Sampling Volunteer Sampling Non-probability sampling Judgement Sampling Quota Sampling Special Topics in Sample Design/Snowball Sampling Power Allocation (Bankier) Optimal Allocation (Neyman) Sample allocation to strata Proportional Allocation Equal Allocation

76 Case 2: GSBPM sub-process “5.3. Review, validate and edit “

In this comparison, we analyse all objects in the repository that are: o the result of the analyses of two methodological handbooks: “Survey Methods and Practices” and “Recommended practices for Editing and Imputation in Cross-Sectional Business Surveys (EDIMBUS)”; o related, directly or indirectly, to the main GSBPM sub-process “5.3. Review, validate and edit”.

First, we compare the different groups of provision that have been defined from the two handbooks:

SURVEY METHODS AND PRACTICE EDIMBUS

Detection of errors Missing values Data editing Systematic errors Random errors Influential errors Identification and treatment of outliers Outlier

Clearly, there is a more compact way of defining provision groups for “Survey Methods and Practices” than for “EDIMBUS”, but is possible to establish a clear correspondence between the two sets of groups. Now, we consider the different provisions, distinctly for the two handbooks.

SURVEY METHODS AND PRACTICE

GROUP OF PROVISIONS PROVISIONS Data editing 1.Editing should be performed at several stages of the survey. 2. The purpose of editing is to: better understand the survey processes and the survey data; detect erroneous or missing data; follow-up with the respondent; send a record to imputation; delete a record. 3. Editing should be used to provide information about the survey process, either in the form of quality measures for the current survey or to suggest improvements for future surveys. 4. When starting a survey, some assumptions are made about the data. During editing, it is possible to test the validity of these assumptions. For example, it may become obvious that some range edits were too strict or that some sequencing edits failed too f 5. Editing is the application of checks to identify missing, invalid or inconsistent entries that point to data records that are potentially in error. 6. Edits applied throughout collection and processing should be consistent with each other

7. Editing can be automated by means of a computer program.

8. Edits should be developed by staff who have expertise in the subject matter, questionnaire design, data analysis and with other similar surveys. 9. There are three main categories of edits: validity, consistency and distribution edits.

10. Validity edits verify the syntax of responses and check that the coded data lie within an allowed range of values.

11. Consistency edits verify that relationships between questions are respected.

77 12. Distribution edits attempt to identify records that are outliers with respect to the distribution of the data. 13. Information on the types of edits performed and the impact of editing on the survey data should be communicated to users. 14. Quality assurance and quality control procedures should be applied to minimise and correct errors introduced during editing 15. Selective editing practices are recommended, particularly for business surveys (i.e., where the population is skewed and a few businesses dominate the estimates). 16. Follow-up is generally limited to edits failures identified during collection or arising from selective editing 17. An outlier is an observation or subset of observations that appears to be inconsistent with the remainder of the dataset 18. Outliers are detected by measuring their distance from the centre of data

Identification and 19. Outliers detected at the editing stage of the survey process can be treated in various ways. treatment of 20. The goal of outlier treatment is to decrease the impact that the outlier has on the sampling outliers variance of the estimate without introducing too much bias

EDIMBUS

GROUP OF PROVISIONS PROVISIONS 1. Detection of errors consists of identifying values that are not acceptable with respect to some pre-defined logical, mathematical or statistical criteria. Since different error types may Detection of errors contaminate the observed data in a sample of units, usually the error detection process consists of a set of error detection methods dealing each with a specific type of error 2. Missing values stem from questions the respondent did not answer. Nonresponse can be due to several reasons; the respondent may not know the answer, may not be willing to respond or may have simply missed a question. 3. Replacing missing values by zero is neither an acceptable imputation procedure nor an acceptable alternative to flagging missing values

Missing values 4. Variables and cases with many missing values should be studied and a decision on the variables and observations to be imputed should be taken in view of the amount of missingness. It may be necessary to discard certain variables or certain cases from the 5. Appropriate indicators on missing values should be calculated

6. The indicators should be analyzed to gain information on nonresponse mechanisms

7. A systematic error is an error that is reported consistently over time by responding units. It is a phenomenon caused either by the consistent misunderstanding of a question during the collection of data, or by consistent misinterpretation of certain answers in the course of coding 8. Systematic errors should be detected and treated before dealing with random errors, in particular when the Fellegi-Holt method is used and before selective editing Systematic errors 9. The analysis of indicators on edits helps to find systematic errors mechanisms. If systematic error mechanisms are found by examining edits, then appropriate deterministic checking rules to detect errors due to the systematic error mechanism should be add 10. If systematic error mechanisms are found, then improvements to the survey process (Questionnaire, interviewer training, coding, processing) should be made to prevent similar errors 11. Influential errors are errors in values of variables that have a significant influence on Influential errors publication target statistics for those variables.

78 12. Selective editing is an appropriate method to focus attention on critical observations without generating a detrimental impact on data quality

13. Priorities should be reflected in the score functions of selective editing. Score functions should include a risk and an influence component

14. The thresholds of score functions should be chosen carefully during the tuning and testing of the E&I process. The thresholds and parameters of score functions should be revised whenever the survey process is changed (Questionnaire, data entry, removal of systematic errors) 15. The quality of the anticipated value should be assessed at least for a sub-sample of the survey. E&I flags have to be taken into account if data of earlier periods are used as anticipated values

15. Units, for which the score function cannot be calculated have to belong to the critical stream to prevent biasing the estimates

17. Macroediting should be properly documented

18. Important publication aggregates, publications cells and publication sub-populations should be considered in macroediting 19. The quality of reference data and problems like inflation and structural differences (e.g. definitions) should be taken into account

20. Macroediting should be performed before releasing data for final estimation.

21. An outlier is an observation which is not fitted well by a model. The model can be a parametric distribution or the model can be a more loosely defined concept like "close to the center of the data". In the latter case an outlier is an observation which is not close to the center of the data 22. Outliers and influential observations should be detected. Such observations may be errors or correct but sometimes their correctness is unclear. Errors should be treated and also correct or unclear influential or outlying observations may have to be treated and also correct or unclear influential or outlying observations may have to be treated to prevent potential bias and high variability 23. The influence on important results should be controlled even after selective editing and outlier Outliers detection. 24. Outlier detection methods should be robust against outliers. Therefore methods based on means or weighted means and standard deviations, which are not robust, should be avoided

25. Simple univariate methods, graphical displays or more complex multivariate techniques should be used for outlier detection depending on possible models among variables. Different models may have to be applied in different sub-populations 26. When choosing tuning constants for detection or treatment several tuning constants should be tested and the corresponding impact on estimates and their variances observed. Often the treatment of a few outliers is acceptable. 27. Random errors are errors that are not caused by a systematic reason, but by accident. They primarily arise due to in-attention by respondents, interviewers and other processing staff during the various phases of the survey cycle. 28. The data in each record should be made to satisfy all edits by changing the fewest possible items of data (fields) 29. As far as possible the frequency structure of the data file should be maintained

30. Imputation rules should be derived from the corresponding edit rules without explicit Random errors specification

31. If appropriate software is available, random errors should be detected and treated by applying the Fellegi-Holt paradigm

32. If appropriate software for applying the Fellegi-Holt paradigm is not available, random errors may be detected by means of deterministic checking rules. However, the error localization becomes more arbitrary with deterministic checking rules

79

Survey Methods and Practices (SMP) counts 20 provisions, while EDIMBUS contains 32 provisions dealing with sub-process 5.3. It is not surprising, as EDIMBUS is dedicated to the edit and imputation steps of surveys.

By comparing the two sets of provisions, we can classify them in: 1. those dealing with the same arguments, and that could be merged (having verified they are not contradictory): for instance, “selective editing” (provision # 15 in SMP, and provision # 12 in EDIMBUS): these provisions may be potentially merged (“ Selective editing practices are recommended, particularly for business surveys (i.e., where the population is skewed and a few businesses dominate the estimates), as they are appropriate methods to focus attention on critical observations without generating a detrimental impact on data quality ”); 2. those dealing with the same argument, but not compatible: for instance, provision # 17 in SMP and # 21 in EDIMBUS. They are both statements defining outliers: even they are both correct, the first may be considered too generic with respect to the second, that should be adopted; 3. non overlapping provisions: for instance, provisions from # 1 to # 14 in SMP are more general than those in EDIMBUS; or provisions from #7 to # 10 do not have correspondence in SMP. In this case, once verified their validity, they could be all promoted to be part of a new standard for editing.

Let us now consider methodologies and methods:

Survey Methods and Practices EDIMBUS Methodologies Methods Methodologies Methods Questionnaire Score Detection of missing Method values (Berthelot-Latouche) Graphical method Analysis of fatal Selective editing edits Detection of systematic Top-down Analysis of ratio errors edits

Aggregate method Finite mixture models Methods for outliers Relative distance from Detection of influential Macroediting identification the centre of data errors Quartile method Selective editing Hidiroglou-Berthelot method Median absolute deviation rule Regression-based models Detection of outliers Graphical editing

One-step robustified ratio estimator Weighted quantiles Winsorised mean and trimmed mean Localising random Deterministic checking errors rules Fellegi-Holt paradigm

80

Once again, there is a higher number of methodologies and methods defined in EDIMBUS than in SMP. Despite the different naming adopted in the two handbooks, it is possible to establish some equivalences between the defined objects: - “Selective editing ” methodology in SMP corresponds to “ Detection of influential errors ” methodology, but also to “ Selective editing ” method, in EDIMBUS; - “Methods for outliers identification ” in SMP corresponds to “ Detection of outliers ” in EDIMBUS; - the couple “ Top-down ” and “ Aggregate method ” methods in SMP correspond to “Macroediting ” in EDIMBUS; - “Graphical method ” in SMP corresponds to “ Graphical editing ” in EDIMBUS (but the former is linked to the “ Selective editing ” methodology, while the latter to the “ Detection of outliers ” methodology).

As an example of integration, we propose to consider the union of all methodologies and methods, with the necessary re-classification in order to avoid inconsistencies (for instance, “ Selective editing ” should be considered as a methodology, making use of different methods). In the following table a possible integrated solution is reported. This unified list of methodologies and methods could be the starting point for the integration also of the provisions and provision groups, when constructing the final “standard” for this particular sub-process.

Methodologies Methods Detection of missing values Detection of systematic Analysis of fatal edits errors Analysis of ratio edits Finite mixture models Top-down Detection of influential errors Aggregate method Questionnaire Score Method (Berthelot-Latouche) Selective editing Graphical method Hidiroglou-Berthelot method Median absolute deviation rule Regression-based models Detection of outliers Graphical editing

One-step robustified ratio estimator Weighted quantiles Winsorised mean and trimmed mean Localising random Deterministic checking rules errors Fellegi-Holt paradigm

81

3. Review of support measures

3.1. Introduction

Application of standards is either compulsory or voluntary. Communication and several support tools are necessary to promote and assist implementation in both cases.

3.2. Support measures related to standards

Implementation can be based only on the excellence of a standard and on the benefits its application provides. This can be observed when a local practice – available in an adequate document – becomes a practical standard implemented by a number of partners without any further external pressure. This was the case of Quality guidelines of Statistics Canada 19 . Benefits of a standard will get through if it is widely used; the number of users is relatively high. This is why speed is important, hence support measures make sense. Support measures are different types of activities, connected to standards implementation. Support measures refer to all necessary information, modes and tools for access and special regulations providing scope in better understanding, delivery and application of standards. Support measures are intended to accelerate, facilitate and smoothen the implementation of standards. They intend to bridge the gap between the optimal expected environment and the local, special environment of implementation. Support measures are “soft” tools, with great variability, optimally adapted to the intended user environments. This is the reason why support measures are not integral part of a standard, however contribute to the realisation of standard development principles 20 : transparency and openness. In practice, at the moment when a standard is accepted there are at least plans on a set of support measures on the implementation or bodies assigned to be involved in supporting activities. In case of European Standards, CEN (European Committee for Standardisation) manages several regional and national Technical Assistance programmes financed by the European Union. The assistance aims to build a quality infrastructure (standardization, testing and metrology, conformity assessment and certification) in beneficiary countries 21 . In the field of statistics the general practice of UN Statistical Commission is to take care of support measures, when adapting a new standard. Concerning classifications, as traditional field of international standards, this role is already an agreed rule. “The role of the Statistical Commission and other intergovernmental boards in the area of classifications is to: (a) determine short and medium term objectives for setting standards; (b) work toward harmonisation of existing standards and practices; (c) promote best practices; (d) implement data collection and dissemination strategies that incorporate international classification agreements; and

19 Quality Guidelines, Statistics Canada, Fifth Edition – October 2009 http://www.statcan.gc.ca/pub/12-539-x/12-539-x2009001-eng.pdf 20 The ABC’s of Standards Activities, National Institute of Standards and Technology, Gaithersburg, August 2009 http://gsi.nist.gov/global/docs/pubs/NISTIR_7614.pdf 21 Compass CEN The World of European Standards ftp://ftp.cen.eu/cen/AboutUs/Publications/Compass.pdf 82 (e) monitor the use of classifications.”, states the UNSD document 22 .

The practices of Eurostat are similar when grants, workshops, training courses, coaching, handbooks, collection of best practices are offered to support implementation of new or modified standards.

3.3. Overview of support measures

The direct influence is the motivation using the tools of awarding and sanctioning. The motivation can be cheap and effective if the standard is known and the implementation is easy. Providing support during implementation may help to guarantee success, learning more on standard itself, avoid misuse and fails. In the following we overview all of these types of support measures with examples, although part of them is not yet used for statistical standards.

3.3.1. Motivation for implementation

The decision to use different types of motivation – and in accordance with the type of conformity assessment – can be linked to the level of risk associated with the product/service and the customer requirements. This is not only true in the case of European market but we can observe it in statistics too. We can think about the problems and regulatory consequences of the falsified EDP report (Excessive Deficit Procedure and the financial crisis).

Legal instruments, sanctions: The strictest motivation is obligation with legal instruments. European examples are the „New Approach” laws and directives in EU standardization process requirement for obtaining CE mark to gain access to European market 23 . In the field of statistics, an example is the standard for EU national accounts. In the near future: ESA (standard for EU national accounts) methods will be obligatory to be used for calculations of EDP indicators 24 . The classifications are de jure standards, as usually regulations prescribe their use in the EU, for example NACE Rev.2 and CPA 2008. These regulations state/determine exactly from when and where (in which statistical domains) classifications should be applied. In the field of metadata, there is only one legal instrument available, the 2009/498/EC Commission Recommendation of 23 June 2009 on reference metadata25 for the European Statistical System. This recommendation covers the basic concepts and the structure of the so called ESMS (Euro-SMDX) files (metadata), which have to be sent in line with the corresponding statistical data. This recommendation takes the SDMX standard into consideration, since it might be a world wide standard for metadata and metadata exchange.

22 International Standard Classifications: A Draft Statement of Best Practices. http://unstats.un.org/unsd/class/intercop/expertgroup/1997/AC60-15.PDF 23 http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CONSLEG:1998L0034:20070101:EN:PDF DIRECTIVE 98/34/EC OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 22 June 1998 laying down a procedure for the provision of information in the field of technical standards and regulations http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:1998:217:0018:0018:EN:PDF DIRECTIVE 98/48/EC OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 20 July 1998 amending Directive 98/34/EC laying down a procedure for the provision of information in the field of technical standards and regulations 24 About sanctions see: COM(2010)0522 Proposal for a COUNCIL REGULATION (EU) No …/… amending Regulation (EC) No 1467/97 on speeding up and clarifying the implementation of the excessive deficit procedure Article 11 and article 12 (page15) 25 http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2009:168:0050:0055:EN:PDF 83 At the META Working Group held in Luxembourg in 2011, a new proposal was made for legislation on metadata and metadata exchange – as a cross-cut legislation – in order to make the metadata of Member States more harmonised. Supposedly those instruments will be regulations, therefore compulsory for every Member State. Legislation can support implementation of standards not only by obligation but by making reference too. An example from CEN-related activities: “In addition many standards are developed to support European legislation. ‘Reference to standards’ within a legislative text is viewed as a more effective means of ensuring that products meet the essential health and safety requirements of legislation than writing of detailed laws. This allows both processes to support each other, without causing a slowdown.” 26 We can find similar examples in statistics: in some cases there is no legal instrument which makes the application obligatory, but according to legislation a set of permissions are necessary preconditions to start certain activities, and the scope is prescribed by referring NACE categories.

Awards Beyond obligation and sanctions, positive approach can be useful with rewarding. In the fields, where no obligation is possible, motivation can be supported by giving awards, like international and national quality awards built on quality management standards (ISO 9001, EFQM). Examples: The European Quality Award 27 , Malcolm Baldrige National Quality Award 28 , The Deming Prize 29 .

Communication Knowing about standards is a precondition for the decision on implementation. Intended users and their partners should be aware of the existing and forthcoming standards, therefore communication and promotion are crucial, even if implementation is supported by regulations or awards.

3.3.2. Making the implementation easy

Implementation of a standard is easier if it is available, understandable, efficient and consistent and is accompanied by proper training activities.

Availability Easy availability is a characteristic of standards, however it does not mean the standards are free of charge, especially if they are linked with patterns. Easy availability means free access, downloadable documentation in the case of statistical standards, such as statistical classifications and methodological handbooks (RAMON 30 on Eurostat homepage, Methods and classifications on UN homepage 31 ). SDMX standards are downloadable from the SDMX website (www.sdmx.org\standards ). A repository of standards with built-in search features, containing not only actual but different versions could improve availability of standards for potential users. Generally, a “responsible body” has a repository of its own standards (accepted by the body). In general, standards relevant for a unit are originated from different responsible bodies. Potential users are interested in a repository,

26 Compass CEN The World of European Standards 27 Established in 1992 and offered by the European Foundation for Quality Management http://www.efqm.org/en/ 28 Established in 1987 and offered by the National Institute of Standards and Technology in the USA http://www.baldrigepe.org/foundation/ 29 Established 1951 (International from 1984) and offered by the Japanese Union of Scientists and Engineers http://www.juse.or.jp/e/deming/ 30 http://ec.europa.eu/eurostat/ramon/index.cfm?TargetUrl=DSP_PUB_WELC 31 http://unstats.un.org/unsd/methods.htm 84 where all relevant standards are available. In the EU, an NSI has to be aware of the EU, UN, UN- ECE, OECD, IMF, ILO and some other bodies’ statistical standards.

Understandability Understandability can be achieved by using adequate language (plain English, local), by providing additional materials different by length and detail, according to the needs of the intended users (managers, experts, IT people). Example on different complementary, promotion materials are those of the CEN: documents, compass-leaflet, articles, homepage 32 . Besides European standards, CEN produces other reference documents like: workshop agreements, technical specifications, technical reports and guides. Another good example for useful support tool is a glossary – definitions specify how the terms are to be understood in the context of the given standard – even if it is not necessarily exhaustive.

Efficiency The technical implementation may be hindered by the costs of developing tools to implement, to fit in the local IT environment. This burden can be reduced by developing interfaces, IT tools, application methods. In the field of statistics, the implementation of the recommendations on seasonal adjustment is supported by Demetra, a supporting IT tool, downloadable from Eurostat website, fit to implement recommendations. In the case of SDMX standard, there are guidelines and software downloadable from the SDMX website ( www.sdmx.org\standards , and www.sdmx.org\guidelines ). These software solutions can be used for the implementation of the SDMX standard for metadata exchange. In order to understand the basic principles and concepts of the standards, guidelines are also available. This handout is the so called Content–oriented guidelines. At European level, applications are also available, such as the National Reference Metadata Editor (NRME) that can be used by Member States for the production and transmission of metadata files. This is a web application, but there are also other software solutions for utilisation. These can be used for mapping the national metadata to the SDMX concepts, classifications and their items, database table columns, etc.

Consistency In any business, a number of standards are applied and they have to be able to work as a consistent system. They have to communicate with each other in an efficient, cost effective way. For users, it is more convenient if the connecting standards have mutual recognition, and some kind of complementarities can be expected among them. The consistency of European standards and national standards is assured in the following way: after ratified by CEN, each of the National Standard Bodies adopts the European standard as an identical national standard and withdraws any national standards that conflict with the new European standard. 33 . Other approach is the case – generally and typically in statistics – when global standards do not go into details and local stakeholders are encouraged to develop additional, more detailed standards or guidelines on their specific activities 34 . Other examples in statistics are: the derived classification the General Industrial

32 http://www.cen.eu/cen/AboutUs/Pages/default.aspx

33 COMPASS Cen The world of European Standards 34 Brian A. Harris-Kojetin – Katherine Wallman (2009): Revision and Implementation of Standards for Statistical Surveys for the U. S. Federal Statistical System. http://www.scb.se/Grupp/Produkter_Tjanster/Kurser/ModernisationWorkshop/Final/B_1_QA_systems_Harris- Kojetin.pdf 85 Classification of Economic Activities within the European Communities (NACE) based upon ISIC, the International Standard Industrial Classification of All Economic Activities adopted as UN standard, and Member States are allowed to adopt a national version of NACE. Similar relation links SNA as UN standard and ESA as Eurostat standard. The standards with more restricted geographical scope are more detailed and more concrete. Generally speaking, the more homogenous the area of application the more detailed and concrete the standard can be. As conformity analysis of the different standards can be very expensive, users can be supported with information on their conformity. This is included as a requirement in the Policy of standards of Statistics Canada: “A standard shall be accompanied by a statement of conformity to relevant internationally recognized standards or a description of the deviations from such a standard and, where possible, a concordance with the referenced standard.” 35 In the case of classifications, the corresponding tables are intended to be used as a key between different classifications, but in the same time they also provide information on their conformity.

Training Training provides organised support for understanding and implementation of standards. Examples in statistics are the ESTP training courses related to methodological standards, however, ESTP courses make few references to methodological handbooks. Examples: There are several workshops (e.g.: METIS workshops) in case of SDMX standards. In 2011, it was held in Washington D.C. In the Conference, SDMX sponsors expressed their intentions „to continue providing training and learning opportunities, also supporting for SDMX implementers a “learning by doing” exercise to promote adoption of the standards and guidelines.” 36 There is a course for better understanding the European software solutions for implementing and using SDMX standard and the corresponding software. It consists of the courses: a) SDMX Basic course for Statisticians and b) SDMX for IT developers.

3.3.3 Supporting right implementation

In order to support the right implementation we can find responsible body , which takes care of the standard itself, its promotion, certification, training material, maintenance. This is the case of SDMX, ISO, CEN and to some extent, this is the role of Eurostat in European statistics. The responsible body should provide feedback possibilities for users. Feedback possibilities on success or fails or critics are crucial for further development of standards. Support during implementation can have different forms, like recommendation on implementation, helpdesk, forum for consultation, financial support or combination of these. An example: on recommendation is the handbook on the methods of implementation of NACE Rev.2 in calculation of time series of different statistical domains. Helpdesk function is not always typical; generally private consultant companies provide this kind of support. In statistics, Eurostat provides temporary support with grants. Further example is the CIRCA Forum of classifications, which operates like a helpdesk and gives prompt assistance in right application of NACE Rev.2 and CPA2008. Usually more than hundred questions are consulted and answered every year, therefore decision can be achieved in one week instead of waiting for the working group meeting once a year.

35 Policy on standards, Statistics Canada, 2004 http://www.statcan.gc.ca/about-apercu/policy-politique/standards-normes-eng.htm 36 http://sdmx.org/wp-content/uploads/2011/05/SDMX-Global-Conference-2011-Summary-Report-2-4-May- 2011.pdf 86 Conformity assessment and certification provide feedback for users on the right implementation of the standard. Conformity assessment is typical in case of ISO, CEN. “For conscientious manufacturers and service providers, having their products and services assessed in accordance with ISO and IEC International standards allows them to distinguish themselves from less reputable suppliers. When public health, safety or the environment may be at stake, conformity assessment is often made obligatory by government regulations. Without appropriate assessment and approval, goods may be barred from sale, or suppliers disqualified from bidding for government procurement contracts.” 37 Not only supply providers but consumers and regulators as well, benefit from conformity assessment. See chapter 5 for more details. In the field of statistics, conformity assessment to international statistical standards is not very common. However, we can find some examples from the last years: Code of Practice, where self assessments were complemented with peer review. In the case of EDP report, audits are the usual tools. At national level there are different practices. Community of users, regular fora support right implementation, common understanding and further development. Users’ fora with regular meetings, conferences are places to share experiences, discuss problems and motivate reluctant people. In statistics, according to the survey 38 carried out in 2010 among methodologists in NSIs of Member States, the mostly used standard handbooks were on quality, seasonal adjustment and disclosure control. There are working groups, expert groups, task forces functioning in these fields as well as METIS workshops and the META Working Group. In the case of classifications, the Task Force on NACE critical classes were set up to assist right and common interpretation of NACE critical classes, therefore a common application of NACE Rev.2 in EU.

3.4. Summary

Standards become real standards if they are implemented. Having the standards in place, they represent an international consensus on good practices and therefore facilitate the common understandings and mutual recognition. Support measures form the basis for the enhancement and acceleration of implementation.

Support measures related to standards

Motivation Implementation for implementation Making easy Supporting Communication, promotion Availability Responsible body Awards Understandability Feedback possibilities Legal instruments Efficiency Support during implementation Consistency of standards Conformity assessment, Training Community of users, regular fora

The forms of support measures collected from the world of standards are summarised in the table above, but still, part of them is used in the field of statistical methodological standards.

37 ISO and conformity assessment, ISO Central Secretariat, Genève, 2005. p 2 http://www.iso.org/iso/casco_2005.pdf 38 Results were presented at the Eurostat workshop on standardisation October 2010. Analysis is downloadable from the workshop webpage. 87

88

4. Quality of statistical standards and criteria for the assessment

4.1. Introduction

One of the sub-tasks of WP 1 is a proposal for the definition of quality of statistical standards and – derived from that – criteria for the assessment of potential standards. Based on the definition of standard as a normative document the ESSnet has analysed six methodological handbooks in order to identify provisions referring to methodology, IT-tools, classifications and definitions. Additionally, the underlying data model contains (statistical) methods linked with methodology (based on) and IT-tools (implemented in). The following considerations refer explicitly to quality criteria on statistical methods and methodological standards. How they can be transformed to quality criteria on standards in general will be discussed later.

4.2. The concept of quality of standards

By defining standards the ESSnet follows the ISO concept and its definitions. Besides a definition of standards the ISO also provides a definition of quality as the ‘totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs’ (ISO 8402). The Qualistat Sub-Group ‘Quality of Norms’ 39 referred to this ISO definition already. One of their conclusion has been, that the ability of a norm (synonym for standard) to satisfy the needs of its users is the best criterion by which to assess its quality. Applying this definition to methodological standards or statistical methods as potential standards one has to identify the users of these potential standards as a first step and determine their needs. As for users of methodological standards the origin of the standardisation initiative is important. The initiative is part of the joint ESS strategy for the implementation of the Commission Communication on the production method for EU statistics (Vision). The idea behind the vision is that interested partners of the ESS community could work together in the development of methodology or other areas of co-operation leading to the application of methodological standards. As expressed in the subscription of the Eurostat grant for the ESSnet Preparation of Standardisation, one of the actions in the Hague action plan is to increase the efficiency and quality of statistical production by further standardisation of processes and methods. Having in mind the origin of the standardisation initiative, the ESS members as producers of statistics can be seen as the users of methodological standards 40 . Their needs – in respect to standards in general – may be simply summarised as the production of statistics as good and as efficient as possible (in absolute terms) or to increase quality and efficiency (in relative terms). 41

39 The Qualistat Sub-Group was created of the Quality Assurance Group to propose a definition of quality for norms and the means to measure that quality. In July 1998 the Sub-Group presented its final report. 40 There has been general agreement that not only NSIs but all producers of ESS statistics should be users of standards. 41 The Qualistat Sub-Group (see the previous footnote) has identified three main groups of users of norms. Besides producers of statistics, data suppliers and users of statistics were mentioned. In the context of the standardisation initiative described here, the latter ones could be regarded not as direct users of 89 This leads to defining the quality of methodological standards as the ability to contribute to the production of qualitative good statistical output in an efficient way (increase quality and efficiency of statistical production) for the ESS members. It is clear that a methodological standard as one single element in the whole production process of statistics does not determine output quality and production efficiency alone, but contributes to both in a certain extent. Thus, the marginal extent of this contribution should be the quality criterion. To do so the methodological standard must be applicable for its users. It is much easier to make a standard successful if its users benefit directly 42 .

The definition above leads to three main quality criteria: 1. The ability for ESS members to produce statistical output by using the methodological standard (applicability) 2. The quality of the statistical output produced by applying methodological standards (effectiveness) 3. The efficiency of the statistical production by applying the methodological standards (efficiency) In order to define criteria for the assessment of potential standards these criteria have to be specified further. 43

4.2.1. Applicability

The first requirement for methodological standards must be their applicability for the users (fitness for purpose) in one or more specific processes. Statistical methods have to fulfil several conditions to be methodological standards. A good and cost-efficient method for one partner may not be a good method for other partners. The requirements for methodological standards must be higher than for methods in a specific context because they are aimed at “common and repeated use” in the “given context” of the ESS. The given context especially includes the national statistical systems in different aspects (organisational, infrastructural, …). Several conditions can be derived for the applicability of standards in the ESS as adequate universal conditions. Relevant criteria are: ─ Relevance - A need for the standard exists - The standard is appropriate as far as the application leads to the goal by reasonable effort (see criteria 2 and 3) - The standard is credible (e. g. has a solid, professional basis to provide trust of users) - The standard is generally accepted ─ Clarity, specificity and accessibility 44 : the standard is - well-known - completely documented - clearly specified (in terms of preconditions, input, interfaces, functionality, …) - understandable for all users

methodological standards. Nevertheless, their needs will be reflected through the quality of statistical output (users) and the response burden (data suppliers). 42 Sponsorship on standardisation, Doc PG 2011/40/4.1, 40th Meeting of the Partnership Group, 25 March 2011 43 The issue about the quality of the standardisation process, which may also influence the quality of methodological standards, is not dealt here. 44 The document ISO/IEC Directives, Part 2, Rules for the structure and drafting of International Standards, lists some important features for documents to become standards, which are relevant in this context.

90 - readily and easily available to all users ─ Conformity and coherence: the standard is - conform with legal requirements (regulations, data protection, …) - conform with technical and organizational requirements - free of contradictions to other standards 45 - complimentary with other standards ─ Universality and stability - Application must be possible for all users or at least a relevant group of users - Applicability should be possible over a certain time ─ Exhaustiveness - Degree in which a standard can be applied for a certain task (fitness to the aim of standard) - The standards covers all necessary features ─ Credibility: the standard is - mutually recognized - the result of common work - state of the art (based on newest results of science, technology and experience) ─ Fitness for conformity assessment - The standard is valuable regarding their conformity with requirements ─ Robustness and simplicity: the standard is - easy to implement and apply - robust in relation to application errors

4.2.2. Effectiveness

Statistical methods are applied with the purpose to produce statistical output for users. Naturally, this implies that they are applicable for the production of the intended output. The degree to which this aim could be achieved is directly related to the quality of the statistical output. Consequently, the quality criteria for methodological standards are the quality dimensions of statistics to which they commit: ─ relevance, ─ accuracy, ─ timeliness and punctuality, ─ accessibility and clarity, ─ comparability and coherence.

4.2.3. Efficiency

This criterion is directly linked to cost efficiency as one of the expected gains of standardisation. As the increase of efficiency of statistical production is mentioned explicitly, the production costs must be criteria for methodological standards. Costs should include any kinds of efforts which will be generated by the application of methodological standards. Thus, both the costs for the producer (ESS member) as well as the costs for respondents (burden) and other data providers should be involved.

45 This criterion does not exclude to define different alternative standard methods for the same purpose. 91 Given the choice between different methods leading to the same quality of output, one would choose the statistical method with the lowest costs. It is also important to consider here relations between costs and benefits of standards in order to evaluate efficiency.

4.3. Relations between the proposed quality criteria on methodological standards

Whenever different criteria will be applied the question on their relations arises. Additionally, all criteria themselves have more than one dimension. The relations between the quality dimensions of statistics are well known, which is true for the relations between costs and burden as well. These questions will not be dealt here. As for methodological standards we propose to define minimum requirements for their applicability – independent from quality and efficiency of the statistical methods. Every statistical method which should become a methodological standard has to fulfill these minimum requirements. Concerning the quality of statistics and the cost-efficiency, there will be a trade-off. It seems clear, that ceteris paribus both criteria apply. The method with higher quality of output will be preferred, if the alternatives have equal cost efficiency. Under methods with equal cost efficiencies the method leading to the highest output quality will be preferred. The choice between two methods, one leading to better quality, the other with better cost efficiency is the most interesting question. The issue, how priorities should be generally set for standardisation has to be discussed at strategic level (Sponsorship Group).

Quality of statistical standards other than statistical methods Which other types of standards are considered will be determined in WP 2. We therefore propose to deal with the extension of quality criteria for those types of standards afterwards as part of WP 2.

4.4. Main problems connected with standards that may have quality character

To complete proposals on quality criteria regarding standards it is worth mentioning some problems that may have quality character. They are following: 1. Standard was created too late When national or international standard is created too late, individual creation of the standard is too expensive. When standards are created too late they are not able to respond on current and future users needs. 2. Standard is not adapted to the specific of the information area Standards should be used in the areas for which they were prepared. This guarantees that they are well adjusted to concrete area. 3. Requirements of the standard are too high It is difficult to meet the requirements, for example: a requirement for small enterprises to submit statistical declaration with information in constant prices. Standards should be based on realistic requirements. Hence, good standards include at least minimum requirements that are easy to fulfil. 4. Standard does not supply full documentation

92 For example, very often instructions, methodological notes are not complete, clear, comprehensible, or mistaking. Standards should be well-documented in order to fulfil criteria on accessibility and clarity. 5. Standard is out of date (obsolete) If the process of standard creation is long-lasting, standards may have lost their immediate interest. When standards have been created for a long period, there is a danger for standards to be irrelevant to current and future needs of users. 6. Standard is too general For example: too broad definition demands additional explanations and interpretations. The level of details of standards should be adequate for their aim. Too general or too detailed standards can bring some problems. 7. Standard is too detailed Too detailed standard is not applied or is a basis for individual/ local standard). See comment above. 8. Standard is unknown Standard should be popularised with information concerning advantages from standardisation and disadvantages from lack of standards. One attribute of standards is that they are well-known. Hence, dissemination and promotion actions for standards are very important. 9. Standard is premature If conditions for the implementation (technical, legal and organizational ones) do not exist, the standard is premature. Creation of standards is connected with sequence of actions which should have be well done. Hence, providing technical, organizational, legal conditions for these actions are key issues. 10. Implementation of the standard is too costly (expensive) When implementation is beyond financial means of potential standard users, or standards are too costly they can meet financial barriers. 11. Standard is unnecessary (needless) Standard is needless when it is not connected with users needs

93 5. Conformity to standards

5.1. Definition

The US standard organisations, CEN or ISO consider that conformity assessment is at the core of standardisation. They refer to similar reasons to explain to non-specialists why conformity assessment is important.

For NIST, “ Standards promote understanding between buyer and seller and make possible mutually beneficial commercial transactions. A product‘s conformance to accepted standards readily provides an efficient method of conveying complex information on the product‘s suitability .” 46

For the ISO , “Products and services are like promises. Business customers, consumers, users and public officials have expectations about products and services relating to features like quality, ecology, safety, economy, reliability, compatibility, interoperability, efficiency and effectiveness. Conformity assessment means evaluating and confirming such features as defined in standards, regulations and other specifications. In this way, conformity assessment makes sure that products and services deliver on their promises”. 47

“ISO/IEC 17000:2004 - Vocabulary and general principles”, defines conformity assessment as “demonstration that specified requirements relating to a product, process, system, person or body are fulfilled .” 48 Conformity assessment consists in conducting activities that are defined below, at the end of the paragraph on “ Standards in the EU and conformity assessment ”.

CEN’s glossary states that conformity evaluation means the “s ystematic examination of the extent to which a product, process or services fulfils specified requirements .”

The conformity of a product (respectively process, service) to a standard means the extent to which this product (respectively process, service) fulfils specified requirements of this standard .

Whereas there is little communication on conformity to standards in itself, the assessment of this conformity is at the heart of many references.

5.2. Conformity assessment is considered during the development of standards

ISO lists among the principles for drafting standards, the principle of verifiability 49 : Whatever the aims of a product standard, only such requirements shall be included as can be verified.

46 : in « THE ABC’s OF STANDARDS ACTIVITIES », NIST. 47 : in « ISO and conformity assessment », ISO 48 : In the previous version (1996), it was defined as «"any activity concerned with determining directly or indirectly that relevant requirements are fulfilled." 49 : In “ ISO/IEC Directives, Part 2 - “Rules for the structure and drafting of International Standards”, Annex A” 94 Requirements in documents shall be expressed in well-defined values (see 6.6.11). Phrases such as “sufficiently strong” or “of adequate strength” shall not be used.

Another consequence of the principle of verifiability is that the stability, reliability or lifetime of a product shall not be specified if no test method is known by means of which compliance with this requirement can be verified in a reasonably short time. A guarantee by the manufacturer, although useful, is not a substitute for such requirements. Guarantee conditions are considered to be outside the aspects to be included, being a commercial or contractual, and not a technical, concept.

Further, ISO includes this conformity assessment in its drafting rules, in chapter 6.7 50 . It has developed specific rules (standards ISO 17000 and 17025) that apply to the conformity assessment to all standards:

Aspects of conformity assessment

Documents containing requirements for products, processes, services, persons, systems and bodies All documents containing requirements for products, processes, services, persons, systems and bodies shall be written in accordance with the "neutrality principle", such that conformity can be assessed by a manufacturer or supplier (first party), a user or purchaser (second party), or an independent body (third party).

NOTE 1 : First-party, second-party and third-party conformity assessment activities are defined in ISO/IEC 17000.

NOTE 2: The term "documents" is defined in Clause 1.

Such documents shall not include requirements related to conformity assessment other than requirements which are necessary to provide repeatable and reproducible conformity assessment results. Committees wishing to specify additional conformity assessment requirements for the product, process, service, persons, systems or bodies may only do so in a separate document or in a separate part of the document provided that the separate parts can be applied independently. Prior to commencing work on a separate document or separate part, a committee shall seek the approval of the ISO Committee on conformity assessment (ISO/CASCO) […]. No document containing requirements for products, processes, services, persons, systems and bodies shall make conformity dependent on a quality management systems standard, i.e. it shall not, for example, make normative reference to ISO 9001.

Conformity assessment schemes and systems

Committees shall not develop documents providing general requirements for conformity assessment schemes and systems. Development of such documents is the responsibility of the ISO policy committee ISO/CASCO […].

50 : In ISO/IEC Directives, Part 2 - “Rules for the structure and drafting of International Standards”, chapter 6 95 Committees wishing a) To propose the establishment of a conformity assessment scheme or system, or b) To prepare documents specifying conformity assessment systems or schemes or sector-specific operating procedures for use by conformity assessment bodies and others for conformity assessment purposes, shall consult with the secretariat of ISO/CASCO or IEC/CAB or both as appropriate, prior to commencement of the work to ensure that any documents developed are in line with the conformity assessment policies and rules approved by ISO/CASCO and IEC/CAB as relevant.

References to ISO/IEC conformity assessment documents

When a committee develops a document relating to conformity assessment systems or schemes, or any other document addressing conformity assessment aspects, the document shall make normative reference to the relevant published ISO/IEC documents for conformity assessment procedures, including ISO/IEC 17000 and ISO/IEC 17025. The committee may include verbatim text from the ISO/IEC documents for conformity assessment procedures but the committee shall not delete, change or interpret them. Committees shall consult with the ISO/CASCO[…], for advice on correctly referencing the ISO/IEC conformity assessment documents. Any request for addition, deletion, change or interpretation shall be submitted to the secretariats of ISO/CASCO […] for decision.

Since national/ regional standards might be considered by the World Trade Organisation (WTO) as barriers to trade, there are strict rules that have to apply in order to avoid this shortcoming. The definition of cross-cutting rules for conformity assessment makes it easier to guarantee the conformity with WTO rules.

5.3. Standards in the EU and EU conformity assessment

5.3.1. Framework for EU standardisation

The EU with its New approach directives refers extensively to standards for the monitoring of the EU single market 51 . “Harmonised standards” define the conditions that apply to some kind of goods, for the monitoring of Health and Security in the EU.

The EU has defined its own principles, in the COUNCIL DECISION of 22 July 1993 concerning the modules for the various phases of the conformity assessment procedures and the rules for the affixing and use of the CE conformity marking 52 . There are about 10 procedures for conformity assessment which are to be used in the technical harmonization directives relating to the marketing of industrial products. The possible assessment modules are listed in this decision.

There are a number of principles that the technical harmonisation directives have to follow when choosing a conformity assessment procedure . Those of particular interest for this work are:

51 : For a comparison with the US system, and a short summary of the EU system, see A Guide to EU Standards and Conformity Assessment, NIST 52 See http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:1993:220:0023:0039:EN:PDF 96 a- the essential objective of a conformity assessment procedure is to guarantee the health and safety of users and consumers; b- conformity assessment can be subdivided into modules which relate to the design phase of products and to their production phase; c- as a general rule a product should be subject to both phases before being able to be placed on the market if the results are positive; d- it is necessary to take into consideration, the appropriateness of the modules to the type of products, the nature of the risks involved, the economic infrastructures of the given sector (e.g. existence or non-existence of third parties), the types and importance of production, etc. e- the cost efficiency of the procedures has to be taken into account

5.3.2. Modules for conformity assessment

Technical harmonisation directives refer to one module in the list below. Modules A, C and H can be used with additional provisions. Module C is designed to be used in combination with module B (called EC type-examination ). Modules D, E and F will also normally be used in combination with module B; however, in special cases they may be used on their own.

A summary of the provisions given in the Council directive is given below.

Module A ( internal production control )

1. The manufacturer ensures and declares that the products concerned satisfy the requirements of the directive that apply to them. The manufacturer draws up a written declaration of conformity. 2. The manufacturer must establish the technical documentation described in paragraph 3 and must keep it for a period ending at least 10 years for inspection purposes. 3. Technical documentation must enable the conformity of the product with the requirements of the directive to be assessed . It must, as far as relevant for such assessment, cover the design, manufacture and operation of the product.

Module Aa

This module consists of module A, plus the following supplementary requirements: For each product manufactured one or more tests on one or more specific aspects of the product must be carried out by the manufacturer or on his behalf. The tests are carried out on the responsibility of a notified body 53 chosen by the manufacturer.

Module B ( EC type-examination )

1. This module describes that part of the procedure by which a notified body ascertains and attests that a specimen , representative of the production envisaged, meets the provisions of the directive that apply to it.

53 Notified body = an organisation that is specialised in conformity assessment and that is registered by CEN= “third party”. 97 2. The application for the EC type-examination must be lodged by the manufacturer with a notified body of his choice. The applicant must place at the disposal of the notified body a specimen, representative of the production envisaged and hereinafter called 'type'. 3. The technical documentation must enable the conformity of the product with the requirements of the directive to be assessed. It must, as far as relevant for such assessment, cover the design, manufacture and operation of the product. 4. The notified body must: 4.1. verify that the type has been manufactured in conformity with the technical documentation and identify the elements which have been designed in accordance with the standards, as well as the components which have not. 4.2. for the latter components, perform the appropriate examinations and necessary tests to check whether the solutions adopted by the manufacturer meet the essential requirements of the directive; 4.3. perform the appropriate examinations and necessary tests to check whether for the former the standards have actually been applied; 5. Where the type meets the provisions of the directive, the notified body must issue an EC type- examination certificate to the applicant. Module C ( conformity to type ) 1. This module describes that part of the procedure whereby the manufacturer ensures and declares that the products concerned are in conformity with the type as described in the EC type-examination certificate and satisfy the requirements of the directive that applies to them. 2. The manufacturer must take all measures necessary to ensure that the manufacturing process assures compliance of the manufactured products with the type as described in the EC type- examination certificate and with the requirements of the directive that apply to them.

Possible supplementary requirements: For each product manufactured one or more tests on one or more specific aspects of the product must be carried out by the manufacturer. Module D ( production quality assurance ) 1. This module describes the procedure whereby the manufacturer ensures and declares that the products concerned are in conformity with the type as described in the EC type-examination certificate . 2. The manufacturer must operate an approved quality system for production, final product inspection and testing. 3. Quality system 3.1. The quality system must ensure compliance of the products. The quality system documentation must permit a consistent interpretation of the quality programmes, plan, manuals and records. 3.2. The notified body must assess the quality system to determine whether it satisfies the requirements. The auditing team must have at least one member with experience of evaluation in the product technology concerned. The evaluation procedure must include an inspection visit to the manufacturer's premises. 3.3. The manufacturer must undertake to fulfil the obligations arising out of the quality system as approved and to uphold it so that it remains adequate and efficient. 4. Surveillance under the responsibility of the notified body 4.1. The manufacturer must allow the notified body entrance for inspection purposes to the locations of manufacture, inspection and testing, and storage and must provide it with all necessary information.

98 4.2. The notified body must periodically carry out audits to make sure that the manufacturer maintains and applies the quality system and must provide an audit report to the manufacturer. 4.3. Additionally the notified body may pay unexpected visits to the manufacturer. Module E ( product quality assurance ) 1. This module describes the procedure whereby the manufacturer ensures and declares that the products concerned satisfy the requirements of the directive that apply to them. 2. The manufacturer must operate an approved quality system for final product inspection and testing. 3. Quality system 3.1. The manufacturer must lodge an application for assessment of his quality system for the products concerned, with a notified body of his choice. 3.2. Under the quality system, each product must be examined and appropriate tests as set out in the relevant standard (s) or equivalent tests shall be carried out in order to ensure its conformity with the relevant requirements of the directive. 3.3. The notified body must assess the quality system to determine whether it satisfies the requirements. The auditing team must have at least one member experienced as an assessor in the product technology concerned. The assessment procedure must include an assessment visit to the manufacturer's premises. 4. Surveillance under the responsibility of the notified body 4.1. The manufacturer must allow the notified body entrance for inspection purposes to the locations of inspection, testing and storage. 4.3. The notified body must periodically carry out audits to ensure that the manufacturer maintains and applies the quality. 4.4. Additionally, the notified body may pay unexpected visits to the manufacturer. Module F ( product verification ) 1. This module describes the procedure whereby a manufacturer checks and attests that the products are in conformity with the type as described in the EC-type examination. 2. The manufacturer must take all measures necessary in order that the manufacturing process ensures conformity of the products. 3. The notified body must carry out the appropriate examinations and tests in order to check the conformity of the product with the requirements of the directive either by examination and testing of every product or by examination and testing of products on a statistical basis at the choice of the manufacturer. 4. Verification by examination and testing of every product All products must be individually examined and appropriate tests as set out in the relevant standard or equivalent tests shall be carried out in order to verify their conformity. 5. Statistical verification 5.1. The manufacturer must present his products in the form of homogeneous lots. 5.2. A random sample shall be drawn from each lot. Products in a sample shall be individually examined and appropriate tests as set out in the relevant standard. Module G ( unit verification ) 1. This module describes the procedure whereby the manufacturer ensures and declares that the product concerned, which has been issued with the certificate referred to in point 2, conforms to the requirements of the directive that apply to it. 2. The notified body must examine the individual product and carry out the appropriate tests as set out in the relevant standard or equivalent tests, to ensure its conformity with the relevant requirements of the directive. The notified body must affix, or cause to be affixed, its identification

99 number on the approved product and shall draw up a certificate of conformity concerning the tests carried out. Module H ( full quality assurance ) 1. This module describes the procedure whereby the manufacturer declares that the products concerned satisfy the requirements of the directive that apply to them. 2. The manufacturer must operate an approved quality system for design , manufacture and final product inspection and testing and shall be subject to surveillance. 3. Quality system 3.1. The quality system must ensure compliance of the products with the requirements of the directive that apply to them. It must contain in particular an adequate description of: - the quality objectives and the organizational structure, responsibilities and powers of the management with regard to design and product quality, - the technical design specifications, including standards, that will be - the design control and design verification techniques , processes and systematic actions that will be used when designing the products, - the corresponding manufacturing, quality control and quality assurance techniques , processes and systematic actions that will be used, - the examinations and tests that will be carried out before, during and after manufacture, and the frequency with which they will be carried out. 3.2. The notified body must assess the quality system to determine whether it satisfies the requirements. The auditing team must have at least one member experienced as an assessor in the product technology concerned. The evaluation procedure shall include an assessment visit to the manufacturer's premises. 4. EC surveillance under the responsibility of the notified body 4.1. The manufacturer must allow the notified body entrance for inspection purposes to the locations of design, manufacture, inspection and testing, and storage 4.3. The notified body must periodically carry out audits to make sure that the manufacturer maintains and applies the quality system and shall provide an audit report to the manufacturer. 4.4. Additionally the notified body may pay unexpected visits to the manufacturer. At the time of such visits, the notified body may carry out tests or have them carried out in order to check the proper functioning of the quality system where necessary.

Possible supplementary requirements:

Design examination

1. The manufacturer must lodge an application for examination of the design with a single notified body . 2. The application must enable the design, manufacture and operation of the product to be understood , and shall enable conformity with the requirements of the directive to be assessed. It must include: - the technical design specifications, including standards, that have been applied, - the necessary supporting evidence for their adequacy, in particular where the standards have not been applied in full. This supporting evidence must include the results of tests carried out by the appropriate laboratory of the manufacturer or on his behalf. 3. The notified body must examine the application and where the design meets the provisions of the directive that apply to it must issue an EC design examination certificate to the applicant.

100 We can see much similarity with the tools used by ISO that gives the following summary of conformity assessment activities:

Conformity assessment may consist of any one of, some of, or all of the following : sample testing, inspection, process evaluation, supplier’s declaration of conformity, management system certification/registration, personnel certification, product certification, mutual recognition of results and the accreditation and peer assessment of the competence of the organizations conducting these activities – collectively known as “conformity assessment bodies”. ISO and IEC jointly develop standards and guides for all those activities.

5.4. How to assess conformity to methodological standards in statistics?

5.4.1. Problem statement in the ESS

It is good practice to investigate how successful methods can be transposed to our domain. We will consider the EU practice in conformity assessment, even if some limits have to be kept in mind: • the EU harmonised directives refer to the standards for goods, not for services. Some investigation of conformity assessment in the domain of services has been made, but it seems that the framework for such activities is still under development 54 . • they are dealing with market activities. As in other chapters of the report from the ESSnet on the Preparation of standardisation, we recall that the group of institutions interested in the development of standards for the implementation of Eurostat’s vision for the next decade is a much smaller group and differs from a market.

The methodological standards are intended for the production of statistical specifications, especially in the domain of statistical processing (sampling, detection of reporting errors and imputation of missing values). Such statistical specifications are intermediate products in the production chain under the responsibility of NSIs. These products will have to be assessed for conformity to methodological standards. They are the result of a design activity .

In addition, such products are “one of a kind”. All aspects related to mass production, and that lead to techniques like control of samples are not directly relevant for a design activity.

Among the principles mentioned by the Council decision on conformity assessment, the last two refer to the evaluation of risks and to the efficiency and cost of the assessment procedure. In short, it makes sense to envisage procedures whose cost is proportional to the risk attached to non- conformity.

As a benchmark for the price to pay for conformity assessment, we can consider existing practice. Conformity to standards is already assessed in the following domains: • implementation of the ESS Code of practice • production of Gross National Income statistics

54 See Annex B in “CEN - INTERNAL REGULATIONS - PART 3 - TERMS OF REFERENCE OF THE CEN CERTIFICATION BOARD (2006-9)” or “ CEN’s Horizontal European Services Standardization Strategy CHESSS Feasibility Study CONSOLIDATED REPORT”. 101 • production of statistics for the Excessive Deficit Procedure, ie. Deficit and Debt of public administrations.

In the last two domains, the conformity assessment involves very strict procedures. This is due to the strategic nature of these statistics: EDP statistics are vital for the euro currency and the trust in debts of national public administrations. Hence it is as well crucial for the whole EU economy, since many EU economic actors have liabilities connected to these debts. In this case, inspections of other administrations than the NSIs have been recently granted to assess the quality of these statistics. Regular visits are organised by Eurostat, which is in charge of the control of conformity to the standard. In this respect, the standard is part of the ESA, which is the the standard for EU natinal accounts.

The production of Gross National Income statistics is very important as well, since it defines the contribution of each Member State to the EU budget. It is regularly controled both by Eurostat and the European Court of Auditors. In addition to the description of methods used by NSIs, some visits can be organised by the assessors on the premices of NSIs.

As for the ESS Code of practice, the assessment of its implementation is currently 55 conducted through self-evaluation of NSIs, that answer to questionnaires sent out either by Eurostat or by the European Statistical Governance Advisory Body (ESGAB).

For the production of other statistics, the assessment of conformity to standards (regulations defining statistics are mandatory standards) is in general very light, made of periodic quality reports, and/ or of internal quality reviews, ie. based on self-assessment by the NSI 56 . These reports tend to focus on specific/ limited aspects of the features of these statistics. For a few statistics - like Harmonised Consumer Price Index - conformity is monitored more closely, with the involvement of Eurostat.

What is the risk attached to non-conformity to standards for statistical specifications, in the framework of the vision, ie. of the integration of the statistical production inside the ESS?

Some hypotheses have to be made on such an integration. We consider here the following possible organisation: • several NSIs would have agreed to integrate their production for one kind of statitics • all would share the workload between themselves • they would all keep responsibility for data collection (due to linguistic, cultural and legal reasons) • one NSI would be responsible for specifying the statistical aspects of the production • downstream, several types of organisation could be envisaged: 1) every NSI implements independently these specifications on its own IT system 2) another NSI develops portable IT programs that can be implemented by all NSIs 3) any intermediate solution between 1) and 2) could be envisaged.

Whereas delegating the development of specifications might reduce the costs (provided the specifications needed by the cooperating NSIs are similar), every non-conformity in the specification would be implemented in all production systems. Hence, the risk attached to non- conformity in the specifications developed by one singular NSI is highly increased.

55 : This procedure should be revised along the proposals made by the Commission in its Communication 211/2011. 56 : A list of legal requirements for quality reports can be found in the ESS Handbook for quality reports, chapter 3. 102 This organisation would then require conformity assessment procedures for common interest of ESS: • for Eurostat, a non-conformity in statistics produced in one single NSI can be of limited impact on EU-wide statistics - and can even be partially compensated using the statistics produced by other NSIs. If several NSIs implemented one single set of specifications that is not conform to standards, then this would have larger consequences on EU agregates and it would be much more difficult to adjust for shortcomings in the methods. • NSIs involved in the cooperation would inherit the non-conformity due to the NSI in charge of the specification.

As a consequence, it seems quite necessary to look for a conformity assessment scheme that would limit the risk to the levels currently known, ie in the absence of integration of the development of specifications. However, the cost of this assessment has to be limited, so as to keep the benefit of this integration, ie. cost reduction.

5.4.2. Proposal for conformity assessment

We have to deal with two features in conformity assessment: which procedures could be chosen and which “notified body” could be chosen? The ESSnet is not in a position to make proposals in such matters. It limits itself to propose a technical assessment of pros and cons of several options, in order to assist the Sponsorship on standardisation in making the corresponding proposals.

5.4.2.1. Procedure for conformity assessment:

We will consider all procedures used by EU harmonised directives.

First, we have to take into account that statistical specifications can be tested for conformity by a third body only if they are provided with full documentation, and if assistance is provided to those who are in charge of checking the conformity. These conditions are mentioned for the option “design examination” of module H in the Council decision on conformity assessment.

Second, we will take into account the fact that statistical specifications are “one of a type” productions, whereas the types of conformity assessment of the EU decision on conformity assessment cover both mass production and one of a type production.

If we consider that there is no “ESS standard” on the way to write statistical specifications, it is difficult to imagine controls of conformity relying on sampling, or even quality assurance methods. If in the medium or long term such standard on the drafting of specifications were agreed, it might be possible to define homogeneous subsets of one specification 57 , and to check only a few subsets chosen at random. In a similar way, it would then be possible to define a quality common assurance system for this activity.

We are dealing here only with the assessment of conformity to methodological standards . This conformity will not guarantee that such specifications are fully satisfactory. They have as well to be

57 : For instance, such standards could lead to group all specifications dealing with one characteristic into one lot. Such a lot would be clearly identified. The whole set of specifications would be made of some pre-processing or post- processing and a series of lots, each one corresponding to one characteristic for which statistics have to be produced. It would then be possible to check conformity independently for each characteristic. Some care should however be paid to the cross-cutting specifications, to be sure that they do not alter this conformity. 103 checked for logical consistency and completeness . This is out of the scope of this chapter. However, developing an ESS standard for the drafting of statistical specifications and defining a common system for quality assurance would as well be useful for this other important concern.

Type of conformity Case of “one of a type” production/ Content (summary) assessment/ statistical specifications (module) internal self-declaration by producer possible in the short term production control (A) EC type- examination of a specimen by an possible if standards on drafting examination(B) independent body specifications are developed with Design examination conformity to self-declaration of control of the possible if standards on drafting type (C) with regularity specifications are developed Design examination production production quality assurance possible if standards on drafting quality audited by an independent body specifications and if a quality assurance (D) assurance system are developed product quality production, inspection and testing possible if standards on drafting assurance (E) quality assurance audited by an specifications and if a quality independent body assurance system are developed product random testing by an independant possible if standards on drafting verification (F) body specifications are developed with Design examination unit verification every product tested by an possible in the short term (G) with Design independant body examination full quality identical to production quality assurance (H) assurance since the production activity is a design activity

As a conclusion, it seems that the choice of procedures would be very limited in the short term: either the NSI in charge of the drafting of specifications guarantees himself the conformity, or a third party assesses this conformity, with the assistance of the author so as to understand the specification.

5.4.2.2. Who should be in charge of the assessment of conformity?

We could envisage in theory three kind of actors: 1. an organisation independent from the ESS 2. a NSI 3. Eurostat

104 It seems difficult to envisage the involvement of an independent organization, a “notified body” as mentioned in the EU Council decision. First, they have not worked with standards in the domain of public statistics up to now. Even in private statistics, their activity is very limited. Since the amount of activity would be very limited, it would be very costly to hire competent people in a position to work in many languages 58 … The financial issue should as well be addressed (who should pay for the assessment…).

As for relying on an NSI, it would be difficult as well: • some languages issues might be met 59 , • NSIs have to reach agreements in the negotiation process inside the ESS, and the independence between them would be difficult to secure • the NSI in charge of the assessment would have to invest heavily in the domain, and a one- off involvement would be probably very costly due to fixed costs (specific training costs in the domain of conformity assessment) • however, the technique involved are in general practiced by NSIs, and they should have competent statisticians in the domains covered by the methodological standards.

As for relying on Eurostat for this assessment, the situation is the following: • it is in the culture of Eurostat staff to have an independent view on work done by NSIs • Eurostat would not much less hampered than NSIs if the work had to be conducted in various languages and located in several NSIs • Eurostat has more interest in promoting the implementation of the vision (and such standardisation) than some NSIs • however, in general, Eurostat is not in charge of many aspects of methodology related to the production of statistics (sampling, editing, imputation). Some investment in Human resources would be necessary as well • there is very few success in the development of long term cooperation between NSIs without the support of Eurostat 60 . Financial compensation between NSIs would probably be much more complicated than securing money from the EU budget under the responsibility of Eurostat…

58 : Unless there is an agreement to develop these specifications only in English 59 : Unless there is an agreement to develop these specifications only in English 60 : The exception seems to be found in the cooperation between Nordic NSIs, with a tool like PC-Axis 105 Annex 1 - ISO/ IEC Guide 2 - Standardization and related activities - General vocabulary

Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission) form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC participate in the development of International Standards through technical committees established by the respective organization to deal with particular fields of technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the work International Standards are drafted in accordance with the rules given in the ISI/IEC Directives, Part 2.

Draft Guides adopted by the responsible Committee or Group are circulated to national bodies for voting.

Publication as a Guide requires approval by at least 75 % of the national bodies casting a vote.

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.

The first edition of this Guide was published by ISO in 1976. It was prepared by the United Nations Economic Commission for Europe (UN/ECE) in consultation with ISO, primarily to facilitate the work of the UN/ECE aimed at removal of barriers to international trade arising from lack of harmonization of standards or inadequate international application of standards.

The second, third and fourth editions of ISO Guide 2 added new terms and definitions in more or less isolated groups, within no overall planned framework. Work on a systematically structured revision was authorized by the ISO Council in 1981 as the principal task of a working group within the ISO Committee on standardization principles (ISO/STACO), including IEC and UN/ECE spokesmen and in cooperation with the ISO Committee on conformity assessment (ISO/CASCO) and the International Laboratory Accreditation Conference (ILAC). The main reason for this activity was a request from UN/ECE for complete and urgent revision of ISO Guide 2 to take into account a number of specific problems raised by the UN/ECE Government Officials Responsible for Standardization Policies as well as the proposed terminology being prepared in the UN/ECE Working Party on Building. Moreover, ISO was publishing an increasing number of Guides on standardization policies and procedures, several of them in conjunction with IEC. To guard against terminological inconsistencies, a clear understanding of basic concepts needed to be established, and unambiguous general terms and definitions provided.

The resulting fifth edition of Guide 2 was issued, for the first time as a joint ISO/IEC publication, in 1986.

The sixth edition (1991) incorporated amendments in response to specific questions raised by ISO and IEC members, requests from the General Agreement on Tariffs and Trade (GATT) and the UN/ECE, and proposals from ISO/CASCO. The seventh edition (1996) preserved the structure of its recent predecessors by grouping related concepts under generic clause headings. Thus, Clauses 1 to 11 concerned standardization,

106 proceeding from the more general to more specific aspects. Similarly, the terms and definitions given in the revised Clauses 12 to 17 covered the broad spectrum of conformity assessment. These latter clauses were modified on the recommendations of ISO/CASCO to reflect developments in conformity assessment activities, and were approved by the national bodies of ISO and IEC in 1994. They were not intended to provide a comprehensive classification but included, in particular, concepts related to the following typical sequence: a) determination of product, process or service characteristics (Clause 13); b) comparison of characteristics with specified requirements, i.e. conformity evaluation (Clause 14); c) assurance of conformity, e.g. by supplier's declaration or by certification (Clause 15).

This eighth edition (2004) is identical to the seventh edition except that the terms and definitions on conformity assessment (in Clauses 12 to 17 in the 7th edition) have been deleted and replaced by a normative reference to ISO/IEC 17000, Conformity assessment — Vocabulary and general principles .

107 Introduction

Whilst ISO and IEC jointly provide the international framework for standardization, covering a wide range of mainly technical matters, the general concept of standardization extends beyond their terms of reference. For ISO and IEC purposes, the definition of standardization (1.1) should accordingly be read in conjunction with the definitions of standard (3.2) and consensus (1.7). The general terminological principles guiding the revisions of Guide 2 since 1986 have been those established by Technical Committee ISO/TC 37, Terminology and other language resources .

Terms expressing more specific concepts may generally be constructed by a combination of terms representing more general concepts. The latter terms thus form “building blocks”, and the selection of terms and the construction of definitions within this Guide has been based on this approach in cases where equivalent English, French and Russian combined terms contain the same “building blocks”. In this way, additional terms can be readily constructed within the framework of the Guide. For example, the term safety standard can be defined as a standard (3.2) concerned with freedom from unacceptable risk of harm (definition of safety in 2.5). Words placed between parentheses “(. . . . .)” within some terms may be omitted if it is unlikely that this will cause confusion. The definitions included in this Guide are directly equivalent in the English, French and Russian languages.

They have been drawn up as concisely as practicable. Where they first occur in other definitions, terms which have themselves been defined in this Guide are printed in bold type. The notes appended to certain definitions offer clarification, explanation or examples to facilitate understanding of the concepts described. In certain cases, the notes may differ in different languages for linguistic reasons, or additional notes may be given. In science and technology, the English word “ standard ” is used with two different meanings: as a normative document as defined in 3.2 (in French “ norme ”, in Russian “ стандарт ”) and also as a measurement standard (in French “ étalon ”, in Russian “ эталон ”). This Guide is concerned solely with the first meaning. The second is the province of the International vocabulary of basic and general terms in metrology (VIM ).

108 Standardization and related activities General vocabulary

Scope

This Guide provides general terms and definitions concerning standardization and related activities. It is intended to contribute fundamentally towards mutual understanding amongst the members of ISO and IEC and the various governmental and nongovernmental agencies involved in standardization at international, regional and national levels. It is intended also to provide a suitable source for teaching and for reference, briefly covering basic theoretical and practical principles of standardization, certification and laboratory accreditation. It is not the aim of this Guide to duplicate definitions of terms adequately defined for general purposes in other authoritative international vocabularies.

NOTE 1 From this point of view, particular attention is drawn to the International vocabulary of basic and general terms in metrology (VIM )jointly prepared by ISO, IEC, BIPM,IFCC, IUPAC, IUPAP and OIML, and published in 1993 (second edition).

NOTE 2 In addition to the terms given in the official languages of ISO and IEC (English, French and Russian),equivalent terms provided by the relevant member bodies are given in the following languages: • German (de); • Spanish (es); • Italian (it); • Dutch (nl); • Swedish (sv).

Normative references

The following referenced documents are indispensable for the application of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies. ISO/IEC 17000, Conformity assessment — Vocabulary and general principles

1 Standardization

1.1 standardization activity of establishing, with regard to actual or potential problems, provisions for common and repeated use, aimed at the achievement of the optimum degree of order in a given context

NOTE 1 In particular, the activity consists of the processes of formulating, issuing and implementing standards .

NOTE 2 Important benefits of standardization are improvement of the suitability of products, processes and services for their intended purposes, prevention of barriers to trade and facilitation of technological cooperation.

1.2 subject of standardization topic to be standardized

NOTE 1 The expression “product, process or service” has been adopted throughout this Guide to encompass the subject of standardization in a broad sense, and should be understood equally to cover, for example, any material, component, equipment, system, interface, protocol, procedure, function, method or activity.

109

NOTE 2 Standardization may be limited to particular aspects of any subject. For example, in the case of shoes, criteria could be standardized separately for sizes and durability.

1.3 field of standardization domain of standardization - (deprecated) group of related subjects of standardization

NOTE Engineering, transport, agriculture, and quantities and units, for example, could be regarded as fields of standardization.

1.4 state of the art developed stage of technical capability at a given time as regards products, processes and services, based on the relevant consolidated findings of science, technology and experience

1.5 acknowledged rule of technology technical provision acknowledged by a majority of representative experts as reflecting the state of the art

NOTE A normative document on a technical subject, if prepared with the cooperation of concerned interests by consultation and consensus procedures, is presumed to constitute an acknowledged rule of technology at the time of its approval.

1.6 level of standardization geographical, political or economic extent of involvement in standardization

1.6.1 international standardization standardization in which involvement is open to relevant bodies from all countries

1.6.2 regional standardization standardization in which involvement is open to relevant bodies from countries from only one geographical, political or economic area of the world

1.6.3 national standardization standardization that takes place at the level of one specific country

NOTE Within a country or a territorial division of a country, standardization may also take place on a branch or sectoral basis (e.g. ministries), at local levels, at association and company levels in industry and in individual factories, workshops and offices.

1.6.4 provincial standardization standardization that takes place at the level of a territorial division of a country

NOTE Within a country or a territorial division of a country, standardization may also take place on a branch or sectoral basis (e.g. ministries), at local levels, at association and company levels in industry and in individual factories, workshops and offices.

1.7 consensus general agreement, characterized by the absence of sustained opposition to substantial issues by any important part of the concerned interests and by a process that involves seeking to take into account the views of all parties concerned and to reconcile any conflicting arguments NOTE Consensus need not imply unanimity.

2 Aims of standardization

110

NOTE The general aims of standardization follow from the definition in 1.1. Standardization may have one or more specific aims, to make a product, process or service fit for its purpose. Such aims can be, but are not restricted to, variety control , usability, compatibility , interchangeability , health, safety , protection of the environment , product protection , mutual understanding, economic performance, trade. They can be overlapping.

2.1 fitness for purpose ability of a product, process or service to serve a defined purpose under specific conditions

2.2 compatibility suitability of products, processes or services for use together under specific conditions to fulfill relevant requirements without causing unacceptable interactions

2.3 interchangeability ability of one product, process or service to be used in place of another to fulfill the same requirements

NOTE The functional aspect of interchangeability is called “functional interchangeability”, and the dimensional aspect “dimensional interchangeability”.

2.4 variety control selection of the optimum number of sizes or types of products, processes or services to meet prevailing needs

NOTE Variety control is usually concerned with variety reduction.

2.5 safety freedom from unacceptable risk of harm

NOTE In standardization , the safety of products, processes and services is generally considered with a view to achieving the optimum balance of a number of factors, including non-technical factors such as human behaviour, that will eliminate avoidable risks of harm to persons and goods to an acceptable degree.

2.6 protection of the environment preservation of the environment from unacceptable damage from the effects and operations of products, processes and services

2.7 product protection environmental protection (deprecated) protection of a product against climatic or other adverse conditions during its use, transport or storage

111 3 Normative documents

3.1 normative document document that provides rules, guidelines or characteristics for activities or their results

NOTE 1 The term “normative document” is a generic term that covers such documents as standards , technical specifications , codes of practice and regulations .

NOTE 2 A “document” is to be understood as any medium with information recorded on or in it.

NOTE 3 The terms for different kinds of normative documents are defined considering the document and its content as a single entity.

3.2 standard document, established by consensus and approved by a recognized body , that provides, for common and repeated use, rules, guidelines or characteristics for activities or their results, aimed at the achievement of the optimum degree of order in a given context NOTE Standards should be based on the consolidated results of science, technology and experience, and aimed at the promotion of optimum community benefits.

3.2.1 Publicly available standards

NOTE By virtue of their status as standards , their public availability and their amendment or revision as necessary to keep pace with the state of the art , international, regional, national and provincial standards (3.2.1.1, 3.2.1.2, 3.2.1.3 and 3.2.1.4) are presumed to constitute acknowledged rules of technology .

3.2.1.1 international standard standard that is adopted by an international standardizing/ standards organization and made available to the public

3.2.1.2 regional standard standard that is adopted by a regional standardizing/ standards organization and made available to the public

3.2.1.3 national standard standard that is adopted by a national standards body and made available to the public

3.2.1.5 provincial standard standard that is adopted at the level of a territorial division of a country and made available to the public

3.2.2 Other standards

NOTE Standards may also be adopted on other bases, e.g. branch standards and company standards. Such standards may have a geographical impact covering several countries.

3.3 pre-standard document that is adopted provisionally by a standardizing body and made available to the public in order that the necessary experience may be gained from its application on which to base a standard

3.4 technical specification document that prescribes technical requirements to be fulfilled by a product, process or service

112

NOTE 1 A technical specification should indicate, whenever appropriate, the procedure(s) by means of which it may be determined whether the requirements given are fulfilled.

NOTE 2 A technical specification may be a standard , a part of a standard or independent of a standard.

NOTE 3 Note 3 applies to the Russian version only.

3.5 code of practice document that recommends practices or procedures for the design, manufacture, installation, maintenance or utilization of equipment, structures or products NOTE A code of practice may be a standard , a part of a standard or independent of a standard.

3.6 regulation document providing binding legislative rules, that is adopted by an authority

3.6.1 technical regulation regulation that provides technical requirements , either directly or by referring to or incorporating the content of a standard , technical specification or code of practice

NOTE A technical regulation may be supplemented by technical guidance that outlines some means of compliance with the requirements of the regulation, i.e. deemed-to-satisfy provision .

4 Bodies responsible for standards and regulations

4.1 body responsible for standards and regulations legal or administrative entity that has specific tasks and composition

NOTE Examples of bodies are organizations , authorities , companies and foundations.

4.2 organization body that is based on the membership of other bodies or individuals and has an established constitution and its own administration

4.3 standardizing body body that has recognized activities in standardization

4.3.1 regional standardizing organization standardizing organization whose membership is open to the relevant national body from each country within one geographical, political or economic area only

4.3.2 international standardizing organization standardizing organization whose membership is open to the relevant national body from every country

4.4 standards body standardizing body recognized at national, regional or international level, that has as a principal function, by virtue of its statutes, the preparation, approval or adoption of standards that are made available to the public

113 NOTE A standards body may also have other principal functions.

4.4.1 national standards body standards body recognized at the national level, that is eligible to be the national member of the corresponding international and regional standards organizations

4.4.2 regional standards organization standards organization whose membership is open to the relevant national body from each country within one geographical, political or economic area only

4.4.3 international standards organization standards organization whose membership is open to the relevant national body from every country

4.5 authority body that has legal powers and rights

NOTE An authority can be regional, national or local.

4.5.1 regulatory authority authority that is responsible for preparing or adopting regulations

4.5.2 enforcement authority authority that is responsible for enforcing regulations NOTE The enforcement authority may or may not be the regulatory authority .

5 Types of standards

NOTE The following terms and definitions are not intended to provide a systematic classification or comprehensive list of possible types of standards . They indicate some common types only. These are not mutually exclusive; for instance, a particular product standard may also be regarded as a testing standard if it provides test methods for characteristics of the product in question.

5.1 basic standard standard that has a wide-ranging coverage or contains general provisions for one particular field

NOTE A basic standard may function as a standard for direct application or as a basis for other standards.

5.2 terminology standard standard that is concerned with terms, usually accompanied by their definitions, and sometimes by explanatory notes, illustrations, examples, etc.

5.3 testing standard standard that is concerned with test methods , sometimes supplemented with other provisions related to testing , such as sampling, use of statistical methods, sequence of tests

5.4 product standard standard that specifies requirements to be fulfilled by a product or a group of products, to establish its fitness for purpose

NOTE 1 A product standard may include in addition to the fitness for purpose requirements, directly or by reference, aspects such as terminology, sampling, testing , packaging and labelling and, sometimes, processing requirements.

114

NOTE 2 A product standard can be either complete or not, according to whether it specifies all or only a part of the necessary requirements. In this respect, one may differentiate between standards such as dimensional, material, and technical delivery standards.

5.5 process standard standard that specifies requirements to be fulfilled by a process, to establish its fitness for purpose

5.6 service standard standard that specifies requirements to be fulfilled by a service, to establish its fitness for purpose

NOTE Service standards may be prepared in fields such as laundering, hotel-keeping, transport, car- servicing, telecommunications, insurance, banking, trading.

5.7 interface standard standard that specifies requirements concerned with the compatibility of products or systems at their points of interconnection

5.8 standard on data to be provided standard that contains a list of characteristics for which values or other data are to be stated for specifying the product, process or service

NOTE Some standards, typically, provide for data to be stated by suppliers, others by purchasers.

6 Harmonization of standards

NOTE Technical regulations can be harmonized like standards . Corresponding terms and definitions are obtained by replacing “standards” by “technical regulations” in the definitions 6.1 to 6.9, and “ standardizing bodies ” by “ authorities ” in definition 6.1.

6.1 harmonized standards equivalent standards standards on the same subject approved by different standardizing bodies , that establish interchangeability of products, processes and services, or mutual understanding of test results or information provided according to these standards

NOTE Within this definition, harmonized standards might have differences in presentation and even in substance, e.g. in explanatory notes, guidance on how to fulfil the requirements of the standard, preferences for alternatives and varieties.

6.2 unified standards harmonized standards that are identical in substance but not in presentation

6.3 identical standards harmonized standards that are identical in both substance and presentation

NOTE 1 Identification of the standards may be different.

NOTE 2 If in different languages, the standards are accurate translations.

115

6.4 internationally harmonized standards standards that are harmonized with an international standard

6.5 regionally harmonized standards standards that are harmonized with a regional standard

6.6 multilaterally harmonized standards standards that are harmonized between more than two standardizing bodies

6.7 bilaterally harmonized standards standards that are harmonized between two standardizing bodies

6.8 unilaterally aligned standard standard that is aligned with another standard so that products, processes, services, tests and information provided according to the former standard meet the requirements of the latter standard but not vice versa

NOTE A unilaterally aligned standard is not harmonized (or equivalent) with the standard with which it is aligned.

6.9 comparable standards standards on the same products, processes or services, approved by different standardizing bodies , in which different requirements are based on the same characteristics and assessed by the same methods, thus permitting unambiguous comparison of differences in the requirements

NOTE Comparable standards are not harmonized (or equivalent ) standards .

7 Content of normative documents

7.1 provision expression in the content of a normative document , that takes the form of a statement , an instruction , a recommendation or a requirement

NOTE These types of provision are distinguished by the form of wording they employ; e.g. instructions are expressed in the imperative mood, recommendations by the use of the auxiliary “should” and requirements by the use of the auxiliary “shall”.

7.2 statement provision that conveys information

7.3 instruction provision that conveys an action to be performed

7.4 recommendation provision that conveys advice or guidance

7.5 requirement provision that conveys criteria to be fulfilled

116 7.5.1 exclusive requirement mandatory requirement (deprecated) requirement of a normative document that must necessarily be fulfilled in order to comply with that document

NOTE The term “mandatory requirement” should be used to mean only a requirement made compulsory by law or regulation .

7.5.2 optional requirement requirement of a normative document that must be fulfilled in order to comply with a particular option permitted by that document

NOTE An optional requirement may be either a) one of two or more alternative requirements; or b) an additional requirement that must be fulfilled only if applicable and that may otherwise be disregarded.

7.6 deemed-to-satisfy provision provision that indicates one or more means of compliance with a requirement of a normative document

7.7 descriptive provision provision for fitness for purpose that concerns the characteristics of a product, process or service

NOTE A descriptive provision usually conveys design, constructional details, etc. with dimensions and material composition.

7.8 performance provision provision for fitness for purpose that concerns the behavior of a product, process or service in or related to use

8 Structure of normative documents

8.1 body of a normative document set of provisions that comprises the substance of a normative document

NOTE 1 In the case of a standard , the body comprises general elements relating to its subject and definitions, and main elements conveying provisions.

NOTE 2 Parts of the body of a normative document may take the form of annexes (“normative annexes”) for reasons of convenience, but other (informative) annexes may be additional elements only.

8.2 additional element information that is included in a normative document but has no effect on its substance

NOTE In the case of a standard , additional elements may include, for example, details of publication, foreword and notes.

9 Preparation of normative document

9.1 standards programme 117 working schedule of a standardizing body , that lists its current items of standardization work

9.1.1 standards project specific work item within a standards programme

9.2 draft standard proposed standard that is available generally for comment, voting or approval

9.3 period of validity period of time for which a normative document is current, that lasts from the date on which it becomes effective (“effective date”), resulting from a decision of the body responsible for it, until it is withdrawn or replaced

9.4 review activity of checking a normative document to determine whether it is to be reaffirmed, changed or withdrawn

9.5 correction removal of printing, linguistic and other similar errors from the published text of a normative document

NOTE The results of correction may be presented, as appropriate, by issuing a separate correction sheet or a new edition of the normative document.

9.6 amendment modification, addition or deletion of specific parts of the content of a normative document

NOTE The results of amendment are usually presented by issuing a separate amendment sheet to the normative document.

9.7 revision introduction of all necessary changes to the substance and presentation of a normative document

NOTE The results of revision are presented by issuing a new edition of the normative document.

9.8 reprint new impression of a normative document without changes

9.9 new edition new impression of a normative document that includes changes to the previous edition

NOTE Even if only the content of existing correction or amendment sheets is incorporated into the text of a normative document, the new text constitutes a new edition.

10 Implementation of normative documents

NOTE A normative document can be said to be “implemented” in two different ways. It may be applied in production, trade, etc., and it may be taken over, wholly or in part, in another normative document. Through the medium of this second document, it may then be applied, or it may again be taken over in yet another normative document.

118 10.1 taking over an international standard (in a national normative document) publication of a national normative document based on a relevant international standard , or endorsement of the international standard as having the same status as a national normative document, with any deviations from the international standard identified

NOTE The term “adoption” is sometimes used to cover the same concept as “taking over”, e.g. “adoption of an international standard in a national standard ”.

10.2 application of a normative document use of a normative document in production, trade, etc.

10.2.1 direct application of an international standard application of an international standard irrespective of the taking over of that international standard in any other normative document

10.2.2 indirect application of an international standard application of an international standard through the medium of another normative document in which it has been taken over

11 References to standards in regulations

11.1 reference to standards (in regulations) reference to one or more standards  in place of detailed provisions within a regulation

NOTE 1 A reference to standards is either dated, undated or general, and at the same time either exclusive or indicative.

NOTE 2 A reference to standards may be linked to a more general legal provision referring to the state of the art or acknowledged rules of technology . Such a provision may also stand alone.

11.2 Precision of reference

11.2.1 dated reference (to standards) reference to standards that identifies one or more specific standards in such a way that later revisions of the standard or standards are not to be applied unless the regulation is modified

NOTE The standard is usually identified by its number and either date or edition. The title may also be given.

11.2.2 undated reference (to standards) reference to standards that identifies one or more specific standards in such a way that later revisions of the standard or standards are to be applied without the need to modify the regulation NOTE The standard is usually identified only by its number. The title may also be given.

11.2.3 general reference (to standards) reference to standards that designates all standards of a specified body and/or in a particular field without identifying them individuall

11.3 Strength of reference

11.3.1 exclusive reference (to standards)

119 reference to standards that states that the only way to meet the relevant requirements of a technical regulation is to comply with the standard(s) referred to

11.3.2 indicative reference (to standards) reference to standards that states that one way to meet the relevant requirements of a technical regulation is to comply with the standard(s) referred to

NOTE An indicative reference to standards is a form of deemed-to-satisfy provision .

11.4 mandatory standard standard the application of which is made compulsory by virtue of a general law or exclusive reference in a regulation

12 Conformity assessment

The terms and definitions given in ISO/IEC 17000 apply.

120 Alphabetical index A acknowledged rule of technology 1.5 additional element 8.2 amendment 9.6 application of a normative document 10.2 authority 4.5 B basic standard 5.1 bilaterally harmonized standards 6.7 body 4.1, 8.1 C code of practice 3.5 comparable standards 6.9 compatibility 2.2 consensus 1.7 correction 9.5 D dated reference (to standards) 11.2.1 deemed-to-satisfy provision 7.6 descriptive provision 7.7 direct application of an international standard 10.2.1 domain of standardization (deprecated) 1.3 draft standard 9.2 E enforcement authority 4.5.2 environmental protection (deprecated) 2.7 equivalent standards 6.1 exclusive reference (to standards) 11.3.1 exclusive requirement 7.5.1 F field of standardization 1.3 fitness for purpose 2.1 G general reference (to standards) 11.2.3 H harmonized standards 6.1 I identical standards 6.3 indicative reference (to standards) 11.3.2 indirect application of an international standard 10.2.2 instruction 7.3 interchangeability 2.3 interface standard 5.7 international standard 3.2.1.1 international standardization 1.6.1 international standardizing organization 4.3.2 international standards organization 4.4.3 internationally harmonized standards 6.4 L level of standardization 1.6 M mandatory requirement (deprecated) 7.5.1 mandatory standard 11.4 multilaterally harmonized standards 6.6 N national standard 3.2.1.3 national standardization 1.6.3 national standards body 4.4.1 new edition 9.9 normative document 3.1 O optional requirement 7.5.2 organization 4.2 P performance provision 7.8 period of validity 9.3 prestandard 3.3 process standard 5.5 product protection 2.7 product standard 5.4 protection of the environment 2.6 provincial standard 3.2.1.4 provincial standardization 1.6.4 provision 7.1 R recommendation 7.4 reference to standards (in regulations) 11.1 regional standard 3.2.1.2 regional standardization 1.6.2 regional standardizing organization 4.3.1 regional standards organization 4.4.2 regionally harmonized standards 6.5 regulation 3.6 regulatory authority 4.5.1 reprint 9.8 requirement 7.5 review 9.4 revision 9.7 S safety 2.5 service standard 5.6 standard 3.2 standard on data to be provided 5.8 standardization 1.1 standardizing body 4.3 standards body 4.4 standards programme 9.1 standards project 9.1.1 state of the art 1.4 statement 7.2 subject of standardization 1.2 T taking over an international standard (in a national normative document) 10.1 technical regulation 3.6.1 technical specification 3.4 terminology standard 5.2 testing standard 5.3

122 U undated reference (to standards) 11.2.2 unified standards 6.2 unilaterally aligned standard 6.8 V variety control 2.4

123 Annex 2 - Extracts referring to standards with another meaning

1: In the US OMB document on “ STANDARD S AND GUIDELINES FOR STATISTICAL SURVEYS” (September 2006):

Standards are very similar to the recommendations of the ESS Code of practice. In the latter, the wording “principle” is used instead of “standard”. The US OMB document states that (see page 1):

Standards for Federal statistical programs serve both the interests of the public and the needs of the government. These standards document the professional principles and practices that Federal agencies are required to adhere to and the level of quality and effort expected in all statistical activities. Each standard has accompanying guidelines that present recommended best practices to fulfill the goals of the standards . Taken together, these standards and guidelines provide a means to ensure consistency among and within statistical activities conducted across the Federal Government.

2: In the ESS Code of practice:

2.1: Statistics comply with the European quality standard s and serve the needs of European institutions, governments, research institutions, business concerns and the public generally.

Here, “standard” means “a certain level of quality”, not to rules to measure or assess quality (which do exist), nor to rules that would define the rules that apply to the quality level for each statistic (such rules do not exist)…

2.2: Timeliness meets the highest European and international dissemination standard s. “Standard” means here “a certain level of quality”.

3- In the ESA 2010 regulation:

Article 338 of the Treaty on the Functioning of the European Union provides the legal basis for the European statistics. Acting in accordance with the ordinary legislative procedure, the European Parliament and the Council shall adopt measures for the production of statistics, where necessary, on the performance of the activities of the Union. It sets out the requirements relating to the production of European statistics, stating that they must conform to standard s of impartiality, reliability, objectivity, scientific independence, cost- effectiveness and statistical confidentiality.

Here, standards mean “levels of quality/ requirement”.

It is particularly surprising that the treaty states that “The production of Union statistics shall conform to impartiality, reliability, objectivity, scientific independence, cost- effectiveness and statistical confidentiality ”. There is no mention of standard!!!

Here, standard means level of quality.

4) Ecofin recommandations - Nov 2010

124

These conclusions state as well that “ Council calls upon […] to reinforce the binding nature of the Code of Practice by enshrining in a legal act some of the minimum standard s, including professional independence.

standards = level of quality

5) The regulation 223/2009 uses as well the term “standards”:

‘objectivity’, meaning that statistics must be developed, produced and disseminated in a systematic, reliable and unbiased manner; it implies the use of professional and ethical standard s, and that the policies and practices followed are transparent to users and survey respondents;

In this paragraph, “standard” means “principles”.

Specific quality requirements, such as target values and minimum standard s for the statistical production , may be laid down in sectoral legislation. .

“standards” mean here “level of quality”.

6) The “Communication of the Commission on the vision for the next decade” (404/2009):

The first component remains Community legislation, which will continue to be mainly output- oriented and to set minimum standard s for the production of statistics in a particular area.

“standards” mean here “level of quality”.

7) The Communication of the Commission on the robustness of statistics (211/2011):

Recent developments, in particular, the inaccuracy of the Greek government deficit and debt statistics, have however demonstrated that the system for fiscal statistics did not sufficiently mitigate the risk of substandard quality data being notified to Eurostat.

“standard” refers here again to a certain level of quality.

125

Annex 3 – Review of literature on standards

Introduction:

Objectives of the work

Aim: defining the concept of standard and, in particular, infer from the general definition the definition of standard corresponding to statistical methods Final objective: making a proposal to define precisely what is a statistical standard Here is the adopted method: o Taking stock of existing references in the literature for domains when standardisation is well implemented o Defining a common structure for the analysis of the chosen references.

Discussions took place on the structure and terms of the template. The template was aimed at collecting from the references all information pertaining to questions raised in Eurostat call for proposals.

Consensus was eventually reached on 5 main sections: General description of the document, Concept and main characteristics of standards, Process of standardisation, Possibility to standardise a design activity & Any other relevant content

This work was performed within the scope of WP 1 (but it is also consistent with the tasks foreseen in WP2)

Choice of reference documents

The search for references was organised as follows: o give priority to general reviews on theory or practice in standardisation o look for references recommended by famous standard organisations (nationally or internationally) o look for EU agreements when possible o look for references either in statistics or in related domains (services, etc…).

The result covers a large scope in many respects: o An international selection: documents from the EU, USA, Canada, Japan… o Large variety of documents: Sponsorship group’s report, leaflet, Code of practice, contribution at conferences, real ISO standard, guidelines… o Different types of intended audience: NSIs, businesses, institutional bodies, general public… o Variety of scope of the documents: opinion surveys, multimedia, industrial sector…

I/ Elements to explore for WP1

Definition of a standard Most documents do not offer an explicit definition for the word « standard ». However some definitions were found: o Technical standards may be described by answering six questions. (The six dimensions of standards) 126 o Documentary standards (…) are written agreements containing technical specifications or other precise criteria that may contain rules, guidelines, or definitions of characteristics. (The ABC of standards activities) o Standard = Practices that should be requirements for all telephone studies conducted by Government of Canada (The Advisory Panel on Telephone Public Opinion Survey Quality)

In addition, some documents refer to the ISO definition ( i.e. « a standard is a document, established by consensus, approved by a recognized body, that provides for common and repeated use, rules, guidelines or characteristics for activities or their results, aimed at the achievement of the optimum degree of order in a given context »)

Some documents also provide definitions for « guidelines »: o Guidelines = Practices that are recommended, but would not be requirements; that is, known good practices or criteria that serve as a checklist to ensure quality research but are not necessarily applied to every study (The Advisory Panel on Telephone Public Opinion Survey Quality) o Guidelines (…) present recommended best practices to fulfill the goals of the standards (Standards and Guidelines for Statistical Surveys)

Binding nature of a standard

Most documents dealt with the possible binding nature of the standard: o no clear description of the binding nature of a standard (it can either be «voluntary», «required», «not mandatory», « mandatory »…) o 2 documents retain attention because they provide specifications as to the binding nature of standards: o The ISO defines that a standard is not mandatory in general. It will be mandatory only if a national/regional law requires it. However there are binding requirements that are defined by the standard. Thus they are binding for companies that claim implementing the standard (ISO standard 2052:2006) o Distinction between « voluntary standards, which by themselves impose no obligations regarding use, and mandatory standards. Mandatory standards are set by government regulatory agencies at all levels » (The ABC’s of standards activities) Other characteristics of standards: - Intended application: it seems that there is no limitation of application for standards in terms of geography, domains or users. A standard can also be concerned with facilities and equipment, products, activities, customer perception or processes. - Degree of freedom that remains when the norm is fully implemented : here again, the analysed documents did not provide a clear view, and did not provide an easy answer (examples: “high degree of freedom”, “none”, “it depends on type of standard involved”). However, it is important to note that in the ISO scope, the standard has to be implemented in full in order to gain ISO accreditation (Independent, Transparent, Externally Audited: the ISO approach to Survey Process Quality Control). Moreover the ISO 20252:2006 reads that “there are strict requirements to secure a minimum level of quality except aspects due to quota method. (…) Higher requirements are possible. The quota method leads to much freedom in the assessment of uncertainty.” - Items to be included in a standard: as noticed before, the scope is very wide. A standard may include precise description of product, methods for reliability, testing and safety, the integration of existing sectoral standards, characteristics of a product, the methodology to assess conformity, or even the conformity assessment process itself. - Measures of implementation: once again, as the scope of standard is very large, the measures of implementation are numerous. One may retain that control can by secured by a public authority and/or a certification board. - Quality of standards criteria: The ABC’s of Standards Activities warns about the issues of poorly written standards which can raise costs, reduce product safety and quality and create barriers to trade. Independent, Transparent, Externally Audited: the ISO approach to Survey Process Quality Control is also of interest as it explains what a “good standard” is. - Conformity to standards: quality assessment is carried out regularly according to defined rules (CEN Workshop Agreement Certification rules for manufactured products in Internal 127 Regulations). Moreover, in the scope of a quality marking scheme, one may identify three assessment options: self-declaration, assessment by an interested party or assessment by an independent body (CEN’s Horizontal European Services Standardisation Strategy, CHESS feasibility Study).

II/ Elements to explore for WP2

Identification of needs, decision on the development of a special standard: One major aspect of standardisation is the origin of such a process. Some of the analysed documents allow for an identification of the needs behind the development of standards. It appears, from the CEN leaflet, that the initiative may be on the part of many different categories such as the industry, consumers, SMEs, boards of certification. In addition, the reasons for this initiative are numerous. One may put forward the need to support legislation (CEN leaflet), to assess quality (Advisory Panel on Telephone Public Opinion Survey Quality) or to maximise profit (International Standardization as a Strategic Tool). It seems obvious that all these reasons are particularly induced by the context of globalisation (Independent, Transparent, Externally Audited: the ISO approach to Survey Process Quality Control).

Identification of the influence of the expected use of a standard on the content: the influence appears to be very strong ( Advisory Panel on Telephone Public Opinion Survey Quality ) and the need for conformity assessment is very important in matters like health and security ( CEN leaflet, A Guide to EU Standard and Conformity Assessment ). It is also of a major importance that the production process be modified in order to become modular ( International Standardization as a Strategic Tool ).

Process of development of a standard: this appears as a major issue as many documents deal with it and offer different elements. First of all, it appears that there is no predefined time in the product cycle for introducing standards. Standards can be anticipatory, participatory or responsive (Six Dimensions of Standards ). Whatever the product cycle, the process of development must however include certain steps: proposal, acceptance (after identification of the purpose of the standard and of the requirements to be included - CHESS Feasibility Study), drafting, enquiry, adoption and publication; this process may take up to three years (CEN leaflet). There are certain elements of standard development which are also mentioned, these are consensus, transparency, balance, due process and openness ( ABC’s of Standards Activities ).

Other elements to be taken into account: - Adoption of standards: votes seem widespread, voters may be from working groups (Independent, Transparent, Externally Audited: the ISO approach to Survey Process Quality Control ) or part of national standard organisations (A Guide to EU Standards and Conformity). - Maintenance: little is said about this aspect, however ABC’s of Standards Activities warns of the possible problems that may arise from the lack of understanding of reviewers as to how the document was first prepared. - Conformity Assessment: ISO and Conformity Assessment and The ABC of Standards Activities provide interesting clues regarding this aspect and in particular its signification, the benefits it produces and how it can be performed. One may also want to refer to A Guide to EU Standards and Conformity Assessment which delves deeper into the concepts of risk assesment and self-certification.

128 - Possibility to standardise a design activity: there seems to be an agreement here, as in International Standardization as a Strategic Tool it is explained that design activity is not standardised but rather kept under control. Advisory Panel on Telephone Public Opinion Survey Quality reads that only very broad standards and guidelines are required for this activity.

Conclusion: This constituted the first work performed in this ESSnet project. The definition of a common structure for the analysis allowed partners to discuss definitions and concepts and to exchange their views on the expected output of the project. This common structure appears to be an efficient tool in order to gather as many relevant elements as possible on standards and the process of standardisation. In conclusion, the analysis of such a variety of reference documents has led the consortium to refine its objectives and to get a clearer picture of what was to be done within the scope of WP1 and WP2.

In general, the underlying concepts in official standardisation seem fairly harmonised on both sides of the Atlantic Ocean (and probably beyond). This can be easily understood: since companies expect their products or services to be accepted worldwide, the trade organisations have engaged in mutual recognition of standardisation, which has lead to similar practices at the regional/ national level. In the EU, the single market has fostered such a convergence.

The analyses are presented below and are organized according to two types of documents: reference documents which have a broad scope and those which are specific to the ESS (documents n° 13 & 14).

129 Bibliography

- “A Guide to EU Standards and Conformity Assessment” - NIST: http://gsi.nist.gov/global/docs/EU_Stds&CA_2000.pdf - “Draft ESS Quality Assurance Framework” - ESS Sponsorship Group on Quality (SQ): internal draft not to be distributed outside the SQ - “Independent, Transparent, Externally Audited: the ISO Approach to Survey Process Quality Control” - Bill Blyth: http://www.tnsglobal.com/_assets/files/Paper_Blyth.pdf - “Architecture-based approaches to international standardization and evolution of business models”, Shintaku, J. Ogawa, K. Yoshimoto, T. (2006), http://www.iecchallenge.org/papers/pdf_iecchallenge/shintaku.pdf “ - The Six Dimensions of Standards - Contribution towards a Theory of Standardization” - E. Baskin, K. Krechmer & M.H. Sherif: http://www.csrstds.com/theory.html - “ CEN’s Horizontal European Services Standardization Strategy, CHESS Feasibility study” - AENOR, BSI, DIN, DS, EVS, NEN, CMC & Cap Gemini: http://www.cen.eu/cen/Services/Business/Value/CHESSS/Documents/CHESSSBrusselsSem inarGeneralPresentationApril2008.pdf - “Annex C: CEN Workshop Agreement (CWA) Certification Rules for Manufactured Products in Internal Regulations” - CEN: ftp://ftp.cen.eu/cen/Services/Conformity/Certification/CENIRpart3.pdf - “ISO and Conformity Assessment” - ISO: www.iso.org/iso/casco_2005.pdf - “European Statistics Code of Practice” - SPC: http://www.dzs.hr/Eng/international/code_of_practice_en.pdf - “The ABC of Standards activities”, USA National Institute of Standards and Technology: http://gsi.nist.gov/global/docs/pubs/NISTIR_7614.pdf - “Presentation of CEN - General information” - CEN: ftp://ftp.cen.eu/CEN/AboutUs/Publications/Compass.pdf - “ISO standard 20252:2006 Market studies, social and opinion surveys” - ISO: http://www.iso.org/iso/fr/search.htm?qt=20252%3A2006&searchSubmit=Rechercher&sort= rel&type=simple&published=true - “The Advisory Panel on Telephone Public Opinion Survey Quality” - Public Works and Government Services Canada: http://www.tpsgc-pwgsc.gc.ca/rop-por/rapports- reports/documents/comitephone-panelphone-eng.pdf - “ Standards and Guidelines for Statistical Surveys” - USA Office of Management and Budget : http://www.whitehouse.gov/sites/default/files/omb/inforeg/statpolicy/standards_stat_surveys .pdf

130 N°1a Summary of reference document: A Guide to EU Standards and Conformity Assessment

Version prepared by GUS

1. General description of the A Guide to EU Standards and Conformity document Assessment, NIST (USA)-Special Publication 951 The aim of this guide is to present the general principles and concepts behind the European Union’s (EU) “New Approach” laws and directives in order to help business and government officials understand the new laws, the EU’s standardization process and the relationship between the European Commission and the European standardization bodies in the European Union. The document focuses mainly on following aspects of standards: standardization as technical harmonization, process of standardization and conformity assessments for obtaining CE mark to gain access to the European Market. 2. Concept, main characteristics of standards (WP1) a. Definition of Direct definition is not mentioned. Some key “standard” elements of definition are mentioned e.g. standardisation by consensus and committee (standards bodies), harmonization aspects b. Types of standards (Concept of categorisation, and categories) The aim is to identify classifications! Report if any from i.-viii. are mentioned, list categories, and add under ix. if any additional types are mentioned! i. How binding Presented standards are voluntary and in some cases they are? (i.e. mandatory with regard to essential health and safety voluntary, requirements in New Approach Directives. mandatory) ii. Form of Standards have documentary character and in some appearance cases also physical measurement character when 1.physical they include ergonomic requirements. measurement standards 2.documentary standards iii. Manner in Requirements are specified with regard to design which they (1.) and production process (2.). specify “Some manufacturers approach risk assessment by requirements using a design standard.” 1.design 2.performance iv. What it relates Presented standards are related with product, service, to (ie. process in business cycle. terminology, classification, product, service, process (if

131 statistical process, which GSBPM item), … complex) v. Link with Coverage of production cycle: design phase, analysis of production architecture of the overall production process vi. Intended user User groups of presented standards are: industry group in (manufacturers, exporters) and government officials. geographical Hence, we can classified them as company and hierarchy 1. national. Because of standards are used also in company international environment we can distinguish 2. national international user group e.g. CEN works with 3. regional Member state-Standard Organisations. 4. international vii. Intended Industry, government, services application domain (i.e. general, industry, statistics, business statistics, SBS) viii. Degree of The New Approach Directives are voluntary. They freedom that are evaluated on the basis of conformity assessment remains when to obtain product liability CE mark. the norm is fully implemented (How can we assess it?) ix. Any others to - be listed c. Items to be included in Items that should be included in a standard include a standard technical characteristics of products-essential requirements, quality levels, etc. “A standard can determine, by defining and delineating technical characteristics, what conditions will permit a product on the market”. “There is a vast body of European Standards (ENs) that is not mandated by the Commission; and these standards are not necessarily directed toward essential requirements. In theory, their use is voluntary, just as the use of harmonized standards is voluntary. These standards may or may not address health or safety aspects of products . They may define other characteristics, such as durability, appearance, quality levels, or even cultural preferences . They may be test

132 methods, or measurement guides .” d. Measures on There are procedures. implementation For example the procedure for EC type examination: - compiling a technical file - affixing the CE marking “Before affixing the CE mark, therefore, the manufacturer must make sure the product complies with the law by (1) determining which New Approach Directives govern the product, (2) applying the appropriate standards, and (3) carrying out the appropriate method of conformity assessment. The CE mark does not disclose which directive(s) or standards apply to the product, nor will it indicate the method of conformity assessment used to bring the product into compliance. This information is provided by other accompanying documents, such as the Declaration of Conformity.” -drawing up of a Declaration of Conformity -signature the Declaration of Conformity -keeping the Declaration of Conformity available for national surveillance authorities - certification “Most products covered by New Approach Directives can be self-certified by the manufacturer and do not require the intervention of a Notified Body. To self- certify, the manufacturer must assess the conformity of products to the applicable directives, and to the standards if applied.” “There are products, however, which may not be self- certified, but must be subjected to an EC type- examination. This examination involves the inspection of a representative example by a Notified Body within the EEA.” e. Specificity Specificity can be linked with technical construction files and certain aspects of conformity assessment. “The following main elements should be present in a Technical File: • a general description of the product; • design and production drawing and diagrams; • detailed technical data for essential aspects of the product; • a risk assessment; • a list of standards and or solutions applied; • reports of calculations and tests that have been carried out (calculations might include a

133 bearing, belt, or pulley based on a chart in a catalog); • certificates and inspection reports; • in the case of series production, internal conditions that have been observed to safeguard compliance with the directive; • a user's manual and a Declaration of Conformity.” “The Declaration of Conformity must contain: • the product (name, type or model number, lot, batch or serial number); • the name and address of the manufacturer or his Authorized Representative; • the name and identification number of the Notified Body if required by the directive; • standards or other normative documents applied to verify compliance with the directive(s). Declaration should also include reference to other directives if applicable; • the date of issue; and • the signature and title.” f. Quality of standards, Elements mentioned in the text that are connected criteria with quality criterias of standard: simplification (one law replaced many laws), government of family of products (the same rules for family of products), fulfilment of essential requirements, mutual recognition. These features have impact on following quality dimensions: a) relevance e.g. acceptance principle- mutual recognition, consensus, the same rules applied for family of products, b) accessibility and clarity e.g. application of harmonized standards, generic format, government families of products by New Approach Directives, integrated scheme for technical harmonization; c) comparability and coherence e.g. application of European, harmonized standards, the same rules applied for family of products

3. Process of standardisation (WP2) a. Identification “The philosophy of standardization by committee and of needs, consensus is the same in the EU as it is in the United decision on the States. Technical experts and others participate

134 development a voluntarily, and without compensation. The makeup special of committees may be organized differently and roles standard (Who may vary, but they generally follow a pattern that will decide?) includes input from producers, users, government, and academia. On both sides, committees are fairly autonomous; and both sides have processes for the creation of subcommittees, drafting of standards, disseminating draft documents for comment, voting, and appeals. On both sides, decisions are reached by consensus. And on both sides, standards organizations provide management, administrative, and logistical support for standards activities. They also provide for the editing, printing, publishing, sale, and distribution of standards documents. “ b. Identification The identification of the influence of the expected of the used of the standard on the content is connected with influence of the procedure of conformity assessment the expected use of the standard on the content c. Process of the It is possible to distinguish following stages in the development process of the development of a standard: of a standard a) the creation of subcommittees; (i.e. principles, b) drafting of standards; participants, c) disseminating draft documents for comment, financing) and voting and appeals; related d) decisions reached by consensus; organisational e) standard organization provides management, matters administrative and logistical support for standards activities; f) editing, printing, publishing, sale, distribution of standards documents.

In addition, process of standardization includes: -design review; -elimination risks; -compiling the technical file of which a list of standards or solutions applied; -affixing the CE marking; -user’s manual; -language; -declaration of conformity; -EC type-examination certificate by notified body within the EEA; -declaration of incorporation “The Declaration of Incorporation includes a statement that protects the subassembly manufacturer, and transfers the responsibility for safe incorporation to the builder of the larger system”.

135 -product liability; - marks; -principle of activity surveillance authorities. d. Adoption, Adoption of standards is based on self-certification repository of a by producer or is established by a notified body. standard Repository of a standard is maintained by a standard organization. e. Maintenance - of a standard f. Promotion and - support for application g. Conformity “Conformity assessment is defined by the assessment International Organization for Standardization/International Electrotechnical Commission Guide 2: 1996 as: "any activity concerned with determining directly or indirectly that relevant requirements are fulfilled." Typical examples of conformity assessment activities are sampling, testing and inspection, evaluation, verification and assurance of conformity (supplier's declaration), certification, registration, accreditation, and approval as well as their combinations. Conformity assessment may also be the process by which it is determined that a product's design meets a specification or standard. “

The paper presents conformity assessment procedures with regard to: a) the module decision – general rules e.g. a. “the essential objective of a conformity assessment procedure is to enable the public authorities to ensure that products placed on the market conform to the requirements as expressed in the provisions of the directives, in particular with regard to the health and safety of users and consumers;

b. conformity assessment can be subdivided into modules which relate to the design phase of products and to their production phase;

c. as a general rule a product should be subject to both phases before being able to be placed on the market if the

136 results are positive….”

b) design of products- conformity assessment procedures in European Legislation -design: - internal control of production: manufacturer keeps technical documentation at the disposal of national authorities, intervention of notified body; - type examination: manufacturer submits to notified body: technical documentation, type, notified body: ascertains conformity with essential requirements, carries out tests, if necessary, issues EC type- examination certificate, -unit verification: manufactures submits technical documentation, -full quality assurance- EN 29001 manufactures operates an approved quality system (QS) for design, notified body carries out surveillance of the QS, verifies conformity of the design, issues EC design examination certificate. c)production – conformity assessment procedures in European Legislation- production: - internal control of production- manufacturer declares conformity with essential requirements, affixes the CE mark, notified body tests on specific aspects of the product, product checks at random intervals, -conformity to type-manufactures declares conformity with approved type, affixes the CE mark; -production quality assurance- EN 29002, manufacturer operates an approved quality system (QS) for production and testing, declares conformity with approved type, affixes the CE mark, notified body approves the QS, carries out surveillance of the QS; - product quality assurance- EN 29003, manufacturer operates an approved quality system (QS) for inspection and testing, declares conformity with approved type or to essential requirements, affixes the CE mark, notified body approves the QS, carries out surveillance of the QS; - product verification – manufactures declares conformity with approved type or with essential requirements, affixes the CE mark, notified body verifies conformity, issues certificate of conformity; - unit verification- manufacturer submits product, declares conformity, affixes the CE mark, notified body verifies conformity with essential requirements, issues certificate of conformity; -full quality assurance- manufacturer operates an approved QS for production and testing, declares

137 conformity, affixes the CE mark, notified body carries out surveillance of the QS. 4. Possibility to standardise a “Some manufacturers approach risk assessment by design activity using a design standard.”

5. Any other relevant content Conformity assessment - the process by which compliance with essential requirements is determined. This process can be carried out with or without use of standards. Presumption of conformity – is a legal concept surrounding Harmonized Standards that denotes the relationship between the legislative and standardization processes. The EC type-examination – is a procedure by which the Notified Body ascertains and attests that a specimen representative of production meets the provisions of the directive that applies to it. A Declaration of Conformity - is a document informing Surveillance Authorities that the product meets the essential requirements of the applicable directives, or that the product has been issued an EC type-examination certificate, and meets the essential requirements of the applicable directives.

138

N° 1b Summary of reference document: A guide to EU Standards and Conformity assessment

Version prepared by Insee

1. General description of A Guide to EU Standards and Conformity the document Assessment NIST (USA) -Special Publication 951 Aim : This document is designed to help businesses and government officials understand the new laws, the EU's standardization process , and the relationship between the European Commission and the European standardization bodies in the European Union. It also provides information on the EU's approach to conformity assessment and requirements for obtaining CE mark to gain access to the European Market. 2. Concept, main characteristics of standards (WP1) a. Definition of The philosophy of standardization by committee and “standard” consensus is the same in the EU as it is in the United States . - (page 19) b. Types of standards (Concept of categorisation, and categories) The aim is to identify classifications! Report if any from i.-viii. are mentioned, list categories, and add under ix. if any additional types are mentioned! i. How binding For some standards, covered by the so-called “New they are? (i.e. approach” - defined by a few directives- and when voluntary, the products are linked to safety, health and mandatory) environment, there is a high level of requirements and binding conditions.. Standardization, when used specifically to translate laws into technical solutions for implementation, is tantamount to technical regulation . (page 19)-

ii. Form of Not mentioned appearance 1.physical measurement standards 2.documentary standards iii. Manner in Not mentioned which they specify requirement 1.design 2.performance

139 iv. What it relates Risk assessment, with a view to security and health to (ie. of users. terminology, classification, product, service, process (if statistical process, which GSBPM item), … complex) v. Link with Not mentioned analysis of architecture of the overall production process vi. Intended user This is a two-level organisation: at the EU level, group in CEN works with Member state - Standard geographical Organisations. Nationally, Standard organisations hierarchy work with industry representatives or businesses. 1. company 2. national 3. regional 4. international vii. Intended All market domains application domain (i.e. general, industry, statistics, business statistics, SBS) viii. Degree of Risk assessment under the responsibility of the freedom that producer (risk-assessment defined by a specific remains when standard) + legal liability for the producer the norm is fully implemented (How can we assess it?) ix. Any others to be listed c. Items to be included in a standard d. Measures on The specific standards required by the directive on implementation the New approach are controlled by the EU in order to secure that their implementation will be in line with the general market policy of the EU. e. Specificity

140 f. Quality of standards, criteria 3. Process of standardisation (WP2) a. Identification of Needs are identified by CEN - this is linked with needs, decision on the National Standardisation Organisations . In addition, development a special due to agreements with ISO/ IEC, some standards standard (Who will are recognised automatically. decide?) b. Identification of the Control by CEN for links with security and health + influence of the environment expected use of the Need for conformity assessment in matters like standard on the health / security content c. Process of the Same as in the US… (page 19). development of a standard (i.e. principles, participants, financing) and related organisational matters d. Adoption, repository vote of national standard organisations at CEN of a standard e. Maintenance of a standard f. Promotion and support for application

141 g. Conformity Three levels depending from the risk for health and assessment security: i. self-certification (Declaration of conformity) ii. self-certification + quality assurance certification iii. assessment by an independent institution. + definition of standards for risk assessment: EN1050 for machinery and EN1441 for Medical devices

self-certification : (page 24) The following main elements should be present in a Technical File: • a general description of the product; • design and production drawing and diagrams; • detailed technical data for essential aspects of the product; • a risk assessment; • a list of standards and or solutions applied; • reports of calculations and tests that have been carried out (calculations might include a bearing, belt, or pulley based on a chart in a catalog); • certificates and inspection reports; • in the case of series production, internal conditions that have been observed to safeguard compliance with the directive; • a user's manual; and • a Declaration of Conformity.

In case of an incident or random check, […] the manufacturer, his authorized representative or importer, must be able to provide the Technical File within seven to ten days as of the date of request by the surveillance authority. (page 30) 4. Possibility to standardise a design activity 5. Any other relevant product liability: The legal obligation to use good content engineering practices in safety matters is not embedded in directives, but exists independently of them.“ (page 36) declaration of incorporation: The general principle behind a Declaration of Incorporation is that the manufacturer of a subassembly cannot be required to foresee all the ways the subassembly may be incorporated, and, more importantly, may not be able to provide all the safety aspects of the larger machine . (page 30)

142

N°2

Summary of reference document: Independent, Transparent, Externally Audited: the ISO approach to Survey Process Quality Control

Prepared by the UK.

1. General description of the Independent, Transparent, Externally Audited: the ISO document (Who prepared, approach to Survey Process Quality Control. when, for what purpose) Bill Blyth, TNS plc, UK

http://www.tnsglobal.com/_assets/files/Paper_Blyth.pdf

The document is not concerned with the standard itself – rather the reasons behind the development of the standard and the process for achieving and assessing compliance with the standard (ISO 20252). 2. Concept, main ISO 20252 is a Process Quality Standard that is specific to characteristics of standards the conduct of Survey Research, both quantitative and (WP1) qualitative. It covers the process components of survey research. a. Definition of standard The complete standard is a lengthy document. The core content is sections on: • Quality management system requirements • Executive elements • Data collection • Data management and processing • Project documentation b. Types of standards ? (Concept of categorisation, and categories) iv. How binding Voluntary. However a similar UK standard has a constant they are? (i.e. 90%+ take-up. voluntary, mandatory) v. Form of Documented process standards (e.g. interviewer recruitment appearance and training, validation of interviews, data entry, coding, (physical etc) measurement standards, documentary standards) vi. Manner in All – very detailed which they specify requirements (design, performance)

143 vii. What it relates Statistical processes, covering all GSBPM stages from to (ie. initial user request to the provision of final deliverables – terminology, doesn’t include evaluation (I think – check documentation classification, when available) product, Examples given in paper are: service, • Responding to research requests process (if • Contents of research proposals statistical • Project schedule process, which • Assistance by and cooperation with clients/users GSBPM • Managing sampling and data processing item),…compl • Monitoring the execution of research ex) • Research reporting • Research records • Recruitment and training of interviewers (including content, duration and documentation of training) • Ongoing training and appraisal • Identification • Data protection and confidentiality • Validation • Data entry • Coding • Editing • File management • Analysis • Delivery • Back-up • Retention • Data security • Project documentation viii. Intended user International – specifically designed to be capable of group in implementation anywhere in the world where survey geographical research is undertaken. meaning (company, national, regional, international) ix. Intended Statistical surveys application domain (i.e. general, industry, statistics, business statistics, SBS) x. Degree of Standard has to be applied in full in order to gain ISO freedom that accreditation remains when the norm is fully 144 implemented (How can we assess it?) xi. Any others to be listed c. Items to be included in Not covered in document a standard d. Measures on “It is critical to the credibility and success of the standard implementation that it is implemented in a manner that results in consistent and comparable assessment and certification between countries, assessment bodies, applicant enterprises and locations.” “.. there is a need to invest greater thought and effort into the assessment procedure [than for physical specifications]” “One possible approach … is detailed expansion of the content of each section and a description how practically these procedures could be put into place and managed. Where they wished to, companies had available the services of a number of consultants with experience of both the industry and assessment procedures who could provide support to this process. In parallel detailed guidelines were drawn up to specify procedures for inspection agencies.” e. Conformity Proposal is to use independent inspection to ensure that the assessment concerns of consistency and comparability are met, using guidelines such as those set out above f. Quality of standards, ? criteria g. Link with the analysis Covers whole production process for statistical surveys – of architecture of the although limited coverage is given to self-completion and overall production internet interviewing process h. Influence of the use of Consistent and comparable survey research from countries the standard on the in all regions of the world. Provides a minimum guarantee content of quality. 3. Process of standardisation (WP2) a. Identification of needs, Response to international market and survey research decision on the community to the requirement of users of survey research development a special for the specification of minimum common standards for the standard (Who will process components of survey research. This demand was decide?) particularly encouraged by the globalisation of the activities of many of the major users of survey research. b. Process of the One of the standards’ organisations led the initiative, development of a providing both the necessary legal, drafting and associated standard (i.e. expertise together with the secretarial support. principles, participants, Invitations to participate. A working party of twenty financing) countries. Drafting process was lengthy, detailed and consultative. At the initial meeting a number of broad principles were discussed and identified. Built on existing standards. Consultation was iterative with voting taking place where there were substantive issues that required a

145 clear decision amongst the members. Process took three years and involved five face-to-face meetings. Financing (I think!) is borne by the members of the working group. c. Adoption, repository Final draft of the standard was voted on by members of the and maintenance of a working group, then by all ISO members. standard Stored in ISO online catalogue. Not explicit how maintained – think it is driven by users of survey data or the standard requesting an update. The guidelines are currently being revised to cover internet interviewing. d. Promotion and support Not covered in paper – driven by user demand for for application consistency and known quality? e. Conformity assessment See 2e above (?) f. Organisational matters See 3b above 4. Any other relevant content The agreement of the standard was the culmination of over fifteen years of work with a history that originated some thirty years ago! 5. Possibility to standardise a Not obvious from paper. Will investigate further when have design activity copy of ISO standard itself.

146

N°3 Summary of reference document: Architecture-based approaches to international standardization and evolution of business models

Prepared by INSEE

1. General description of the Architecture-based approaches to international document (Who prepared, standardization and evolution of business models when, for what purpose) Junjiro Shintaku, lead author; Koichi Ogawa, Tetsuo Yoshimoto, co-authors The university of Tokyo- Manufacturing Management Research Center, Japan

The document is about standards mainly in industry, involving modularization, leading growth in global economy itself . Standardization is seen from a technical and architectural point of view. The benefits of standardization are also analysed.

2. Concept, main Business model based on international standards characteristics of standards (WP1) a. Definition of standard Business model : the product itself, result of the process, is determined as a standard (ex. : DVD). The business model is cut into functions allocated to some components

Modular architecture (as opposed to closed architecture): • Open technology and open design rule • Weak interdependency among components • World wide installed base • Only one function is allocated to each component Closed local standards =closed architecture: • integral architecture • complex and strong interdependency among components • more than one function is allocated to each component • closed technology & closed design rules

b. Types of standards Examples of standards are found in technological (Concept of industry, as DVD player or different kind of CD’s categorisation, and production. In a full chain of production, newly categories) industrialized economies (NIEs) are specialized in some components and advanced countries cannot operate without : collaboration is needed.

147 i. How binding Voluntary : open to every firm in the world, not only they are? (i.e. to advanced countries but also to developing voluntary, countries. mandatory) But every participant is obliged to respect the standardized final product and the way to reach it. ii. Form of Documentary appearance Based on an alliance between firms. They allocate (physical modules according to the expertise of each partner. measurement standards, documentary standards) iii. Manner in Based on the division of labour. The industrial which they process is cut into sub-processes and then a modular specify architecture is built. However, it is not clear-cut if it requirements is a process standard or a product standard. (design, performance) iv. What it refers Industrial processes mainly about new technologies. to (ie. terminology, Related to a global economic growth model. classification, product, The standardization is about the final product : the service, process (if rules apply to the product itself through its business statistical process, process (a set of functions to be realized), related to which GSBPM technical components. item),…complex) v. Link with Strongly linked, architecture and business process analysis of architecture can’t be separated. of the overall process This requires a modular architecture, i.e. the association between one component and one single function of the final machine. Open international standards accelerated modularisation of products and promoted the international division of labour. vi. Intended user International group in geographical Company (in the case of local standards) meaning (company, national, regional, international) vii. Intended Industry application domain (i.e. general, industry, statistics, business statistics, SBS) viii. Degree of None. freedom that remains when the norm is fully implemented (How can we assess it?)

148 ix. Any others to categories: be listed • International - open • Local -closed c. Items to be Precise description of the product included in a standard d. Measures on Market growth implementation e. Specificity ? f. Quality of standards, criteria 3. Process of standardisation (WP2) a. Identification of International market and maximisation of profit - needs, decision on more precisely, how to create a bigger market for the the development a final product, while sharing commercial risks special standard without loosing the know-how in the most advanced (Who will technology. decide?) b. Identification of The expected use requires to avoid sharing highly the influence of advanced technology, whereas cheap realisation of the expected use less advanced technology (including assembly) is of the standard on crucial for the market development. The process has the content to be modified in order to become modular. Some modules are then standardised, not all. c. Process of the As said above, the main principle is labour’s division development of a and participation of several international firms. The standard (i.e. leaders are the western company. principles, participants, financing) and related organisational matters d. Adoption, The standard of the product has been decided in an repository of a international open forum (the DVD forum) by 200 standard firms from 20 countries. e. Maintenance of a ? standard f. Promotion and This kind of standard permits to create new business support for opportunities for NIEs. application g. Conformity assessment (?) 4. Possibility to standardise a In the paper, the examples are taken from optical design activity storage industry. The advanced design activity is kept under control, and not standardised. 5. Any other relevant content

149

N°4 Summary of reference document: The six dimensions of standards - Contribution towards a theory of standardization

Prepared by INSEE

1. General description of the Paper presented at the Seventh International Conference on document (Who prepared, Management of Technology, February 20, 1998 when, for what purpose) Authors: E. Baskin, K. Krechmer and M. H. Sherif

This document reviews the different approaches to multimedia communications to understand how this could affect standardization. It notes that the interests of standard seekers and standards creators are at opposite ends. To resolve that, it presents a framework for standards, to help organizations to plan their standard strategy. 2. Concept, main The overlap of technologies in multimedia communications characteristics of standards has caused many standards groups to address the issues (WP1) independently and with little coordination leading to application-specific approaches. In such a fragmented environment, a better understanding of the theory of standards becomes more important, to focus resources in a productive way. a. Definition of standard Technical standards may be described by answering six questions: The strategic questions are: • Why seek a standard? • What is the category of product or service to be standardized? • When in the product cycle to standardize? The tactical questions are: • Which is the appropriate Standards Development Organization (SDO)? • How will consensus be reached? • Where will the standard be used? b. Types of standards (Concept The document depicts a layered taxonomy of technical of categorisation, and standards into four strata: reference standards, similarity categories) standards, compatibility standards and etiquette standards. i. How binding they There are no regulatory obligations but a will, for the are? (i.e. enterprises, to win or dominate markets. To increase their voluntary, activity, organizations have to ensure compatibility. mandatory) Consensus based standards rely on the positive self-interest of the parties involved. ii. Form of Reference standards include unit standards which define appearance measurable physical qualities, e.g., ohm, volt, watt,.. or (physical other norms like ASCII, OSI,...

150 measurement Similarity standards define the nominal value and the allowed standards, variation for a set of similar entities. documentary Compatibility standards define the "transparency" of an standards) interface between two or more mating elements that communicate with each others. Modem recommendations (V.32, V.34, etc.) or the packet switching recommendation X.25 are examples of compatibility standards: they define interfaces, sufficient aspects of the transmitter and the receiver pairs necessary to ensure communication. Etiquette standards define the range and open ended variability of protocols. iii. Manner in which The applicability of a standard can be assessed in terms of they specify market, industry or geography requirements (design, performance) iv. What it refers to To ensure an access “anywhere any time” for many traffic (ie. terminology, types, the design and engineering activities of many types of classification, offerings are involved: digital, wireless, multimedia product, service, communications with new capabilities and services. process (if statistical process, which GSBPM item),…complex) v. Link with analysis of architecture of the overall process vi. Intended user International.. group in geographical meaning (company, national, regional, international) vii. Intended Multimedia communications application domain (i.e. general, industry, statistics, business statistics, SBS) viii. Degree of It depends of the type of standard involved freedom that remains when the norm is fully implemented (How can we assess it?) ix. Any others to be listed c. Items to be included in a In the field of multimedia communications, standards specify standard the methods for interaction, the measures of equivalence,

151 the limits of performance, quality, or degrees of compatibility. They may also include safety, reliability and test methods. d. Measures on implementation e. Specificity f. Quality of standards, criteria 3. Process of standardisation (WP2) a. Identification of needs, Precursor products or services provide sufficient proof of the decision on the development technology or market interest to stimulate work on standards. a special standard (Who will decide?) b. Identification of the influence of the expected use of the standard on the content c. Process of the development In terms of their position in the product cycle, there are of a standard (i.e. principles, different kinds of standards : anticipatory standards, participants, financing) participatory standards and responsive standards. Anticipatory standards are those standards that must be created before widespread acceptance of the device or services. They can emerge from the cooperation among existing competitors, or by new entrants that offer a precursor of the potential standard product or service. Participatory standards are developed, tested and used in an interactive environment. Responsive standards occur to codify a product or service that has been sold with some success. d. Adoption, repository of a standard e. Maintenance of a standard f. Promotion and support for application g. Conformity assessment (?) 4. Possibility to standardise a design activity 5. Any other relevant content

152

N°5 Summary of reference document

Prepared by KSH

1. General description of the document Title: CEN’s Horizontal European Services Standardization Strategy, CHESSS Feasibility Study 61 Author: Partnership of the National Standards Bodies of Spain (AENOR), Britain (BSI), Germany (DIN), Denmark (DS), Estonia (EVS) and the Netherlands (NEN), and participation from CEN Management Centre (CMC) and Cap Gemini Funded by: European Commission and European Free Trade Association (EFTA) 2008 Purpose: determine the feasibility of the development of a generic service standardization in Europe 2. Concept, main characteristics of standards (WP1) a. Definition of “standard” Concerning the intended standardisation the project team recommends in the feasibility study that the single generic European service standard be written as a requirement specification, because it enables service providers to be assessed against objectively verifiable requirements in the standard and demonstrate compliance b. Types of standards (Concept of categorisation, and categories) The aim is to identify classifications! Report if any from i.-viii. are mentioned, list categories, and add under ix. if any additional types are mentioned! i. How binding they are? (i.e. Possible interfaces of legislation and standards: voluntary, mandatory) • Including legal requirements directly in service standards • Including references to technical regulations (for example, by mentioning the identification code and the title of the technical regulation, with its date of approval and / or the official journal in which it was published) in the service standards • Standards are developed in order to provide solutions that enhance compliance with legislation • Public administrations, which are responsible for developing regulations, can refer to the existing voluntary standards in their technical

61 The template was filled in according to the general report and to annex G (Guidance Document for the Preparation of Service Standards) of module 1 (Guidance in the preparation of service standards) 153 regulations, if they consider that said documents can be beneficial to society ii. Form of appearance documentary standards 1.physical measurement standards 2.documentary standards iii. Manner Performance in services in which they specify require ments 1.desig n 2.performance iv. What it relates to (ie. all services in EU terminology, classification, product, service, process (if service standard can include requierements statistical process, which which refer to: GSBPM item), … complex) • Facilities and equipment (hotels, golf courses, etc.) • Products (fish retail trade, translation services – the text, postal services, etc.) • Activities (market research) • Customer perception (retail trade, customer call centres, etc.) • Processes (UNE draft standard on hotels – Tourism)

v. Link with analysis of it covers aspects of service provision that architecture of the overall are generic across all service sectors: production process • design • information provision to customers • billing • complaint & redress • innovation & review

vi. Intended user group in European geographical hierarchy 1. company 2. national 3. regional 4. international vii. Intended application domain services (i.e. general, industry, statistics, business statistics, SBS) viii. Degree

154 of freedo m that remain s when the norm is fully implem ented (How can we assess it?) ix. Any others to be listed Classification: • generic service standards (horizontal standards - a applicable across multiple or all service sectors) • service-specific standards (apply to a single service or a specific aspect within a single service)

c. Items to be According to the feasibility study the aim is included in a the integration of existing sectoral standards standard in a comprehensible, logical and transparent way to the generic service standardisation. In addition for each services, standard can include requirements which refer to: • Facilities and equipment (hotels, golf courses, etc.) • Products (fish retail trade, translation services – the text, postal services, etc.) • Activities (market research) • Customer perception (retail trade, customer call centres, etc.) • Processes (UNE draft standard on hotels – Tourism)

d. Measures on implementatio n e. Specificity f. Quality of If a quality mark is created according to the standards, generic standard, the certification and quality criteria marking scheme must allow for all three assessment options: ― 1st party (i.e. self-declaration) ― 2nd party (i.e. assessment by an interested party)

155 ― 3rd party (i.e. assessment by an independent body)

3. Process of standardisation (WP2) a. Identification of needs, decision on the The aim is that the standard promotes the development a special standard (Who will increase of trade of services in the EU decide?) (customers’ confidence, comparability, possibility of assessment). The project was funded by European Commission and European Free Trade Assossiation (EFTA) b. Identification of the influence of the expected In the case of services the assessment of the use of the standard on the content output is much more difficult than in the case of products, service standards are related to the processes of production of this services and circumstances of services. c. Process of the development of a standard (i.e. 1) identifyig in advance the purpose of the principles, participants, financing) and related standard and subsequently identifying the organisational matters requirements and recommendations to be considered in the document 2) identifying requierements to be included in the standard (according to the service lifecycle)

Service standard can be drafted following different methodologies; these are some of the possible ones: - According to internal processes, business areas or functional areas, e.g., some Spanish or Chilean standards for tourist services, or UNI 11067 Standard on Managing consulting, criteria for supply and check, or AS / NZS Standard on Laundry practice. - According to the service provision sequence, e.g., Restaurant services (UNE 167…), Translation services (EN 15038), Event stewarding (BS 8406). - According to customer needs and expectations;the standard sets requirements in accordance with the different generic customer expectations (reliability, response capability, competence, courtesy, etc.) e.g., UNE Standards on retail trade.

d. Adoption, repository of a standard e. Maintenance of a standard f. Promotion and support for application g. Conformity assessment 4. Possibility to standardise a design activity The service standard can be drafted following different methodologies; these are some of the possible ones: 156 - According to internal processes, business areas or functional areas, e.g., some Spanish or Chilean standards for tourist services, or UNI 11067 Standard on Managing consulting, criteria for supply and check, or AS / NZS Standard on Laundry practice. - According to the service provision sequence, e.g., Restaurant services (UNE 167…), Translation services (EN 15038), Event stewarding (BS 8406). - According to customer needs and expectations;the standard sets requirements in accordance with the different generic customer expectations (reliability, response capability, competence, courtesy, etc.) e.g., UNE Standards on retail trade.

5. Any other relevant content Scope of sector specific service standard: What does the standard describe? • The entire service provided by the service provider (e.g. an airline service) • One or some of the subservices ( e.g. on-board service on a plane) • All or some service categories (e.g. Business class, safety-related services) Who shall meet the standard requirements? • The service provider • Entities subcontracted by the provider • Who shall offer guarantees of compliance and is held liable in case of non-compliance Whom is it addressed to? • All or some customer segments (e.g. elderly, children, disabled)

Structure of the service standard: • Service is described following a service provision sequence, • Service requirements are established to directly meet customer‘s needs and expectations, • Service is described depending on the internal organization of the service provider, • Open model, • Other models different from those

157 mentioned above, • A mixture of two or more of the above.

There is a strong interface between service standards and management system standard.

158

N°6 Summary of reference document

Prepared by DESTATIS and KSH

1. General description of the Title: “ Annex C: CEN Workshop Agreement document (CWA) Certification rules for manufactured products in Internal Regulations PART 3 TERMS OF REFERENCE OF THE CEN CERTIFICATION BOARD”, prepared by CEN (2006) to describe the conformity assessment process. The CWA Certification System is a voluntary European certification system for demonstrating the compliance of products with the requirements of the relevant CWA(s) 2. Concept, main characteristics of standards (WP1) a. Definition of reference to ISO definition (ISO/IEC 17000:2004)? “standard” b. Types of standards (Concept of categorisation, and categories) The aim is to identify classifications! Report if any from i.-viii. are mentioned, list categories, and add under ix. if any additional types are mentioned! i. How voluntary binding they are? (i.e. voluntary, mandatory) ii. Form of documentary standard appearance 1.physical measureme nt standards 2.documentary standards iii. Manner in n.a. which they specify requiremen ts 1.design 2.performance iv. What it to certification of manufactured products on the relates to basis of CEN Workshop Agreements only (ie. terminolog y, classificati on, product,

159 service, process (if statistical process, which GSBPM item), … complex) v. Link with n.a. analysis of architectur e of the overall production process vi. Intended European & user group CEN Management Centre (CMC) ensures the in necessary liaisons with the other European and geographic international standardization bodies, and various al other organizations and reporting to CCB hierarchy 1. company 2. national 3. regional 4. international vii. Intended Industry. application domain (i.e. general, industry, statistics, business statistics, SBS) viii. Degree of n.a. freedom that remains when the norm is fully implement ed (How can we assess it?) ix. Any others n.a. to be listed c. Items to be n.a. included in a standard

160 d. Measures on CEN Certification Board is monitoring the implementation implementation in CEN of the CWA Certification System and overseeing its progress and effectiveness with the assistance of the CEN Management Centre (CMC) e. Specificity n.a. f. Quality of general competence criteria and conditions are standards, criteria specified for certification bodies and their own or sub-contracted testing laboratories and/or inspection/assessment bodies

3. Process of standardisation (WP2) a. Identification of needs, CEN Certification Board is enlisting the advice of decision on the development a CEN Workshops, where appropriate, and industry special standard (Who will associations decide?) b. Identification of the influence n.a. of the expected use of the standard on the content c. Process of the development of CEN Certification Board is responsible a standard (i.e. principles, participants, financing) and related organisational matters d. Adoption, repository of a common rules are applied to manage the standard administration of CWA certification e. Maintenance of a standard CEN Certification Board is responsible. f. Promotion and support for CEN Certification Board is responsible application g. Conformity assessment CEN Certification Board is responsible.

CCB is responsible for the overall administration of the CWA Certification System among others: - monitoring the implementation in CEN of the CWA Certification System and overseeing its progress and effectiveness with the assistance of the CEN Management Centre (CMC); - examining and deciding on proposals for new CWA Certification schemes and evaluating and deciding on the acceptance of existing certification schemes as being suitable for granting the CWA Certificate of conformity and in compliance with the CWA Certification rules;

In detail:

• CWA Certification System Rules, procedures and management for carrying out certification related to products on the basis of CWAs adopted by CEN.

161 • CWA Certification scheme Set of specific product-related requirements for the conformity assessment procedure to be applied to the CWA Certification System for carrying out certification of products which are in conformity with (a) specific CWA(s) and to which the same specified requirements, rules and procedures apply. • CWA Certification licence Document issued under the rules of the CWA Certification System, by which an empowered certification body grants to a supplier the right to use the CWA Certificate of conformity for its products in accordance with the rules of the relevant CWA Certification scheme. • CWA Certificate of conformity Document issued under the rules of the CWA Certification System, providing confidence that a duly identified product is in conformity with the relevant CWA(s).

4. Possibility to standardise a n.a. design activity 5. Any other relevant content The document deals also with the followings: • content of the certification scheme • administration rules • certification procedure (including surveillance procedure) • rules managing the modification both in the production process and in the conformity assessment scheme • correction actions • validity of certificates • complaints

162

N°7 Summary of reference document

Prepared by DESTATIS (Kai Lorentz) and KSH

1. General description of the Title: “ISO and conformity assessment”, prepared document by ISO (2005) to describe the general benefits (user’s point of view) from conformity assessment to standards, regulations or other specifications.

2. Concept, main characteristics of standards (WP1) a. Definition of “standard” no b. Types of standards (Concept of categorisation, and categories) The aim is to identify classifications! Report if any from i.-viii. are mentioned, list categories, and add under ix. if any additional types are mentioned! i. How voluntary binding they are? (i.e. voluntary, mandatory) ii. Form of documentary standard appearance 1.physical measureme nt standards 2.documentary standards iii. Manner in design which they specify requiremen ts 1.design 2.performance iv. What it Products, services, materials, processes, systems and relates to personnel measures – as scope of conformity (ie. assessment terminolog Process - as Conformity assessment as such is a y, standard on a process classificati on, product, service, process (if statistical process, which GSBPM item), … complex)

163 v. Link with n.a. analysis of architectur e of the overall production process vi. Intended all user group in geographic al hierarchy 1. company 2. national 3. regional 4. international vii. Intended General, but typically industry and services application Any type of conformity assessment, especially domain conformity to ISO/IEC standards (i.e. general, industry, statistics, business statistics, SBS) viii. Degree of Not general rule, form depending on the results of freedom risk assessment. The ISO standards described that originally are voluntary. remains when the norm is fully implement ed (How can we assess it?) ix. Any others n.a. to be listed c. Items to be included in a n.a. standard d. Measures on This is a general description on the objectives of implementation conformity assessment and its mechanisms. Conformity assessment items are covered in several ISO norms, which are referred in this document. e. Specificity n.a. f. Quality of standards, n.a. criteria 3. Process of standardisation (WP2) a. Identification of n.a.

164 needs, decision on the development a special standard (Who will decide?) b. Identification of n.a. the influence of the expected use of the standard on the content c. Process of the n.a. development of a standard (i.e. principles, participants, financing) and related organisational matters d. Adoption, n.a. repository of a standard e. Maintenance of a n.a. standard f. Promotion and n.a. support for application g. Conformity The document presents ISO conformity assessment, assessment concepts, goals, content, types and organisational matters. Conformity assessment means checking whether products, services, materials, processes, systems and personnel measure up to the requirements of standards, regulations or other specifications. Conformity assessment provides benefits for • manufacturers and service providers: allows them to distinguish themselves from less reputable suppliers • consumers: basis for selecting products and services supported by a formal supplier declaration, a mark or certificate of conformity, that attest to quality, safety and other desirable characteristics. • Regulators: gives them means to enforce legislation and achieve public policy goals. Transparent conformity assessment avoid costly multiple testing and/or certification of products. Conformity assessment may consist of any one of, some of, or all of the following: sample testing, inspection, process evaluation, supplier’s declaration of conformity, management system certification/registration, personnel certification, product certification, mutual recognition of results and the accreditation and peer assessment of the competence of the organizations conducting these activities – collectively known as “conformity assessment bodies” 4. Possibility to standardise a n.a. design activity

165 5. Any other relevant content This document can be considered as a standard on conformity assessment (in this case WP1 items are relevant) and as a standard-related activity, as conformity assessment of international standards (in this case WP2 is the relevant phase). On the web page other useful pages are available like “How conformity assessment works” – with first, second and third party assessment characteristics, the role of risk in the selection of type. “Mechanisms for performing conformity assessment” includes the references to the concrete ISO norms describing declaration, certification rules in different cases.

166

N°8 Summary of reference document: The ABC’s of standards activities

Prepared by Insee

1. General description of the The ABC’s of standards activities document National Institute of Standards and Technology (USA) NISTIR 7614

The report is intended to provide basic information to help educate U.S. government agency officials, legislative staff, industry, and other standards professionals to make informed decisions regarding standards development policies and the effective allocation of limited standards development resources. It is also hoped that this report will serve as a starting point for further study and discussion on some of these important issues. 2. Concept, main characteristics of standards (WP1) a. Definition of “standard” page 5: „Documentary standards [, which] are written agreements containing technical specifications or other precise criteria that may contain rules, guidelines, or definitions of characteristics. Standards ensure that materials, products, personnel qualifications, processes, and services are: • adequate for their purpose, • compatible and/or interchangeable, • if necessary; o ensure public health and safety; o protect the environment; o and/or improve economic performance.”

167 b. Types of standards (Concept of categorisation, and categories) The aim is to identify classifications! Report if any from i.-viii. are mentioned, list categories, and add under ix. if any additional types are mentioned! i. How Yet another classification scheme distinguishes binding between voluntary standards , which by themselves they are? impose no obligations regarding use, and mandatory (i.e. standards . Mandatory standards are set by voluntary, government regulatory agencies at all levels -- state, mandatory) local, and federal. They are usually included within the regulations of the government agency with applicable jurisdiction. Such regulations or mandatory ―standards, generally establish requirements for public health and safety, consumer protection, environmental protection, national security, or other similar criteria. However, the distinction between these two categories may be lost when voluntary consensus standards are referenced in government regulations, effectively making them ―mandatory standards 62 . (page 7) ii. Form of appearance 1.physical measureme nt standards 2.documentary standards iii. Manner in Those standards that describe how a product is which they supposed to function are called performance specify standards . In contrast, design standards define requiremen characteristics or how the product is to be built. ts 1.design It should also be noted that the determination of 2.performance conformance to performance standards may be more difficult than for design standards. [...] Performance standards are also more difficult to write. Performance standards are also more likely to allow the inclusion of technological innovations in the product and to prevent unnecessary barriers to trade . (page 7)

62 :: Note by Insee: example ( see the EU ”pet regulation” 438/2010 that requires the use of ISO standards 11784 and 11785 for the identification of EU pets). 168 iv. What it The NIST refers explicitly to the ISO/IEC Guide relates to 2:2004, which is analysed in another form. (ie. terminolog y, classificati on, product, service, process (if statistical process, which GSBPM item), … complex) v. Link with analysis of architectur e of the overall production process vi. Intended Standards may also be classified by the intended user group user group or by the standard‘s developer. There are in company standards, developed for use by a company geographic or organization for its own products or for the al products it purchases. There are also international hierarchy standards, most of which are developed and 1. company promulgated by international governmental and 2. national nongovernmental organizations, such as the 3. regional International Telecommunication Union (ITU) 4. international (governmental) and ISO (nongovernmental). (page 6)

The U.S. system is also tremendously diverse and the result is a system that is largely sectoral in its focus. This is a logical approach because each industrial sector, such as the information technology, telecommunications, automotive, medical devices, and building technology sectors, is most likely to understand that sector‘s needs and to know what standards best meet those needs. (page11).

169 vii. Intended application domain (i.e. general, industry, statistics, business statistics, SBS) viii. Degree of It should also be noted that the determination of freedom conformance to performance standards may be more that difficult than for design standards. [...] Performance remains standards are also more difficult to write. when the Performance standards are also more likely to allow norm is the inclusion of technological innovations in the fully product and to prevent unnecessary barriers to implement trade. (page7) ed (How can we assess it?) ix. Any others page 6: to be listed Government regulatory standards are those designed to be used by federal regulatory agencies in rulemaking and related activities. These should not be confused with Federal and Military Specifications , which are officialdocuments used by agencies and by the Department of Defense respectively, to support government procurement. Specifications are a set of conditions and requirements that provide a detailed description of a procedure, process, material, product, or service for use primarily in procurement and manufacturing c. Items to be included in a Standards can cover many aspects of the conformity standard assessment process. They can describe characteristics of the product for which conformity is sought; the methodology (e.g., test, inspection or other assessment methods) used to assess that conformity; or even the conformity assessment process itself (e.g., how a certification program or conformity assessment body should be operated). page 22 d. Measures on implementation e. Specificity

170 f. Quality of standards, „However, when standards are poorly written, they criteria can cause significant economic damage. Poorly written standards can raise transaction costs, reduce product safety and quality, and create barriers to trade. They can also constrain innovation; entrench inferior technologies; and impede the development of interoperable products and systems.” (page 2)

„The U.S. standards development system serves the national interest well. In most cases, it supports efficient and timely development of product and process standards that meet economic and public interests.” (page 10)

Problems can also occur when standards are not based on sound science. (page 24) 3. Process of standardisation (WP2) a. Identification of needs, The process is described for CEN (page 20): decision on the development a CEN is a private, not-for-profit organization, special standard (Who will composed of the national standards bodies in the EU decide?) and EFTA. CEN develops standards in all fields except the electrotechnical area. In addition to its market driven standards efforts, CEN also develops standards in support of EU and EFTA governmental policies and regulations (directives) in response to formal governmental mandates. Under the so-called New Approach , EU regulations (directives) are limited to establishing ―essential requirements. These essential requirements are obligatory and are formulated in general terms. The detailed technical specifications necessary for the implementation of directives are entrusted to European, voluntary standards organizations like CEN. b. Identification of the influence of the expected use of the standard on the content

171 c. Process of the However, certain principles for standards development of a standard development are widely accepted. (i.e. principles, participants, Consensus. Consensus means that all views are financing) and related heard and the resultant standard is generally agreed organisational matters to by those involved. Consensus is characterized by the absence of sustained opposition to substantive issues. However, it does not necessarily imply unanimity. Transparency . Transparency means: (a) providing advance public notice of a proposed standards development activity; (b) identifying the scope of work to be undertaken; (c) providing information on conditions for participation; (d) and providing an opportunity for all interested parties to comment prior to final approval and adoption. Balance. Balance means that no one interest, including the government, should dominate. It should be noted that balance can be affected not only by the number of participants in particular categories but also by the funding source. The provider of the funding in standards development work can sometimes end up dominating the process. This is particularly true if the funding is from a government entity. If funding is to be provided by a government agency or other entity, care must be taken to avoid undue influence on the outcome of the process by the funding provider. Due Process. Due process means that any person (organization, company, government agency, individual, etc.) with a direct and material interest has a right to: (a) express a position and the basis for it; (b) have that position considered; and (c) appeal if adversely affected. Due process ensures equity and fair play in the standards development process. Openness. The standards development process should be to open to participation by all materially affected interests. (page 8). d. Adoption, repository of a standard

172 e. Maintenance of a standard Unless the original technical experts that developed the standard participate in its revision, the reviewers may not be able to fully understand how the document was prepared, what was eliminated from consideration, and the reasons or assumptions underlying decisions and the resultant provisions. Problems can also occur in the application of specific provisions if the intent behind them is unclear. Rationale statements, which sometimes accompany a standard, are specifically designed to define the purpose and scope of the standard, to explain the criteria used in developing its requirements and to provide all other relevant information at the disposal of the developers. (page 24) f. Promotion and support for application

173 g. Conformity assessment „A product‘s conformance to accepted standards readily provides an efficient method of conveying complex information on the product‘s suitability.” (page 2)

Conformity assessment includes any activity concerned with determining directly or indirectly that requirements for products, services, systems, personnel qualifications and organizations are fulfilled. Conformity assessment includes: sampling and testing; inspection; supplier's declaration of conformity; certification; and management system assessment and registration. It also includes accreditation of laboratories, certifiers, inspection bodies, and management system registrars, and the recognition of the competence of accreditation bodies. Conformity assessment activities may be conducted by the manufacturer/supplier (first party), by the buyer (second party) either directly or by another party acting on the supplier's or buyer's behalf, or by a body not under the control or influence of either the buyer or the seller (third party). It can also be conducted by a government agency acting in a regulatory capacity. (page 22).

Knowing what aspects of the product will be evaluated in a conformity assessment process and whether there are other aspects which might impact quality, safety, or performance allows the user of the conformity assessment data to evaluate the data‘s significance. (page 23) h. standard developing Compared with umbrella-type standards organisation organizations that operate in other nations or at the international level, the more specialized U.S. SDO 63 s also tend be quicker to generate standards needed by industry. (page 11)

Among the private sector bodies, the International Organization for Standardization (ISO) is probably the largest producer of International Standards, having issud over 16,000 standards. (page 16) 4. Possibility to standardise a design activity 5. Any other relevant content

63 SDO= Standard developing organisations 174

N°9 Summary of reference document: CEN leaflet

Prepared by Insee

1. General description of the Presentation of CEN - general communication document version September 2010 Explains the way it is operating. 2. Concept, main characteristics of standards (WP1) a. Definition of “standard” No b. Types of standards (Concept of categorisation, and categories) The aim is to identify classifications! Report if any from i.-viii. are mentioned, list categories, and add under ix. if any additional types are mentioned! i. How binding they are? (i.e. voluntary, mandatory) ii. Form of appearance 1.physical measurement standards

2.documentary standards iii. Manner in which they specify requirements 1.design 2.performance iv. What it relates to (ie. terminology, classification, product, service, process (if statistical process, which GSBPM item), … complex) v. Link with analysis of architecture of the overall production process vi. Intended user The CEN is coordinated with the NSOs of Member group in geographical states, and has agreements with ISO. CEN, hierarchy 1. company CENELEC (electrotechnical standardisation) and 2. national ETSI (standardisation for information and 3. regional communication technology) have close links. 4. international vii. Intended application Covers 22 domains. One of them is “Services”: domain (i.e. general, industry, tourism, facility management, postal services, statistics, business statistics, SBS) cinematographic works, real estate agents, customer contact centres, supply chain security, (engineering) consultancy services, sheltered housing for the elderly, services of chiropractors, aesthetic surgery services, airport and aviation security services etc.

175 ix. Degree of freedom that remains when the norm is fully implemented (How can we assess it?) i. Any others to be listed c. Items to be included in a standard d. Measures on implementation e. Specificity f. Quality of standards, criteria 3. Process of standardisation (WP2) a. Identification of needs, Most standards are initiated by industry. Other decision on the development a standardization projects can come from consumers, special standard (Who will Small and Medium Enterprises or associations, to decide?) name some other sources. In addition, many standards are developed to support European legislation. (page 4) b. Identification of the ‘Reference to standards’ within a legislative text is influence of the expected use of viewed as a more effective means of ensuring that the standard on the content products meet the essential health and safety requirements of legislation than the writing of detailed laws. This allows both processes to support each other, without causing a slowdown. (page 4) c. Process of the page 7: development of a standard (i.e. 1- Proposal principles, participants, 2- Acceptance financing) and related 3- Drafting organisational matters 4- CEN Enquiry 5- Adoption by weighted vote 6- Publication of the European Standard (EN ) duration: 3 years d. Adoption, repository of a standard e. Maintenance of a standard f. Promotion and support for application g. Conformity assessment specific procedure with the keymark 4. Possibility to standardise a design activity 5. Any other relevant content page 7: Two intermediate products: CEN Workshop agreement: agreement between participants, but not the status of a standard - no obligation for NSOs to adopt them as national standards CEN Technical specification: pre-standard for innovative features or coexistence of various 176 alternatives.

177

N°10 Summary of reference document: Market studies, social and opinion surveys- Vocabulary and service requirements (ISO standard 20252:2006)

Prepared by Insee

1. General description of the document Since many customers ask for internationally or regionally comparable information, such a standard helps in guaranteeing the comparability of the information. This comparability will then be traceable. The documents contains the definition of 65 terms. 2. Concept, main characteristics of standards (WP1) a. Definition No (there is one ISO definition of s standard). of “standard” b. Types of standards (Concept of categorisation, and categories) The aim is to identify classifications! Report if any from i.-viii. are mentioned, list categories, and add under ix. if any additional types are mentioned! i.How binding they are? (i.e. voluntary, The standard is not mandatory in general, by mandatory) definition of an ISO standard. It will be mandatory only if a national/ regional law requires this. However, there are many binding requirements that are defined by the standard. Thus, they are binding for companies that claim implementing this standard. ii.Form of appearance documentary. 1.physical measurement standards 2.documentary standards iii.Manner in which they specify Mainly process standard, some aspects are of a requirements 1.design “product standard” kind: this is the case for the 2.performance response rate. Overall, ISO would qualify it as a service standard. iv.What it relates to (ie. terminology, Terminology, complete process from design to classification, product, service, process (if dissemination. It deals as well with statistical process, which GSBPM item), encompassing aspects like a management quality … complex) system, training of staff, data protection, quality reviews, continuous improvement, ... v.Link with analysis of architecture of the The norm covers quite typically surveys as they overall production process are conducted by companies for their customers. When the process is only partially covered by the company - but subcontracted under its responsibility - the company is responsible for the work done by the subcontractor. vi.Intended user group in geographical International hierarchy 1. company 2. national 3. regional 4. international

178 vii.Intended application domain (i.e. general, opinion surveys, market analysis, social industry, statistics, business statistics, surveys… SBS) viii.Degree of freedom that remains when the The standard requires much transparency with norm is fully implemented (How can we the client and much traceability. If the client does assess it?) not require much control, the degree of freedom for the surveying company is higher. There are strict requirements to secure a minimum level quality, except aspects due to the quota method. Higher requirements are possible… The quota method leads to much freedom in the assessment of uncertainty. ix.Any others to be listed c. Items to be included in a standard • quality control by the customer, transparency • reliability of results • access to the questionnaire and the guidelines for field staff, the translation of questionnaires • pilot surveys are highly recommended • training of interviewers • traceability of all actions d. Measures on implementation all the quality control and the requirements for traceability are closely linked to the implementation. e. Specificity f. Quality of standards, criteria 3. Process of standardisation (WP2) a. Identification of needs, decision on the development a special standard (Who will decide?) b. Identification of the influence of the expected use of the standard on the content c. Process of the development of a standard (i.e. principles, participants, financing) and related organisational matters d. Adoption, repository of a standard e. Maintenance of a standard f. Promotion and support for application g. Conformity assessment Possible for the data collection, coding, even imputation. Very few on the choice of estimators and quotas. 4. Possibility to standardise a design No clue in this standard. activity 5. Any other relevant content

179

N°11 Summary of reference document: The Advisory Panel On Telephone Public Opinion Survey Quality (Final report Feb. 2007)

Prepared by ISTAT

1. General description of the The Advisory Panel On Telephone Public Opinion document Survey Quality (Final report Feb. 2007)

The purpose is to provide advice and guidance to Public Opinion Research Directorate (PORD) on survey quality standards and benchmarks appropriate to public opinion telephone survey research conducted for the Government of Canada. For this purpose a Technical Advisory Panel was established, consisting of eight members belonging to: - Government of Canada - Market Research Industry - Academic Community. 2. Concept, main characteristics of standards (WP1) b. Definition of “standard” Standard = Practices that should be requirements for all telephone studies conducted by Government of Canada Guidelines = Practices that are recommended , but would not to be requirements; that is, known good practices or criteria that serve as a checklist to ensure quality research but are not necessarily applied to every study.

c. Types of standards (Concept of categorisation, and categories) The aim is to identify classifications! Report if any from i.-viii. are mentioned, list categories, and add under ix. if any additional types are mentioned! The report is organized in five sections, corresponding to the five main phases of a telephone public opinion survey: - Pre-field Planning, preparation and Documentation - Sampling and Data Collection - Response Rate - Data Management and Processing - Data Analysis/Reporting and Survey Documentation. For each of these sections, it defines Standards and/or Guidelines . i. How binding they are? (i.e. voluntary, mandatory)

180 ii. Form of Documented process standards. appearance 1.physical measurement standards 2.documentary standards iii. Manner in which VeryDetailed description for each element (activity, they specify method, etc.) pertaining to each section requirements 1.design 2.performance iv. What it relates to It relates to statistical process, to all the GSBPM (ie. terminology, items, even if not explicitly referred to. classification, product, service, process (if statistical process, which GSBPM item), … complex) v. Link with analysis There is a link through GSBPM of architecture of the overall production process vi. Intended user Company (commissioned by the Canadian group in Government) geographical hierarchy 1. company 2. national 3. regional 4. international vii. Intended Statistics application domain (i.e. general, industry, statistics, business statistics, SBS) viii. Degree of Limited degree of freedom freedom that remains when the norm is fully implemented (How can we assess it?) ix. Any others to be listed d. Items to be included in a standard

181 e. Measures on They are to be included in all calls for tender of the implementation Canadian administration f. Specificity g. Quality of standards, criteria 3. Process of standardisation (WP2) a. Identification of needs, decision on The need was to ensure a common approach to the development a special standard assessing data quality and reporting data quality (Who will decide?) indicators in survey reports commissioned by Government of Canada, as there were not standards or benchmarks concerning these aspects. b. Identification of the influence of the Page 10 of the paper: expected use of the standard on the content The objectives for the Panel were to: • Review and discuss current Canadian and international standards, benchmarks, and documentation practices related to telephone survey quality, with a particular focus on response rates and population coverage. • Review and discuss the Government of Canada context for undertaking public opinion research. • Advise on appropriate telephone survey quality standards for Government of Canada public opinion research . • Advise on specific benchmark levels of quality indicators for public opinion research telephone surveys conducted by private sector suppliers on behalf of the Government of Canada. These benchmarks will be considered for inclusion in the 2007 renewal of the contracting tools for public opinion research.

There is a very strong influence of the expected use on the content of the standard itself. c. Process of the development of a Already explained in box 1 standard (i.e. principles, participants, financing) and related organisational matters d. Adoption, repository of a standard page 8: “ The role of the Panel was also to reach consensus where possible, although this was not an essential outcome of the work of the Panel.” e. Maintenance of a standard f. Promotion and support for application g. Conformity assessment

182 4. Possibility There is for instance a very interesting summary of to standardise a design activity this issue on page 18:

There was agreement that for questionnaire design only very broad standards and guidelines are required, and several reasons were given for why this overall approach should be adopted. • It is unnecessary to develop standards/guidelines for questionnaire design because this is primarily the responsibility of the research firms. a) The industry has good “rules of thumb” to follow relating to every aspect of questionnaire design. b) Research firms already need to demonstrate their capabilities in the RFSO or an RFP by the requirement to provide both project credentials and references. • It is both unrealistic and undesirable to impose either standards or guidelines on an activity, i.e., questionnaire development, that: a) Is as much art as it is science b) Needs to be tailored to the specific information needs of each GOC survey and the different departmental requirements in certain areas (e.g., inclusion of references to specific pieces of legislation is required in some departments but not in others) • It is best left to the research industry to develop a set of standards and guidelines for questionnaire design, not the POR community in the GOC. Notably, though, the point was also made that the research industry itself does not have enough of a consensus on best practices upon which to base standards. • There is a need to avoid creating a process that becomes unduly burdensome for both GOC and suppliers in terms of the level of effort, time and financial resources that are devoted to developing and finalizing survey questionnaires. 5. Any other relevant content

183

N°12 Summary of reference document: Office of Management and Budget 64 - standards and guidelines for statistical surveys

Prepared by Insee

1. General description of the page 1: document In 2002, the U.S. Office of Management and Budget (OMB), in response to [...] the Information Quality Act , issued government-wide guidelines that “provide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies ” (67 FR 8452-8460; February 22, 2002). Federal statistical agencies worked together to draft a common framework to use in developing their individual Information Quality Guidelines . 2. Concept, main characteristics of standards (WP1) a. Definition of No, but they are exhaustively listed, accompanied by “standard” guidelines . page 1: Standards for Federal statistical programs serve both the interests of the public and the needs of the government. There is no indication of a consensus , this is required by the government.

Each standard has accompanying guidelines that present recommended best practices to fulfill the goals of the standards.

b. Types of standards (Concept of categorisation, and categories) The aim is to identify classifications! Report if any from i.-viii. are mentioned, list categories, and add under ix. if any additional types are mentioned! i. How binding they are? page 1: These standards document the professional (i.e. voluntary, principles and practices that Federal agencies are mandatory) required to adhere to and the level of quality and effort expected in all statistical activities.

page 4: Federal agencies are required to adhere to all standards for every statistical survey , even those that have already received OMB approval. Agencies should provide sufficient information in their Information Collection Requests (ICR) to OMB under the Paperwork Reduction Act (PRA) to demonstrate whether they are meeting the standards .

64 : USA 184 OMB recognizes that these standards cannot be applied uniformly or precisely in every situation. Consideration will be given to the importance of the uses of the information as well as the quality required to support those uses. If funding or other contingencies make it infeasible for all standards to be met , agencies should discuss in their ICR submissions the options that were considered and why the final design was selected. The agency should also include in the standard documentation for the survey, or in an easily accessible public venue , such as on its web site, the reasons why the standard could not be met and what actions the agency has taken or will take to address any resulting issues. ii.Form of appearance 1.physical measurement standards 2.documentary standards iii. Manner in which they specify requirements 1.design 2.performance iv.What it relates to (ie. page 2: classical steps of the statistical process: terminology, classification, product, • Development of concepts, methods, and service, process (if design statistical process, • Collection of data which GSBPM item), • Processing and editing of data … complex) • Production of estimates and projections • Data analysis • Review procedures • Dissemination of Information Products.

v.Link with analysis of architecture of the overall production process vi.Intended user group in applies to all US statistical agencies geographical hierarchy 1. company 2. national 3. regional 4. international vii.Intended application cross-domains domain (i.e. general, industry, statistics, business statistics, SBS) viii. Degree of freedom that budget allocation + choice of professional & 185 remains when the technical methods norm is fully implemented (How can we assess it?) ix.Any others to be listed c. Items to be included in a page2: standard The standards and guidelines are not intended to substitute for the extensive existing literature on statistical and survey theory, methods, and operations. When undertaking a survey, an agency should engage knowledgeable and experienced survey practitioners to effectively achieve the goals of the standards . Persons involved should have knowledge and experience in survey sampling theory, [...].

page 4: Additional information relevant to the standards can be found in other more specialized publications, and references to other Federal guidance documents or resources and the work of the Federal Committee on Statistical Methodology are provided in this document . d. Measures on implementation e. Specificity f. Quality of standards, criteria not relevant here. 3. Process of standardisation (WP2) a. Identification of needs, decision on the development a special standard (Who will decide?) b. Identification of the influence of the expected use of the standard on the content c. Process of the development of a standard (i.e. principles, participants, financing) and related organisational matters d. Adoption, repository of a standard e. Maintenance of a standard f. Promotion and support for application g. Conformity assessment 4. Possibility to standardise a Two standards deal with this issue (page i): design activity

186 Survey Planning - Standard 1.1: Agencies initiating a new survey or major revision of an existing survey must develop a written plan that sets forth a justification, including: goals and objectives; potential users; the decisions the survey is designed to inform; key survey estimates; the precision required of the estimates (e.g., the size of differences that need to be detected); the tabulations and analytic results that will inform decisions and other uses; related and previous surveys; steps taken to prevent unnecessary duplication with other sources of information; when and how frequently users need the data; and the level of detail needed in tabulations, confidential microdata, and public-use data files.

Survey Design - Standard 1.2: Agencies must develop a survey design, including defining the target population, designing the sampling plan, specifying the data collection instrument and methods, developing a realistic timetable and cost estimate, and selecting samples using generally accepted statistical methods (e.g., probabilistic methods that can provide estimates of sampling error). Any use of nonprobability sampling methods (e.g., cut-off or model-based samples) must be justified statistically and be able to measure estimation error. The size and design of the sample must reflect the level of detail needed in tabulations and other data products, and the precision required of key estimates. Documentation of each of these activities and resulting decisions must be maintained in the project files for use in documentation (see Standards 7.3 and 7.4).

Technically speaking, this is a process standard that lists all steps to be implemented. It refers as well to the documentation of this design, that has to be made partly public . 5. Any other relevant content These standards refer to very general principles, and have some similarity with the ESS Code of practice. However the latter is more general, whereas the former refers to some professional issues is several steps of the process. These US standards refer to the FCSM methodological documents. The ESS code of practice refers to international standards, which should all be used in the ESS, without any limitation. There is no reference to the ESS handbooks.

187

N°13 Summary of reference document

Prepared by DESTATIS and CBS NL

6. General description of the Title: “ ESS Quality Assurance Framework ”, document prepared by the Sponsorship on Quality (SQ) Final Version 1 from June 2011 (internal version, not yet distributed outside the SQ

The QAF is based on the proposal for a new version of the Code of Practise (CoP). Its aim is to assist the implementation of the CoP. While the CoP sets the principles and indicators as standards, the QAF describes, for each indicator, activities/tools/methods that facilitates the implementation of the CoP. The QAF has been developed to support the implementation of principles 4 and 7 to 15 of the CoP in its revised version. For each indicator of these principles the QAF contains:

A. Methods at institutional level B. Methods at product/survey level C. Reference Documentation

7. Concept, main characteristics of standards (WP1) a. Definition of “standard” b. Types of standards (Concept of categorisation, and categories) The aim is to identify classifications! Report if any from i.-viii. are mentioned, list categories, and add under ix. if any additional types are mentioned! i. How binding The binding character is not clear yet. It may not they are? (i.e. have a legally binding nature, but be used for peer voluntary, reviews and other forms of assessment about mandatory) compliance with the CoP. ii. Form of 2 appearance 1.physical measurement standards 2.documentary standards iii. Manner in Framework for Statistical Offices (performance?) which they specify requirements 1.design 2.performance iv. What it relates Methods/tools/activities at to (ie. a. Institutional level of

188 terminology, ESS members classification, b. Product/Survey product, level service, process (if statistical process, which GSBPM item), … complex) v. Link with ? analysis of architecture of the overall production process vi. Intended user Users are all members of the ESS, both Eurostat and group in the national statistical authorities geographical hierarchy 1. company 2. national 3. regional 4. international vii. Intended Mainly general, in some parts for specific domains application (business register, household register) domain (i.e. general, industry, statistics, business statistics, SBS) viii. Degree of High degree of freedom. Very often the QAF states, freedom that that the NSA should establish a procedure to …(e. g. remains when validate the questionnaire). The choice of the the norm is procedure or method is left open. fully implemented (How can we assess it?) ix. Any others to ? be listed c. Items to be included in ? a standard d. Measures on Possibly Peer Reviews or Self assessments? implementation e. Specificity ? f. Quality of standards, ? criteria 8. Process of standardisation (WP2) h. Identification The original idea was to find a common language for of needs, quality assurance inside the ESS.

189 decision on the Decision about the QAF and its application can be development a expected by the ESSC. special standard (Who will decide?) i. Identification of the influence of the expected use of the standard on the content j. Process of the development of a standard (i.e. principles, participants, financing) and related organisational matters k. Adoption, repository of a standard l. Maintenance of a standard m. Promotion and support for application n. Conformity assessment 9. Possibility to standardise a design activity 10. Any other relevant content

190

N°14 Summary of reference document: European Code of Practice

Prepared by CBS NL

1. General description of the The CoP is a set of principles related to different document aspects of quality and trust in statistics within the ESS. These aspects are the chapters of the CoP and are related to the highest institutional level till ‘lowest’ methodological aspects. 2. Concept, main characteristics of standards (WP1) a. Definition of “standard” No b. Types of standards (Concept of categorisation, and categories) The aim is to identify classifications! Report if any from i.-viii. are mentioned, list categories, and add under ix. if any additional types are mentioned! x. How The standard is not mandatory (yet). binding they are? (i.e. voluntar y, mandato ry) xi. Form of documentary. appeara nce 1.physic al measure ment standard s 2.documentary standards xii. Manner Design and organisation of mostly organisational en in which institutional aspects of production of statistics to they ensure trust and reliability. specify require ments 1.design 2.performance

191 xiii. What it relates to (ie. terminol ogy, classific ation, product, service, process (if statistica l process, which GSBPM item), … complex ) xiv. Link with analysis of architect ure of the overall producti on process xv. Intended International, national and company. user Focus of CoP is on the national level. A country is group in responsible for reliable statistics, enough budget etc. geograp Eurostat and NSI’s, the company level, are using the hical CoP as reference model for their production process. hierarch International the CoP is used to build trust in y 1. statistics. compan y 2. national 3. regional 4. international

192 xvi. Intended general applicati on domain (i.e. general, industry , statistics , business statistics , SBS) xvii. Degree Indicators are defined for each principle. The Quality of Assurance Framework is needed to assess the degree freedom of implementation. that remains when the norm is fully impleme nted (How can we assess it?) xviii. Any others to be listed c. Items to be included in a standard d. Measures on The QAF is meant to measure the implementation. implementation The QAF is under development. For the time being peer reviews and reports on CoP are measuring implementation. e. Specificity f. Quality of standards, criteria 3. Process of standardisation (WP2) a. Identification of needs, ESSC decides. Is owner of the standard. decision on the development a special standard (Who will decide?) b. Identification of the influence of the expected use of the standard on the content

193 c. Process of the development of Could be special task force or ESSnets. Now a a standard (i.e. principles, Sponsorship group is dealing with revision. participants, financing) and Participants are ESS members, normally from NSI’s. related organisational matters But also ECB, ESGAB and others are consulted. d. Adoption, repository of a Adoption by ESSC standard e. Maintenance of a standard Revision on initiative of ESSC f. Promotion and support for CoP is used as a reference document for all NSI’s. application g. Conformity assessment 4. Possibility to standardise a No clue in this standard. design activity 5. Any other relevant content

194