NAS Audit Evidence Subgroup

Version 1 rev.3 – 25-26/02/2015

REFERENCE DOCUMENT FOR NAS NETWORK

WORKING GROUP - AUDIT EVIDENCE

MEETING 25 AND 26 FEBRUARY 2015

FVO, GRANGE

NAS Audit Evidence subgroup - working document Page 1 Version 1 rev.3 – 25-26/02/2015

The National Audit Systems (NAS) Network

The NAS network is a network of officials (auditors) from national competent authorities, responsible for the performance of audits of official control systems as provided for by article 4(6) of Regulation (EC) No 882/20041. The networks meet regularly, under the chairmanship of, and facilitated by, the FVO to exchange experiences in implementing national audit systems on official control activities. During the course of these exchanges; discussions, workshops etc. good principles and practices are identified and agreed by the network.

To enable dissemination of information the network, working in plenary session and through sub-groups, facilitated by the FVO, consolidate agreed principles and good practices on specific topics into reference documents. These reference documents may be used as guidance documents, however, they do not constitute an audit standard and are not legally binding.

Audit Evidence

OBJECTIVES

The objective of this document is to guide and support Competent Authorities (CA) and audit bodies in managing audit evidence. The aim is:  To provide principles and definitions regarding audit evidence  To identify characteristics, types, and sources of audit evidence  To discuss evidence collection planning  To give principles/discuss verification of audit evidence This document is intended to assist in the implementation of Section 6 of the Annex to Commission Decision 2006/677/EC.

SCOPE AND INTENDED AUDIENCE

This guidance applies to planning of audits as required by Article 4(6) of Regulation (EC) No 882/2004. It is intended for use by CAs / audit bodies that carry out audits on official control (systems) according to the requirements of Article 4(6) of Regulation (EC) No 882/2004. It supports the development of good practice in audit evidence collection and verification in the area of official control activities e.g. feed, food, animal health and welfare and plant health.

1 OJ L 191, 28.5.2004

I. BACKGROUND AND CONTEXT

{where does evidence fit in the audit cycle – evidence and evidence collection plan} {The objective of collecting evidences : measure, compare, evaluate to meet the objective, in order to support audit conclusions} {audit criteria and how it links with audit objectives}

The collection of audit evidence is a familiar but important step in the audit process. The quality of the evidence collected has a direct and significant effect on the audit findings and conclusions. The audit team should, at the audit planning stage of an audit, consider what audit evidence should be required. During the audit process, the audit team should verify the audit evidence collected and ensure it is appropriate and sufficient to achieve the audit objectives. Audit evidence needs to be compared to the audit criteria and the audit objectives to allow the audit team produce audit findings and present persuasive audit conclusions. Only audit evidence that is appropriate and sufficient will effectively support audit findings and conclusions which are capable of withstanding challenge and satisfy internal and external scrutiny. {Audit objectives: (IIA) 2210.A1- Internal auditors must conduct a preliminary assessment of the risks relevant to the activity under review. Engagement objectives must reflect the results of this assessment} {Audit criteria: means the set of policies, procedures or requirements used as a reference against which audit evidence is compared, i.e. the standard against which the auditee’s activities are assessed.} . (ISO) – (IIA) 2210.A3- Adequate criteria are needed to evaluate controls. Internal auditors must ascertain the extent to which management has established adequate criteria to determine whether objectives and goals have been accomplished. If adequate, internal auditors must use such criteria in their evaluation. If inadequate, internal auditors must work with management to develop appropriate evaluation criteria. For me the approach seems different, the criteria are linked for the IIA to the measure of the organisation’s goals, the criteria used for the appreciation of the evidences come from the key internal control points of the procedures (in the risk matrix of the audit team)}

{Different terminology may be used by different MS for the same processes} {reference to ISO to help with language versions} {Evaluation of Evidence: Is this within the scope of this document or does it belong to the next phase ? Possibly dealt with in a separate reference document on “Recommendations”}

II. DEFINITION(S)

This document should be read in conjunction with the definitions contained in Regulation (EC) No 882/2004 and Commission Decision 2006/677/EC bearing in mind that the definitions of those documents apply. Audit Evidence: records, statements of fact or other information which are relevant to the audit criteria and verifiable. (ISO 19011:2011 from ISO 9000:2005)

Effectiveness: is the extent to which official controls produce an (intended) effect / achieve an objective2. In this particular context the objectives are those of Regulation (EC) 882/2004. Effectiveness is not to be confused with efficiency, which is normally used when we want to refer to input-output ratio i.e. cost and/or resources required to produce an output. (“Auditing effectiveness of official control systems” NAS network document) Findings: results of the evaluation of the evidence collected during the audit against the applicable standard (e.g. legislation), described in an objective manner. (FVO SOP Audit Performance)

Conclusions: statements made by the audit team concerning the outcome of the audit which are based on and after consideration of all the findings and the audit objectives but which do not propose any course of action. (FVO SOP Audit Performance)

{Note: characteristics not mentioned in this section are better described in the next chapter, less prescriptive and explaining the concept}

III. AUDIT EVIDENCE

Include statement on usefulness of evidence, i.e. when it helps to reach goals of the audit "An effective audit has persuasive findings and conclusions. The quality of audit findings and conclusions relies on the judgements the auditor makes and these judgements are directly dependent on the quality of the audit evidence collected and the competence of the auditor collecting it” Audit Evidence: information used by the auditor in arriving at the conclusion on which the auditor's opinion is based…."(international Standard on Auditing (UK and Ireland) Audit Evidence: Audit evidence is the information internal auditors obtain through observing conditions, interviewing people, and examining records. Audit evidence should provide a factual

2 Objectives may be at a strategic or operational level.

NAS Audit Evidence subgroup - working document Page 4 Version 1 rev.3 – 25-26/02/2015 basis for audit opinions, conclusions, and recommendations. (IIA - SAWYER) / Audit evidence is the information that supports or refutes an audit objective (IIA – David O’Regan)”. The nature of audit evidence in systems audits {particularities of systems audits vs financial or compliance audits} For financial or compliance audits, evidence only needs to be collected to demonstrate activities are being carried out to planned arrangements. For systems audits, evidence needs to be collected to verify the effective implementation of planned arrangements

Quantitative versus qualitative

A. Characteristics of audit evidence {Some text?} Description The persuasiveness of evidence is linked to its appropriateness (relevant Persuasive and reliable) and sufficiency. (Linked also to target audience and findings) Appropriateness The appropriateness of the evidence is the measure of the quality of the / Usefulness evidence determined by its reliability and relevance. When there is enough evidence to persuade a reasonable person that the audit findings and conclusions are valid, and that the recommendations are appropriate. [IIA] Amount of evidence considered enough: [Scoping paper] Sufficient i) for the auditor to form a reasonable opinion (sample size, representativeness) ii) to convince interested parties/stakeholders of validity of auditors opinions (persuasive) When the evidence is clearly and logically related to the audit questions, audit criteria and audit findings. [IIA] Relevant Extent to which the information bears a clear and logical relationship to the audit findings (and audit objectives). [Scoping paper] Reliable When evidence is obtained through the use of appropriate techniques. When the same findings arise when alternative techniques are used or when information is obtained from different sources. The best obtainable information through the use of appropriate engagement techniques. [IIA] The degree to which evidence can be considered trustworthy (accurate and credible), the likelihood of coming up with the same answers if audit test is repeated or information is obtained from a different source or test. [Scoping paper]

Description Continuity and integrity of evidence. e.g. in a laboratory the reliability of results could be in question if sample identification, documentation and/or security is suspect. Verifiable MA Objective MA Representative Representative of the audit universe and … time… {UK’s example} Logical/ Linked to persuasiveness MA Rational/ Reasonable/ Sound Reference to Annex I – Audit Evidence Mind Map

B. Types Examples of Type Description Considerations evidence/Techniques Examples: Visual control

Photo? Whilst usually the most persuasive Sample evidence, the auditor must be aware Information that a risk exists that his/her Techniques: gathered by the presence may distort or prejudice auditor Direct inspection or what would normally occur, thus Observed through observation of reducing the quality of the (or Physical) personal people, property or evidence.* observation of events. people, events Ways to record this type of and physical.  Listening, smelling? evidence – photo, notes? Cross On-site verification reference with Section V. Verification of Evidence. Shadow inspection / Witness audit Review audit3 Documentary Information Exemples : This evidence may be in electronic prepared by Documents or hardcopy format. * others than the containing routines However, useful information may auditor. website information, not always be documented, thus Documentary etc. necessitating the use of other information approaches also.* can exist both Photos in paper and Internal/external Be sure to record the date on which electronic the information was gathered as the form.  3 Audis of the FBO without the presence of the inspector

Examples of Type Description Considerations evidence/Techniques Paper/electronic Legal/work ISO definition of “document”? Techniques: information may change later on. Review of documents, reports, manuals, literature, external and internal websites, postal or web-based surveys. Examples: Information gathered from Oral / written Oral evidence is generally people through interview important in performance audits, as information obtained in this manner interviews and Single / group focus groups. is up-to-date and may not be Oral / Such Techniques: available elsewhere.* Inquiry information Interviews However, information should be may take the Presentations corroborated and statements form of written confirmed if they are being used as or oral Questionnaires? evidence.* statements.  Knowledge/facts? Indirect or Examples: derived evidence / Comparison information Computation constructed by Such evidence is obtained by using Ratio the auditor professional judgement to evaluate combining Crossing physical, documentary and oral Analytical evidence. * information Techniques from different Be aware of importance of Audit sources and Analysis through experience and skills analysing that reasoning, information to reclassification, reach a computation and conclusion. comparison  Based on page 96 of “Internal Audit Practice”, chapter on “Gathering and analysing information” * based on page 60 of Court of Auditors’ “Performance Audit Manual”, Chapter 4 Examination Phase Note: Types are not related to description of ways to record evidence as this aspect may be covered by internal procedures.

C. Sources Source Type of Evidence Examples / Techniques Considerations Obtained Observed (and Direct inspection, The auditors can determine the directly Physical) methods that will provide the best by the On-site verification quality of evidence for the particular audit. However, their auditors Observation skills in designing and applying the methods will determine the quality Oral / Inquiry of the evidence. * Interviews,

Preparation of questionnaires

Analytical Previous audit reports from the audit bodies,

Analysis

Provided Documentary Information from Auditors must determine the reliability by the databases, documents, of data that is significant to the audit auditee activity statements and questions by review and corroboration, files (e.g. procedures, and by testing the auditee's internal instructions, legal acts, controls over information, including inspection reports, general and application controls over management reviews, computer-processed data. * organisational and planning documents, certifications).*

Answers to questionnaires

Oral replies during interviews

Oral / Inquiry

Provided Documentary Information which may The degree to which such by third have been verified by information can be used as audit parties others or whose quality evidence depends on the extent to is well known, e.g. which its quality can be established national statistical and its significance in relation to data.* the audit findings. *

Source Type of Evidence Examples / Techniques Considerations Information belonging to third parties (Business Operators, Customs, Stakeholder representatives, other CAs, etc.) Third parties audit reports Websites

Answers to questionnaires

Oral replies during interviews

Oral / Inquiry

* based on page 59 of Court of Auditors’ “Performance Audit Manual”, Chapter 4 Examination Phase

{Maybe we could add a point for the common deficiencies: failing to scrutinize important point, failing to maintain auditor independence, failing to supervise work}

IV. EVIDENCE COLLECTION PLANNING

Why do we need it? Main purpose is to allow a targeted evidence gathering to support the audit findings. This should focus on the audit objective and scope. Reference to Annex II – Diagram of Audit Process What is the benefit? To gather enough evidence and not more than needed. Plan the audit so that enough (sufficient) evidence can be obtained to be able to draw conclusions that have a bearing on the object of the audit. (RdH - link evidence sufficiency to the audit objectives) How do we do it? Which methodologies are used? Is there a “good practice” that we can identify? Iterative approach? Use of external experts (e.g. in data analysis)?

Knowledge of, and information available to, internal auditors vs. external auditors. Bias of auditors? Importance of on-site Note: important to link with characteristics of evidence (how to ensure we get useful information) Factors to consider when judging the quality and quantity of audit evidence: a higher standard is required for evidence supporting the purpose for which the evidence will audit findings than for background information provided be used in the audit report the level of the significance of the audit in general, the higher the level of significance, the higher finding the standard of evidence that is required the degree of independence of the source greater reliance can be placed on evidence which of the evidence emanates from independent sources the cost (money or time) of obtaining at some point, the cost of obtaining more evidence will additional evidence relative to likely outweigh the improved persuasiveness of the total body benefits in terms of supporting findings of evidence and conclusions

the greater the risk of legal action, controversy or the risk involved in making incorrect surprise from reporting an audit finding, the higher the findings or reaching invalid conclusions standard of evidence needed the care taken in collecting and analysing Including the extent of the auditors' skills in these areas the data

* based on page 58 of Court of Auditors’ “Performance Audit Manual”, Chapter 4 Examination Phase

Reference to Annex III – example of evidence matrix

When does it take place? Create a timeline with audit steps and where evidence collection planning takes place: Planning - Audit Objectives + scope - Phase 1 (desk study) – Risk analysis/desk study results – Phase 2 (test – on-site activities) – Audit report? Diagram? Planning – preparation – execution – reporting? Diagram? Evidence collection planning may take place at different stages, depending on the audit planning approach (Desk-based / on-site). Refined, adapted and developed along the audit process. On-site evidence collection is particularly important where the audit is being used to confirm/verify the effective implementation of planned arrangements. Retention of Audit Evidence.

This would have particular significance in respect of independent scrutiny, evaluation of or challenge to an audit system and /or its findings. Should be kept during a period described by the audit body or national rules. Link to Section III, table “A. Types”, type “Observed (and Physical)”, ways to record this type of evidence.

V. VERIFICATION OF AUDIT EVIDENCE

A. Verification: (ISO 9000 definition?) Is the evidence really “evidence” The information collected is not audit evidence until it has been verified. (meeting its characteristics – described above in Section III.A)? Root-cause-analysis – link with the evidence, runs along with the collection and verification of evidence.4Reference to Annex IV – to be developed. Who does it? Auditors (and their managers?). When to do it? Along with evidence gathering. Importance of on-site (do we need to emphasise here ; need additional text, refer to the document on effectiveness) How to verify? Cross-checking; / Review of auditor’s work (own review or supervision);) / Quality checks;/ Peer review.

[in BTSF CB-D3-P04]

4 Reference to “Root-cause analysis of non-compliance – outcome of the workshop (MANCP WG-meeting 21- 22/11/2012)”

B. Validation: (ISO 9000 definition?) Who validates? Auditors (and their managers?). Importance of competence and roles/responsibilities when validating evidence. When to validate? How to validate? Supervision of auditor’s work; Peer review

Annex I - Example of a mind map on Audit Evidence (to be adapted)

Annex II – Diagram of Audit Process

NAS Audit Evidence subgroup - working document Page 14 Annex III - Example of an evidence matrix5:

1 - From the risks cartography of the competent authority showing a high level of criticality on the subject of “species substitution”, the audit team planned a mission

2 – The audit team analyses the process of the CA to deal with that subject. The audit team elaborates a risk matrix of the process to identify the key points and the criteria showing they are under control. The audit team also reduces the scope of the mission to the horse meat, because it appears to be, one of the easiest products to substitute with a cheaper one, hard to detect and enables quick and strong profits.

3 – The process of audit can be summarized with the following matrix

Audit Steps and criteria Audit evidence Objective Type of evidence Level of evidence document observat testimonia Analytical enough Too Too ary ion l weak much Is the CA Plannin All the The Interview Data Yes efficient in g country is annual of the registered horse meat covered, all control meat in the species year long plan board informatio substitution manager n system controls? From the The Y-1 related to Yes production synthesis the meat to the control distribution plan chain

the orders Message Local Yes are and interview efficiently instruction of meat transmitted s given to agents to the agents the agent through the country

Executi The control Local Interviews Local Intervi on plan is planning of results vs ew of respected declining managem local agents (quantity, the ent and objectives is quality, data national agents in the useles recorded) instruction informatio s s n system

The agent Training Interviews Need knows how records of of agents to add to make a agents on an on- sampling the subject site observ Agent ation evaluation to s conclu de

5 This matrix is linked to a specific kind of audit and can be adapted to other cases.

NAS Audit Evidence subgroup - working document 15 Audit Steps and criteria Audit evidence Objective Type of evidence Level of evidence document observat testimonia Analytical enough Too Too ary ion l weak much The agent Quality of Yes knows the local product, the records law, the internal procedures,

Analyse The labs used a the right equipment

The analysts have the competencie s

The lab is referenced

Prosecut The level ion correspond to the level of the fraud

The rate of prosecution is homogeneo us on the territory

The rate of validation by the court is high

We can also add a column to the matrix to write the findings and another one for conclusions

NAS Audit Evidence subgroup - working document 16 Annex IV – Audit map and root-cause analysis

Alternative diagram

NAS Audit Evidence subgroup - working document 17