Identification of Problems of Performance Audit

Identification of Problems of Performance Audit For the Implementation of Principles of Democracy

Stasys Puskorius

Mykolas Romeris University Ateities str. 20, 08303 Vilnius

The determination of the problems of the performance audit of the public sector institutions is analyzed. The results are measured by widely used 3E criteria: economy, efficiency, and effectiveness. The content of performance audit as a particularly new branch of public administration is highlighted. The process of formulation of some interior and exterior audit problems is investigated. It suggested that the performance audit is a very important means that helps to develop democratic processes in society. A special research was carried out to determine preliminary audit problems to fight corruption cases in the customs of Lithuania.

Key words: performance audit, economy, efficiency, effectiveness, internal audit, external audit.

Introduction Performance audit is a new control form of audit, which covers different activities or systems of public administration institutions and in some cases the whole functioning of any government. Most countries in the world apply performance audit, but, as a rule, in different ways. In the wide sense the performance audit is a systematic analysis and estimation of economy, efficiency and effectiveness (3E criteria) of some public administration institutions aimed at making proposals how to increase the values of these criteria. The performance audit is a very complex process. For this reason special international audit organizations are established. They create common audit standards, study and summarize practical experience, seek to unify applied methodology, methods, models, and procedures. As it is declared in Performance audit standards “Performance auditing is concerned with the audit of economy, efficiency and effectiveness and embraces: (a) audit of the economy of administrative activities in accordance with sound administrative principles, and management policies; (b) audit of the efficiency of utilization of human, financial and other resources, including examination systems, performance measures and monitoring arrangements, and procedures followed by audited entities for remedying identified deficiencies; and (c) audit of the effectiveness of performance in relation to achievement of the objectiveness of the audited entity, and audit of the actual impact of activities compared with the intended impact”[1, p.6; 3, p.19]. 2

According to the International Organization of Supreme Audit Institutions (INTOSAI) performance auditing is “an audit of economy, efficiency and effectiveness with which the audited body uses its resources in carrying out its responsibilities” [2, p.17] The goals of performance audit are much more different comparing to financial auditing. Moreover financial auditing procedures are standardized quit well. To create such procedures for performance auditing is very difficult due to such reasons as follows: the kinds of public activities are quite different, the functions exercised differ in content, they are very complex, the terms to do some functions are quite long, and so forth. To solve these audit goals it is necessary to create special research methods, to widely use system analyses principles, to develop special methodology, appropriately use qualitative and quantitative data, and have a great concern about audit conclusions, recommendations and their reliability and accuracy. The necessity to check any entities work is quite difficult to forecast. It depends on some set of reasons. Most important of them are as follows: when some stagnancy activities have arisen, when low level of economy has been noticed in terms of efficiency or effectiveness, when forecasting and planning errors are observed, when management structure is too complex, when some functions are dubbed, when gaps of responsibility and accountability are expected, when quite large gaps from world standards are observed, when some law regulations don’t fit contemporary requirements, when some cases of resources used are not clear or transparent, especially in large amounts, when we see the violation of public and private interests. Above mentioned obstacles separate performance auditing as quite specific, new and thus it is very important to analyze these obstacles in general systematic way and determine the clear goals, content, procedures, stages, to choose appropriate methodology and research methods, different data collecting and computing technology.

Performance auditing criteria

In each auditing procedure there are used specific, the so called 3E performance quality criteria: economy – the minimization of resources used to reach some results while quality of this outcome is not less than established; efficiency – the ratio of desirable outcomes to complex resources used to reach these results; effectiveness – the level of goals reached in accordance of the resources applied [4; 5, p.746; 6, p.746]. These criteria are very complex, closely related with each other, but quite different, so they should be examined thoroughly to discover differences and similarities they have. Economy criterion means that any result may be compared according to the quantity of resources used, if the quality of the result is acceptable. Thus, it is necessary to take the cheapest way to reach some concrete result. Economy and Efficiency are closely linked, but the first term is narrower than Efficiency. The resources to be used may be very different and the amount of them may be quite large. Every of such resource may be estimated using the economy criterion. If we have such a set of criteria and recognize that not all of them are as good as we want, we need to choose some criteria of a different kind using those by which it is possible to estimate all economy criteria together. As a rule all these criteria of economy should be united and a new criterion constructed. Such a joint criterion is named the efficiency criterion. 3

The problem of creating an efficiency criterion is quite difficult. If the situation is simple and all resources used may be estimated by amount of expenditure, that criterion of efficiency is the sum of economy criteria. But such situation can rarely be met because each variant of performance depends not only on expenditure, but, for example, on time, technology used, qualification of employees, their motivation, environmental factors and so on. Most authors do not determine clear boundaries between economy, efficiency and effectiveness, while such boundaries really exist. These criteria are distributed in hierarchy order. Economy is the simplest of these criteria. Firstly, it depends on less different factors than efficiency or effectiveness. Secondly, all factors are linked only with expenditure aspects. Thirdly, this criterion may be estimated in a quantitative way quite easily. The criterion of efficiency includes factors, linked with economy, but, firstly, scrutinizes the wider and more distinguished sphere of factors, secondly, estimates the results of performance in more systematic way, thirdly, this criterion is “closer” to the established goals. Efficiency may be estimated in a quantitative way too, but it is a more complex task, than that of estimating economy. Effectiveness is the criterion of the highest level, because, firstly, it includes the full range of criteria of efficiency, secondly, it determines the impact of each efficiency criterion on results of performance, and, thirdly, mainly, estimates the degree of established goals reached. Whenever we try to estimate the efficiency and effectiveness it’s very important to choose the quantitative criteria, because it is the only way to determine the degree of established goals reached. Of course, to construct such criteria in some situations is impossible, particularly if we want to estimate the effectiveness, but efforts to construct quantitative criteria as a rule are useful even though these criteria do not reflect quite exactly the goals reached.

Formulation of audit problems

Performance auditors face a large number of different problems, thus they need to have some creative features to find the right decision in a quite unclear situation. They should know the structure of auditing organization, its mission, the functions to be done, vision, the particulars of work, and so forth. To formulate audit problems is both easy and difficult. Easy, because they arise from real life if some functioning results don’t satisfy the society, government, parliament or needs of some system. The words “don’t satisfy” should be explained. The results of performance may be quite good, satisfactory, and bad. If these results are good or satisfactory, it’s useful to determine the ways to increase these results. Obviously, if these results are bad, someone needs to find appropriate ways to change this situation immediately. Such means by auditors may be discovered at the end of auditing. Formulation of the internal audit problems Performance auditing may be internal and external. The procedures of finding and selection of internal and external audit problems are quite different. These differences become obvious, if we look at the process of identification of audit problems attentively. 4

In any case the auditing procedure starts from setting the problems to be verified. The activities to be made are as follows. When internal auditing problems are being stated an auditor: 1) names all or at least the largest part of them, 2) seeks to cover as many as possible performance fields of an organization or system; 3) invoke the main specialists and managers of entity, 4) range identified problems or set their priorities, 5) estimate internal auditors ability to do necessary job and may be necessity to invite some external experts, 6) determine the duration, money, and other resources we need. The process of determining internal audit problems is much simpler than in case of determining the external audit problems. It specifies such reasons: the study object fulfilling some functions is quite clear, the links among different parts of auditing system are clear enough, it is easy to determine available distribution of functions, accountability and responsibility of the auditing system, quite often the handicaps of functioning are known, the managers of auditing institution understand their organizations vision, mission, and role of strategic planning. The simplest way to determine the framework of internal or external problem is to stipulate insufficient proceeding of some organization branches. If preliminary audit problem is stated, it is necessary to revise it. The way to formulate an audit problem is quite complex and difficult. Consequently, some recommendations are useful. Firstly, when tentative audit problem is formulated it is advisable to spread it into some expansion constituent problems or tasks. Secondly, identify relationships among these constituents. Thirdly, discriminate functions fulfilled to reach established goals. Fourthly, separate the entities, departments and systems responsible for these functions. Fifthly, formulate the tentative hypothesis to explain insufficient results of functioning. Sixthly, generalize the tentative hypotheses and adjust the audit problem. It is advisable to create the so called “goals tree”. This tree covers the spirit of preliminary determined audit problems. In this case anyone sees the interpretation of partial goals and tasks. It is advisable to invite the managers, some specialists, and heads of different departments, who are interested to participate in this process. The “goals tree” should be continually improving until the audit problem would be highlighted. “ Goals tree” helps to determine links among partial goals or tasks. However it’s not enough. It is necessary to determine horizontal correlations among different goals and tasks, and connections among different management levels, who participate implementing performance activities. This study may be done by the same working group, which created the “goals tree”. It is necessary foresee the distribution of responsibility, functions, structure of any organization to be audited. The steps to be analyzed to determine the functions the auditing organization should fulfill are as follows:  To name all functions of some organization or system which influence economy, efficiency and effectiveness;  To discuss, if these functions are distributed properly among different departments and management levels;  To discriminate dubbing or redundant functions;  To stipulate, which additional functions should be exercised and who should do it;  To search the aspects of distribution of accountability and responsibility; 5

 If additional departments are suggested to establish, what functions should be attached and how their responsibility should be determined. Obviously the most complex stage of formulation of the audit problem is setting the hypotheses or reasons of uneconomic, inefficient or ineffective performance. It is recommended to propose as many as possible hypotheses linked with any shortcomings. When such set of hypotheses are stated it is recommended to make a chart of those. With a chart, it is easier to identify dubbing and closely related hypotheses, to distribute them according to some features and thus to formulate an audit problem. When audit problems are determined someone should look at large numbers of factors that restrict the abilities to execute internal audit tasks. Although it is very important to carry out a thorough examination of all functions, we need to determine our real possibilities to do it. Thus we should evaluate the number of audit problems, to compare this number with the number of auditors we have, to determine their abilities to do this job, the term it may be done, recourses we have and so forth. As a rule the possibilities of internal auditors are not quite sufficient to make estimations we would like to do. In this case some experts should be invited. The main functions of internal auditors are to make preliminary estimations, to prepare special audit methods, spread them among different departments, explain the ways to use them, pick up different data, to analyze them and make conclusions and recommendations. Formulation of the external audit problems The way of formulation of external audit problems is slightly different. This follows from many factors, which take place in this situation. The object of the audit may be very large and complex, it is very difficult to determine the problems of appropriate distribution of accountability and responsibility, which functions are to be assigned to some department or level of management, who and how should coordinate the common activities, and so forth. It is advisable to begin the analysis in such a way:  make a list of shortcomings of different ministries, departments or systems;  cover the most important areas of performance, which are not quite transparent, have low value output in economy, efficiency and effectiveness;  invite independent experts and foreign specialists;  state the priorities of each problem according of their actuality, importance, image of government, international obligations and expected effect;  Evaluate external, independent and foreign auditors possibilities to accomplish this audit;  Plan the duration of each audit. According to the mentioned ways the formulation of preliminary audit problems should include:  Identification of performance shortcomings the main ministries, departments or systems;  Making the list of named above preliminary audit problems;  Formulation each of these problems;  Setting the priorities of these problems;  Evaluation of resources, finance, duration and other limitations;  Evaluation of real capacities of supreme and other audit institutions;  Making the plan of external audit for second year and strategic plan. 6

It is not very difficult to identify preliminary audit problems, but to determine the reasons why they exist is quite a challenge. Nevertheless it is not very easy to identify preliminary external audit problems, thus it is useful to lean upon the practical experience any auditor or audit institution has. The appropriate steps may be stated as follows. Firstly, everybody needs to examine the main and most reliable sources of information, which include conclusions and recommendations about economy, efficiency and effectiveness of auditing institution. Of course, it is difficult to imagine the most reliable and necessary information would be found. If it were we would not need to plan and exercise any audit. As a rule the features of openness are characteristic of official government reports, statistical data, inquiry results of independent social research institutions, the results of different scientific research, scientific and practical conferences, the conclusions and opinions presented in media, and so forth. Secondly, the economy, efficiency and effectiveness of any institution may be estimated quite differently. These gaps arise due to different methods used, criteria chosen, data utilized and so forth. Thus it is very important rightly to compare the information, gained from different sources. The common recommendations rarely proved out, but it is better than without such recommendations. Few of these recommendations are as follows: more sources of information support any conclusion, better its reliability; if reliability of some sources of information strongly differs it is advisable to use only most reliable information; if data of different sources of information equally confirm and deny some conclusion, it is advisable to formulate two hypotheses and verify each of them. Thirdly, it is purposeful to study similar situations in different countries and to look attentively at the way they formulate any audit problem. Some of them may be used without changes, the others – after some corrections. When all preliminary audit problems are determined, it is necessary to look at them as a whole. In this situation it is advisable to make the list of these problems, to settle the correlations among them, to draw the chart and to discriminate the common ones, repeated ones, specific and so forth. This job needs to be done thoroughly because we can gain much useful information. Such adjusted list of preliminary audit problems should to be discussed further. Next it is necessary to foresee the ways to solve these audit problems. Of course, it is difficult to believe, that these ways would be the best ones, but premeditate opinions when the serious research is not yet made, have some advantages. Auditors in this case are not tied in traditional ways to solve these problems and thus suggested decisions may be more innovative and better. Next step is to determine the priorities of chosen audit problems. At the beginning they may be ranged in order according their importance. Then should be estimated the possibilities to perform the audit we need to solve this audit problem. When audit problems are separated and discriminated according to the mentioned factors, the list of really useful audit problems may be set up. The last stage includes determination the links and correlations among chosen problems and setting the sequence to check them. When the priorities of preliminary problems are set, it is advisable to choose the most important of them and count the money, time and other resources we need to make this audit. It may emerge that for some reasons this audit problem is impossible to resolve. In this case the problem should be delayed, until the limiting reasons would be eliminated. 7

When the described analysis of preliminary audit problems is accomplished and the possibilities of audit institution additionally evaluated, the audit plan is to be formed. This plan should be confirmed, declared and sent for each institution to be audited.

Structure of a Pre-Study Memo

After the audit problem is formulated the next step is to characterize it properly and clearly. Such description of audit problem as a rule is named a memorandum of audit (Pre-Study Memo) [3, p.163]. This memorandum includes two parts: general part and work part (plan). The general part includes: “…background to the pre-study, design of pre-study, description of the audit area, results from the pre-study, selection of audit problem”. Work plan cover: “…audit problem, audit design (audit scope, hypotheses, assessment criteria, methods of data collection), organization of the audit, time schedule, project budget, monitoring the progress of the audit, cooperation with the auditee, the possible results of the audit” [3, p. 163,164]. The general problems should be presented in terms of arguments for necessity to execute a pre-study. The design of the pre-study includes the rules to choose the respondents, creating the questions to be asked, what records to study etc. The description of the audit area should provide any interested person with sufficient background information to understand the pre-study goals. The results from the pre-study should be used as the basis for a list of audit problems. The audit problem should be limited in different ways (audit scope), namely in terms of: relevant audit object, geographical dimensions, time period to be covered. It is very important to formulate audit hypotheses. This will help to determine whether the problems can be audited or not. As a rule no more than ten hypotheses should be formulated. It is useful to describe the particularities of a preliminary audit: what, who and how was evaluated choosing the possible audit problems, what factors were estimated, what methods were used, what data or information sources were used, who participated in formulating audit problems, what principles were followed selecting the appropriate audit problems. The main attention is paid to explain what data and how they were gained, what was the content of questionnaire, who participated in the inquiry, how they were grouped, how the data were arranged, how the reliability of audit conclusions was vouchsafed, and so forth. The object of an audit is to be described in detail. It is important to name all directly or indirectly linked institutions, the goals of these institutions and links among different objectives, as well as integral goals of the whole system.

An example of the formulation of audit problems Identification of system problems in the Customs

Let us look at the customs institutions’ work in Lithuania. The results of social research show that 35 % citizens of Lithuania claim they have to pay illegal taxes passing customs procedures. 60 % of businessmen and 56% of population judge customs system in 8

Lithuania as very corrupted. Thus it is very important to discover the reasons why it happens, to eliminate them and suggest the means to prevent corruption cases. In 1999 -2003, the Special Research Office of Lithuania sued 20 criminal cases of customs officers. 12 of them are already sentenced to jail. The general data about suspected servants or officers in 2001 are given in Table 1.

Table 1 General data about detected officers or government servants of Lithuania in 2001 who are suspected to have committed a criminal act No Institution Quantity 1 Officers of the system of the Interior 32 2 Servants of the Social Insurance Fund 8 3 Officers of Penitentiary Department 5 4 Servants of municipally institutions 5 5 Officers of the national Defense Ministry 3 6 Servants of the State Tax Inspection 2 7 Other government officials 13 8 Catchpoles 1 9 Procurators 3 10 Customs officers 9

As we can see mostly corrupt officers are from the interior system, other government institutions and the customs. Suppose that the performance audits are already exercised in the system of the interior and in some other government institutions. A supreme audit institution needs to determine the expected audit problems in customs institutions. The way to do it is to perform a preliminary audit of these institutions. Preliminary audit is recommended to starte with framing some questionnaire, which may help to determine the main problems linked with corruption in customs system and reasons challenging corruption cases. These groups of questions should be reflected in the questionnaire: 1. The questions which seek to determine if the officers were faced with cases of corruption during duty time and in which circumstances it happened. 2. The questions, which help to determine the main luck of the customs activities challenging the assumptions to occur the corruption cases. 3. The questions which seek to determine the means to fight the corruption cases. The results of questionnaire (preliminary research) are tabled for approval in Tables 2 and 3.

Table 2 Results of questionnaire: “Have you faced corruption cases when exercising your duties?” [8, p.75, 76]. Nr. Question Officers of Officers of administration posts Yes No Yes No 1 Has someone directly proposed you illegal 6 23 10 28 reward at your workplace? 2 Has someone indirectly proposed you illegal 6 23 16 22 reward at your workplace? 3 Has someone directly proposed you illegal 2 27 1 37 9

reward after work? 4 Has someone indirectly proposed you illegal 2 27 3 35 reward after work? 5 Was the illegal award proposed in the 6 23 13 25 daytime? 6 Was the illegal award proposed in the 1 28 11 27 nighttime? 7 Did you receive some instructions not to 4 25 3 35 exercise some necessary functions for an illegal award? 8 Did you inform an appropriate institution or 0 29 2 33 person about attempt to graft you? 9 Did you take an illegal award? 1 28 3 32

Table 3 Main performance limitations of customs officers which may challenge corruption cases [8, p.79, 80] Nr Question Officers of Officers of posts . administration Yes No No Yes No No response response 1 Are the contents of procedures 7 7 15 11 8 19 accordingly used suitable or not 2 Are the qualifications of the 4 15 10 2 30 6 personnel good enough? 3 Is the cooperation with the 11 6 10 7 21 10 clients good enough? 4 Do the customs suffer from 19 4 6 27 4 6 deficit of money and management of resources is not effective? 5 Is coordination and cooperation 11 6 12 22 7 8 among subdivisions not effective?

The analysis of these data enables us to make some very important conclusions. For example, if we look at the second question in Table 2, we see that indirectly proposed illegal reward at an officer’s workplace differs very much: the percent of such illegal actions is 42 for post officers and 20.7 for administration officers. Thus it is possible to formulate such hypotheses: the post officers are much more corruptible than administration units. Thus the audit problem may be formulated as follows: are the customs post officers and administration ones differently corrupt on illegal reward at an officer’s workplace. If we look at the question 5 table 2, we see that the illegal award at the posts was proposed 65% more often then at the administration subdivisions. Thus it is possible to formulate such a hypothesis: the post officers are much more corruptible than administration units. As we can see the problem of audit is the same as formulated above. 10

If we look at question 7, Table 2, we can see that the officers of administrative subdivisions more often (13.8%) received some instructions not to exercise some necessary functions for an illegal award than the officers of the customs posts (7.9%). On the basis of this information two hypotheses may be formulated: 1. Instructions not exercise some obligatory functions are given in an increasing range which depends of the management level. 2. At the lower level there are unorganized graft taking cases when going up in the hierarchy the number of such cases increases. If we look at question 2, Table 3, we can see that officers of administrative subdivisions more often (14%) than post officers (5%) stated that they did not have appropriate qualifications. The audit problem may be as follows: qualification requirements of customs officers going up in the hierarchy are met inadequately. If we look at questions 3 and 5, Table 3, we may observe the quite large differences between opinions of officers at administration subdivisions and those at posts. Thus according to those differences audit problems may be formulated. Thus we can see the ways of formulating some preliminary audit problems which are very important means that help to develop democratic processes in the society.

References 1. Implementation Guidelines for Performance Auditing Standards. Draft. Standards and Guidelines for Performance Auditing based on INTOSAI’S Auditing Standards and Practical Experience. The Swedish National Audit Office, 2001. 2. INTOSAI Auditing Standards. Issued by Auditing Standards Committee International Organization of Supreme Audit Institutions, June 1992. 3. Handbook in Performance Auditing: Theory and Practice. Second Edition. Stockholm, Sweden, 1999. 4. Stasys Puškorius. Criteria for Estimation of the Quality of Management. National Economic HORIZONS, ESF MU BRNO, 2004 Nr.1, p.61 – 69. 5. International Encyclopedia of Public Policy and Administration. Jay M. Shafritz (Editor in Chief). Westview Press. A Division of Harper Collins Publishers, 1998. Vol.1-4. 6. Dierdre O’Neill. Efficiency // International Encyclopedia of Public Policy and Administration Jay M. Shafritz (Editor in Chief) Westview Press. A Division of Harper Collins Publishers, 1998, Vol.1. 7. Stasys Puškorius. Veiklos auditas: monografija.– Vilnius: Lietuvos teisės universiteto Leidybos centras, 2004.. 8. Vytautas Brendulis. Viešojo administravimo institucijų antikorupcinės veiklos įgyvendinama prevencija: modeliavimas Lietuvos muitinės pavyzdžiu. Magistro baigiamasis darbas. LTU, Vilnius, 2004.