Evaluation Research in Occupational Health Services: General Principles and a Systematic Review of Empirical Studies

Home , Industrial (First), Industrial (Second)

Occup Environ Med 1999;56:361–377 361 Occup Environ Med: first published as 10.1136/oem.56.6.361 on 1 June 1999. Downloaded from METHODOLOGY Series editors: T C Aw, A Cockcroft, R McNamee

Evaluation research in occupational health services: general principles and a systematic review of empirical studies

Carel T J Hulshof, JosHAMVerbeek, FrankJHvanDijk, Willeke E van der Weide, Ingrid T J Braam

Abstract Keywords: review; occupational health services; evalua- Objectives—To study the nature and ex- tion of research tent of evaluation research in occupational health services (OHSs) Health services research Methods—Literature review of evaluation As 45% of the world’s population belong to the research in OHSs. On the basis of a workforce, occupational injuries and work conceptual model of OHS evaluation, related diseases have an important impact on empirical studies are categorised into health. Other diseases, although not primarily aspects of input, process, output, out- caused by work, may inﬂuence the working come, and OHS core activities. ability. Occupational health services (OHSs) —Many methods to evaluate OHSs are supposed to play an important part in pre- Results vention and control of occupational diseases or OHS activities exist, depending on the and injuries and in occupational rehabilitation. objective and object of evaluation. The In the World Health Organisation (WHO) glo- amount of empirical studies on evaluation bal strategy for “occupational health for all”, of OHSs or OHS activities that met the governments are asked to prepare actions for non-restrictive inclusion criteria, was re- providing competent OHSs for all people at markably limited. Most of the 52 studies work and for eVective implementation of were more descriptive than evaluative. OHSs.1 The terms competent and eVective The methodological quality of most stud- assume knowledge on the required quality of ies was not high. A diVerentiated picture

health care provided by these services. What do http://oem.bmj.com/ of the evidence of eVectiveness of OHSs we know about this? Scientiﬁc evaluation of arises. Occupational health consultations health care is part of health services research. and occupational rehabilitation are hardly Health services research in general seeks to studied despite much time spent on the analyse the functions and objectives of health consultation by occupational physicians in services, including the political, social, and most countries. The lack of eVectiveness economic forces shaping and conditioning the and eYciency of the pre-employment funding, organisation, management, priorities, examination should lead to its abandon- 2

eYciency, and eVectiveness of the services. on October 1, 2021 by guest. Protected copyright. ment as a means of selection of personnel Due to the demand for eVectiveness of care, for by OHSs. Periodic health monitoring or decision making in health programmes, for the Coronel Institute for surveillance, and education on occupa- development of standards and guidelines, and Occupational and tional health hazards can be carried out for the need for cost containment, health serv- Environmental Health, with reasonable process quality. Identifi- Academic Medical ices research has become much more promi- 3 Center, Division cation and evaluation of occupational nent in recent years. Epstein refers to “the Public Health, health hazards by a workplace survey can outcomes movement: the third revolution in University of be done with a high output quality, which, medical care”.4 This development seems not Amsterdam, The however, does not guarantee a favourable yet to be reflected in the field of occupational Netherlands outcome. health. There was a lack of published empirical 56 Correspondence to: Conclusions—Although rigorous study studies on the work in OHSs. In past years, Dr Carel TJ Hulshof, designs are not always applicable or feasi- similar developments in occupational health Coronel Institute for ble in daily practice, much more eVort care can be identified to those in general health Occupational and Environmental Health, should be directed at the scientific evalua- care: budget cuts, market competition, and Academic Medical Center, tion of OHSs and OHS instruments. To decrease or withdrawal of governmental grants. University of Amsterdam, develop evidence-based occupational Because of the specific setting of occupational PO Box 22700, 1100 DE Amsterdam, The health care the quality of evaluation stud- health care in social and economic life, there is Netherlands. Telephone ies should be improved. In particular, an increasing demand for justification of the 0031 20 5665333; fax 0031 process and outcome of consultation and eVectiveness and eYciency of OHSs from out- 20 6977161; email side the profession: employers, branches of [email protected] rehabilitation activities of occupational physicians need to be studied more. economic activity, governments, trade unions, Accepted 12 January 1999 (Occup Environ Med 1999;56:361–377) scientists, and insurance companies. Behrens et 362 Hulshof, Verbeek,van Dijk, et al

al refer to “the path breaking function of evalu- ployee assistance programmes on drugs, alco- Occup Environ Med: first published as 10.1136/oem.56.6.361 on 1 June 1999. Downloaded from ation research”.7 Concerns about quality of hol, or fitness—are not represented in this care, cost, and unnecessary medical care have review. In these areas, several comprehensive also emerged in occupational health care.8 All reviews have been published. Goldenhar and this can be seen as the need for external Schulte reviewed the intervention studies in the evaluation.9 There is also a need for internal field of occupational health and safety pub- evaluation. New developments in working life lished between 1988 and 1993, and concluded and the work environment, and demographic that in particular the number and method- changes in working populations call for new ological rigor of intervention studies has to be strategies and programmes. Changing legisla- increased to identify eVective intervention tion and professional and scientific interest in methods.14 To contribute to the development of the quality of occupational health care can also practice guidelines for occupational physicians, form a stimulus for studying aspects of quality van der Weide et al assessed the level of in OHSs.10 In 1982, a WHO working group evidence of the eYcacy of non-surgical inter- recommended that the evaluation of OHSs ventions for workers with low back pain. Voca- should be a regular activity, fully integrated tional status was a measure of outcome and into the planning and implementation of occu- they concluded that the scientific evidence for pational health and safety programmes.11 How- the eYcacy of interventions for patients with ever, despite a rich history of aetiological low back pain in decreasing rates of sickness research, the field of occupational health and absence or duration of sick leave is limited.15 In safety does not have a long history of research a review on economic implications of pro- on what works and what does not work to pre- grammes that promote health in the workplace, vent and control occupational diseases and Warner et al raised doubts on the evidence of injuries.12 Also in the field of occupational cost eVectiveness of many of these rehabilitation in cases of sickness absence or programmes.16 In another review on health and disability, a considerable lack in scientific cost eVective outcome of promotion of health knowledge on eVective and eYcient strategies at the workplace and disease prevention exists. The need has arisen for studies on the programmes, Pelletier was more optimistic: all eVectiveness of prevention strategies, pro- of the 24 studies included indicated positive grammes, and services. Skov and Kristensen health benefits or positive cost eVects.17 An distinguish between aetiological intervention update of this review in 1993 confirmed these studies seeking causes of diseases, and preven- findings and also reported an important tion eVectiveness studies evaluating the eVec- improvement of research design, data analysis, tiveness of methods for prevention,13 which is and complexity of interventions.18 For the field often inspired or conducted by principles and of occupational health and safety in general, methods in use in evaluation research. the findings of these reviews in both areas pro- Although the term evaluation research is vide important information. However, for commonly used, there is no single or clear cut evaluating the practice of occupational health definition of it. Depending on the context or care, the information is limited. At best, they

the scientific field in which the research is con- oVer an indication of the eYcacy of treatments http://oem.bmj.com/ ducted, various research activities can be or interventions in a well controlled and often categorised under this heading. From social more or less artificial situation. Black recently science publications, clinical or epidemiologi- called attention to the fact that most ran- cal research, and quality assurance, diVerent domised trials are explanatory—that is, they concepts and types of evaluation research can provide evidence of what can be achieved in the be derived. Notwithstanding this diVerence in most favourable circumstances.19 They often scientific origin and terminology, many analo- do not deal with eVectiveness in health care in

gies and overlap between these concepts exist. everyday practice. In this review, emphasis is on on October 1, 2021 by guest. Protected copyright. process and outcome of occupational health Objective care as it is provided in its typical everyday The purpose of this paper is to review the practice setting: the OHS. nature and extent of evaluation research in OHSs. We studied the scientific literature for Methods some general principles and methodological SELECTION OF THE PUBLICATIONS aspects of evaluation research in occupational For the publications on general principles and health care and we reviewed the empirical methodological aspects of evaluation research studies in this field. The main question of this in occupational health care, we used a few review is almost a rhetorical one: how well are essential handbooks and monographs and col- we doing? What is known of input, process, and lected additional scientific literature by check- outcome of occupational health care as it is ing citations in relevant publications and by a provided by OHSs? computerised search in Medline. The available In this paper, we focus on evaluation of the publications on empirical studies in OHSs activities of OHSs. This is excluding a consid- were selected in an automatic search of the erable amount of prevention eVectiveness computerised databases Medline, OSH-ROM, research in the field of occupational health. CIS-DOC, HSE-line, Embase, and Current Research on non-OHS related interventions, Contents. Also, the references in relevant arti- programmes, and policies to reduce workplace cles and in background literature were further health hazards and public health oriented examined. For computer searches we used the research on health promotion at the work following keywords: eVectiveness, evaluation site—for example, hypertension control, em- study, health services research, occupational Evaluation research in occupational health services 363

health services, outcomes research, outcome were more descriptive in nature, the results are Occup Environ Med: first published as 10.1136/oem.56.6.361 on 1 June 1999. Downloaded from assessment, outcome evaluation, outcome and reported as indefinite. process assessment, process evaluation, programme evaluation, pre-employment examina- Results tion, periodic occupational health examination, EVALUATION RESEARCH IN HEALTH CARE; occupational rehabilitation, medical consulta- GENERAL PRINCIPLES tion, audit, quality, and practice guidelines. Terminology The empirical studies had to meet the The history of evaluation research is linked to following inclusion criteria: the growth and standing of the social sciences, (1) The study had to deal specifically with in particular to the evaluation of educational evaluation of OHSs or OHS instruments. The programmes, and to the assessment of public OHS instruments (defined as circumscript and health initiatives to reduce morbidity and mor- formalised working methods and measure- tality from infectious diseases.12 Evaluation of ment protocols, inclusive equipment, and health care is defined as “the assessment of strategies10) were restricted to workplace inves- eVectiveness, eYciency, acceptability, and ac- tigations and evaluation on work related ceptance of a care system or programme in hazards; management consultation; infor- achieving the stated objectives”.11 22 According mation and education of employees on work to this definition, evaluation research is closely related hazards; pre-employment examination; related to intervention research: “the study of periodic occupational health examination or planned and applied activities designed to pro- surveillance; consulting hours; occupational duce designated outcomes”14 and to outcomes rehabilitation, and first aid organisation. research: “study of health care received by (2) The paper had to present original study typical patients with a particular condition to a results; reviews were excluded. range of positive and negative outcomes to (3) The study was published in English in an identify what works best and for whom”.8 international (peer reviewed) journal. EVectiveness of health care is a measure of (4) The work was published between 1985 technical outcome, in terms of health. EY- and 1996. ciency is an economic concept referring to the In particular, the third criterion excluded costs of the care system or programme relative many evaluation studies. Most evaluation to its eVectiveness. Acceptability refers to research actually carried out in OHSs goes whether the care is professionally and socially unpublished or is published in reports in the satisfactory and adequate. Acceptance can be “grey literature”, often exclusively directed at defined as the psychosocial process (individu- financial suppliers, programme funders, or ally or collectively) of accepting health care.22 decision makers. We insisted on this because The study objects of evaluation research in we think that dissemination of research find- health care can be classified in diVerent ways. ings in the scientific and professional field is an According to Donabedian most of these classi- essential prerequisite.20 fications distinguish input (or structure), process, and outcome aspects of health care.23

Input or structure aspects can be divided into http://oem.bmj.com/ QUALITY ASSESSMENT “system characteristics” (administrative, or- Quality assessments in a review can be used as ganisational, physical, and ﬁnancial facilities), a threshold for inclusion, as a possible explana- “provider characteristics” (knowledge, spe- tion for diVerences in results between studies, cialty training, beliefs, and attitudes), and in sensitivity analyses, and as weights in statis- “patient or client characteristics” (age, sex, tical analysis or meta-analysis of the results.21 health habits, preferences, expectations). Pro- In systematic reviews on the eYcacyofa cess refers to the content of the provided care; speciﬁc intervention, often there is an exclusion

technical aspects (activities, continuity of care, on October 1, 2021 by guest. Protected copyright. of studies with a lower methodological quality etc) and treatment aspects like interpersonal or studies are rated to see if they meet some manner and communication style. Outcome minimum (particularly methodological) qual- deals with the eVects of the care on the health ity criteria. In this review, we chose not to use a of patients or populations. In its most basic quality assessment procedure for inclusion or form, the outcome of health care can be classi- weighting of studies. Because of the broad ﬁed under the “ﬁve ds”: death, disease, disabil- focus of this review (the nature and extent of ity, discomfort, and dissatisfaction.22 Recent evaluation research in OHSs) and conse- developments in health services research show quently, the heterogeneity of the studies and the use of other and broader outcome variables study objects, it is very diYcult to adopt a such as functional, general wellbeing, satisfac- quality rating system applicable to the different tion with care, quality of (working) life, know- types of studies in OHSs. ledge, skills, and behaviour outcomes.

PRESENTATION OF PUBLICATIONS Outcome evaluation: aspects of study design In the first part of this paper, we highlight some Outcome evaluation requires an explicit re- general principles and methodological aspects search design. The most important elements and present a conceptual model for evaluation in selecting a design are: the use of comparison of OHSs. This model is used in the second part or control groups (truly experimental, quasi- of the paper to present the empirical studies. experimental, or non-experimental) and the We used the conclusions of the authors to timing of measures: pretest-post-test, post-test report positive or negative findings. If authors only, or time series.24 The most rigorous did not formulate a concrete finding or studies evaluation design is the true experimental 364 Hulshof, Verbeek,van Dijk, et al

25 26 pretest-post-test control group design. In involved, assess the outcome of the Occup Environ Med: first published as 10.1136/oem.56.6.361 on 1 June 1999. Downloaded from clinical research, this design is better known as programme.28 a randomised controlled trial (table 1). It is regarded as the “gold standard” in clinical Process evaluation outcome research, especially because of its Process evaluation is the evaluation of the vari- high internal validity. However, when evaluat- ous components of the health care provided. ing eVectiveness of health care, this may be Process evaluation involves making judgements disputed. In particular, the low external valid- about how well a programme operates.22 Two ity of many randomised controlled trials may basic questions of process evaluation are: does cause problems by oVering an indication of the the intervention reach the target group and was theoretical eYcacy of an intervention rather the intervention carried out in the way it was 19 than its eVectiveness in everyday practice. planned?29 A scale of diVerent measurement This also holds true for evaluation research in methods can be used in process evaluation: occupational health care. In this field, often questionnaire surveys—for example, testing the applicability of such a rigorous research knowledge or attitudes before and after a design is limited. In particular, in activities of health education programme—analysis of reg- OHSs directed to groups (improvement of istered activities, direct observation, measure- working conditions or an educational pro- ment of use, audit, etc. gramme) randomisation at the individual level Process evaluation may sometimes be re- is not possible. This problem may sometimes garded as a proxy measure for judging outcome be solved by randomisation at the population but it remains diYcult to show cause and effect level—for example, plant or department—but between process used and outcome achieved.30 this requires very large sample sizes, often not The worth of process evaluation should not be feasible. In such cases, researchers have to rely underestimated. For new health programmes, on a less rigorous design. Also ethical or legal knowledge of how a succesful or an unsucces- matters in occupational health care can inter- ful outcome was obtained, will have the most fere with the use of a true experimental design. impact on future decision making.31 Especially, Therefore, in health services research, quasi- when outcome findings are negative, a thor- experimental study designs are often chosen. ough process evaluation can provide infor- Such designs, also called non-equivalent mation on the reason for this negative out- pretest-post-test designs, make use of control come; is it a lack of implementation or a lack of groups not selected by random assignment but eYcacy of the service or programme? Compli- by techniques of matching, stratification, etc.25 ance with health programme components is In studies evaluating OHSs, researchers will always an important factor in intervention often assign factories or factory departments studies.32 As a part of process evaluation, it is to experimental and control groups. necessary to document the degree of compli- A non-experimental evaluation design (also ance as much as possible. Measuring the proc- known as single group design) includes an ess of care may in some circumstances be even experimental group only. No control group is more eVective than measuring outcomes. Out-

used in its most basic form: the one group come studies often need to be run for several http://oem.bmj.com/ post-test only design. From this design, one years to detect deﬁciencies in care.33 Some- cannot easily infer that the treatment is related times, traditional outcome measures—for ex- to any kind of change.26 More often a design is ample, accidents in safety performance—are used in which the target populations act as rare events in the statistical sense, and their own control, often on a before-after com- consequently, not sensitive enough to evaluate parison basis: the one group pretest-post-test the eVectiveness of speciﬁc intervention design. Although widely used, the validity of programmes.34 Process data may sometimes be

such a design is limited. A possibility for more sensitive measures of quality than out- on October 1, 2021 by guest. Protected copyright. strenghtening of this design is to increase the come data because a poor outcome does not number of observations before and after the occur every time there is an error in the provi- intervention. Such a time series design ideally sion of care.35 Therefore, in comprehensive includes at least three measurements before evaluation studies, true or quasi-experimental and three after the intervention has taken designs for outcome measurements should be place. The changes in trends must be consist- combined with process evaluation to monitor ent for the diVerent groups but the same inter- how this outcome was achieved. A similar dis- vention must have been introduced at different tinction can be made between summative and times.27 AdiVerent category is formed by the formative evaluation. Summative evaluation judgemental designs. In this design, no objec- has to give a judgement (in quantitative or sta- tive measurements are made, but experts, pro- tistical terms) of the value or outcome of a gramme staV, participants, or other parties programme, mostly by an outside expert. Table 1 Comparability and terminology of study designs in diVerent scientiﬁc disciplines

Rank order Evaluation study (social sciences) Clinical research/observational epidemiology Quality assurance 1 True experimental, pretest-post-test control group design Randomised controlled trial, community intervention trial 2 Quasi-experimental design Cohort study, case control study 3 Non-experimental design: eg time series Patient series, descriptive study 4 Judgemental design Peer review, audit, satisfaction with care Process evaluation Compliance, descriptive study Audit, certiﬁcation, guidelines, satisfaction Evaluation research in occupational health services 365

Formative evaluation is the systematic monitor- many occupational physicians and nurses are Occup Environ Med: first published as 10.1136/oem.56.6.361 on 1 June 1999. Downloaded from ing, often by a member of in house staV,ofan involved in general work site health promotion ongoing programme or policy with the intention programmes. In 1985, the United States Public to control and improve the progress.36 Health Service’s national work site survey showed that 65% of work sites with >50 Quality and audit employees had at least one ongoing health pro- Another contribution to evaluation research in motion programme.44 In analysing this Ameri- health care comes from quality control princi- can phenomenon of health promotion pro- ples. A definition of quality of care is “the grammes, Conrad distinguishes corporate degree to which health services for individual factors—for example, the lack of a national subjects and populations increase the likeli- health insurance system means that most of the hood of desired health outcomes and are con- companies pay for a large portion the general sistent with current professional knowledge”.37 health bill—health factors (the emergence of Also in the assessment of quality of care, the the lifestyle risk factor paradigm in medicine) Donabedian structure, process, and outcome and cultural factors (an improved interest in triad can be applied. Quality assurance is the jogging, fitness, and wellness).45 He draws process that ensures that the standards or level attention to the many pitfalls of this lifestyle of quality which have been specified are met. approach, in particular to the danger of cross- This requires audit and measurement. Audit ing the thin line from individual responsibility involves observing practice and comparing it to blaming the victim. In some other countries, with a standard. Realistic performance stand- OHSs are also involved in curative health care. ards need to be set and performance indicators In Finland, employees use the OHS units have to be developed.38 The most important instead of the municipal healthcare centres critical success factor in this approach is to partly for general practitioners services.46 develop performance indicators and perform Therefore, the question “what is the goal of meaningful measurements. OHSs ?” may lead to diVerent answers in diVerent countries or even in diVerent regions Practice guidelines or companies within one country. Moreover, In the quest for evidence-based medicine, the OHS professionals, OHS managers, employ- development of practice guidelines for health ers, and employees may have diVerent opinions professionals is rapidly gaining popularity. Pro- about the goals of OHSs. fessionalisation, accountability, and eYciency are the most important reasons.39 The degree OHS evaluation models of implementation and use of professional In the scientific literature, a clear distinction is standards or guidelines within OHSs may not always made between evaluation defini- reflect a measure of quality of the care provided tions, types of evaluation, and evaluation mod- and may be subject to evaluation at the process els. Menckel gives an overview of approaches to level. Evidence of change in health outcomes and models for evaluation of OHS activities.47 due to the eVectiveness of practice guidelines She presents a classification of some major

should be the subject of outcome evaluation. In evaluation models, the systems analysis model http://oem.bmj.com/ a systematic review on the eVect of clinical and the behavioural objectives model, being the guidelines on medical practice, Grimshaw and most prominent ones. The aim of the systems Russel concluded that explicit guidelines do analysis model is to provide an evaluation of an improve clinical practice.40 The impact of prac- entire body of activities. It is always summative tice guidelines on quality of care is, however, and is mostly initiated after a programme has often hampered by poor implementation.41 been completed—for example, “can OHSs contribute to a reduction in occupational inju-

Satisfaction ries?”. The behavioural objectives model, com- on October 1, 2021 by guest. Protected copyright. The eVectiveness of health care is not only monly used in health education programmes, is determined by quality variables but also by the more formative in nature. It evaluates the acceptance of the parties involved. Acceptance eVects of a speciﬁc measure taken—for exam- is closely associated with satisfaction of pa- ple, “have back exercises led to improved tients or populations with care. Some authors physical ﬁtness?”. Process criteria are impor- regarded satisfaction as a process measure, tant and OHS personnel may be involved more important as a means to gain acceptance of and directly in this type of evaluation. In general participation in the service being provided.27 health services research, the system analytical Others considered patient satisfaction to be model is a framework often used for evaluation. one of the desired outcomes of care, even an Examples of its use in OHSs are described by element in health status itself.23 Cho et al and by Parillo.48 49 In Finland, Husman et al applied this concept to develop MODELS AND METHODS IN OHS EVALUATION and evaluate a national OHSs system for 9 RESEARCH farmers. Input, process, and outcome compo- Evaluation objectives in OHSs nents were distinguished. To achieve the The formulation of clear objectives is a prereq- ultimate goal, a change in prevalence of work uisite for evaluation research.42 Therefore, it is related diseases, intermediate objectives—for important to know what can be considered as example, change in work methods and work the primary goal of OHSs. The OHSs vary behaviour—were chosen. This shows that it much in structure and function, more so than might be possible to evaluate an OHS system at primary health care or hospital services, even in diVerent levels of objectives, which could industrialised countries.43 In the United States, increase the eYciency of the analysis. 366 Hulshof, Verbeek,van Dijk, et al

Influence OHS importance of combining outcome evaluation Occup Environ Med: first published as 10.1136/oem.56.6.361 on 1 June 1999. Downloaded from Influence company with process measurements. A general model for evaluation of OHSs For practical and for methodological reasons, it is often not feasible to study long term outcome objectives such as a decrease in the prevalence of work related diseases. Therefore, Influence in studies that evaluate OHSs, emphasis will be on intermediate objectives such as changes in exposure or changes in knowledge, skills, attitudes, or work methods in target groups. Activities of OHSs often have only indirect Identification Assessment Control influence on the ultimate outcome on work and Figure 1 Model of changing impact of OHS activities health. The output or product of most OHS during risk identification, assessment, and control of a work activities is advice. This advice may be given to related risk.50 an individual employee, to a group of employees, or to a supervisor or manager. In A more general problem in evaluation stud- evaluation of OHSs, this advice can be ies in occupational health care, is the fact that regarded as an essential link between the proc- in the ultimate outcome of OHSs, other actors ess of delivery of care and the outcome.28 From and factors may play important and sometimes concepts of evaluation research, a general more decisive parts.50 This is schematically model for evaluation of OHSs can be outlined in figure 1. During the successive extracted.51 This general model (illustrated by phases of identification of occupational health an OHSs approach on prevention and control risk, risk assessment and control of a work of noise induced hearing loss), showing the dif- related health risk, which influence OHSs, vary ferent dimensions of aetiological research and considerably. Risk identification and risk as- evaluation or intervention research, is pre- sessment are important tasks of OHSs, and sented in figure 2. occupational health professionals in OHSs are expected to play a competent and active part in EMPIRICAL STUDIES ON EVALUATION OF OHSS OR this. However, the actual control of risk itself— OHS INSTRUMENTS for example, changes in work conditions—is A total of 52 empirical studies met the the direct responsibility of the employer, to a inclusion criteria. Most of the publications much larger degree than of the employee. come from four countries: United States, When the performed activities do reach the United Kingdom, Netherlands, and Finland. final goal, it is not necessarily a failure of the The studies were categorised according to their evaluated OHS system as such. Maybe the main object; 21 studies were directed at input OHS activities were carried out correctly, but or structure of OHSs, 13 studies deal with for some reason the employer totally ignored all processes, six with output, and 13 with advice. In evaluation of the outcome of OHSs outcome of OHSs or OHS instruments. Two http://oem.bmj.com/ or OHS activities such mechanisms have to be publications did refer to the same study and taken into account. This again stresses the one study was directed at both input and

Input/structure: Occupational health money, staff, means, methods service on October 1, 2021 by guest. Protected copyright.

Process: Noise measurements activities Audiometry Health education Intervention

Output: Company advice advice Worker advice

Outcome: Work related occupational and work related health hazards diseases

Aetiology

Noise > 80 dB(A) Threshold shift Noise induced hearing loss Figure 2 General model for evaluation of OHSs.51 Evaluation research in occupational health services 367

output. Tables 2–6 summarise the studies and trition and only to a small part on occupation- Occup Environ Med: first published as 10.1136/oem.56.6.361 on 1 June 1999. Downloaded from their main findings. In 28 studies, general ally related ailments.58 In a study of OHSs in aspects of OHSs were investigated, whereas the San Diego, it was reported that employers other 24 studies dealt with specific OHS activi- responding to the survey cited acute care as the ties or instruments, in particular pre- service most often obtained from outside employment examination and (periodic) occu- providers.52 In Finland workers often use OHS pational health surveillance. units for general practitioners’ services.46 In the United Kingdom, assessment of fitness for Evaluation of input or structure of OHSs work or sickness absence ranked first in use of We have made a distinction between character- physicians’ time,56 and in Norway, 30% of the istics of OHS systems or provider and charac- working time of occupational physicians was teristics of clients (table 2). Almost all of the directed to curative activities.57 This was also studies reviewed on input or structure of OHSs the case in the developed countries, the actual were descriptive, non-experimental, and cross practice does not always follow the demands of sectional. In only two studies, data of repeated the clients or customers of the OHSs. measurements were used and a trend was analysed, although comparable data over time Evaluation of process of OHSs or OHS were few and inaccessibile, as the authors indi- instruments cate themselves.52 60 The study by Woodall et al As in the previous section, most of the studies is the only one that compared the results of the on processes of OHSs used a non-experimental study group (frequent visitors of the OHSs) and cross sectional study design. Although the with a reference group (random sample of emphasis is still on description of the activities non-frequent visitors), but was hampered by a of OHSs (what do they do?), some studies have low response in both groups.52 In most of the a more evaluative nature (how well is it done?). studies, the data were collected by postal or Sugita et al studied the quality of biological interviewer administered questionnaire. In six monitoring methods in use in OHSs and saw a studies, additional health services data or case gradual improvement in scores on a well records were used.52 55 58 59 68 70 Because of the defined evaluation system between 1980 and descriptive character, most studies lacked 1987.71 In an external audit of occupational statistical analyses of the results. In one study,46 medical consultation records, Agius et al used a multivariate analysis was used to explain the set of quality criteria to judge the medical con- findings and in another study,55 Cohen’s ês sultation process.74 Because the “career” occu- were calculated to study the agreement be- pational physicians had significantly better tween physicians’ and employees’ perceptions scores than the “non-career” occupational of work relatedness of the health problems. In a physicians (usually part time general practi- few other studies, descriptive statistics were tioners), the authors stress the importance of presented. further training of physicians practising occu- With the exception of the study on sickness pational medicine. Behrens and Müller evalu- absence and fitness for work by Agius et al,68 the ated the self reported compliance of company

objectives of the studies on input or structure doctors with the workplace related activities as http://oem.bmj.com/ of OHSs were not evaluated against certain required by the German law on work security.72 criteria. The information in most studies in this They found that only one third of the respond- section may therefore be considered more as a ers carried out these activities. By contrast with description of input and structure of OHSs in Agius et al, they saw no significant eVect of the diVerent countries than a real evaluation. A real qualifications of the physicians; more impor- evaluation would require available standards of tant were compulsory factors such as state (best) practice or well defined criteria of care. regulations or a prevention oriented policy in

Despite this general limitation and the hetero- the company. In a study on pre-employment on October 1, 2021 by guest. Protected copyright. geneous character of the study objectives, some examinations, the variability between experi- general trends emerge from the findings. Five enced occupational physicians in a governmen- studies, conducted in diVerent populations, all tal OHS was used as a measure of reproduc- reported a shortage of physicians or specialists ibility and thus as a measure of quality.77 Poor in occupational medicine in the United States agreement was found, suggesting that the and a shift away from in factory OHSs to free- validity of judgement of medical fitness for a standing OHSs, often operating on a commer- job may be seriously questioned, even when cial basis.52–55 60 Examination of and advice on detailed fitness criteria are available. In another matters of work environment and preventive study, the value of haematological screening as health examinations of workers are, in different part of pre-employment examination in health- countries, seen as the most important OHSs care workers was questioned because in half of tasks, in particular by employees.63 66–69 For the cases abnormalities were found, but they other tasks—for example, rehabilitation or almost never aVected the decision on fitness for public health oriented health promotion—less employment.76 More positive conclusions were agreement exists between employers, employ- drawn in a study on the feasibility of pre- ees, and occupational physicians.67 69 These employment screening on occupational aller- preferences or perceptions of the role of OHSs gens in a vocational school of bakers.79 Because are, however, not always reflected in the actual of the specificity of the findings (positive skin use of OHSs or in the work content of occupa- prick tests to wheat flour, rye, and amylase), the tional physicians in practice. In many develop- fact that 4% of the total group of these young ing countries, OHSs are often concentrated on bakers already had respiratory symptoms after the predominant health problems like malnu- short exposure, and the opinion that the social 368 Hulshof, Verbeek,van Dijk, et al

Table 2 Evaluation studies on input or structure of occupational health services (OHSs) Occup Environ Med: first published as 10.1136/oem.56.6.361 on 1 June 1999. Downloaded from

Study Objective Study design

Occupational health services system and provider characteristics: Guidotti and Kuetzing52 1985 (USA) Proﬁle and trends in occupational health services in San Diego between 1974 and 1984 Descriptive study cross sectional (with trend analysis)

Brandt-Rauf et al53 1988 (USA) Current use and perceived future need for occupational physicians in non-industrial Descriptive study cross sectional occupational health services settings

Pransky54 1990 (USA) Characteristics of occupational medicine (occupational medicine) specialists Descriptive study cross sectional

Ducatman et al55 1991 (USA) Variation in occupational physician employment in large companies Descriptive study cross sectional

Agius et al56 1993 (UK) Characteristics of occupational physicians and involvement in audit Descriptive study cross sectional

Hulshof et al50 1993 (Netherlands) Occupational health services involvement in control of adverse eVects of whole body Descriptive study cross sectional vibration

Wannag and Nord57 1993 (Norway) Work content of occupational physicians Descriptive study cross sectional

Isah et al58 1996 (Nigeria) Proﬁle of occupational health services in manufacturing industries in Nigeria Descriptive study cross sectional

Client characteristics and use of occupational health services: Woodall et al59 1987 (USA) Patient characteristics (frequent visitors) Descriptive study 1 year follow up

Pedersen and Sieber,60 1989 (USA) Worker acces to health care as a result of employment Descriptive study cross sectional (with trend analysis)

Spiegel and Yassi,61 1989 (Israel) Employers’ need, use, accessibility and demand of occupational health services’ for Descriptive study cross sectional small workplaces

Barron et al62 1990 (South Africa) Provision of occupational health services in the manufacturing industry Descriptive study cross sectional

Plomp63 1992 (Netherlands) Employees’ attitude towards occupational health services and occupational physicians Descriptive study cross sectional (occupational physician)

Dryson64 1993 (New zealand) Occupational health needs assessment and use of services of workers in small Descriptive study cross sectional companies

Plomp65 1993 (Netherlands) Employees’ and physicians’ perceptions of work relatedness of problems Descriptive study cross sectional

Räsänen et al46 1993 (Finland) Use of occupational health services Descriptive study cross sectional http://oem.bmj.com/ Ritchie and McEwen66 1994 (UK) Employee perception of role of occupational health services Descriptive study cross sectional

Williams et al67 1994 (UK) Perception of role of occupational health services by managers, employee representatives Descriptive study cross sectional and occupational physicians

Agius et al68 1995 (UK) Information from managers or supervisors in referrals for sickness absence or ﬁtness for Performance study stratiﬁed work sample over a 26 month period

Dryson69 1995 (New Zealand) Workers’ preferences in delivery of occupational health services in small industry Descriptive study cross sectional

Plomp70 1996 (Netherlands) Accessibility and use of occupational health services Descriptive study cross sectional on October 1, 2021 by guest. Protected copyright.

cost at this age is more acceptable, the authors rejection or restriction rates in the OHSs which concluded that pre-employment screening in examine more comprehensively.75 Whitaker this particular occupational group may be use- and Aw confirmed the variation in examining ful. Mikovic-Kraus and Macan gave a positive practice, but they found no significant diVer- opinion on the usefulness of pre-employment ence in rejection rates between various assess- patch testing to prevent occupational contact ment methods.78 The authors of both studies allergy in industries at risk.80 The paper is, questioned the eYciency of the current pre- however, not particularly informative, in par- employment practice in the NHS and made ticular with respect to the selection of the recommendations for pre-employment assess- population used. Some of the studies on proc- ments targeted at specific occupational groups. ess are the result of medical audit from quality The study by Agius et al also included an assurance procedures, in particular in occupa- attempt to “audit the audit”; it evaluated the tional health departments within the National possible benefit of audit on the medical Health Service (NHS) in the United consultation process and found it as yet to be Kingdom.74 75 78 The study of Braddick et al only of minor significance.74 Quality assurance shows a distinct variance in comprehensiveness procedures were also used by Udasin et al in of pre-employment examination procedures in evaluating the periodic occupational health diVerent departments with slightly higher surveillance practice.81 By auditing medical Evaluation research in occupational health services 369

Table 2 Continued Occup Environ Med: first published as 10.1136/oem.56.6.361 on 1 June 1999. Downloaded from

Methods Study group/sample size Main ﬁndings

Analysis of demographic and health Occupational health services (facilities and Decline in in-plant occupational health services facilities; increase of services data; questionnaire survey human resources) in San Diego; 130 freestanding industrial medical clinics; certified occupational medicine employers (response: 29% in 1981 and 22% specialists remain few; more approval of acute care, screening and in 1983) employee assistance than preventive services Postal questionnaire Random sample of group medical practices 44 Group practices employ in total only 18 occupational physician’s and 11 (n=100, response 44%) and health occupational health nurses; 35 health maintenance organisations employ maintenance organisations (n=100, 35%) 20 occupational physicians and 7 nurses; a 200% increase of occupational health personnel between 1987 and 1997 is foreseen Postal questionnaire 1056 Physicians occupational medicine certified Number of occupational medicine specialists less than expected; 35% response: 67% employed in one company, 17% academic setting, 12% in federal or military government, 8% independent clinic; younger physicians less often in industry Interview of corporate oYcials and 25 Largest US companies (n= 514 occupational Oil and chemical plants employ largest number of occupational physicians analysis of occupational physician physicians) per employee; positive relation between company profits and number of employment data occupational physicians Postal questionnaire 200 Occupational medicine practitioners in UK Wide diversity in specialty characteristics Assessment of fitness for work or response: 83% sick leave ranked first in physicians’ time 48% engaged in audit; in 18% audit of structure, process and outcome of care Postal questionnaire All oYcially registered occupational health 75% of occupational health services judge their expertise and ability in services (n=166); response rate 67% prevention of eVects related to whole body vibration as insuYcient; small impact of occupational health services Interview by telephone 50 Occupational physicians, (5 workdays per 40% of work time concerned work matters and 30% curative activities; Work occupational physician) situation or type of occupational health services cannot explain observed diVerences in work content Questionnaire survey observation 123 Occupational health services (randomly Poor provision of occupational health services in small industries; only 4.3% (visits) selected) response rate: 91% of cases seen are occupationally related.

Analysis of 1 year occupational health Frequent visitors of occupational health services 16% of employees accounted for 50% of all visits; frequent visitors had services visits and sickness absence of automobile plant (n=235); non-frequent higher absence rates and greater health risk; black and young employees records; Questionnaire survey visitors (random sample n=199) response: were overrepresented 19% Questionnaires in 2 national surveys by 4016 Companies in 1972 (8133 workers) and Increase of delivery of oV site contractual medical care (instead of on site NIOSH in 1972 and 1981 4258 companies in 1981 (1572638 workers) physicians); large increase in the use of screening tests and decreased use of in private sector pre-employment examinations Interview 51 Plant managers, size stratiﬁed sample from a Use of occupational health services by small workplaces is restricted, few community health centre area small workplaces have met mandatory regulations; a community health centre based occupational health service deemed to be acceptable Postal questionnaire 760 Manufacturing organisations (response General deﬁciency in the quantity of occupational health services; salaried rate 51%) employees better oV than wage earners; no substantial change since 1976 Interview 3 companies with diVerent occupational health Examination of workers and working conditions seen as most important service setting, selected sample of employees occupational health services task; lack of clarity about loyalty and (n=310) independence of occupational physician in practice; company and occupational health services characteristics had little eVect on workers’ perception/appreciation Interviewer administered questionnaire 200 Workers in 35 small companies (cluster 15% assessed working conditions as poor, 38% had needed occupational sampling technique) factory response: 70% health information/advice; boss and general practicioner commonest sources of information on occupational health issues Interview Employees in 3 companies (n= 313) and Large disagreement between physicians’ and employees’ judgement; occupational physicians in these companies socioeconomic implications of the label work relatedness is important (n=6) factor in this discrepancy Computer assisted telephone interview Employees, random sample from national Employees often use occupational health services units for general health survey (n=1029) practitioner services; availability of occupational health services did not increase overall use of physicians’ services (coverage of occupational health services: 79%) http://oem.bmj.com/ Postal questionnaire 300 Employees (public sector organisation) Monitoring the working environment by 60% seen as the prime function of response: 46% occupational health services; stress greatest health concern Postal questionnaire 264 Managers, 68 union representatives, and Advice on work environment and on medical retirement seen as most 145 occupational physicians; response: important occupational health services tasks; for other tasks (eg 51%–61% rehabilitation) less agreement Audit by external peer review (Total sample: see Agius74 1994) 162 randomly Referral requests adequate in specifying duration of absence; information on selected referral letters in case records other relevant issues less frequent, only 12% provided employees’ job description Interviewer administered questionnaire 200 workers in 35 small companies (see Good support for work protective or preventive tasks of occupational health Dryson64 1993) services in small industry; little demand for general health promotion activities like lifestyle issues on October 1, 2021 by guest. Protected copyright. Rates of use from medical records; Random sample of employees (n=911) Use of occupational health services is determined by organisation of its acces; Interview interview: selected sample (see Plomp63) open consultation hour has restricted function for occupational health problems

records of 17 diVerent occupational health pational physicians’ response) of OHSs consul- facilities and comparing them against perform- tations on sickness absence and ﬁtness to ance standards, they studied the quality of continue work.68 Although the quality of the medical surveillance programmes for hazard- input was often found to be poor, the quality of ous waste workers. They found the level of the physicians’ response (the way physicians medical surveillance to vary dramatically answered the questions of the managers or among the providers. supervisors and gave advice to both managers and employees) was rated higher. Moreover, Evaluation of output of OHSs the frequency of occupational physicians’ The output of OHSs is an essential link responses was often higher than the frequency between process and outcome. One study of questions posed to them, suggesting added evaluated the output of medical consultations, value of the physicians in the formulation of the one was directed to pre-employment examina- problem. De Kort et al analysed all pre- tions, two to periodic health examinations, and employment examinations of applicants for two studies dealt with the output of workplace governmental functions duringa6year investigations. Agius et al examined both input period.85 Applicants >50 years old were four aspects (the quality of the referrals from man- times more likely to be rejected than applicants agers or supervisors) and the output (the occu- between 20 and 30 years old. Only for Table 3 Evaluation studies on process aspects of occupational health service or occupational health service instruments 370

Study Objective Study design Methods Study group/sample size Main ﬁndings

Various aspects: Sugita et al71 1991 (Japan) Quality control of biological monitoring Performance study time Round robin test; evaluation Between 79 (1980) and 179 (1987) Gradual improvement in evaluation scores during research by occupational health service series analysis based on strict criteria participating occupational health service period; quality control programme promoted adoption of modern and reliable analytical methods Behrens and Müller72 1993 Compliance of occupational physicians Descriptive study cross Postal questionnaire 502 Completed questionnaires out of 3000 Workplace related activities carried out by 1/3 of respondents, (Germany) with workplace related activities as sectional randomly selected physicians with mainly due to demand side factors (eg prevention oriented required by law certification culture in company); supply side (eg qualifications or resources) less influence Menckel73 1993 (Sweden) Activities on accident prevention of Descriptive study cross Questionnaire survey All safety engineers (n=55),occupational Accident related work is very limited in time and scope. Most safety engineers, occupational health sectional (questions concerned the health nurses (n=58) and important obstacles: “time, lack of demand for their nurses and physiotherapists activities in the preceding physiotherapists (n=47) from services”(engineers), “time”(nurses), “pressure to concentrate year) occupational health service units in on provision of treatment”(physiotherapists) slaughter house and meat industry Agius et al74 1994 (UK) Quality of occupational medical Performance study Audit of records by external 324 Consultations in occupational health Adequate occupational history in only 36% of the records; consultation records stratified sample over a peer review service of three health boards from 19 significantly better records in career occupational physicians 26 month period physicians (7 “career” and 12 compared with “non-career” occupational physicians; audit “non-career”= part time general status of occupational health service plays part, but eVect is practicioner) smaller than career status Pre-employment examination: Braddick et al75 1992 (UK) Local practice of pre-employment Descriptive study cross Postal questionnaire 22 Occupational health service Rejection rate: 0.5%; restriction: 1%; self administered examination sectional departments in one NHS region questionnaire used by all; variance in comprehensiveness of (response: 77%) further examination Evans and Aw76 1992 (UK) EYcacy of pre-employment Descriptive study 17 Analysis of pre-employment All pre-employment examination records In 50% of records at least one parameter outside reference range, haematological screening month follow up examination records and consequential actions performed by in only one case aVecting employment haematological one occupational health service during screening in pre-employment examination not eYcacious 17 months (n=988) De Kort et al77 1992 Interobserver variability in Diagnostic quality study Analysis of degree of agreement Panel of 5 occupational physicians Poor agreement between panel and original occupational health (The Netherlands) pre-employment examination on fitness for a job of patient rejudging 180 paper patient cases service physicians (ê: 0.35) and poor agreement between panel cases on paper (randomly selected from a stratified physicians (ê: 0.38); highest and lowest rejection % diVered by sample) of job applicants who were afactorof2 formerly seen by diVerent occupational health service physicians Whitaker and Aw78 1995 Practice of pre-employment Descriptive study 3 Audit by questionnaire on each Random sample of occupational health Rejection rate 0.7%; main reasons: body mass (40%) and skin (UK) examination in the NHS month period pre-employment service units in NHS (n=65) response (21%) Self administered questionnaire alone most common examination; additional rate: 82%; 9139 returned questionnaires method (49%); no significant diVerence in rejection rates medical information on in 3 month study period between various assessment methods rejected applicants De Zotti et al79 1995 (Italy) Feasibility of pre-employment Diagnostic quality study Analysis of prevalence of atopy Trainee bakers (n=144) graphic art Positive skin prick test to wheat flour, rye and amylase specific to examination screening on allergic cross sectional and sensitisation to students (n=81) bakers; 4% of bakers had respiratory symptoms due to flours: occupational respiratory disease occupational allergens in a pre-employment screening at this age is useful in this among trainee bakers vocational school occupational group Milkovic-Kraus and Usefulness of patch testing during Diagnostic study cross Analysis of patch tests and 175 Candidates undergoing 7% Of subjects showed positive reaction; patch testing as a part Macan80 1996 (Croatia) pre-employment examination for at sectional medical histories pre-employment examination for of pre-employment examination may be useful for occupations risk workplaces pharmaceutical industry in at risk workplaces Periodic occupational health examination or surveillance: Udasin et al81 1991 (USA) Quality assurance of medical Performance study one Audit of medical records by 325 Medical records from 17 examining Most facilities completed required paperwork but relevant surveillance programmes for year follow up external peer review occupational health facilities occupational history was often lacking and spirometric hazardous waste workers examination seldom performed correctly; workers were not always informed on the findings; level of medical surveillance varied dramatically among providers Conway et al82 1993 (USA) Practice of (periodic) occupational Descriptive study cross I: Nationwide survey with I: 7177 Workplace establishments (random 56% Of plants > 200 employees have occupational medicines; medical surveillance sectional standardized questionnaire sample) II: 238 establishments with 4% in plants < 20 empl.oyees; general physical examination al et Dijk, van Verbeek, Hulshof, (tel.) II: follow on survey III: comprehensive occupational medical most common component; audiometric testing most common site visits surveillance III: 25 plants with periodic test occupational medical surveillance Conway et al83 1993 (USA) Purposes and health findings of see Conway et al 82 see Conway et al 82 see Conway et al 82 Main occupational medical surveillance purposes: “protect periodic occupational medical general health”, “judge fitness for duty” and “comply with surveillance OSHA or other regulations”; repetitive trauma (8%), hearing loss (7%), and skin disorders (5%) most commonly identified health eVects Broersen et al84 1995 Reliability and generalisability of Diagnostic validity study Comparison of aggregated data 1. Periodical occupational health survey Direct comparisons of questionnaire scores between the two (The Netherlands) periodical occupational health survey collected with a standardised data from one large occupational health data files are biased by regional diVerences; however, data from diVerent regional periodical occupational service (n=36 000) 2; periodical similarities in the relative position of occupations on items with occupational health services health survey questionnaire occupational health survey data in a widespread distribution: generisability of the results plausible from diVerent sources construction industry from five occupational health services spread over

the country (n=11 000)

Occup Environ Med: first published as 10.1136/oem.56.6.361 on 1 June 1999. Downloaded from from Downloaded 1999. June 1 on 10.1136/oem.56.6.361 as published first Med: Environ Occup http://oem.bmj.com/ on October 1, 2021 by guest. Protected by copyright. by Protected guest. by 2021 1, October on Evaluation research in occupational health services 371

Table 4 Evaluation studies on output of occupational health service or occupational health service instruments Occup Environ Med: first published as 10.1136/oem.56.6.361 on 1 June 1999. Downloaded from

Study Objective Study design Methods Study group/sample size Main ﬁndings

Medical consultation: Agius et al68 1995 (UK) Responses and response Performance study Audit by external 162 Consultation Physicians’ response rate high time of occupational stratified sample peer review records from referrals (96%) for ‘likely date of return to physicians in sickness over a 26 month dealing with sickness work’ but lower for other items, absence or fitness for period absence or fitness for eg work limitations; frequency of work consultations work occupational physicians’ responses higher than frequency of questions added value of occupational physicians in formulation of the problem; no correlation between response time and completeness of records Pre-employment examination: de Kort et al85 1991 EYcacy of Case-referent study Analysis of All pre-employment Overall rejection rate 0.6%, (The Netherlands) pre-employment pre-employment examination in a large applicants > 50 years fourfold examination examination governmental increased risk; accepted and (pre-employment records occupational health rejected applicants had diagnoses examination) in a large service during a 6 in common; poor eYcacy of occupational health year time period pre-employment examination in service (101754 cases) reducing absenteeism and disablement Periodic occupational health examination or surveillance: Hessel and Zeiss86 1988 EYcacy of periodic Patient series Reanalysis of Consecutive periodic Only hearing loss and hypertension (South-Africa) examination with respect periodic examinations in occured frequently enough for to screening purposes, examination mining industry screening; periodic examinations assessing fitness for findings by two duringa6month useful for assessing fitness for work, and identifying specialists period (n=7758) work (8 % of workers required compensable diseases consideration), but not in identifying workers with compensable diseases Rose and Bengtsson87 1991 EYcacy of a health Patient series Retrospective 117 White collar Few measures were taken as a result (Sweden) examination programme analysis of case workers selected by of the laboratory examinations. based on laboratory records of age or work related The programme seemed to be of examinations participants risks) from 2000 limited value employees Workplace investigation: Mattila88 1989 (Finland) Usefulness of new Non-experimental Comparison of 8 Site visits on 3 Increase of the number and quality workplace investigation before-after output of building sites (414 of proposals for preventive method in construction workplace job analyses) measures; improvement of industry investigation questionnaire survey occupational health service before and after of workers (n=531; surveillance programme on basis introduction of a response 80%) of new method possible; new new method method superior to previous practice and implemented at moderate cost Peretz et al89 1992 (Israel) Output of occupational Judgemental Interview/ Managers and safety 80% Satisfied with quality of the health hazard surveys (constituency questionnaire oYcers of 100 report 51%; of recommendations and implementation of approach) workplaces (= 79% of fully implemented and 33% not recommendations of an all workplaces at all; implementation not related occupational health surveyed by the unit to actual hazard but to existence service unit in 1988) of regulations covering it

musculoskeletal disorders, was an association safety oYcers with the content and clarity of between diagnostic category and job demands the reports.89 The study was also dealing with http://oem.bmj.com/ apparent. Accepted and rejected applicants an outcome aspect: the extent to which the had diagnoses in common. These findings sug- recommendations, given in the reports, were gest poor eYcacy of the pre-employment implemented after 2 years. Satisfaction with examination for reducing absenteeism and the quality of the reports was high but half of disablement. Hessel and Zeiss evaluated a the recommendations were not or partially car- periodic examination programme in the min- ried out. ing industry and concluded that it was on October 1, 2021 by guest. Protected copyright. probably useful as a means of assessing fitness Evaluation of outcome of OHSs or OHS for work, but not so much in health screening instruments or in identifying compensable occupational The eVects of care delivered by OHSs on work diseases.86 Rose and Bengtsson reported the environment and health status of individual limited value of ECG and laboratory examina- employees or worker populations can be tion as a part of a general health examination of regarded as the ultimate outcome. Although employees.87 Few measures were taken as a seen by some authors as a process measure, the result of these examinations, other than degree of clients’ satisfaction with care is often re-examinations. Mattila studied the output of used as an outcome variable. In this review, we a new systematic method of investigating the have classified three studies on satisfaction with workplace (based on job analysis, worker the care delivered by OHSs under outcome involvement, and group problem solving) used evaluation (table 5). by OHS teams in the construction industry.88 Seven of the outcome studies evaluated care In a non-experimental before-after study de- delivery by OHSs in general. The other sign, the new method was found to be better outcome studies dealt with a specific OHS than the previous examination method: it activity. From a methodological point of view, increased the number and quality of proposals the research designs of most of the outcome to line management for preventive measures evaluation studies are weak. In only one study and improved the occupational healthcare pro- was a quasi-experimental study design gramme. In a questionnaire survey, Peretz et al applied.91 Another study used a before-after evaluated workplace investigation reports, aim- design with repeated measurements.102 Lo- ing to assess the satisfaction of managers and wenthal made internal comparisons in a group Table 5 Evaluation studies on outcome of occupational health service or occupational health service instruments 372

Study Objective Study design Methods Study group/sample size Main ﬁndings

Occupational health service in general: Wood et al90 1987 (Australia) Satisfaction of managers, employees, Descriptive study cross Questionnaire survey 32 Firms with occupational health service; Employees less satisfied with current services and occupational health workers sectional response 54%, 143 usable than managers; occupational health workers with delivery of occupational health questionnaires (32 managers,76 more sceptical of health maintenance service employees, 35 OH workers) programmes Husman et al91 1990 (Finland) Use and eVectiveness of a farmers’ Quasi-experimental Questionnaire survey “before” and 1980-1982: Seven experimental and seven Knowledge, use of personal protective Notkola et al92 1990 (Finland) occupational health service system, pretest-post-test design; “after”: 1979-1982-1986 reference occupational health service equipment and hygienic behaviour increased delivered by municipal health farmers’ occupational health (n=3200 farmers); 1985: 4 experimental significantly among farmers’occupational centres service experiments: occupational health services (n=881) health service-participants; no diVerences in 1980-1982 and 1985 “after” questionnaire surveys improvement of working conditions between (n=10 700, respondants n=2866) participants and non-participants ultimate goal not realised Draaisma et al93 1993 Quality and eVectiveness of Judgemental design cross Interview with occupational health Occupational physician or occupational Consultation in sickness absenteeism takes most (The Netherlands) occupational health service sectional service teams (constituency health teams (n=51), equally distributed time (33%); 85% positive about eVects of activities in selected companies as approach) over occupational health service type, their advice but most occupational health seen by the providers themselves size, and involvement in sickness teams had no criteria for eVectiveness absence certification Rogers et al94 1993 (USA) Employee satisfaction with Descriptive study cross Questionnaire mailed to employees All scheduled occupational health 90% Waited < 15 minutes to see nurse or occupational health service sectional after every occupational health service-visits in 3 months (n=600) in a physician; 2/3 were very satisfied with both service visit large company; response 82% nursing and medical care provided Fitko et al95 1994 (USA) Comparing costs of in-house Cost eVectiveness analysis Comparison of in-house One in house refinery medical Cost for the same services of in-house occupational health service versus occupational health service base department; a total of 109 invoices occupational health service 42% less than of external occupational health expenses with invoices from outside providers (savings of time away from service outside providers work not included) Weel and Slotboom96 1996 Feasibility of and company Judgemental design before Registration of occupational health One year experiment with new programme New approach was feasible for occupational (The Netherlands) satisfaction with a company and after service activities; interviews with of occupational health service activities health service professionals; satisfaction of specific programme of company oYcials; inquiry of in companies (n=7) companies was increased occupational health service occupational health service activities professionals Pachman et al97 1996 (USA) Cost savings of an on-site Questionnaire survey Estimation of saved days of All employees using the occupational On-site occupational health service resulted in occupational health service absenteeism health service over a period of 4 months average reduction of absenteeism of 3.3 days (n=133) per employee per user Pre-employment examination: Lowenthal98 1986 (USA) EVectiveness of diVerent Retrospective follow up study Analysis of worker placement, Records of 200 consecutive candidates No significant diVerence between groups; pre-employment examination follow up period: 2-4 years compensation claims and with comprehensive pre-employment comprehensive pre-employment examination methods healthcare costs examination and 200 candidates with is not a cost-eVective activity minimal pre-employment examination Periodic occupational health examination or surveillance: Conway et al99 1993 (USA) EVects or benefits of occupational Descriptive study cross See Conway et al82 See Conway et al82 In 61% no change in outcome parameters medical surveillance programs as sectional (illnesses, insurance costs); main perceived perceived by respondents to the benefits were “healthier and happier OSHA survey employees”(42%); most firms had no procedure for evaluating eVectiveness of their occupational medical surveillance Ren et al100 1994 (Japan) Impact of periodic health Patient series Analysis of health care insurance Local government employees in public Abnormalities found in 57-70% of which 7-14%

examination on health care claim and utilisation rates after service sectors (n=19,146) in 6 year required serious medical attention; the al et Dijk, van Verbeek, Hulshof, utilisation and costs introduction of annual periodic period (96% participation in periodic comprehensive periodic health examination health examination health examination) played some part in increasing health care utilisation and costs. Workplace investigation and health education: Kahan et al101 1995 (Israel) Consumer satisfaction with quality of Descritive study Postal questionnaire All regular clients (managers, safety High satisfaction (waiting time for last report occupational hygiene services oYcers) of occupational hygiene strongest predictor of satisfaction); safety services during a 2 year period (n=144); improvement, legal requirements, and service response 47% credibility determine contract-continuation Porru et al102 1993 (Italy) EVectiveness of a health education Non-experimental before and Repeated measurements Workers in seven small factories (n=50); Signiﬁcant improvement of blood lead programme on lead after design (questionnaire/blood samples) 34 subjects in complete follow up concentrations and knowledge scores; reduction of blood lead seems to be due to

change in hygienic behaviour

98 of pre-employment examinations. Most of the between participants and non-participants Occup Environ Med: first published as 10.1136/oem.56.6.361 on 1 June 1999. Downloaded from other studies were descriptive. Wood et al were negligible. However, farmers’ knowledge investigated the satisfaction of managers, em- on health hazards, use of protective equipment, ployees, and the OHS workers involved with and occupational hygienic behaviour scored the delivery of care by OHSs in 32 firms in significantly higher among participants than industry.90 Polarised views were found: employ- among the reference group, indicating that on ees were less satisfied than managers, with the process level the OHS system was success- OHS workers in between. By contrast, Rogers ful. As a possible reason for this discrepancy, et al reported high employee satisfaction with the authors suggest that the farmers’ OHS sys- both nursing care and physicians’ services from tem may be concentrated too much on the use an OHS in a large pharmaceutical company.94 of personal protective equipment. Draaisma et High levels of satisfaction were also found in a al evaluated the eVectiveness of OHS activities study of Kahan et al among managers and in selected companies by interviewing the OHS safety oYcers with accessibility, quality, and teams themselves (constituency approach).93 cost of occupational hygiene services, although Most of the teams were positive in their assess- the response rate of only 47% may be ment of the results of their advice to the com- selective.101 The publications of Husman et al panies but their criteria for defining eVective- and Notkola et al, both referring to the same ness were vague and output criteria for their study, described the development and evalua- own activities were lacking. Weel and Slotboom tion of a national farmers’ OHS system in Fin- evaluated a method of delivering diVerential land between 1979 and 1987.91 92 The func- company health care based on the particular tional adequacy (with input and process demands and needs of companies compared aspects) and the eVectiveness of this system with the standard care in seven companies was evaluated. Unfortunately, information on within an OHS.96 The approach was found to the evaluation of OHSs in both articles on be feasible and a trend of increased satisfaction methodology and design of this unique large within the companies was noted. Fitko et al scale study was not optimal; in particular the examined the cost eVectiveness of the trend in assignment of farmers to the experimental and the United States for corporations to switch the control group and the relation between the from in house medical departments to outside questionnaire surveys and the experiments. contract organisations for OHSs.95 They found When improvement in working conditions was the cost for the same services of an in house the ultimate indicator of the outcome evalua- department at a large oil refinery to be 42% less tion, the system was not eVective: diVerences than that of outside providers (other benefits Table 6 Summary of evaluation of occupational health service or occupational health service instruments

Input Process Output Outcome

Occupational health service in general Guidotti and Kuetzing 52 - Husman et al 91- Brandt-Rauf et al 53- Wood et al 90 ± Pransky 54- Draaisma et al 93 ± Isah et al 58- Rogers et al 94 + http://oem.bmj.com/ Spiegel and Yassi 61- Fitko et al 95 + Barron et al 62- Weel and Slotboom 96 + Plomp 63- Pachman et al 97 + Dryson 64- Ducatman et al 55 ± Agius et al 56 ± Wannag and Nord 57 ± Woodall et al 59 ± Pedersen and Sieber 60 ± Ritchie and McEwen 66 ± Williams et al 67 ± on October 1, 2021 by guest. Protected copyright. Räsänen et al 46 + Dryson 69 +

Occupational health consultation Plomp 65-Agiuset al 74- Plomp 70-

Occupational rehabilitation Williams et al 67-Agiuset al 68 + Agius et al 68- Plomp 70 ±

Pre-employment examination Braddick et al 75- De Kort et al 85- Lowenthal 98- Evans and Aw 76- De Kort et al 77- Whitaker and Aw 78 ± De Zotti et al 79 + Milcovic and Macan 80 +

Periodic occupational health examination/surveillance Udasin et al 81- Hessel and Zeiss 86- Conway et al 99 ± Conway et al 8283 ± Rose and Bengtsson 87-Renet al 100 ± Sugita et al 71 + Sugita et al 71 + Broersen et al 84 + Broersen et al 84 +

Workplace investigation or evaluation of hazards Hulshof et al 50- Behrens and Müller 72- Mattila 88 + Kahan et al 101 + Dryson 69 + Menckel 73- Peretz et al 89 +

Occupational health education Hulshof et al- Husman et al 91 +Porruet al 102 + Porru et al 102 +

- = Negative result; + = positive result; ± = indeﬁnite result or descriptive study. 374 Hulshof, Verbeek,van Dijk, et al

not included) and therefore recommend cor- Discussion Occup Environ Med: first published as 10.1136/oem.56.6.361 on 1 June 1999. Downloaded from porations to perform similar analyses before a It is widely recognised that health services decision is made to switch from inside to research and evaluation in OHSs should be outside OHSs, at least if a desire to reduce placed high on the agenda of both researchers expenses is the objective. As well as this, Pach- and practitioners in occupational health. In a man et al assessed the hidden saving in costs of Delphi study among 150 experts from OHSs, an on site medical centre in a large company; in scientific research institutes, governmental and particular absenteeism was found it to be other administrative bodies, and companies in substantial.97 The cost eVectiveness of pre- the Netherlands, design, implementation, and employment examination was also questioned. evaluation of control measures was ranked highest in the priority topics for research in the Lowenthal examined in a retrospective analysis field of occupational health and safety.104 Con- of records in a group of healthcare workers, the sidering the size of the field of occupational outcome of a non-specific comprehensive pre- health care, the social and economical magni- placement health evaluation compared with a 98 tude of occupational health problems, and the minimal evaluation by a nurse. No diVerence growing awareness and position of health serv- in duration of employment, reason for ending ices research, the number of studies on evalua- work, workers’ compensation claims, and use tion of OHSs or OHS activities that met the of healthcare resources in a period of 2–4 years (not very restrictive) inclusion criteria is after the examination was found. He concluded remarkably limited. Moreover, the nature of that comprehensive pre-employment examina- many of the 52 studies included in this review tion is not a cost eVective activity. In a large is more descriptive than really evaluative. nationwide survey in the United States, the Probably, many evaluation studies remain Occupational Safety and Health Administra- unpublished. Cherry refers to the often prevail- tion (OSHA) investigated the prevalence, pur- ing lack of interest of funders of intervention poses, and eVects or benefits from (periodic) programmes in evaluating eVectiveness of occupational medical surveillance pro- OHSs, and the rigor of the scientific commu- grammes. Conway et al reported that most nity rejecting every other approach than 105 responders with an existing programme did not randomised controlled trials. detect a change in outcome variables like Our classification of the studies in input, illnesses or insurance costs as a consequence of process, output, and outcome may be arbitrary. the programme.99 Most of the responders had, A sharp border between process and outcome indicators does not always exist and some however, no procedure for evaluating the eVec- studies deal with diVerent aspects. Input, proc- tiveness of their medical surveillance pro- ess, and outcome are not characteristics or gramme. In the same journal issue, however, variables of quality but they oVer a suitable this OSHA study was severely criticised approach for gaining information in the because of the lack of a clear definition of presence or absence of indicators of quality.23 occupational medical surveillance, leading to For us, in this field with heterogeneous study misinterpretation by responders.103 To test the

objectives, it helped to categorise the evalua- http://oem.bmj.com/ statement that periodic health examination tion studies. leads to an increase in use or costs of health In general, the methodological quality of care (in our opinion not a negative outcome most of the reviewed studies is not high. measure in itself), Ren et al analysed rates of Robust study designs were only occasionally use of healthcare services and insurance claims used. Most of the studies did not have an active of a large group of local government employees intervention or a quasi-experimental design, duringa6yearperiodafter introduction of a did not use control groups, and did not deﬁne

comprehensive periodic health examination standards or criteria against which the study on October 1, 2021 by guest. Protected copyright. programme.100 Confirmation of increasing object was evaluated. Of course, this in itself costs and use was found, especially as a short does not necessarily disqualify these studies. term eVect, but the authors themselves dis- Also qualitative research designs and case cussed distinct limitations of the study—for studies may be of value in studying aspects of example, the lack of an adequate control group. occupational health care as provided by OHSs. An evaluation study of an employee health Evaluation of OHSs can (and has to) be performed at di erent levels. Moreover, we education programme was conducted by Porru V have considered already the fact that in evalua- et al.102 The eVects of health education in work- tion of health care in daily practice, the ers exposed to lead were examined before, 4 applicability of rigorous (intervention) study months after, and 1 year after the education designs, for diVerent reasons, is not always programme was given by the OHSs in seven possible and researchers are forced to make small factories. A highly significant improve- compromises. ment in knowledge of workers about lead poi- When looking at the results of this literature soning and its prevention and also a decrease of review (table 6 summarises the findings of this PbB concentrations was found. Because during review) a diVerentiated picture of the evidence the study period no hygiene improvements or of eVectiveness of OHSs arises. The OHSs or engineering changes were undertaken, the OHS programmes in general are studied from reduction of PbB seemed to be due to a change the input perspective: how many occupational in hygienic behaviour. The authors therefore physicians work in OHSs? The drawback of concluded that this OHS health education these studies and the reason for the many programme was eVective. indefinite results is that they usually remain at Evaluation research in occupational health services 375

the descriptive level. No criteria are used to RECOMMENDATIONS Occup Environ Med: first published as 10.1136/oem.56.6.361 on 1 June 1999. Downloaded from assess the quality of the input. So, the questions There is still much left to be studied more such as “is the number of physicians suYcient thoroughly in studies that evaluate OHSs. In to provide adequate care”, and “are all research programmes, much more eVort branches of industry provided with adequate should be directed at the scientific evaluation services” usually cannot be answered. Out- of the occupational health consultation and come, studied as satisfaction with OHSs in rehabilitation activities of occupational physi- general, shows a slightly positive picture. cians. In this field it is important to use or Despite this satisfaction, input in OHSs is in develop an explicit theoretical basis for such most studies considered to be inadequate. studies. A clear theory on which activity or Evaluation of the eVectiveness of OHS activi- intervention could work best can help research- ties and implementation of adequate measures ers design studies that provide more interpret- may change this lack of adequate input. able and generalisable results. These activities It is striking to see that occupational health lend themselves quite well to rigorous study consultations and occupational rehabilitation designs of methods—such as the randomised are hardly studied. In sharp contrast with the controlled trial. A recent example of such a trial extensive time spent on consultation by is the study by van der Weide et al on the quality of occupational rehabilitation by occupa- occupational physicians in most countries, the 109 process remains more or less a “black box” and tional physicians for low back pain. Studying its outcome is hardly known. Moreover, the few outcome and process quality of the consulta- studies that are conducted on this tend to be tion and rehabilitation activities could give negative on input and process quality. clues for immediate improvement. However, By contrast, the pre-employment examina- this type of study requires the construction and tion has been well studied. Most of the studies implementation of professional guidelines with give a negative result on process quality as well which the usual input and process can be compared. To date we know of few professional as on outcome. Only in specific circumstances guidelines for and process evaluation of OHS may the pre-employment examination be activities. useful—such as for the prevention of occupa- In OHSs activities directed to groups—for tional asthma in certain occupational groups.79 example, an educational programme— However, even in this specific disorder, this can randomisation at an individual level is not pos- be questioned. In 1982, Cockroft et al con- sible. This problem may be solved with a cluded from a study among laboratory animal quasi-experimental approach and assigning workers that pre-employment allergy screening plants or departments to an experimental and a would not substantially reduce the problem of 106 control group of OHSs. We have recently used occupational allergy in this group. More this design in evaluating an OHSs prevention recently, de Kort and van Dijk made a calcula- programme on the eVects of whole body tion based on the validity characteristics of the vibration.110 tests to be used and the available epidemiologi- Much work still remains to be done. More cal data on risk factors relative to the adverse

research is needed on demands and needs, http://oem.bmj.com/ outcome to be prevented, and estimated the policy and practice development, aspects of eVectiveness of pre-employment examination input, process and output (and their interrela- 107 for this disorder to be low. This increasing tions), and eYcacy and eVectiveness of OHSs amount of evidence of lack of eVectiveness and in terms of beneﬁts and harms of interventions. eYciency of the pre-employment examination A theoretical framework for evaluation of should lead to its general abandonment as a OHSs should be discussed and further devel- means of selecting personnel by OHSs. oped, in particular on occupational medical Also, some positive ﬁndings emerge from

consultation and rehabilitation. Researchers on October 1, 2021 by guest. Protected copyright. this review. There is some evidence that and practitioners should collaborate to work on periodic health monitoring or surveillance, appropriate ways to monitor and evaluate per- especially when directed to specific occupa- formance and quality of OHSs in practice. The tional exposures, can be carried out with use of OHS databases for evaluation of eVect reasonable process quality. Whether this leads should be encouraged, and easily measurable to a favourable outcome cannot be inferred outcome measures are needed for small scale from the studies included in this review. In a evaluation by OHSs themselves. There is a small scale evaluation of a periodic occupa- need for new and better performance tional health examination programme of one indicators.111 In another paper, we have de- OHS in The Netherlands, most of the partici- scribed the development and evaluation of a pating employees were positive about the proc- quality assessment instrument for occupational ess quality of the programme, but only 20% physicians.112 Such methods can be used for noticed a clear improvement in working condi- both single evaluations and for a continuing tions as a positive result of the programme.108 process of improving occupational health care. Although based on only a few studies, positive In the quest for evidence-based occupational results were reported on process and outcome health care more and better research on eVec- of education on occupational health tiveness of OHSs is needed but also non- hazards.91 92 102 The identification and evalua- experimental activities—such as quality assur- tion of occupational health hazards by a work- ance or guideline implementation—should be place survey can be done with a perceived high guided by scientific principles. Researchers output quality, which, however, does not guar- must be encouraged to publish the results antee a favourable outcome. internationally. Occupational medical journals 376 Hulshof, Verbeek,van Dijk, et al

should consider measures to facilitate publica- program: design, methods, amd baseline findings from a Occup Environ Med: first published as 10.1136/oem.56.6.361 on 1 June 1999. Downloaded from 3-year randomized clinical trial. Aging Clin Exp Res 1994;6: tions on this topic. 105–20. 33 Davies HTO, Crombie IK. Assessing the quality of care. Measuring well supported processes may be more enlight- 1 World Health Organisation. Global strategy on occupational ening than monitoring outcomes. BMJ 1995;311:766. health for all. Recommendation of the 2nd meeting of the WHO 34 Cohen HH, Jensen RC. Measuring the eVectiveness of an Collaborating Centres in Occupational Health. Beijing, China industrial truck safety training program. J Safety Res 1984; 11–14 October 1994. Geneva: WHO, 1995:95. 15:125–35. 2 International Commission on Occupational Health. Policy 35 Brook RH, McGlynn EA, Cleary PD. Quality of health care. Document of the Scientific Committee on Health Services Part 2: measuring quality of care. N Engl J Med 1996;335: Research and Evaluation in Occupational Health. 966–70. Amsterdam: ICOH, 1993. 36 Tijssen IMJG, Elsinga E. Evaluatie-onderzoek op het terrein 3 Pope C, Mays N. Opening the black box: an encounter in van de gezondheidszorg (Evaluation research in the field of the corridors of health services research. BMJ 1993;306: medicine). In: Maarse JAM, Mur-Veeman (eds). Beleid en 315–8. beheer in de gezondheidszorg (Policy and management in health 4 Epstein AM. The outcomes movement—will it get us where care). Assen, The Netherlands: van Gorcum, 1990. we want to go? N Engl J Med 1990;323:266–70. 37 Welch CE, Grover PL. An overview of quality assurance. 5 Wannag A. Classification system for work in occupational Medical Care 1991;29(suppl):AS8–28. health services. Work content of Norwegian occupational 38 Macdonald EB, Agius RM, McCloy EC, et al ,eds.Quality physicians. Scand J Work Environ Health 1993;19:390–3. and audit in occupational health. London: Faculty of 6 National Institute for Occupational Safety and Health Occupational Medicine of the Royal College of Physicians (NIOSH). National research agenda. Washington: NIOSH, of London, 1995. DHHS, 1996. (Publ No 96–115.) 39 Klazinga N. Compliance with practice guidelines: clinical 7 Behrens J, Westerholm P, Baranski B. Introduction: the new autonomy revisited. Health Policy 1994;28:51–66. occupational health and evaluation research as its path- 40 Grimshaw JM, Russell IT. EVect of clinical guidelines on breaker. In: Behrens J, Westerholm P, eds. Occupational medical practice: a systematic review of rigorous evalua- health policy, practice and evaluation. Bremen: WHO tions. Lancet 1993;342:1317–22. EUR/ICP/EPOL, 1996. 41 Grilli R, Lomas J. Evaluating the message: the relationship 8 Pransky G, Himmelstein J. Outcomes research: implications between compliance rate and the subject of a practice for occupational health. Am J Ind Med 1996;29:573–83 guideline. Medical Care 1994;32:202–13. 9 Husman K. Principles and pitfalls in health services 42 American Public Health Association (APHA) technical research in occupational health systems. Occup Med 1993; report. Criteria for the development of health promotion 43(suppl 1):10–14. and education programs. Am J Public Health 1987;77:89– 10 Van Dijk FJH, de Kort W, Verbeek JHAM. Quality 92. assessment of occupational health care instruments. Occup 43 Karvonen M. Epidemiology in the context of occupational Med 1993;43(suppl 1):28–33. health. In: Karvonen M, Mikheev MI, eds. Epidemiology of 11 World Health Organization. Evaluation of occupational health occupational health. Copenhagen: WHO Regional Publica- and industrial hygiene services. Report on a WHO Working tions, 1986. (European Series No 20.) Group. Copenhagen: WHO, 1982. (Euro Reports and 44 Stonecipher LJ, Hyner GC. Health practices before and Studies 56.) after a work-site health screening. DiVerences among sub- 12 Schulte PA, Goldenhar LM, Connally LB. Intervention populations of employees. J Occup Med 1993;35:297–306. research: science, skills, and strategies. Am J Ind Med 1996; 45 Conrad P. Worksite health promotion: the social context. 29:285–8. Soc Sci Med 1988;26:485–9. 13 Skov T, Kristensen TS. Etiologic and prevention eVective- 46 Räsanen K, Notkola V, Kankaanpää E, et al. Role of the ness intervention studies in occupational health. Am J Ind occupational health services as a part of illness-related pri- Med 1996;29:378–81. mary care in Finland. Occup Med 1993;43(suppl 1):23–7. 14 Goldenhar LM, Schulte PA. Intervention research in occu- 47 Menckel E. Evaluating and promoting change in occupational pational health and safety. J Occup Med 1994;36:763–75. health services - models and applications. Stockholm: The 15 Van der Weide WE, Verbeek JHAM, Tulder MW van. Voca- Swedish Work Environment Fund, 1993. tional outcome of interventions for low-back pain. Scand J 48 Cho KS, Park H, Lee SH, et al. Evaluation of occupational Work Environ Health 1997;23:165–78. health service systems. Seoul: Catholic Industrial Medical 16 Warner KE, Wickizer TM, Wolfe RA, et al. Economic impli- Center/Catholic University Medical College, 1992. cations of workplace health promotion programs: a review (Center File 3-P-89–0326, Final technical report.) of the literature. J Occup Med 1988;30:106–12. 49 Parillo VL. Systems analysis of an occupational health 17 Pelletier K. A review and analysis of the health and department. Recommendations to increase eVectiveness. cost-eVective outcome studies of comprehensive health American Association of Occupational Health Nurses Journal promotion and disease prevention. American Journal of 1993;41:220–7. Health Promotion 1991;5:311–5. 50 Hulshof CTJ, Verbeek JHAM, van Dijk FJH. Development 18 Pelletier K. A review and analysis of the health and and evaluation of an occupational health services pro- http://oem.bmj.com/ cost-eVective outcome studies of comprehensive health gramme on the prevention and control of eVects of promotion and disease prevention programs at the vibration. Occup Med 1993;43(suppl 1):38–42. worksite: 1991–3 update. Am J Health Promot 1993;8:50– 51 Verbeek JHAM, van der Weide WE, Hulshof CTJ. Ervarin- 62. gen met kwaliteitsonderzoek en kwaliteitszorg in de Neder- 19 Black N. Why we need observational studies to evaluate the landse bedrijfsgezondheidszorg (Experiences with quality eVectiveness of health care. BMJ 1996;312:1215–8. assessment and quality assurance in the Dutch occupa- 20 Rossi PH, Freeman HE. Evaluation, a systematic approach, tional health care). Problemen van Arbeidsgeneeskunde (Prob- 3rd ed. Beverly Hills: Sage, 1985. lems in occupational medicine). Leuven, Belgium: Afd 21 Oxman AD, ed. Section VI: Preparing and maintaining Arbeids- en verzekeringsgeneeskunde KU Leuven 1996; systematic reviews. Cochrane Collaboration handbook. Oxford: 31:99–109. Cochrane Collaboration, 1994. 52 Guidotti TL, Kuetzing BH. Competition and 22 Holland WW, ed. Evaluation of health care. Oxford: Oxford despecialization: an analytical study of occupational health on October 1, 2021 by guest. Protected copyright. University Press, 1983. services in San Diego, 1974–84. Am J Ind Med 1985;8: 23 Donabedian A. The quality of care. How can it be assessed? 155–65. JAMA 1988;260:1743–8. 53 Brandt-Rauf PW, Teichman RF. Current and future needs 24 Taylor Fitz-Gibbon C, Lyons Morris L. How to design a pro- for occupational medicine physicians in non-industrial gram evaluation. Newbury Park: Sage, 1987. settings: a survey of multispecialty group medical practices 25 Campbell DT, Stanley JC. Experimental and quasi- and health maintenance organizations. J Occup Med 1988; experimental designs for research. Reprinted from handbook 30:928–33. of research on teaching. Boston: Houghton MiZin, 1963. 54 Pransky G. Occupational medicine specialists in the United 26 Cook TD, Campbell DT. Quasi-experimentation. Design and States: a survey. J Occup Med 1990;32:985–8. analysis issues for field settings. Boston: Houghton MiZin, 55 Ducatman AM, Forman S, Teichman R, et al. Occupational 1979. physician staYng in large US corporations. J Occup Med 27 Guidotti TL, Cowell JWF, Jamieson GG, et al. Program 1991;33:613–8. evaluation. In: American Medical Association. Occupational 56 Agius RM, Lee RJ, Murdoch RM, et al. Occupational physi- health services, a practical approach. Chicago: AMA, 1989. cians and their work; prospects for audit. Occup Med 1993; 28 Draaisma D. A conceptual approach for the evaluation of 43:159–63. the quality and eVectiveness of occupational health care. 57 Wannag A, Nord E. Work content of Norwegian occupa- In: Rantanen J, Lehtinen S, eds. New trends and tional physicians. Scand J Work Environ Health 1993;19: developments in occupational health services. Amsterdam: 394–8. Elsevier, 1991. 58 Isah C, Asuzu MC, Okojie OH. Occupational health 29 Van Dongen M. Evaluatie van GVO-interventies (Evalua- services in manufacturing industries in Nigeria. Occup Med tion of health education interventions). In: Damoiseaux V, 1996;46:333–6. van der Molen HT, Kok GJ. Gezondheidsvoorlichting en ged- 59 Woodall GE, Higgins CW, Dunn JD, et al. Characteristics of ragsverandering (Health education and behavioral change). the frequent visitor to the industrial medical department Assen, The Netherlands: van Gorcum, 1993. and implications for health promotion. J Occup Med 1987; 30 Bloom BS. Does it work? The outcomes of medical 29:660–4. interventions. Int J Technol Assess Health Care 1990;6:326– 60 Pedersen DH, Sieber WK. Some trends in worker access to 32. health care in the United States (1974–83). Am J Ind Med 31 Long RJ, Wilkinson WE. A theoretical framework for occu- 1989;15:151–65. pational health program evaluation. Occupational Health 61 Spiegel J, Yassi A. Community health centre-based occupa- Nursing 1984:257–9. tional health services for the small workplace: an Ontario 32 Rubenstein LZ, Aronow HU, Schloe M, et al. A home-based study of employer acceptability. Can J Public Health 1989; geriatric assessment, follow-up and health promotion 80:355–8. Evaluation research in occupational health services 377

62 Barron P, Steyn SD, Malekela LL, et al. Occupational health occupational hygiene survey reports. Ann Occup Hyg 1992; Occup Environ Med: first published as 10.1136/oem.56.6.361 on 1 June 1999. Downloaded from services in Johannesburg and Randburg. South Afr Med J 36:229–38. 1990;77:581–4. 90 Wood JM, Kelman R, Pilkington W, et al. Comparative per- 63 Plomp HN. Workers’ attitude towards the occupational ceptions of the delivery of occupational health services in physician. J Occup Med 1992;34:893–901. industry. Journal of Occupational Health and Safety— 64 Dryson E. Occupational health needs in small industry in Australia and New Zealand 1987;3:632–8. New Zealand: preferred sources of information. Occup Med 91 Husman K, Notkola V, Virolainen R, et al. Farmers’ occupa- 1993;43:176–9. tional health program in Finland 1979–88: from research 65 Plomp HN. Employees’ and occupational physicians’ diVer- to practice. Am J Ind Med 1990;18:379–84. ent perceptions of the work-relatedness of health problems: 92 Notkola V, Husman K, Tupi K, et al. Farmers’ occupational a critical point in an eVective consultation process. Occup health programme in Finland, 1979–87. Soc Sci Med 1990; Med 1993;43(suppl 1):18–22. 30:1035–40. 66 Ritchie KA, McEwen J. Employee perceptions of workplace 93 Draaisma D, Dam J, Winter CR de. Quality and health and safety issues. Can a questionnaire contribute to eVectiveness of occupational health services evaluated by health needs assessment? Occup Med 1994;44:77–82. 67 Williams N, Sobti A, Aw TC. Comparison of perceived the providers. Occup Med 1993;43(suppl 1):50–5. occupational health needs among managers, employee rep- 94 Rogers B, Winslow B, Higgins S. Employee satisfaction with resentatives and occupational physicians. Occup Med 1994; occupational health services. Results of a survey. American 44:205–8. Association of Occupational Health Nurses Journal 1993;41: 68 Agius RM, Seaton A, Lee RJ. Audit of sickness absence and 58–65. ﬁtness-for-work referrals. Occup Med 1995;45:125–30. 95 Fitko J, Kalina CM, Fisher AM, et al. A cost-eVectiveness 69 Dryson E. Preferred components of an occupational health analysis of an in-house corporate occupational medical service for small industry in New Zealand: health department. J Occup Med 1994;36:161–5. protection or health promotion? Occup Med 1995;45:31–4. 96 Weel ANH, Slotboom MAJ. Quality improvement by means 70 Plomp HN. Accessibility and utilization of occupational of diVerential occupational health care: an experiment in a health services. Scand J Work Environ Health 1996;22:216– regional occupational health centre. Occup Med 1996;46: 22. 20–4. 71 Sugita M, Harada A, Taniguchi M, et al. Quality control 97 Pachman JS, Stempien DE, Milles SS, et al. The hidden program on biological monitoring by Japan Federation of savings of an on-site corporate medical center. J Occup occupational health organizations. Int Arch Environ Health Environ Med 1996;38:1047–8. 1991;62:569–77. 98 Lowenthal G. Medical center worker preplacement 72 Behrens J, Müller R. Supply and demand factors in occupa- screening: a follow-up study. J Occup Med 1986;28:451–2. tional health; determinants of self-reported compliance 99 Conway H, Simmons J, Talbert T. EVects of occupational with the work place related requirements of the German medical surveillance programs as perceived by respondents Work Safety Law. Occup Med 1993;43(suppl 1):47–9. to the Occupational Safety and Health Administration. J 73 Menckel E. Accident prevention by safety engineers within Occup Med 1993;35:687–97. occupational health services in Sweden. Safety Sci 1993;16: 100 Ren A, Okubo T, Takahashi K. Comprehensive periodic 465–84. health examination: impact on health care utilisation and 74 Agius RM, Lee RJ, Symington IS, et al. An audit of occupa- costs in a working population. J Epidemiol Community tional medicine consultation records. Occup Med 1994;44: Health 1994;48:476–81. 151–7. 101 Kahan E, Peretz C, Rybski M, et al. Determination of con- 75 Braddick MR, Atwell CP, Aw T-C. Audit of pre- sumer satisfaction: a basic step for quality improvement of employment health assessment in the National Health an occupational hygiene service. Occup Med 1995;45:193–8. Service. Occup Med 1992;42:36–8. 76 Evans G, Aw TC. Should prospective NHS employees have 102 Porru J, Donato F, Apostoli P, et al. The utility of health routine blood counts? Occup Med 1992;42:79–82. education among lead workers: the experience of one pro- 77 Kort WLAM de, Post Uiterweer HW, van Dijk FJH. Agree- gram. Am J Ind Med 1993;22:473–81. ment on medical ﬁtness for a job. Scand J Work Environ 103 Hathaway JA. Medical surveillance: extent and eVective- Health 1992;18:246–51. ness? J Occup Med 1993;35:698–700. 78 Whitaker S, Aw TC. Audit of pre-employment assessments 104 Van der Beek AJ, Frings-Dresen MHW,van Dijk FJH, et al. by occupational health departments in the National Health Priorities in occupational health research: a Delphi study in Service. Occup Med 1995;45:75–80. The Netherlands. Occup Environ Med 1997;54:504–10. 79 De Zotti R, Molinari S, Larese F, et al. Pre-employment 105 Cherry N. Evaluation of preventive measures. In: McDon- screening among trainee bakers. Occup Environ Med 1995; ald C. Epidemiology of work related diseases. London: BMJ 52:279–83. Publishing Group, 1995. 80 Milkovic-Kraus S, Macan J. Can pre-employment patch 106 Cockroft A, Edwards J, McCarthy P. The role of testing help to prevent occupational contact allergy? pre-employment allergy screening in animal workers. Eur J Contact Dermatitis 1996;35:226–8. Respir Dis 1982;62:42–3. 81 Udasin IG, Buckler G, Gochfeld M. Quality assurance 107 Kort W de, van Dijk F. Preventive eVectiveness of

audits of medical surveillance programs for hazardous pre-employment medical assessments. Occup Environ Med http://oem.bmj.com/ waste workers. J Occup Med 1991;33:1170–4. 1997;54:1–6. 82 Conway H, Simmons J, Talbert T. The Occupational Safety 108 Hulshof C, Remijn B, Leonhard N. Het periodiek bedrijf- and Health Administration’s 1990–1 survey of occupa- sgezondheidkundig onderzoek (PBGO), ervaringen van tional medical surveillance prevalence and type of current deelnemers (Periodic Occupational Health Survey, experi- practices. J Occup Med 1993;35:659–69. ences of participants). T Soc Gezondheidsz 1990;68:397– 83 Conway H, Simmons J, Talbert T. The purposes of occupa- 403. (In Dutch, English abstract.) tional medical surveillance in US industry and related 109 van der Weide WE, Verbeek JHAM, van Dijk FJH. A ran- health ﬁndings. J Occup Med 1993;35:670–86. domised trial of occupational rehabilitation for low-back 84 Broersen JPJ, Weel ANH, van Dijk FJH, et al. The atlas of pain. EYcacy on pain, functional disability, general health health and working conditions by occupation 2. A and sick leave. In: Weide WE van der. comparison with the “Atlas of health and working Quality of occupational rehabilitation for low-back pain [academic conditions in the construction industry”. on October 1, 2021 by guest. Protected copyright. Int Arch Occup thesis] Amsterdam: University of Amsterdam, 1998:85– Environ Health 1995;67:337–42. . 85 Kort WLAM de, Fransman LG, van Dijk FJH. Pre- 99. employment medical examinations in a large occupational 110 Hulshof CTJ, Braam ITJ, Verbeek JHAM, et al. Evaluation health service. Scand J Work Environ Health 1991;17:392–7. of an occupational health prevention program on whole- 86 Hessel PA, Zeiss E. Evaluation of the periodic examination body vibration in forklift truck drivers. In: Hulshof CTJ. in the South Africa mining industry. J Occup Med 1988;30: Prevention and control of adverse eVects of whole-body 580–6. vibration, an evaluation study in occupational health services 87 Rose G, Bengtsson C. Evaluation of a laboratory health [academic thesis]. Amsterdam: University of Amsterdam, examination programme in a Swedish industry (Volvo). 1998:153–79. Scand J Clin Lab Invest 1991;51:155–60. 111 Emmett EA. What is the strategic value of occupational 88 Mattila M. Improvement in the occupational health and environmental medicine? J Occup Environ Med program in a Finnish construction company by means of 1996;38:1124–34. systematic workplace investigation of job load and hazard 112 Van der Weide WE, Verbeek JHAM, van Dijk FJH, et al. analysis. Am J Ind Med 1989;15:61–72. Development and evaluation of a quality assessment 89 Peretz C, Goldberg Ph, Derazne E, et al. Assessment of the instrument for occupational physicians. Occup Environ Med extent of implementation of the recommendations made in 1998;55:375–82.