Evaluating the Effectiveness of Certain Metrics in Measuring the Quality of End User Documentation
Total Page：16
File Type：pdf, Size：1020Kb
Portland State University PDXScholar Dissertations and Theses Dissertations and Theses 1993 Evaluating the Effectiveness of Certain Metrics in Measuring the Quality of End User Documentation Ronald Morrison Portland State University Follow this and additional works at: https://pdxscholar.library.pdx.edu/open_access_etds Part of the Other Computer Sciences Commons Let us know how access to this document benefits ou.y Recommended Citation Morrison, Ronald, "Evaluating the Effectiveness of Certain Metrics in Measuring the Quality of End User Documentation" (1993). Dissertations and Theses. Paper 1326. https://doi.org/10.15760/etd.1325 This Dissertation is brought to you for free and open access. It has been accepted for inclusion in Dissertations and Theses by an authorized administrator of PDXScholar. Please contact us if we can make this document more accessible: [email protected]. EVALUATING THE EFFECTIVENESS OF CERTAIN METRICS IN MEASURING THE QUALITY OF END USER DOCUMENTATION by RONALD MORRISON A dissertation submitted in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY in SYSTEMS SCIENCE: BUSINESS ADMINISTRATION Portland State University © 1993 TO THE OFFICE OF GRADUATE STUDIES The Members of the Committee approve the dissertation of Ronald Morrison presented August 3, 1993. Da APPROVED: Beatrice T. Oshika, Director, System Science Ph.D. Program Roy or Graduate Studies and Research AN ABSTRACT OF THE DISSERTATION OF Ronald Morrison for the Doctor of Philosophy in Systems Science: Business Administration presented August 3, 1993. Title: Evaluating the Effectiveness of Certain Metrics in Measuring the Quality of End User Documentation APPROVED BY THE MEMEBERS OF THE DISSERTATION COMMITTEE Traditional methods of evaluating quality in computer end user documentation have been subjective in nature, and have not been widely 2 used in practice. Attempts to quantify quality and more narrowly define the essential features of quality have been limited -- leaving the issue of quality largely up to the writer of the user manual. Quantifiable measures from the literature, especially Velotta (1992) and Brockman (1990), have been assembled into a set of uniformly weighted metrics for the measurement of document. quality. This measure has been applied to the end user documentation of eighty-two personal computer packages. End user documentation is defined in terms of paper documents only. The research examined only those manuals that were titled lI user gUide,1I IItraining manual,lI IItutorial,lI or similar title. The research examined six categories of software: applications, graphics, utilities, spreadsheets, databases, and word processing. - Following the recommendation of Duffy (1985), a panel of experts was assembled and asked to evaluate several of the 82 end user manuals in order to determine what correlation exists between the set of metrics and the subjective opinion of experts. The eighty-two documents in the sample were scored by the metrics using a convenient random sampling technique. This technique was selected based the consistency of the material in commercial software manuals and the methods ofVeloUa (1992). Data from the metrics suggest that there is little correlation between quality, category, price, page length, version number, and experience. On a scale of 0.0 to 1.0, the minim'.lm total score from the metrics was .2; the maximum score .83; the mean total score was. 70; the median .697 with a standard deviation of .093. The distribution is slightly skewed and leptokurtic 3 (steeper than a normal curve). The metrics further suggest a declining score as the integration of sentences into chapters and chapters into the document progresses. Of the metrics two consistently had lower scores: those relating to the transition between sections of the document; and the reference tools provided. Though not conclusive, the analysis of data from the panel of experts compared with the model results suggests only a moderate correlation. However, by varying the weighting scheme, it is POSSible to improve model performance - essentially by "tuning" the model to match the sample data from the panelists. Further research would be required to verify if these weights have more global application. ACKNOWLEDGMENTS I wish to thank the members of my dissertation committee (Henry Crockett, David Scott, Jack Taylor, Wayne Wakeland, and Lee Haggerty) for their support and their uncompensated time in helping me through my written examinations, oral exatJ1inations, and through the dissertation process. Without their support and time there would be no dissertation. I wish to thank the faculty and staff of the System SCience Department for their efforts. A special acknowledgment is appropriate to the software development firms listed in the Appendix, who provided evaluation copies of software for use in the research. Without their support of higher education and this dissertation in particular, universities would be unable to prOvide graduate level research to create a better trained workforce. I would hope everyone who reads this document will consider purchaSing their products. Finally, I would like to expressly thank the members of my family who supported me both finanCially and morally throughout my graduate study process, which went well beyond the single Masters degree started in 1988. Without their express approval and support none of this would have been possible given the long distances I had to commute and the large number of hours I W"'.r'· required to spend away from home. TABLE OF CONTENTS PAGE ACKNOWLEDGMENTS.................................... ill LIST OF TABLES ....................................... Viii LIST OF FIGURES ....................................... Ix CHAPTER I INTRODUCTION ................................. .1 Plan and Definltions. .1 General Definition of QUality. .1 Definltion of End User Manuals.................. 2 Two Critical Assumptions. .5 II REVIEW OF THE LITERATURE .......................6 Computer Science Literature: The Human- Computer Interface ..............................6 User Manuals: Experimental Studies ...............7 Computer Sctence Graphics Literature. .10 Technical Writing Literature and User Documentation ................................11 Traditional Neglect. .11 Documentation Quality .......................12 Graphic Elements in Documents .................13 Literature Regarding Measurement of Document Quality ......................................14 Measurement............................... 14 Readability. .'. .16 Reference Aids Metrics . .20 Graphical Metrics ............................22 Typographic Metrics. .29 III THE PROPOSED MODEL ..........................32 Model Structure. .32 v Document (Levell). ..........................34 Chapter and Paragraph (Level 2) .................35 Sentence (Level 3) ............................37 Final Score. .40 Explanation of Example .......................40 IV RESEARCH METHODOLOGY .......................44 The Hypothesis and Overall Research Process ..........45 The Research Hypothesis .....................45 The Research Process .........................45 Sampling Issues and the Scoring Process .............47 Selection of the Sample from a Given Document .................................48 Selection of the Evaluation Platform ..............49 Selection of the Specific Software Packages .........53 Processing the User Manuals ...................53 Model Validity .................................55 General Approach ...........................56 Panel of Experts .............................56 Initial Survey of Panel. ........................57 Scoring of Documents by the Panel ...............58 Comparison of the Panel Scores to the Model ........59 Evaluation of the Consistency of the Model and the Panel ..................................59 Testing of the Research Hypothesis ...............60 V FINDINGS .....................................62 Analysis of the Model ............................62 Data from the Model. .........................63 Levels and Categories in the Model . .66 Internal Consistency of Samples .................72 Statistical Analysis of the Model. ................74 Analysis of the }-'anelist Data ......................82 Selection of the Panel of Experts .................82 Scorlng by the Panel. .........................82 Data from the Panel of Experts . .83 Internal Consistency of the Panel ................85 Statistical Analysis of the Panel of Experts .........85 Analysis of the Model and Panelist Data ..............87 Statistical Tests of Hypothesis ...................88 vi Statistical Analysis of Combined Data. .90 Factor Analysis . .93 Observations ..................................98 Obtaining Evaluation Copies ....................98 Volume of Material. ........99 Timing.................................... 99 Formats. .100 Writing Styles. .100 Graphics . .101 Presentation .............................. 101 Style .................................... 102 Training Aids. .102 Reference Aids. .103 Icons .................................... 105 Product Depth. .106 Completeness.............................. 107 Software Support ........................... 107 Descriptive Statistics . .108 VI CONCLUSIONS AND IMPLICATIONS .................109 Conclusions . .109 Potential Relationships That are More Complex 110 Intervening Variables ........................ 110 Limiting Values ............................ 110 IInpl1cations of the Research...................... 111 Indications for Future Research