DOCSLIB.ORG

Report Template

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Report Template Evaluation Design/Planning and Methodology for the NIH Web Site—Phase I Final Report March 31, 2000 Abridged Version QRC DIVISION OF MACRO INTERNATIONAL INC. 7315 WISCONSIN AVENUE • SUITE 400W • BETHESDA, MD 2081 ACKNOWLEDGMENTS This project, Evaluation Design/Planning and Methodology for the NIH Web Site—Phase I, was conducted by QRC Division of Macro International Inc. (formerly Quantum Research Corporation) for the NIH Office of Communication and Public Liaison and the Office of the Deputy Chief Information Officer as Task Order 10 under NIH Contract N01-OD-7-2114. Macro International Inc. and UserWorks, Inc. participated as subcontractors. Macro provided valuable expertise in content analysis and the customer service methodologies, including the pilot online focus group. UserWorks conducted the heuristic review and pilot usability tests of the NIH Web site. We also wish to thank those individuals who participated in the various methodology pilot tests. QRC wishes to acknowledge the valuable assistance and support of Dennis Rodrigues, NIH OCPL, and Anna Perrone, NIH/ODCIO, who served as Task Leaders on this project. The Task Leaders provided insight into the early development of the NIH Web site, assisted with selection of pilot test respondents for the pilot surveys, and offered useful suggestions and comments throughout the project. We hope NIH and the Institutes and Centers will find this Evaluation and Performance Measures Toolkit useful as they assess their respective Web sites. Mary V. Look Project Director QRC i Division of Macro International Inc. CONTENTS Acknowledgments........................................................................................................i 1. Executive Summary..............................................................................................1 2. Introduction...........................................................................................................3 3. Investigative Methodologies.................................................................................5 3.1. Document Analysis......................................................................................5 3.2. Literature and Web Site Reviews ................................................................5 3.3. Site Mapping................................................................................................6 3.4. Webmaster and Staff Survey .......................................................................7 3.5. Expert Critique.............................................................................................8 3.6. Summary of Investigative Methodologies.................................................10 4. User-Centered Methodologies ............................................................................11 4.1. Usability Tests ...........................................................................................11 4.2. Transaction Log Analysis ..........................................................................13 4.3. Email Content Analysis .............................................................................13 4.4. Customer Satisfaction Focus Groups.........................................................15 4.5. Customer Satisfaction Questionnaires.......................................................16 4.6. Summary of User-Centered Methodologies ..............................................17 4.7. Other Emerging Methodologies.................................................................18 5. Performance Measurement .................................................................................20 5.1. Extensiveness.............................................................................................20 5.2. Customer Service.......................................................................................20 5.3. Effectiveness..............................................................................................21 5.4. Impact ........................................................................................................21 6. Conclusion ..........................................................................................................23 APPENDICES Appendix A-1. Literature and Web Site Reviews ....................................................25 Appendix A-2. Sample Survey of Webmasters ........................................................38 Appendix A-3. Heuristic Guidelines for Expert Critique of a Web Site ..................58 Appendix A-4. Web Site Usability Test Plan ...........................................................66 Appendix A-5. Usability Pilot Test Documents .......................................................68 Attachment A: Screening Questionnaire ............................................................69 Attachment B: Demographics Questionnaire .....................................................71 Attachment C: Informed Consent / Video Release Form ...................................73 Attachment D: Facilitator’s Guide and Final Questionnaire ..............................74 Appendix A-6. Email Content Analysis Pilot Test...................................................82 Appendix A-7. Summary of Process for Conducting an Online Focus Group........89 Attachment A: Focus Group Recruitment Screener ...........................................97 Attachment B: Follow-up Email Letter ............................................................100 Attachment C: Reminder Email Letter .............................................................102 Attachment D: Focus Group Moderator’s Guide..............................................104 Attachment E: Focus Group Participant Survey...............................................109 QRC ii Division of Macro International Inc. Appendix A-8. Customer Satisfaction Survey.......................................................111 Appendix A-9. Implementation Details of Web Site Evaluation Methodologies .117 Appendix A-10. Performance Measures................................................................120 QRC iii Division of Macro International Inc. 1. EXECUTIVE SUMMARY Effective health information dissemination strategies and services are essential to extend the value of the knowledge resulting from biomedical research. NIH’s ability to fulfill its mission to discover and disseminate new knowledge leading to improved health for all Americans has been strengthened by the rapid development and accessibility of the World Wide Web. The Web has become an essential tool supporting NIH’s mission to disseminate health information to the public. That tool has matured to the point that it is now advisable to examine and evaluate the current state of the NIH Web site, identify its most important user populations, and learn how the Web site can be used most effectively and efficiently to meet the various needs of different user populations. Web site evaluation can help NIH and the Institutes and Centers (ICs) be responsive to legislative mandates and Executive orders. It can also provide information to help manage information technology resources and respond to Government Performance and Results Act (GPRA) requirements in performance measures, goal setting and goal measuring, and customer service. Evaluation can also help serve NIH and IC missions by ensuring access of information to all segments of the Web-enabled public including those who are accessibility-challenged. The Evaluation and Performance Measures Toolkit described in this document can be used to assess the effectiveness of any Web site. The specific methodologies used for an evaluation depend on what is already known about the site and its users, and the resources available for the evaluation. The methodologies can be summarized as follows according to the sequence in which they are likely to be used and what is learned from each: • Background information • Document analysis • Literature review and Web site reviews • Web site information • Site mapping • Webmaster and staff questionnaires • Usability and accessibility issues • Heuristic review/expert panel • Usability testing • User information • Web server log analysis • Email content analysis • Customer service • Customer satisfaction questionnaire • Customer satisfaction focus group The first steps are collecting background information and information about the characteristics of the site to be evaluated through document analysis, literature reviews, site mapping, and Webmaster questionnaires. To look at customer satisfaction, usability, and accessibility issues QRC 1 Division of Macro International Inc. and conduct focus groups, more specialized skills and knowledge are required. Each of these methodologies is more labor-intensive than the more automated tasks, but each provides high quality information and the evaluator has considerable control over the type of information collected. The overall evaluation design focuses on the experience of the user and is organized around performance measures that can be used in preparing responses to Federal legislation and Executive Orders. Together, the evaluation methodologies are complementary in their focus on specific performance measures—extensiveness, customer service, effectiveness, and impact— that can be used to assess the Web site. QRC 2 Division of Macro International Inc. 2. INTRODUCTION This document describes the final evaluation design, methodologies, performance metrics, and data collection instruments that have been developed and pilot tested for assessing a
Load more

Recommended publications
  • PANTONE® Colorwebtm 1.0 COLORWEB USER MANUAL
    User Manual PANTONE® ColorWebTM 1.0 COLORWEB USER MANUAL Copyright Pantone, Inc., 1996. All rights reserved. PANTONE® Computer Video simulations used in this product may not match PANTONE®-identified solid color standards. Use current PANTONE Color Reference Manuals for accurate color. All trademarks noted herein are either the property of Pantone, Inc. or their respective companies. PANTONE® ColorWeb™, ColorWeb™, PANTONE Internet Color System™, PANTONE® ColorDrive®, PANTONE Hexachrome™† and Hexachrome™ are trademarks of Pantone, Inc. Macintosh, Power Macintosh, System 7.xx, Macintosh Drag and Drop, Apple ColorSync and Apple Script are registered trademarks of Apple® Computer, Inc. Adobe Photoshop™ and PageMill™ are trademarks of Adobe Systems Incorporated. Claris Home Page is a trademark of Claris Corporation. Netscape Navigator™ Gold is a trademark of Netscape Communications Corporation. HoTMetaL™ is a trademark of SoftQuad Inc. All other products are trademarks or registered trademarks of their respective owners. † Six-color Process System Patent Pending - Pantone, Inc.. PANTONE ColorWeb Team: Mark Astmann, Al DiBernardo, Ithran Einhorn, Andrew Hatkoff, Richard Herbert, Rosemary Morretta, Stuart Naftel, Diane O’Brien, Ben Sanders, Linda Schulte, Ira Simon and Annmarie Williams. 1 COLORWEB™ USER MANUAL WELCOME Thank you for purchasing PANTONE® ColorWeb™. ColorWeb™ contains all of the resources nec- essary to ensure accurate, cross-platform, non-dithered and non-substituting colors when used in the creation of Web pages. ColorWeb works with any Web authoring program and makes it easy to choose colors for use within the design of Web pages. By using colors from the PANTONE Internet Color System™ (PICS) color palette, Web authors can be sure their page designs have rich, crisp, solid colors, no matter which computer platform these pages are created on or viewed.
    [Show full text]
  • The Internet As a Course Support Tool in Pharmaceutical Sciences Education: a Primer
    The Internet As a Course Support Tool in Pharmaceutical Sciences Education: A Primer David J. McCaffrey III Alicia S. Bouldin Kathryn F. Gates SUMMARY. The move toward increasing student-centered learning efforts and improving students’ cognitive interaction with the course content is becoming increasingly evident in U.S. higher education. Pharmacy education is not an exception. However, such interactive learning may require consideration of alternative ways to deliver course content, especially for the large lecture class. For these and a variety of other reasons, pharmaceutical sciences educators have started looking to the Internet to support or supplant traditional instructional methodol- ogies. The integration of Internet-based elements into pharmaceutical science course offerings familiarizes students with technologies and behaviors that are likely to persist and improve throughout their ca- David J. McCaffrey III, Ph.D., R.Ph., is Assistant Professor of Pharmacy Admin- istration and Research Assistant Professor, Research Institute of Pharmaceutical Sciences, School of Pharmacy, University of Mississippi, University, MS 38677. Alicia S. Bouldin, Ph.D., R.Ph., is Research Assistant Professor of Instructional Assessment and Advancement, Research Institute of Pharmaceutical Sciences, School of Pharmacy, University of Mississippi. Kathryn F. Gates, Ph.D., is Director of Support Services and Research Assistant Professor, Office of Information Technology, University of Mississippi. The authors acknowledge the efforts of the anonymous reviewers whose com- ments served to improve this manuscript. [Haworth co-indexing entry note]: ‘‘The Internet As a Course Support Tool in Pharmaceutical Sciences Education: A Primer.’’ McCaffrey, David J. III, Alicia S. Bouldin, and Kathryn F. Gates. Co-published simultaneously in Journal of Pharmacy Teaching (Pharmaceutical Products Press, an imprint of The Ha- worth Press, Inc.) Vol.
    [Show full text]
  • Chapter 2 Creating Web Pages: XHTML
    Chapter 2 Creating Web Pages: XHTML A Web page is a document, identi¯ed by an URL, that can be retrieved on the Web. Typically, a Web page is written in HTML, the Hypertext Markup Language. When a Web browser receives an HTML document, it can format and render the content for viewing, listening, or printing. The user can also follow embedded hyperlinks, or simply links, to visit other Web pages. HTML enables you to structure and organize text, graphics, pictures, sound, video, and other media content for processing and display by browsers. HTML supports headings, paragraphs, lists, tables, links, images, forms, frames, and so on. The major part of a website is usually a set of HTML documents. Learning and understanding HTML is fundamental to Web Design and Programming. To create HTML ¯les you may use any standard text editor such as vi, emacs, word (MS/Windows), and SimpleText (Mac/OS). Specialized tools for creating and editing HTML pages are also widely available. After creating an HTML ¯le and saving it in a ¯le, you can open that ¯le (by double-clicking the ¯le or using the browser File>Open File menu option) and look at the page. XHTML (Extensible Hypertext Markup Language) is a modern version of HTML that is recommended for creating new Web pages. Having evolved from version 2.0 to 4.01, HTML now gets reformulated in XML (Extensible Markup Language) and becomes XHTML 1.0. 41 42 CHAPTER 2. CREATING WEB PAGES: XHTML XML conforming documents follow strict XML syntax rules and therefore become easily manipulated by programs of all kinds{a great advantage.
    [Show full text]
  • Web Site Authoring Tools
    Web site 15 authoring tools Despite their lofty educational pretensions, many e-learning courses at their heart are just special-purpose Web sites. Many are created by Web site authoring tools. Web site authoring tools build and link individual Web pages to create a Web site. They are the successors to the simple HTML page editors of several years ago. Most sport sophisticated capabilities to create and maintain complex sites of thousands of pages. Some let you create interactive animations and database connections without any programming. Joining these veteran tools is a relatively new type of Web authoring tool called a blog. Blogs make creating ongoing Web journals simple enough for anyone. On our tools framework, we put Web site authoring tools in the Create column, spanning both the Page and Lesson rows. They peek up into the course level, but lack the sophisticated collaboration and tracking capabilities needed to completely cover this square. However, with the database connections built into some of these tools, you can, with enough hard work and cleverness, extend the scope of these tools to cover courses and curricula. Web site authoring tools do not work on their own. Their purpose is to create Web sites that are, in turn, offered by Web servers. To create these Web pages, they rely on media editors for the graphics, animations, and other media that appear. Sometimes they are used in conjunction with course authoring tools (and the course authoring capabilities of some offering tools) to prepare pages more efficiently than the Web site authoring tool can. 305 306 W Web site authoring tools W E-learning Tools and Technologies WHY CREATE E-LEARNING WITH WEB SITE TOOLS? Your first reaction to this chapter might well have been, “This book is supposed to be about e-learning technology.
    [Show full text]
  • Working with the Nifti Data Standard in R
    Working with the NIfTI Data Standard in R Brandon Whitcher Volker J. Schmid Pfizer Worldwide R&D Ludwig-Maximilians Universit¨at Munchen¨ Andrew Thornton Cardiff University Abstract The package oro.nifti facilitates the interaction with and manipulation of medical imaging data that conform to the ANALYZE, NIfTI and AFNI formats. The S4 class framework is used to develop basic ANALYZE and NIfTI classes, where NIfTI extensions may be used to extend the fixed-byte NIfTI header. One example of this, that has been implemented, is an XML-based “audit trail” tracking the history of operations applied to a data set. The conversion from DICOM to ANALYZE/NIfTI is straightforward using the capabilities of oro.dicom. The S4 classes have been developed to provide a user-friendly interface to the ANALYZE/NIfTI data formats; allowing easy data input, data output, image processing and visualization. Keywords: export, imaging, import, medical, visualization. 1. Introduction Medical imaging is well established in both the clinical and research areas with numerous equipment manufacturers supplying a wide variety of modalities. The ANALYZE format was developed at the Mayo Clinic (in the 1990s) to store multidimensional biomedical images. It is fundamentally different from the DICOM standard since it groups all images from a single acquisition (typically three- or four-dimensional) into a pair of binary files, one containing header information and one containing the image information. The DICOM standard groups the header and image information, typically a single two-dimensional image, into a single file. Hence, a single acquisition will contain multiple DICOM files but only a pair of ANALYZE files.
    [Show full text]
  • CS Guide a Guide for the Communications Services Officer
    U.S. COAST GUARD AUXILIARY DEPARTMENT OF INFORMATION TECHNOLOGY CS Guide A Guide for the Communications Services Officer January 2009, Version 2.0 Page 1 CS Guide This page intentionally blank Page 2 CS Guide Table of Contents (Click on any subject within the Table of Contents to hyperlink to that section) Table of Contents Background information ............................................................................................... 10 Privacy Statement ......................................................................................................... 10 Notice/Disclaimer ......................................................................................................... 10 Introduction .................................................................................................................. 12 I-Department Mission & Vision .................................................................................... 12 CS Guide Change Page .................................................................................................. 13 How to use this Guide ................................................................................................... 14 For the new CS Officer .................................................................................................. 16 Section 1. CS Program Overview ............................................................. 20 1.1 The Communications Services (CS) Program .......................................................... 20 1.1.1 Purpose ...........................................................................................................
    [Show full text]
  • Publishing to the World Wide Web
    PUBLISHING TO THE WORLD WIDE WEB Copyright © 1999 The President and Fellows of Harvard College All Rights Reserved TABLE OF CONTENTS PUBLISHING TO THE WORLD WIDE WEB ................................................................... 1 Direct HTML Editing vs. Web Design Tools ................................................................ 1 Creating a Web Directory In Your Unix Account ........................................................... 1 Using Pico ................................................................................................................. 2 Setting File Permissions ............................................................................................ 2 INTRODUCTION TO HTML........................................................................................ 3 THE BASIC WEB PAGE ......................................................................................... 4 Useful HTML tags ....................................................................................................... 4 Lists ........................................................................................................................... 5 Using Tags with Parameters ....................................................................................... 5 URLs .......................................................................................................................... 6 The Anchor Tag .......................................................................................................... 6 Mailto: Links
    [Show full text]
  • Summary of UNIX Commands Furnishing, Performance, Or the Use of These Previewers Commands Or the Associated Descriptions Available on Most UNIX Systems
    S u mmary of UNIX Commands- version 3.2 Disclaimer 8. Usnet news 9. File transfer and remote access Summary of UNIX The author and publisher make no warranty of any 10. X window kind, expressed or implied, including the warranties of 11. Graph, Plot, Image processing tools commands merchantability or fitness for a particular purpose, with regard to the use of commands contained in this 12. Information systems 1994,1995,1996 Budi Rahardjo reference card. This reference card is provided “as is”. 13. Networking programs <[email protected]> The author and publisher shall not be liable for 14. Programming tools damage in connection with, or arising out of the 15. Text processors, typesetters, and This is a summary of UNIX commands furnishing, performance, or the use of these previewers commands or the associated descriptions available on most UNIX systems. 16. Wordprocessors Depending on the configuration, some of 17. Spreadsheets the commands may be unavailable on your Conventions 18. Databases site. These commands may be a 1. Directory and file commands commercial program, freeware or public bold domain program that must be installed represents program name separately, or probably just not in your bdf search path. Check your local dirname display disk space (HP-UX). See documentation or manual pages for more represents directory name as an also df. details (e.g. man programname). argument cat filename This reference card, obviously, cannot filename display the content of file filename describe all UNIX commands in details, but represents file name as an instead I picked commands that are useful argument cd [dirname] and interesting from a user’s point of view.
    [Show full text]
  • Overview and Preview Tools for Navigating the World-Wide Web SILS Technical Report TR-1999-03
    Overview and Preview Tools For Navigating the World-Wide Web SILS Technical Report TR-1999-03 By: Benjamin D. Brunk July, 1999 Benjamin D. Brunk Animated Web Overviews CONTEXT AND PROBLEM STATEMENT ......................................................................................2 RELATED WORK ................................................................................................................................4 WHAT IS A “WEB SITE?”........................................................................................................................4 SITEMAPS ...............................................................................................................................................5 ANIMATED PREVIEWS ...................................................................................................................26 FIRST PROTOTYPE ...............................................................................................................................27 FEASIBILITY STUDY..............................................................................................................................28 SECOND PROTOTYPE (FUTURE WORK) ...............................................................................................30 CONCLUSION ....................................................................................................................................31 ACKNOWLEDGMENTS ...................................................................................................................31
    [Show full text]
  • Miklos Systems, Inc. 10300 Eaton Place, Suite 300 Fairfax, Virginia 22030 Phone: 703-383-1633 Fax: 888-805-1064 Email: [email protected]
    AUTHORIZED INFORMATION TECHNOLOGY SCHEDULE PRICELIST GENERAL PURPOSE COMMERCIAL INFORMATION TECHNOLOGY EQUIPMENT, SOFTWARE AND SERVICES SPECIAL ITEM NUMBER 132-51 - INFORMATION TECHNOLOGY (IT) PROFESSIONAL SERVICES FPDS Code D302 IT Systems Development Services FPDS Code D306 IT Systems Analysis Services FPDS Code D307 Automated Information Systems Design and Integration Services FPDS Code D308 Programming Services FPDS Code D310 IT Backup and Security Services FPDS Code D311 IT Data Conversion Services FPDS Code D316 IT Network Management Services FPDS Code D399 Other Information Technology Services, Not Elsewhere Classified SPECIAL ITEM NUMBER 70 500 – ORDER-LEVEL MATERIALS (OLMs) Note: Contractor has been awarded under the Cooperative Purchasing and Disaster Recovery Programs. Miklos Systems, Inc. 10300 Eaton Place, Suite 300 Fairfax, Virginia 22030 Phone: 703-383-1633 Fax: 888-805-1064 Email: [email protected] http://www.miklos.com Contract Number: GS-35F-0305M Period Covered by Contract: February 28, 2017 – February 27, 2022 Price list current through modification #PA-0041, dated March 23, 2018 General Services Administration Federal Acquisition Service Products and ordering information in this Authorized FAS Information Technology Schedule Pricelist are also available on the GSA Advantage! System. Agencies can browse GSA Advantage! by accessing the Federal Acquisition Service’s Home Page via the Internet at http://www.gsa.gov/fas INFORMATION FOR ORDERING ACTIVITIES 1a. Table of awarded the special item number with appropriate cross-reference to item descriptions and awarded price. Special Item No.132-51 Information Technology Professional Services Special Item No. 70 500 Order-Level Materials (OLMs) 1b. Identification of the lowest priced model number and lowest unit price for that model for each special item number awarded in the contract.
    [Show full text]
  • The Intranet Web: Short on Distance Long on Education
    Session 2358 The Intranet Web: Short on Distance - Long on Education Gregory M. Dick University of Pittsburgh at Johnstown Abstract The World Wide Web and Electronic Mail List Servers are Internet communications tools available to educators at most colleges and universities. This paper is a case study of one application of these tools to the education process; it represents the first step in the process of learning how to prepare effective, Web compatible multimedia presentations. Introduction The Internet is a global data communications infrastructure. The use of Internet functionality within an organization is referred to as the Intranet. Intranet communications are often selective (i.e., access to information is only granted to those within the organization). The principal goal of this project is communications among class participants; however, no access filter has been applied. The information referenced is globally available and offered as an example to others who may wish to incorporate these techniques into their own classes. Electronic mail and the World Wide Web are communications tools enabled by Internet - Intranet technology which are available to support the distribution of course materials. The World Wide Web is a platform independent communications system layered on top of the Internet infrastructure. The Web is enabled by the Hyper Text Transfer Protocol (HTTP) and related technologies. Among its notable characteristics is its ability to easily link together several related documents, which may reside at sites that are physically distant. A List Server works in cooperation with an electronic post office. It provides the capability of creating a globally accessible mailing list to which interested parties may subscribe without requiring any action on the part of the list owner.
    [Show full text]
  • A. the CD-ROM
    A. The CD-ROM ~ll complete!' said the Toad triumphantly, pulling open a locker. 'You see ... everything you can possibly want...you'll find that nothing whatever has been forgotten ... ' Kenneth Grahame, The Wind in the Willows The disk that accompanies this book contains software and documenta­ tion for systems designed for use with SGML, and also for some which are more general-purpose but have some specific SGML applications. Although I have tried to make the documentation as obvious as pos­ sible, I have avoided starting on the bottom rung of trying to teach the use of the keyboard, the mouse, and the operating system (see the assumptions alluded to earlier, which are in section 1.3). Not all the programs described in the book are on the disk, as many of them are regular commercial products for which no free or demo version exists. By the same token, not all the programs on the disk are necessarily described in the book, as I have tried to make the disk as up to date as possible, and the lead-time for printing is longer than that for making the disk. If you don't have access to MS-Windows to use the Synex ViewPort browser provided (see next section), you can read the same information by using your regular Web browser to open the INDEX. HTM file on the CD-ROM. The disk is written in ISO 9660 format, so that it is usable on Macs, PCs, and Unix systems. 384 Peter Flynn A.1. BrOwsing the CD-ROM: using SyneX/lnso ViewPort Synex, makers of the ViewPort SGML browser engine for MS-Windows, have kindly provided a copy of their demonstration browser for you to use with this CD-ROM.
    [Show full text]