A Comparison of Usability Between Voting Methods
Total Page:16
File Type:pdf, Size:1020Kb
A Comparison of Usability Between Voting Methods Kristen K. Greene, Michael D. Byrne, and Sarah P. Everett Department of Psychology Rice University, MS-25 Houston, TX 77005 USA {kgreene, byrne, petersos}@rice.edu ABSTRACT From a human factors and usability standpoint, In order to know if new electronic voting voting is an arena where research could and systems truly represent a gain in usability, it is should have great practical impact. Voting is a necessary to have information about the challenging usability problem for two main usability of more traditional voting methods. reasons: characteristics of the user population The usability of three different voting methods and characteristics of the task itself. The user was evaluated using the metrics recommended population is extremely diverse, representing a by the National Institute of Standards and wide range of ages and varying greatly in Technology: efficiency, effectiveness, and education, socioeconomic status, familiarity satisfaction. Participants voted with two with technology, etc. In addition to this different types of paper-based ballots (one open- extremely diverse voter population, there are response ballot and one bubble ballot) and one subpopulations of users—such as those those mechanical lever machine. No significant who are visually impaired or physically differences in voting completion times or error disabled, or those who do not speak and/or read rates were found between voting methods. English—for whom voting has historically been However, large differences in satisfaction ratings particularly challenging. Also important from a indicated that participants were most satisfied usability standpoint are the numerous first time with the bubble ballot and least satisfied with the voters and the relatively infrequent nature of the lever machine. These data are an important step task—even the most zealous voter will not have in addressing the need for baseline usability the opportunity to vote more than a few times measures for existing voting systems. per year. This makes it especially crucial that voting systems are usable. One of the promises Author Keywords of electronic voting technology is that it may Usability, voting, paper ballots, mechanical lever help address this issue. machine INTRODUCTION However, there is currently no baseline usability Usability is a critical issue for voting systems; information for existing voting technologies to even if a voting system existed that was which new systems can be compared. The goal perfectly secure, accurate, reliable, and easily of the current study is to begin addressing this auditable, it could still fail to serve the needs of need for baseline data. It is the second in a series the voters and the democratic process. Voting of experiments we are conducting that compares systems must also be usable by all voters. A the usability of multiple voting technologies: truly usable voting technology should be a walk- paper ballots, optical scan ballots, mechanical up-and-use system, enabling even first time lever machines, punchcard voting systems, and voters to cast their votes successfully. It should DREs. be accessible for those voters with disabilities, and it should not disenfranchise particular The three usability measures used in the current groups of voters. The Help America Vote Act study are those metrics recommended by the (HAVA) of 2002 was a landmark piece of National Institute of Standards and Technology legislation in terms of calling public attention to (NIST) (Laskowswki, et al., 2004): efficiency, issues such as these. effectiveness, and satisfaction. These are also suggested by the International Organization for Standardization (ISO 9241-11, 1998). Endorsement by NIST is important, because this previous research by using the same three unlike other usability domains, decisions about measures to assess the usability of mechanical acceptability are made by governmental lever machines, in addition to measuring standards bodies. usability for two of the three previously-studied paper ballots. Efficiency is defined as the relationship between the level of effectiveness achieved and the The three types of paper ballots evaluated by amount of resources expended, and is usually Everett, Byrne, & Greene (2006) were bubble measured by time on task. Efficiency is an ballots, arrow ballots, and open response ballots objective usability metric that measures whether (see Figure 1 for examples of bubble and open a user’s goal was achieved without expending an response ballots). Every participant voted on all inordinate amount of resources (Industry three ballot types. Each participant saw either Usability Reporting Project, 2001). Did voting realistic candidate names or fictional candidate take an acceptable amount of time? Did voting names, but never both. Realistic names were take a reasonable amount of effort? those of people who had either announced their intention to run for the office in question, or who Effectiveness is the second objective usability seemed most likely to run based on current metric recommended by NIST, and is defined as politics. Fictional names were produced by a the relationship between the goal of using the random name generator. Participants were also system and the accuracy and completeness with provided with one of three different types of which this goal is achieved (Industry Usability information. They were given a slate that told Reporting Project, 2001). In the context of them exactly who to vote for, given a voter voting, accuracy means that a vote is cast for the guide, or were not given any information about candidate the voter intended to vote for, without candidates at all. error. Completeness refers to a vote actually being finished and officially cast. Effectiveness The three dependent measures used in the is usually measured by collecting error rates, but previous study were ballot completion time, may also be measured by completion rates and error rates, and satisfaction. Ballot completion number of assists. If a voter asks a poll worker time (efficiency) measured how long it took the for help, that would be considered an assist. participant to complete his or her vote. No significant differences were found between Satisfaction is the lone subjective usability ballot types (bubble, arrow, or open response) or metric recommended by NIST, and is defined as candidate types (realistic or fictional) on the the user’s subjective response to working with ballot completion time measure. Significant the system (Industry Usability Reporting differences were observed between information Project, 2001). Was the voter satisfied with his/ types; participants given a voter guide took her voting experience? Was the voter confident reliably more time to complete their ballots than that his/her vote was recorded? Satisfaction can those who received a slate or those who did not be measured via an external, standardized receive any information about candidates. instrument, usually in the form of a Likert scale questionnaire. The System Usability Scale, Errors (our measure of effectiveness) were (SUS), (Brooke, 1996) is a common recorded when a participant marked an incorrect standardized instrument that has been used and response in the slate condition or when there verified in a multitude of domains. was a discrepancy between their responses on the three ballot types. Overall, error rates were Everett, Byrne, & Greene (2006) used measures low but not negligible, with participants making of efficiency, effectiveness, and satisfaction to errors on slightly less than 4% of the races. compare the usability of three different types of paper ballots. The current study expands upon Satisfaction was measured by the participant’s participation and all other participants were responses to the SUS questionnaire. The bubble compensated $25 for the one-hour study. ballot produced significantly higher SUS scores than the other two ballots, while the open Design response and arrow ballot were not significantly A mixed design with one between-subjects different from one another. variable and one within-subjects variable was used. The between-subjects manipulation was Given that we found no significant differences information condition. This meant that each on ballot completion times between realistic or participant received only one of three types of fictional candidate names, for the current study information: slate, voter guide, or no only fictional names were used. This decision information. A slate was a paper that listed was made to prevent advertising campaigns and exactly who the participant was to vote for, as media coverage from differentially affecting well as whether to vote yes/no on the data from one study to the next. In addition, the propositions. The voter guide used was based on continued use of realistic candidate names those produced by the League of Women Voters. would have quickly rendered the study materials It was left completely up to the participant obsolete, making comparisons between studies whether she/he chose to actually read and use with new materials difficult. Using fictitious the voter guide. In the control condition, names circumvents both these issues for the participants were not given any information current study and future research. Since no about the candidates or propositions. The within- significant differences were found between subjects variable was voting method. This meant ballot completion times for the three paper ballot that every participant voted using each of the types in our