A STUDY OF VOTE VERIFICATION TECHNOLOGY CONDUCTED
FOR THE
PART II: USABILITY STUDY
January 2006
The Center for American Politics and Citizenship
1108 Tawes Hall
301-405-4123
The Human-Computer Interaction Lab
301-405-2764
Executive Summary
The
The major findings from the expert review by human-computer interaction experts are:
· There was a perceived trade-off between usability and security. In all cases, the verification system appeared to reduce the usability of the voting process compared to the Diebold AccuVote-TS, which had no verification unit.
· The Diebold AccuVote-TSx with the AccuView Printer Module (paper printout, referred to as AccuView Printer) was rated most favorably. However, suggestions were made for improvement and questions were raised about the paper record’s utility when used for a long ballot.
· Privacy concerns were raised about each of the four vote verification systems.
The major findings from the field test involving more
than 800 Marylanders are:
· All of the systems were viewed favorably, including the Diebold AccuVote-TS with no verification unit.
· The Diebold with AccuView Printer was rated the most favorably in terms of voter satisfaction, but not substantially better than the AccuVote-TS with no verification unit or the VoteHere Sentinel.
· The MIT (audio) system was found to be distracting and it failed to generate as much confidence as other systems. It also was criticized by some users because of sanitary concerns related to the repeated use of the same headset.
· Participants needed the least amount of help when using the Diebold AccuVote-TS system (no verification unit). The Diebold with AccuView Printer system (paper trail) came next. Voters received more help using the VoteHere (internet or telephone), MIT (audio), and Scytl (monitor) systems.
The major findings concerned with election administration
are:
· Adding any of the four verification systems greatly increased the complexity of administering an election.
· The paper spool in the Diebold AccuView Printer had to be changed frequently, and changing it was fairly complex.
· It was difficult and time consuming to set up the Scytl system.
· The Scytl, MIT, and Diebold AccuVote-TS with no verification unit were out of commission for some portions of the study (but not enough to affect the results).
·
Diebold provided outstanding response to service
calls. Scytl (based in
Recommendations
·
On the basis of usability and some
administrative considerations we cannot recommend that the State of
· We recommend that the voter interface of AccuVote-TS (with no printer unit) be modified to incorporate some of the improvements made to the interface of the AccuVote-TSx with the AccuView Printer system.
· The AccuVote-TS with no verification unit became inoperative while an individual was voting under normal circumstances. This had a direct impact on the usability of the system and caused concern among voters. An explanation was provided but it was beyond the scope of this study to confirm it. We recommend this situation be addressed.
Introduction
To assist the Maryland State Board of Elections (SBE) in its
Study of Independent Verification Systems, the
All of the systems but the Diebold AccuVote-TSx with the
AccuView Printer Module were prototypes in various stages of development. The
Diebold AccuVote-TS, a direct recording electronic (DRE) system, which has no
vote verification unit and is presently used in the State of
To meet the requirements of democratic elections, voting systems must enable voters to register their voting intentions both accurately and confidently, record votes correctly, count votes accurately, and prevent coercion or vote tampering. Usability research is concerned with the first three criteria. Security studies are concerned with the last two.
Usability studies typically include assessments of “learnability,” efficiency, “memorability,” errors, accuracy, speed, and user satisfaction. Here, we focus on the public opinion component of usability: user satisfaction. Understanding how voters react to technologies—including ease of use, confidence that one’s vote is recorded accurately, and the need to request help when casting a vote—is crucial for an evaluation of the effectiveness of current electoral processes and new processes under consideration.
Usability research is
particularly relevant for voting systems because voting is an infrequent,
unspecialized practice in which virtually all
Usability concerns about voting systems are just as important as security issues. A voting system can be absolutely perfect in terms of its ability to count votes and protect them from tampering, but such considerations decline in importance when citizens are not confident their votes were accurately recorded or have serious concerns about their experience at the polls. This situation is analogous to investing millions of dollars to protect a bank vault at the same time that customers have little confidence about what has been deposited into their individual accounts.
This report has six parts: I) an overview of the systems tested, II) a review of the vote verification systems by experts in human-computer interaction, III) field tests involving Maryland voters, IV) a discussion of issues relevant to officials who would be involved in administering elections using these systems, V) recommendations, and VI) appendices that include information about the methodology used in the study. Parts II and III provide comparative assessments of the different vote verification systems, as well as the voting system that has no vote verification unit. Part IV covers the following practical concerns: the integration of vote verification systems to other voting equipment, experiences from the field test relevant for Election Day activities, and support provided by vote system manufacturers.
I. Overview of Vote Verification Systems
A brief overview of the interfaces of the vote verification systems follows.
Diebold AccuVote-TSx with AccuView Printer Module
This system combines a paper printout that lists the
candidates’ names with a touchscreen voting system. After voters complete their
selections and submit their votes, they are given the opportunity to review
their selections. They can lift the cover to the printer to see a printout of
their selections behind a sheet of glass. An attached magnifying glass is
available for closer viewing. Once a person looks at the paper printout they
have the option of casting their ballot or making changes. If the person
decides to cast the ballot, the paper sheet is rolled into the machine. If the
person wishes to revise the ballot, once they are done resubmitting their votes
the new set of selections appears on the paper sheet and the old set is rolled
into the machine. At this point, a person can cast the ballot.
The paper rolls can be used for election auditing and recounts. Because they are kept intact, the order in which individuals voted is kept intact, creating possible violations of secret voting.
The system we tested is presently commercially available and in use at some polling sites around the country. It uses Diebold’s newer interface, and so the details of the voter’s interaction were a little different than the other four systems.
VoteHere Sentinel
This system uses a mathematical cryptographic-based technique for voter verification. The system has an independent computer and a stand-alone printer that are hooked up to the voting system. It has two verification approaches: simple and advanced. Most voters are given the opportunity to use the simple verification system, which in fact is not really a verification at all, but rather just a receipt with a cryptographic code that shows that the voter voted and can be used on the web or telephone to verify the vote was counted (but who was actually voted for is not revealed). The simple verification system is not very useful by itself, but it is a required part of the advanced verification system. The advanced verification system (which would be used by some arbitrary number of voters per precinct--500 was recommended) requires voters to actually verify who they voted for race by race. They get a more complicated receipt that includes printed codes for each race. This proves to the voter that the actual people they voted for were recorded properly (both at the station, and later through the web or telephone). However, because the receipt encodes each race, it can not be used to verify to anyone else who they voted for. Thus votes can not be coerced or sold. The reason this complex set of two verification systems works is because the 500 voters who use the advanced verification system are chosen randomly at the time of voting. Furthermore, each system prints some cryptographic codes on the receipt before that choice is made. This combination of features is what enables VoteHere to assure its statistical integrity of verification and auditing.
Because the verification system results in individuals’ votes being recorded on an independent computer it can be used for election auditing and recounts. Election officials can compare the vote totals from the verification system with the vote totals from the voting system to check for discrepancies in vote counts.
We tested a mock-up of the system that provided the voter with an accurate polling place experience. This was a unit that consisted of the casing and printer a voter would encounter along with a printed receipt, but not the verification itself (i.e., they could not actually check their vote).
Scytl Pnyx
This system has a small independent computer and computer monitor that are connected to the voting system. The verification system captures individuals’ votes once they are submitted to the voting system. After verifying their selections on the small monitor, race by race, the voters have the opportunity to cast their ballot or change it. Voters who change their selections are presented their new selections on the small monitor and then they can cast their ballot.
Because the verification system results in individuals’ votes being recorded on an independent computer, election officials can compare the vote totals from the verification system with the vote totals from the voting system to check for discrepancies in vote counts.
The vote images are stored in a random order on the vote verification systems’ computer monitor. This protects the privacy of their vote. We tested what appeared to be an early prototype.
MIT Audio System
This system has a set of headphones that is attached to a voice-activated portable analog cassette tape recorder and a small computer unit that attaches to the voting system. Using the headphones, voters are able to hear a computerized voice that repeats their candidate selection to them immediately after it is selected on the voting system. Voters who change their selections hear audio confirmation that they have changed votes. When they are done with their selections and cast their vote, the audio tape stops.
The audio tape can be used for election auditing and recounts. Because an audio tape is used, the order in which individuals voted is kept intact, creating possible violations of secret voting.
We tested what appeared to be a fairly well-developed prototype.
Diebold AccuVote
We tested the Diebold AccuVote-TS that is presently used in
II. Expert Review
This reports the results of an expert review we performed on
The purpose of an expert review is to have a small number of people who are professionals in interface design and evaluation examine a specific system with the goal of identifying likely problem areas based on their expertise and experience with other similar systems. In this case, the review focused on usability and perception of the verification systems. The voting system itself as well as the security, accessibility and other features of the verification systems were beyond the scope of this study.
Participants
The following people participated as experts in the review. All
were from the
· Marguerite Autry User-Centered Design, Inc.
· Elizabeth Buie UserWorks
· John Cugini Cugini Consultants
· Dick Horst User Works
· Scott Gilkeson Usability Consultant
· Hilary Hutchinson UMD, Human-Computer Interaction Lab
· Bill Killam User-Centered Design, Inc.
· Bill Kules Takoma Software, Inc.
· Christy Mylks UCD / Usability Consultant
Systems Examined
The four vote verification systems selected by the Maryland State Board of Elections were examined. We used each system as delivered, and performed some programming work to connect the verification systems to voting systems as needed. In order to make the evaluation as fair and consistent as possible, we connected three of the verification systems to a single prototype DRE voting system that was provided by the Massachusetts Institute of Technology. The prototype DRE was designed to simulate the commercial Diebold DRE voting system because the other two systems we looked at were provided by Diebold. In this fashion, all five verification systems worked with essentially the same voting system.
Testing Structure
/ Instructions
The experts were brought in as a group to UMD where all the machines were set up in a single large room. The five systems were positioned side by side against one wall of the room.
Each expert was given the same voter pamphlets used in the broader field study. They were instructed to go through the entire voting and verification process, and to specifically be sure to:
· Change their vote after it was cast
· Enter write-in candidates
· Verify their vote
· Cancel their vote during the verification process
In addition to the above-mentioned voter pamphlet, the experts were also given the following list of heuristics and perspectives to keep in mind, and a paper and clipboard with one sheet of paper per machine to write down their concerns. They were encouraged to write down any concerns they had in addition to responses to these heuristics. They were asked to rate each concern with a severity rating of 1 (least severe) to 5 (most severe).
After each expert evaluated the machines individually and wrote down their comments, we had a group discussion where we discussed the concerns and trade-offs of each machine in turn.
General Heuristics
· Simple and natural language – messages should not contain information that is irrelevant or rarely needed
· Speak the user’s language – Dialogues should be expressed clearly in words, phrases, and concepts familiar to the user rather than in system-oriented terms.
· Minimize the user’s memory load – The user should not have to remember information from one part of the interface to another.
· Consistency – Users should not have to wonder whether different words, situations, or actions mean the same thing.
· Feedback – The system should always keep users informed about what is going on through appropriate feedback within reasonable time.
· Reversal of actions – As much as possible, actions should be easily reversible.
·
Clearly
marked exits – Users should be able to leave the system at will and need
support to do so.
·
Good
error messages – All errors should be avoided if possible, but any errors
messages should be expressed in plain language (no codes) precisely indicating
the problem and constructively suggesting a solution.
·
Help
– Even though it is better if the system can be used without documentation, it
may be necessary to provide help. Any such information should be easy to
navigate and should be focused on the user’s task.
Personal Perspective
· Novice computer users – how will people with no or minimal computer experience be able to these systems?
· Poor language skills – how will people with poor language skills in the language of the user interface be able to these systems?
· Error handling – how do the systems help users through error situations, unexpected user input, or changes in user input?
· Stress – how will people that are under stress (i.e., because there is a long impatient line of people behind them) be able to use these systems?
· Older voters – how will older voters be able to use these systems?
Results
We summarize the concerns and positive comments about each vote verification system, taken from the reports and discussion. The specific concerns follow each summary. Each concern was listed with a severity from 1 (low) to 5 (high). A description of each issue that was raised is listed with its severity. If similar concerns were raised by multiple experts, then the number of times that comment was made is listed along with the sum of the severity rankings.
There was general discussion about the fundamental trade-off in the different kinds of verification. The experts saw a trade-off in actual security vs. perception. While they knew that the cryptographic solutions were likely to be more secure in practice (or at least could theoretically be made more secure), they felt that many voters would not be able to understand this. As such, they felt that voters would perceive that they were just being asked to replace trust of one group (voting machine vendors) with another (cryptographic verification system vendors). Overall, they felt that the only way that most voters would truly trust the system would be if they could understand it themselves. Thus, they argued for a paper verification system which every voter could understand. Even if they did not or could not read the verification, the fact that it was there and could be understood was felt likely to give voters high confidence.
In addition, there was a discussion about the trade-off between sequential (provided by the Diebold wth AccuView Printer, Scytl Pnyx, and VoteHere Sentinel) and simultaneous verification (provided by MIT Audio system). There was general skepticism about the benefit of simultaneous verification because they felt that this would be a continuous distraction from the process of voting which would slow down the voter, and possibly result in more errors due to the distraction. In addition, they felt that it would be perceived as just more feedback of the vote rather than actual third-party verification. Again, they brought up the concern of actual vs. perceived security of the vote.
While there was not an explicit discussion about which system would most likely detect an error, there is a general principle in favor of simultaneous audio verification (of which the MIT system is an example) over after-the-fact verification (of which all the other systems are examples). In after-the fact verification, the voter must manually and explicitly compare the original vote to the vote on the verification system. The original and verified vote are in all cases physically separated which means that voters must use their “short term memory” to remember one while looking at the other. Furthermore, voters must correlate the races in each system so they are comparing the same things against each other. Scholarly research has shown that people have relatively poor short term memory [Atkinson and Shiffrin, 1968]. There is a fair amount of variance among individuals. Individuals with poor short-term memory, including the elderly, would generally find after-the-fact verification somewhat challenging, and thus would be expected to occasionally make mistakes.
On the other hand, simultaneous audio verification which gives audio feedback immediately upon selecting an individual candidate puts a much lower demand on the voter’s short-term memory because the feedback comes immediately after a candidate is selected and at the same time the voter is still looking at the candidate’s name. We can thus expect there to be fewer memory-related errors, and so this approach could result in better verification than after-the-fact verification, especially for older voters and others with poor short-term memory.
However, in practice, the same people that have poor short term memory may have poor hearing, and even impaired ability to integrate information from multiple sources. Given the additional characteristic of simultaneous audio verification being distracting, it is difficult to predict whether this approach would actually work better in practice for a wide audience. In addition, there is evidence synthetic speech places greater demands on short term memory [Smither 1993]. Given that the MIT system is implemented with synthetic speech, we would expect it to perform poorly.
Specific comments about each machine follow:
Diebold AccuVote-TSx with AccuView Printer Module (with paper
printout)
In general, the experts liked this verification system. It received the largest number of written positive comments and came out favorably in the discussion. While there was some concern that the verification system was made by the same vendor as the voting system and thus the systems were not independent, several experts brought up the fact that this would likely inspire confidence in many voters as it would feel like a more solid and robust system. Also, since the verification was paper, which was readily understandable and not easily tampered with by the system, it did not matter that the verification system was from the same vendor as the voting system.
As for details of the printer, almost everyone commented on the fact that they were surprised that could only reject the ballot twice, and it was accepted without choice on the third try. While this information was given at the beginning, there was no warning before the third try, which they felt would be very disconcerting to voters who got to this stage.
There was also a broad concern about how the printer would deal with long ballots since the viewing window was relatively small, and there were no controls to manipulate which part of the paper was viewed. We did not test this since our ballot was short enough to view all at once, but this must be addressed and tested.
The text quality in the printout was pretty good, but there was concern that it was too small. It was definitely appreciated that there was a magnifying glass to make the printout easier to read, but there were concerns that in practice, this magnifying glass would be problematic. It was a bit flimsy, there were concerns that people in need would not notice it in the first place – and more importantly, it did not work very well. It distorted the text, and made it harder to read sometimes – especially around the edges.
There was some concern about the barcode that was printed on the paper. We presumed that the bar code encoded the text and would be used for a recount. However, if this is the case, then the human-readable text doesn’t really serve much purpose. If voters notice this barcode and think about it, there was concern that it might decrease confidence in the paper trail.
There also were concerns about the privacy of recording on paper rolls. If votes were printed in continuous rolls, then privacy would be lost as the sequence of voters could be matched to the roll.
There was also concern about how the system would deal with paper jams and physical malfunctions. How reliable are these machines? How would these malfunctions affect privacy? What if it jammed in such a way that a pollworker had to look at a voter’s paper?
|
Count |
Severity Sum |
Description |
|
7 |
25 |
After 2 tries, the vote is cast
without warning |
|
5 |
22 |
Long ballots may be
hard/impossible to read on paper |
|
4 |
13 |
Magnifier distorts, hard to read |
|
2 |
8 |
A few inconsistencies in writing
as far as casting vote, casting ballot, printing ballot, etc. |
|
1 |
4 |
Printing casts the ballot -
confusing |
|
1 |
3 |
Unlikely to notice magnifier |
|
1 |
3 |
"rejected" printed but
not explained |
|
1 |
2 |
Font on paper is too small |
|
1 |
2 |
Bar codes on accepted paper are
confusing |
|
|
|
|
|
Positive Comments: |
|
|
|
|
|
Prefer format of instructions to
that of other systems |
|
|
|
Format/layout for Pres & VP
etc, good |
|
|
|
Good feedback for write-in
candidate |
|
|
|
"Please wait" message is
good to have |
|
|
|
Good that "print ballot"
changes to "cast ballot" |
|
|
|
Verification of actual vote on
paper is reassuring |
|
|
|
Yellow flashing was noticeable |
|
|
|
Noise of printer makes you look
there |
|
|
|
Liked the integrated form factor,
makes me a bit more confident |
|
|
|
Much higher feeling of trust
because paper is transparent |
|
|
|
Overall a positive impression of
this verification process, nice to have a record on paper |
|
|
|
Confident that it recorded voter
accurately |
VoteHere Sentinel
This verification system brought up significant concern among the experts. The concern centered on the fact that the “simple” version of the verification we tested was not in fact a verification. Rather, it is a cryptographic-based technique used by the vendor to ensure that the actual “complex” version of the verification (which really is a verification) works mathematically. Because this is a complex verification model that is not exposed to the voters, the experts felt that the voters would likely be very confused since the system tells the voter they are verifying their vote, but actually all they are doing is pressing a button saying they verified the vote in a way that is a nearly exact duplicate of the vote confirmation process which the voters would just have completed. In general, it was felt that it would be very difficult for voters to understand how this system worked. For example, they would not be able to understand why some of the receipt was printed early in the process, and the rest printed later. They might not understand why there were two codes (“ballot lookup number” and “ballot tracking code”). And the fact that there was an “Official Use Only” part of the receipt was intimidating and might make some voters believe that they were not allowed to take the receipt with them. Finally, the fact that a website was given might falsely give some voters the impression that their vote would not be private.
The experts recommended that if this system were to be used, then its purpose be made known more precisely to the voters. This could be done by saying something to the effect of “The election administrators ask you to press the ‘review’ button and take the printed receipt with you to help us verify the accuracy of your vote.”
In addition, there were some other specific concerns about this system. The main one was that the poll worker has to intervene if a voter rejects the ballot during the review process. This was felt to be a potentially very slow procedure that would make the voter feel that their privacy was violated.
|
Count |
Severity Sum |
Description |
|
7 |
23 |
Poll worker has to reset machine
if person rejects ballot |
|
5 |
18 |
Why are there 2 steps to cast your
ballot? Confirm and verify? |
|
2 |
9 |
Why did paper show up before
complete verification? Paper doesn't have anything to do with verify |
|
2 |
7 |
What is "sentinel audit
device?" If you're not going to explain it then say something simpler |
|
2 |
7 |
Verification looks too much like
the review of the vote, confusing |
|
2 |
7 |
What is website on paper for? |
|
2 |
6 |
Why is there "Official use
only" on the receipt? Am I allowed to take the receipt from the room? |
|
2 |
6 |
To verify "your vote
counted" is not the same as verify. I voted for who I thought I voted
for. |
|
2 |
5 |
People might be confused by having
both a ballot lookup and a ballot tracking code |
|
1 |
4 |
The paper codes are confusing
especially because of explanation |
|
Positive Comments: |
|
|
|
|
|
There were no explicit positive comments |
Scytl Pnyx
While this machine brought up some the same general concerns as the VoteHere Sentinel because it relies on a cryptographic approach that voters would probably not understand, it was generally perceived as better because there was an actual verification process that the voter could understand.
The larger concerns about this verification system centered on the specific design of the verification interface. It had a small screen that made it tedious to review a long ballot, and a very confusing interface with an unusual use of “confirm” and “cancel” buttons. The experts all felt that this on-screen review process was slower and more tedious than the review of a paper record. As such, they felt that many voters would not finish it (or perhaps not even notice it as it was separate from the voting system), and thus the voter’s privacy might be violated, or the vote might not be counted if they left while the verification was not completed.
|
Count |
Severity Sum |
Description |
|
8 |
35 |
"Confirm" and
"cancel" are final sounding activities but the actions they perform
are not final - they allow you to go back. When you finally do submit your
vote, you hit "continue" |
|
3 |
14 |
On the main screen it shouldn't
say "cast ballot," it should say something like "proceed to
verification" |
|
3 |
12 |
Buttons need to give scope, what
does "cancel" cancel? |
|
2 |
9 |
No way to tell if I've really
voted or not until done verifying |
|
2 |
8 |
The verification process is not
transparent. I don't understand how the voting machine and the module are
connected |
|
3 |
8 |
Instructions and verification
module text should be simpler, they are confusing and complex |
|
2 |
8 |
Can think you voted without
actually voting - just got to end of names |
|
2 |
6 |
Need to see poll workers to vote
again |
|
2 |
6 |
Cannot return to initial
instructions |
|
1 |
5 |
Privacy and no-vote problems are
likely if people don't finish the verification |
|
1 |
5 |
No parties on candidates in
confirmation module |
|
1 |
5 |
No warning that you can no longer
change vote |
|
1 |
4 |
Verification module -
confirm/cancel next to each other |
|
1 |
3 |
Verification screen is too small |
|
Positive Comments: |
|
|
|
|
|
There were no explicit positive comments |
MIT Audio
The experts thought that the idea of simultaneous audio verification was novel, and definitely worth a thorough consideration. However, they had near-unanimous concerns with this verification system for three core reasons: 1) For the audio to be loud enough for voters to hear reliably (even with good headphones), they felt that it would not be private, and that other people nearby would be able to hear who they were voting for; 2) The simultaneity of the audio was a continuous distraction to the voting process, and would likely slow down the voter and possibly cause errors; and 3) The simultaneous nature of the verification made it feel more like feedback rather than verification and thus much of the potential benefit in giving voter’s confidence would be lost. This last point was especially a concern if the voter did not approve of the verification. How would they respond to an inconsistency between the audio verification and what they selected on the voting device?
While this last issue is relevant for most verification systems, it is especially important for the MIT system because the simultaneous and audio nature of the verification implies that 1) having a poll worker come over would invade the voter’s privacy and could influence their vote since their vote would still be on the screen and would not yet be cast; and 2) there is no way to repeat the audio verification, so it would be impossible to demonstrate to the poll worker what the problem is without rewinding and replaying the audio tape at that moment.
In addition, there were other specific concerns with the implementation such as the voice quality not being good enough and it being machine generated rather than recorded from a human voice. In addition, there was no way to repeat the audio if someone did not hear the voice.
There were also concerns about the process of using audio tape. If one tape were used per machine, then privacy would be lost as the sequence of voters could be matched to the recording. If the tapes had to be physically moved from one machine to another between voters, it would introduce likely technical problems with wires and plugs being crossed or broken, or not used properly, etc.
|
Count |
Severity Sum |
Description |
|
6 |
28 |
Feels like privacy violation, others
could overhear |
|
4 |
16 |
Voting is a slow process with this
method, have to wait for the voice to stop talking |
|
3 |
11 |
Voice quality poor - hard to
understand |
|
2 |
10 |
No help or instructions on how to
use the recording device |
|
2 |
9 |
Common headphones won't be used by
some. Must have new pairs for each person |
|
2 |
9 |
Would an analog audio tape really
be able to recount 100,000 votes? Robustness concern |
|
2 |
8 |
No way to repeat audio if you
don't understand |
|
2 |
8 |
Unclear what is happening when you
go to cast your ballot |
|
2 |
6 |
Voice is mechanical, prefer a
non-machine voice |
|
1 |
5 |
What if audio feedback does not
match what you think you did? How does voter say "no that’s wrong"? |
|
1 |
5 |
Operation of the recording device
is a potential problem. Who stops it? Who starts it? |
|
1 |
5 |
Users who don't speak English well
or at all will be distracted |
|
2 |
4 |
Audio is not an after-vote
verification, but simply redundant feedback during the voting process |
|
1 |
3 |
Simultaneous feedback not
valuable, potentially distracting |
|
1 |
2.5 |
"Submit ballot" not good
as the last audio - does not let you know you have already finished |
|
1 |
2 |
Seemed to read write-in name but
hard to understand |
|
|
|
|
|
Positive Comments: |
|
|
|
|
|
"Next"
is good vocally |
|
|
|
Review
and submit ballot has good contrast to view |
|
|
|
Trust
this machine more than the other ones with verification |
|
|
|
Liked
that machine gave pronunciation of write -in |
Diebold AccuVote-TS (no verification unit)
Since there was no verification unit, there were no explicit comments about a verification system for this machine. However, several experts said that in general they felt that this voting system and process were trustable, and that if a verification step were added that were confusing, it could result in overall decreased confidence in the process.
Recommendations
Based on the expert review, we have some specific
suggestions as for improving
each verification system:
Diebold AccuVote-TSx: The magnifying window over the printer should be more solidly
constructed. It also should be made clearer to improve readability and reduce distortion.
There should be a long enough window to accommodate any length election, or
less preferably, a mechanism to let voters control the printout so they can see
a verification report that is longer than the window. There should be a clearer
description to the
voter that after the third try, the ballot would be
accepted no matter what. The barcode should be removed from the verification
report, or there should be a way for voters to learn what is encoded in that barcode.
The verification reports should be stored internally in random order so as to
remove the possibility of privacy infringement. Finally,
it should be easier for poll workers to replace the
paper rolls.
VoteHere Sentinel:
The purpose of the "simple" verification version should be clarified
to voters so they understand how the system works. This would avoid the potential
for voters to have negative feelings as a result of first being told they were verifying
their votes and later learning they are not actually doing this. Information
needs to be provided to voters to avoid a situation in which they mistakenly
think that their votes will be publicly available on the website or telephone
number. Because the receipt is to provided to the voter, the "Official Use
Only" part of the receipt might confuse the voter as to whether he or she
could take the receipt. As a result, it should be removed. .
Scytl Pnyx:
The write-in votes should be displayed as entered by the voter instead of the
system only notifying the voter that he or she had cast a write-in vote. The
interface, in
general, should be improved to have more consistent
and clear language
and button labels.
MIT Audio:
The voice should be recorded by a human rather than relying on a synthesizer.
The overall audio quality should be improved. The headphone coverings that
touch the ear should be replaceable for each voter to avoid health concerns.
The audio should be recorded in a non-sequential order so as to remove possible
infringement of voters’ privacy.
III.
Field Tests
The purpose of a field test is to collect data that can be used to generalize to individuals in the larger population. Field tests generally involve a large and heterogeneous group of individuals engaging in an activity in a fairly natural setting. In addition to an overview of all participants’ views, field tests enable researchers to compare the assessments of different types of participants. In this case, the field study involved observations of voters using the vote verification systems and the administration of post-voting questionnaires. The data analysis provides an overview of voters’ assessments of the different systems and comparisons of different groups of voters’ responses to those systems.
We recruited a diverse sample of 815 Marylanders to participate in the field tests of the vote verification systems. Participants included individuals with little-to-no experience using computers, individuals with little voting history, senior citizens, individuals with less than a college education, minority groups and women, individuals for whom English is not their native language, and individuals with disabilities. We made extra efforts to include these groups of voters because we tested computerized vote verification systems, and research on the digital divide identifies members of these groups as having less familiarity with and experience using computers. We recruited participants from shopping malls, office buildings, college campuses, and assisted living homes for senior citizens. (More information about group demographics and the testing sites is provided in the appendix.)
Overview
of Research Methodology
After receiving a short orientation to the study, which explained the purpose of the study and gave an overview of the operations of the vote verification system, each participant was asked to vote on one system and immediately complete a short questionnaire about the verification system before proceeding to the next system. All five voting systems (including the system with no vote verification system) were evaluated using a core set of questions. Additional customized questions were used to ascertain participants’ views about specific aspects of each of the four vote verification systems (The questionnaires appear in the appendix).
Voters were asked the following questions about the vote verification systems:
· How easy it was to use
· Their understanding of how it works
· Their confidence that their vote was accurately recorded.
· Their confidence that votes will be properly counted
· The privacy with which their vote was cast
· Their ability to confirm their write-in vote
· The ease of reading (or understanding for the audio version) their votes on the verification system. (This question was not used for the VoteHere Sentinel because it does not provide individuals with the opportunity to review their actual vote choices.)
· Whether their votes on the voting system matched those on the verification system. (This question was not used for the VoteHere Sentinel for the reason mentioned above.)
· Whether they needed help
· Whether they considered the vote verification system distracting
· The amount of time they spent voting
· Whether they experienced any technical problems while voting
Our
questionnaires did not include a question that asked participants whether they
actually paid attention to or used the vote verification systems because our
research required them to pay attention to the systems and use them. Thus,
participants were directly instructed to do so. Nevertheless, our observations
were that most participants did not spend the time needed to verify every
selection. Observations of voters in Nevada in a 2004 election who used systems
with paper records (conducted by the Los Angeles County Registrar/Recorder)
suggests that the majority of voters do not use the paper trail to verify all
of their selections. Observations of voters who participated in a study
conducted by members of this research team in
After
voting on the systems and completing the aforementioned questionnaires,
participants completed a questionnaire requesting information about their prior voting experience, computer usage patterns, age,
race, education, sex, disability status, native language, partisanship, and other
relevant background information. The questionnaire also queried participants
about whether they thought vote verification systems were important to the
voting process and if they had read about or discussed these systems. These
questions were included in order to gauge participants’ awareness of the issue
and the impact of lobbying and other efforts to publicize the vote verification
issue.
Finally, participants were asked by a member of the research team whether they needed assistance in using or understanding the vote verification systems and, if so, the specific type of assistance they needed. This final question, which was verbally delivered at the end of the experiment, was designed to elicit information on the specific aspects of the vote verification systems participants found challenging.
In
order to avoid bias due to learning effects,
voter fatigue, or other order effects, we randomly assigned participants
to each of the five voting systems using a 5 X 5
Latin square design. Statistical controls also were used in the analysis
discussed below to control for the possibility of order effects.
The data collected in the field tests has been
analyzed using appropriate statistical techniques. The analysis that follows
was designed to facilitate comparisons of the four vote verification options
and the voting system that had no vote verification system.
Results
First, an overview of the results that makes broad comparisons among the vote verification systems is presented. Next, some comparisons are made about the responses of voters who have different background characteristics, including their level of computer usage and familiarity and concern about vote verification systems. Finally, we create a voter satisfaction index based on participants’ assessments of whether their votes were accurately recorded, properly counted, and cast in private—the three core variables we used to assess all four verification systems and the voting system that did not have a verification system. This index (created by averaging the three scores together) is used to assess the impact of voters’ background characteristics on their level of satisfaction with the four verification systems and the voting system that has no verification unit.
Overview
Table 1 summarizes the evaluations of the four vote verification systems and the control system (which has no verification unit). The numbers in the table are averages of the voters’ level of agreement with the statements about the systems. The lowest level of agreement is 1 and the highest level of agreement is 7. Systems can be assessed by comparing the averages for each system across the different evaluative criteria.
The most notable result is that all of the systems were evaluated very favorably. Every system was evaluated between 5 and 7 on virtually every criterion, well above the 4 midpoint of the 7-point scale.
The first three criteria—concerned with voting accuracy, vote tallies, and privacy—are issues that are directly connected to the sanctity of the vote and are applicable to the four verification systems and the voting system without a verification system. For all five systems, participants very strongly agreed that their vote was accurately recorded. Participants expressed the highest level of confidence in the Diebold with AccuView Printer (with the paper trail), but the differences between it and the Diebold AccuVote-TS (no verification unit) and the VoteHere Sentinel (internet verification) are very small. The Scytl (separate computer monitor) and MIT Audio were evaluated somewhat lower, but they were evaluated very positively.
Similarly, all of the systems received high overall evaluations in terms of people’s votes being properly counted. Once again, the Diebold with AccuView Printer was evaluated the most favorably, followed the Diebold AccuVote-TS and the VoteHere Sentinel. The Scytl Pnyx and MIT Audio were evaluated somewhat lower, but very positively.
The results for casting one’s vote privately were somewhat different. The Diebold AccuVote-TS was rated most highly in terms of privacy, followed closely by the Diebold with AccuView Printer, and the Sentinel. The MIT and Scytl systems received somewhat lower ratings. These results are not surprising. The AccuVote-TS has no extra verification system, and so there is no extra medium (paper or otherwise) for others to observe how an individual cast his or her vote. The paper sheet on the AccuView Printer is angled so as to be difficult for others to view it, which probably accounts for its high rating. Because the Sentinel provides the voter with a sheet with an encrypted serial number it stands to reason that participants believed their vote was safe from prying eyes, but clearly at least a few thought that others might be able to learn about their vote after dialing the toll-free number or logging on the internet. The MIT and Scytl systems were somewhat lower, presumably because voters were concerned that these verification systems would allow others to hear or see how their votes were cast.
The next row in the table contains the satisfaction index for all five systems. Recall this is an average of participants’ assessments of whether their votes were accurately recorded, properly counted, and cast in private. The index demonstrates that all of the systems were evaluated favorably on these three core issues. The results for the AccuView Printer and AccuVote-TS, the two most favorably rated systems, are substantively indistinguishable. The Sentinel ranks next. It is followed by the Scytl and MIT systems, which have almost identical ratings.
The next two criteria are concerned with the ease with which voters could use the vote verification systems and understand how they operate. Because these evaluations were directly concerned with those systems, data were not collected for the voting system without a verification system. Participants provided very favorable assessments of all four systems in terms of their ease of use. They rated the Diebold AccuView Printer system as the easiest to use, followed by the VoteHere system. The Scytl was rated somewhat lower, conceivably because of the difficulty of reading the Scytl screen, which was small and did not match the screen on the DRE voting system. The MIT system was rated last, presumably because the large amount of static and the low quality of the computerized voice made listening a challenge.
Once again, most participants stated that it was easy for them to understand the workings of all four vote verification systems. Although the differences in evaluations were small, the Diebold AccuView Printer, with its simple paper trail, was considered the easiest system to use, followed by the VoteHere system, and then the Scytl system.
The next two questions are concerned with using the Diebold AccuView Printer, MIT, and Scytl verification systems to verify one’s votes. (The VoteHere system was omitted from the analysis because it is impossible for voters to verify that their votes had been cast from the polling place.) There are some noteworthy differences in the voters’ assessments of the three systems. First, the AccuView Printer received substantially higher evaluations then the other two systems in terms of verifying a write-in vote. This is to be expected given that the system prints out the name of the write-in candidate. The MIT system was rated more than one full point lower, presumably because the sound quality made it difficult to understand the name of the write-in candidate. Rated even lower, the Scytl system merely produced the word “write-in” when a person wrote in the name of the candidate. It is quite possible that the system could be programmed to print out the name of a write-in candidate as entered by the voter, but the system we tested was not programmed with this feature. (Its programming made it very difficult to integrate with the voting system, in general. See section IV on Election Administration Issues.) Because most voters do not cast write-in votes, the Maryland State Board of Elections and other election officials may wish to de-emphasize this criterion.
For the three systems for which the question was relevant, participants responded favorably to the statement about their ability to read their votes on the verification system (or understand them on the audio system). The Diebold AccuView Printer system (paper printout) received the most positive evaluation, followed by the Scytl system (small computer monitor). The MIT system was assessed substantially less favorably, presumably because of the static computerized voice and overall low quality of the sound.
|
Table
1. Average Ratings for
Usability of Vote Verification Systems |
|||||
|
|
DIEBOLD ACCU-VIEW PRINTER |
VOTEHERE SENTINEL |
SCYTL PNYX |
MIT AUDIO |
DIEBOLD ACCUVTE-TS |
|
I am confident that my vote was accurately recorded |
5.99 |
5.66 |
5.33 |
5.23 |
5.79 |
|
I think everyone’s votes will be properly counted |
5.81 |
5.51 |
5.14 |
5.02 |
5.60 |
|
No one else could tell how I voted |
5.43 |
5.32 |
4.99 |
5.16 |
5.57 |
|
Satisfaction index |
5.75 |
5.50 |
5.15 |
5.14 |
5.65 |
|
The machine was easy to use |
5.95 |
5.59 |
5.05 |
4.94 |
N/A |
|
I understand how the verification system works |
6.02 |
5.70 |
5.45 |
5.42 |
N/A |
|
The verification system made it easy to confirm the write-in vote |
5.95 |
N/A |
4.77 |
4.94 |
N/A |
|
It was easy to read/understand my votes on the verification system |
5.83 |
N/A |
5.44 |
4.90 |
N/A |
|
The verification system was distracting (Note: lower numbers are more favorable for this question) |
2.90 |
3.20 |
3.35 |
3.99 |
N/A |
|
N (range) |
(771-792) |
(765-793) |
(601-733) |
(684-776) |
(693-769) |
|
Note: Figures are averages of responses from the survey questions provided. A response of 1 represents “Strongly Disagree” and a response of 7 represents “Strongly Agree.” |
|||||
The last evaluation in Table 1 is concerned with whether voters considered the verification system distracting. For this question, lower numbers—representing stronger disagreement—are more desirable. Participants found all of the verification systems to be
distracting to some degree. The printer on the Diebold with AccuView Printer was evaluated as the least distracting, perhaps because it was fully integrated into the system, had a cover on it, and did not actually print out the voter’s selections until he or she was ready to review the paper sheet. The MIT system was considered the most distracting, most likely because it stated a candidate’s name immediately after the candidate was selected and before the voting process was completed. Members of the research team do not consider distractions that take place during the voting process desirable, particularly those that take place between voter selections of candidates (as is the case with the MIT system). However, some people may consider distractions to be useful because voters might be more likely to use a vote verification system if it distracted them.
Table 2 summarizes the evaluations of the four vote verification systems on matters related to participants’ perceptions about their need for help using the systems and the degree to which they believed the votes on the verification system matched those on the voting system. Participants reported needing the least help when using the system that had no verification unit, undoubtedly because the voting process on that system was less complex and consisted of considerably fewer steps. The Diebold with AccuView Printer (paper trail) was ranked next, probably because all a voter had to do was read a sheet of paper. That participants needed somewhat more help with the VoteHere system may reflect the fact that a poll worker had to reset the system if the voter rejected the ballot. The need for somewhat more help on the MIT system may be attributed to the audio system; many participants asked for help trying to adjust the audio so they could better understand it, and some forgot to put on the headphones. The relatively poor performance of the Scytl system (small computer monitor) may have to do with confusion over the “confirm” and “cancel” buttons and the slowness and tediousness of the review process—a point that was made by the expert reviewers. It should be noted that the final three systems were tested using a mock Diebold voting systems rather than actual systems. The touchscreens on the mock systems were not as good as the actual Diebold voting system and some participants needed help using them. Although we emphasized to participants on several occasions that they were to answer the questions for the vote verification system, not the voting system, responses to this question may have been affected by the voting system itself.
Feeling that some assistance would be helpful and having sufficient difficulty to ask for or attract the assistance of a pollworker are two different things. We explored this second possibility by having our staff record whether a participant actually received help with the vote verification systems (this did not include help with the voting system or the mock voting system). The results of this analysis once again show that the participants received the least amount of help when using the Diebold AccuVote-TS system. The AccuView Printer system (paper trail) came next. Voters received more help using the VoteHere (internet), MIT (audio), and Scytl (monitor) systems.
Voters were only capable of comparing the selections they made on the voting system with those reported on two verification systems: the Diebold with AccuView Printer and Scytl Pnyx. (Recall the VoteHere system only prints a serial number, not the voters’ selections.) Ninety-seven percent of the participants said that the votes on the AccuView Printer matched their actual votes, a significantly higher match than that reported for Scytl. This may be due to the weaknesses of Scytl reported in the expert review. The fact that Scytl did not report the actual name of the write-in candidate and instead presented the words “write-in” also was undoubtedly relevant.
Table 2. Assessments
of the Need for Help and Accuracy
|
|
DIEBOLD ACCU-VIEW PRINTER |
votehere Sentinel |
Scytl PNYX |
MIT Audio |
Diebold AccuVote-TS |
|
Did you feel the need to get help? |
19.8 |
27.5 |
41.6 |
29.5 |
14.0 |
|
Instances where voters actually received help using verification systems |
4.5 |
5.9 |
7.0 |
6.9 |
1.8 |
|
Did the votes on the voting system match those reported by the verification system? |
97.0 |
N/A |
90.8 |
N/A |
N/A |
|
N (range) |
(777-815) |
(765-815) |
(686-815) |
(765-815) |
(802-815) |
Note: Figures are the percentage of respondents who answered yes to the given survey.
Although the research team was not commissioned to study technical problems, when such problems arise they disrupt the voting process and have substantial effects on the usability of voting systems. Our previous research on voting systems, and the findings of other researchers, demonstrate that technical problems can and do arise when voters are using them under normal circumstances. As such, we included a question to address this issue and report the technical problems we observed.
The technical problems addressed in Table 3 include issues ranging from difficulties that voters reported to situations where a vote verification system or voting system became inoperative under normal use. The large number of technical problems reported on the Scytl and MIT vote verification systems and the Diebold AccuVote-TS (no verification unit) to a large degree reflect the fact that these systems became inoperative and were out of commission for portions of the study. (See the final section of the report for more information about system difficulties.) When the Scytl and MIT vote verification and Diebold AccuVote-TS with no verification unit became inoperative they had a direct impact on the usability of the voting systems and caused concern among voters—a clear usability issue. This study did not, and was not intended to, focus on technical problems, but we report them because we observed them.
Table 3. Evidence of
Technical Problems
|
|
DIEBOLD ACCU-VIEW PRINTER |
votehere Sentinel |
Scytl PNYX |
MIT Audio |
Diebold AccuVote-TS |
|
Did the voting machine you just used have any technical problems? |
9.2 |
14.1 |
37.9 |
26.3 |
17.0 |
|
(N) |
(804) |
(803) |
(805) |
(800) |
(807) |
Note: Figures are the percentage of respondents who answered yes to the given survey or who did not have the opportunity to vote on the system because it was out of commission due to hardware or software problems.
The Impact of Voters’
Backgrounds and Other Related Factors
The analysis of the impact of participants’ backgrounds on the satisfaction index (comprising their beliefs about their votes being properly recorded, counted, and cast in private) includes the following information collected from participants (the exact question wording is presented in the appendix):
Have you read about or discussed voter verification systems
with anyone? |
|
Necessity of voter verification systems |
|
Have you previously voted in an election? |
|
Have you previously used a touch screen voting system? |
|
Have you previously used a "Paper" voting system? |
|
Have you ever used an ATM to do your banking? |
|
How often, on average do you use a computer? |
|
What is the highest level of education you have completed? |
|
Do you
have any physical challenges or disabilities? (impaired eyesight, dyslexic,
impaired hearing, tremors or shaky hands, wheelchair bound, cognitive
disability, other) |
|
How old are you? |
|
Language speaks most regularly |
|
Are you male, female? |
|
Black (nonHispanic) |
|
Hispanic |
|
Income (low to high) |
|
Strength of partisan attachments |
|
Democrat |
|
Republican |
The results of the data analysis (see Tables A14-A18 in the appendix) demonstrate that generally speaking, the aforementioned background factors did not lead participants to provide differing assessments of the vote verification systems or the control system (no verification unit). Few of the variables had a systematic impact on participants’ opinions. Only those that are statistically significant are discussed.
Participants with higher levels of education provided slightly lower evaluations for all of the verification systems and the control system than did other individuals. Frequent computer users also provided slightly lower evaluations than those who use computers infrequently. These differences, though extremely small, suggest that individuals with more exposure to computers are slightly more skeptical about electronic voting machines in general, regardless of whether they have vote verification systems.
Individuals’ race or ethnicity had little effect on their evaluations of the systems. Hispanics voiced somewhat less satisfaction with all of the vote verification systems and the control system than did non-Hispanics. However, their evaluations were roughly the same for each of the five systems. African Americans were slightly less impressed with the Scytl and MIT systems than were others (most of whom were white). But in general, race and ethnicity did not lead to substantial differences in participants evaluations of the voting systems.
The findings for individuals with disabilities are largely inconclusive. Although participants identified themselves as having a wide range of disabilities, most were related to seeing and hearing impairments. We tested for these two impairments and combined the rest (dyslexia, tremors or shaky hands, wheelchair bound, cognitive disability, etc.) into one “other” category. None of the three sets of disabilities had any effect on individuals’ satisfaction with the Diebold AccuView Printer (paper), AccuVote-TS (control) or Scytl (computer monitor) systems. Voters with visual impairments provided slightly lower ratings for the MIT (audio) system, and those reporting any of the three disabilities provided slightly lower evaluations for the VoteHere (internet) system, but it must be emphasized that these evaluations were small enough to be considered inconsequential.
Participants who said they had read about or discussed vote verification systems were slightly less approving of the four verification systems than others. They also gave slightly lower evaluations for the Deibold AccuVote-TS which had no verification unit (the control system) than they did for the systems with vote verification systems. Nevertheless, the differences were substantively inconsequential. Ironically, individuals who strongly indicated that vote verification systems were necessary gave slightly higher evaluations to all five systems (including the system that had no verification unit) than did those who indicated they believe verification systems are unnecessary. Overall, these findings suggest that study participants’ evaluations were not strongly influenced by the efforts of interest groups and others to raise the issue of vote verification.
IV. Election Administration Issues
Integration of Vote Verification Systems
Three vote verification systems, from Scytl, VoteHere and MIT, were tested with a mock Diebold interface. The Diebold mockup we chose to use was one of several sample DRE interfaces developed at MIT. The Scytl and VoteHere systems had to be integrated with the MIT Diebold mockup. (The MIT system was already integrated with it.) Each piece of technology had its own set of challenges, summarized below.
Scytl Pnyx
This system was at an early stage in development. While prototypes are bound to have problems, it had more than anticipated, and we were not notified of the issues upfront. Many problems emerged when we sought to integrate the verification system with the voting system and during the field tests.
The programming challenges and modifications with Scytl include:
· The source code provided was limited. The system came with much high-level documentation, but the source code lacked important simple “getting started” instructions.
· Folder structure of source code was confusing and had to be changed in order to use with Eclipse, the development environment used by Scytl.
· Essential connection parameters were buried deep on the development machine, with no instructions on where they were.
· Frequent requests to Scytl for simple build instructions with a simple example and/or the compiled code were ignored.
·
Several calls to the company, in
· Source code provided on the CD was an earlier version, which was incompatible with the hardware we were given. The update we received was a later version, which was also incompatible.
· Once we were given the compiled code, the technical representative did not know what all of the dependencies were and had to do some testing to figure it out.
· During development, the system stopped working. The representative later discovered that the Certificate Authority in the voting terminal had expired.
· During testing, we were notified by Scytl of a 50 vote limit per session at which point the system needed to be reset.
· During testing, we were notified by Scytl of a 250 vote limit per election, at which point the compact flash card needed to be cleared.
· No way to pass the names of write-in candidates to the verification module. As a result, “Write-in” was displayed on the verification module for candidates whose names had been written in.
· Titles of offices had to be shortened to fit on the screen with the default fonts.
· Added three screens to the mock Diebold interface for “votes accepted,” “votes cancelled” and “communication errors.” Used the text from the mock voting terminal provided by Scytl.
Because this system was in such an early stage in development it was very difficult to integrate with the prototype voting system. This resulted in numerous problems during the programming stage and during the field tests themselves.
Time Spent on Integration: conservative estimate 110 hours
Programmer’s evaluation: Poor
VoteHere Sentinel
For simplicity and to meet the limited needs of our field study, we decided not to integrate the voting system with the full-blown Sentinel verification unit. Instead, we were given a simple printer unit, which looked like the Sentinel. We then tuned the mock Diebold interface and the details printed by the mock Sentinel to mimic the interaction with the real prototype. Hence, we emulated the front end of the vote verification process only. Emulating the entire the process, including the production of a receipt that voters could use to check their votes via telephone or on the Internet might have been more complex, but it was unnecessary for our study.
The technical representative from VoteHere was very helpful in working with us to design the mock interface and connect to the printer.
The programming challenges and modifications with VoteHere include:
· Had to drive the printer unit from java.
· Had to determine whether to show simple or advanced verification and what the interface should look like.
· Decided to only demonstrate the simple verification, on the basis that most states would not allow user to do the advanced verification.
· Changed “Cast Ballot” to “Confirm Ballot” for the confirmation page
· Added screens for “verification,” “votes verified,” and “votes rejected.” Used the text from the mock DRE provided by VoteHere.
·
Provided the ability to cancel for simple
verification, which was not in the original VoteHere design.
Only a front-end mock-up of the system was integrated into the voting system. This was relatively easy to perform and it was done with the help of a VoteHere representative.
Time Spent on Integration: 3-4 hours
Programmer’s evaluation: Good
MIT Audio System
This system was already integrated with the Diebold mockup and provided no difficulties.
Time Spent on Integration: 0 hours
Programmer’s evaluation: not applicable
Diebold AccuVie-TSx with
AccuView Printer Module and Diebold AccuVote-TS
These systems arrived fully integrated and provided no difficulties.
Time Spent on Integration: 0 hours
Programmer’s evaluation: not applicable
Experiences from the Field Relevant to Election Day
Activities
Except for the Diebold AccuVote-TSx with the AccuView Printer Module, the vote verification systems we tested were prototypes at various levels of development. (The Diebold AccuVote-TS with no verification unit also was not a prototype.) Setting up each piece of technology had its own set of challenges, summarized below.
Diebold AccuVote-TSx
with the AccuView Printer Module
This system was very easy to set up. The paper spool had to be changed between every 80 and 100 votes, much more frequently than anticipated. Changing the paper spool was a complex task. Among other things, the printer would only work on one side of the carbonized paper. Staff needed the assistance of the Diebold staff to figure this out at one point.
Scytl Pnyx
This system required many parts to be connected. The prototype delivered required that the date be reset to a previous date in order to be operated. It took considerably longer to set up than the other systems. It was out of commission due to software problems for a portion of the study.
VoteHere Sentinel
The mock VoteHere system was very easy to set up and operate.
MIT Audio System
The MIT system was not that hard to set up, but it was out of commission for a portion of the study due to hardware problems with the tape recorder and with the mock DRE.
Some participants raised sanitary concerns about putting on the headphones.
Diebold AccuVote TS (no
verification unit)
The system was easy to set up. Although it was only subject to normal use by voters it became inoperative while an individual was voting during the seventh day of testing. This resulted in the system having to be removed from the site and unusable for one day of testing while repairs were made.
It should be noted that this study did not, and was not intended to, focus on technical problems, but we report them because we observed them.
Manufacturer’s Support
Diebold (with AccuView Printer and
AccuVote-TS)
The manufacturer’s representative was extremely responsive. He visited a test site on a Saturday to help staff install the paper in the printer. He visited a test site on a Sunday to try to repair the system, took the system back to the State Board of Elections to have the problem addressed, and returned it a day and a half later.
Grade: Excellent
Scytl Pnyx
The
technical representative responded to emails, but often failed to address the
questions asked. Frequent requests for simple instructions/examples were
ignored. In the end it took several phone calls to the company in
Grade: Poor
VoteHere Sentinel
The technical representative was very helpful in working with us to design the mock Sentinel used in the test and connect it to the printer.
Grade: Excellent
MIT Audio Verification
The MIT system arrived integrated into the Mock Diebold System. No support was needed to integrate the system. MIT’s staff were very helpful in that they provided the Mock Diebold System used in the field experiment.
Grade: Not Applicable because no support needed.
V.
Recommendations
On the basis of usability and some administrative
considerations we cannot recommend that the State of
The expert review and the field test demonstrated that the Diebold AccuView Printer system (paper printout) performed best followed closely by the Diebold AccuVote-TS (no verification) system. The differences between these two systems were insignificant. The VoteHere Sentinal (internet or telephone), Scytl Pnyx (computer monitor), and MIT (audio) systems all performed well, but received somewhat lower evaluations in the expert review or the field test. Specific recommendations for each vote verification system are provided at the end of the expert review (Section II).
Voters’ backgrounds had little impact on their evaluations. Computer usage patterns and demographic characteristics did not lead any group of voters to provide significantly lower evaluations for any one of the vote verification systems or the voting system with no verification unit. Individuals who were more familiar with or concerned about issues related to vote verification did not voice strong support for any one vote verification system (or the system with no verification unit) over any other, despite the efforts of some grassroots and lobbying organizations.
Considerations related to integrating the vote verification systems with a voting system (and in some cases programming the vote verification systems), setting up the verification systems for use by voters, and the support provided by some of the system’s manufacturers reinforce our conclusion that the four vote verification systems detract from the usability of the AccuVote-TS voting system.
Finally, we observed some shortcomings in the Diebold
AccuVote-TS system, which is currently used by the state of
VI. Appendix
Integration of the Vote Verification Systems with the
Voting Systems
Originally, we proposed to test seven vote verification systems and the Diebold AccuVote-TS, which has no verification unit. When three manufacturers chose not to participate in a timely fashion, we ended up testing four vote verification systems and the Diebold AccuVote-TS. The verification systems were to be fully integrated into a voting unit or a computer with voting unit software , and delivered to CAPC in September. These integrated systems were to be used in the expert reviews and field tests. The field tests were to involve testing the systems during the fall 2005 elections at six or more polling places and at a variety of locations before and after election day.
We had to make some modifications to our original research methodology. First, because the systems did not arrive on time and were not integrated a voting unit or a computer with voting unit software, CAPC and SBE worked together to make two changes in our testing strategy. First, we integrated the MIT, Scytl, and VoteHere vote verification systems with laptop computers and touch screen monitors that were programmed to emulate the interface on the Diebold AccuVote-TS (with no verification unit). (The Diebold with AccuView Printer system (paper printout ) and the Diebold AccuVote-TS (with no verification unit) did not need to be programmed to emulate the Diebold interface.) Substituting the computerized touchscreens for actual voting systems may have introduced some measurement error into the study, but we are confident that they did not prevent expert reviewers and voters from evaluating the vote verification systems accurately.
Field Study Methodology
Second, because the systems arrived too late to be deployed
on election day, we increased the amount of testing we did at other sites. We
tested the voting systems on college campuses, office buildings, shopping
malls, and senior citizen assisted living centers in
The computer usage levels and background characteristics of the field study participants are presented in the tables below (see Tables A1-A13).
Table A1. Frequency of Computer Use
|
|
Percent |
|
Never |
4.9 |
|
Once a month |
4.2 |
|
Once every two weeks |
2.5 |
|
One or two days a week |
4.3 |
|
Three or four days a week |
9.4 |
|
Five to seven days a week |
73.5 |
|
(N) |
(815) |
Table A2. ATM Use
|
|
Percent |
|
No |
9.2 |
|
Yes |
90.8 |
|
(N) |
(807) |
Table A3. Respondents’ Voting History
|
|
Previously Voted |
Used Touchscreen |
Used Dials and Knobs |
Used Lever |
Used Punchcard |
Used Paper |
Used Other |
|
Yes |
79.3 |
54.7 |
6.1 |
30.3 |
22.6 |
19.0 |
5.6 |
|
No |
19.4 |
45.3 |
93.9 |
69.7 |
77.4 |
80.9 |
94.4 |
|
(N) |
(804) |
(814) |
(815 |
(813) |
(815) |
(813) |
(815) |
Table A4. What is the language you speak most regularly?
|
|
Percent |
|
English |
94.1 |
|
Spanish |
3.6 |
|
Other |
2.2 |
|
(N) |
(802) |
Table A5. Respondents’ Gender
|
|
Valid Percent |
|
Male |
38.6 |
|
Female |
61.4 |
|
(N) |
(801) |
Table A6. Respondents’ Age
|
|
Percent |
|
18-24 |
31.1 |
|
25-34 |
16.4 |
|
35-49 |
24.8 |
|
50-64 |
20.5 |
|
65-74 |
4.3 |
|
75+ |
2.9 |
|
(N) |
(805) |
Table A7. Respondents’ Race/Ethnicity
|
|
Percent |
|
White |
43.2 |
|
Black |
35.6 |
|
Hispanic/ Latino |
7.6 |
|
Asian |
6.9 |
|
Other |
6.6 |
|
(N) |
(812) |
Table A8. Respondents’ Education
|
|
Percent |
|
Some high school |
3.4 |
|
High School diploma or GED |
20.2 |
|
Some college, no degree |
40.2 |
|
4-year degree |
14.8 |
|
Some post-graduate work |
5.7 |
|
Master's degree |
12.2 |
|
Doctoral, Law, or Medical degree |
3.5 |
|
(N) |
(805) |
Table A9. Respondents’ Total Household Income
|
|
Percent |
|
$0-$14,999 |
10.7 |
|
$15,000-$34,999 |
19.2 |
|
$35,000-$49,999 |
16.8 |
|
$50,000-$64,999 |
12.2 |
|
$65,000-$84,999 |
10.8 |
|
$85,000 or more |
21.1 |
|
Don't know |
9.0 |
|
(N) |
(785) |
Table A10. Respondent Disabilities
|
|
Percent |
|
Impaired eyesight |
10.2 |
|
Dyslexic |
.7 |
|
Impaired hearing |
2.5 |
|
Tremors or shaky Hands |
.5 |
|
Wheelchair bound |
.6 |
|
Cognitive disability |
.5 |
|
Other |
1.8 |
|
(N) |
(815) |
Table A11. Necessity of voter verification systems
|
|
Percent |
|
1 Unnecessary |
2.0 |
|
2 |
1.8 |
|
3 |
3.5 |
|
4 |
10.5 |
|
5 |
15.1 |
|
6 |
18.8 |
|
7 Necessary |
48.3 |
|
(N) |
(797) |
Table A12. Have you read about or discussed voter verification systems with anyone?
|
|
Percent |
|
No |
69.0 |
|
Yes |
31.0 |
|
Total |
(807) |
Table A13. Seven-point partisan scale
|
|
Percent |
|
1 Strong Democrat |
32.1 |
|
2 |
18.7 |
|
3 |
11.0 |
|
4 |
16.2 |
|
5 |
6.1 |
|
6 |
4.9 |
|
7 Strong Republican |
7.0 |
|
Other |
4.0 |
|
Total |
(798) |
Multivariate Analyses of the Impact of Participant’s
Background Factors on Satisfaction Index (Tables A14-A18)
|
Table A14. Paper (Diebold AccuVote-TSx with AccuView Printer) |
|
|
|
|
|
Model
Summary |
|
|
|
|
|
R |
|
Adjusted |
Std.
Error of the Estimate |
|
|
0.225156644 |
0.050696 |
0.015097 |
1.325976 |
|
|
|
|
|
|
|
|
|
Unstandardized
Coefficients |
Standardized
Coefficients |
Sig. |
|
|
|
B |
Std.
Error |
Beta |
|
|
(Constant) |
5.173 |
0.495 |
|
0.000 |
|
disaeye2 recoded disabeye (Disability: impaired
eyesight) to set 4 cases coded as no disab and eye disab to missing |
-0.003 |
0.181 |
-0.001 |
0.987 |
|
disabhea Disability: Impaired hearing |
0.339 |
0.371 |
0.040 |
0.362 |
|
disaoth2 combined other disability (dyslexia,
tremors, wheelchair, cognitive, other) (can include eyes and ears) |
-0.299 |
0.304 |
-0.042 |
0.326 |
|
readdiss Have you read about or discussed voter
verification systems with anyone |
-0.223 |
0.120 |
-0.078 |
0.064 |
|
needveri Necessity of voter verification systems |
0.074 |
0.041 |
0.077 |
0.070 |
|
prevvote Have you previously voted in an election? |
-0.095 |
0.177 |
-0.027 |
0.591 |
|
prevuse1 Have you previously used a touch screen
voting system? |
-0.043 |
0.124 |
-0.016 |
0.731 |
|
prevuse5 Have you previously used a
"Paper" voting system? |
0.053 |
0.148 |
0.016 |
0.722 |
|
atmuse Have you ever used an ATM to do your
banking? |
0.324 |
0.240 |
0.063 |
0.177 |
|
compuse How often, on average do you use a
computer? |
-0.090 |
0.051 |
-0.086 |
0.079 |
|
educatio What is the highest level of education you
have completed? |
-0.103 |
0.045 |
-0.109 |
0.023 |
|
age How old are you? |
-0.011 |
0.053 |
-0.011 |
0.835 |
|
english language speaks most regularly dummy |
0.449 |
0.271 |
0.071 |
0.097 |
|
gender Are you male, female? |
0.112 |
0.117 |
0.041 |
0.338 |
|
black_nonhisp black only & non-hispanic |
-0.093 |
0.130 |
-0.034 |
0.475 |
|
hispanic_rc other race and hispanic, excluding 1 white
& 1 black, to use as hispanic variable |
-0.576 |
0.420 |
-0.059 |
0.170 |
|
income_rc income recoded with dk as missing |
0.062 |
0.036 |
0.081 |
0.087 |
|
pstren partisan strength |
-0.010 |
0.081 |
-0.008 |
0.903 |
|
democrat democratic PID 1 thru 3 |
0.027 |
0.241 |
0.010 |
0.912 |
|
repub republican PID 5 thru 7 |
0.150 |
0.250 |
0.045 |
0.547 |
|
order_p |
0.065 |
0.039 |
0.069 |
0.097 |
|
Dependent
Variable: indexp Paper: index of
record, count, and privacy |
||||
|
|
|
|
|
|
|
N = 582 |
|
|
|
|
|
|
|
|
|
|
|
Table A15. Internet (VoteHere) |
|
|
|
|
|
|
|
|
|
|
|
R |
|
Adjusted |
Std.
Error of the Estimate |
|
|
0.260997486 |
0.06812 |
0.03336 |
1.49691 |
|
|
|
|
|
|
|
|
|
Unstandardized
Coefficients |
Standardized
Coefficients |
Sig. |
|
|
|
B |
Std.
Error |
Beta |
|
|
(Constant) |
5.895 |
0.546 |
|
0.000 |
|
disaeye2 recoded disabeye (Disability: impaired
eyesight) to set 4 cases coded as no disab and eye disab to missing |
-0.329 |
0.208 |
-0.066 |
0.114 |
|
disabhea Disability: Impaired hearing |
-0.661 |
0.433 |
-0.066 |
0.128 |
|
disaoth2 combined other disability (dyslexia,
tremors, wheelchair, cognitive, other) (can include eyes and ears) |
-0.528 |
0.352 |
-0.063 |
0.134 |
|
readdiss Have you read about or discussed voter
verification systems with anyone |
-0.171 |
0.137 |
-0.052 |
0.212 |
|
needveri Necessity of voter verification systems |
0.074 |
0.046 |
0.068 |
0.105 |
|
prevvote Have you previously voted in an election? |
-0.230 |
0.199 |
-0.056 |
0.249 |
|
prevuse1 Have you previously used a touch screen
voting system? |
-0.050 |
0.140 |
-0.016 |
0.718 |
|
prevuse5 Have you previously used a
"Paper" voting system? |
-0.247 |
0.166 |
-0.064 |
0.138 |
|
atmuse Have you ever used an ATM to do your
banking? |
0.453 |
0.269 |
0.078 |
0.093 |
|
compuse How often, on average do you use a
computer? |
-0.122 |
0.058 |
-0.103 |
0.035 |
|
educatio What is the highest level of education you
have completed? |
-0.092 |
0.051 |
-0.085 |
0.072 |
|
age How old are you? |
-0.043 |
0.060 |
-0.037 |
0.478 |
|
english language speaks most regularly dummy |
-0.256 |
0.301 |
-0.036 |
0.395 |
|
gender Are you male, female? |
0.187 |
0.131 |
0.060 |
0.155 |
|
black_nonhisp black only & non-hispanic |
-0.114 |
0.146 |
-0.036 |
0.437 |
|
hispanic_rc other race and hispanic, excluding 1 white
& 1 black, to use as hispanic variable |
-0.883 |
0.473 |
-0.079 |
0.062 |
|
income_rc income recoded with dk as missing |
0.048 |
0.041 |
0.055 |
0.243 |
|
pstren partisan strength |
0.039 |
0.091 |
0.029 |
0.671 |
|
democrat democratic PID 1 thru 3 |
-0.170 |
0.271 |
-0.054 |
0.532 |
|
repub republican PID 5 thru 7 |
-0.239 |
0.279 |
-0.062 |
0.393 |
|
order_i |
0.116 |
0.044 |
0.110 |
0.008 |
|
Dependent
Variable: indexi Internet: index of
record, count, and privacy |
||||
|
|
|
|
|
|
|
N = 585 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Table A16. Scytl |
|
|
|
|
|
|
|
|
|
|
|
R |
|
Adjusted |
Std.
Error of the Estimate |
|
|
0.221677167 |
0.049141 |
0.008306 |
1.630948 |
|
|
|
|
|
|
|
|
|
Unstandardized
Coefficients |
Standardized
Coefficients |
Sig. |
|
|
|
B |
Std.
Error |
Beta |
|
|
(Constant) |
5.191 |
0.628 |
|
0.000 |
|
disaeye2 recoded disabeye (Disability: impaired
eyesight) to set 4 cases coded as no disab and eye disab to missing |
0.075 |
0.243 |
0.014 |
0.757 |
|
disabhea Disability: Impaired hearing |
0.440 |
0.518 |
0.041 |
0.397 |
|
disaoth2 combined other disability (dyslexia,
tremors, wheelchair, cognitive, other) (can include eyes and ears) |
-0.101 |
0.385 |
-0.012 |
0.792 |
|
readdiss Have you read about or discussed voter
verification systems with anyone |
-0.256 |
0.158 |
-0.073 |
0.107 |
|
needveri Necessity of voter verification systems |
0.144 |
0.052 |
0.126 |
0.006 |
|
prevvote Have you previously voted in an election? |
-0.085 |
0.231 |
-0.019 |
0.715 |
|
prevuse1 Have you previously used a touch screen
voting system? |
-0.196 |
0.161 |
-0.059 |
0.225 |
|
prevuse5 Have you previously used a
"Paper" voting system? |
-0.054 |
0.192 |
-0.013 |
0.778 |
|
atmuse Have you ever used an ATM to do your
banking? |
0.367 |
0.308 |
0.060 |
0.234 |
|
compuse How often, on average do you use a
computer? |
-0.136 |
0.065 |
-0.113 |
0.036 |
|
educatio What is the highest level of education you
have completed? |
-0.107 |
0.060 |
-0.093 |
0.075 |
|
age How old are you? |
-0.038 |
0.069 |
-0.032 |
0.579 |
|
english language speaks most regularly dummy |
0.006 |
0.347 |
0.001 |
0.985 |
|
gender Are you male, female? |
0.075 |
0.154 |
0.022 |
0.628 |
|
black_nonhisp black only & non-hispanic |
-0.243 |
0.172 |
-0.072 |
0.159 |
|
hispanic_rc other race and hispanic, excluding 1 white
& 1 black, to use as hispanic variable |
-0.720 |
0.538 |
-0.061 |
0.182 |
|
income_rc income recoded with dk as missing |
0.053 |
0.048 |
0.056 |
0.275 |
|
pstren partisan strength |
0.001 |
0.107 |
0.000 |
0.995 |
|
democrat democratic PID 1 thru 3 |
-0.091 |
0.320 |
-0.027 |
0.777 |
|
repub republican PID 5 thru 7 |
0.074 |
0.329 |
0.018 |
0.823 |
|
order_s |
0.023 |
0.052 |
0.019 |
0.664 |
|
Dependent
Variable: indexs Scytl: index of
record, count, and privacy |
||||
|
|
|
|
|
|
|
N = 511 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Table A17. Audio (MIT) |
|
|
|
|
|
|
|
|
|
|
|
R |
|
Adjusted |
Std.
Error of the Estimate |
|
|
0.240577768 |
0.057878 |
0.022036 |
1.647649 |
|
|
|
|
|
|
|
|
|
Unstandardized
Coefficients |
Standardized
Coefficients |
Sig. |
|
|
|
B |
Std.
Error |
Beta |
|
|
(Constant) |
6.019 |
0.595 |
|
0.000 |
|
disaeye2 recoded disabeye (Disability: impaired
eyesight) to set 4 cases coded as no disab and eye disab to missing |
-0.329 |
0.231 |
-0.060 |
0.154 |
|
disabhea Disability: Impaired hearing |
0.063 |
0.462 |
0.006 |
0.892 |
|
disaoth2 combined other disability (dyslexia,
tremors, wheelchair, cognitive, other) (can include eyes and ears) |
0.195 |
0.386 |
0.022 |
0.613 |
|
readdiss Have you read about or discussed voter
verification systems with anyone |
-0.271 |
0.151 |
-0.076 |
0.074 |
|
needveri Necessity of voter verification systems |
0.070 |
0.051 |
0.059 |
0.167 |
|
prevvote Have you previously voted in an election? |
0.000 |
0.223 |
0.000 |
0.999 |
|
prevuse1 Have you previously used a touch screen
voting system? |
0.071 |
0.154 |
0.021 |
0.645 |
|
prevuse5 Have you previously used a
"Paper" voting system? |
-0.316 |
0.186 |
-0.075 |
0.090 |
|
atmuse Have you ever used an ATM to do your
banking? |
0.109 |
0.304 |
0.017 |
0.720 |
|
compuse How often, on average do you use a
computer? |
-0.134 |
0.066 |
-0.102 |
0.042 |
|
educatio What is the highest level of education you
have completed? |
-0.116 |
0.057 |
-0.099 |
0.041 |
|
age How old are you? |
-0.027 |
0.067 |
-0.021 |
0.687 |
|
english language speaks most regularly dummy |
0.108 |
0.332 |
0.014 |
0.746 |
|
gender Are you male, female? |
-0.093 |
0.146 |
-0.027 |
0.524 |
|
black_nonhisp black only & non-hispanic |
-0.413 |
0.163 |
-0.121 |
0.012 |
|
hispanic_rc other race and hispanic, excluding 1 white
& 1 black, to use as hispanic variable |
-0.949 |
0.521 |
-0.078 |
0.069 |
|
income_rc income recoded with dk as missing |
-0.002 |
0.045 |
-0.002 |
0.966 |
|
pstren partisan strength |
-0.106 |
0.102 |
-0.072 |
0.300 |
|
democrat democratic PID 1 thru 3 |
0.248 |
0.304 |
0.072 |
0.415 |
|
repub republican PID 5 thru 7 |
0.049 |
0.314 |
0.012 |
0.876 |
|
order_a |
0.045 |
0.050 |
0.038 |
0.367 |
|
Dependent
Variable: indexa Audio: index of
record, count, and privacy |
||||
|
|
|
|
|
|
|
N = 574 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Table A18. No verification unit (Diebold AccuVote
TS) |
|
|
||
|
|
|
|
|
|
|
R |
|
Adjusted |
Std.
Error of the Estimate |
|
|
0.2844319 |
0.080902 |
0.041512 |
1.508558 |
|
|
|
|
|
|
|
|
|
Unstandardized
Coefficients |
Standardized
Coefficients |
Sig. |
|
|
|
B |
Std.
Error |
Beta |
|
|
(Constant) |
5.381 |
0.593 |
|
0.000 |
|
disaeye2 recoded disabeye (Disability: impaired
eyesight) to set 4 cases coded as no disab and eye disab to missing |
0.030 |
0.223 |
0.006 |
0.893 |
|
disabhea Disability: Impaired hearing |
0.407 |
0.445 |
0.043 |
0.361 |
|
disaoth2 combined other disability (dyslexia,
tremors, wheelchair, cognitive, other) (can include eyes and ears) |
-0.298 |
0.412 |
-0.033 |
0.469 |
|
readdiss Have you read about or discussed voter
verification systems with anyone |
-0.506 |
0.147 |
-0.154 |
0.001 |
|
needveri Necessity of voter verification systems |
0.076 |
0.051 |
0.068 |
0.132 |
|
prevvote Have you previously voted in an election? |
-0.070 |
0.214 |
-0.017 |
0.744 |
|
prevuse1 Have you previously used a touch screen
voting system? |
0.090 |
0.152 |
0.029 |
0.553 |
|
prevuse5 Have you previously used a
"Paper" voting system? |
-0.178 |
0.179 |
-0.046 |
0.320 |
|
atmuse Have you ever used an ATM to do your
banking? |
0.365 |
0.297 |
0.061 |
0.218 |
|
compuse How often, on average do you use a
computer? |
-0.127 |
0.065 |
-0.101 |
0.050 |
|
educatio What is the highest level of education you
have completed? |
-0.093 |
0.055 |
-0.084 |
0.095 |
|
age How old are you? |
-0.054 |
0.064 |
-0.046 |
0.405 |
|
english language speaks most regularly dummy |
0.403 |
0.333 |
0.054 |
0.227 |
|
gender Are you male, female? |
0.124 |
0.143 |
0.039 |
0.385 |
|
black_nonhisp black only & non-hispanic |
-0.122 |
0.158 |
-0.038 |
0.438 |
|
hispanic_rc other race and hispanic, excluding 1 white
& 1 black, to use as hispanic variable |
-0.780 |
0.530 |
-0.067 |
0.142 |
|
income_rc income recoded with dk as missing |
0.099 |
0.043 |
0.114 |
0.023 |
|
pstren partisan strength |
0.117 |
0.098 |
0.087 |
0.233 |
|
democrat democratic PID 1 thru 3 |
-0.272 |
0.288 |
-0.086 |
0.346 |
|
repub republican PID 5 thru 7 |
0.284 |
0.300 |
0.072 |
0.344 |
|
order_c |
0.009 |
0.048 |
0.008 |
0.852 |
|
Dependent
Variable: indexc Control: index of
record, count, and privacy |
||||
|
|
|
|
|
|
|
N = 512 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Questionnaires Used in the Field Test
[Diebold AccuView]
Acknowledgements: We thank the following organizations for allowing us to
field test the voting systems on their premises: Harmony Hall (
Organizations
declining to participate:
Research
Team
Paul
S. Herrnson, Director, Center for American Politics and Citizenship and
Professor, Dept. of Government and Politics,
Benjamin
B. Bederson, Director, Human-Computer Interaction Lab, Associate Professor, Dept.
of Computer Science, Univ. of
Charles
D. Hadley, Research Professor and Chair, Department of Political Science,
Richard
G. Niemi, Professor, Department of Political Science,
Michael
J. Hanmer, Assistant Professor, Department of Government,
CAPC Staff: Mario P. Stylianou, Jr., Elizabeth A. Bentley-Smith,
James B. Kelley, Nathan S. Bigelow, Gabriel Ferreyros, Lauren Azebu, Joshua
Kaminski, Kelly Zavala, Eric Shutt, Matthew Stempek, Matthew Verghese, Juliana
Menasce-Horwitz, Seth Petersen (Georgetown University).
HCIL Staff: Aaron Clamage, Kiki Schneider.
About the
Center for American Politics and Citizenship
The
Center for American Politics and Citizenship (CAPC) provides citizens and
policy-makers with research on critical issues related to the nation's
political institutions, processes, and policies. These include election
administration, election campaigns, money and politics, political debates, and
voting technology and ballot design. CAPC is a nonpartisan research institution
within the Department of Government and Politics at the
About the
Human-Computer Interaction Lab
The
Human-Computer Interaction Lab (HCIL) at the