American Statistical Association Endorses Risk-limiting Audits of Election Results
Read the ASA press release here.
American Statistical Association Statement on Risk-Limiting Post-Election Audits
Poorly marked ballots, computer glitches, and voting system configuration errors can make machine vote counts diverge from voters’ intentions. By comparing hand counts of randomly selected ballots with machine tallies, we can judge whether a full hand count would show the same winners. Such audits can improve trust in our elections. Statisticians can help design efficient audits that save taxpayers’ money and election officials’ time.
Many states now require auditing a fixed percentage of all ballots. While any auditing is good, fixed-percentage audits frequently select more ballots than needed – and sometimes not enough – to provide strong evidence that the machine-count1 winner(s) truly received the most votes. In contrast, “risk-limiting” audits are designed to always have an acceptably small probability of failing to correct a wrong machine-count outcome. When a machine-count outcome is correct, a risk-limiting audit can often confirm the result after examining only a small fraction of the ballots cast.
The American Statistical Association (ASA) recommends that risk-limiting audits be routinely
conducted and reported in all federal, most state-wide, and at least a sampling of other governmental election contests. The ASA urges state and local officials to seek statistical advice on how to sample and analyze data to efficiently attain the desired level of risk control.
This statement builds upon previous actions of ASA’s Board of Directors. In March 2008, the Board issued a position statement2 on election integrity, saying (in part) that “… the integrity of central vote tabulations [must] be confirmed by audits of voter-verified hardcopy records … to provide high – and clearly specified – levels of confidence in electoral outcomes.” In August 2008, the Board endorsed the statistics-relevant parts of Principles and Best Practices for Post-Election Audits.3
Risk-Limiting Audits and Sampling
Post-election audits generally rely on random sampling of batches of ballots, with requirements:
• Sampling occurs after machine-count batch totals have been reported
• Batches and ballots to be selected are not known in advance
• All ballots are taken into account
To be risk-limiting, the overall procedure must ensure that if the machine-count electoral outcome is incorrect, there is a large, pre-specified chance that the audit will reveal the correct outcome. For further discussion and requirements, see Principles and Best Practices for Post-Election Audits.2
Credible audits are not possible if votes are cast and counted only electronically because there is no original record of voter intent with which to compare. An audit based on examining a sample of paper records is a good check on the machine tallies because the two ways of counting are vulnerable to different kinds of errors.
Checking a non-random sample only tells us about these particular ballots; it says nothing about the remaining ones. In contrast, the errors found in a random sample allow inferences about errors in not-yet-examined ballot batches. Statistics provides tools for evaluating the strength of the evidence that an outcome is correct despite errors found.
The risk we seek to limit is the chance of certifying the machine-count outcome of any contest for
which a full hand count would yield a different outcome. Statistics can be used to design audits that meet the desired risk threshold efficiently. The total number of ballots that must be examined to confirm a correct machine-count outcome generally increases with 1) a smaller margin of victory, 2) larger batch sizes (see below), and 3) the number of errors present.
For simplicity and efficiency, a risk-limiting audit can be conducted in stages. First, choose a large enough sample so that if the sampled hand counts and machine counts are sufficiently similar, we have strong evidence that the machine-count outcome is correct. If so, stop; if not, sample more. The audit protocol specifies rules for sampling and deciding whether to stop or to continue, so that we (eventually) either 1) determine that there is an acceptably low risk that the outcome is incorrect, or 2) count all the votes by hand, thereby directly determining the winner.
Sampling Small Batches
A batch is a group of ballots (or even one ballot) for which machine and hand counts can be
compared. Currently, batches often correspond to precincts or groups of paper ballots that are
counted and stored together. Generally, using smaller batches reduces the number of ballots that must be counted to confirm the machine-count outcome when that outcome is in fact correct.
Thus, to improve the efficiency of audits, we urge vendors and election officials to work together so that new voting equipment allows votes to be counted in arbitrarily small batches, down to individual ballots. Meanwhile, we should use the smallest feasible batches, taking care to maintain ballot secrecy. For example, a precinct that uses several machines should report totals separately for each machine, rather than as a single batch.
Timely, detailed audit findings should be reported in meaningful groups (such as, by precinct and
machine) in standard, machine-readable formats. All new electoral machinery should be able to
report the data needed to identify changes that can continuously improve our elections.
Designing and Legislating Risk-Limiting Audits
Both the science of auditing and the mechanics of elections continue to evolve. Thus, statutory audit requirements should focus on specifying good approaches and principles. These include:
• Comparing counts from hand-tallied batches of paper records with the contribution of the
same ballots to the machine-count outcome
• Requiring additional counting (even a full hand count) if needed to limit the risk of failing
to correct a wrong outcome
• In the event of a full hand count, clarifying that the hand count determines the outcome
• Reporting machine counts and hand counts for the audited batches in standard, machine-readable format (small batch totals might be combined in public reports to protect voter
anonymity)
• Ensuring verifiability. For example, publishing source code for algorithms used in audit
calculations allows the calculations to be replicated; publishing machine counts for all
batches prior to their being sampled demonstrates that the numbers that the audit checks are
the same as the subtotals included in the machine-count outcome.
Statisticians should be involved in designing audit protocols that address state election needs. Draft protocols should be open to comment with adequate time to accommodate the reasonable concerns of election officials, candidates, and other citizens. The goal is to conduct risk-limiting post-election audits that provide confidence that certified electoral outcomes match what would have resulted from a full hand-count of the ballots.
1 The machine-count is the outcome that will become official unless a full hand count determines it is wrong.
2 http://www.amstat.org/about/pressreleases/asacallsforaudits.pdf
3 http://electionaudits.org/principles
