The Dirty Little Secrets of Voting System Testing Labs by Avi Rubin

Reproduced under the Fair Use exception of 17 USC § 107 for noncommercial, nonprofit, and educational use.

| Chapter 4 — Trust Our Election Officials? |

| Next — U.S. Bars Colorado Lab From Testing Electronic Voting Machines by Christopher Drew |

December 16, 2005 — A couple of weeks ago, I spoke at a voting system testing summit hosted by the Secretary of State of California, Bruce McPherson. It was an event that included members of the US Election Assistance Commission, Secretaries of State, local election officials, vendors, voting machine testers, representatives from NIST, social scientists who study voting issues, and computer scientists, such as myself.

Most notable by their absence were Wyle Laboratories and Ciber Inc. Let me explain.

Before election officials can purchase voting systems, those systems need to be certified by a federally accredited lab called an Independent Testing Authority (ITA). There are three such labs in the US: Ciber, Wyle Labs, and Systest. These labs are tasked with testing any proposed voting systems against federal standards, in this case, the 2002 federal standards, soon to be replaced by the 2005 voluntary voting system guidelines (VVSG). You would think that these labs would be very interested in attending a summit such as this, and in fact, they were all invited. Only Systest showed up.

There were several overriding themes that emerged at the voting systems testing summit. Perhaps the most prevalent one was that the ITAs consistently decline to appear at these meetings. Why? Well the main reason is that they are fraught with conflict of interest and incompetence. In fact, had they shown up, they would have been raked over the coals by some of the voting system examiners that attended the summit. For instance, an examiner from Pennsylvania wanted to know how come so many systems that passed the ITA testing still had serious security and even operational flaws. The Systest representative, who had the misfortune of representing his entire industry alone, replied that they were only required to test against the standard. When pressed about whether or not the ITAs would fail a system if a serious flaw was found, the reply was that a memo would be written, but that the system would still pass. I couldn't believe it. The company that was tasked with certifying machines for elections in the United States would still pass them, even if a serious flaw was found, as long as the machine did not violate any aspects of the standard. Unbelievable.

Now, let me talk a bit about the conflict of interest. As a friend of mine put it, the ITAs are not independent and they have no authority. So Independent Testing Authority is a misnomer. Thankfully, NIST is going to change the name next year. Here's where it gets bad. The ITAs are hired by and paid by — the vendors. That is, when a vendor has a voting machine that they want certified, they find an ITA who is willing to certify the voting machine. Any memos about flaws that are discovered remain confidential. There is no requirement to disclose any problems that are found with the machines. In fact, the entire ITA report is considered proprietary information of the voting machine vendor. After all, they paid for it. This provides an incentive for ITAs to certify machines, to satisfy their clients.

Two years ago, my research team got our hands on the code that runs inside of Diebold's AccuVote machines. We performed a source code analysis and reported all kinds of serious security problems. It was incredible to me that such machines were actually deployed and used in elections. Equally confounding was that a national testing lab, in this case Wyle Labs, actually certified this machine. Either they did not know the first thing about cryptography and security, or they did not look at the source code. In fact, according to the 2002 standards, they were not required to examine the code.

So, the current state of affairs is grim. The ITA model provides an incentive to certify bad systems, and clearly such systems are being certified all the time. When the ITAs find a serious problem, it is relayed, confidentially to the vendor, and the only thing that the public ever learns is that a machine was certified. If a machine is not certified, nobody ever learns about it. The ITAs are aware enough of how broken the system is that they mostly hide from public events where they might be taken to task.

Here's how I would reform the system. First off, I would have all the vendors pay a tax to NIST. NIST would then hire real independent testers to examine any voting machine proposed by a vendor. The testers would be paid more for finding problems with the machines than for certifying them. Thus, you can be sure that the testers tried every way of failing a machine before passing it. Everything done by the testers, every test performed, as well as the result, would be public. Occasionally, to keep the testers on their toes, NIST would throw a machine at the testers with a known serious problem, just to see if the testers could find it, and testers who did not find the problem would be penalized. The whole process would be open and transparent to the public. I doubt systems such as the 2003 Diebold AccuVote would have ever made it to a polling station in that model.

I learned a lot at the voting system testing summit, and I applaud Secretary McPherson for the dialogue that he opened up. I sincerely hope that in such events in the future, there will be no stakeholders who are afraid or ashamed to show their faces.

Top

| Chapter 4 — Trust Our Election Officials? |

| Next — U.S. Bars Colorado Lab From Testing Electronic Voting Machines by Christopher Drew |

| Back — Almost Heaven, West Virginia |

Added August 6, 2006

Last modified 6/14/09