The Cupertino, Calif. company will no longer seek the VB100 certification, Raimund Genes, chief technology officer for Trend Micro, told SecurityFocus on Friday. The certification, which is administered by antivirus-industry watcher Virus Bulletin, tests whether antivirus software can detect a small set of viruses encountered by experts on the Internet, known as the WildList, without flagging non-viruses as malicious.

The problem, Genes said, is that the certification ignores the fact that, at present, the most significant threats are not viruses, but Trojan horse programs and bot software. In addition, antivirus software updates itself over the Internet, but the testing does not allow Internet connectivity for safety reasons, Genes said. Finally, companies have to deal with a massive influx of new threats every year -- in 2007, the number of malware variants topped 500,000 -- but the test only checks for fewer than 1,000 threats.

"I'm okay being tested against a million pieces of malware, because it gives me an idea of where the product stands," Raimund Genes, chief technology officer for Trend Micro, told SecurityFocus. "But I am not okay being tested against 700 outdated pieces of malware."

In response to Trend Micro's statements, Virus Bulletin underscored that the VB100 certification is not intended to measure a product's performance against the largest number of computer threats, but to set a baseline for antivirus products to regularly exceed.

"We have a simple test requirement of detecting the full WildList without false positives," John Hawes, technical consultant for Virus Bulletin, said in an e-mail interview with SecurityFocus. "By monitoring a product's performance over time our results should give an idea of the ongoing competency and reliability of the vendor."

If a product fails to detect any of the viruses from the WildList, which included 678 viruses in April, or categorizes a non-virus as a threat, the software does not receive certification. While vendors occasionally miss spotting a tricky signature or mistakenly flag a harmless file as a virus, regularly passing the test "is the mark of a well-maintained product," Hawes said.

The pass-fail nature of the test has angered software companies in the past. Most notably, antivirus firm Panda has not tested for certification since 2002. Trend Micro has failed to pass its last three VB100 certification tests, starting with a single case of mistaking a non-malicious file as a virus -- known as a false positive -- in August 2007 and culminating in missing three WildList viruses and two false positives in the most recent test in April 2008. Trend Micro and Panda compete with Symantec, the owner of SecurityFocus, in the antivirus software market.

Concerns about the manner in which testing firms evaluate anti-malware software led antivirus companies to make the debate a major topic last year at a conference in Reykjavik, Iceland. In January, security software makers and independent and media-sponsored testing labs agreed to create an industry group the Anti-Malware Testing Standards Organization (AMTSO) to establish best practices and standards in the testing and rating of antivirus software.

The concerns are not new. Writing about the discussions last year, Randy Abrams, director of technical education for antivirus firm Eset, argued that the time has come for antivirus software tests to evolve beyond the WildList.

"Agreement was virtually unanimous that the WildList is no longer useful as a metric of the ability of a product to protect users," he wrote in June 2007 issue of Virus Bulletin. "The Wild List brought a standard of scientific repeatability and credibility to testers however, if the sentiments of test and research alike are to be acted upon, the WildList will evolve or die."

Yet, Abrams stressed that while there are questions about using the WildList as the sole dataset for the test, the certification is still relevant.

"A single success in passing the Virus Bulletin 100 does not mean anything," Abrams said. "A string of successes typically means that you are paying attention to quality control."

While VB100 certification is a bit like taking a test to which you already had the answer key, companies do fail to get certified occasionally, said Abrams. In the best known recent case, Microsoft failed to gain certification for Windows OneCare in February 2007, but has passed its last two certification tests, according to Virus Bulletin.

"When it happens repeatedly, that gets embarrassing," Abrams said. "When you fail a test, you have customers that call up and ask what is going on."

Abrams' company, Eset, recently boasted in a press release that it had attained its 50th VB100 certification.

Major antivirus-software makers agreed that the testing of their products leaves something to be desired, but none of the companies planned to join Trend Micro in boycotting the VB100 certification. Microsoft stated that the company would continue to test its products for the certification. McAfee, whose VirusScan Enterprise failed its latest test because it missed a single WildList entry, will continue to apply for the certification, a company spokesperson told SecurityFocus.


"The correlation between all tests and real-world performance has dropped in recent years due to the scale of the malware problem," the spokesperson said. "It is unlikely future tests will provide perfect results, but we're working with the anti-malware and testing industry to help improve the situation (as much as) possible."

Symantec, the owner of SecurityFocus, will also continue to submit its products for VB100 certification, the company said in a statement. The security software maker last failed VB100 certification in September 1999.

"Our hope is that current tests will adopt the newer test methodologies on top of their existing test strategies and so give a more comprehensive picture of product protection effectiveness," the spokesperson said.

Virus Bulletin has already taken the industry's criticism to heart and the certification will evolve in the coming months, said Hawes. The WildList, on which the test is based, will begin to include Trojan horse programs encountered in the wild to better reflect the current landscape of threats. However, Virus Bulletin's small size and amount of work required to overhaul the tests have made progress slow, he said.

"We are certainly aware that there are a range of important factors which are not covered by our current testing regime," Hawes said. "Plans to expand and improve the information we offer have been underway for some time."

Until more changes are incorporated into the certification, it's unlikely that Trend Micro will return to testing, the company's Genes said.

"A lot of people have asked them to change, and they haven't," he said. "So we think it is totally meaningless for our customer base to continue testing."

If you have tips or insights on this topic, please contact SecurityFocus.