Suprising results over which is the best AV

[zero2]

Kobra's Antivirus SHOWDOWN results.

Comments: I found this posted in another forum, and thought it might be useful here. Please read the entire article which can be found @ the bottom of this article.


Kobra's 6-14-04 AV Test

Premise of test:

I test because I want to know what works and what doesn't and whats hype and whats real. I'm sick of the hype and marketing BS some of these AV companies are throwing out. Talk of "100% never failed on this test" and "Fastest and most accurate scanners" and other crap, and frankly, its just that, crap. My opinion, take it or leave it.

Testbed consisted of 321 Viruses, Trojans and Worms, all for the Windows32 environment, and all reasonably new samples. I don't have any data on whether some of these are zoo, or ITW, but they are all real threats I feel someone is likely to encounter, since I got them off the internet (and i've verified they are real as each sample must be detected by at least 4 AV's for me to consider it). All scanners were installed on a clean system, without any traces of other anti-virus softwares - between each test the system and directories were cleaned, and the registry was sweeped. Each AV product was treated with a double-reboot, one before, and one after installation. Each scanner was set at its highest possible settings, and was triple checked for proper options and configuration. Most products were the full registered version when possible, others were fully functional unrestricted trials. All products were tested with the current version as of 6-14-04, and the latest definitions for that date. Each product was run through the test set a minimum of 3 times to establish proper settings and reliability, the only product to exhibit some variance on this was F-Secure, which had one scan come up less than the other two without any settings changes indicating a possible stability issue.

The final standings:

1) eXtendia AVK
2) McAfee VirusScan 8.0
3) F-Secure
4) Kaspersky 5.0
5) GData AVK
6) RAV + Norton (2 way tie)
7) Dr.Web
8) CommandAV + F-Prot + BitDefender (3 Way Tie)
9) ETrust
10) Trend
11) Panda
12) Avast! Pro
13) KingSoft
14) NOD32
15) AVG Pro
16) AntiVIR
17) ClamWIN
18) UNA
19) Norman
20) Solo
21) Proland
22) Sophos
23) Hauri
24) CAT Quickheal
25) Ikarus

Heuristics seemed to play some of a roll in this test, as no AV had every virus in my test in their definitions, and products with stronger heuristics were able to hold their position towards the top of the test. Double/Multi engined products put up strong showings as well, proving to me that the redundacy method works, and I think more AV companies should considering double-engines. The strongest heurisitical AV I noticed was F-Prot/Command, picking up only 247 samples with definitions but they were able to power through 67 additional hits on "Possible Virus" indicators - very strong! Norton with BloodHound activated had 30 Heuristical pickups, and DrWeb rounded up the pack with 20 heuristical pickups. eXtendia AVK grabs the number one slot with double engine scanning, anything the KAV engine missed, the RAV engine picked up with great redundancy on the double engine/definition system. McAfee actually missed only 2 samples with its definitions, but picked those 2 up as "Suspicious File", and therefore, scores nearly perfect as well.

The biggest dissapointments for me were Norman and Nod32. Even with Advanced-Heuristics enabled, NOD32 failed to pick up a large portion of the samples. Norman, while finding some of the toughest samples, managed to completely miss a large portion of them! Showing that their sandbox-emulation system has great potetential, but its far from complete.

Actual test numbers were:

Total Samples/Found Samples (321 total possible) + Number Missed + Detection Percentage

1) eXtendia AVK - 321/321 0 Missed - 100%
2) McAfee VirusScan 8.0 - 319/321 + 2 (2 found as joke programs - heuristically) - 100%
3) F-Secure - 319/321 2 Missed - 99.37%
4) Kaspersky 5.0 - 318/321 3 Missed - 99.06%
5) GData AVK - 317/321 4 Missed - 98.75%
6) RAV + Norton (2 way tie) - 315/321 6 Missed - 98.13%
7) Dr.Web - 310/321 11 Missed - 96.57%
8) CommandAV + F-Prot + BitDefender (3 Way Tie) - 309/321 12 Missed - 96.26%
9) ETrust - 301/321 20 Missed - 93.76%
10) Trend - 300/321 21 Missed - 93.45%
11) Panda - 298/321 23 Missed - 92.83%
12) Avast! Pro - 292/321 29 Missed - 90.96%
13) KingSoft - 288/321 33 Missed - 89.71%
14) NOD32 - 285/321 36 Missed (results identical with or without advanced heuristics) - 88.78%
15) AVG Pro - 275/321 46 Missed - 85.66%
16) AntiVIR - 268/321 53 Missed - 83.48%
17) ClamWIN - 247/321 74 Missed - 76.94%
18) UNA - 222/321 99 Missed - 69.15%
19) Norman - 215/321 106 Missed - 66.97%
20) Solo - 182/321 139 Missed - 56.69%
21) Proland - 73/321 248 Missed - 22.74%
22) Sophos - 50/321 271 Missed - 15.57%
23) Hauri - 49/321 272 Missed - 15.26%
24) CAT Quickheal - 21/321 300 Missed - 6%
25) Ikarus - Crashed on first virus. - 0%

Interesting also to note, is the detection level of the US AVK version with KAV+RAV engines was higher than the German version with KAV+BitDefender engines. Several vendors have free versions of their for purchase AV's, we didn't test the free versions, as it would serve no purpose for this test, but based on the results, none of the free versions would have been very impressive anyway. The term "Heuristics" seems like it should be taken very liberally, as some products that claim to be loaded with Heuristics scored miserably on items they clearly didn't have definitions for. Scanning speed was not measured, as it was totally irrelevant to my testing, and on-access scanners were not tested, as it would have been too time consuming, but considering most products have similar on-access engines as on-demand, and use the same database, results most likely, would be very similar.

Cut through the hype, cut through the marketing schemes, this was a real test, with real samples, and none of these samples were provided to the antivirus software vendors in advance. This is real world, and these are likely badguys you'll encounter, since I got them in my real encounters, and all were aquired on the internet in daily activities which anyone out there might be involved in. (Installing shareware, filesharing, surfing, etc). Keep in mind that with ITW tests the AV vendors have full disclosure of what they will be tested on in advance, not so here, so heuristics and real detection algorithms will play a big part, as well as the depth and scope of their definition database.

Honestly, I was *HOPING* to be surprised by a ton of things in this test, and really all I did was re-enforce many of the other testing sites on their results, mine are very close to theres, which actually shocked me, because i'm sure my samples aren't the same. This tells me overall, I think this might be a great guage of these products.

Also, I wanted to test the multi-engined products against the others, since most testers seem to not like testing them. Strong showings by F-Secure, and the AVK' brothers proved this idea works, and works incredibly well. The strenght of the KAV engine cannot be denied as well, since all but one of the top 5 products use the KAV engine. I forgot to add, one product I tested was called V-Catch, and turned out to be a trojan downloader and spyware application masking as a AV product.. LOL! Thankfully it was the last product I tested, and I just reformatted, I think it downloaded 30 trojans to my system. 8-)

I did NOT test any Dos viruses, as this is completely retarded to test these in a windows based environment, it tells us nothing. I cannot understand why Clementi bothers to test them, all they do is skew his test results badly. For example on his test, NOD32 scored 95.51%, but without DOS or other OS samples, NOD32 scored only 87.71%. Which amazingly enough, is within 1% variance of *MY* results. So i'm oblivious as to why he skews his own results for no real purpose? Who the hell cares what a product scores on DOS?!?


Update:

Greetings folks. There have been some updates to the test results. I've re-tested a few products with different switches and settings on the prodding of the users of those products.

* CAT Quickheal was retested, and yielded the same results, the developer is investigating.

* Avast was retested with different command line switches, and improved to 299 Detections. Bumping Panda down one notch.

* Kaspersky5.0 was re-tested with "Extended Database" downloaded and in place, and was upgraded with only 1 miss.

*Ahn V3 Pro was requested to be tested, I tested it, and it detected only 109 out of the 321 samples for a 33.95% rating. (although the options and interface rocked. Heh)

1) eXtendia AVK - 321/321 0 Missed - 100%
2) Kaspersky 5.0 - 320/321 1 Missed - 99.70% (with Extended Database ON)
2) McAfee VirusScan 8.0 - 319/321 + 2 (2 found as joke programs - heuristically) - 99%
3) F-Secure - 319/321 2 Missed - 99.37%
4) GData AVK - 317/321 4 Missed - 98.75%
5) RAV + Norton (2 way tie) - 315/321 6 Missed - 98.13%
6) Dr.Web - 310/321 11 Missed - 96.57%
7) CommandAV + F-Prot + BitDefender (3 Way Tie) - 309/321 12 Missed - 96.26%
8) ETrust - 301/321 20 Missed - 93.76%
9) Trend - 300/321 21 Missed - 93.45%
10) Avast! Pro - 299/321 22 Missed - 93.14%
11) Panda - 298/321 23 Missed - 92.83%
12) KingSoft - 288/321 33 Missed - 89.71%
13) NOD32 - 285/321 36 Missed (results identical with or without advanced heuristics) - 88.78%
14) AVG Pro - 275/321 46 Missed - 85.66%
15) AntiVIR - 268/321 53 Missed - 83.48%
16) ClamWIN - 247/321 74 Missed - 76.94%
17) UNA - 222/321 99 Missed - 69.15%
18) Norman - 215/321 106 Missed - 66.97%
19) Solo - 182/321 139 Missed - 56.69%
20) V3 Pro - 109/321 212 Missed - 33.95%
21) Proland - 73/321 248 Missed - 22.74%
22) Sophos - 50/321 271 Missed - 15.57%
23) Hauri - 49/321 272 Missed - 15.26%
24) CAT Quickheal - 21/321 300 Missed - 6%
25) Ikarus - Crashed on first virus. - 0%

Ironically, since doing this test, i've checked around, and my results aren't really all that different than some of the independant test houses. What *IS* very different, are how places like Virus Bulletin can rate something like CAT Quickheal at having 90% range detection, when it can't find the most simplest bug I throw at it. Ironically, the most recent test of NOD32 scored it in the 80 percentile range, exactly where it fell in my testing.

As for eXtendia AVK, its quite a feature rich and configurable product, and probably offers the only true 100% detection ability out there. Both engines cranking away, sweeping everything, and with each engine having its own 100,000ish definition database comparing with each other, double heuristics doing comparatives.. I think its pretty safe to say, your chances of infection are zero? I've personally run into a couple of bugs that Kaspersky missed, and the RAV side of AVK picked it up. So I do believe the product works, and provides an incredible level of layered protection for its price.

Keeping in mind though, AVK does use the KAV+RAV engine, and you can run either/or in any configuration you want, so for example you could run RAV for on-access to get the speed, and run KAV+RAV Double-Mode for on-demand to get the incredible depth of scans. Also, my testing showed AVK updates directly from their sources, i've seen it update 8-10 times per day if you set it to "Hourly" in the configurations. But I will tell you this, after running NOD32 for 3 months, I installed AVK and found 5 trojans - so don't be surprised if it finds something on your box if you were running one of the other AV's.

I'd say my personal choices are KAV5.0 or AVK. I can't wait to see what KAV5-Pro looks like, its not due till September though.

Regards

PS: I don't test for a living, this was done to satisfy my OWN curiosity about which AV product would be best for me to run. I got tired of paying cash for products like Norton and NOD32 and being horribly dissappointed - and in fact being left to reformat due to infections! Also, i'm a bit sick of magazines and websites talking really lousy products. So much misleading information out there its quite annoying. Agree with my test results or not, its up to you, but its strictly done to satisfy my own curiosity, and was posted merely because I thought others might benefit.

Tech support findings:

Yea, eXtendia AVK support definately needs work. They *DO* have a voice phone number, and i've heard registered users are responded to within 1-24 hours via email. The times i've emailed, i've usually recieved a reply back the same day, a couple times its dragged on and on.

Kaspersky has one of the best support of any AV product out there IMHO. I've recieved responses back in minutes at times, and usually by someone that knows what they are talking about, even if i'm not registered. Avast has similar levels of support.

So you really have to weigh it out.. If you really need fast support, get Kaspersky. If you can deal with off and on support, then maybe eXtendia would be fine for you. Theres *ALOT* to consider when buying an antivirus, and I don't pretend to have the answers.

Oh, I actually logged my support responses from AV companies over the last couple months.. Heres what I logged. =)

Kaspersky – Generally within 1-2 hours, seldom more than 12 hours.
Panda – 1-2 days
F-Secure – 7 Days (WTF?)
NOD32 – Varies between 3-5 days or NO RESPONSE (More common)
BitDefender – 6 Days!
Norman – 1-2 Hours, never more than 12 hours.
AVK – Varies, sometimes hours, sometimes days, sometimes never.
RAV – No more than 5 hours delay. (too bad their product isn’t sold anymore)
BOClean – 1-2 Hours, sometimes LESS, never more than 5 hours.
McAfee – I've yet to get a response whatsoever, its been 3 weeks.
Dr.Web – 4-7 Days Wait, usually a Russian response, never helpful. Last response I got, a guy said "Who u? say self or no help)
Avast – 1-12 hours usually, they even reply to virus submittals!

I tried eScan, and it caused me to reformat. So I tried it again, and had to reformat a second time. That product is an accident waiting to happen! Anyway, I got some updates:

Latest Update

Updated testing results, several additional products tested. Special note to the changes in first place. Notes on the changes:

Discovered and tested MKS-Vir2004, from Poland. Surprisingly, this one with caught every sample perfectly on Medium Heuristics. Specifically, nearly 50 samples were picked up Heuristically giving it a perfect score of 321/321. However, when I increased Heuristics to "Super Deep", it picked up an addition 10 more suspicious files. Upon further investigation, it was found that it was picking up signatures of hacktool utilities left over in some of the archives and flagging those files. Indeed, this is impressive. MKS-Vir2004 exhibits the most advanced detection algorithms i've ever seen, clearly it only had signatures for 271 of my samples, but through code emulation, it was able to pick up all 321 samples!! It clearly labeled the Heuristically found ones as things as "Likely Win32 Trojan" or "Highly Suspicious Acting File". In addition, its scanning speed was incredibly quick, and its memory footprint was quite small. Impressive! Furthermore, this is a full featured and fairly polished product that appears to update at least once per day, and tech support responded to me within 5-15 minutes on my emails. Unfortunately, it appears to not be available in the US for purchase at this time.

Tested other additional products, Antidote, PerAV, Vir.IT, FireAV, and VirusBuster. Results are below.

1a) MKS_Vir 2004 - 321/321 0 Missed - 100%
1b) eXtendia AVK - 321/321 0 Missed - 100%
2a) Kaspersky 5.0 - 320/321 1 Missed - 99.70% (with Extended Database ON)
2b) McAfee VirusScan 8.0 - 319/321 + 2 (2 found as joke programs - heuristically) - 99%
3) F-Secure - 319/321 2 Missed - 99.37%
4) GData AVK - 317/321 4 Missed - 98.75%
5) RAV + Norton (2 way tie) - 315/321 6 Missed - 98.13%
6) Dr.Web - 310/321 11 Missed - 96.57%
7) CommandAV + F-Prot + BitDefender (3 Way Tie) - 309/321 12 Missed - 96.26%
8) ETrust - 301/321 20 Missed - 93.76%
9) Trend - 300/321 21 Missed - 93.45%
10) Avast! Pro - 299/321 22 Missed - 93.14%
11) Panda - 298/321 23 Missed - 92.83%
12) Virus Buster - 290/321 31 Missed - 90.34%
13) KingSoft - 288/321 33 Missed - 89.71%
14) NOD32 - 285/321 36 Missed (results identical with or without advanced heuristics) - 88.78%
15) AVG Pro - 275/321 46 Missed - 85.66%
16) AntiVIR - 268/321 53 Missed - 83.48%
17) Antidote - 252/321 69 Missed - 78.50%
18) ClamWIN - 247/321 74 Missed - 76.94%
19) UNA - 222/321 99 Missed - 69.15%
20) Norman - 215/321 106 Missed - 66.97%
21) Solo - 182/321 139 Missed - 56.69%
22) Fire AV - 179/321 142 Missed - 55.76%
23) V3 Pro - 109/321 212 Missed - 33.95%
24) Per_AV - 75/321 - 246 Missed - 23.36%
25) Proland - 73/321 248 Missed - 22.74%
26) Sophos - 50/321 271 Missed - 15.57%
27) Hauri - 49/321 272 Missed - 15.26%
28) CAT Quickheal - 21/321 300 Missed - 6%
29) Vir_iT - 10/321 311 Missed - 3%
30) Ikarus - Crashed on first virus. - 0%


Source & Full Article: "Kobra's Antivirus Showdown results

[Flesh]

/cheer for kaspersky.. good show.

[yatzr]

good stuff. I'm interested in checking out that MKS_Vir 2004. It just sounds like a dream come true.

[Eric640]

Wow very informative post. Thanks

[Slavakion]

Thanks. I switched from Norton 2003 to Kaspersky (I tried to get eXtendia, but any serials were invalid. I'm looking for a "free" solution ). Kaspersky seems much more streamlined.

[yakimushi]

EXCELLENT post! Thanks!

[MichaelFarker]

This is a great article. Thank you for posting it here. I found it helpful and thought provoking.
Broadband, the site it was originally posted on, had some additional advice I found useful. http://www.dslreports.com/faq/8428

[Silverbrain]

Trojans are becomming a grey area it seems, with a lot of trojans also picked up as simple malware or spyware with adaware and other programs. I have been using symantec programs for years now, and I know they dont pick up everything, in fact at the comp store I work at I scan the systems with trendmicro's and symantec. I refuse to use Mcafee as they sell your email address and its the only way to get updates, and the fact their interface is far too bloated. I have never heard of the top 2 programs, but will look into them further. Thanks for the info )

[Mephisto2]

Originally posted by Slavakion
Thanks. I switched from Norton 2003 to Kaspersky (I tried to get eXtendia, but any serials were invalid. I'm looking for a "free" solution ). Kaspersky seems much more streamlined.

You want a good AV solution, to protect you against "bad people" on the internet who write viruses, yet you're perfectly willing to try to steal software?

Sheesh...

Mr Mephisto

[JStrider]

just got the kaspersky scanner...its still on my C drive and its already found 3 things symantec missed!!!!

crazy...

[tropple]

Thank you very much for posting this information. It's great.