========================================= File 7EVALW2k.TXT Evaluation of VTC Scanner Test "2001-10": ========================================= Formatted with non-proportional font (Courier) Content of this file: ===================== ********************************************************************** Eval W2k: Development of detection rates under Windows-2000: ********************************************************************** Eval W2k.01: Development of W-2000 Scanner Detection Rates Table W2k-A: Comparison Macro/Script virus detection rates Eval W2k.02: In-The-Wild Detection under W-2000 Eval W2k.03: Evaluation of overall W-2000 AV detection rates Eval W2k.04: Evaluation of detection by virus classes under W-2000 W2k.04.2 Grading the Detection of macro viruses under W-2000 W2k.04.3 Grading the Detection of script viruses under W-2000 Eval W2k.05: Detection of Packed Macro Viruses under W-2000 Eval W2k.06: Avoidance of False Alarms (Macro) under W-2000 Eval W2k.07: Detection of Macro Malware under W-2000 Eval W2k.SUM Grading of W-2000 products ********************************************************************** This part of VTC "2001-10" test report evaluates the detailed results as given in sections (files): 6hW2k.TXT File/Macro Viruses/Malware results W-2000 (W2k) The following *18* products participated in this scanner test for W-2000 products: -------------------------------------------------------- Products submitted for aVTC test under Windows-2000: -------------------------------------------------------- AVA v(def): 6.263 sig: unknown AVG v(def): 6.0.263 sig: June 22,2001 AVK v(def): 10.0.67 sig: June 21,2001 AVP v(def): 3.5.133.0 sig: June 01,2001 CMD v(def): 4.61.5 sig: June 21,2001 DRW v(def): 4.25 sig: June 20,2001 FPR v(def): 3.09d sig: June 25,2001 FPW v(def): 3.09d sig: June 25,2001 FSE v(def): 1.00.1251 sig: June 20,2001 scan eng fprot: 3.09.507 scan eng avp: 3.55.3210 scan eng orion: 1.02.15 INO v(def): 6.0.85 sig: June 14,2001 MR2 v(def): 1.17 sig: June 2001 MCV v(def): 0.5 sig: June 22,2001 NAV v(def): 4.1.0.6 sig: June 22,2001 NVC v(def): 5.00.25 sig: June 19,2001 PAV v(def): 3.5.133.0 sig: June 23,2001 RAV v(def): 8.2.001, scan eng:8.3 sig: June 25,2001 SCN v(def): 4144 scan eng:4.1.40 sig: June 20,2001 VSP v(def): 12.22.1 sig: June 25,2001 -------------------------------------------------------- Eval W2k.01: Scanner Detection Rates under Windows-2000: ======================================================== The number of scanners running under Windows 2000 is growing. Evidently, AV producers invest now more work into the development of the W32-related platforms, and here into the detection of macro viruses (with minor improvements) and script viruses (with major improvements). The following table summarizes results of file, macro and script virus detection under Windows-2000: Table W2k-A: Comparison: File/Macro/Script Virus Detection Rate: ================================================================ Scan I = File Virus = + ======= Macro Virus ======= + ===== Script Virus ======= ner I Detection I Detection I Detection -----+-----------------+-----------------------------+--------------------------- Test I 0104 Delta I 0008 0104 0110 Delta I 0008 0104 0110 Delta -----+-----------------+-----------------------------+--------------------------- ANT I - - I 93.3 - - - I 53.9 - - - AVA I 95.0 - I 94.1 95.7 97.7 +2.0 I 15.0 29.1 29.6 +0.5 AVG I 81.9 - I 97.9 98.3 98.4 +0.1 I 45.8 57.9 62.9 +5.0 AVK I 99.8 - I 100.0~ 100.0~ 100.0% 0.0 I 91.5 99.8 100.0% +0.2 AVP I 99.9 - I 100.0~ 100.0~ 100.0~ 0.0 I 88.2 99.8 100.0% +0.2 AVX I - - I 99.0 - - - I 61.4 - - - CLE I - - I - - - - I 4.2 - - - CMD I 97.8 - I 100.0% 100.0% 100.0~ 0.0 I 93.5 96.9 93.2 -3.7 DRW I - - I 97.5 - 99.5 - I 59.8 - 95.4 - FPR I 97.8 - I - 100.0% 100.0~ - I - 96.9 94.6 -2.3 FPW I 97.8 - I 100.0% 100.0% 100.0~ 0.0 I 90.8 96.9 94.6 -2.3 FSE I - - I 100.0% 100.0% 100.0% 0.0 I 96.7 100% 100.0% 0.0 INO I 97.9 - I 99.8 99.7 99.9 +0.2 I 78.1 93.1 93.9 +0.8 MCV I - - I - - 88.5 - I - - 27.7 - MR2 I - - I - - 0.7 - I - - 83.3 - NAV I 93.9 - I 97.7 97.0 99.5 +2.5 I 36.6 54.5 94.2 +39.9 NVC I 98.1 - I 99.9 99.8 99.8 0.0 I 83.7 88.5 91.3 +2.8 PAV I 97.5 - I 100.0~ 99.4 100.0% -0.6 I 90.2 98.5 100.0% +1.5 PER I - - I 85.0 68.2 - - I 0.0 22.0 - - PRO I 70.6 - I 69.1 67.1 - - I 12.1 40.7 - - QHL I - - I 0.0 - - - I 6.9 - - - RAV I 93.5 - I 96.9 99.6 99.5 -0.1 I 47.1 84.9 82.5 -2.4 SCN I 89.0 - I 100.0% 100.0% 100.0% 0.0 I 95.8 100% 99.8 -0.2 VSP I - - I - 0.0 0.~ 0.0 I - 85.3 84.0 -1.3 -----+-----------------+-----------------------------+---------------------------- Mean : 97.6% - I 99.9% 89.7 88.0% +0.3% I 57.6 79.4 84.8% +2.5% Without extreme results: (98.9%) I (91.9%)(-0.2%) -----+-----------------+-----------------------------+---------------------------- Remark: for abbreviations of products (code names), see appendix A5CodNam.txt. Concerning macro viruses, "mean" detection rate is slightly reduced "in the mean" to a still inacceptably low level (<90%) though there is some slight improvement (+0.3%) for those products which participated also in last test; when one does not count 2 products with extreme low detection arte (<30%), mean results are acceptable if not very good (98.9%). Now, 4 scanners detect ALL MACRO zoo viruses, and 4 more detect almost all. Concerning script viruses which is presently the fastest growing sector, detection rate is improving though still low (84.8% mean) but those (15) products which also participated in last VTC test have improved their detection rates; but the impressing figure (+2.5%) is essentially influenced by one porduct which upgraded its detection rate by 39.9% (NAV) to now reach 94.2%. Now, 4 products detect ALL script zoo viruses. **************************************************************** Findings W2k.1: For W-2000, macro and script zoo virus detection rates need further work. ------------------------------------------------ Concerning macro zoo viruses: 4 products detect ALL macro zoo viruses in all files and are rated "perfect": AVK,FSE,PAV,SCN 9 products detect almost all macro viruses in almost all files and are rated "excellent": AVP,CMD,FPR,FPW,INO,NVC,DRW,NAV,RAV ------------------------------------------------ Concerning script zoo viruses: 4 products detect ALL script zoo viruses in all files and are rated "perfect": AVK,AVP,FSE,PAV 2 product detects almost all script viruses in almost all files and is rated "excellent": DRW,SCN ------------------------------------------------- Overall: 3 W2k products now detect ALL zoo macro and script virus "perfectly" (100% detection rate): AVK,FSE,PAV 1 WNT product detects >99% of zoo macro and script viruses "excellently": SCN **************************************************************** Eval W2k.02: In-The-Wild (Macro,Script) Detection under W-2000 =============================================================== Concerning "In-The-Wild" viruses, the following grid is applied: - detection rate is 100% : scanner is "perfect" - detection rate is >99% : scanner is "excellent" - detection rate is >95% : scanner is "very good" - detection rate is >90% : scanner is "good" - detection rate is <90% : scanner is "risky" 100% detection of In-the-Wild viruses also esp. detecting ALL instantiations of those viruses is now ABSOLUTE REQUIREMENT, for macro and script viruses (it must be observed that detection and identification is not completely reliable). The following 9 W-2000 products (of 18) reach 100% for ITW macro and script virus detection both concerning viruses and infected objects (files) and are rated "perfect" in this category (alphabetically ordered): ITW Viruses&Files ( Macro Script) ------------------------- "Perfect" W2k ITW scanners: AVK ( 100.0% 100.0%) AVP ( 100.0% 100.0%) DRW ( 100.0% 100.0%) FSE ( 100.0% 100.0%) INO ( 100.0% 100.0%) NAV ( 100.0% 100.0%) NVC ( 100.0% 100.0%) PAV ( 100.0% 100.0%) SCN ( 100.0% 100.0%) Several more scanners detect all macro and script ITW viruses but not in all infected objects; the following are rated "excellent" as they detect at 100% ITW viruses and >99.9% infected files (both macro and script): "Excellent W2k ITW scanners: FPR ( 100.0% 100.0/99.1%) FPW ( 100.0% 100.0/99.1%) AVG ( 100.0% 100.0/97.2%) Only concerning macro ITW viruses, 14 (out of 18) scanners detect ALL macro ITW viruses in ALL files and are rated "perfect": ITW Viruses&Files ( Macro ) ----------------- AVG ( 100.0%) AVK ( 100.0%) AVP ( 100.0%) CMD ( 100.0%) DRW ( 100.0%) FPR ( 100.0%) FPW ( 100.0%) FSE ( 100.0%) INO ( 100.0%) NAV ( 100.0%) PAV ( 100.0%) SCN ( 100.0%) ----------------- And concerning script ITW viruses only, 9 (out of 18) products detect ALL script ITW viruses in ALL files and are rated "perfect": ITW Viruses&Files ( Script) ----------------- AVK ( 100.0%) AVP ( 100.0%) DRW ( 100.0%) FSE ( 100.0%) NAV ( 100.0%) NVC ( 100.0%) PAV ( 100.0%) SCN ( 100.0%) ----------------- **************************************************************** Findings W2k.2: Now 8 AV products (out of 18) detect ALL In-The-Wild macro and script viruses in >99.9% files: AVK,AVP,DRW,FSE,INO,NAV,NVC,PAV,SCN And 3 products can be rated "perfect" con- cerning detection of macro and script viruses but they still fail to detect all >99.9% of script viral files (objects): FPR,FPW,AVG **************************************************************** Eval W2k.03: Evaluation of overall W-2000 AV detection rates (zoo,ITW) ====================================================================== The following grid is applied to classify scanners: - detection rate =100% : scanner is graded "perfect" - detection rate above 99% : scanner is graded "excellent" - detection rate above 95% : scanner is graded "very good" - detection rate above 90% : scanner is graded "good" - detection rate of 80-90% : scanner is graded "good enough" - detection rate of 70-80% : scanner is graded "not good enough" - detection rate of 60-70% : scanner is graded "rather bad" - detection rate of 50-60% : scanner is graded "very bad" - detection rate below 50% : scanner is graded "useless" To assess an "overall AV grade" (including macro and script virus virus detection, for unpacked objects), the lowest of the related results is used to classify each scanner. Only scanners where all tests were completed are considered. (For problems in test: see 8problms.txt). Besides grading products in related categories according to their performance, it is interesting to compare how products developed. In comparison with previous results (VTC test "2000-04") and with respect to macro and script viruses, it is notified whether some product remained in the same category (=), improved into a higher category (+) or lost some grade (-). The following list indicates those scanners graded into one of the upper three categories, with macro and script virus detection rates in unpacked samples, and with perfect ITW virus detection (rate=100%). (zoo: macro/script; macro/script:ITW) ---------------------------------------------- "Perfect" W-2k scanners: AVK ( 100% 100% ; 100% 100% ) (+) FSE ( 100% 100% ; 100% 100% ) (+) PAV ( 100% 100% ; 100% 100% ) (+) ---------------------------------------------- "Excellent" W-2k scanners: AVP ( 100~ 100% ; 100% 100% ) (=) SCN ( 100% 99.8 ; 100% 100% ) (+) ---------------------------------------------- "Very Good" W-2k scanners: DRW ( 99.5 95.4 ; 100% 100% ) (+) ---------------------------------------------- Few more scanners are "perfect" or "excellent" with respect to macro virus detection but have insufficient detection rates(at best "very good" or "good") for script virus detection: (zoo: macro/script; macro/script:ITW) ---------------------------------------------- NAV ( 99.5 94.2 ; 100% 100% ) NVC ( 99.8 91.3 ; 100% 100% ) ---------------------------------------------- FPR ( 100~ 94.6 ; 100% 98.2 ) FPW ( 100~ 94.6 ; 100% 98.2 ) NAV ( 99.5 94.2 ; 100% 100% ) INO ( 99.9 93.9 ; 100% 98.2 ) ---------------------------------------------- ****************************************************************** Findings W2k.3: Now, 3 W2k product are overall rated "perfect" in last test: no product): AVK,FSE,PAV 2 "excellent" overall scanners: AVP,SCN 1 "very good" overall scanner: DRW ****************************************************************** Eval W2k.04: Evaluation of detection by virus classes under W-2000: =================================================================== Some scanners are specialised on detecting some class of viruses (either in deliberately limiting themselves to one class, esp. macro viruses, or in detecting one class significantly better than others). It is therefore worth notifying which scanners perform best in detecting macro and script viruses. Products rated "perfect" (=100%), "excellent" (>99%) and "very good" (>95%) are listed (where ITW detection must be 100%). W2k.04.2 Grading the Detection of macro viruses under W2k --------------------------------------------------------- "Perfect" W2k macro scanners: AVK (100.0%) FSE (100.0%) PAV (100.0%) SCN (100.0%) "Excellent" W2k macro scanners: AVP ( 100~ ) CMD ( 100~ ) FPR ( 100~ ) FPW ( 100~ ) INO ( 99.9%) NVC ( 99.8%) DRW ( 99.5%) NAV ( 99.5%) RAV ( 99.5%) "Very Good" W2k macro scanners: AVG ( 98.4%) W2k.04.3 Grading the Detection of Script viruses under W2k: ----------------------------------------------------------- "Perfect" W2k script scanners: AVK (100.0%) AVP (100.0%) FSE (100.0%) PAV (100.0%) "Excellent" W2k script scanners: SCN ( 99.8%) "Very Good" W2k script scanners: DRW ( 95.4%) *********************************************************************** Finding W2k.4: Performance of W2k scanners by virus classes: Perfect scanners for macro zoo: AVK,FSE,PAV,SCN Excellent scanners for macro zoo: AVP,CMD,FPR,FPW,INO,NVC,DRW,NAV,RAV Perfect scanners for script zoo: AVK,AVP,FSE,PAV Excellent scanners for script zoo: SCN *********************************************************************** Eval W2k.05: Detection of Packed Macro Viruses under W-2k ========================================================= Detection of macro viruses within packed objects becomes essential for on-access scanning, esp. for incoming email possibly loaded with malicious objects. It seems therefore reasonable to test whether at least ITW viral objects compressed with given popular methods are also detected. IIt seems therefore reasonable to test whether at least ITW viral objects compressed with 6 popular methods (PKZIP, ARJ, LHA, RAR, WinRAR and CAB) are also detected. Tests are performed only on In-The-Wild viruses packed once (no recursive packing). As last test showed that AV products are rather far from perfect detection of packed viruses, testbed has essentially be unchanged to ease comparison and improvement. A "perfect" product would detect ALL packed viral samples (100%) for all (6) packers: ---------------------------------------------------- "Perfect" packed virus detectors: AVK,AVP,CMD,FPR,FPW,PAV,SCN ---------------------------------------------------- An "excellent" product would reach 100% detection of packed viral samples for at least 5 packers: -------------------------------------------------------- "Excellent" packed macro virus detector: INO,RAV -------------------------------------------------------- A "very good" product would detect viral samples for at least 4 packers: ------------------------------------------------------ "Very Good" packed macro virus detector: AVG,DRW ------------------------------------------------------ Remark: Much more data were collected on precision and reliability of virus detection in packed objects. But in the present state, it seems NOT justified to add differentiation to results discussed here. ************************************************************************** Findings W2k.5: Detection of packed viral objects shows significant improvements as now 7 products detect "perfectly" in ALL packed viruses (in last test: 4). Perfect packed macro virus detectors: AVK,AVP,CMD,FPR,FPW,PAV,SCN Excellent packed macro virus detectors: --- Very Good packed macro virus detectors: AVG,DRW,INO ************************************************************************** Eval W2k.06: Avoidance of False Alarms (File, Macro) under W-2000: ================================================================== First introduced in VTC test "1998-10", a set of clean (and non-malicious) objects has been added to the file and macro virus testbeds to determine the ability of scanners to avoid False-Positive (FP) alarms. This ability is essential for "excellent" and "very good" scanners as there is no automatic aid to customers to handle such cases (besides the psychological impact on customerīs work). Therefore, the grid used for grading AV products must be significantly more rigid than that one used for detection. The following grid is applied to classify scanners: - False Positive rate = 0.0%: scanner is graded "perfect" - False Positive rate < 0.5%: scanner is graded "excellent" - False Positive rate < 2.5%: scanner is graded "very good" - False Positive rate < 5.0%: scanner is graded "good enough" - False Positive rate <10.0%: scanner is graded "rather bad" - False Positive rate <20.0%: scanner is graded "very bad" - False Positive rate >20.0%: scanner is graded "useless" ------------------------------------------------------------------- "Perfect" FP avoiding W2k scanners: AVA,AVG,AVK,INO,PAV,SCN ------------------------------------------------------------------- Moreover, MCV and VSP avoid FPs, but at low level of virus detection. **************************************************************** Findings W2k.6: Avoidance of False-Positive Alarms is improving though still regarded insufficient. FP-avoiding perfect W-2k scanners: AVA,AVG,AVK,INO,PAV,SCN **************************************************************** Eval W2k.07: Detection of Macro and Script Malware under W-2k ============================================================= Since test "1997-07", VTC tests also the ability of AV products to detect non-viral malware. An essential argument for this category is that customers are interested to be also warned about and protected from non-viral and non-wormy malicious objects such as trojans etc, the payload of which may be disastrous to their work (e.g. stealing passwords). Since VTC test "1999-03", malware detection is a mandatory part of VTC tests, both for submitted products and for those downloaded as free evaluation copies. A growing number of scanners is indeed able to detect non-viral malware. The following grid (admittedly with reduced granularity) is applied to classify detection of file and macro malware: - detection rate =100% : scanner is "perfect" - detection rate > 90% : scanner is "excellent" - detection rate of 80-90% : scanner is "very good" - detection rate of 60-80% : scanner is "good enough" - detection rate of < 60% : scanner is "not good enough" Concerning Macro AND Script malware detection: ------------------------------------------------------ 2 "Perfect" macro/script malware detectors under W2k: Macro/Script PAV (100% 100%) SCN (100% 100%) ------------------------------------------------------- 3 "Excellent" macro/script malware detectors under W2k: AVK (99.8% 100.0%) AVP (99.8% 100.0%) FSE (98.8% 100.0%) ------------------------------------------------------ 1 "Very Good" macro/script malware detector under W2k: RAV (97.7% 81.8%) ------------------------------------------------------ Concerning only macro malware detection, several more products (with problems concerning script malware detection) can be reported as "excellent" or "very good": --------------------------------------------------- "Perfect" macro malware detectors under W2k: PAV (100.0%) SCN (100.0%) --------------------------------------------------- "Excellent" macro malware detectors under W2k: AVK ( 99.8%) AVP ( 99.8%) FSE ( 99.8%) CMD ( 99.5%) FPR ( 99.5%) FPW ( 99.5%) NVC ( 98.8%) RAV ( 97.7%) INO ( 92.3%) DRW ( 90.8%) --------------------------------------------------- "Very Good" macro malware detectors under W2k: AVA ( 88.5%) NAV ( 86.4%) AVG ( 82.6%) --------------------------------------------------- An concerning script malware detection only, few products behave better than in macro malware detection: "Perfect" script malware detectors under W2k: ---------------------------------------------------- AVK (100.0%) AVP (100.0%) FSE (100.0%) PAV (100.0%) SCN (100.0%) ---------------------------------------------------- "Excellent" script malware detectors under W2k: NONE ---------------------------------------------------- "Very Good" macro malware detectors under W2k: RAV ( 81.8%) --------------------------------------------------- ******************************************************************* Findings W2k.7: Macro/Script Malware detection under W2k is slowly improving but still more efforts are needed: 2 products are "perfect": PAV, SCN 3 products are "excellent": AVK, AVP, FSE 1 product is rated "very good": RAV *************************************************** Concerning only macro malware detection, 2 products are rated "perfect": PAV, SCN And concerning macro malware detection only, 10 more products are rated "excellent": AVK,AVP,FSE,CMD,FPR,FPW,NVC,RAV,INO,DRW **************************************************** Concerning only script malware detection, 5 products are rated "perfect": AVK, AVP, FSE, PAV, SCN ******************************************************************* Eval W2k.SUM: Grading of W-2000 products: ========================================= Under the scope of VTCs grading system, a "Perfect W2k AV/AM product" would have the following characteristics: Definition (1): A "Perfect AntiVirus (AV) product" -------------------------------------------------- 1) Will detect ALL viral samples "In-The-Wild" AND in at least 99.9% of zoo samples, in ALL categories (file, boot and script-based viruses), with always same high precision of identification and in every infected sample, 2) Will detect ALL ITW viral samples in compressed objects for all (6) popular packers, and 3) Will NEVER issue a False Positive alarm on any sample which is not viral. Definition (2): A "Perfect AntiMalware (AM) product" ---------------------------------------------------- 1) Will be a "Perfect AntiVirus product", That is: 100% ITW detection AND >99% zoo detection AND high precision of identification AND high precision of detection AND 100% detection of ITW viruses in compressed objects, AND 0% False-Positive rate, 2) AND it will also detect essential forms of malicious software, at least in unpacked forms, reliably at high rates (>90%). ********************************************************************* In VTC test "2001-10", we found ** 2 perfect W98 AV products: AVK,PAV ********************************************** but we found **** No perfect WNT AM product **** ********************************************************************* But several products seem to approach our definition on a rather high level (taking into account the highest value of "perfect" defined on 100% level and "Excellent" defined by 99% for virus detection, and 90% for malware detection): Test category: "Perfect" "Excellent" ------------------------------------------------------------------ W2k zoo macro test: AVK,FSE,PAV,SCN AVP,CMD,FPR,FPW, INO,NVC,DRW,NAV,RAV W2k zoo script test: AVK,AVP,FSE,PAV SCN W2k ITW tests: AVK,AVP,DRW,FSE, FPR,FPW,AVG INO,NAV,NVC,PAV,SCN W2k pack-tests: AVK,AVP,CMD,FPR,FPW,PAV,SCN ----- W2k FP avoidance: AVA,AVG,AVK,INO,PAV,SCN ----- ----------------------------------------------------------------- W2k Macro Malware Test: PAV,SCN AVK,AVP,FSE,CMD,FPR, FPW,NVC,RAV,INO,DRW W2k Script Malware Test: AVK,AVP,FSE,PAV,SCN ----- ----------------------------------------------------------------- In order to support the race for more customer protection, we evaluate the order of performance in this W2k test with a simple algorithm, by counting the majority of places (weighing "perfect" twice and "excellent" once), for the first places: ************************************************************ "Perfect" W-2000 AntiVirus product: AVK,PAV (10 points) ************************************************************ "Excellent" W-2000 AV products: 3rd place: SCN ( 9 points) 4th place: AVP ( 7 points) 5th place: FSE ( 6 points) 6th place: INO ( 5 points) 7th place: FPR,FPW ( 4 points) 9th place: CMD,DRW,NAV,NVC ( 3 points) 13th place: AVA,AVG ( 2 points) 15th place: RAV ( 1 point ) ************************************************************ "Perfect" W-2000 AntiMalware product: PAV (14 points) ************************************************************ "Excellent" W-2000 AntiMalware product: 2nd place: AVK,SCN (13 points) 4th place: AVP (10 points) 5th place: FSE ( 7 points) 6th place: INO,FPR,FPW ( 5 points) 9th place: CMD,DRW,NVC ( 4 points) 12th place: RAV ( 2 points) ************************************************************