========================================= File 7EVAL-WNT.TXT ========================================= Evaluation of results for Macro and Script Virus/Malware detection under Windows-NT in Heureka-2 test ========================================= Formatted with non-proportional font (Courier) Forword: This test is based on VTC test "2001-09". Products submitted for that test (with engines and signatures dated before June 25, 2001) were applied to 2 differential testbeds: Testbed ".017" contained all those macro/script viruses/malware which were reported between May 1, 2001 and July 31, 2001. Testbed ".01A" contained all those macro/script viruses/malware which were reported between August 1 and October 31, 2001. For both testbeds, only those viruses were icluded in In-The-Wild tests which have been newly reported as "In-The- Wild" during the related period. The goal of test "2002-02" is to determine to which degree AV products are able to reliably detect macro/script viruses and malware found after delivery of engines and signatures. In order to enable compa- rison with VTC test "2001-09", products were used in the same mode, esp. with the same options and parameters as in that test. Therefore, this is NOT EXACTLY a test for heuristic detection (where one should set switches/options accordingly, e.g. to exclude detection by signa- tures). ("Heureka" is a word from ancient Greek meaning "I found it"). Content of this file: ************************************************************************ Eval WNT.MZ: Development of Macro Zoo (MZ) virus detection rates Table WNT.MZ: Development of MZ detection rates Analysis of Macro Zoo virus detection rates Result Heureka-2.MZ (Macro Zoo viruses) Eval WNT.MI: Development of Macro ITW (MI) virus detection rates Table WNT.MI: Development of MI detection rates Analysis of Macro ITW virus detection rates Result Heureka-2.MI (Macro ITW viruses) Eval WNT.MM: Development of Macro Malware (MM) detection rates Table WNT.MM:Development of MM detection rates Analysis of Macro Malware detection rates Result Heureka-2.MM (Macro Malware) Eval WNT.SZ: Development of Script Zoo (SZ) virus detection rates Table WNT.SZ: Development of SZ detection rates Analysis of Script Zoo virus detection rates Result Heureka-2.SZ (Script Zoo viruses) Eval WNT.SI: Development of Script ITW (SI) virus detection rates Table WNT.SI: Development of SI detection rates Analysis of Script ITW virus detection rates Result Heureka-2.SI (Script ITW viruses) Eval WNT.SM: Development of Script Malware (SM) detection rates Table WNT.SM: Development of SM detection rates Analysis of Script Malware detection rates Result Heureka-2.SM (Script Malware) COMPARE: Comparison of Heureka-1 and Heureka-2 test results Eval WNT.SUM Grading of WNT products according to "Heureka-2" results ************************************************************************ This part of VTC "2002-02" test report evaluates the detailed results as given in section (file): 6GWNT.TXT Macro/Script Viruses/Malware results (W-NT) The following (21) products participated in this special "heuristic" scanner test for WNT products: -------------------------------------------------------- Products submitted for aVTC test under Windows-NT: -------------------------------------------------------- ANT v(def): 6.8.0.2 sig: June 05,2001 AVA v(def): unknown sig: unknown AVG v(def): 6.0.263 sig: June 22,2001 AVK v(def): 10.0.167 sig: June 21,2001 AVP v(def): 3.5.133.0 sig: June 01,2001 AVX v(def): 6.1 sig: June 18,2001 CMD v(def): 4.61.5 sig: June 25,2001 DRW v(def): 4.25 sig: June 20,2001 FPR v(def): 3.09d sig: June 25,2001 FPW v(def): 3.09d sig: June 25,2001 FSE v(def): 1.00.1251 sig: June 20,2001 scan eng fprot: 3.09.507 scan eng avp: 3.55.3210 scan eng orion: 1.02.15 IKA v(def): 5.01 sig: June 25,2001 INO v(def): 6.0.85 sig: June 14,2001 MR2 v(def): 1.17 sig: June 25,2001 NVC v(def): 5.00.25 sig: June 19,2001 PAV v(def): 3.5.133.0 sig: June 23,2001 QHL v(def): 6.02 sig: June 28,2001 RAD v(def): 8.1.001 sig: June 25,2001 RAV v(def): 8.2.001, scan eng:8.3 sig: June 25,2001 SCN v(def): 4144 scan eng:4.1.40 sig: June 20,2001 VSP v(def): 12.22.1 sig: June 25,2001 -------------------------------------------------------- One products (NAV) was withdrawn from this test, due to "new engines". Eval WNT.MZ: Development of Macro Zoo virus detection rates: ============================================================ ------------------------+---------------+---------+---------------+---------- Viruses | New viruses | Loss in | New viruses | Loss in Scanner detected | detected | 3 months| detected | 6 months ------------------------+---------------+---------+---------------+---------- Status: April 30,2001 I July 31,2001I IOctober 31,2001I Testbed 6762 100.0% | 357 100.0% I | 176 100.0% I ------------------------+---------------+---------+---------------+---------- ANT 6566 97.1% | 221 61.9% | -35.2% | 77 43.8% | -53.3% AVA 6604 97.7% | 254 71.1% | -26.6% | 97 55.1% | -42.6% AVG 6651 98.4% | 318 89.1% | -9.3% | 117 66.5% | -31.9% AVK 6762 100.0% | 288 80.7% | -19.3% | 69 39.2% | -60.8% AVP 6761 100.0% | 292 81.8% | -18.2% | 70 39.8% | -60.2% AVX 6703 99.1% | 343 96.1% | -3.0% | 166 94.3% | -4.8% CMD 6760 100.0% | 324 90.8% | -9.2% | 128 72.7% | -27.3% DRW 6725 99.5% | 344 96.4% | -3.1% | 169 96.0% | -3.5% FPR 6760 100.0% | 322 90.2% | -9.8% | 127 72.2% | -27.8% FPW 6760 100.0% | 322 90.2% | -9.8% | 127 72.2% | -27.8% FSE 6762 100.0% | 341 95.5% | -4.5% | 151 85.8% | -14.2% IKA 6451 95.4% | 290 81.2% | -14.2% | 107 60.8% | -34.6% INO 6755 99.9% | 339 95.0% | -4.9% | 167 94.9% | -5.0% MR2 44 0.7% | 6 1.7% | 1.0% | 5 2.8% | 2.1% NVC 6751 99.8% | 223 62.5% | -37.3% | 50 28.4% | -71.4% PAV 6762 100.0% | 292 81.8% | -18.2% | 70 39.8% | -60.2% QHL 0 0.0% | 0 0.0% | 0.0% | 0 0.0% | 0.0% RAV 6726 99.5% | 330 92.4% | -7.1% | 134 76.1% | -23.4% SCN 6762 100.0% | 349 97.8% | -2.2% | 167 94.9% | -5.1% VSP 1 0.0% | 0 0.0% | 0.0% | 1 0.6% | 0.6% ------------------------+---------------+---------+---------------+---------- Mean ALL: 85.7% 72.8% -11.5% 56.8% -27.6% Mean rel: 99.2% 85.6% -10.5% 66.6% -25.1% ------------------------+---------------+---------+---------------+---------- Remarks: "Mean ALL" is the mean value of virus and file identification calculated over ALL related entries. "Mean rel" is the relative mean value of virus and file identification calculated only for those entries with a minimum detection rate "minrate", where minrate = 65% for zoo viruses, minrate = 95% for ITW viruses, and minrate = 60% for malware. Definition of "loss vector": Loss in 3 months = Loss vector #1 = detection rate in month 1-3 minus detection rate in reference test Loss in 6 months = Loss vector #2 = detection rate in month 4-6 minus detection rate in reference test Analysis of results for Macro Zoo viruses: ========================================== (1) For zoo macro viruses, best products are able to detect more than 90% of those viruses reported within first 3-month period and more than 80% within second 3-month period after product/signature delivery: SCN (100.0% 97.8% 94.9%) FSE (100.0% 95.5% 85.8%) INO ( 99.9% 95.0% 94.9%) DRW ( 99.5% 96.4% 96.0%) AVX ( 99.1% 96.1% 96.0%) (2) During the first 3 months, mean loss in detection ability is 11.5% (overall), and it is slightly better (10.5%) when products with extremely low detection rates are not counted. The following products behave best in first 3-month period: SCN (97.8%) DRW (96.4%) AVX (96.1%) FSE (95.5%) INO (95.0%) (3) In months 4-6, the loss in detection quality is fastly growing, with a mean loss of 27.6% (overall) and 25.1% when products with extremely low detection rates are not counted. For viruses first reported in the next 3 months (=fourth to sixt month after product delivery), best products still detect more than 90% of zoo macro viruses: DRW (96.0%) INO (94.9%) SCN (94.9%) AVX (94.3%) (4) In order to classify product behaviour, we grade products according to loss in detection quality. When considering only products with losses up to 20% after 6 months (ordered according to highest detection rates after 6 months), the following products behaved best in "Heureka-2" test: ------------------------------------ detection rate loss in loss in AV product in ref-test month 1-3 month 4-6 ------------------------------------ DRW 99.5% -3.1% -3.5% SCN 100.0% -2.2% -5.1% INO 99.9% -4.9% -5.0% AVX 99.1% -3.0% -4.8% FSE 100.0% -4.5% -14.2% ------------------------------------ ************************************************************* Result "Heureka-2.MZ": concerning new zoo macro viruses, the following 4 products miss less than 10% over 6 months: ----------------------------------------------------- DRW after 3 months: - 3.1% after 6 months: - 3.5% ----------------------------------------------------- SCN after 3 months: - 2.2% after 6 months: - 5.1% ----------------------------------------------------- INO after 3 months: - 4.9% after 6 months: - 5.0% ----------------------------------------------------- AVX after 3 months: - 3.0% after 6 months: - 4.8% ************************************************************** And the following product misses less than 20% over 6 months: FSE after 3 months: - 4.5% after 6 months: -14.2% ************************************************************** Eval WNT.MI: Development of Macro ITW virus detection rates: ============================================================ ------------------------+---------------+---------+---------------+---------- Viruses | New viruses | loss in | New viruses | loss in Scanner detected | detected | 3 months| detected | 6 months ------------------------+---------------+---------+---------------+---------- Status: April 30,2001 July 31,2001 IOctober 31,2001 Testbed 143 100.0% | 17 100.0% | 7 100.0% ------------------------+---------------+---------+---------------+---------- ANT 142 99.3% | 14 82.4% | -16.9% | 5 71.4% | -27.9% AVA 143 100.0% | 16 94.1% | -5.9% | 5 71.4% | -28.6% AVG 143 100.0% | 17 100.0% | 0.0% | 7 100.0% | 0.0% AVK 143 100.0% | 17 100.0% | 0.0% | 7 100.0% | 0.0% AVP 143 100.0% | 17 100.0% | 0.0% | 7 100.0% | 0.0% AVX 143 100.0% | 16 94.1% | -5.9% | 6 85.7% | -14.3% CMD 143 100.0% | 17 100.0% | 0.0% | 7 100.0% | 0.0% DRW 143 100.0% | 17 100.0% | 0.0% | 7 100.0% | 0.0% FPR 143 100.0% | 17 100.0% | 0.0% | 7 100.0% | 0.0% FPW 143 100.0% | 17 100.0% | 0.0% | 7 100.0% | 0.0% FSE 143 100.0% | 17 100.0% | 0.0% | 7 100.0% | 0.0% IKA 142 99.3% | 17 100.0% | 0.7% | 7 100.0% | 0.7% INO 143 100.0% | 16 94.1% | -5.9% | 7 100.0% | 0.0% MR2 13 9.1% | 0 0.0% | -9.1% | 0 0.0% | -9.1% NVC 143 100.0% | 17 100.0% | 0.0% | 6 85.7% | -14.3% PAV 143 100.0% | 17 100.0% | 0.0% | 7 100.0% | 0.0% QHL 0 0.0% | 0 0.0% | 0.0% | 0 0.0% | 0.0% RAV 143 100.0% | 17 100.0% | 0.0% | 7 100.0% | 0.0% SCN 143 100.0% | 17 100.0% | 0.0% | 7 100.0% | 0.0% VSP 0 0.0% | 0 0.0% | 0.0% | 0 0.0% | 0.0% ------------------------+---------------+---------+---------------+---------- Mean ALL: 86.7% 86.7% -2.2% 80.7% -4.7% Mean rel: 99.9% 99.9% -2.4% 95.0% -5.2% ------------------------+---------------+---------+---------------+---------- Remark: concerning calculation of mean values: see 1st table "Eval WNT.MZ" Analysis of results for Macro ITW viruses: ========================================== (0) Due to the small number of ITW Macro viruses detected in each 3-month period, we just discuss findings but dont grade products based on such potentially insignificant figures. (1) For macro ITW viruses, the majority of products detect all ITW viruses even after 6 months. The following products consistently detect ALL macro ITW viruses at reference test as well after 3 and 6 months, ALL with perfect detection vectors (100% 100% 100%): AVG, AVK, AVP, CMD, DRW, FPR, FPW, FSE, INO, PAV, RAV, SCN. (2) In comparison with Heureka-1 test, where 6 products detected ALL ITW macro viruses, the situation has improved significantly. ************************************************************* Result "Heureka-2.MI": concerning new Macro ITW viruses, the following 13 products miss NO ITW virus during 6months: AVG, AVK, AVP, CMD, DRW, FPR, FPW, FSE, INO, PAV, RAV, SCN. ************************************************************** Eval WNT.MM: Development of macro malware detection rates: ========================================================== ------------------------+---------------+---------+---------------+---------- Viruses | New viruses | loss in | New viruses | loss in Scanner detected | detected | 3 months| detected | 6 months ------------------------+---------------+---------+---------------+---------- Status: April 30,2001 I July 31,2001I IOctober 31,2001I Testbed 426 100.0% | 22 100.0% | 7 100.0% ------------------------+---------------+---------+---------------+---------- ANT 378 88.7% | 10 45.5% | -43.2% | 4 57.1% | -31.6% AVA 377 88.5% | 11 50.0% | -38.5% | 3 42.9% | -45.6% AVG 352 82.6% | 14 63.6% | -19.0% | 1 14.3% | -68.3% AVK 425 99.8% | 12 54.5% | -45.3% | 3 42.9% | -56.9% AVP 425 99.8% | 14 63.6% | -36.2% | 3 42.9% | -56.9% AVX 392 92.0% | 18 81.8% | -10.2% | 7 100.0% | 8.0% CMD 424 99.5% | 15 68.2% | -31.3% | 4 57.1% | -42.4% DRW 387 90.8% | 18 81.8% | -9.0% | 7 100.0% | 9.2% FPR 424 99.5% | 15 68.2% | -31.3% | 4 57.1% | -42.4% FPW 424 99.5% | 15 68.2% | -31.3% | 4 57.1% | -42.4% FSE 425 99.8% | 17 77.3% | -22.5% | 4 57.1% | -42.7% IKA 383 89.9% | 16 72.7% | -17.2% | 5 71.4% | -18.5% INO 398 93.4% | 15 68.2% | -25.2% | 5 71.4% | -22.0% MR2 135 31.7% | 0 0.0% | -31.7% | 2 28.6% | -3.1% NVC 421 98.8% | 12 54.5% | -44.3% | 3 42.9% | -55.9% PAV 426 100.0% | 14 63.6% | -36.4% | 3 42.9% | -57.1% QHL 0 0.0% | 0 0.0% | 0.0% | 0 0.0% | 0.0% RAV 416 97.7% | 17 77.3% | -20.4% | 5 71.4% | -26.3% SCN 426 100.0% | 17 77.3% | -22.7% | 3 42.9% | -57.1% VSP 1 0.2% | 0 0.0% | -0.2% | 0 0.0% | -0.2% ------------------------+---------------+---------+---------------+---------- Mean ALL: 83.4% 56.8 -25.8% 50.0% -32.6% Mean rel: 95.0% 66.8% -28.7% 55.6% -36.2% ------------------------+---------------+---------+---------------+---------- Remark: concerning calculation of mean values: see 1st table "Eval WNT.MZ" Analysis of results for Macro Malware: ====================================== (0) Due to the small number of Macro Malware detected in each 3-month period, we just discuss findings but dont grade products based on such potentially insignificant figures. (1) For non-replicant Macro Malware, detection quality is - in the mean - significantly less developed than the detction of replicative malware (aka viruses & worms). The mean malware detection rate of tested products (except those with extremely insufficient detection rates) degrades from 95.0% (in reference test) to 66.8% (after 3 months) further down to 55.6% (after 6 months). (2) Some products even improve their detection rates, as the following detection vectors indicate: DRW ( 90.8% 81.8% 100.0%) AVX ( 92.0% 81.8% 100.0%) This may indicate that the heuristic mechanism of these products are very well developed, but with the relative small set of samples (7 for months 4-6), it canNOT be determined whether this results is an artefact of the statistical evaluation. (3) The following products loose less than 20% detection rate over each 3-month period but they they dont start with optimum detection rate in the reference test: DRW ( 90.8% 81.8% 100.0%) AVX ( 92.0% 81.8% 100.0%) IKA ( 89.9% 72.7% 71.4%) (4) Those products which detected almost all malware samples with "fresh" signatures in the reference test (esp. PAV and SCN) lost significantly more detection rate compared to mean loss. This may indicate that these products apply mechanisms of exact identification instead of heuristics. (5) In comparison with Heureka-1 test results, those products then scoring best (FSE, SCN: loss after 6 months: -31.7%) have now much larger loss in detection rate (-42.7%, -57.1%). ******************************************************************* Result "Heureka-2.MM": The persistency of non-replicative malware detection needs significant improvement. Only 3 products loose less than 40% detection quality over six months, but all three products have less than optimum detection rates in the reference test. ******************************************************************* Eval WNT.SZ: Development of Script Zoo virus detection rates: ============================================================= ------------------------+---------------+---------+---------------+---------- Viruses | New viruses | loss in | New viruses | loss in Scanner detected | detected | 3 months| detected | 6 months ------------------------+---------------+---------+---------------+---------- Status: April 30,2001 I July 31,2001I IOctober 31,2001I Testbed 588 100.0% | 164 100.0% | 102 100.0% ------------------------+---------------+---------+---------------+---------- ANT 481 81.8% | 42 25.6% | -56.2% | 12 11.8% | -70.0% AVA 174 29.6% | 32 19.5% | -10.1% | 11 10.8% | -18.8% AVG 370 62.9% | 85 51.8% | -11.1% | 40 39.2% | -23.7% AVK 588 100.0% | 126 76.8% | -23.2% | 52 51.0% | -49.0% AVP 588 100.0% | 126 76.8% | -23.2% | 49 48.0% | -52.0% AVX 412 70.1% | 89 54.3% | -15.8% | 31 30.4% | -39.7% CMD 552 93.9% | 104 63.4% | -30.5% | 46 45.1% | -48.8% DRW 561 95.4% | 136 82.9% | -12.5% | 72 70.6% | -24.8% FPR 558 94.9% | 104 63.4% | -31.5% | 46 45.1% | -49.8% FPW 556 94.6% | 104 63.4% | -31.2% | 46 45.1% | -49.5% FSE 588 100.0% | 141 86.0% | -14.0% | 71 69.6% | -30.4% IKA 457 77.7% | 104 63.4% | -14.3% | 49 48.0% | -29.7% INO 559 95.1% | 78 47.6% | -47.5% | 34 33.3% | -61.8% MR2 490 83.3% | 93 56.7% | -26.6% | 45 44.1% | -39.2% NVC 537 91.3% | 74 45.1% | -46.2% | 27 26.5% | -64.8% PAV 588 100.0% | 126 76.8% | -23.2% | 47 46.1% | -53.9% QHL 1 0.2% | 1 0.6% | 0.4% | 1 1.0% | 0.8% RAV 485 82.5% | 0 0.0% | -82.5% | 0 0.0% | -82.5% SCN 587 99.8% | 134 81.7% | -18.1% | 58 56.9% | -42.9% VSP 494 84.0% | 93 56.7% | -27.3% | 45 44.1% | -39.9% ------------------------+---------------+---------+---------------+---------- Mean ALL: 78.7% 54.6% -27.2% 38.3% -43.6% Mean rel: 86.6% 60.7% -28.6% 42.5% -45.8% ------------------------+---------------+---------+---------------+---------- Remark: concerning calculation of mean values: see 1st table "Eval WNT.MZ" Analysis of results for Script Zoo viruses: =========================================== (1) Heuristic detection of script viruses is significantly less developed than detection fo macro viruses, as comparison of mean losses (without those products with inadequate detection rates) in detection rates shows: reference after after test 3 months 6 months detection rate of macro viruses 99.2% 85.6% 66.6% detection rate of script viruses 86.6% 60.7% 42.5% For zoo script viruses, best products are able to detect more than 80% of those zoo viruses reported within 3-months after products delivery and 60% of those viruses reported after 6 months: FSE (100.0% 86.0% 69.9%) DRW ( 95.4% 82.9% 70.6%) In addition, the following products (which detected at least 90% in the reference test) lost less than 20% in the first 3-month period but lost more than 40% in the second period: SCN ( 99.8% 81.7% 56.9%) (2) During the first 3 months, mean loss in detection ability is 28.6%. In months 4-6, the loss in detection quality is fastly growing, with a mean loss of 45.8%. (3) In order to classify product behaviour, we grade products according to loss in detection quality. When considering only products with losses up to 40% after 6 months (ordered according to highest detection rates after 6 months), the following products behaved best in "Heureka-2" test: ------------------------------------ detection rate loss in loss in AV product in ref-test month 1-3 month 4-6 --------+------------------------------------- DRW 95.4% -12.5% -24.8% FSE 100.0% -14.0% -30.4% ------------------------------------ SCN 99.8% -18.1% -42.9% ------------------------------------ ******************************************************************** Result "Heureka-2.SZ": Zoo script virus detection is significantly less well developped compared with macro virus detection; losses in detection rates are more than 5 times higher than with macro viruses. The following 2 products miss less than 15% after 3 months and about 30% after 6 months: DRW after 3 months: - 12.5% after 6 months: - 24.8% -------------------------------- FSE after 3 months: - 14.0% after 6 months: - 30.4% For AV companies, there is strong need to improve persistent detection methods esp. as this category adresses many mass-emailing viruses! For customers, the strong evidence is to update AV products for script virus detection much more often than for script viruses. ******************************************************************** Eval WNT.SI: Development of Script ITW virus detection rates: ============================================================= ------------------------+---------------+---------+---------------+---------- Viruses | New viruses | loss in | New viruses | loss in Scanner detected | detected | 3 months| detected | 6 months ------------------------+---------------+---------+---------------+---------- Status: April 30,2001 I July 31,2001I IOctober 31,2001I Testbed 19 100.0% | 10 100.0% | 6 100.0% ------------------------+---------------+---------+---------------+---------- ANT 19 100.0% | 10 100.0% | 0.0% | 2 33.3% | -66.7% AVA 18 94.7% | 10 100.0% | 5.3% | 2 33.3% | -61.4% AVG 19 100.0% | 10 100.0% | 0.0% | 4 66.7% | -33.3% AVK 19 100.0% | 10 100.0% | 0.0% | 4 66.7% | -33.3% AVP 19 100.0% | 10 100.0% | 0.0% | 4 66.7% | -33.3% AVX 19 100.0% | 10 100.0% | 0.0% | 2 33.3% | -66.7% CMD 19 100.0% | 10 100.0% | 0.0% | 5 83.3% | -16.7% DRW 19 100.0% | 10 100.0% | 0.0% | 4 66.7% | -33.3% FPR 19 100.0% | 10 100.0% | 0.0% | 5 83.3% | -16.7% FPW 19 100.0% | 10 100.0% | 0.0% | 5 83.3% | -16.7% FSE 19 100.0% | 10 100.0% | 0.0% | 5 83.3% | -16.7% IKA 18 94.7% | 10 100.0% | 5.3% | 4 66.7% | -28.0% INO 19 100.0% | 7 70.0% | -30.0% | 1 16.7% | -83.3% MR2 17 89.5% | 10 100.0% | 10.5% | 1 16.7% | -72.8% NVC 19 100.0% | 10 100.0% | 0.0% | 3 50.0% | -50.0% PAV 19 100.0% | 10 100.0% | 0.0% | 4 66.7% | -33.3% QHL 1 5.3% | 1 10.0% | 4.7% | 1 16.7% | 11.4% RAV 18 94.7% | 0 0.0% | -94.7% | 0 0.0% | -94.7% SCN 19 100.0% | 10 100.0% | 0.0% | 4 66.7% | -33.3% VSP 17 89.5% | 10 100.0% | 10.5% | 1 16.7% | -72.8% ------------------------+---------------+---------+---------------+---------- Mean ALL: 89.5% 89.0% -4.4% 50.8% -43.7% Mean rel: 98.2% 98.3% -4.4% 53.5% -45.4% ------------------------+---------------+---------+---------------+---------- Remark: concerning calculation of mean values: see 1st table "Eval WNT.MZ" Analysis of results for Script ITW viruses: =========================================== (0) Due to the small number of ITW Script viruses detected in each 3-month period, we just discuss findings but dont grade products based on such potentially insignificant figures. (1) For script ITW viruses, the majority of products detect all ITW viruses after 3 months but detection rates are signi- ficantly reduced after 6 months. ********************************************************** Result "Heureka-2.SI": concerning new script ITW viruses, detection rates degrade much faster after 3 months than for macro ITW viruses. ********************************************************** Eval WNT.SM: Development of Script Malware detection rates: ============================================================ ------------------------+---------------+---------+---------------+---------- Viruses | New viruses | loss in | New viruses | loss in Scanner detected | detected | 3 months| detected | 6 months ------------------------+---------------+---------+---------------+---------- Status: April 30,2001 I July 31,2001I IOctober 31,2001I Testbed 22 100.0 | 37 100.0% | 73 100.0% ------------------------+---------------+---------+---------------+---------- ANT 0 0.0% | 7 18.9% | 18.9% | 5 6.8% | 6.8% AVA --- 0% | 1 2.7% | 2.7% | 2 2.7% | 2.7% AVG 5 22.7% | 4 10.8% | -11.9% | 4 5.5% | -17.2% AVK 22 100.0% | 25 67.6% | -32.4% | 20 27.4% | -72.6% AVP 22 100.0% | 28 75.7% | -24.3% | 20 27.4% | -72.6% AVX 2 9.1% | 10 27.0% | 17.9% | 8 11.0% | 1.9% CMD 14 63.6% | 8 21.6% | -42.0% | 4 5.5% | -58.1% DRW 8 36.4% | 19 51.4% | 15.0% | 21 28.8% | -7.6% FPR 14 63.6% | 8 21.6% | -42.0% | 4 5.5% | -58.1% FPW 14 63.6% | 8 21.6% | -42.0% | 4 5.5% | -58.1% FSE 22 100.0% | 32 86.5% | -13.5% | 24 32.9% | -67.1% IKA 8 36.4% | 15 40.5% | 4.1% | 11 15.1% | -21.3% INO 15 68.2% | 9 24.3% | -43.9% | 14 19.2% | -49.0% MR2 5 22.7% | 4 10.8% | -11.9% | 10 13.7% | -9.0% NVC 2 9.1% | 5 13.5% | 4.4% | 4 5.5% | -3.6% PAV 22 100.0% | 25 67.6% | -32.4% | 20 27.4% | -72.6% QHL 1 4.5% | 1 2.7% | -1.8% | 1 1.4% | -3.1% RAV 18 81.8% | 0 0.0% | -81.8% | 0 0.0% | -81.8% SCN 22 100.0% | 27 73.0% | -27.0% | 21 28.8% | -71.2% VSP 5 22.7% | 4 10.8% | -11.9% | 10 13.7% | -9.0% ------------------------+---------------+---------+---------------+---------- Mean ALL: 49.6% 32.4% -32.4% 14.2% -36.0% Mean rel: (63.1%) (37.8%) (-46.0%) (22.3%) (-68.2%) ------------------------+---------------+---------+---------------+---------- Remark: concerning calculation of mean values: see 1st table "Eval WNT.MZ" Analysis of results for Script Malware: ======================================= (0) When comparing the numbers of macro and script malware detected within two consecutiv 3-month periods, much more samples of script malware have been detected for the latter. (1) For non-replicant Script Malware, detection quality starts at a significantly lower level (63.1%), and quality degrades much faster than for replicative malware (aka viruses & worms). The mean malware detection rate of tested products (except those with extremely insufficient detection rates) degrades from 63.1% (in reference test) to 37.8% (after 3 months) further down to 22.3% (after 6 months). ******************************************************************* Result "Heureka-2.SM": The persistency of non-replicative malware detection needs significant improvement. Customers of AV products are advised to update there products much faster for detection of trojanic script malware than for macro malware. ******************************************************************* Comparing results of test Heureka-1 and Heureka-2: =================================================== The following table lists essential results of Heureka tests: Mean detection rates in: Heureka-1 Heureka-2 ---------------------------+----------------------------- after after after after reference 3 months 6 months reference 3 months 6 months -------------------------------------------------+----------------------------- MZ=Macro zoo viruses 90.8% 73.7% 66.0% 99.2% 85.6% 66.6% MI=Macro ITW viruses 91.8% 89.7% 83.0% 99.9% 99.9% 95.0% MM=Macro Malware 87.1% 61.8% 56.4% 95.0% 66.8% 55.6% -------------------------------------------------------------------------------- SZ=Script zoo viruses 83.4% 61.0% 49.3% 86.6% 60.7% 42.5% SI=Script ITW viruses --- --- --- 98.2% 98.3% 53.5% SM=Script Malware --- --- --- 63.1% 37.8% 22.3% -------------------------------------------------------------------------------- Concerning detection rates for macro viruses, both In-The-Wild and in zoo, AV products have improved their detection rates both generally and after first 3-month period significantly. But loss of detection quality in the second 3-month period is much stronger than before (the results for ITW viruses may be influenced to the small number of newly found viruses). Concerning zoo script viruses, detection rates are stable on an insufficient level. Eval WNT.SUM Grading of WNT products according to "Heureka-2" results ===================================================================== In comparing products, some behave "rather well". Over two 3-months periods, the following products behaves best (although they also need significant improvement): DRW and FSE Moreover, the following products behave best in the first 3-month period: DRW, FSE, SCN and AVX, INO. But as some testbeds were rather small, and as the loss in script viruses detection quality was so dominant, we decided NOT to grade any product in VTCs grading scheme. We nevertheless hope that AV companies do their best to improve the generic and heuristic detection mechanisms. And we strongly advise customers, to upgrade their products and signatures as often as possible.