========================================= File 7EVAL.TXT Evaluation of VTC Scanner Test "1998-10": ========================================= Formatted with non-proportional font (Courier) This part of VTC "1998-10" test report evaluates the detailed results as given in sections (files): 6BDOSFIL.TXT File Virus/Malware results DOS 6CDOSBOO.TXT Boot Virus results DOS 6DDOSMAC.TXT Macro Viruses/Malware results DOS 6EW95.TXT File/Macro Viruses/Malware results Win 95 6FW98.TXT File/Macro Viruses/Malware results Win 98 6GWNT.TXT File/Macro Viruses/Malware results Win NT &HCMP32.TXT Comparison File/Macro results Win 95/98/NT Eval #1: Evaluation of DOS Scanner Improvement between last tests: ================================================================== Concerning performance of DOS scanners, a comparison of virus detection results in previous tests "1997-02/07" and "1998-02" with "1998-10" shows how scanners behave and how manufacturers work in adapting their products to the growing threat of new viruses and malware. The following table lists the development of the detection rate of scanners (most actual versions in each test), and it calculates the change (+ indicating improvement) in detection rates between the last (1998-02) and the actual test (1998-10). This comparison concentrates on file and macro virus detection quality. VTC test do NOT TEST for physical boot sector detection (see 4testcon.txt), so results may be unfair for those scanners which analyse physical layout of boot viruses. Therefore, boot virus detection results are not discussed here in detail (although results are avalable: 6CDOSBOO.TXT). For reasons of fairness, it must be noted that improvement of those products which have yet reached a very high level of detection and quality (say: more than 90 or 95%) is much more difficult to achieve than for those products which reached lower detection rates. Some products have incorporated new engines and included formerly separate scanners (e.g. on macro viruses) which lead to improved performance. Generally, changes in the order of about +-2% are less significant as this is about the growth rate per month, so detection depends strongly upon whether some virus is reported (and analysed and included) just before a new update is delivered. ************************************************************** Finding #1.1) In comparison with last VTC test ("1998-02"), the ability of scanners to detect file viruses under DOS was only slightly improved (mean value up 1% to 85.4%). Equally, only 6 out of 20 scanners in test detected 100% of ITW file viruses. #1.2) On the better side, the ability of scanners to detect macro viruses improved significantly (by almost 6%) to now 89.6 (mean detection rate). Now, 2 scanners detect ALL Zoo viruses, and 17 (out of 21) scanners detect ALL ITW macro viruses. This indicates that contemporary macro viruses are not technically difficult to process. #1.3) Evidently, most AV producers invest relatively more work into detection of macro viruses than of file viruses. There is a risk that threats of file viruses are underestimated! ************************************************************** Table E1: Improvement of DOS scanners from 1997-02 to 1998-02: ============================================================== ---- File Virus Detection --- ---- Macro Virus Detection --- SCAN 97/02 97/07 98/02 98/10 DELTA 97/02 97/07 98/02 98/10 DELTA NER % % % % % % % % % % ----------------------------------------------------------------- ALE 98.8 94.1 89.4 - - 96.5 66.0 49.8 - - AVS 98.9 97.4 97.4 97.9 +0.5 99.3 98.2 80.4 97.2 +16.8 AVG 79.2 85.3 84.9 87.6 +2.7 25.2 71.0 27.1 81.6 +54.6 AVK - - - 90.0 - - - - 99.7 - AVP 98.5 98.4 99.3 99.7 +0.4 99.3 99.0 99.9 100.0 +0.1 ANT 73.4 80.6 84.6 75.7 -8.3 58.0 68.6 80.4 56.6 -23,8 DRW 93.2 93.8 92.8 93.1 +0.3 90.2 98.1 94.3 99.3 +5.0 DSS 99.7 99.6 99.9 99.9 0.0 97.9 98.9 100.0 100.0 +0.0 FMA - - - - - 98.6 98.2 99.9 - - FPR 90.7 89.0 96.0 95.5 -0.5 43.4 36.1 99.9 99.8 -0.1 FSE - - 99.4 99.7 - - - 99.9 90.1 -9.8 FWN - - - - - 97.2 96.4 91.0 85.7 -5.3 IBM 93.6 95.2 96.5 - - 65.0 88.8 99.6 - - INO - - 92.0 93.5 +1.5 - - 90.3 95.2 +4.9 IRS - 81.4 74.2 - - - 69.5 48.2 -22.3 ITM - 81.0 81.2 65.8 -15.4 81.8 58.2 68.6 76.3 +7.7 IVB 8.3 - - - - - - - - HMV - - - - - - - 98.2 99.0 +0.8 NAV 66.9 67.1 97.1 98.1 +1.0 80.7 86.4 98.7 99.8 +1.1 NVC 87.4 89.7 94.1 93.8 -0.3 13.3 96.6 99.2 90.8 -8.4 PAN - - 67.8 - - - - 73.0 - - PAV - 96.6 98.8 - - - 93.7 100.0 - - PCC - - - - - - 67.6 - - - PCV 67.9 - - - - - - - - - RAV - - - 71.0 - - - - 99.5 - SCN 83.9 93.5 90.7 87.8 -0.9 95.1 97.6 99.0 98.6 -1.4 SWP 95.9 94.5 96.8 98.4 +1.6 87.4 89.1 98.4 98.6 -0.2 TBA 95.5 93.7 92.1 93.2 +1.1 72.0 96.1 99.5 98.7 -1.2 TSC - - 50.4 56.1 +5.7 - - 81.9 17.0 -64.9 TNT 58.0 - - - - - - - - - VDS - 44.0 37.1 - - 16.1 9.9 8.7 - - VET - 64.9 - - - - 94.0 97.3 97.5 +0.2 VRX - - - - - - - - - - VBS 43.1 56.6 - 35.5 - - - - - - VHU 19.3 - - - - - - - - - VSA - - 56.9 - - - - 80.6 - - VSP - - - 76.1 - - - - - - VSW - - 56.9 - - - - 83.0 - - VTR 45.5 - - - - 6.3 - - - - XSC 59.5 - - - - - - - - - ------------------------------------------------------------------ Mean 74.2 84.8 84.4 85.4 69.6 80.9 83.8 89.6 ------------------------------------------------------------------ Eval #2: Evaluation for overall virus detection rates under DOS: ================================================================ The following grid is applied to classify scanners: - detection rate =100% : scanner is "perfect" - detection rate above 95% : scanner is graded "excellent" - detection rate above 90% : scanner is graded "very good" - detection rate of 80-90% : scanner is graded "good enough" - detection rate of 70-80% : scanner is graded "not good enough" - detection rate of 60-70% : scanner is graded "rather bad" - detection rate of 50-60% : scanner is graded "very bad" - detection rate below 50% : scanner is graded "useless" Overall AV grade of DOS scanners: ================================= To assess an "overall AV grade" (including file and macro virus detection, for unpacked objects), the lowest of the related results is used to classify the resp. scanner. If several scanners of the same producer have been tested, grading is applied to the most actual version (which is in most cases the version with highest detection rates). Only scanners where all tests were completed are considered. The following list indicates those scanners graded into one of the upper three categories, with file and macro virus detection rates in unpacked forms, and with perfect ITW virus detection (rate=100%): (file/macro zoo; file/macro ITW) -------------------------------- "Perfect" DOS scanners: =NONE= "Excellent" DOS scanners: DSS ( 99.9% 100.0%; 100.0% 100.0%) AVP ( 99.7% 100.0%; 100.0% 100.0%) SWP ( 98.4% 98.6%; 100.0% 100.0%) "Very Good" DOS scanners: AVS ( 97.9% 97.2%; 99.2% 100.0%) FPR ( 95.5% 99.8%; 99.2% 100.0%) INO ( 93,5% 95.2%; 99.2% 100.0%) NVC ( 93.8% 90.8%; 100.0% 100.0%) TBA ( 93.2% 98.7%; 99.2% 100.0%) DRW ( 93.1% 99.3%; 99.2% 100.0%) FSE ( 99.7% 90.1%; 100.0% 100.0%) AVK ( 90.0% 99.7%; 100.0% 100.0%) ************************************************************** Finding #2) The overall virus detection quality of DOS scanners has reached a very acceptable level also for viruses which are not "in-the-wild" but with a bias to better macro virus detection. ************************************************************** Eval #3: In-The-Wild Detection under DOS: ========================================= Concerning "In-The-Wild" viruses, a much more rigid grid must be applied to classify scanners, as the likelyhood is significant that a user may find such a virus on her/his machine. For evaluation, the following grid is applied: - detection rate is 100% : scanner is "perfect" - detection rate is >99% : scanner is "excellent" - detection rate is >95% : scanner is "very good" - detection rate is >90% : scanner is "good" - detection rate is <90% : scanner is "risky" The following DOS products reach 100% both for file and macro virus detection and are rated "perfect" in this category (alphabetically ordered): "Perfect" DOS ITW scanners: AVK (100.0% 100.0%) AVP (100.0% 100.0%) DSS (100.0% 100.0%) FSE (100.0% 100.0%) NVC (100.0% 100.0%) SWP (100.0% 100.0%) Several scanners miss the highest category for ITW-detection marginally; the following scanners musts be rated "excellent": "Excellent" DOS ITW scanners: AVG ( 99.2% 100.0%) AVS ( 99.2% 100.0%) DRW ( 99.2% 100.0%) FPR ( 99.2% 100.0%) INO ( 99.2% 100.0%) TBA ( 99.2% 100.0%) As macro-only products, VET and HMV also reach "perfect" 100% ITW detection. ************************************************************** Finding #3) In-The-Wild detection of best DOS scanners has been improved since last test, esp. concerning macro virus detection. ************************************************************** Eval #4: Evaluation for detection by virus classes under DOS: ============================================================= Some scanners are specialised on detecting some class of viruses (either in deliberately limiting themselves to one class, esp. macro viruses, or as that part is significantly better than other parts). It is therefore worth notifying which scanners perform best in detecting file, boot and macro viruses. Compared to the last test, the number of "excellent" macro virus detectors has significantly grown (as has the class of "good" ones which is not listed here); in contrast, "standard" file viruses (and even more: boot) viruses seem to be comparably less carefully handled in product upgrading. With no product rated "perfect" (=100%), those products with grade "excellent" (>95%) are listed below. 4.1 Detection of file viruses: ------------------------------ "Very Good" DOS scanners: DSS ( 99.9%) AVP ( 99.7%) FSE ( 99.9%) SWP ( 98.4%) NAV ( 98.1%) AVS ( 97.9%) FPR ( 95.5%) 4.2 Detection of macro viruses: ------------------------------- "Perfect" DOS scanners: AVP (100.0%) DSS (100.0%) "Excellent" DOS scanners: FPR ( 99.8%) NAV ( 99.8%) AVK ( 99.7%) RAV ( 99.5%) DRW ( 99.3%) HMV ( 99.0%) TBA ( 98.7%) SCN ( 98.6%) SWP ( 98.6%) VET ( 97.5%) AVS ( 97.2%) INO ( 95.2%) ************************************************************** Finding #4: Specialised scanners (esp. those specialising on macro viruses) are not superior to best overall scanners, even concerning large collections such as VTCs "zoo" testbeds. ************************************************************** Eval #5: Detection of Packed File and Macro Viruses under DOS: ============================================================== Detection of viruses within packed objects becomes essential for on-access scanning, e.g. for incoming email possibly loaded with malicious objects. Before testing on-access detection in email (planned in a forthcoming VTC test), it seems essential to test whether viral objects compressed with given popular methods (PKZIP, ARJ, LHA and, new in this test, RAR) are also detected. Concerning results (see 6BDOSFIL.TXT and 6DDOSMAC.TXT), results are presently rather DISAPPOINTING: Only 9 scanners (out of 20) detected at least ONE file virus packed with at least ONE compressing method. And only 8 scanners (out of 21) detected at least ONE macro virus packed with at least ONE compressing method. The following table list ALL scanners which detect file and macro viruses in objects compressed with AT LEAST TWO packing methods: ----- Packed Files --- ----- Packed Macros ---- %ZIP %LHA %ARJ %RAR %ZIP %LHA %ARJ %RAR -------------------------------------------------- ALL Methods: AVP (93.2 93.2 93.2 93.2) (100.0 99.9 100.0 100.0) FSE (93.2 93.2 93.2 93.2) (100.0 99.9 100.0 100.0) 3 Methods: DSS (99.5 99.5 99.5 0.0) (100.0 99.9 100.0 0.0) AVK (89.6 89.6 89.6 89.6) ( 99.7 99.7 99.7 99.7) 2 Methods: FPR (95.0 0.0 95.0 0.0) ( 99.0 0.0 99.0 0.0) NVC (93.1 0.0 93.1 0.0) ( 90.6 0.0 90.6 0.0) -------------------------------------------------- Remark: Much more data were collected on precision and reliability of virus detection in packed objects. But in the present state, it seems NOT justified to add differentiation to results discussed here. ************************************************************* Finding #5) VERY FEW products have reached an acceptable level of detecting viruses in packed infected objects with given compression methods. Signi- ficant investment of wrok is needed here. ************************************************************** Eval #6: False Positive Detection in Clean Files and Macros: ============================================================ As new category in VTC test "1998-10", a set of clean (and non- malicious) objects has been added to the file and macro virus tes- beds to determine the ability of scanners to avoid False Positive (FP) alarms. This ability is essential for "excellent" and "very good" scanners as there is no automatic aid to customers to handle such cases (besides the psychological impact on customers` work). Therefore, the grid used for grading AV productws must be signifi- cantly more rigid than that one used for detection (see Eval #2). The following grid is applied to classify scanners: - False Positive rate = 0.0%: scanner is graded "perfect" - False Positive rate < 0.5%: scanner is graded "excellent" - False Positive rate < 2.5%: scanner is graded "very good" - False Positive rate < 5.0%: scanner is graded "good enough" - False Positive rate <10.0%: scanner is graded "rather bad" - False Positive rate <20.0%: scanner is graded "very bad" - False Positive rate >20.0%: scanner is graded "useless" Regarding the ability of scanners to avoid FP alarms, the following AV products running under DOS reported NO SINGLE False Positive alarm both in file and macro zoo testbeds and are therefore rated "perfect": FP-avoiding "perfect" DOS scanners: AVP, DSS, NVC, RAV and SCN. Several more DOS scanners gave NO FP alarm on clean files or macros: Perfect FP-avoidance on DOS clean file testbed: AVP, AVK, DSS, FPR, FSE, NVC, RAV, SCN and SWP Perfect FP-avoidance on DOS clean macro file testbed: ANT, AVP, AVS, DSS, NVC, RAV, SCN and TBA Comparing related results with behaviour of 32-bit scanner engines aand esp. using results produced under Win-NT, there is just ONE AV product which avoids ANY FP alarm in both clean file and macro objects: FP-avoiding "perfect" Win-NT scanner: DSS Several more scanners also gave NO FP alarm on clean files or macros: Perfect FP-avoidance on Win-NT clean file testbed: AVK, AVP, DSS, FSE, NVC, PAV, RAV, SCN and SWP Perfect FP-avoidance on Win-NT clean macro file testbed: AVS, DSS, FMA(FPR), FWN, IBM and TBA. Presently, only ONE AV product avoids ANY False Positive alarm both for clean file and macro objects under both DOS and Win-NT: Overall perfect FP-avoiding scanner: DSS ************************************************************* Finding #6.1) VERY FEW products reliably avoid ANY False Positive alarm on clean file and macro objects, both under DOS and Win-NT. There are several products which have excellent FP avoidance in one category (either clean file or macro objects). #6.2) Only ONE product - DSS - didnot give ANY FP alarm for both categories (file, macro objects) under BOTH DOS and Win-NT. #6.3) AV producers should intensify work to avoid FP alarms. ************************************************************** Eval #7: Evaluation of File and Macro Malware detection: ======================================================== Since test "1997-07", VTC tests also the ability of AV products to detect non-viral malware. An essential argument for this category is that customers are interested to be warned and protected not only about viruses but also about other malicious objects such as trojans etc, the payload of which may be disastrous to their work. Regrettably, consciousness of AV producers to protect their users against related threats is still underdevelopped. Manifold arguments are presented why AV products are not the best protection against non-viral malware; from a technical point, these arguments may seem conclusive but at the same time, almost nothing is done to support customers with adequate AntiMalware software. Some AV producers have asked NOT to publish Malware detection results of VTC tests. Until this test, it was VTCs policy to follow such requests. Consequently, results for such AV producers are not published. This policy will no longer be followed by VTC; any producer requesting non-publication of malware results will no longer remain in our tests, starting with VTC test "1999-02". Several scanners are indeed able to detect also non-viral malware. The following grid is applied to classify detection of file and macro malware: - detection rate =100% : scanner is "perfect" - detection rate > 90% : scanner is "excellent" - detection rate of 80-90% : scanner is "very good" - detection rate of 60-80% : scanner is "good enough" - detection rate of < 60% : scanner is "not good enough" In comparison with last VTC test (where no product was rated as "excellent" in this category) under DOS, there are now several scanners detecting over 90% of malicious non-viral software in VTCs (partial) malware testbeds. As this applies to both DOS and Win-NT, the following table list scanners with related performance under BOTH operating systems: ===== Malware Detection ===== = under DOS == = under W-NT = (File/Macro-mw;File/Macro-mw) ------------------------------ "Excellent" DOS scanners: DSS (98.4% 100.0%; 98.4% 100.0%) AVP (94.5% 96.4%; 94.5% 96.4%) FSE (94.5% 96.4%; 93.5% 100.0%) AVK (93.1% 96.4%; 93.1% 96.4%) Moreover, the following scanners reach 90%-detection either for file or macro malware for at least one operating system (DOS or Win-NT): "Excellent" scanners in at least one category/under one OS: ACU, AVK, AVP, AVS, DRW, DSS, FSE, FPR(FMA), FWN, INO, PAV, RAV and TBA. ************************************************************** Finding #7.1: Obviously, several AV producers have worked very hard to protect their customers also from non-viral malware. 4 scanners reached a con- siderable level (>90%) for file and macro malware detection under both DOS and Win-NT. #7.2: With continuous growth of malware testbeds, AV producers are well advised to improve their products also in this area. ************************************************************** Eval #8: Overall virus detection rates under Windows-95: ======================================================== The number of scanners running under Windows 95 is slowly growing (last test: 14 products; this test: 15 products). The following table compares results of file and macro virus detection under Windows 95 in last 3 tests ("1997-07","1998-02" and actual test), including relative improvement (DELTA) since last test: --------------------------------------------------------------- Scan === File Virus Detection === === Macro Virus Detection == ner 97/07 98/02 98/10 Delta 97/07 98/02 98/10 Delta --------------------------------------------------------------- ANY - - - - 72.3 - 72.1 - ANT 88.9 85.0 91.3 +6.3 92.2 - 95.7 - AVK - - 99.6 - - - 99.6 - AVS - 97.4 96.6 -0.8 - 91.9 96.7 +4.8 AVG - 84.8 - - 70.5 - - - AVP 97.7 97.2 99.9 +2.2 94.8 99.9 96.7 -3.2 DSS 99.5 99.9 99.9 +0.0 95.3 100.0 100.0 +0.0 FPR - - - - 89.3 - 92.4 - FSE - - - - 99.9 - 100.0 - IBM 95.2 96.5 92.8 -3.7 92.9 99.8 94.5 -5.3 INO - 91.8 92.4 +0.6 - 89.7 88.1 -1.6 IRS - 96.2 96.7 +0.5 - 99.0 99.0 +0.0 NAV 86.5 97.1 - - 95.6 98.7 95.3 -3.4 NVC 89.6 93.9 93.6 -0.3 96.6 99.2 - - PAN - 69.8 - - 68.5 - - - PAV 97.7 99.4 98.4 -1.0 93.5 100.0 99.5 -0.5 PER - - - - 96.3 91.0 - - RAV - - 84.9 - - - 13.9 - SCN 93.8 91.6 87.8 -3.8 97.6 99.1 98.9 -0.2 SWP 94.5 96.8 98.4 +1.6 89.1 98.4 98.6 +0.2 TBA 95.2 - 92.6 - 96.1 - 98.7 - VBS - - - - - - 88.7 - VDS 52.9 - - - - - - - VET 64.4 - - - 93.5 - - - VSA - - - - - 84.4 - - VSW - - - - - 84.4 - - --------------------------------------------------------------- Mean: 88.0% 92.7% 94.6% - 90.2 95.4% 90.5% - --------------------------------------------------------------- Generally, the ability of Win-95 scanners to detect file viruses "in the mean" has slightly improved (+1,9%) in comparison with last test. At the same time, the "mean" detection of macro viruses has relatively decreased (different from DOS scanners!) at rate of 4.9%. Indeed, several scanners performing well in VTC test "1998-02" are now showing less macro virus detection rates. The same grid as for the DOS classification is applied to classify scanners according to their ability to detect file and macro viruses under Windows 95. The following list indicates those scanners under Windows 95 graded into the upper category "excellent" upon detecting file and macro viruses: "Excellent" Windows 95 scanners: DSS (99.9% 100.0%) AVK (99.6% 99.6%) PAV (98.4% 99.5%) AVP (99.7% 99.6%) SWP (98.4% 98.6%) IRS (96.7% 99.0%) AVS (96.6% 96.7%) Moreover, SCN (98,9%) and TBA (98,7%) reached "excellent" performance level on macro virus detection. ************************************************************** Finding #8: With still few scanners working under Windows 95, a level of excellency is reached by only few AV products. Indeed, the mean detection level of file viruses improved (now at 94.6%) whereas the mean level of macro virus detection was significantly reduced (-4.9% to now only 90.5%). ************************************************************** Eval #9: Overall virus detection rates under Windows-NT: ======================================================== The number of scanners running under Windows NT is still small, though growing. Significantly less products were available for these tests, compared with the traditional DOS scene. The following table compares results of file and macro virus detection under Windows-NT in test "1998-10" as compared with last one (1998-02): --------------------------------------------------------------- Scan === File Virus Detection === === Macro Virus Detection == ner 97/07 98/02 98/10 Delta 97/07 98/02 98/10 Delta --------------------------------------------------------------- ANT 88.9 69.2 91.3 +22.1 92.2 - 85.7 - ANY - - 69.7 - - - 70.5 - AVK - - 99.6 - - - 99.6 - AVP - - 83.7 - - - 100.0 - AVS - 97.4 96.6 -0.8 - 91.9 97.2 +5.3 AW - 56.4 - - - - 61.0 - DSS 99.6 99.7 99.9 +0.2 99.0 100.0 100.0 +0.0 FPR(FMA) - 96.1 - - - 99.9 99.8 -0.1 FSE - 85.3 - 99.8 - 99.9 - - FWN - - - - - - 99.6 - HMV - - - - - - 99.0 - IBM 95.2 95.2 77.2 -18.0 92.9 92.6 98.6 +6.0 INO - 92.8 - - - 89.7 - - IRS - 96.3 - - - 99.1 - - IVB - - - - - - 92.8 - NAV 86.5 97.1 - - 95.6 98.7 99.9 +1.2 NVC 89.6 93.8 93.6 -0.2 96.6 99.2 - - PAV 97.7 98.7 98.4 -0.3 93.5 98.8 99.5 +0.7 RAV - 81.6 84.9 +3.3 - 98.9 99.5 +0.6 PCC 63.1 - - - - 94.8 - - PER - - - - - 91.0 - - SCN 94.2 91.6 71.4 -20.2 97.6 99.1 97.7 -1.4 SWP 94.5 96.8 98.4 +1.6 89.1 98.4 97.5 -1.1 TBA - 93.8 92.6 -1.2 96.1 - 98.7 - TNT - - - - - - 44.4 - VET 64.9 - - - - 94.0 - - VSA - 56.7 - - - 84.4 - - --------------------------------------------------------------- Mean: 87.4% 88.1% 89.0% - 94,7% 95,9% 91.6% --------------------------------------------------------------- Generally, the ability of Win-NT scanners to detect file viruses "in the mean" has slightly improved (+0,9%) in comparison with last test. At the same time, the "mean" detection of macro viruses has relatively decreased (different from DOS scanners!) at rate of 4.3%. Indeed, several scanners performing well in VTC test "1998-02" have now yielded lesser macro virus detection rates. Btw: same tend is also observable with Win-95 scanners. The same grid as for the DOS classification is applied to classify scanners according to their ability to detect file and macro viruses under Windows NT. The following list indicates those scanners under Windows NT graded into the upper category "excellent" upon detecting file and macro viruses: "Excellent" Windows-NT scanners: DSS (99.9% 100.0%) AVK (99.6% 99.6%) PAV (98.4% 99.5%) SWP (98.4% 97.5%) AVS (96.6% 97.6%) Moreover, besides those mentioned above, FSE (99.8%) reached "excellent" performance level on file virus detection. And the following scanners reached "excellent" performance level on macro virus detection under Windows-NT (in addition to those mentioned above): AVP (100.0%), NAV (99.9%), FPR/FMA (99.8%), FWN (99.6%), RAV (99.5%), HMV (99.0%), TBA (98.7%), IBM (98.6%) and SCN (97.7%) ************************************************************* Finding #9.1: With a growing number of scanners working under Windows-NT, some have reached a level of "excellency" (although not yet "perfect"), both in detecting file and macro viruses. Some more scanners have also excellent results either in detection of file or macro viruses. Finding #9.2: Generally, the "mean" detection rate of Win-NT scanners for file viruses dropped slightly but dropped seriously for macro viruses. Concerning macro virus detection, Win-NT scanners behave "in the mean" significantly less favourable compared to DOS scanners. ************************************************************** Eval #9: File/Macro Virus detection under 32-bit engines: ========================================================= Concerning 32-Bit engines as used in Windows 95/98 and Windows NT, it is interesting to test the validity of the hypothesis that related engines produce same detection and identification quality. For details see 6HCOMP32.TXT. When comparing results from related tests (as far as applicable), it is interesting to observe that identical results are presently achieved only for very few (6 out of 19) products of equal quality for zoo file viruses whereas few more (11 out of 19) equally well detect ITW file viruses under all 32-Bit engines. Concerning macro viruses, results are more favourable as 18 (out of 24) products detect ITW macro viruses equally well whereas as few as 6 (out of 24) products detect zoo macro viruses with equal quality. Indeed, only 2 products were found to perfectly equally detect both in zoo and ITW file and macro virus testbeds: AVP and TBAv. Few scanners achieve results which are not too far apart from perfect quality. ************************************************************* Finding #9: The assumption that 32-bit engines in scanners produce the same detection rate for different instantiations of 32-bit operating systems (esp. concerning Windows-95, Windows-98 and Windows-NT) holds only for 2 scanners (AVP and TBA) is NOT correct, at least presently. ************************************************************** Eval #10: Evaluation for malware virus detection under Windows 95/NT: ===================================================================== With Windows 95 and Windows-NT often used for downloading potentially hazardous objects from Internet, it is interesting to measure the ability of AntiVirus products to also act as AntiMalware products. The following table lists those products which reached a "very good" (>90%) detection threshhold at for both categories of file and macro malware, both under Windows 95 and Windows NT: Detection of File Malware Detection of Macro Malware Scanner Win95 WinNT Win95 WinNT ---------------------------------------------------------------------- AVK 93.1% 93.1% 96.4% 96.6% AVP 94.5% 94.5% 96.4% 96.4% DSS 98.0% 98.4% 100.0% 100.0% FSE 93.6% 93.5% 100.0% 100.0% PAV 91.6% 91.6% 96.4% 96.4% --------------------------------------------------------------------- Moreover, the following additional products also reached a "very good" level of macro malware detection under both Win-95 and Win-NT: FPR/FMA (98.2% 98.2%), FWN (96.4% 96.4%), AVS (93.7% 93.7%) and TBA (91.9% 91.0%) Evidently, some AV products are able to help protecting users by detecting file and macro-related malware at some relevant level. Fortunately, related products also show good to excellent results in Detecting viral malware. ************************************************************** Finding #10: Some AntiMalware producers help customers detecting also non-viral malware under 32-bit operating systems, esp. Win-95 and Win-NT. But most AV-products are far from supporting their customers against related threats. **************************************************************