========================================= File 7 mEVAW32.TXT ----------------------------------------- Comparison of detection results under W32 (W2k and WXP) platforms ========================================= Formatted with non-proportional font (Courier) ****************************************************************** Content of this file: ****************************************************************** Comp ALL: Comparison of Detection rates under ALL platforms: ****************************************************************** Background of this test Test Hypothesis: Excellency in engine and database design, implementation and maintenance Eval W32-Harmonicity: Equality of results for all W32 platforms Grading W32-Harmonicity: Grading of W32 harmonical products ****************************************************************** This part of VTC "2004-07" test report evaluates the detailed results as given in section (file): 6ncmpW32.TXT Comparison of Detection Quality of Scanning Engines under W32 (Win-2000 / Win-XP) platforms Background of this test: ======================== With the fast deployment of new versions of Microsoft Windows-32 (in past 5 years from W-NT to W-95, W-98, W-2000 and W-XP (and beyond), customers needing protection and producers of security-enhancing software (AntiVirus and AntiMalware) can only cope with the pace when they essentially reuse engines prepared for previous W32 platforms and simply "adapt" them to the intrinsics of the new platforms. Otherwise, "rewriting" the resp. software would consume too much time and efforts, and customers would receive "adapted" products only with some delay. AV/AM testers cannot determine the characteristics of the algorithms in scanning engines, either in following legal objectives (which, in most Copyright laws, prohibit reverse-engineering of proprietory code, except for specific reasons such as collecting evidence for a court case or teaching related techniques, as in Hamburg university IT Security curriculum), or for complexity of related code (and in many cases, for unsufficient p rofessional knowledge of testers). It is therefore worthwhile to analyse whether those AV/AM products versions of which are available for all W32 platforms behave EQUALLY concerning detection and identification of viral and malicious code. Test Hypothesis: "W32-harmonical" behaviour of W32 products: ============================================================ We assume that those products which participate for all W32 platforms in this test (W98 and W2k) for ALL categories shall yield IDENTICAL results (argument for this assumption: likelihood of reuse of engines running on the same platform). We call product behaviour following this hypothesis "W32-harmonical". Eval W32-Harmonicity: Equality of results for all W32 platforms: ================================================================ In comparison with last VTC test, not much progress can be reported. Equal detection this test last test -----------------------------+-----------+------------ of zoo file viruses: 21 (of 25) 9 (of 18) of zoo infected files: 20 (of 25) 7 (of 18) of ITW file viruses: ALL (of 25) 17 (of 18) of ITW infected macro files: ALL (of 25) 17 (of 18) of zoo file malware: 20 (of 25) 13 (of 18) Equal detection this test last test -----------------------------+-----------+------------ of zoo macro viruses: 22 (of 25) 16 (of 18) of zoo infected macro objects: 22 (of 25) 16 (of 18) of ITW macro viruses: ALL (of 25) ALL (of 18) of ITW infected macro files: ALL (of 25) ALL (of 18) of zoo macro malware: ALL (of 25) 15 (of 18) Equal detection this test last test -----------------------------+-----------+------------ of zoo script viruses: 24 (of 25) 16 (of 18) of zoo script viral objects: 24 (of 25) 15 (of 18) of ITW script viruses: ALL (of 25) ALL (of 18) of ITW script viral objects: ALL (of 25) ALL (of 18) of ITW script malware: ALL (of 25) 16 (of 18) Findings W32-Harmonicity: -------------------------------------------------------------- Concerning detection of FILE viruses: many though not all (20 of 25) products behave "W32-harmonically" in all categories: ANT,AVA,AVG,AVK,BDF,CMD,FIR,FPR,FSE,GLA, INO,NAV,PAV,PER,PRO,RAV,SCN,SWP,VBR,VSP Concerning file malware detection, also many though not all (20 of 25) products behave "W32-harmonically" in all categories: ANT,AVG,AVK,BDF,CMD,DRW,FIR,FPR,FSE,GLA, IKA,INO,NAV,PAV,PER,PRO,SCN,SWP,VBR,VSP -------------------------------------------------------------- Concerning detection of MACRO viruses: many though not all (22 of 25) products behave "W32-harmonically" in all categories: ANT,AVA,AVG,AVK,AVP,BDF,CMD,DRW,FIR,FPR,FSE, GLA,INO,NAV,NVC,PAV,PER,PRO,RAV,SCN,SWP,VSP Concerning macro malware detection, ALL 25 products behave in W32-harmonical form: ANT,AVA,AVG,AVK,AVP,BDF,CMD,DRW,FIR,FPR,FSE,GLA, IKA,INO,NAV,NVC,PAV,PER,PRO,QHL,RAV,SCN,SWP,VBR,VSP -------------------------------------------------------------- Concerning detection of SCRIPT viruses: ALMOST ALL (24 of 25) products behave "W32-harmonically" in all categories: ANT,AVG,AVK,AVP,BDF,CMD,DRW,FIR,FPR,FSE,GLA,IKA, INO,NAV,NVC,PAV,PER,PRO,QHL,RAV,SCN,SWP,VBR,VSP Concerning script malware detection: ALL 25 products behave in W32-harmonical form: ANT,AVA,AVG,AVK,AVP,BDF,CMD,DRW,FIR,FPR,FSE,GLA, IKA,INO,NAV,NVC,PAV,PER,PRO,QHL,RAV,SCN,SWP,VBR,VSP -------------------------------------------------------------- Conclusion: regarding economy of AV/AM testing, it seems sufficient to include only AV/AM products at the upper end of the W32 development chain (presently Windows XP). Grading W32-Harmonicity: Grading of W32 harmonical products: ============================================================ The following grid is used to grade W32 products concerning their ability for IDENTICAL detection for ALL categories on ALL W32 platforms: A "perfect" W32-harmonical AV product will yield IDENTICAL results for all categories (file, macro and script viruses). (Assigned value: 5). A "perfect" W32-harmonical AM product will be a perfect AV product and yield IDENTICAL results for all categories (file, macro and script malware). (Assigned value: 2). ANT,AVG,AVK,BDF,CMD,FIR,FPR,FSE,GLA, IKA,INO,NAV,PAV,PER,PRO,SCN,SWP,VSP Grading W32-harmonical AntiVirus products: =========================================================== Grade: "Perfect" W32-harmonical detection: ANT,AVG,AVK,BDF,CMD,FIR,FPR,FSE, GLA,IKA,INO,NAV,PAV,PER,PRO,SCN,SWP,VSP =========================================================== Grading W32-harmonical AntiMalware products: =========================================================== Grade: "Perfect" W32-harmonical detection: ANT,AVG,AVK,BDF,CMD,FIR,FPR,FSE, GLA,IKA,INO,NAV,PAV,PER,PRO,SCN,SWP,VSP =========================================================== ************************************************************* "Perfect" W32-harmonical AntiVirus products: 1st place: ANT,AVG,AVK,BDF,CMD,FIR,FPR,FSE, GLA,IKA,INO,NAV,PAV,PER,PRO,SCN,SWP,VSP (5 points) ************************************************************* "Perfect" W32-harmonical AntiMalware products: 1st place: ANT,AVG,AVK,BDF,CMD,FIR,FPR,FSE, GLA,IKA,INO,NAV,PAV,PER,PRO,SCN,SWP,VSP (7 points) *************************************************************